6aaff6211f
feat: use union member selector for authn config schema
2023-02-04 07:56:29 +01:00
15035f7eb0
refactor: remove lazy type for authentication
...
The idea of using lazy type for authentication was to make the
config check easy to extend at runtime. However the reality
is: it's overly complicated.
It's more likely that people will just continue to implementing
the auth hook callbacks instead of injecting config schema
at runtime.
2023-02-04 07:56:29 +01:00
ccb0c7ebaa
Merge remote-tracking branch 'origin/release-50' into 0202-merge-release-50-back-to-master
2023-02-03 16:12:11 +01:00
d45a0cb2ee
chore: remove <<>> when logging username
2023-02-03 16:54:18 +08:00
1e0fe270bc
chore: bump emqx app vsn to 5.0.17
2023-02-02 20:32:09 +01:00
94768c9f44
Merge remote-tracking branch 'origin/master' into 0202-merge-release-50-back-to-master
2023-02-02 20:21:26 +01:00
5fe3a0b1b3
Merge pull request #9898 from zmstone/0202-upgrade-to-dashboard-e1.0.3
...
chore: upgrade dashboard to v1.1.7 for ce and e1.0.3 for ee
2023-02-02 17:41:52 +01:00
42462db8b8
chore: bump version to v5.0.16 and e5.0.0
2023-02-02 17:26:00 +01:00
3d6756cbe8
test: fix addrinuse errors in listeners API test SUITE
2023-02-02 16:37:56 +01:00
a71d983ff8
test: fix assertMatch patter to make rebar3 fmt work
2023-02-02 16:37:45 +01:00
7ea140599a
Merge pull request #9894 from id/ci-always-run-static-checks
...
ci: always run static_checks
2023-02-02 16:33:19 +01:00
feca4cc0a5
Merge pull request #9892 from zmstone/0202-docs-cosmetic
...
0202 docs cosmetic
2023-02-02 15:43:58 +01:00
85fb53dcf0
refactor: use compile switch to diff functions between ce and ee
2023-02-02 15:15:31 +01:00
0d7cd2f0c5
docs: Update apps/emqx_conf/i18n/emqx_conf_schema.conf
...
Co-authored-by: Ivan Dyachkov <dev@dyachkov.org>
2023-02-02 14:20:07 +01:00
980ba2fd35
docs: Update apps/emqx/i18n/emqx_schema_i18n.conf
...
Co-authored-by: Ivan Dyachkov <dev@dyachkov.org>
2023-02-02 14:19:37 +01:00
0e299642cb
chore: function renamed but forgot to update xref excemption
2023-02-02 14:16:23 +01:00
58627b7958
chore(emqx_resource_manager): ignore unused return value for dialyzer
2023-02-02 14:11:12 +01:00
9a9943d35d
chore(emqx_mgmt_cache): ignore unused return value for dialyer
2023-02-02 14:11:12 +01:00
1c748070c1
fix(emqx_connector_mqtt_worker): publish_async can only return ok # Please enter the commit message for your changes. Lines starting
2023-02-02 14:11:12 +01:00
c0d478bd41
fix(buffer_worker): type spec
2023-02-02 14:11:12 +01:00
ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async
...
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
2023-02-02 21:08:01 +08:00
d5c482b0b0
docs: remove timer unit from description
...
the user input has time unit. e.g. "5s" for 5 seconds etc.
2023-02-02 13:49:20 +01:00
5dbee76af4
Merge pull request #9890 from zmstone/0202-fix-kafka-status-check
...
0202 fix kafka status check
2023-02-02 13:48:44 +01:00
031b36da32
feat(banned): filter out banned messages from mqueue
2023-02-02 20:20:52 +08:00
2b308af354
Merge pull request #9889 from keynslug/fix/EMQX-8857/force-clean-start-mqtt-bridge
...
fix(mqtt-bridge): stop respecting `clean_start` config parameter
2023-02-02 12:55:58 +01:00
9864587389
fix: send to buffer-supported connector even when disconnected
2023-02-02 12:04:17 +01:00
13ef30c46c
Merge pull request #9884 from savonarola/resource-fixes
...
fix(resources): fix resource lifecycle
2023-02-02 12:02:34 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
2cf193e2fd
Merge pull request #9839 from kjellwinblad/kjell/fix/Authorization_header_log_leak_webhook/EMQX-8791
...
fix: Authorization header leak in log entries for webhook
2023-02-02 10:38:42 +01:00
1c9035d24c
test: remove async from redis ct
2023-02-02 17:37:18 +08:00
22cc1cc745
fix: make spell_check happy
2023-02-02 17:37:18 +08:00
f8936013b7
chore: replace async with sync
2023-02-02 17:37:18 +08:00
22c3f50020
fix: add query_mode_sync_only for mysql pgsql redis mongodb bridge
2023-02-02 17:37:18 +08:00
ca5c192f4b
Merge pull request #9882 from fwup/fix/no-mqtt-bridge-middleman
...
refactor(mqtt-worker): avoid unnecessary abstraction
2023-02-02 13:11:31 +04:00
e5bc787467
Merge pull request #9872 from zmstone/0131-improve-help-text
...
docs: improve docs for system monitoring config
2023-02-02 14:45:03 +08:00
1a90c1654c
chore: bad typo
2023-02-02 11:43:04 +08:00
6fd30a7e81
docs: improve docs for system monitoring config
2023-02-01 20:21:08 +01:00
14f528cc86
fix(resources): fix resource lifecycle
...
* do not resume all buffer workers on successful healthcheck
* do not pass undefined state to resource healthcheck callback
2023-02-01 18:26:13 +02:00
ec3031271b
Merge pull request #9883 from zmstone/0201-upgrade-enterprise-dashboard-to-e1.0.2
...
0201 upgrade enterprise dashboard to e1.0.2
2023-02-01 16:26:21 +01:00
32d18ec594
Merge pull request #9878 from zmstone/0201-upgrade-ehttpc-0.4.6
...
fix(ehttpc): fix crash when webhook bridge uses `hash` pool type.
2023-02-01 15:00:28 +01:00
5fd7f65a1f
test(bufworker): make testcase simpler to follow
...
The confusion was due to the fact that subsequent query was missing
`async_reply_fun` and thus, was not accumulating in the results.
2023-02-01 16:52:47 +03:00
38ef37d109
chore: bump release version to e5.0.0-rc.4
2023-02-01 14:52:08 +01:00
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly
2023-02-01 16:28:16 +03:00
5ebceb20d2
test(mqtt-bridge): also test reconfiguration
2023-02-01 16:23:58 +03:00
ad88938d34
refactor: reuse some parts of test code for brewity
2023-02-01 16:22:19 +03:00
f0395be383
refactor(mqtt-worker): avoid unnecessary abstraction
...
So the code is easier to follow.
2023-02-01 14:46:26 +03:00
13511d2782
Merge pull request #9856 from zhongwencool/alarm-percent-decimals
...
fix: /api/nodes is timeout when emqx in high load
2023-02-01 19:40:53 +08:00
979e9804be
Merge pull request #9879 from keynslug/fwup/fix/buffer-worker-testcase
...
test(bufworker): fix testcase flapping due to data races
2023-02-01 15:33:59 +04:00
ff473e0f1b
test(bufworker): fix testcase flapping due to data races
2023-02-01 12:57:46 +03:00
96a18e7105
chore: upgrade to hocon 0.35.3
2023-02-01 10:52:01 +01:00
6382915036
chore: repalce fresh by refresh
2023-02-01 17:36:30 +08:00
9f6b6cedc6
fix(webhook): pick worker according to pool type
2023-02-01 09:53:25 +01:00
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover
...
fix: fix redis cluster resource recovery
2023-02-01 10:38:52 +02:00
f4381d90ca
refactor: http cliet should not need to retry HTTP requests for bridge
2023-02-01 09:30:30 +01:00
6ca816f7a4
docs: remove impl details from user doc
2023-02-01 09:30:10 +01:00
3c4d09a752
fix: get_memory_cache return {ok, Mem} | stale
2023-02-01 16:00:55 +08:00
ced55719ef
chore: only run t_sys_mem_check ct in linux
2023-02-01 14:47:41 +08:00
56b9238645
fix: only cache sys_memory in linux
2023-02-01 12:21:34 +08:00
95ef1e806c
chore: don't crash when add_handler failed
2023-02-01 11:24:27 +08:00
9cbad5297c
fix: primary logger level not update
2023-02-01 11:16:30 +08:00
71f00f2962
test: ctl suite failed
2023-02-01 10:07:22 +08:00
7a9f477267
feat: use emqx_mgmt_cache to cache sys_memory
2023-02-01 10:07:22 +08:00
c2bdb9faa7
test: multiple_nodes case failed
2023-02-01 10:07:22 +08:00
b6e6315b50
feat: change loads from string to float
2023-02-01 10:07:22 +08:00
0b19be074c
feat: cache OTP_VERSION in persistent_term
2023-02-01 10:07:22 +08:00
6162f90610
fix: don't crash when OTP_VERSION file is missing
2023-02-01 10:07:22 +08:00
5783127c30
test: cpu_sup:load mock test
2023-02-01 10:07:22 +08:00
2d67bb3fb6
fix: /api/nodes is timeout if emqx in high load
2023-02-01 10:07:22 +08:00
7e8253e3af
chore: bump version to e5.0.0-rc.3
2023-01-31 20:45:28 +01:00
78028a7fcf
Merge pull request #9876 from zmstone/0131-avoid-persisting-logging-configs-if-no-change
...
0131 avoid persisting logging configs if no change
2023-01-31 20:43:32 +01:00
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback
...
feat(mqtt-bridge): avoid middleman process
2023-01-31 19:12:18 +01:00
ddfbc0a19f
refactor: refresh logger config before starting listeners
2023-01-31 18:50:16 +01:00
605d9972e4
fix(config): avoid calling emqx:update_config/2 for logger refresh
...
Prior to this fix, whatever configured for loggers are persisted to
cluster-override.conf.
This may cause troulbe for users who changes boot mode.
For example if the node is once started in console mode,
the logging config is persisted, with console enabled, but file
disabled.
Then if the user decide to start in daemon mode, all the logs
will silently go to erlang.log.N and emqx.log.N will be empty.
After this fix, only changes really made into cluter-override.conf
will take effect.
2023-01-31 18:50:16 +01:00
b0ac924ca9
refactor: less copy-paste
2023-01-31 18:40:55 +01:00
3d59be6ab6
Merge pull request #9871 from lafirest/fix/authz_rules
...
feat(authz): allow the placeholder to be anywhere in the topic for authz rules
2023-02-01 00:12:06 +08:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
c76311c9c3
fix(buffer): count inflight batches properly
2023-01-31 18:30:42 +03:00
9f4c36ecbc
chore: bump version && update changes
2023-01-31 23:30:22 +08:00
c034cbf6de
feat(authz): allow the placeholder to be anywhere in the topic for authz rules
2023-01-31 23:30:12 +08:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
fce1e74c3d
fix(connector): fix redis cluster resource recovery
2023-01-31 16:55:05 +02:00
f6dafc20ea
feat: export API to read cluster and local overrides
2023-01-31 14:20:34 +01:00
ade5316419
Merge pull request #9802 from sstrigler/EMQX-8469-enable-proxy-protocol-support-in-emqx-dashboard-schema
...
feat: support HAProxy protocol for dashboard API
2023-01-31 14:19:56 +01:00
3786bb8086
Merge pull request #9824 from olcai/internal-error-api-get-topic
...
fix(emqx_management): handle multiple routes in topics/{topic} API
2023-01-31 13:50:42 +01:00
4d146c521b
fix(mqtt-bridge): ensure proper feedback on async forwards
...
So that buffer worker would notice a connection loss in time, and
recycle inflight messages subsequently.
2023-01-31 11:09:39 +03:00
35c429ef1d
refactor: drop a couple of unused macros / includes
2023-01-31 11:09:39 +03:00
b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback
...
0127 refactor buffer worker no need to keep request for reply callback
2023-01-31 08:44:01 +01:00
5b3a77e3c7
test(emqx_management): fix flaky route handling in get topic test
...
This reworks a test case to use a second slave node. This ensures that
an added route is permanently in the routing table. The old version
reverted the routing table quickly since the node name given wasn't a
real node.
2023-01-31 08:28:55 +01:00
03cabf6b26
chore: bump app VSN
2023-01-31 08:27:54 +01:00
33e011aff5
fix(emqx_management): handle multiple routes in topics/{topic} API
...
The topics/{topic} API endpoint would return 500 - Internal Error if a
topic had multiple routes. This is now fixed by returning a list of
routes.
2023-01-31 08:27:33 +01:00
1a1c3afa81
Merge pull request #9852 from savonarola/fix-redis-desc
...
Fix Redis configuration field description
2023-01-30 23:26:37 +02:00
3ce6cbcd6a
chore: bump vsn
2023-01-30 18:09:10 +01:00
0b1483040a
test: add actual test for haproxy
2023-01-30 18:07:06 +01:00
e288424924
Merge pull request #9864 from lafirest/fix/exclusive-unsub-5.0
...
fix: the exclusive topics aren't removed when the session has already been cleaned
2023-01-31 00:33:45 +08:00
0c4134c423
test: add unit test case for redact function in http connector
2023-01-30 11:51:28 +01:00
27881064dc
fix: increase dropped.queue_full by number of messages
2023-01-30 11:37:35 +01:00
3e6cc40525
Merge pull request #9843 from lafirest/fix/speed_up_test
...
fix: speed up banned test cases
2023-01-30 18:17:27 +08:00
3bdffca488
fix: the exclusive topics aren't removed when the session has already been cleaned
2023-01-30 18:17:00 +08:00
bb636394e1
chore: add debug log for raw data
2023-01-30 16:35:06 +08:00
b73d11675e
chore: log the bad mqtt packet(frame error)
2023-01-30 15:05:49 +08:00
3d07271ea5
fix: crash when mfa not found
2023-01-30 14:30:20 +08:00
25090563af
chore: use brackets to wrap the mqtt packet when logging
2023-01-30 12:29:00 +08:00
ce32ea7334
feat: Make the log output format order fixed
2023-01-30 12:18:22 +08:00
6cbad047cd
fix: don't log CONNECT twice when debug
2023-01-30 12:16:17 +08:00
d47941601d
refactor(buffer_worker): rename trace points
2023-01-28 11:52:11 +01:00
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
...
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
fc38ea9571
refactor(buffer_worker): do not keep request body in reply context
...
the request body can be potentially very large
the reply context is sent to the async call handler and kept
in its memory until the async reply is received from bridge
target service.
this commit tries to minimize the size of the reply context
by replacing the request body with `[]`.
2023-01-27 17:12:55 +01:00
72f39b9b72
fix(docs): correct Redis conf field description
2023-01-27 17:39:16 +02:00
578271ea3d
refactor: use lists:map instead of lc for safty
2023-01-27 15:15:46 +01:00
f793807bc1
refactor(buffer_worker): rename function
...
batch_reply_after_query to handle_async_batch_reply
2023-01-27 15:04:28 +01:00
262c3a2869
refactor(buffer_worker): rename function
...
from reply_after_query to handle_async_reply
2023-01-27 15:03:18 +01:00
6a58bafcb0
chore: bump release version to e5.0.0-rc.2
2023-01-27 14:38:21 +01:00
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker
...
EMQX 8774 failure to handle timeout error in resource worker
2023-01-27 14:36:44 +01:00
514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak
...
0127 fix influxdb bridge atom leak
2023-01-27 14:30:20 +01:00
96ed725a55
Merge pull request #9849 from zmstone/0127-refactor-buffer-worker-simplify-caller-reply
...
0127 refactor buffer worker simplify caller reply
2023-01-27 14:06:56 +01:00
c47be57c59
fix(bridge): ensure all bridge resources are stopped before app stop
2023-01-27 12:39:05 +01:00
d53106145f
fix: stop resource when resource manager terminates
2023-01-27 12:39:05 +01:00
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching
...
feat(redis): disable batching in redis_cluster bridges
2023-01-27 15:27:57 +04:00
7005b71ddf
style: fix typo in comment
2023-01-27 11:43:51 +01:00
2d62de5188
test: fix expected result from timeout error
2023-01-27 11:43:48 +01:00
a180bd9aa5
fix: catch error, not exit
2023-01-27 11:40:06 +01:00
b7e3f9d5a6
fix: try-case-of rather than try-of
...
try-of catches only what happens within but not after
2023-01-27 11:40:06 +01:00
db2f631a8a
refactor(buffer_worker): simplify caller reply
2023-01-27 11:33:45 +01:00
965236c888
Merge pull request #9845 from zmstone/0126-reply-caller-for-buffer-overflow-queue-items
...
0126 reply caller for buffer overflow queue items
2023-01-27 11:32:59 +01:00
d4fab92b72
refactor(buffer_worker): no need to keep request for REPLY macro
2023-01-27 10:41:30 +01:00
71f996b9d5
refactor(mqtt-bridge): unwrap single statem actions
...
So that the code would be easier to follow and harder to break.
Also drop a couple of unused macrodefs.
2023-01-26 21:13:18 +03:00
c95f979413
fix(mqtt_bridge): use correct gen_statem reply action
2023-01-26 21:12:05 +03:00
1f799dfd59
fix: reply with {error, buffer_overflow} when discarded
2023-01-26 17:15:36 +01:00
ed28789164
refactor(buffer_worker): no need to return after collect into buf queue
2023-01-26 14:50:40 +01:00
25b4821adc
refactor: move the the per-message overflow log from error to info level
2023-01-26 14:48:43 +01:00
bb26632c8a
fix(buffer_worker): fix a wrong assertion
...
the assertion is to ensure queue items are not binary
but should not assert the queue itself
2023-01-26 14:33:16 +01:00
f6b3b930b0
chore: improve a error log
2023-01-26 14:21:27 +01:00
86cfbfb43c
fix: Authorization header leak in log entries for webhook
...
There might be another possibility for leakage. If the resource mangager
for the webhook resource crashes, OTP might log the spec for the
resource manager which contains the Config and thus the Authorization
header. This is probably an issue for other resources as well and should
be fixed in another commit. The following issue has been created for
that:
https://emqx.atlassian.net/browse/EMQX-8794
Fixes:
https://emqx.atlassian.net/browse/EMQX-8791
2023-01-26 12:44:16 +01:00
908ac8d948
fix: speed up banned test cases
2023-01-26 13:04:55 +08:00
2ee00b75a7
fix(redis): unwrap pipeline queries against redis cluster
...
This is an additional safety measure in addition to the disabled
batching on the bridge level.
2023-01-25 17:28:11 +03:00
805d08e823
fix: reduce log level from error to warning in several places
...
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
2023-01-25 14:49:50 +01:00
8836494542
fix: redact influxdb tokens in a few logs
2023-01-25 14:48:32 +01:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
6175076f6f
Merge pull request #9835 from olcai/add-influxdb-test-files
...
fix: add influxdb test files and fixes
2023-01-24 17:02:23 +01:00
e5b65087af
Merge pull request #9834 from zmstone/0123-fix-idle_timeout-infinity
...
fix(emqx_connection): crash when idle_timeout is set to infinity
2023-01-24 16:05:09 +01:00
9d20431257
fix(emqx_resource): fix crash while flushing queue
...
We used next_event for flushing the queue in emqx_resource, but this
leads to a crash. We now call flush_worker/1 instead.
2023-01-24 14:13:35 +01:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
7575120ea6
test: use snabbkaffe retry macro
2023-01-24 10:54:20 +01:00
140cda2f13
fix(emqx_connection): crash when idle_timeout is set to infinity
2023-01-24 10:14:35 +01:00
a0100c0a4e
chore: bump app versions
2023-01-23 16:20:21 +01:00
a83ed2fe94
Merge remote-tracking branch 'origin/master' into 0123-chore-sync-release-50-to-master
2023-01-23 16:20:01 +01:00
727100e094
chore: prepare for v5.0.15 release
2023-01-20 16:42:01 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
57607ca0ce
chore: prepare for v5.0.15 release
2023-01-20 11:20:34 +01:00
1c3e055b13
Merge pull request #9822 from JimMoen/fix-schema-typo
...
chore: i18n typo fix
2023-01-20 11:11:18 +01:00
16f45a60fd
chore: i18n typo fix
2023-01-20 11:50:01 +08:00
a6ad97e6a5
Merge pull request #9807 from thalesmg/buffer-worker-rename-v50
...
buffer work refactoring follow up (part 1)
2023-01-19 14:17:10 -03:00
abe7a69696
Merge remote-tracking branch 'origin/master' into release-50
2023-01-19 17:51:38 +01:00
6bde1173af
chore: add translations
2023-01-19 13:26:36 +01:00
d755b43c77
fix(jwt_worker): handle exceptions when decoding jwk from pem
...
Returns a more controlled error if users attempt to use the Service
Account JSON from the GCP PubSub example from swagger, which is
redacted.
2023-01-19 09:24:45 -03:00
63748aba3c
Merge pull request #9804 from emqx/release-50
...
Merge release-50 (candidate of e5.0.0-rc.1) back to master.
2023-01-19 08:48:41 +01:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test
...
chore(ee bridge): fix Redis bridge test flakyness
2023-01-18 20:56:13 +02:00
1716a5da99
chore: bump version to e5.0.0-rc.1
2023-01-18 17:22:05 +01:00
fb763ecebd
feat: support HAProxy protocol for dashboard API
2023-01-18 15:49:57 +01:00
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
33b3c4fa9a
Merge pull request #9753 from feat/EMQX-8738/convert-ordered-sets
...
feat: turn tables queried with search APIs into ordered sets
2023-01-18 18:05:57 +04:00
44a6e5ed15
chore(resources): add missing parameters to emqx_resource schema
2023-01-18 14:33:45 +02:00
7e8381f4c7
Merge pull request #9785 from savonarola/fix-authn-handling
...
fix(authn): stop authn handling when emqx_authentication provides a result
2023-01-18 13:24:22 +01:00
c46d7f3404
Merge pull request #9801 from ieQu1/ekka-0.13.9
...
chore(ekka): Bump version to 0.13.9
2023-01-18 13:23:59 +01:00
167b623497
Merge pull request #9699 from thalesmg/fix-buffer-clear-replayq-on-delete-v50
...
fix(buffer): fix `replayq` usages in buffer workers (5.0)
2023-01-18 09:08:50 -03:00
46fc69cd48
Merge pull request #9781 from olcai/delete-zip-file-from-trace-log-download
...
fix(emqx_management): delete files after trace log download
2023-01-18 13:05:39 +01:00
d7242739e0
chore(ekka): Bump version to 0.13.9
2023-01-18 12:01:03 +01:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
8e1475addb
Merge pull request #9798 from zhongwencool/dashboard-document
...
chore: improve the dashboard's configuration
2023-01-18 18:20:00 +08:00
42182279b7
fix(emqx_management): ensure trace file dir is deleted on zip exception
2023-01-18 10:20:41 +01:00
fb84d5b817
chore: make spellcheck happy
2023-01-18 17:06:46 +08:00
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames
...
fix: ensure no colon in filenames
2023-01-18 09:36:16 +01:00
0d852d9122
docs: improve the dashboard's document
2023-01-18 16:28:35 +08:00
faf5916ed6
test: relax recoverable/unrecoverable error check
...
for now, treat all other errors unrecoverable
2023-01-18 07:52:28 +01:00
bc9d97ea53
Merge pull request #9791 from zhongwencool/crash-dump-doc
...
chore: more detail about crash dump config
2023-01-18 09:56:21 +08:00
5c2ac0ac81
chore: don't cancel inflight items upon worker death; retry them
2023-01-17 19:50:30 -03:00
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
...
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
dea0c8230e
Merge pull request #9787 from lafirest/fix/webhook_bridge_cfg_upgrade
...
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-18 04:47:08 +08:00
4ed7bff33f
chore: fix dialyzer warnings
2023-01-17 16:49:16 -03:00
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
b82009bc29
refactor: use monotonic times as refs and store initial times when creating ets
...
with this, we may measure latencies in the future.
2023-01-17 16:48:48 -03:00
3ba65c4377
feat: poke the buffer workers when inflight is no longer full
...
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
2023-01-17 16:48:48 -03:00
b5aaef084c
refactor: enter running state directly
...
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
2023-01-17 16:48:48 -03:00
bd0e2a74ba
refactor: rename inflight_name field to inflight_tid
2023-01-17 16:48:48 -03:00
006b4bda97
feat(buffer_worker): monitor async workers and cancel their inflight requests upon death
2023-01-17 16:48:48 -03:00
731ac6567a
fix(buffer_worker): don't retry all kinds of inflight requests
...
Some requests should not be retried during the blocked state. For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
2023-01-17 16:48:48 -03:00
5425f3d88e
refactor: rm unused fn
2023-01-17 16:48:48 -03:00
5dd24a64c3
refactor(buffer_worker): check if inflight is full before flushing
2023-01-17 16:48:48 -03:00
344eeebe63
fix: always ack async replies
...
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
2023-01-17 16:48:48 -03:00
bd95a95409
refactor: remove redundant `BlockWorker` arg, change boolean to ack/nack
...
`BlockWorker` was always false (ack). Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/ )
2023-01-17 16:48:48 -03:00
478fcc6ffd
test: fix flaky test
2023-01-17 16:48:48 -03:00
30a227bd38
refactor: rename `resume` state timeout to `unblock`
2023-01-17 16:48:48 -03:00
7401d6f0ce
refactor: rename ack fn
2023-01-17 16:48:48 -03:00
196bf1c5ba
feat: mass collect calls from mailbox also when blocked
2023-01-17 16:48:48 -03:00
d4724d6ce9
refactor: remove redundant function
...
`retry_queue` does basically what the running state does, now that we
refactored the buffer workers to always use the queue.
2023-01-17 16:48:48 -03:00
d6a9d0aa48
fix: set queuing to 0 after buffer worker termination
2023-01-17 16:48:48 -03:00
81fc561ed5
fix(buffer_worker): check for overflow after enqueuing new requests
2023-01-17 16:48:48 -03:00
4cb83d0c9a
fix: fix some expressions after refactoring
2023-01-17 16:48:48 -03:00
fecdbac9a8
refactor: rename a few functions
2023-01-17 16:48:48 -03:00
cdd8de11b0
chore: fix a typo in function name
2023-01-17 16:48:48 -03:00
618b97870b
refactor: call local function queue_count everywhere
2023-01-17 16:48:48 -03:00
249c4c1c79
refactor: use 'bufs' for resource worker replayq dir
2023-01-17 16:48:48 -03:00
af6807e863
refactor: cancel flush timer sooner
...
Avoids the cancellation being delayed.
2023-01-17 16:48:48 -03:00
477c55d8ef
fix: sanitizy replayq dir filepath
...
Colons (`:`) are not allowed in Windows.
2023-01-17 16:48:48 -03:00
4c04a01370
refactor(buffer_worker): remove `?Q_ITEM` wrapping and use lightweight size estimate
2023-01-17 16:48:48 -03:00
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
ff23d25e8b
chore(replayq): update replayq -> 0.3.6 and use `clean_start` for buffer workers
...
So we can truly avoid resuming work after a node restart.
2023-01-17 16:48:48 -03:00
c383558467
fix(buffer): fix `replayq` usages in buffer workers (5.0)
...
https://emqx.atlassian.net/browse/EMQX-8700
Fixes a few errors in the usage of `replayq` queues.
- Close `replayq` when `emqx_resource_worker` terminates.
- Do not keep old references to `replayq` after any `pop`s.
- Clear `replayq`'s data directories when removing a resource.
2023-01-17 16:48:48 -03:00
fc992f28bc
test: add test coverage
2023-01-17 20:13:01 +01:00
fff6bf921f
refactor(authz): call emqx_resource:simple_sync_query
...
there is no need to route the request through the buffer workers
2023-01-17 20:01:45 +01:00
7a471faf67
refactor(authn): use simple sync query for authn
...
there is no need to route the authn queries through the buffer worker
2023-01-17 19:57:14 +01:00
1d7364dab1
fix(authn): fix authn hook chain evaluation
...
Stop authentication handling when
`emqx_authentication` provides a definitive result.
2023-01-17 20:06:20 +02:00
d164e5bc0f
fix: don't require ssl conf
2023-01-17 15:29:19 +01:00
e54f2f83b3
test: use same default timeout as elsewhere
2023-01-17 15:29:19 +01:00
30a5cfaa83
fix(emqx_management): remove trace files after zip download
...
We only deleted the resulting zip after a trace file download, not the
actual trace files. This adds a deletion of the uncompressed trace files
as well. It also creates unique directories when collecting trace files
so that concurrent downloads doesn't overwrite files in transit.
2023-01-17 14:24:10 +01:00
94296258b4
test(emqx_management): refactor test suite to use common test utils for API calls
2023-01-17 14:24:10 +01:00
f98786c91f
fix: bad crash_dump_file default value in rpm
2023-01-17 20:47:54 +08:00
cf99089f84
chore: add some trace func to user_default
2023-01-17 20:33:01 +08:00
5a56013193
docs: more detail about crashdump config
2023-01-17 20:33:01 +08:00
9f80eb2497
Merge pull request #9777 from zmstone/0116-refactor-config_files-is-readonly
...
refactor: config_files is a readonly config
2023-01-17 20:32:39 +08:00
263deae1f3
refactor: add a more generic name for password_converter
2023-01-17 11:12:17 +01:00
2afbf6a406
test: unlink process which is getting shutdown
2023-01-17 11:12:17 +01:00
4a7e74f5d6
fix(schema): add password converter to ensure its binary() type
2023-01-17 11:12:17 +01:00
8b3ec48406
refactor: node.config_files should be hidden as it's not configurable
...
this config schema exists only for app-env mapping.
users can not change it because the bootstraping config
has to be etc/emqx.conf
2023-01-17 11:08:40 +01:00
74ae7c4264
Merge pull request #9782 from sstrigler/EMQX-8361-fix-ct-suite-for-plugins-api
...
test: get plugin template as artifact
2023-01-17 11:08:07 +01:00
c3d5c25c26
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-17 17:42:10 +08:00
cb5314bb9e
Merge pull request #9774 from zhongwencool/dashboard-password-validate-v50
...
feat: add dashboard password validation
2023-01-17 17:02:51 +08:00
bc69a00329
test: get plugin template as artifact
2023-01-17 09:50:17 +01:00
44f635300d
chore: update mysql-otp to 1.7.2
2023-01-17 13:36:26 +08:00
3dfdad9cc9
test: dashboard default password
2023-01-17 11:30:42 +08:00
b9911e281e
docs: Update apps/emqx_dashboard/src/emqx_dashboard_admin.erl
2023-01-17 11:30:42 +08:00
c549379b7c
chore: apply suggestions from code review
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-01-17 11:30:42 +08:00
82af854c4a
test: dashbashboard new password failed
2023-01-17 11:30:42 +08:00
a523fa2fa2
feat: add dashboard password validate
2023-01-17 11:30:42 +08:00
33be7672ad
test: recover boot_modules app env after test
2023-01-17 00:43:25 +01:00
676f017ec0
fix: ensure no colon in filenames
2023-01-16 21:27:01 +01:00
5fd157fb1d
Merge pull request #9776 from keynslug/chore/bump-epgsql-otp-25
...
chore: update epgsql to 4.7.0.1 with full OTP-25 compat
2023-01-16 23:51:09 +04:00
e668044ebc
fix: version number previs in restricted shell
2023-01-16 13:47:00 +01:00
34571c779d
feat: make `suboption` table ordering more natural
2023-01-16 15:39:10 +03:00
ce2dba15b4
feat: turn tables queried with search APIs into ordered sets
...
This is needed to ensure more or less consistent client experience
for the new planned cursor-based search APIs.
2023-01-16 15:39:10 +03:00
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs
...
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
2023-01-16 13:37:29 +01:00
54cea3b3f2
chore: update epgsql to 4.7.0.1 with full OTP-25 compat
2023-01-16 15:35:51 +03:00
67909f0b40
fix: testing metrics for emqx_bridge_mqtt_SUITE
2023-01-16 12:10:06 +01:00
a19226326c
fix: don't test for strict values as that makes test flaky
2023-01-16 12:10:06 +01:00
9a53410efb
fix: don't add empty list element to options proplist
2023-01-16 12:00:03 +01:00
edfcea7f43
fix: add log for timeout
2023-01-16 12:00:03 +01:00
47414e0d53
docs: improve kafka key and value field description
2023-01-16 11:32:09 +01:00
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions
...
0112 docs fix some descriptions
2023-01-16 09:56:56 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
089389100e
Merge pull request #9763 from zmstone/0114-fix-crash-when-password-undefined
...
0114 fix crash when password undefined
2023-01-15 12:41:02 +01:00
df202c69ad
Merge pull request #9746 from zmstone/0113-fix-mqtt-connect-packet-format-when-password-is-empty
...
fix(emqx_packet): log empty string if password value is missing
2023-01-14 19:42:07 +01:00
0f2f5fbbe0
fix(authn): no exception when password is 'undefined'
2023-01-14 17:58:55 +01:00
7073c62dd9
feat: add a 'this' for rule engine put env
...
now a dot is allowed in front of a var (path)
e.g. ${.clientid} is equivalent to ${clientid}
${.} however, means everything.
the parsed var is interally represented as $_THIS_
to make it easier to read/search.
2023-01-14 11:03:58 +01:00
002f5dbd89
Merge pull request #9750 from zhongwencool/bootstrap-app-env
...
fix: save app's env when bootstrapping and reloading
2023-01-14 07:33:13 +01:00
b07befbf56
test: conf_app ct failed
2023-01-14 01:03:57 +08:00
6fe09447ed
fix: stale test using old resource paths after merge
2023-01-13 17:23:25 +01:00
09160383f5
chore: bump emqx_resource to 0.1.5
2023-01-13 23:42:41 +08:00
588140b842
chore: why we should ingore save kernel app env when init
2023-01-13 23:35:45 +08:00
1fe0061123
fix: node.config_files type is hocon:array(string()))
2023-01-13 23:35:45 +08:00
c9f0355f5a
fix: save app env when bootstrap and reload
2023-01-13 23:35:45 +08:00
1085492171
chore: don't refresh when CT is running
2023-01-13 23:35:45 +08:00
91d38b410d
fix: refresh logger handler when starting
2023-01-13 23:35:45 +08:00
8f5881d1a5
fix: remove stale request/3 from merge error
2023-01-13 16:19:35 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
61e98900be
chore: bump app vsn of emqx_resource
2023-01-13 15:13:35 +01:00
c3133fb6a2
fix(emqx_bridge): small fixes from review
2023-01-13 14:54:54 +01:00
a77d33b97d
Merge pull request #9751 from lafirest/fix/un_cleared_ca
...
fix that obsoleted cert file will not be deleted after the listener is updated/deleted
2023-01-13 14:38:55 +01:00
1ac03ab208
Merge pull request #9730 from kjellwinblad/kjell/fix/resource_atom_leak/EMQX-8583
...
fix: remove atom leaks
2023-01-13 14:38:28 +01:00
bb3dceb456
Merge pull request #9749 from keynslug/fix/count-respect-matchspec
...
fix(paging): respect matchspec even if qs is empty when counting
2023-01-13 14:29:03 +01:00
8dd52e5a6e
chore: add translations to schemas
2023-01-13 14:19:23 +01:00
f1c58c34ed
test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE
2023-01-13 14:19:23 +01:00
2974269f2a
Merge pull request #9723 from emqx/release-50
...
chore: sync release-50 to master
2023-01-13 14:13:12 +01:00
0a965879dd
fix: fix that obsoleted cert file will not be deleted after the listener is updated/deleted
2023-01-13 18:42:27 +08:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge
2023-01-13 18:12:53 +08:00
734e6b9c96
chore: fix flaky test cases, log labels and review comments
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-01-13 11:05:02 +01:00
b3e62bd8f8
fix(paging): respect matchspec even if qs is empty when counting
2023-01-13 12:56:29 +03:00
383a6812d1
Merge pull request #9748 from zhongwencool/listener-max-connections-default-value
...
fix: default max_connections in raw config should be binary not atom
2023-01-13 17:55:22 +08:00
59ac0b1424
fix(mqtt-bridge): transmit raw msg payload with empty template
2023-01-13 17:23:04 +08:00
0049b4a294
Merge branch 'master' into release-50
2023-01-13 16:39:35 +08:00
e7883e0ee2
Merge pull request #9741 from id/fix-description-for-duration-fields
...
fix: remove time unit from duration fields description
2023-01-13 09:31:48 +01:00
eacdaffe3c
fix: default max_connections in raw config should be binary not atom
2023-01-13 16:01:57 +08:00
33d88c95de
Merge pull request #9745 from keynslug/fix/EMQX-8702/changelog
...
chore: add changelog for the fuzzy search API fix
2023-01-13 11:59:37 +04:00
2bc19c3a62
fix(emqx_packet): log empty string if password value is missing
...
to keep it a consistent behaviour comparing when logging username
2023-01-13 08:21:09 +01:00
153d64068a
refactor: rename root level cache type to authz_cache
2023-01-13 07:47:54 +01:00
c7ac55520d
docs: update pool_size doc
2023-01-13 07:46:23 +01:00
b40ce0fc2d
Merge pull request #9626 from id/fix-enable-authz-cache-by-default
...
fix: enable authorization cache by default
2023-01-13 07:39:38 +01:00
6b75077c47
Merge pull request #9734 from terry-xiaoyu/influxdb-default-value-of-write-syntax
...
refactor: remove default value of timestamp field of influxdb
2023-01-13 10:14:14 +08:00
42f42de4d9
feat(emqx_bridge): add separate endpoint for enable/disable of bridge
...
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
c2fd1a4482
feat(emqx_bridge): shorten operation endpoint URLs
...
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
90e837783f
fix(paging): rename `partial` to `overflow` for clarity
2023-01-12 19:25:40 +03:00
a26c05f4f6
Merge pull request #9586 from lafirest/fix/disable_basic_auth_api
...
fix: disable basic auth for HTTP API
2023-01-12 23:02:29 +08:00
370b6b0d2f
Merge pull request #9735 from kjellwinblad/kjell/fix/password_log_leak/EMQX-8718
...
fix: remove password from info logs
2023-01-12 15:34:16 +01:00
1711823487
refactor: remove default value of timestamp field of influxdb
2023-01-12 22:25:44 +08:00
c5f557e315
fix: disable basic auth for HTTP API
2023-01-12 21:35:49 +08:00
2a81fa1522
fix: remove redundant log message
2023-01-12 14:29:11 +01:00
04f46f5227
feat: make http connector report errors
2023-01-12 14:26:36 +01:00
77f043dedf
fix: don't require username and password
...
not enforced by schema so we shouldn't either
2023-01-12 14:26:36 +01:00
b5d3e9d8b8
fix: remove time unit from duration fields description
2023-01-12 14:18:55 +01:00
f90c41f769
fix: set default value in schema
2023-01-12 13:49:52 +01:00
1f57e7b538
fix: enable authorization cache by default
2023-01-12 12:36:31 +01:00
9e9d97b4d8
Merge pull request #9726 from keynslug/fix/EMQX-8702/fuzzy-search-paging
...
fix(api): augment paged search responses with `hasnext` flag
2023-01-12 13:35:59 +04:00
82e80e5610
fix: remove password from info logs
...
Information log messages for several connectors contained sensitive
password information. This commit fixes this by hiding the sensitive
information from the log messages.
Fixes: https://emqx.atlassian.net/browse/EMQX-8718
2023-01-12 09:28:01 +01:00
13fdbd695e
Merge pull request #9722 from HJianBo/more-confs-for-prometheus
...
feat: more confs for prometheus pushing
2023-01-12 16:06:21 +08:00
50c9321b9e
fix(api): anticipate empty set of nodes handling cluster query
2023-01-12 10:57:50 +03:00
c89b227687
fix(test): adapt affected testcases
2023-01-12 10:57:50 +03:00
abf6f143e5
fix(paging): return subset of rows if page start is in the middle
...
This change should cover this example scenario where select ended
returning 4 rows.
Row 3
Row 4
Row 5 <- page start
Row 6
Here, only rows 5 and 6 should end up in the response. Before this
change some rows could be duplicated across adjacent search pages.
2023-01-12 10:57:50 +03:00
e07aa2086b
fix(api): augment paged search responses with `hasnext` flag
...
This flag indicates whether there are more results available on the
next pages. It is needed in cases when the total number of search
results is not known in advance.
Also, in such cases there's no `count` field in responses anymore
because responding with `0` was confusing for clients: it's not possible
to differentiate between "there are no results" and "we don't know how
much".
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-01-12 10:57:30 +03:00
1194d07a07
refactor: update emqx_prometheus's headers from map() to list()
2023-01-12 12:48:26 +08:00
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field
...
refactor: remove the auto_reconnect field
2023-01-12 11:11:00 +08:00
8c482e03d1
fix: remove atom leaks
...
Both emqx_resource_managers and emqx_resource_workers leaked atoms as they
created an unique atoms to use as registered names. This is fixed by
removing the need to register the names.
Fixes: https://emqx.atlassian.net/browse/EMQX-8583
2023-01-11 17:03:28 +01:00
417d95d76f
Merge pull request #9708 from sstrigler/EMQX-8588-refactor-emqx-resource-manager
...
EMQX 8588 refactor emqx resource manager
2023-01-11 16:13:55 +01:00
3e9c4f444f
refactor: remove the auto_reconnect field
2023-01-11 21:47:06 +08:00
254b92a3c1
docs: add comma to please spellcheck
2023-01-11 10:31:23 -03:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
d5a462486e
feat: tag another emqx schema as `emqx`
2023-01-11 09:31:38 -03:00
e6a8be1ff6
docs: fix closing tag
2023-01-11 09:10:04 -03:00
48e1ba4832
feat(docs): add tags to schemas
...
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
2023-01-11 09:10:03 -03:00
fde8752452
Merge pull request #9710 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master_fix
...
chore: upgrade mongodb-erlang to v3.0.19
2023-01-11 12:41:11 +01:00
9f6df27759
chore: bump app.src vsn
2023-01-11 17:47:06 +08:00
b1292d6d4f
feat(prom): support configuring job_name for pushing mode
2023-01-11 17:31:58 +08:00
50b7ac6a22
feat(prom): support headers for pushing
2023-01-11 17:31:58 +08:00
fdc0682c3f
fix: elvis warning
2023-01-11 17:30:45 +08:00
6f0e228a9b
Merge pull request #9717 from zmstone/0110-do-not-start-resource-after-recreate-if-disabled
...
fix(emqx_resource_manager): do not start when disabled
2023-01-11 16:12:53 +08:00
85a8eff90b
fix(emqx_resource_manager): do not start when disabled
2023-01-11 08:33:48 +01:00
d9429bba00
fix: bump hocon to 0.35.0(don't log overriding structs)
2023-01-11 14:44:34 +08:00
ba738d7707
chore: bump to 5.0.14
2023-01-11 14:39:32 +08:00
ce2b4d48ad
Merge pull request #9714 from zhongwencool/auto-subscribe-api-schema
...
fix: bad auto_subscribe api schema
2023-01-11 14:32:40 +08:00
e7d6d26186
Merge pull request #9713 from zhongwencool/api-keys-bootstrap-file
...
feat: introduce API keys bootstrap_file config
2023-01-11 11:30:45 +08:00
52ccd0762e
test: QOS ct failed
2023-01-11 10:34:09 +08:00
6bbb5edb53
Merge pull request #9667 from olcai/remove-clientid-in-publish-request-body
...
fix(mgmt_api): remove possibility to set clientid in /publish API
2023-01-10 22:31:35 +01:00
ee89de816c
fix: upgrade bridge config converter
2023-01-10 20:42:30 +01:00
fb97096405
fix(mgmt_api): deprecate clientid field instead of removing it
2023-01-10 16:48:04 +01:00
a2be9947f5
chore: update auto_subscribe app version
2023-01-10 22:12:23 +08:00
5415e341fb
fix: bad auto_subscribe api schema
2023-01-10 21:53:54 +08:00
918ba0dffd
chore: apply suggestions from code review
...
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-01-10 20:39:31 +08:00
21dbc21dc8
chore: add changelog for api_key.bootstrap_file
2023-01-10 19:15:01 +08:00
5d4a1933a2
chore: add more test for api_key
2023-01-10 19:04:00 +08:00
f6a47e5cf6
chore: support api_key.bootstrap_file config
2023-01-10 19:03:49 +08:00
142826c4c3
chore: limit the maximum number of api keys to 100
2023-01-10 19:03:39 +08:00
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
...
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
12de104033
Merge pull request #9712 from emqx/fix-bulk-subscribe-failed-in-client.connected-hook
...
fix: bulk subscribe topics failed in the client.connected hook
2023-01-10 18:22:15 +08:00
13f1ab6344
Merge pull request #9671 from ieQu1/metrics
...
feat(emqx_metrics): Sliding window samples
2023-01-10 11:08:43 +01:00
f27f573109
refactor: move to /authorization/sources/built_in_database/rules
2023-01-10 11:00:22 +01:00
73643e932f
fix: bulk subscribe topics failed in the client.connected hook
2023-01-10 16:43:24 +08:00
52343e58c7
Merge pull request #9703 from emqx/subscribe-api-function-clause
...
fix: missing default value for qos field of API /clients/:clientid/subscribe
2023-01-10 10:13:58 +08:00
724015af47
fix: remove the default value of webhook body field
2023-01-10 09:45:25 +08:00
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection
...
0101 authz schema union member selection
2023-01-09 22:17:51 +01:00
0bd5bec982
chore: upgrade mongodb-erlang to v3.0.19
...
This upgrades mongodb-erlang from v3.0.18 to v3.0.19. This fixes a bug
that occurred when one issued a command specified as a map*. This fix
should not affect present EMQX as this functionality is currently not
used by EMQX. However, I think it is good to do the upgrade anyway in
case we will use the fixed functionality in the future. There is no need
to update the changelog.
* https://github.com/emqx/mongodb-erlang/pull/36
2023-01-09 18:14:11 +01:00
f1ebfe40a5
Merge pull request #9697 from lafirest/fix/ldap_i18n
...
chore: add i18n file for ldap connector
2023-01-09 22:54:09 +08:00
7f5150b2e9
Merge pull request #9689 from thalesmg/fix-http-authz-failure-v50
...
fix(authz_http): handle `ignore` results (request failures)
2023-01-09 11:07:52 -03:00
c3635f537a
ci: wait for redis in emqx_ee_bridge_redis_SUITE
2023-01-09 14:39:56 +01:00
0697c692ed
refactor: mongo_type and redis_type are not mandatory
...
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:
[
{mongo_type = rs,
...
}
{another backend config}
]
however the brdige types are maps like, for example:
mongodb_rs.$name {
mongo_type = rs
...
}
in which case, the mongo_type is not required.
in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.
However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
2023-01-09 14:26:16 +01:00
e52f9d5920
refactor: use union member type selector for authz sources
2023-01-09 14:26:16 +01:00
ba8c304723
Merge pull request #9691 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master
...
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-09 14:15:48 +01:00
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
...
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
c85a988a43
fix: split 'server' param into host and port for mqtt bridge
2023-01-09 13:27:42 +01:00
0db31f30a1
Merge pull request #9675 from keynslug/chore/otp-25-compat-deps
...
chore: bump deps versions with full OTP-25 compat
2023-01-09 14:54:14 +04:00
50717a5567
fix: set the topic field of API /clients/:clientid/subscribe as requried
2023-01-09 17:57:39 +08:00
c42c99f94f
fix: set a timeout for tcp connect
2023-01-09 10:01:35 +01:00
96ca0d9f49
feat(emqx_bridge): add /bridges_probe API endpoint
2023-01-09 10:01:33 +01:00
903ae9a644
style: fix typo in fun name
2023-01-09 09:59:42 +01:00
3319a8d28e
fix(mgmt_api): remove possibility to set clientid in /publish API
...
To avoid security confusion, we remove the possibility to specify the
client ID in the request body for /publish and /publish/bulk.
2023-01-09 09:30:34 +01:00
8d53d154d9
refactor: remove license from configs API
...
license has its own config APIs
2023-01-09 09:16:19 +01:00
7ed1384623
fix: missing default value for qos field of API /clients/:clientid/subscribe
2023-01-09 16:02:10 +08:00
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip
...
test: do not use tc_user_skip for test cases
2023-01-09 09:02:06 +01:00
6f5057b9dd
Merge pull request #9637 from HJianBo/fix-clients-api
...
fix(clients): fix expiry_interval unit error
2023-01-09 09:49:16 +08:00
336b310d56
Merge pull request #9687 from emqx/function_clause_when_send_msg_to_bridges
...
fix: function_clause when sending messages to bridges
2023-01-09 09:23:17 +08:00
1336a57b3b
chore: bump to emqx/eredis_cluster 0.7.5
...
* Full OTP-25 compat.
* Password censoring in status and crash reports.
* No more crashes on query timeout.
2023-01-08 13:13:31 +03:00
5653d113ac
Merge pull request #9663 from zmstone/0102-fix-stopped-nodes-detection
...
0102 fix stopped nodes detection
2023-01-08 10:03:48 +01:00
d2456f9fbe
chore: add i18n file for ldap connector
2023-01-06 17:28:06 +08:00
f7b50c56da
feat(bridges): integrate PostgreSQL into bridges
...
support both simple and batch query
2023-01-06 11:47:22 +08:00
5d55e3a7b3
chore: updage mgmt app vsn
2023-01-06 10:31:27 +08:00
65729cd640
fix(clients): fix expiry_interval unit error
...
It should be second not millisecond
2023-01-06 10:31:27 +08:00
13b2f45405
fix: function_clause when sending messages to bridges
2023-01-06 09:24:38 +08:00
7a4e7a6de9
test: wait for redis
2023-01-05 22:47:08 +01:00
5f12cdff6c
ci: should not skip tests in github actions
2023-01-05 22:47:08 +01:00
22707495ac
Merge pull request #9690 from thalesmg/fix-flaky-jwt-test
...
test(flaky): fix flaky jwt worker test
2023-01-05 14:00:39 -03:00
5266722e51
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-05 16:40:45 +01:00
b59c4c34c5
fix(Bridge REST API): no feedback when deleting bridge
...
This fixes https://emqx.atlassian.net/browse/EMQX-8648 . The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:
```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist '
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```
After the fix, deleting a non existing bridge will give the following
response:
```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```
Closes: EMQX-8648
2023-01-05 16:19:46 +01:00
c6b8e614df
fix(authz_http): handle `ignore` results (request failures)
...
Related issue: https://github.com/emqx/emqx/issues/9683
When the HTTP request for authz fails (e.g.: resource is down or
server is down), then the HTTP authorizer returns `ignore`, which was
not handled correctly by the authorization callback.
2023-01-05 11:34:23 -03:00
cf6996cbea
test(flaky): fix flaky jwt worker test
2023-01-05 11:19:43 -03:00
70eb5ffb58
refactor: remove unused function
2023-01-05 10:16:01 -03:00
56437228dc
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections.
2023-01-05 10:16:01 -03:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
bf3983e7c4
feat(buffer_worker): use offload mode for `replayq`
...
To avoid confusion for the users as to what persistence guarantees we
offer when buffering bridges/resources, we will always enable offload
mode for `replayq`. With this, when the buffer size is above the max
segment size, it'll flush the queue to disk, but on recovery after a
restart it'll clean the existing segments rather than resuming from
them.
2023-01-05 10:11:59 -03:00
5d9f9671e9
feat(emqx_metrics): Sliding window samples
2023-01-05 13:55:52 +01:00
fa4f90d43e
Merge pull request #9676 from thalesmg/fix-change-retainer-storage-type-v50
...
fix(retainer): change mnesia table storage types during update
2023-01-05 09:07:10 -03:00
93c205db7c
fix(router_helper): Reduce log verbosity of expected events
2023-01-05 12:17:27 +01:00
51ad27cb4b
test(retainer): assert that retained messages are not lost when changing storage type
2023-01-04 13:52:52 -03:00
2f13bfd452
fix(retainer): change mnesia table storage types during update
...
https://emqx.atlassian.net/browse/EMQX-8650
2023-01-04 13:52:52 -03:00
86a4ac87a7
Merge pull request #9674 from thalesmg/reset-rule-metrics-on-disable-v50
...
chore(rule_engine): reset metrics when disabling a rule
2023-01-04 13:30:39 -03:00
00b91d8a60
Merge pull request #9664 from thalesmg/fix-mongo-duration-field-descs-v50
...
docs(mongodb_connector): fix duration field descriptions
2023-01-04 10:03:38 -03:00
eb7dca3691
chore(rule_engine): reset metrics when disabling a rule
...
https://emqx.atlassian.net/browse/EMQX-8502
When a bridge is disabled, its metrics are reset. With this change,
we make rule actions behave like that: disabling a rule will reset its
metrics.
2023-01-04 09:30:47 -03:00
f767db4d8f
Merge pull request #9660 from olcai/refactor-ingress-bridge-counter
...
refactor(emqx_resource): ingress bridge counter
2023-01-04 10:43:07 +01:00
c3cfbae3c2
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections
2023-01-03 11:38:09 -03:00
b43be50a98
test: add the `redacted` test case for bridges api
2023-01-03 22:16:34 +08:00
b016695cb1
fix(bridges): obfuscate the password in bridges API responses
2023-01-03 22:09:29 +08:00
7bbdc20993
docs(mongodb_connector): fix duration field descriptions
...
https://emqx.atlassian.net/browse/EMQX-8642
Adds different descriptions for the different fields of type duration
for MongoDB.
Based off: https://www.mongodb.com/docs/manual/reference/connection-string/
2023-01-03 10:01:56 -03:00
ced5a27053
test: fix emqx_conf test cases
2023-01-02 21:30:40 +01:00
2f208c3ab9
fix(cluster_rpc): Detect stopped replicant nodes
2023-01-02 21:30:40 +01:00
7985cd3536
Merge pull request #9638 from lafirest/fix/mysql_dup
...
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
2023-01-02 23:15:44 +08:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
bc37556963
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
2023-01-02 21:46:24 +08:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
231d6da4dc
Merge pull request #9644 from id/fix-kafka-bridge-connectivity-detection
...
fix(kafka): detect connectivity in on_get_status
2022-12-30 21:54:56 +01:00
305ed68916
chore: bump app vsns
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
b30f2ea58b
fix(statsd schema): a field is not 'required' when it has 'default'
2022-12-30 14:20:24 +01:00
981f75dc73
test(emqx_modules): do not expect badmatch in test cases!!!
2022-12-30 14:20:24 +01:00
0ce1ca89b7
refactor: use string type for server and servers
2022-12-30 14:20:23 +01:00
cac7e0c5f0
refactor: move version prefix string detection to a static fun
...
this is to make dialyzer happy
2022-12-30 14:19:03 +01:00
a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50
...
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
2022-12-30 09:33:41 -03:00
f410201dc3
chore: fix flaky test case in emqx_connector_jwt_worker_SUITE
2022-12-30 11:45:13 +01:00
f92b1fb8f4
Merge pull request #9633 from zhongwencool/fix-json-fmt
...
fix: bad best fmt json []
2022-12-30 09:08:13 +08:00
446a4c74d0
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
...
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652
Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.
Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
0b43ae621d
ci: dump docker-compose log if failed to run ct
2022-12-29 09:23:11 +01:00
4e62aff3ab
fix: bad best fmt json []
2022-12-29 14:53:00 +08:00
2790d5697d
test: add a 2 seconds sleep between listener sotp and restart
...
to lower the chance of eaddrinuse
2022-12-28 18:48:53 +01:00
4819794401
test(refactor): decrease test teardown noise
2022-12-27 10:30:14 -03:00
6fdcba641e
test(refactor): no need for monitor
...
the janitor is already linked to the parent
2022-12-27 10:30:14 -03:00
a777bac38c
build: prepare for v5.0.13 release
2022-12-27 11:54:09 +01:00
15bd797b36
ci: bump to v5.0.13-rc.3
2022-12-27 11:24:47 +01:00
580137de0d
Merge remote-tracking branch 'origin/master' into release-50
2022-12-27 11:06:17 +01:00
2ff6ef01f1
Merge pull request #9599 from lafirest/fix/blacklist_default_exp
...
fix(banned): ensure the default expiration time of `banned` is large enough
2022-12-27 17:30:10 +08:00
0517541f34
ci: bump to v5.0.13-rc.2
2022-12-27 09:54:34 +01:00
18246442aa
chore: prepare v5.0.13-rc.1
...
bump dashboard version to v1.1.4
2022-12-26 21:45:48 +01:00
81b226a801
Merge pull request #9608 from zmstone/1123-pin-hocon-0.33.0
...
1123 pin hocon 0.33.0
2022-12-26 19:22:21 +08:00
af3f0c9bb0
fix(banned): ensure the default timeout of `banned` is large enough
2022-12-26 10:17:31 +08:00
f93c22045d
fix: non-empty field should not be undefined
2022-12-24 11:41:45 +01:00
269a40a9ea
fix: format psk user lookup fun back to string
2022-12-23 19:22:15 +01:00
d3efb0c0ba
chore: bump app versions
2022-12-23 15:10:16 +01:00
350023e757
fix(config): option only_fill_defaults renamed to make_serializable
2022-12-23 14:27:04 +01:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
fa7433628c
fix(logs): set password field sensitive for connectors
2022-12-23 12:46:03 +01:00
59970272ff
chore: pin hocon 0.33.0
2022-12-23 12:07:36 +01:00
6292d7ec68
refactor: remove dead code
2022-12-23 12:07:36 +01:00
13942f5c49
refactor: rename error return in mysql connector
2022-12-22 10:29:12 +01:00
7df24000a0
test: add more EE mysql bridge test cases
2022-12-22 10:27:38 +01:00
23ac426608
fix(emqx_connector): check for key among prepared statements on query
...
An infinite loop was triggered in the mysql connector when a query
used a prepared statement key that was not among the defined prepared
statements on start. We now check that the key is defined among the
prepared statements before recursing. It seems that this bug was never
triggered in any production code flow and simply found while writing
tests.
An error return spell fix is also included as well as a FIXME comment
regarding running mysql:prepare and not distinguishing between
transient failures and syntax errors. Syntax errors should not be
retried.
2022-12-22 10:27:38 +01:00
f660724bf7
Merge pull request #9582 from lafirest/fix/fix_field_name
...
fix(limiter): change limiter `cache` to `buffer`
2022-12-21 18:12:40 +08:00
a8e020af58
fix(limiter): change limiter `cache` to `buffer`
2022-12-20 22:30:52 +08:00
516147ad00
Merge pull request #9581 from zmstone/1219-fix-mqtt-bridge-config
...
1219 fix mqtt bridge config
2022-12-20 14:47:02 +01:00
Zaiming (Stone) Shi