b2ca59dfa7
fix(bridge_gcp): Don't print config to the logs
2023-12-05 15:48:53 +01:00
e5a4be11f5
chore: stop downgrading action id in rule API responses
...
Fixes https://emqx.atlassian.net/browse/EMQX-11405
2023-12-05 11:48:46 -03:00
1b68fbff05
fix(api): return list of dependent rule ids when trying to delete bridge/action
...
Fixes https://emqx.atlassian.net/browse/EMQX-11523
2023-12-05 11:47:26 -03:00
286d483a3a
Merge pull request #12055 from keynslug/ft/EMQX-11474/subopts
...
fix(sessds): respect subscription options when publishing
2023-12-05 15:45:25 +01:00
e995b3948e
chore: bump emqx_gateway_coap version
2023-12-05 15:17:53 +01:00
b6e8f5a37c
Merge pull request #12095 from thalesmg/fix-rule-test-api-m-20231204
...
fix(rule_api): fix sql test when testing against events
2023-12-05 08:57:54 -03:00
7b9d12f20e
Merge pull request #12048 from SergeTupchiy/fix-coap_sub-opts
...
Fix COAP sub opts
2023-12-05 10:27:04 +02:00
c1c7d1246f
Merge pull request #12084 from thalesmg/refactor-resource-status-state-m-20231201
...
refactor(resource_manager): use macros and better differentiate status from state
2023-12-04 17:55:27 -03:00
d594b38ceb
Merge pull request #12083 from thalesmg/fix-connector-bws-m-20231201
...
fix(connector): don't start buffer workers for the connector itself
2023-12-04 16:14:15 -03:00
984e2ccc74
Merge pull request #12082 from thalesmg/test-ensure-res-opts-conn-m-20231201
...
add missing `resource_opts` fields to connectors schemas
2023-12-04 16:13:32 -03:00
cea857cb8e
fix(rule_api): fix sql test when testing against events
...
Fixes https://emqx.atlassian.net/browse/EMQX-11449
2023-12-04 15:24:09 -03:00
b2cd8aaab7
refactor: use single clause
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-12-04 10:02:05 -03:00
ac2c99f996
test(connector): add test to ensure connector schema has `resource_opts`
...
https://emqx.atlassian.net/browse/EMQX-11509
2023-12-04 10:02:05 -03:00
bb04a2d89e
fix(bridge_redis): add `resource_opts` to connector schema
2023-12-04 10:02:05 -03:00
87a13aa316
fix(bridge_mongo): add `resource_opts` to connector schema
2023-12-04 10:02:05 -03:00
7a1288e802
fix(bridge_postgres): add `resource_opts` to connector schema
2023-12-04 10:02:05 -03:00
3078de6e97
fix(bridge_syskeeper): add `resource_opts` to connector schema
2023-12-04 10:02:05 -03:00
5bdfac52b8
fix(bridge_kafka_producer): add missing `resource_opts` sub-fields to connector schema
2023-12-04 10:02:05 -03:00
626bcc1c62
fix(bridge_http): add `resource_opts` to connector schema
2023-12-04 10:02:05 -03:00
10b1a8bbda
feat(connector): add some `resource_opts` schema facilities
2023-12-04 10:02:05 -03:00
9920841deb
chore: upgrade greptimedb client to v0.1.6 to fix the failed test
2023-12-04 20:30:33 +08:00
1d6ea685ef
fix: another try to fix t_write_failure
2023-12-04 20:30:33 +08:00
97592e2e40
fix: test case t_write_failure
2023-12-04 20:30:33 +08:00
fbeb56e475
fix: test case t_write_failure
2023-12-04 20:30:33 +08:00
547acef01d
feat: upgrade greptimedb client to v0.1.5
2023-12-04 20:30:33 +08:00
a694f3c92d
fix: tests
2023-12-04 20:30:33 +08:00
8bc874c261
feat: supports async mode for greptimedb data bridge
2023-12-04 20:30:33 +08:00
90d0081df4
feat: upgrade greptimedb and client
2023-12-04 20:30:33 +08:00
130a5a5442
fix(ds): pass topics to `emqx_topic:words/1` before feeding LTS tree
...
So that empty levels in topics will be properly mapped into `''` atoms.
2023-12-04 13:39:01 +03:00
92c4b29a4c
feat(topic): match empty topic levels more loosely
...
So that the result of `emqx_topic:tokens/1` would be perfectly
matchable with the result of `emqx_topic:words/1` of a topic filter
with empty levels.
2023-12-04 13:38:55 +03:00
b5c485cd55
test(sessds): add empty level topic subscription testcase
2023-12-04 13:38:42 +03:00
e455433694
test(emqx_gateway): fix test failures caused by grpc-erl upgrade
2023-12-04 12:17:58 +02:00
cb992fcf55
Merge pull request #12059 from JimMoen/1130-fix-vm-time-warp
...
fix: vm.args use `multi_time_warp`
2023-12-04 11:56:08 +08:00
627c58b07d
fix(sessds): handle `expire_awaiting_rel` common timer with no-op
...
Otherwise, the channel crashes when the timer is triggered.
2023-12-02 15:00:30 +03:00
47bc747323
fix(sessds): drop everything related to session on CleanStart
2023-12-02 15:00:30 +03:00
ee9a98f0a4
chore: bump emqx_gateway_mqttsn to 0.1.7
2023-12-02 15:00:29 +03:00
fd26e690b8
refactor(sessds): move parts of message processing to replayer
...
To simplify the processing flow, reducing the number of back-and-forth
between the session and the replayer.
2023-12-02 15:00:29 +03:00
29ec73847a
fix(utils): make `flattermap/2` results less variative
2023-12-02 14:38:04 +03:00
6255ee0833
fix(sessmem): avoid defining unsafe defaults in `#session`
2023-12-02 14:38:04 +03:00
b5ad0f9815
fix(test): subscribe with QoS 2 to receive QoS 2 message
2023-12-02 14:38:04 +03:00
1948719eec
test(sessds): unskip testcase related to QoS2 handling
2023-12-02 14:38:04 +03:00
3265d2f2aa
fix(sessds): respect subscription options when publishing
2023-12-02 14:38:03 +03:00
b5f39f89e3
feat(utils): add `flattermap/2` as slightly more generic `flatmap/2`
2023-12-02 14:36:21 +03:00
508346f095
refactor(topicidx): split persistent term stuff off gbt-based index
2023-12-02 14:36:21 +03:00
88103c5f0e
refactor(session): pass config to `SessionImpl:open/3` as well
...
* Anticipate that connection info may change during session takeover.
* Avoid persisting session conf as part of persistent session state.
2023-12-02 14:36:21 +03:00
6eba082992
Merge pull request #12075 from keynslug/fix/EMQX-11474/function-clause
...
fix(channel): expect there's no session yet on disconnect
2023-12-02 14:32:50 +03:00
dc5e3b939c
refactor(resource_manager): use macros and better differentiate status from state
...
Internally in `emqx_resource_manager`, there seems to be many points where the
`gen_statem` states are conflated with resource status, since their names coincide. While
that works for now, introducing a new `gen_statem` state, an internal state, shouldn't
necessarily imply a new, externally facing resource status.
Here we also introduce the usage of some macros to avoid the pitfalls of making a typo in
a state/status name.
2023-12-01 18:23:05 -03:00
55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all
...
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 21:35:40 +01:00
29ae45c39d
fix(connector): don't start buffer workers for the connector itself
...
Fixes https://emqx.atlassian.net/browse/EMQX-11448
2023-12-01 17:20:38 -03:00
6e30718818
test: allow join cluster later
2023-12-01 20:33:39 +01:00
3d2e95fe3e
Merge pull request #12081 from ieQu1/dev/gen-rpc-3.3.0
...
chore(gen_rpc): Bump version to 3.3.0
2023-12-01 17:57:33 +01:00
476c300ecf
Merge pull request #12070 from ieQu1/dev/ds-min-max-batch-size
...
Various minor fixes
2023-12-01 17:57:15 +01:00
d3b7478d70
chore(gen_rpc): Bump version to 3.3.0
2023-12-01 17:10:12 +01:00
ab7d57e1c5
test: delete debug level logging
2023-12-01 17:07:14 +01:00
0e27d031a8
chore: 5.4.0-alpha.1
2023-12-01 16:54:03 +01:00
6a9cc20d4d
test: wait for all nodes clustered
2023-12-01 16:37:39 +01:00
956b25cc4c
Merge pull request #12071 from zmstone/1201-sync-release-54
...
1201 sync release 54
2023-12-01 14:38:47 +01:00
cb60880bab
Merge pull request #12067 from thalesmg/fix-kafka-check-client-m-20231130
...
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
2023-12-01 09:55:12 -03:00
6b17920fea
fix(channel): expect there's no session yet on disconnect
2023-12-01 15:29:05 +03:00
cce474ce98
fix(session): include session term in stacktrace
2023-12-01 15:29:05 +03:00
1a78e7ae79
fix(connector): ensure webhook bridge convert to http connector
2023-12-01 11:56:37 +01:00
90b7486624
fix(redis): ensure schema namespace
2023-12-01 11:37:06 +01:00
e1ec560639
test(emqx): Fix flaky emqx_takeover_SUITE:t_takeover testcase
...
This testcase uses QoS1, so it must account for possible duplication
of the messages.
2023-12-01 10:06:18 +01:00
60d70b22dc
chore: bump app vsn for emqx_auth_redis
2023-12-01 08:37:37 +01:00
7f4d91d490
Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54
2023-12-01 08:32:22 +01:00
e238602533
fix(ds): Update README
2023-12-01 08:27:05 +01:00
0ae618d010
fix(ds): Use emqx_rpc for calls that work with large binaries
2023-12-01 08:27:05 +01:00
0e625d814a
feat(sessds): Make batch size configurable
...
This change affects flow control. It allows to configure maximum size
of a batch, as well as fetch threshold.
2023-12-01 08:27:05 +01:00
4717e56fb6
fix(sessds): Schedule poll immediately upon receiving an ack
...
This commit affects the flow control, and improves the throughput by
removing a delay between freeing up space in the in-flight window and
polling new messages.
2023-12-01 08:27:05 +01:00
38800c0260
refactor(sessds): Store timers in the session
2023-12-01 08:27:05 +01:00
22f7cc1622
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 08:07:09 +01:00
ee25d9bd9e
fix: vm.args use `multi_time_warp`
...
See also:
https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
2023-12-01 14:52:31 +08:00
69f1ca43c3
fix(sessds): Create a timer even if it's present in the map
2023-12-01 06:11:13 +01:00
1897e5c31b
refactor(session): Use common naming conventions
2023-12-01 06:11:08 +01:00
d9e7544070
refactor(sessds): Introduce macros for the timers
2023-12-01 06:11:08 +01:00
0d245acdc1
Merge pull request #12066 from keynslug/fix/sessds-tc-flaky
...
test(sessds): wait client disconnect propagates to broker
2023-12-01 00:13:38 +01:00
113a4ad4b1
Merge pull request #12057 from thalesmg/ds-session-gc-m-20231128
...
feat(ds): add session gc process
2023-11-30 17:51:52 -03:00
0388e1c1c4
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
...
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-11-30 17:51:32 -03:00
9fd2fa95a8
chore: bump apps versions
2023-11-30 20:01:12 +01:00
684d637fed
test(bridge_api): workaround strange config syncing problem
...
For some unknown reason, this test has difficulties in syncing the config correctly
between the nodes, while the equivalent in bridge_v2_api_SUITE doesn't.
2023-11-30 19:51:35 +01:00
ec10c51073
Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53
2023-11-30 19:51:12 +01:00
dcb1c0680b
Merge pull request #12053 from zmstone/1129-prepare-for-otp26
...
1129 prepare for OTP 26
2023-11-30 19:38:40 +01:00
ec471c0557
test(sessds): wait client disconnect propagates to broker
2023-11-30 20:14:57 +03:00
f54bda7063
fix(emqx_mgmt_api_data_backup): add count and hasnext meta paging params
2023-11-30 19:10:12 +02:00
81e75cf068
test: fix bulk-kick test case flakyness
2023-11-30 17:30:25 +01:00
5427ebc5f1
fix: log error when failed to install plugin
...
the error return is a map with details.
prior to this change only the reason is returned in the api response,
now the error map is logged at error level to help troubleshooting
2023-11-30 16:49:30 +01:00
6dd92c382f
chore: 5.3.2 release
2023-11-30 16:18:44 +01:00
cf6cb3e4ad
chore: set low importance to config
2023-11-30 11:55:57 -03:00
1ab009f081
refactor: rename supervisor
2023-11-30 11:55:48 -03:00
eed253af82
test: implement a new node restart helper func
2023-11-30 15:11:38 +01:00
880f5e8f89
feat(ds): add session gc process
...
Fixes https://emqx.atlassian.net/browse/EMQX-9744
2023-11-30 09:38:59 -03:00
cf72c04fdd
test: fix peer node stop and plugin SUITE typo
2023-11-30 09:53:36 +01:00
f2db4cc7fc
chore: upgrade to gen_rpc 3.2.2
2023-11-30 09:42:44 +01:00
640b0df319
test: do not add -master erl flag for peer nodes
2023-11-30 08:49:57 +01:00
773bd8ecb2
Merge pull request #12047 from thalesmg/test-more-flaky-tests-r53-20231128
...
test(flaky): more adjustments
2023-11-30 09:33:55 +08:00
4ecfe2be30
test: use peer module for slave and ct_slave
2023-11-29 21:45:53 +01:00
62b763a8f8
test(gcp_pubsub_consumer): even more adjustments
2023-11-29 15:12:47 -03:00
7f5433f6dd
chore: 5.3.2-rc.1
2023-11-29 18:20:15 +01:00
fdafe2493e
Merge pull request #12041 from thalesmg/ds-estimate-last-alive-m-20231128
...
fix(ds_session): take conservative estimate of `last_alive_at` when bumping
2023-11-29 13:43:59 -03:00
6f35f25163
chore: upgrade esockd to 5.9.8 for OTP 26
2023-11-29 16:15:18 +01:00
1a563b4f65
chore: fix 0.0 match for OTP 26
2023-11-29 16:15:18 +01:00
14644988e0
chore: change triple-quotes to single-quotes
2023-11-29 16:15:18 +01:00
e6eb97e104
chore: upgrade dependency brod_gssapi to work with OTP 26
2023-11-29 16:15:18 +01:00
41194cacc8
Merge pull request #12030 from zhongwencool/redis-connector-v2-540
...
Redis connector v2 540
2023-11-29 16:13:09 +01:00
a750fbbc52
Merge pull request #12051 from HJianBo/fix-http-bridge-bugs
...
Fix http bridge bugs
2023-11-29 15:48:51 +01:00
89cdfbca63
fix(emqx_vm): trim new-line in otp version string
2023-11-29 15:32:25 +01:00
ccef91437d
Merge pull request #12029 from keynslug/ft/EMQX-11049/qos2
...
feat(sessds): provide QoS2 message replay support
2023-11-29 14:10:16 +03:00
009a15c7d0
fix(emqx_mgmt_api_data_backup): validate empty file in upload API
...
fixes EMQX-11488
2023-11-29 12:12:42 +02:00
d8691f1d64
refactor(sessds): rename marker → committed offset
...
For better clarity.
2023-11-29 13:04:42 +03:00
0aec2f7605
feat: redis bridge v2
2023-11-29 16:06:46 +08:00
8ec3857f4d
Merge pull request #12046 from savonarola/1128-fix-redis-54
...
[5.4] Fix Redis connector
2023-11-29 16:06:23 +08:00
e90e843ff3
Merge pull request #12040 from qzhuyan/dev/william/quic-0.0.300
...
feat(quic): bump to quicer-0.0.303
2023-11-29 08:48:47 +01:00
d2e5f302a8
chore(http): add description for bridges v1
2023-11-29 13:47:59 +08:00
72bc046063
chore: avoid unnecessary default values being filled when querying via the v1 api.
...
see: https://emqx.atlassian.net/browse/EMQX-11482
2023-11-29 12:01:59 +08:00
d7bf8e97d2
chore: add tests case
2023-11-29 11:18:51 +08:00
b0d4ba183e
test: use `emqx_cth_suite` in `emqx_coap_api_SUITE`
...
`emqx_coap_api_SUITE` was failing with `emqx_authn_schema_not_injected`
(missing authn schema mods in persistent_term) if it was run standalone.
2023-11-28 20:02:21 +02:00
3530ed66d5
fix(emqx_gateway_coap): take subscribe options from uri_query
...
Closes #12031
2023-11-28 20:02:21 +02:00
095e7c4ecb
test(flaky): more adjustments
2023-11-28 13:41:31 -03:00
f6f61f4353
Merge pull request #12036 from thalesmg/test-more-flaky-tests-r54-20231127
...
test: attempting to stabilize more flaky tests
2023-11-28 13:40:49 -03:00
e5508105c7
fix(redis): start and load eredis app
2023-11-28 18:51:53 +03:00
55218e2df2
fix(redis): start and load eredis app
2023-11-28 18:51:19 +03:00
d93fdd8e9b
Merge pull request #12039 from zmstone/1128-sync-release-54
...
Sync release-54 to master
2023-11-28 15:52:49 +01:00
64f6220cf5
feat(quic): bump to quicer-0.0.303 and emqtt 1.9.7
2023-11-28 14:43:47 +01:00
8a065ba6d0
Merge pull request #12027 from ieQu1/dev/ds-meta-persist-shard
...
fix(ds): Persist shard table
2023-11-28 14:33:57 +01:00
f5c4fb5860
fix(ds_session): take conservative estimate of `last_alive_at` when bumping
...
Addresses https://github.com/emqx/emqx/pull/12024#discussion_r1407432154
2023-11-28 09:58:03 -03:00
bd40b5c553
test(flaky): more adjustments
2023-11-28 09:48:40 -03:00
2e11ab6a16
chore: 5.3.2-alpha.2
2023-11-28 13:46:28 +01:00
05e47254e2
fix(ds): Fixes related to the shards table
2023-11-28 12:59:09 +01:00
6c85e62d26
fix(schema): add namespaces
2023-11-28 12:07:42 +01:00
1b1cea24f3
Merge remote-tracking branch 'origin/release-54' into 1128-sync-release-54
2023-11-28 11:43:30 +01:00
34c9c022d0
chore(replayer): add comment describing what "until" means
2023-11-28 11:47:54 +03:00
bd6e9503e6
fix(http): compose the url and path in correctly format
2023-11-28 16:15:46 +08:00
6e6e68c902
Merge pull request #12037 from zhongwencool/license-watermark-check
...
fix: validate error when set license's watermark to 100%
2023-11-28 16:04:39 +08:00
29bcdb9a4a
fix: validate error when set license's watermark to 100%
2023-11-28 13:16:16 +08:00
891ecc179d
chore: fix flaky tests
2023-11-28 10:34:44 +08:00
c8b5c51bbc
chore: fix failed test cases
2023-11-28 09:53:46 +08:00
cdb90ebe6b
feat: rename webhook bridge to http bridge
2023-11-28 09:53:46 +08:00
dc99651690
test(bridge): ensure almost test cases passed
2023-11-28 09:53:46 +08:00
8954450c0b
chore: fix compile warnings
2023-11-28 09:53:46 +08:00
96af7a74e8
feat: impl the http bridge v2
2023-11-28 09:53:46 +08:00
f9a1e747fd
chore(http): break the bridge confs to connector and action parts
2023-11-28 09:53:46 +08:00
3b1ae0f3df
Merge pull request #12017 from SergeTupchiy/EMQX-11319-data-backup-http-api
...
EMQX-11319 data backup http api
2023-11-27 19:23:54 +02:00
e3fbf6c958
test: attempting to stabilize more flaky tests
2023-11-27 14:12:48 -03:00
26e59f9508
Merge pull request #12024 from thalesmg/ds-session-expiry-m-20231124
...
feat(ds): session expiry
2023-11-27 14:11:38 -03:00
ac370fed61
Merge pull request #12035 from sstrigler/EMQX-11382-a-new-field-should-be-added-to-the-interface-of-the-connector-to-return-the-actions-associated-with-it
...
feat(emqx_connector): add field 'actions' in API response
2023-11-27 17:39:44 +01:00
d933d0b9e0
Merge pull request #12016 from zmstone/1122-load-license-file
...
1122 load license file
2023-11-27 16:43:51 +01:00
70978df5bb
fix: use parse_id
2023-11-27 16:32:15 +01:00
89fdbf3e73
fix(emqx_mgmt_cli): remove duplicated word in the log message
2023-11-27 17:30:58 +02:00
64ee29af81
feat(emqx_management): implement data backup API
2023-11-27 17:30:58 +02:00
86685bdce2
feat(sessds): use integer tags for pubrange types
2023-11-27 18:27:15 +03:00
bb05281adb
refactor(sessds): add dedicated `#ds_pubrange.tracks` field
...
This slightly simplifies the replayer code.
2023-11-27 18:24:57 +03:00
d88deb9ceb
feat(ds): add session timer to bump last alive at timestamp
2023-11-27 12:19:52 -03:00
048f4724a9
feat(emqx_connector): add field 'actions' in API response
...
Also unify schemas, use emqx_connector_schema for the generic parts.
2023-11-27 16:09:17 +01:00
09c4e40511
refactor(ds): rename `disconnected_at` to `last_alive_at`, add more assertions
2023-11-27 11:48:44 -03:00
f2dbddc315
test: attempting to stabilize more flaky tests
2023-11-27 11:36:32 -03:00
6acdc6e432
feat: add `/actions/:id/metrics/`, `/actions/:id/metrics/reset` APIs
...
Fixes https://emqx.atlassian.net/browse/EMQX-11381
2023-11-27 11:33:11 -03:00
923898eadf
chore(chan): leave a TODO note for PUBREC handler
2023-11-27 09:55:46 +03:00
41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs
2023-11-27 09:55:46 +03:00
46475fac66
feat(sessds): provide QoS2 message replay support
2023-11-27 09:55:45 +03:00
ce59cb71bb
chore: bump emqtt to 1.9.6
2023-11-27 09:55:45 +03:00
b5a00ec6b2
Merge pull request #12023 from emqx/master
...
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
bd7a84fe3e
revert(ds): Don't duplicate the clean start in session_ds
2023-11-26 19:18:59 +01:00
c85004b7ef
chore: remove obsolete TODO
2023-11-24 21:22:27 +01:00
c6c1d886f0
fix: make pgsql action schema properly
2023-11-24 21:19:26 +01:00
839f9dbedb
feat(ds): session expiry
...
Fixes https://emqx.atlassian.net/browse/EMQX-11048
2023-11-24 16:31:54 -03:00
66945dcc5c
fix: address more comments from @thalesmg
2023-11-24 20:31:29 +01:00
30e248061f
fix: problems found by @thalesmg in code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-24 20:07:37 +01:00
f070d80b1a
fix: swagger examples for PostgreSQL, Matrix and Timescale
2023-11-24 18:50:52 +01:00
d03674a505
fix: duplicate key in example
2023-11-24 17:59:56 +01:00
fc7bedb81a
fix: remove duplicated entry
2023-11-24 17:37:58 +01:00
e920160805
fix: add enable and description fields to PostgreSQL connector
2023-11-24 17:33:39 +01:00
3dca83c854
fix: all missing descriptions
2023-11-24 17:33:39 +01:00
c5e281b84b
fix: emqx_auth_postgresql test suites
2023-11-24 17:33:39 +01:00
6ef9c6fe4a
fix: ops
2023-11-24 17:33:39 +01:00
f7296d549f
fix: elvis problem
2023-11-24 17:33:39 +01:00
64c015cf6f
fix: unify the schema modules in emqx_bridge_pgsql
2023-11-24 17:33:39 +01:00
b8f510d956
fix: add env variables about action info
2023-11-24 17:33:39 +01:00
2e3028a8f8
fix(emqx_postgresql): fix lifecycle test
2023-11-24 17:33:39 +01:00
dbe73c70b1
fix: dialyzer problem
2023-11-24 17:33:38 +01:00
d5b62eead0
feat: split pgsql, matrix and timescale into connector action
...
This commit splits the bridges pgsql, matrix and timescale into
connector and action.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
2023-11-24 17:33:35 +01:00
f8f8cf9f30
Merge pull request #11963 from sstrigler/EMQX-11156-bridge-v-2-mongo-db-support
...
EMQX 11156 bridge v2 mongo db support
2023-11-24 16:49:57 +01:00
8f548f4cbb
Merge pull request #12020 from thalesmg/sync-r53-m-20231124
...
chore: sync `release-53` to `master`
2023-11-24 22:54:53 +08:00
c1ef773e77
fix: check for sane state after regular shutdown
2023-11-24 15:09:35 +01:00
4e077c951b
feat(emqx_bridge_mongodb): port mongodb to shared connector and actions
2023-11-24 15:09:35 +01:00
261fe8a831
Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124
2023-11-24 10:10:09 -03:00
2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start
...
Discard sessions when client connects with clean start = true
2023-11-24 13:35:39 +01:00
e95ec5b150
test: fix another flaky test
2023-11-24 09:24:21 -03:00
f8fd95c683
Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124
2023-11-24 09:22:24 -03:00
6a3c438194
Merge pull request #12009 from thalesmg/fix-kafka-get-bridges-r53-20231123
...
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-24 08:56:24 -03:00
9ebbc9bbea
refactor(emqx_bridge): use more simplistic function to validate connector
2023-11-24 10:07:11 +01:00
d2901afd1b
fix(emqx_bridge_kafka): match example in api schema
2023-11-24 10:07:11 +01:00
e616e0746a
feat(ds): Implement a function for dumping persistent session state
2023-11-24 04:39:46 +01:00
8dfcb69e52
feat(ds): Discard session when client connects with CleanStart=1
2023-11-24 04:17:28 +01:00
e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0
...
Support QoS0 messages in persistent_session_ds
2023-11-23 23:43:18 +01:00
449bafc27e
fix(ds): LTS trie handles empty topic levels
2023-11-23 22:16:38 +01:00
a158f25a40
fix(ds): Fix return type of emqx_persistent_session_ds:publish
2023-11-23 22:16:38 +01:00
c5bb86db67
feat(ds): Support QoS 0
2023-11-23 22:16:38 +01:00
1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message
...
feat(ds): Don't store #message record in the DB
2023-11-23 18:56:45 +01:00
1c11a02a76
Merge pull request #12010 from sstrigler/EMQX-11415-e-5-3-2-api-should-return-404-when-the-built-in-database-authentication-doesnt-exist
...
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
2023-11-23 18:08:48 +01:00
c69b82455e
feat(ds): Don't store #message record in the DB
2023-11-23 17:56:35 +01:00
14077ec43b
feat(license): allow setting 'default' license key
2023-11-23 17:52:17 +01:00
caaf8113fc
feat(license): support loading license from file
2023-11-23 17:52:17 +01:00
4597e1c5b9
Merge pull request #12011 from thalesmg/fix-kafka-get-bridges-m-20231123
...
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-23 13:15:39 -03:00
1ced8786fd
feat(ds): Make session poll interval configurable
2023-11-23 17:10:03 +01:00
82e74d0201
feat(ds): Add a flag that forces all sessions to become durable
2023-11-23 17:10:03 +01:00
8ba116d378
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
2023-11-23 16:15:03 +01:00
ce77a12cd3
Merge pull request #12012 from savonarola/1123-fix-callback-types
...
fix(ds): impose more flexible types for ds implementations
2023-11-23 17:03:56 +02:00
fdfa3213cc
fix(bridge_api): don't mangle configs, use correct type as argument
...
Fixes https://emqx.atlassian.net/browse/EMQX-11412
- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 11:41:26 -03:00
7375bc5f9b
fix(license): allow CRLF in license keys
2023-11-23 15:39:18 +01:00
48b69bd60c
chore: delete a TODO from perfectionism
2023-11-23 15:39:18 +01:00
29d42506d5
fix(ds): impose more flexible types for ds implementations
2023-11-23 17:16:11 +03:00
8e8d6d2192
Merge pull request #12006 from thalesmg/test-debug-flaky-test-r53-20231122
...
test(gcp_pubsub_consumer): fix flaky test
2023-11-23 10:57:22 -03:00
c89ec0b1f7
fix(bridge_api): don't mangle configs, use correct type as argument
...
Fixes https://emqx.atlassian.net/browse/EMQX-11412
- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 10:25:36 -03:00
f3693e5dbc
fix(gcp_pubsub_producer): add missing references to api specs
2023-11-23 09:02:52 -03:00
209077593d
Merge pull request #11970 from zmstone/1108-add-v2-schema-dump
...
1108 add v2 schema dump
2023-11-22 23:42:54 +01:00
db83457d13
test: fix flaky test
...
The cause was that the call `sys:terminate/2` was timing out...
`exit/2` doens't always work:
```
2023-11-22 19:14:40.974
killed async workers
Error: -22T19:14:40.974563+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15908.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 28, reductions: 1822; neighbours:
Error: -22T19:14:40.998051+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15908.7>.
2023-11-22T19:15:41.088752+00:00 [critical] Run stage failed: error:{badmatch,{timeout,#{expected_remaining => 1,mailbox => {messages,[]},msgs_so_far => []}}}, Stacktrace: [{emqx_bridge_gcp_pubsub_consumer_SUITE,'-t_async_worker_death_mid_pull/1-fun-17-',3,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1576}]},{emqx_bridge_gcp_pubsub_consumer_SUITE,t_async_worker_death_mid_pull,1,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1505}]}], Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-11-22_19.14.27/snabbkaffe/1700680540975786370.log", mfa: undefined
Error: -22T19:15:46.095702+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15934.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 1471; neighbours:
Error: -22T19:15:46.095192+00:00 [error] Supervisor: {local,ehttpc_sup}. Context: shutdown_error. Reason: killed. Offender: id={ehttpc_pool_sup,<<98,114,105,100,103,101,58,103,99,112,95,112,117,98,115,117,98,95,99,111,110,115,117,109,101,114,58,116,95,97,115,121,110,99,95,119,111,114,107,101,114,95,100,101,97,116,104,95,109,105,100,95,112,117,108,108,45,53,55,54,52,54,48,55,53,50,51,48,51,52,50,50,55,53,49>>},pid=<0.15903.7>.
Error: -22T19:15:46.095470+00:00 [error] Supervisor: {<0.15906.7>,ehttpc_worker_sup}. Context: shutdown_error. Reason: killed. Offender: id={worker,1},pid=<0.15924.7>.
Error: -22T19:15:46.096762+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15934.7>.
Warning: 2T19:15:46.098278+00:00 [warning] msg: remove_local_resource_failed, mfa: emqx_resource:remove_local/1(362), error: {error,timeout}, resource_id: <<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>
Error: -22T19:15:46.149090+00:00 [error] Generic server <0.15904.7> terminating. Reason: killed. Last message: {'EXIT',<0.15903.7>,killed}. State: {state,<<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>,1,random}.
Error: -22T19:15:46.149525+00:00 [error] crasher: initial call: ehttpc_pool:init/1, pid: <0.15904.7>, registered_name: [], exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,909}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.15903.7>,ehttpc_sup,<0.15731.7>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 3428; neighbours:
```
2023-11-22 17:41:32 -03:00
d9f964a44f
test: fix test cases after schema type namespace change
2023-11-22 16:58:05 +01:00
fc849f0c05
ci(test): add info to help diagnose flaky test
2023-11-22 12:36:10 -03:00
6c9417efe0
Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122
2023-11-22 12:02:34 -03:00
e065217e9a
Merge pull request #11997 from zmstone/1121-do-not-allow-special-char-as-prefix-for-resource-name
...
1121 do not allow special char as prefix for resource name
2023-11-22 15:46:11 +01:00
274a254624
Merge pull request #12002 from thalesmg/multiple-action-infos-per-app-m-20231122
...
feat(actions): allow multiple action info modules per application
2023-11-22 10:42:56 -03:00
bb549cdf8b
feat(actions): allow multiple action info modules per application
2023-11-22 09:33:01 -03:00
38d3a1d7d0
feat(actions): allow multiple action info modules per application
2023-11-22 09:30:37 -03:00
db33bc616a
feat(schema): Add v2 scheam JSON dump
2023-11-22 13:12:35 +01:00
1b2c052646
docs: add type namespaces
2023-11-22 13:12:35 +01:00
d9f95cdc56
feat: integrate OpenTelemetry log handler
2023-11-22 13:59:30 +02:00
a1b9a14fa1
feat(ds): allow fdb implementation for durable storage
2023-11-22 14:49:38 +03:00
3261a12140
fix(emqx_resource): do not allow leading _ or - as resource name
2023-11-22 10:58:54 +01:00
ee191803ea
Merge pull request #11990 from ieQu1/dev/ds-shards
...
Implement a prototype of durable message sharding
2023-11-22 00:57:54 +01:00
869e73d637
Merge pull request #11966 from thalesmg/fix-kafka-parameters-r53-20231117
...
fix(kafka): don't return `parameters` from `/bridges` API
2023-11-22 00:36:39 +01:00
4fb2d3a503
Merge pull request #11996 from zmstone/1121-prepare-for-e5.3.2-alpha.1
...
chore: bump release version to e5.3.2-alpha.1
2023-11-21 22:01:05 +01:00
39791511fc
chore: remove obsolete workaround
2023-11-21 17:38:13 -03:00
01895fd660
Merge pull request #11995 from thalesmg/fix-action-api-bad-name-r53-20231121
...
fix(actions_api): don't crash on validation errors
2023-11-21 17:27:38 -03:00
5ebd954b16
chore: bump release version to e5.3.2-alpha.1
2023-11-21 21:19:05 +01:00
8fbdcab118
Merge pull request #11987 from zmstone/1120-do-not-crash-on-einval-after-check-cache
...
fix(emqx_connection): handle socket activation error return
2023-11-21 20:31:56 +01:00
3a8c332805
fix(actions_api): don't crash on validation errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-11394
2023-11-21 16:27:54 -03:00
3165b4f645
fix(ds): Abort application startup when rocksdb is not avialable
2023-11-21 20:27:07 +01:00
11ec1a30a0
test(flaky): fix flaky pulsar test
2023-11-21 16:00:19 -03:00
b3dffa4390
fix(kafka): don't return `parameters` from `/bridges` API
...
Fixes https://emqx.atlassian.net/browse/EMQX-11412
2023-11-21 16:00:15 -03:00
3d823beb11
fix(ds): Apply review remarks
2023-11-21 19:37:01 +01:00
4d47490734
chore(ds): Rebase configuration
2023-11-21 19:37:01 +01:00
f5c71e8068
refactor(ds): Add a wrapper to the store batch API
2023-11-21 19:37:01 +01:00
2a1f7d946a
feat(ds): Shard messages by publisher client ID
2023-11-21 19:37:01 +01:00
62542e5844
feat(ds): Metadata storage for the replication layer
2023-11-21 19:37:01 +01:00
b0d670aaa9
Merge pull request #11992 from kjellwinblad/kjell/backport/action_upgrade_downgrade_hooks
...
fix(action): upgrade and downgrade strategy
2023-11-21 19:14:37 +01:00
9e1796ec4f
feat(gcp_pubsub_producer): migrate GCP PubSub producer to actions
...
Fixes https://emqx.atlassian.net/browse/EMQX-11157
2023-11-21 14:22:42 -03:00
fa91bacdfb
Merge pull request #11975 from zmstone/1119-fix-socket-close-race-condition
...
fix(emqx_channel): do not log stale sock_close event as error
2023-11-21 16:47:03 +01:00
fa7151f255
fix: port emqx_utils_maps:rename function from master
...
The emqx_utils_maps:rename function is needed by action
upgrade/downgrade hoos.
2023-11-21 15:18:22 +01:00
6c9a8461f7
Merge pull request #11989 from thalesmg/confluent-action-m-20231120
...
feat: add confluent connector/action
2023-11-21 11:01:40 -03:00
cd72dc11dd
fix: missing emqx_action_info module mapping
2023-11-21 14:22:29 +01:00
068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120
...
sync `release-53` to `master`
2023-11-21 09:02:51 -03:00
759468efcf
Merge pull request #11986 from thalesmg/split-resource-opts-actions-r53-20231120
...
refactor: split `resource_opts` fields between connector and actions
2023-11-21 09:01:29 -03:00
75ee3ced68
feat: add confluent connector/action
...
Fixes https://emqx.atlassian.net/browse/EMQX-11044
See also: https://emqx.atlassian.net/wiki/spaces/EMQX/pages/712179934/Confluent+Integration
2023-11-21 08:59:42 -03:00
6030bf6fa5
fix(action): upgrade and downgrade strategy
...
This commit adds upgrade and downgrade hooks that are called when
upgrading from a bridge V1 to connector and action or the other way
around. The automatic translation is used if the callback is not
defined.
NOTE: Backported from master
2023-11-21 11:18:21 +01:00
c6fd1e4c75
Merge pull request #11971 from savonarola/1116-no-auth-rebalance-status
...
feat(rebalance): improve rebalance usability
2023-11-21 12:10:18 +02:00
cd48c283db
Merge pull request #11988 from keynslug/ft/EMQX-11222/basic-config
...
feat(schema): introduce separate root for new session persistence
2023-11-21 15:02:21 +07:00
9716b051d9
fix(ocpp): fix bad webscoket schema for HTTP API
2023-11-21 11:24:23 +08:00
6f8630304d
Merge pull request #11980 from kjellwinblad/kjell/fix_fixup_callbacks/EMQX-11428
...
fix: bridge to action upgrade fixup hook should run after upgrade
2023-11-20 16:19:43 -03:00
7fb5ade832
fix(kafka_producer): hocon renames kafka field
2023-11-20 19:20:13 +01:00
d214ae8772
fix: problems found by @thalesmg in code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-20 18:59:57 +01:00
110a5a4896
feat(schema): introduce separate root for new session persistence
...
With some knobs to choose a storage backend. Support only builtin
RocksDB-based backend with minimal configuration for now.
2023-11-20 23:49:14 +07:00
b02711af79
refactor(emqx_ws_connection): rename cache to buffer for limiter
2023-11-20 17:47:20 +01:00
8e107ffe45
Merge pull request #11958 from keynslug/ft/EMQX-9745/preserve-acks
...
feat(sessds): preserve acks / replays in session state
2023-11-20 23:40:22 +07:00
edbfe090c9
feat: add custom callback for creating connector from bridge V1 config
2023-11-20 17:23:51 +01:00
ec19247271
refactor: rename limiter buffer related messages and var names
2023-11-20 16:55:26 +01:00
9684e79ee0
fix(sessds): ensure dup flag is on for replayed messages
2023-11-20 22:54:40 +07:00
8ec3b1db5d
fix(emqx_connection): handle socket activation error return
2023-11-20 16:51:39 +01:00
3aa8044475
fix(action): upgrade and downgrade strategy
...
This commit fixes the upgrade and downgrade strategy when upgrading
from a bridge V1 to connector and action or the other way around so that
the custom callbacks get the complete unchanged input instead of the
result of the automatic translation. The automatic translation is used
if the callback is not defined.
2023-11-20 16:40:27 +01:00
f40f6bc5dd
refactor: split `resource_opts` fields between connector and actions
2023-11-20 12:37:36 -03:00
1f1d9e58c6
fix(emqx_connector): don't crash in API on delete with active channels
2023-11-20 16:23:46 +01:00
53e796bbd0
Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120
2023-11-20 11:49:52 -03:00
ef7cfd0202
feat(sessds): add field to `ds_pubrange` for forward-compat
2023-11-20 19:56:55 +07:00
b42d4e5ab0
Merge pull request #11950 from thalesmg/fix-bridge-v1-examples-r53-20231114
...
fix(bridges_v1): avoid merging action examples for non-v1 bridges
2023-11-20 09:12:23 -03:00
926078b82c
Merge pull request #11960 from thalesmg/fix-kafka-producer-channel-status-r53-20231116
...
fix(kafka_producer): make status `connecting` while the client fails to connect
2023-11-20 09:12:05 -03:00
79a764f117
fix: bridge to action upgrade fix up hook should run after upgrade
...
This commit changes how the `emqx_action_info` callback
`bridge_v1_to_action_fixup/1` works. It is now called after the
automatic upgrade instead of before. Since the full Bridge V1 config
might be needed to do the fixup, it is provided in a special field
`<<"__bridge_v1_conf__">>`. The `<<"__bridge_v1_conf__">>` field is
removed after the callback is called and can thus be ignored if it is
not needed.
2023-11-20 12:06:21 +01:00
84ff7b0b38
feat(emqx_bridge): action_info with dynamic lookup
...
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
d7974d835f
fix(sharesub): anticipate messages w/o redispatch header
...
For instance, `emqx_session_mem` will push almost all of the messages
that are still in the state to `emqx_shared_sub:redispatch/1`. After
this commit, the session will no longer crash during channel terminate.
2023-11-20 15:53:24 +07:00
7081f1951f
refactor(sessds): use `ds_pubrange` record as is
...
Instead of converting it into almost similar runtime representation.
2023-11-20 15:40:43 +07:00
a5ff4144fe
test(sessds): add complex testcase for session replay
2023-11-20 15:40:43 +07:00
1246d714c5
feat(sessds): preserve acks / ranges in mnesia for replays
2023-11-20 13:33:48 +07:00
5b40304d1f
chore(sessds): simplify subscriptions handling
...
There's currently no point in storing parsed topic filters in the
subscriptions table.
2023-11-20 13:33:48 +07:00
648b6ac63e
chore(sessds): rename iterators -> subscriptions
...
Also try to make clearer the difference between 2 flavors of topic
filter representation in use.
2023-11-20 13:33:48 +07:00
e73bf716ae
fix(emqx_channel): do not log stale sock_close event as error
...
In some cases, EMQX may decide to close socket and mark connection
at 'disconnected' state, for example, when DISCONNECTE packet is
received, or, when failed to write data to socket.
However, by the time EMQX decided to close the socket, the socket
might have already been closed by peer, and the `tcp_closed` envet
is already delivered to the process mailbox -- causing EMQX to
handle sock_close event at 'disconnected' state.
2023-11-19 22:27:25 +01:00
e93e9ed108
feat(rebalance): improve rebalance usability
...
* make availability API endpoint public
* allow connections during wait_health_check interval
* make availability status calculation more consistent and lightweight
* refactor test to get rid of some mocks and to use cth
2023-11-18 21:29:16 +03:00
eb3f54184e
refactor: address review comments and avoid transformations without schema knowledge
2023-11-17 16:16:34 -03:00
86c126ffcd
feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade
...
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
2023-11-17 16:10:08 -03:00
9feba802e9
chore: add convenience function for creating action schemas
2023-11-17 16:10:08 -03:00
58437cd35a
fix(mongodb): fix deadlock while stopping mongodb resource
2023-11-17 12:34:36 +03:00
b2e168d508
feat: add prometheus.push_gateway.enable config
2023-11-17 09:48:39 +08:00
c4eb9f86e7
chore: add refactor prometheus refactor changelog
2023-11-17 08:23:51 +08:00
b343653769
chore: support list(tuple()) for schema_with_examples/3
2023-11-17 08:23:51 +08:00
0c2610fa61
feat: support prometheus.enable_basic_auth
2023-11-17 08:23:51 +08:00
c376a5db29
refactor: prometheus config
2023-11-17 08:23:51 +08:00
b92821188b
fix(kafka_producer): make status `connecting` while the client fails to connect
...
Fixes https://emqx.atlassian.net/browse/EMQX-11408
To make it consistent with the previous bridge behavior.
Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
4c54efd8b1
fix(auth): fix deadlock while stopping mongodb resource
2023-11-16 19:29:08 +03:00
36b5d58957
test: reorganize test suite a bit
2023-11-16 13:16:02 -03:00
90571b7d8e
test: fix noise about undefined unofficial callbacks
2023-11-16 13:16:02 -03:00
656d575e2d
ci(router): fix flaky testcase
2023-11-16 17:53:20 +07:00
d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default
...
feat(router): switch to v2 routing store by default
2023-11-16 15:56:26 +07:00
893e90b372
fix(kafka): use safe publish in consumer
...
Routing with v2 schema is actually more strict with respect to input to
`emqx_router` module routines. This causes Kafka consumer bridge to
crash when it tries to publish a message to a topic that looks like a
topic filter.
2023-11-15 22:37:00 +07:00
98f947f4f3
ci(router): fix flaky testcase
2023-11-15 22:28:52 +07:00
8919b08207
fix(utils): rename `emqx_utils_stream:take/2` to `consume/2`
...
Which is more neutral and harder to confuse with a destructive `take` in
collections.
2023-11-15 17:20:40 +07:00
65ba381fd8
Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53
2023-11-15 09:25:03 +01:00
f5545bbdf3
fix(schema_types): add missing `secret()` type
2023-11-14 16:08:13 -03:00
5275a01727
fix(syskeeper): refactor action to new action info scheme
2023-11-14 19:49:28 +01:00
d2cf29fae4
fix(schema): use `hoconsc:union/1`
2023-11-14 19:49:08 +01:00
423fb32171
test: fix noise about undefined unofficial callbacks
2023-11-14 19:48:39 +01:00
7c0e345d3a
Merge remote-tracking branch 'upstream/release-54'
2023-11-14 19:38:21 +01:00
2f1d88d414
fix(bridges_v1): avoid merging action examples for non-v1 bridges
...
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
2023-11-14 13:56:50 -03:00
88637f81d1
Merge pull request #11936 from zmstone/1112-readable-types
...
refactor(schema): keep type converters close
2023-11-14 17:56:07 +01:00
124c0e2dba
fix(ds): Respect receive_maximum from the connection info
2023-11-14 16:06:19 +01:00
bbf00c2f71
Merge pull request #11946 from id/1114-sync-r53
...
sync release-53 to master
2023-11-14 13:38:24 +01:00
01a2a3b1c0
Merge pull request #11930 from thalesmg/ds-session-discard-m-20231110
...
feat(ds): implement session discard
2023-11-14 09:30:34 -03:00
d6e9bbb95c
fix(connector): validate connector name before converting ssl certs
...
Fixes https://emqx.atlassian.net/browse/EMQX-11336
See also: https://github.com/emqx/emqx/pull/11540
2023-11-14 09:29:59 -03:00
6812ee9d0f
fix(mgmt): hide route selection behind router interface
...
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
2023-11-14 18:36:25 +07:00
28a577ad09
chore: bump apps versions
2023-11-14 11:02:26 +01:00
1c57993c91
Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53
2023-11-14 10:44:35 +01:00
d1c3b1c659
fix(bridges): handle secrets more consistently
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 16:05:54 +07:00
d4dd4508c9
feat(greptimedb): accept wrapped secrets as passwords
2023-11-14 16:05:53 +07:00
5d620465e2
feat(influx): accept wrapped secrets as passwords
2023-11-14 16:05:53 +07:00
7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords
2023-11-14 16:05:53 +07:00
7817502b8b
feat(rocketmq): accept wrapped secrets as passwords
2023-11-14 16:05:53 +07:00
2449d54b1f
feat(pulsar): accept wrapped secrets as passwords
2023-11-14 16:05:52 +07:00
e2b7b33d14
feat(kinesis): accept wrapped secrets as passwords
2023-11-14 16:05:52 +07:00
93eaf0caee
feat(dynamo): accept wrapped secrets as passwords
2023-11-14 16:05:52 +07:00
7250059c60
feat(sqlserver): accept wrapped secrets as passwords
2023-11-14 16:05:52 +07:00
06861e377f
feat(ldap): accept wrapped secrets as passwords
2023-11-14 16:05:52 +07:00
fda395014c
test(kafka): verify file-based secrets work
2023-11-14 16:05:25 +07:00
aa458b65d6
test(kafka): simplify consumer testsuite matrix setup
2023-11-14 16:05:25 +07:00
18cd98def6
chore(test): fix formatting quirks
2023-11-14 16:05:25 +07:00
c300eb41a7
feat(kafka): accept wrapped secrets as passwords
2023-11-14 16:05:25 +07:00
9c5856029f
feat(cassa): accept wrapped secrets as passwords
2023-11-14 16:05:24 +07:00
8b4ac8eb4f
feat(oracle): accept wrapped secrets as passwords
2023-11-14 16:05:24 +07:00
34aeeab041
feat(rabbitmq): accept wrapped secrets as passwords
2023-11-14 16:05:24 +07:00
a69a78d024
feat(clickhouse): accept wrapped secrets as passwords
2023-11-14 16:05:24 +07:00
fc340a276e
feat(mongo): accept wrapped secrets as passwords
...
Also test authorization with mongo in bridge / auth test suites.
2023-11-14 16:05:24 +07:00
f827df2821
test(mysql): slightly simplify bridge testsuite
2023-11-14 16:05:23 +07:00
36e57a479d
feat(mysql): accept wrapped secrets as passwords
2023-11-14 16:05:23 +07:00
ad72ab7a8c
feat(redis): accept wrapped secrets as passwords
2023-11-14 16:05:23 +07:00
4385b2f020
feat(pgsql): accept wrapped secrets as passwords
...
That are coming from `emqx_schema_secret`. Also adapt pgsql-related
connectors.
2023-11-14 16:05:23 +07:00
2b36b48f98
feat(connector): use `emqx_schema_secret` as password field schema
2023-11-14 16:05:23 +07:00
a49aea3b56
chore: bump app versions
2023-11-14 09:27:04 +01:00
d682e6e23c
test: fix test mock by calling exported function
2023-11-14 09:20:46 +01:00
3bea3496af
test: fix test case mock that broke due to removal
2023-11-14 09:20:46 +01:00
093c8b0c6e
docs: add missing copyright headers
2023-11-14 09:20:46 +01:00
ca3e5eab11
refactor: remove emqx_bridge_v2_enterprise.erl
2023-11-14 09:20:46 +01:00
e93b71d8d5
fix: problems found by @thalesmg in code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 09:20:46 +01:00
49fdfef8c3
fix: azure event hub names are defined in one place
2023-11-14 09:20:46 +01:00
5e8e407017
refactor: action schema retrival after PR feedback
2023-11-14 09:20:46 +01:00
ab078647a5
chore: fix formatting problem
2023-11-14 09:20:46 +01:00
bdab421885
fix(bridge_v2): name and copyright header
2023-11-14 09:20:46 +01:00
ca2cdbc08d
fix: bapi static check error
2023-11-14 09:20:46 +01:00
7839f0cbc8
chore(bridge_v2): make dialyzer and bpapi happy
2023-11-14 09:20:46 +01:00
d26a1b9afb
fix(bridge_v1): no hard coded downgrade and upgrade type functions
2023-11-14 09:20:46 +01:00
a6aa81b548
fix(emqx_bridge_v2): dialyzer found bad format parameter error
2023-11-14 09:20:46 +01:00
3c778121a5
fix: bridge V1 type lookup issue
2023-11-14 09:20:46 +01:00
77aaff137a
test(emqx_bridge_v2_SUITE): fix incorrect assumption about return type
...
This commit fixes an incorrect assumption about the return type of
emqx_bridge_v2:lookup/2 in emqx_bridge_v2_SUITE:t_load_config_success/1.
2023-11-14 09:20:46 +01:00
14e305e2a6
fix(emqx_bridge_v2): xref warning
2023-11-14 09:20:46 +01:00
9eaee8f333
refactor(emqx_bridge_v2): make independent of Kafka
...
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
2023-11-14 09:20:46 +01:00
cd5b1f9b96
docs(bridge_V2): type specs for operations
2023-11-14 09:20:46 +01:00
99031f0dae
refactor(bridge_v2): prefix compatibility functions with bridge_v1
2023-11-14 09:20:46 +01:00
729e7df0d5
docs(bridge_v2): add specs for CRUD functions
2023-11-14 09:20:46 +01:00
a9cf8fe78c
chore: 5.3.1
2023-11-14 08:44:41 +01:00
ab1f3b6b2f
fix(emqx_conf_schema_types): address review comments
2023-11-13 21:55:03 +01:00
fc2891d6de
Merge pull request #11937 from sstrigler/EMQX-11318-crash-in-get-bridges-v-2-if-a-broken-bridge-is-configured
...
Emqx 11318 crash in get bridges v 2 if a broken bridge is configured
2023-11-13 17:28:56 +01:00
4c5d64abc2
refactor(schema): keep type converters close
2023-11-13 15:26:30 +01:00
17b18849de
test(emqx_bridge): test broken config
2023-11-13 14:52:06 +01:00
9b9908780c
Merge pull request #11926 from kjellwinblad/kjell/fix_bridge_v2_to_action_cluster_hocon/EMQX-11353
...
fix: reflect bridge V1 upgreade in cluster.hocon
2023-11-13 14:18:57 +01:00
0b3645057d
fix(emqx_bridge): don't try to handle what we don't understand
2023-11-13 14:08:54 +01:00
0d33284672
fix(emqx_bridge): don't crash uninstallling broken action
2023-11-13 14:08:54 +01:00
4fe1d332b3
fix(emqx_bridge): don't crash if there's no status field
2023-11-13 14:08:54 +01:00
45dad2ed3a
feat(ds): implement session discard
...
Fixes https://emqx.atlassian.net/browse/EMQX-9739
Fixes some issues to ensure the session is discarded when the client connects with
`clean_start = true`, and added some cleanup to subscriptions/routes/iterators/streams.
> There is an API that session garbage collector can use to perform cleaning
We already have `emqx_session:destroy/1`, which could serve as an API for a periodic
session GC to use.
2023-11-13 09:06:22 -03:00
26fc9ed89d
fix: log warning message after suggestion from @zmstone
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-11-13 12:53:09 +01:00
03843c6071
feat(router): switch to v2 routing store by default
2023-11-13 17:43:30 +07:00
5838568d5d
fix: hard coded call to schema module
...
The user can configure the configuration loader and schema module so we
should not hard code a call to the default schema module.
2023-11-13 11:41:40 +01:00
518b02fc70
Merge pull request #11931 from zmstone/1110-delete-stale-type-converters
...
1110 delete stale type converters
2023-11-13 10:27:51 +01:00
65efa2672e
test(schema): delete stale reference to emqx_schema:file() type
2023-11-13 09:28:26 +01:00
2fda91ec59
refactor: delete default type converter for dashboard schema json
...
prior to this commit, dashboard scheam json type conversion
has a 'fallback' type for all unknown types, which is 'string'
this commit removes this fallback and add all possible types
to the function clauses.
also, some of the old types which are no longer in use
are deleted from both swagger spec converter and dashboard
sechema converter
2023-11-12 20:08:53 +01:00
952ab08991
fix: make dialyzer happy
2023-11-12 21:44:24 +08:00
20bf9ed26d
fix(gw_gbt32960): mountpoint should not with `/` suffix
2023-11-12 21:44:23 +08:00
867d5aaea6
chore: rm dbg funcs in SUITE
2023-11-12 21:44:23 +08:00
70fcfdddcb
test(gw_jt808): fix test auth server bootstrap
2023-11-12 21:44:23 +08:00
dc64068e41
test(gw_jt808): frame parser and channel SUITE
2023-11-12 21:44:23 +08:00
4d493292fb
feat(gw_jt808): port jt808 gateway from 4.4
2023-11-12 21:44:18 +08:00
526faa0362
fix(gw_gbt32960): clean frame parser state acc data
2023-11-12 21:43:12 +08:00
d30f0aa4a0
fix(gw_gbt32960): parse state atom key error
2023-11-12 21:43:11 +08:00
a4da1d260a
docs(gw_jt808): data exchange guide cn
2023-11-12 21:43:11 +08:00
17edeead6e
fix: erlang_ls complains gateway hrl file can not be found
2023-11-12 21:43:11 +08:00
a218ff6dd8
fix(gw_gbt32960): undefined type `emqx_gbt32960_frame:frame()`
2023-11-12 21:43:11 +08:00
aef5b17ed5
fix(gw_lwm2m): lwm2m channel behaviour
2023-11-12 21:43:11 +08:00
caade4a7e5
chore: emacs mode entry comment for non-erl files
2023-11-12 21:43:11 +08:00
25e277bdc6
chore: fix schema desc and README.md typo
2023-11-12 21:43:10 +08:00
523bb17fc8
test: mark proper types gateway connection modules need be fixed
2023-11-12 21:43:10 +08:00
b7d2c38213
refactor: delete stale type converters
...
Some of the types are refactored to use typerefl alias
or hoconsc:enum, no need to keep the special function
clauses to translate them to swagger spec
2023-11-12 11:39:44 +01:00
a32cd20758
refactor(emqx_authn_jwt_schema): add alias for verify_claims
...
verifiy_claims is a list after converted, but input is a map
2023-11-12 11:39:44 +01:00
021f7e6b49
refactor(schema): comma_separated_list is list(string()) not list(any())
2023-11-12 11:37:58 +01:00
06e440260f
refactor(schema): add args to map alias
2023-11-12 11:29:33 +01:00
8be718b22f
refactor(emqx_schema): delete unused type bar_separated_list
2023-11-12 11:29:33 +01:00
ff5916e33f
docs: fix bad html tag
2023-11-12 10:31:37 +01:00
7bb3ca5519
Merge pull request #11909 from emqx/bugfix-plugins
...
fix: emqx failed to start if plugin dir is removed by the user
2023-11-11 10:51:26 +08:00
b4e45f0189
Merge pull request #11923 from thalesmg/sync-m-to-r54-20231109
...
chore: sync `master` to `release-54`
2023-11-10 13:04:55 -03:00
088de9476c
fix(ds): use integer for tag values
...
Follow up to https://github.com/emqx/emqx/pull/11906#discussion_r1389115973
2023-11-10 11:26:32 -03:00
5170267fef
Merge pull request #11924 from ieQu1/dev/emqx-ds-behavior
...
feat(ds): Introduce emqx_ds behavior
2023-11-10 15:23:28 +01:00
6ebf0d1bdd
fix: reflect bridge V1 upgrade in override configurations
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11353
2023-11-10 14:52:29 +01:00
101990b8fc
test: fix schema tests after types change
2023-11-10 14:23:43 +01:00
e653c6b4e0
chore: bump app versions
2023-11-10 13:41:51 +01:00
855b3c5b29
test: ensure atom exists
2023-11-10 13:41:51 +01:00
f1de0aa176
fix(schema): add namespace to authn schemas
2023-11-10 13:41:51 +01:00
86110824eb
feat: upgrade hocon to 0.40.0 which supports union type display name
2023-11-10 13:41:51 +01:00
8a4fba431e
refactor(emqx_limiter_schema): use typerefl alias
2023-11-10 13:41:51 +01:00
d603de10e6
refactor(schema): add namespace/0 to all schema modules
2023-11-10 13:41:51 +01:00
7b59d46854
fix(schema): add namesapce to sso schema
2023-11-10 13:41:51 +01:00
6b3aaf5b24
refactor(emqx_bridge_gcp_pubsub): use typerefl alias
2023-11-10 13:41:51 +01:00
b24b66081a
refactor(authn/authz_http_schema): use typerefl alias
2023-11-10 13:41:51 +01:00
2ade6da7a6
fix(ds): Remove redundand type
2023-11-10 12:53:30 +01:00
feef23fc08
feat(ds): Pass store_batch through RPC
2023-11-10 12:16:39 +01:00
8dc8237331
feat(ds): Introduce emqx_ds behavior
2023-11-10 12:16:39 +01:00
0fc4bcaf0d
Merge pull request #11912 from qzhuyan/test/william/quic-multistream-flaky
...
test(quic): fix flaky
2023-11-10 11:13:35 +01:00
d1623f491c
Merge pull request #11925 from lafirest/docs/gbt
...
chore(gbt32960): add EN documentation of the data exchange guide
2023-11-10 16:43:49 +08:00
3e2ce57cb1
refactor(emqx_bridge_greptimedb): use typerefl alias
2023-11-10 08:45:20 +01:00
46fffa401e
refactor(emqx_prometheus_schema): use typerefl alias
2023-11-10 08:45:20 +01:00
ae24b45d1b
fix(schema): add type namespce to emqx_dashboard_sso_ldap
2023-11-10 08:45:20 +01:00
a2700771c9
refactor(emqx_s3_schema): make type readable
2023-11-10 08:45:20 +01:00
b3df8604fc
refactor(emqx_conf_schema): use hocon enum type for log levels
2023-11-10 08:45:20 +01:00
df96798c53
refactor(http_bridge): use more readable type for pool_type
2023-11-10 08:45:20 +01:00
40c2ca134d
refactor(schema): delete unused type cipher()
2023-11-10 08:45:20 +01:00
7a0a0bee98
refactor(schema): change file() type to string()
2023-11-10 08:45:20 +01:00
af3dfc91b1
refactor(schema): change log_lvel to enum type
2023-11-10 08:45:20 +01:00
3d20e566d8
docs: add namespace to authn and authz schema
2023-11-10 08:45:20 +01:00
0299e9033a
chore(gbt32960): add EN documentation of the data exchange guide
2023-11-10 15:22:28 +08:00
9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109
...
fix(actions_api): change bridge v2 occurrences to action
2023-11-10 07:35:35 +01:00
447c933ff8
fix(syskeeper): homogenize the Syskeeper bridge and update license data
2023-11-10 10:42:53 +08:00
38d004e06e
Merge pull request #11913 from ieQu1/dev/fix-hooks-comment
...
chore(hooks): Fix comment
2023-11-10 03:15:54 +01:00
548e39e799
refactor(sysk): refactor the syskeeper to v2 style
2023-11-10 09:52:38 +08:00
abc7233a77
test(sysk): add test suite for syskeeper
2023-11-10 09:52:38 +08:00
e93216fa62
chore(sysk): fix spellchecks & update change & more logs
2023-11-10 09:52:32 +08:00
ce83079c6b
feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend
2023-11-10 09:51:43 +08:00
6500d21d98
Merge pull request #11885 from HJianBo/ocpp-gw
...
feat: Port OCPP gateway from v4
2023-11-10 09:32:40 +08:00
ced7be61f0
Merge pull request #11917 from ieQu1/dev/fix-packet-id-to-seqno-trans
...
fix(ds): Fix packet id -> sequence number translation
2023-11-09 22:57:16 +01:00
7a237f6a3a
Merge branch 'master' into sync-m-to-r54-20231109
2023-11-09 18:49:52 -03:00
371a49304d
Merge branch 'release-53' into sync-r53-to-m-20231109
2023-11-09 17:19:23 -03:00
5aa9d026df
fix(ds): Apply review remarks
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-09 21:16:59 +01:00
b812db1e3c
fix(ds): Fix packet id -> sequence number translation
2023-11-09 20:12:39 +01:00
2d539463cd
test: fix flaky test
...
```
Testing lib.emqx.emqx_cm_SUITE: *** FAILED test case 24 of 31 ***
%%% emqx_cm_SUITE ==> t_stepdown_session_takeover_begin_timeout: FAILED
%%% emqx_cm_SUITE ==>
Failure/Error: ?assertEqual([<0.6635.8>,<0.6520.8>], lists : sort ( emqx_cm : lookup_channels ( ClientId ) ))
expected: [<0.6635.8>,<0.6520.8>]
got: [<0.6520.8>,<0.6635.8>]
line: 309
```
2023-11-09 15:32:52 -03:00
ee87f90b0d
Merge pull request #11906 from thalesmg/ds-avoid-records-m-20231108
...
chore(ds): avoid using records in persistence / rpc
2023-11-09 14:25:27 -03:00
7977ae95c0
fix(actions_api): change bridge v2 occurrences to action
...
Fixes https://emqx.atlassian.net/browse/EMQX-11338
2023-11-09 14:20:08 -03:00
eabd09051a
feat(actions_api): add `/action_types` API
2023-11-09 13:54:57 -03:00
42a9f0eb6b
Merge pull request #11914 from ieQu1/dev/speed-up-poll
...
fix(ds): Speed up polling for the new messages
2023-11-09 16:40:06 +01:00
42536bb5f7
chore(ds): avoid using records in persistence / rpc
...
Fixes https://emqx.atlassian.net/browse/EMQX-11279
2023-11-09 12:00:25 -03:00
cf4a46a78b
fix(ds): Speed up polling for the new messages
...
Poll immediately if the previous poll returned non-empty result
2023-11-09 14:22:32 +01:00
3350bda446
chore(hooks): Fix comment
2023-11-09 14:05:13 +01:00
2b97800c8b
fix(limiter): simplify the descriptions for fields of the limiter
2023-11-09 18:00:09 +08:00
f7a2359329
test(quic): fix flaky
2023-11-09 10:56:16 +01:00
d55f1e0813
chore: fix mix compiling failures
2023-11-09 16:54:41 +08:00
e0f4a2014d
fix: emqx failed to start if plugin dir is removed by the user
2023-11-09 16:01:20 +08:00
0bf5deaed9
chore: merge master into release-54
2023-11-09 08:32:26 +01:00
d6104b37cf
test: fix flaky tests
2023-11-09 15:01:12 +08:00
4b18631d49
chore: fix gateway failed test cases
2023-11-09 13:44:20 +08:00
2d958beae9
chore: try to fix spellcheck
2023-11-09 09:51:08 +08:00
42fa1289a1
Merge pull request #11905 from thalesmg/tag-e531-alpha5-r53-20231108
...
chore: prepare to tag `e5.3.1-alpha.5`
2023-11-08 23:39:33 +01:00
f95058a3e7
Merge pull request #11897 from zmstone/1107-fix-cluster-conf-sync-wait-loop
...
1107 fix cluster conf sync wait loop
2023-11-08 23:37:09 +01:00
1e3500ffd2
fix(emqx_conf_app): ensure log data fields
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-08 21:19:22 +01:00
8540566eba
chore: prepare to tag `e5.3.1-alpha.5`
2023-11-08 13:02:19 -03:00
f9e9748cec
fix(cluster-rpc): boot from local config if table loaded
...
When EMQX boots up, it tries to get latest config from peer (core type)
nodes, if none of the nodes are replying, the node will decide
to boot with local config (and replay the committed changes) if
the commit table is loaded from disk locally (an indication of the
data being latest), otherwise it will sleep for 1-2 seconds and
retry.
This lead to a race condition, e.g. in a two nodes cluster:
1. node1 boots up
2. node2 boots up and copy mnesia table from node1
3. node1 restart before node2 can sync cluster.hocon from it
4. node1 boots up and copy mnesia table from node2
Now that both node1 and node2 has the mnesia `load_node` pointing
to each other (i.e. not a local disk load).
Prior to this fix, the nodes would wait for each other in a dead loop.
This commit fixes the issue by allowing node to boot
with local config if it does not have a lagging.
2023-11-08 15:06:47 +01:00
788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session
...
Refactor emqx_durable storage application and introduce learned topic structure storage
2023-11-08 14:27:40 +01:00
f5456135aa
fix(bridge_v2_api): take status and error from bridge, not the connector
...
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-08 09:53:42 -03:00
d061d64c70
fix(plugins): fix backward compatibility
2023-11-08 12:36:09 +03:00
0c91bec98d
chore: merge 'upstream/release-53'
2023-11-08 09:24:38 +01:00
a1cdbaa76d
fix(ds): Address code review remarks
2023-11-08 04:31:27 +01:00
c030188eb7
chore(ds): Add rebar.config file to app/emqx_durable_storage
2023-11-08 03:04:19 +01:00
7cb0322856
fix(emqx): Move bpapi and emqx message record to emqx_utils app
2023-11-08 03:04:19 +01:00
74cb43f8b1
fix(ds): Add unique ID to the key
2023-11-08 03:04:19 +01:00
54951c273f
feat(ds): mix safe cutoff into keymapper filter during iteration
2023-11-08 03:04:19 +01:00
7a94db25c3
fix(ds): don't iterate over incomplete epoch in bitmask lts storage
2023-11-08 03:04:19 +01:00
46d8301bc0
feat(emqx): expose timestamp function in `emqx_message`
...
So that the code that relies on it would not need to guess clock source
and precision.
2023-11-08 03:04:19 +01:00
38b7f516bc
fix(ds): Change the directory where message database is stored
2023-11-08 03:04:19 +01:00
8e5dda40be
fix(ds): Fix static checks
2023-11-08 03:04:19 +01:00
99329e1243
refactor(ds): Address review remarks
2023-11-08 03:04:19 +01:00
2de79dd9ac
feat(ds): Replay QoS1 messages
2023-11-08 03:04:19 +01:00
e745e42093
test(ds): Explore full range of keys when testing ratchet function
2023-11-08 03:04:19 +01:00
87689890ff
chore(ds): Fix linter and compilation warnings
2023-11-08 03:04:19 +01:00
465e8a90dd
revert(ds): Remove change from the old protocol file
2023-11-08 03:04:19 +01:00
ef46c09caf
feat(ds): Implement ratchet function for bitmask keymapper
2023-11-08 03:04:19 +01:00
164ae9e94a
feat(ds): LTS bitfield storage passes all tests
2023-11-08 03:04:19 +01:00
56b6b176c2
fix(ds): LTS shall keeps the concrete topic indexes
2023-11-08 03:04:19 +01:00
7428e7037b
feat(ds): Bitfield + Learned Topic Structure
2023-11-08 03:04:19 +01:00
ac91dbc58f
feat(ds): Restore LTS trie from a dump
2023-11-08 03:04:19 +01:00
f1ab7c8a7c
feat(ds): Add persist callback to LTS trie
2023-11-08 03:04:19 +01:00
c149e0e2df
fix(ds): Pass topic filter to emqx_ds:make_iterator call
2023-11-08 03:04:19 +01:00
51a6f623fd
refactor(ds): Split out bitfield keymapper to a different module
2023-11-08 03:04:19 +01:00
903b3863d1
chore(ps_ds): make persistent session module use new `emqx_ds` APIs
2023-11-08 03:04:19 +01:00
2972bf14ee
refactor(ds): Implement create_generation gen_rpc storage layer call
2023-11-08 03:04:19 +01:00
6d65707d41
refactor(ds): Implement drop_db function
2023-11-08 03:04:19 +01:00
c6a721a7eb
refactor(ds): Passthrough open_db and get_channels to storage layer
2023-11-08 03:04:19 +01:00
59d01dc823
refactor(ds): Implement emqx_ds:open_db
2023-11-08 03:04:19 +01:00
7095cb8583
refactor(ds): Refactor storage layer
2023-11-08 03:04:19 +01:00
c91df2f5cd
refactor(ds): Create a prototype of replication layer
2023-11-08 03:04:19 +01:00
f1454bb57e
feat(ds): learned topic structure
2023-11-08 03:04:19 +01:00
7ab57824dc
chore(ds): change return type of `storage_layer:next/{1,2}`
...
Part of https://emqx.atlassian.net/browse/EMQX-10942
The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.
From:
```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```
To:
```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.
-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
75704513d8
fix(plugins): fix backward compatibility
2023-11-07 19:06:37 +03:00
c4c9d7ceb3
chore: prepara to tag `e5.3.1-alpha.4`
2023-11-07 13:04:46 -03:00
b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
...
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
7dcdbc9e51
fix(resource): take error from action/connector before attempting query
...
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
fa740a1927
docs: fix inconsistencies for bridge and connector operations
...
It's only `start` allowed for now. Also fixed some wordings.
2023-11-07 13:58:23 +01:00
f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config
...
Rename connector channel related configs in bridge_v2 to 'parameters'
2023-11-07 13:41:55 +01:00
9742dae30d
Merge pull request #11889 from thalesmg/rename-bridgev2-to-action-apis-r53-20231106
...
chore: rename `bridges_v2` -> `actions` in the public facing APIs
2023-11-07 09:29:06 -03:00
d76ed585b0
chore: correct the application vsn
2023-11-07 11:11:20 +08:00
7bd5579980
chore: fix the documentation generation and example conf checking
2023-11-07 11:09:39 +08:00
4ef156d69e
chore: fix dialyzer warnings
2023-11-07 09:04:50 +08:00
0d9e0bd3fe
chore: fix lot of running bugs
2023-11-07 08:52:10 +08:00
bb8f80f256
test: add test case to pin bridge schema json version 0.1.0
2023-11-06 21:59:36 +01:00
9a26c03a5c
refactor: unify top level field names for bridge v2
2023-11-06 21:45:08 +01:00
2b8cf50a1d
chore: rename `bridges_v2` -> `actions` in the public facing APIs
...
Fixes https://emqx.atlassian.net/browse/EMQX-11330
After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.
- HTTP API
- Hocon schema root key
2023-11-06 15:37:07 -03:00
6af6309301
chore: upgrade to hocon 0.39.19
2023-11-06 19:16:19 +01:00
f17b762596
chore: don't disable rule that references non-existent bridge
...
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
2023-11-06 10:09:14 -03:00
6f51b9f842
chore: ensure elvis pass
2023-11-06 12:31:54 +08:00
8871738756
chore: ensure emqx_gateway_ocpp tests pass
2023-11-06 12:26:58 +08:00
bea0acd929
fix(ocpp): ensure ocpp gateway options works
2023-11-06 11:56:50 +08:00
5881e34d4e
Merge pull request #11874 from zmstone/1103-validate-bridge-type-and-name-at-entry
...
fix(bridge): do not allow dot in bridge name
2023-11-03 23:04:10 +01:00
0ff4465c78
Merge pull request #11875 from thalesmg/fix-kafka-connecting-r53-20231103
...
fix(kafka_producer): don't return `disconnected` when there are connections issues while starting the bridge
2023-11-03 17:56:00 -03:00
a064eecb68
Merge pull request #11879 from thalesmg/fix-rule-api-downgrade-bridge-r53-20231103
...
fix(rules_api): downgrade bridge id
2023-11-03 17:26:09 -03:00
21a167e1a5
Merge pull request #11873 from zmstone/1103-unhide-nodelay-field
...
fix(kafka): do not hide nodelay field
2023-11-03 20:49:29 +01:00
600747b7e5
fix(bridge): do not allow dot in bridge name
...
also validate name at the API entry
2023-11-03 20:44:57 +01:00
0ee74cd8ae
fix(rules_api): downgrade bridge id
...
Fixes https://emqx.atlassian.net/browse/EMQX-11312
2023-11-03 15:57:13 -03:00
4265ef66cc
fix(kafka_producer): don't return `disconnected` when there are connection issues while starting the bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixex https://emqx.atlassian.net/browse/EMQX-11298
We don't enforce the connection to be up when starting/creating the bridge, otherwise the
status will be `disconnected` for a possibly transient reason such as network issues or
Kafka broker restart.
Same applies for Azure Event Hub Producer bridge, as they share the same module.
2023-11-03 15:49:36 -03:00
dd8085d3a4
test(emqx_bridge_v2_api_SUITE): fix broken test case
2023-11-03 17:49:25 +01:00
8a1b370a5e
Merge pull request #11876 from sstrigler/EMQX-11317-creating-a-bridge-for-a-non-existent-or-non-matching-connector-results-in-a-crash
...
handle errors from pre/post_config_update
2023-11-03 16:26:21 +01:00
953d483c24
Merge pull request #11864 from thalesmg/test-enable-rule-check-deps-r53-20231101
...
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 11:01:38 -03:00
45bfcea6ae
Merge pull request #11866 from sstrigler/EMQX-11294-e-5-3-1-swaggers-several-misdescriptions-of-ap-is
...
EMQX 11294 swaggers several misdescriptions of APIs
2023-11-03 14:55:46 +01:00
dc11008993
fix(emqx_bridge): handle errors from pre/post_config_update
2023-11-03 14:11:25 +01:00
fcecd3e9d4
fix(kafka): do not hide nodelay field
...
otherwise:
it's returned in the GET response,
but it's not allowed in the POST request
2023-11-03 13:35:52 +01:00
c84e4a4187
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 09:24:53 -03:00
65ba0b9de4
Merge pull request #11869 from kjellwinblad/kjell/shared_con/start_feedback/EMQX-11304
...
fix(bridge_v2): start operation should return an error when unsuccessful
2023-11-03 13:16:32 +01:00
4dd054b0a2
test(emqx_bridge_kafka_impl_consumer_SUITE): fix flaky
2023-11-03 11:48:50 +01:00
dd11a30383
docs(emqx_bridge_azure_event_hub): remove timestamp from message template example
2023-11-03 11:18:54 +01:00
a89b0c8cf4
fix(emqx_bridge_azure_event_hub): make ssl mandatory (and give an example)
2023-11-03 11:18:38 +01:00
ebb5997a8c
test(emqx_bridge_v2_api_SUITE): fix cluster test
2023-11-03 11:11:27 +01:00
7cab269e0b
feat: port the ocpp gateway from version 4
2023-11-03 16:51:21 +08:00
fdbf439df8
Merge pull request #11859 from cisiqo/patch-1
...
fix content type returned by Stomp
2023-11-03 08:21:08 +01:00
afe6f79a66
feat(bridge): add description field to bridge and connector
2023-11-03 06:53:42 +01:00
d14d0fbcb1
fix(bridge_v2 operations): better error message
2023-11-03 06:35:42 +01:00
36640263ba
fix(bridge_v2): start operation should return an error when unsuccessful
...
The bridge V2 HTTP API start operation should return a 400 error if the
start is unsuccessful.
The bridge V1 HTTP API compatibility layer for Bridge V2 should return a
400 error if the start or restart operation is unsuccessful.
This commit fixes the above and adds tests that checks this for the V2
HTTP API.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11304
2023-11-02 18:28:22 +01:00
910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder
...
feat(tpl): split `emqx_placeholder` into a couple of modules
2023-11-02 22:50:05 +07:00
729c6edff6
chore(repub): simplify error handling in MQTT props templating
2023-11-02 20:13:32 +07:00
f1847fe494
chore(tpl): drop unnecessary binding
2023-11-02 20:13:32 +07:00
e521a9f5fc
fix(utils): denote `emqx_jsonish` follows access module behaviour
...
Defined in `emqx_template`.
2023-11-02 20:13:32 +07:00
b5b6c3f8cc
fix(tpl): ensure full backward compat in basic connectors
2023-11-02 20:13:32 +07:00
02c1bd70b6
feat(tpl): factor out loose json concept into a separate module
...
Which is called `emqx_jsonish`. Also introduce an _access module_
abstraction to extract information from such data during rendering.
2023-11-02 20:13:31 +07:00
fbd27eda6a
fix(Stomp): content type returned
2023-11-02 11:21:27 +00:00
69cfa740ea
fix(ruleeng): ensure full backward compatibility
2023-11-02 17:11:12 +07:00
75cc663786
chore(ruleeng): streamline application dependencies
2023-11-02 17:11:12 +07:00
8e4585d64f
chore: move template modules to `emqx_utils`
...
Even though most of the time these modules will be used by
connectors, there are exceptions (namely, `emqx_rule_engine`).
Besides, they are general enough to land there, more so given
that `emqx_placeholder` is already there.
2023-11-02 17:11:12 +07:00
343b679741
feat(tpl): make escaping mechanism more foolproof
...
Treat "${$}" as literal "$". This allows to template express
strings, for example, of the form "${some_var_value}" where
`some_var_value` is interpolated from bindings.
2023-11-02 17:11:11 +07:00
f689d6c233
fix(tpl): ensure backward compat in `emqx_rule_engine`
...
Missing bindings in string templates will be rendered as "undefined",
as before. Rendering still assumes that missing binding with implicit
default (`undefined`) is an error.
This will also restore complete backward compat in `emqx_prometheus`.
2023-11-02 17:11:11 +07:00
1fcdfe991c
fix(tpl): add few missing `@doc`s + correct typespecs
2023-11-02 17:11:11 +07:00
a9693eada7
fix(tpl): rename `trivial` -> `is_const`
...
This is clearer. Former naming was a bit misleading.
2023-11-02 17:11:11 +07:00
7bb995f0c6
feat(tpl): support `:n` SQL parameters
2023-11-02 17:11:11 +07:00
dfb7faf6a8
fix(tpl): ensure rough backward compat in `emqx_prometheus`
...
But also emit warning when job name template renders with errors.
2023-11-02 17:11:10 +07:00
49fba40ee7
fix(tpl): ensure backward compat with authz / authn templates
...
This commit leans heavy into discouraging the former approach where
only part of placeholders were interpolated, depending on `placeholders`
option.
2023-11-02 17:11:10 +07:00
49f5325c67
feat(tpl): unify validations / errors var representations
2023-11-02 17:11:10 +07:00
b812f9af5a
feat(tpl): use `emqx_connector_template` in `emqx_rule_engine` app
2023-11-02 17:11:10 +07:00
e1bca5844f
feat(tpl): use `emqx_connector_template` in `emqx_prometheus` app
2023-11-02 17:11:10 +07:00
0538a77700
feat(tpl): use `emqx_connector_template` in `emqx_authn`, `emqx_authz`
...
This slightly changes semantics: now the attempt to create authenticator
with illegal bindings in templates will fail, instead of treating them
as literals. The runtime behaviour on the other hand should be the same.
2023-11-02 17:11:10 +07:00
35902dc72d
feat(tpl): switch basic connectors to `emqx_connector_template`
...
Also avoid `filename:join/2` in HTTP connector since it's both OS specific
and an overkill.
2023-11-02 17:11:09 +07:00
28d55d72ca
feat(tpl): split `emqx_placeholder` into a couple of modules
...
Located under `emqx_connector` for now. The APIs of the new modules
are slightly different from the old ones. The new ones are more
explicit in terms of error handling. Also copy the according testsuite
from `emqx_plugin_libs` and enrich it for better coverage.
2023-11-02 17:11:09 +07:00
73dd2f0ffd
chore: bump version to e5.3.1-alpha.3
2023-11-02 10:36:41 +01:00
f5818a86e3
Merge pull request #11860 from thalesmg/fix-delete-shared-bridgev2-r53-20231101
...
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-02 10:30:47 +01:00
a44f8eb583
Merge pull request #11861 from zmstone/1101-do-not-repeat-print-shell-warning
...
fix(shell): do not repeat print shell warnings
2023-11-02 10:28:56 +01:00
34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293
...
fix(bridge_v2 API): optional cascading delete operation
2023-11-02 10:28:43 +01:00
ec2d339355
Merge pull request #11857 from kjellwinblad/kjell/shared_con/EMQX-11270
...
fix(bridge_v2): channels should not be removed when status is connecting
2023-11-01 16:46:47 +01:00
5b964ef415
Merge pull request #11858 from id/1101-emqx-11288-fix-kafka-replaq-dir-conflict
...
emqx 11288 fix kafka replaq dir conflict
2023-11-01 16:45:02 +01:00
4bea65bf97
fix(bridge_v2_api): don't disable schema check
2023-11-01 12:44:40 -03:00
2dd5061643
fix(kafka): kafka bridge replaq dir conflict
2023-11-01 15:44:28 +01:00
357b664c8d
fix(bridge_v2): more fixes thanks to PR comments from @thalesmg
2023-11-01 15:27:54 +01:00
b06d05eaac
test(bridge_v2): fix test case after new API
2023-11-01 15:27:54 +01:00
96d6c6db49
test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change
2023-11-01 15:27:53 +01:00
edb1d37e67
chore(bridge_v2): make fixes thanks to PR comments from @thalesmg
2023-11-01 15:27:53 +01:00
95f3b94ac3
fix(bridge_v2): channels should not be removed when status is connecting
...
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
1e935e9eb4
fix(bridge_v2 API): optional cascading delete operation
...
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
2023-11-01 15:03:18 +01:00
c38725facd
fix(shell): do not repeat print shell warnings
2023-11-01 14:44:18 +01:00
17544dc410
Merge pull request #11852 from lafirest/feat/gbt_gw
...
feat(gbt32960): Port the GBT32960 gateway from v4
2023-11-01 21:03:18 +08:00
c03979a48a
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-01 09:51:26 -03:00
6b50fdcf9e
chore(gbt32960): update changes
2023-11-01 19:35:44 +08:00
30a72f557f
fix(gateway): improve gateway schema modules
...
1. enhances the gateway name as an enum
2. make the schema more flexible and extensible without some hardcode
2023-11-01 18:55:36 +08:00
10b2894b6e
chore(bridge_v2_api): remove misplaced handling code
2023-11-01 11:31:17 +01:00
ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched
...
perf: no need to format event message if no bridge matched
2023-11-01 10:22:06 +01:00
aa8a6f2e26
Merge pull request #11853 from thalesmg/test-bridge-v1-compat-layer-r53-20231030
...
test(bridges): add bridge v1 compatibility layer test suite, and other fixes
2023-11-01 09:51:10 +01:00
29683072a1
fix(emqx_connector): remove `stop` and `restart` operations
2023-11-01 09:18:50 +01:00
5e314d4ef1
test(gbt32960): add test suites
2023-11-01 13:17:01 +08:00
45a39d97c6
fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1
...
Fixes https://emqx.atlassian.net/browse/EMQX-11291
2023-10-31 16:39:47 -03:00
7092c75597
Merge pull request #11809 from keynslug/ft/EMQX-10808/file-secrets
...
feat(mqttbridge): support file-sourced secrets as passwords
2023-11-01 00:29:30 +07:00
b420b53075
fix(bridges_v2): check bridge name length before attempting atom conversion
...
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
8eb822d898
test(bridges): add bridge v1 compatibility layer test suite
2023-10-31 13:21:01 -03:00
db5cdb6742
perf: no need to format event message if no bridge matched
2023-10-31 16:48:10 +01:00
3e9f48d10f
fix: remove unneeded clause
2023-10-31 16:05:31 +01:00
2a4eaf41ba
docs: fix examples for PUT operations on bridge_v2 and connectors
...
Examples would show `type` and `name` properties in the request body, which is
not accepted by the schema.
Also fixes some minor inconsistencies in the example names of connectors and
bridges.
2023-10-31 15:10:11 +01:00
84e78f5d2e
fix(bridge_api): fix response status code for some operations
2023-10-31 10:40:56 -03:00
c90ca1ea53
fix(bridge_v1): always delete connector when deleting v1 bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-11287
2023-10-31 10:40:06 -03:00
d278486416
fix(secret): dedicate a specific loader module for file secrets
...
To make code employing `emqx_secret` easier to follow.
2023-10-31 19:26:04 +07:00
b35cd57034
fix(gbt32960): make elvis happy
2023-10-31 19:04:21 +08:00
3ec811e828
feat(gbt32960): Port the GBT32960 gateway from v4
2023-10-31 18:25:43 +08:00
b1ab213081
chore: merge 'upstream/release-53' into 1031-sync-r53
2023-10-31 11:06:25 +01:00
39f49d425c
test(emqx_bridge): fix test for bridge_v2_type_to_connector_type
2023-10-31 08:34:05 +01:00
63606922f2
refactor(emqx_bridge): only one place to generate same error
2023-10-31 08:34:05 +01:00
b256241650
test(emqx_bridge_azure_event_hub): fix legacy v1 bridge API tests
2023-10-31 08:34:05 +01:00
176bbe88bc
fix: use `_producer` for AEH bridge type
2023-10-31 08:34:03 +01:00
aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import
...
EMQX-11272 fix kafka bridge v1 import
2023-10-31 07:58:54 +01:00
21e4f918aa
Merge pull request #11833 from JimMoen/fix-shared-sub-topic-or-subscription-searching
...
Fix shared sub topic or subscription searching
2023-10-31 08:16:36 +08:00
9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271
...
fix: crash when listing non bridge_v1 compatible bridge_v2
2023-10-30 20:29:36 +01:00
c07cf9051e
Merge pull request #11844 from zmstone/1030-downgrade-bridge-type-for-old-api
...
1030 downgrade bridge type for old api
2023-10-30 19:47:24 +01:00
124d79a1ca
Merge pull request #11843 from thalesmg/test-kafka-producer-nits-r53-20231030
...
test(kafka_producer): minor adjustments to test suite
2023-10-30 18:27:10 +01:00
350e8433ab
fix(bridge_v2): downgrade bridge type for GET api results
2023-10-30 18:17:35 +01:00
0562585c8f
fix(emqx_mgmt_data_backup): implement importer modules ordering
...
`emqx_bridge_v2` depends on `emqx_connector`, so connectors must be imported first.
2023-10-30 19:14:58 +02:00
bc8c291820
fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2
2023-10-30 19:14:58 +02:00
d94193ac15
fix(emqx_connector_schema): add kafka alias for kafka_producer
2023-10-30 19:14:58 +02:00
0935bb6225
fix(emqx_connector): fix badarity error
2023-10-30 17:52:39 +02:00
d1cd5dd817
fix(emqx_mgmt_data_backup): upgrade raw conf before validating and importing
2023-10-30 17:52:39 +02:00
8fb426e57f
fix: crash when listing non bridge_v1 compatible bridge_v2
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
2023-10-30 16:32:04 +01:00
a60b96c5fd
test(kafka_producer): minor adjustments to test suite
...
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
3f6c09b195
Merge pull request #11780 from savonarola/1017-fix-pbkdf2-validation
...
fix(authn): fix pbkdf2 option validation
2023-10-30 16:37:37 +02:00
5a6e55e7af
fix(emqx_bridge_kafka): schema fixes for kafka_producer in bridge_v2
2023-10-30 14:59:06 +01:00
f463eff02d
chore: e5.3.1-alpha.2
2023-10-30 14:49:08 +01:00
0656b6be3c
refactor(emqx_config): use dynamic callback to upgrade raw config
2023-10-30 14:49:08 +01:00
9dc3a169b3
feat: split bridges into a connector part and a bridge part
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.
Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
e896957803
Merge pull request #11824 from zhongwencool/audit-log-reviewed
...
refactor: audit log review refactor
2023-10-30 17:51:15 +08:00
3a09fdc495
refactor: check match topic before do subscriptions query
2023-10-30 16:45:29 +08:00
e9de7316b6
test: shared-sub topics/subscription api
2023-10-30 14:42:59 +08:00
814e22feb3
fix: topics/subscripton mgmt api searching
2023-10-30 14:42:47 +08:00
0634ff61c0
fix: dialyzer warning
2023-10-30 09:57:30 +08:00
a2015f37ae
Merge pull request #10976 from JimMoen/fix-shared-sub-unsub
2023-10-27 21:22:27 +08:00
e63602c4b0
Merge pull request #11811 from lafirest/fix/api_rbac
...
fix(rbac): for compatibility with old data schema, extend the existing field as an extra
2023-10-27 17:56:54 +08:00
d563121284
refactor: move ?REDISPATCH_TO macro to emqx_mqtt.hrl
2023-10-27 17:54:14 +08:00
4ba34f8f3e
chore(rbac): fix CI errors & update change
2023-10-27 17:17:04 +08:00
3b5cc912e7
fix: add `redispatch_to` header to all msgs when deliver shared-sub
...
- to find correct SubOpts for shared-sub dispatch
- use previous key `redispatch_to` to ensure rolling upgrade compatibility
2023-10-27 16:15:23 +08:00
22223dc536
fix: return 400 when audit log is disabled
2023-10-27 16:11:30 +08:00
995948f0e8
refactor: remove seq from audit record
2023-10-27 14:07:19 +08:00
9bb22507df
fix: dont use transation on audit log
2023-10-27 11:41:12 +08:00
53383991d9
fix: rewrite #share{} and return #share{}, not formated $share/group/topic
2023-10-27 09:23:44 +08:00
afec6fa2f6
fix: TopicFilter may modified by `client.subscribe` hook
2023-10-27 09:23:44 +08:00
802a36c670
fix: find SubOpts by shared_record, not deliver topic
2023-10-27 09:23:44 +08:00
eaa5459509
chore: bump apps vsn
2023-10-27 09:23:43 +08:00
0ca725ff25
fix: deprecated `share` in `subopts`
2023-10-27 09:23:43 +08:00
b5411da770
refactor: subscribe process to fix shared-sub
2023-10-27 09:23:43 +08:00
44b4205561
fix(secret): do not treat missing file secrets as config error
...
They are intended to be used mostly in the context of resources, which
have their own feedback mechanism: statuses, retries, etc.
Also turn the error into a throw exception, so that it can be
interpreted as a regular error condition, for example by the resource
manager.
2023-10-26 14:37:14 +07:00
cbfd02d1b0
refactor(ldap): merge the `ldap-bind` backend as a type for the `ldap` backend
2023-10-26 11:04:44 +08:00
97601171a5
Merge pull request #11818 from zmstone/1025-fix-flaky-test
...
test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked
2023-10-25 20:46:31 +02:00
76a8b0efd4
test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked
2023-10-25 16:08:31 +02:00
f52b4a9e32
fix: gen_rpc tcp client port align with server port
2023-10-25 12:11:35 +02:00
37dd44c3d7
Merge pull request #11813 from qzhuyan/fix/william/genrpc-ssl-port-align
...
Fix/william/genrpc ssl port align
2023-10-25 09:04:19 +02:00
6346e0d28a
fix(gen_rpc): ssl client port align with server port
2023-10-24 22:33:52 +02:00
ec41479633
feat(rbac): supports setting role in API bootstrap file
2023-10-24 23:03:18 +08:00
e175c213a1
fix(rbac): for compatibility with old data schema, extend the existing field as extra
2023-10-24 22:28:53 +08:00
abcb3166f6
fix: use binary string for raw default values
2023-10-24 14:46:01 +02:00
7a8a5926ab
fix(rbac): adjust the role names
2023-10-24 19:01:12 +08:00
6e12569260
Merge pull request #11797 from sstrigler/EMQX-1057-fix-authz-api-donot-return-200
...
fix: return 404 if built_in_database not configured as auth source
2023-10-24 12:08:53 +02:00
52f4519eeb
feat(mqttbridge): support file-sourced secrets as passwords
2023-10-24 16:24:52 +07:00
1c2f9321d1
feat(emqx): add file-sourced generic secrets
...
These secrets follow the same `emqx_secret` convention of 0-arity
functions. Also provide a simple HOCON schema module for use in
application schemas.
2023-10-24 16:24:51 +07:00
ea9228108b
fix: use minirest filter
2023-10-24 09:57:08 +02:00
99fab8dc5d
chore: missing change log for duplicated apikey fixing
2023-10-24 09:44:39 +08:00
d392c9989f
Merge pull request #11793 from thalesmg/test-rule-engine-foreach-m-20231019
...
test(rule_engine): add test to exemplify `foreach` with json array payload
2023-10-23 13:43:33 -03:00
4e0e755b28
fix: return 404 if built_in_database not configured as auth source
2023-10-23 16:26:41 +02:00
8c6f6b56b8
chore(tests): report more friendly about unset PROFILE variable
2023-10-23 13:48:42 +03:00
edde661da3
fix(authn): fix pbkdf2 option validation
2023-10-23 10:26:11 +03:00
a9b42af8b3
Merge pull request #11796 from qzhuyan/fix/william/gen_rpc-driver
...
fix(gen_rpc): ensure client/server use same driver
2023-10-23 08:24:17 +02:00
e6576951ef
test: cleanup duplicated apikey with different name
2023-10-20 22:33:26 +08:00
d467289bb2
fix: avoid duplicated apikey from data import
2023-10-20 21:13:04 +08:00
dd5807b797
fix(gen_rpc): ensure client/server use same driver
...
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
ef692596f7
fix: don't crash when 401 and 403 unauthorize
2023-10-20 15:22:30 +08:00
6a8b2dc1f9
fix: bad high frequency request name
2023-10-20 14:57:31 +08:00
c97fe796e3
chore: update emqx_audit's license
2023-10-20 14:47:24 +08:00
f381961108
fix: macro EMQX_RELEASE_EDITION when `emqx` run as standalnoe app
2023-10-20 14:47:24 +08:00
32c1f1cca6
chore: inlude emqx as emqx_audit's deps
2023-10-20 14:47:24 +08:00
1d7aa9495a
chore: make spellcheck happy
2023-10-20 14:47:24 +08:00
befc484544
chore: add changelog entry and fix xref warning
2023-10-20 14:47:24 +08:00
141061c1e2
feat: add max_filter_size and ignore_high_frequency_request config
2023-10-20 14:47:24 +08:00
926c804314
feat: add /audit http api to filter audit log
2023-10-20 14:47:24 +08:00
8b5ec2a4d5
fix: xref failed
2023-10-20 11:46:31 +08:00
281add1464
Merge branch 'master' into sync-master-to-release-54
2023-10-20 10:11:39 +08:00
74442b0d31
Merge pull request #11766 from lafirest/feat/rbac
...
feat(api_key): add RBAC feature for the API key
2023-10-20 09:24:41 +08:00
8c4beec6f0
test(rule_engine): add test to exemplify `foreach` with json array payload
...
Relates to https://emqx.atlassian.net/browse/EMQX-11174
2023-10-19 18:19:42 -03:00
0b4600c293
feat(auth): improve redis command parsing and validation
2023-10-19 23:01:35 +03:00
eda82caf65
Merge pull request #11791 from thalesmg/fix-coap-heartbeat-m-20231018
...
fix(coap): increase received packet counter for keepalive
2023-10-19 13:12:41 -03:00
8d82c30b00
Merge pull request #11771 from savonarola/1015-validate-bcrypt-schema-in-api
...
feat(authn): allow authn providers to define a separate schama for API
2023-10-19 15:34:34 +03:00
8f9ac6c1f3
Merge pull request #11776 from savonarola/1017-auth-cleanup-2
...
chore(authn): remove dead code
2023-10-19 15:28:35 +03:00
5b9866f630
fix(coap): increase received packet counter for keepalive
...
Fixes https://emqx.atlassian.net/browse/EMQX-11193
Fixes https://github.com/emqx/emqx/issues/11779
2023-10-18 16:35:33 -03:00
2388d36b09
fix: allow viewers to change their own passwords
2023-10-18 18:40:02 +08:00
e1c8317779
chore(authn): remove dead code
2023-10-18 10:57:55 +03:00
1f8478623f
chore: bump esaml to support Entra ID && bump app.src
2023-10-18 10:01:34 +08:00
b2948666fb
chore: add saml sso redirect login log
2023-10-18 10:01:34 +08:00
c0ebaf9b6f
Merge pull request #11541 from savonarola/0829-duplicate-ft-responses
...
Support asynchronous mode of receiving responses to FT operations
2023-10-17 17:08:00 +03:00
6354f3b04f
feat(authn): allow authn providers to define a separate schama for API
2023-10-17 13:19:11 +03:00
55618dd28a
Merge pull request #11731 from zmstone/1008-expose-file-transfer-host-conf
...
fix(api-docs): add file-transfer config to hot-config scope
2023-10-17 11:23:16 +02:00
467a2bc9b1
test(rbac): add test cases for RBAC in REST API
2023-10-17 16:23:38 +08:00
26ec860d96
feat(backup): add migration mechanism when import backup data
2023-10-17 16:23:32 +08:00
dfa5909392
Merge pull request #11752 from zmstone/1011-change-default-rpc-module-for-mria
...
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
2023-10-17 10:21:14 +02:00
db3915d472
fix(rbac): update changes && fix CI errors
2023-10-17 13:44:37 +08:00
162b901be6
Merge pull request #11763 from zmstone/1114-sync-release-53
...
Merge release-53 back to master
2023-10-16 21:42:02 +02:00
e095de7367
feat(api_key): add RBAC feature for the API key
2023-10-16 14:10:01 +08:00
16cc816bd3
Merge pull request #11757 from zhongwencool/trace-api-return-500
...
fix: 500 error response when downloading non-existent trace files
2023-10-16 13:40:18 +08:00
6eb3bb7cff
Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53
2023-10-14 10:16:38 +02:00
7c022c2c6a
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
...
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-10-14 10:13:31 +02:00
cd2752117c
Merge pull request #11762 from savonarola/1013-fix-mnesia-authz-destroy
...
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 20:54:14 +03:00
bde8800f2e
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 18:29:45 +03:00
90a0c093bf
Merge pull request #11750 from savonarola/1011-redact-auth-http-request
...
Do not trace authn http request bodies
2023-10-13 17:25:28 +03:00
4f14e8df22
chore(source dir): split out postgresql connector
2023-10-13 16:58:57 +03:00
7a6c756b7c
Merge pull request #11754 from thalesmg/fix-postgres-error-reason-log-m-20231011
...
fix(postgres): format unicode error messages from driver
2023-10-13 09:58:28 -03:00
0144ed9981
Merge pull request #11753 from keynslug/chore/smaller-ci-artifacts
...
chore(ci): try to reduce CI artifacts footprint
2023-10-13 16:56:47 +07:00
51c57a5f0a
fix(cass): avoid using aggregates in healthcheck query
...
Otherwise, this query will produce warning message per each
healthcheck, telling the user:
```
WARN <...> - Aggregation query used without partition key
```
2023-10-13 16:50:19 +07:00
083e2da347
chore(bpapicheck): make some failures more user-friendly
2023-10-13 15:42:08 +07:00
0b9ac24c1e
fix(bpapicheck): stop depending on git index
2023-10-13 15:40:49 +07:00
96c546c187
Merge pull request #11725 from lafirest/feat/os-ldap
...
chore: change the LDAP integration to opensource
2023-10-13 15:05:39 +08:00
dd03be29a6
Merge pull request #11751 from zmstone/1011-refactor-move-emqx_license-app-to-apps
...
1011 refactor move emqx_license app to apps
2023-10-13 06:45:49 +02:00
33ff5d5588
chore: update auth header file
2023-10-13 10:16:47 +08:00
68f31a9da2
fix(authn): do not trace authn requests
2023-10-12 16:32:16 +03:00
08795f559c
Merge pull request #11336 from savonarola/0724-trace-authz
...
Trace non-resultative authz calls
2023-10-12 14:51:15 +03:00
6413afd0bc
fix(gcpdev): restore original shard in mnesia tab definition
...
Changing the shard is not backward compatible, and leads to a crash
when upgrading from 5.2.x.
2023-10-12 17:30:07 +07:00
4ecd5e17a2
chore(authz): trace non-resultative authz calls to backend modules
2023-10-12 12:29:39 +03:00
3e658b3da9
chore: update changes
2023-10-12 17:15:47 +08:00
9b4def885a
chore: change the LDAP integration to opensource
2023-10-12 16:37:32 +08:00
1021088f5f
fix: 500 error response when downloading non-existent trace files
2023-10-12 15:23:23 +08:00
b07dddd49e
fix(postgres): format unicode error messages from driver
...
Fixes https://emqx.atlassian.net/browse/EMQX-11024
Sample error:
```
{error, error, <<"42501">>, insufficient_privilege,
<<229,175,185,232,161,168,32,109,113,116,116,95,117,115,101,114,32,230,157,131,233,153,144,228,184,141,229,164,159>>,
[]}
```
2023-10-11 17:56:12 -03:00
534c9bdc13
feat(ft): add additional operation status report channel
2023-10-11 19:51:36 +03:00
d9859b6b49
Merge pull request #11749 from thalesmg/test-fix-flaky-http-suite-r53-20231011
...
test: attempt to fix flaky http bridge suite
2023-10-11 16:07:17 +02:00
e6358ca18c
Merge pull request #11747 from qzhuyan/dev/william/quicer-0.0.202
...
bump to quicer 0.0.202
2023-10-11 15:43:47 +02:00
e069680bca
refactor: move emqx_license app from lib-ee to apps
2023-10-11 15:41:35 +02:00
ab662f506d
test: attempt to fix flaky http bridge suite
...
```
=CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
crasher:
initial call: ehttpc:init/1
pid: <0.23358.2>
registered_name: []
exception error: bad argument
in function persistent_term:get/1
called as persistent_term:get({emqx_bridge_http_SUITE,
do_t_async_retries,attempts})
*** argument 1: no persistent term stored with this key
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-11 09:13:25 -03:00
57655854d1
Merge pull request #11743 from savonarola/1010-auth-leftovers
...
chore(auth): cleanup code
2023-10-11 15:09:07 +03:00
a69c53455f
fix: bump to quicer 0.0.202
...
bring in urgent fixs from msquic 2.2.3
2023-10-11 13:12:32 +02:00
cc392cadb8
chore: update apps/emqx_retainer/src/emqx_retainer_mnesia_cli.erl
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-10-11 18:40:09 +08:00
03ae5bf3c8
chore(auth): cleanup code
2023-10-11 13:13:50 +03:00
f4a88f717b
feat: support retainer Start Length command
2023-10-11 11:47:37 +08:00
8e7ba16c3a
fix: nothing show when run clients list command
2023-10-11 11:47:15 +08:00
574dc2f243
fix: observer load command crash when loading noexist module
2023-10-11 10:36:33 +08:00
a2e86c67db
Merge pull request #11733 from keynslug/fix/EMQX-10827/sesson-takeover
...
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
2023-10-10 19:03:29 +02:00
0f07f678e7
Merge pull request #11742 from zmstone/1010-sync-release-53-to-master
...
1010 sync release-53 to master
2023-10-10 18:56:23 +02:00
196b24c0cb
Merge pull request #11741 from thalesmg/test-fix-flaky-http-test-m-20231010
...
test: attempt to fix flaky http bridge suite
2023-10-10 13:52:08 -03:00
ee45145fb5
Merge pull request #11724 from thalesmg/fix-kprodu-sync-metrics-m-20231006
...
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers
2023-10-10 12:13:33 -03:00
ec588f94e2
test: attempt to fix flaky http bridge suite
...
```
=CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
crasher:
initial call: ehttpc:init/1
pid: <0.23358.2>
registered_name: []
exception error: bad argument
in function persistent_term:get/1
called as persistent_term:get({emqx_bridge_http_SUITE,
do_t_async_retries,attempts})
*** argument 1: no persistent term stored with this key
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-10 09:50:53 -03:00
bfb2218392
chore: fix release version
2023-10-10 14:50:49 +02:00
cf2075d7d8
chore: remove mention of `is_buffer_supported` from typespec
2023-10-10 09:49:18 -03:00
143673b703
chore: bump release version to 5.3.1-alpha.1
2023-10-10 14:35:46 +02:00
0a9e1dfa88
Merge pull request #11734 from zmstone/1002-support-ipv6-only-clustering
...
Make IPv6 clustering options configurable
2023-10-10 14:29:52 +02:00
5fff2ffe45
test(kafka): try to stabilize `t_dynamic_mqtt_topic/1` testcase
2023-10-10 17:18:24 +07:00
9b573834f9
Merge pull request #11736 from keynslug/fix/eval-erl-audit
...
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 15:49:19 +07:00
2626d793a7
test: try to resolve schema module from PROFILE in tests
2023-10-10 10:24:10 +02:00
808237364c
fix(api-docs): add file-transfer config to hot-config scope
2023-10-10 10:24:10 +02:00
e7e696cd66
feat(rpc): add ipv6_only config
2023-10-10 10:05:14 +02:00
e07937a3ef
fix(ldap): escape the escape character (\)
2023-10-10 13:49:36 +08:00
2a291dfd27
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 12:31:19 +07:00
522302fee1
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
...
Which was accidentally broken in bf164175
2023-10-10 01:34:10 +07:00
1e93d2f1fc
chore: upgrade to ekka 0.15.16 gen_rpc 3.2.0
...
* ekka 0.15.16 supports 'inet6_tls' as ekka.proto_dist
* gen_rpc 3.2.0 supports true | false as gen_rpc.ipv6_only
2023-10-09 20:03:21 +02:00
497e08448d
feat(cluster): support ipv6 and tls on ipv6 for clustering
...
Made possible to configure inet6_tls for Erlang distribution
Also, added support to configure ipv6 listener for gen_rpc
2023-10-09 20:03:21 +02:00
03d8e06ff7
chore(emqx_rule_funcs): regroup export functions
2023-10-09 20:03:21 +02:00
d6781efee2
fix(resource): change how buffer workers are started
2023-10-09 15:02:25 -03:00
902b1d6ec5
fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
...
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
eebfb44f72
fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
...
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
79cf0a2ced
fix(kafka_producer): correctly handle metrics for connector that have internal buffers
...
Fixes https://emqx.atlassian.net/browse/EMQX-11086
There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).
We use simple_sync_query to call the Kafka Producer bridge. If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
c60915293a
Merge pull request #11634 from savonarola/0919-auth-refactor
...
Auth refactor
2023-10-09 19:51:26 +02:00
8f4cdc3fcf
Merge pull request #11732 from lafirest/test/sso_cli
...
test(ldap): add test suite for SSO CLI
2023-10-09 20:35:23 +08:00
9ab49a7ae3
Merge remote-tracking branch 'origin/master' into 0919-auth-refactor
2023-10-09 14:18:10 +02:00
2c7e5eb1cb
Merge pull request #11728 from lafirest/fix/ldap_filter
...
fix(ldap): improve the filter lex && parse
2023-10-09 18:24:57 +08:00
e3550fc07b
test(ldap): add test suite for SSO CLI
2023-10-09 16:05:56 +08:00
0c89b6b213
docs: swagger api tags should camel case
2023-10-09 15:21:51 +08:00
b2a6724dc2
fix(ldap): improve the filter lex && parse
...
1. auto escape special chars in the filter
2. fix a bug that the value can't be `dn`
2023-10-08 18:39:20 +08:00
482e82f914
fix: cli's args in audit log should be array
2023-10-07 17:38:02 +08:00
34186fcc74
fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
...
Fixes https://emqx.atlassian.net/browse/EMQX-11085
Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
398a62031e
chore: update app versions
2023-10-05 13:41:50 +03:00
5dff36474d
chore(auth): get rid of hardcoded schema modules in auth
2023-10-05 13:41:50 +03:00
c2c56ba481
chore(auth): update tests
2023-10-05 13:41:50 +03:00
1eb75b43c4
chore(auth): split emqx_authn and emqx_authz apps
2023-10-05 13:41:50 +03:00
fd75dc895d
fix(authn): "authentication" importance should not be "hidden"
2023-10-03 11:18:21 +02:00
ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
...
0928 sync release 53
2023-10-02 21:57:32 +02:00
3f6e0e890b
chore: bump emqx_durable_storage and emqx_license app vsn
2023-10-02 20:13:09 +02:00
2358d67908
refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
...
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
bce8fd2fbc
chore: bump app versions
2023-09-29 18:56:52 +02:00
ed5b456d62
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-29 18:24:44 +02:00
a852400fb3
chore: add 5.3.bpapi
2023-09-29 16:34:04 +02:00
fe01aaff4f
chore: prepare for e5.3.0 promotion
2023-09-29 13:32:57 +02:00
ce5bd0a3ce
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-29 11:36:32 +02:00
c64e599e81
docs: document how to retrieve peercert
2023-09-29 10:33:57 +02:00
dc147fd310
fix(rule-engine): console action has no args field
2023-09-29 10:33:33 +02:00
1177a32310
chore: bump version to 5.3.0-rc.2
2023-09-29 09:49:27 +02:00
6891234390
chore: return simplified error reason for less logging
2023-09-29 09:48:15 +02:00
c2d750aa09
fix(resource): redact query args in exception log
2023-09-29 09:20:42 +02:00
02ef854f0f
fix(ldap): no crash when no query result is empty list
2023-09-29 08:54:41 +02:00
b59a7ff2dd
Merge pull request #11706 from zmstone/0928-fix-ldap-sso-logging-level
...
0928 fix ldap sso logging level and reject muti-match results
2023-09-29 02:40:37 +02:00
4a4730ad46
fix(ldap): handle invalidCredentials in ldap authn
2023-09-29 00:51:05 +02:00
9ee2cb9c79
fix(ldap): return unrecoverable_error if more than on match found
2023-09-28 23:58:34 +02:00
cc5dab1dc7
chore: fix code style
2023-09-28 21:29:59 +02:00
b267fc2588
chore: bump release version to 5.3.0
2023-09-28 21:22:33 +02:00
922d5a9a83
fix(ldap): do not allow multi-matches to proceed
...
if ldap query returns more than on match
we should reject the auth request instead of picking
the first one
2023-09-28 21:20:50 +02:00
d858f8af39
test: fix openldap docker runs
2023-09-28 18:40:03 +02:00
b28e781c50
fix(ldap-sso): do not log error level when invalid user credentials
2023-09-28 18:37:11 +02:00
36f3052be1
Merge pull request #11698 from zmstone/0928-disable-audit-log-by-default
...
fix(audit): disable audit log by default
2023-09-28 17:28:23 +02:00
39820be5ff
Merge pull request #11703 from lafirest/fix/ldap_bind
...
fix(ldap): use the search result as bind target
2023-09-28 14:48:53 +02:00
1a13b2ac56
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-28 13:53:03 +02:00
43ea367df8
fix(ldap): use the search result as bind target
2023-09-28 18:49:00 +08:00
98409c9b1e
Merge pull request #11702 from zmstone/0928-minor-chore
...
0928 minor chore
2023-09-28 12:39:55 +02:00
26cadb2248
fix(audit): disable audit log by default
2023-09-28 12:36:14 +02:00
b14425d865
fix(sso/saml): ensure lower case http header name
2023-09-28 11:40:21 +02:00
7ddcd71412
docs(ssl_dist.conf): update doc link
2023-09-28 11:16:18 +02:00
dca8fdb17f
fix(resource): respect the start_timeout
2023-09-28 16:36:41 +08:00
b77e5e880a
Merge pull request #11697 from ieQu1/dev/gen-rpc-3.1.1
...
chore(gen_rpc): Bump version to 3.1.1
2023-09-28 10:03:36 +02:00
6d2adfc259
chore(audit): emit the message field
2023-09-28 11:04:42 +08:00
859b122cdd
chore: format codes
2023-09-28 10:25:37 +08:00
b52e4ac99d
chore: update apps/emqx_machine/src/emqx_restricted_shell.erl
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-28 10:25:37 +08:00
bdf24d0ec8
chore: remove the `unauthorized` type for `from` field
2023-09-28 10:25:37 +08:00
71acf121ba
chore(audit): distinguish requests from rest_api or dashboard
2023-09-28 10:25:37 +08:00
dd9938114c
chore(audit): add from field
2023-09-28 10:25:37 +08:00
ece7d5b52a
Merge pull request #11690 from lafirest/fix/ldap_parse_cfg
...
fix(ldap): improve the LDAP `parse_config` function
2023-09-28 10:24:52 +08:00
b0d86eecd6
Merge pull request #11691 from lafirest/fix/sso_ssl
...
fix(sso): support for SSL update && ensure update is atomic
2023-09-28 10:22:34 +08:00
57781d0544
fix(ldap): remove the parse_config, it never work
2023-09-28 09:38:16 +08:00
9dee2dc31e
fix(sso): clear last error first before update && fix the `running`
2023-09-28 08:56:16 +08:00
0aa3ccdd65
feat(gen_rpc): Add schema for the TLS versions and cipher suites
2023-09-28 00:36:59 +02:00
afdda107af
fix(logger): json format log encode binary list as string array
2023-09-27 23:40:01 +02:00
6f7a4344dc
fix: do not gc sso saml SP singing keys
2023-09-27 23:19:39 +02:00
34367fc4ec
fix(audit_log): pretty print shell args
2023-09-27 23:19:39 +02:00
bb49914fd6
fix(sso): add convet_certs callback for sso backends
...
must convert certs in pre_config_update so the cert path refernces
are stored in raw config, otherwise the files might get gc:ed
2023-09-27 22:41:39 +02:00
45caa3bf01
fix(sso): make sp_private_key sensitive
...
so it will not be logged
2023-09-27 21:27:59 +02:00
c8cbbff044
fix(logger): no need for special handling of empty string
...
when formating json logs, there is no need to handle empty strings
special, already covered by unicode handling
2023-09-27 21:26:47 +02:00
3c37f19105
chore(gen_rpc): Bump version to 3.1.1
2023-09-27 21:09:00 +02:00
66d2107007
fix(sso): refactor update logic
2023-09-28 00:09:09 +08:00
bd3277c51b
Merge pull request #11683 from zmstone/0925-test-refactor-emqx-rpc-test
...
test: refactor emqx_rpc unit tests
2023-09-27 15:46:27 +02:00
08ad09a68f
fix(sso): refactor backen update logic
...
1. valid config always can update successfully
2. the `running` endpoint only return successfully created backend
3. enhancement of the `/sso` endpoint, and will check is the resource online
2023-09-27 20:53:10 +08:00
ea8d54fd8b
test: ensure atom exists in test module
2023-09-27 12:58:06 +02:00
6e8c73258f
test: refactor emqx_rpc unit tests
2023-09-27 12:58:06 +02:00
c9194cd6b2
fix(saml_sso): donot load IDP metadata when disabling saml
2023-09-27 18:46:24 +08:00
af9e87c025
fix: saml callback should check saml state
2023-09-27 18:34:48 +08:00
b2699c687b
fix(sso): support for SSL update && ensure update is atomic
...
1. support update SSL key and cert files
2. increase connection timeout
3. ensure the update is atomicity, everything will be consistent
2023-09-27 15:48:11 +08:00
1d0e789e4d
Merge pull request #11679 from keynslug/ft/EMQX-10942/custom-session-timers
...
feat(session): add custom session timers mechanism
2023-09-26 19:09:53 +03:00
0c33df3912
fix(ldap): improve the LDAP `parse_config` function
2023-09-26 23:28:37 +08:00
3fa18d6935
fix: lower sso saml redirect http header
2023-09-26 23:25:59 +08:00
6a9bb7c3ae
Merge pull request #11681 from zhongwencool/audit-log-fix-2
...
fix: don't need to change audit log's level
2023-09-26 23:12:28 +08:00
ac5eb5bc29
Merge pull request #11687 from lafirest/fix/sso_timeout
...
fix(sso): Handle backend update timeout and fix create errors
2023-09-26 22:43:38 +08:00
1a96a5990b
chore: Update apps/emqx_enterprise/src/emqx_enterprise_schema.erl
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 21:44:40 +08:00
403714d44e
fix(sso): Handle backend update timeout and fix create errors
...
1. correctly handle the timeout when call update on a backend
2. fix that config update always returns success
3. do not ignore start failures and ensure start is atomic
2023-09-26 21:43:30 +08:00
aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
...
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
bce2142df8
test(session): add custom session timers testcase
2023-09-26 16:22:06 +03:00
4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
...
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
fdcd73e20c
fix(file_logger): change file logger type depending on rotation size
...
Fixes https://emqx.atlassian.net/browse/EMQX-11036
From `logger_disk_log_h:open_disk_log`:
```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
case filelib:ensure_dir(File) of
ok ->
Size =
if Type==halt -> MaxNoBytes;
Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
end,
Opts = [{name, Name},
{file, File},
{size, Size},
{type, Type},
{linkto, self()},
{repair, false},
{format, external},
{notify, true},
{quiet, true},
{mode, read_write}],
case disk_log:open(Opts) of
```
Affects all file loggers (audit included):
```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
{badarg,
[{size,
{infinity,
10}},
{type,
wrap},
{linkto,
<0.1952.0>},
{repair,
false},
{format,
external},
{notify,
true},
{quiet,
true},
{mode,
read_write}]}}
```
2023-09-26 09:04:44 -03:00
14983ec14a
chore(hooks): validate hookpoints and document hook callbacks
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
5e6397b843
chore: remove result field for CLI audit logs
2023-09-26 19:24:35 +08:00
a73c3b8e1e
Merge remote-tracking branch 'ce/release-53' into audit-log-fix-2
2023-09-26 19:05:47 +08:00
e9785a6863
Merge pull request #11680 from thalesmg/audit-not-on-ce-r53-20230925
...
fix(audit): only support audit log on enterprise edition
2023-09-26 18:57:22 +08:00
7d810c2107
test: fix test case match pattern
2023-09-26 11:40:30 +02:00
d31bfc70fb
Merge pull request #11659 from zhongwencool/fix-listener-ssl-create-500
...
fix: create ssl listener return 500 crash
2023-09-26 11:38:07 +02:00
8cc626d33f
Merge pull request #11686 from lafirest/fix/sso_user_backend
...
fix: fix update SSO user && improve SSO user deletion commnad
2023-09-26 16:04:51 +08:00
56917fee5f
fix(sso): Improved SSO user deletion command
2023-09-26 14:15:41 +08:00
d1af81c86e
fix(sso): fix the failure to update SSO users
2023-09-26 12:14:45 +08:00
8135e27586
fix: redact sensitive data in SSO and LDAP
2023-09-26 11:06:47 +08:00
17206f8c75
Merge pull request #11662 from thalesmg/port-scan-check-proto-dist-m-20230922
...
chore: check ekka proto dist module type when resolving node address
2023-09-25 17:01:43 -03:00
5d212e1086
fix(audit): only support audit log on enterprise edition
...
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
0498e59c45
test: fix flaky test
2023-09-25 13:35:17 -03:00
95060302fd
fix: don't need to change audit log's level
2023-09-26 00:28:47 +08:00
ff7f37ccf5
test(cth): allow defining schema to load for app
2023-09-25 13:22:41 -03:00
b1f144ab8b
feat(session): add custom session timers mechanism
...
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
...
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
5e6996dc05
refactor: log formatter format mfa+line as m:f/a(line)
...
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
6a557980e4
refactor(logger): ensure JSON log field order
...
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
5f45ba50ff
refactor: delete log formatter config for audit log handler
...
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
1fed38c248
fix(logger): write 'json' format logs as JSON
2023-09-25 16:55:22 +02:00
aea1e80290
feat: add 'format' as alias for log 'formatter'
2023-09-25 16:55:22 +02:00
9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify
2023-09-25 15:48:01 +02:00
806017ef90
chore: check ekka proto dist module type when resolving node address
...
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917
Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
cfdb25b213
fix(sso): updates the SSO backend when the `[dashboard]` has updated
2023-09-25 20:30:47 +08:00
f1bc3b68b6
Merge pull request #11674 from id/0925-e5.3.0-alpha.2
...
e5.3.0 alpha.2
2023-09-25 14:21:46 +02:00
6f29bbf997
Merge pull request #11673 from HJianBo/redact_audit_log_from_cli
...
Redact audit log from CLI
2023-09-25 20:03:45 +08:00
81cf619f07
fix(ftconf): also mark `secret_access_key` key as sensitive
2023-09-25 14:52:57 +03:00
36d3a3a524
fix: bump to quicer 0.0.201
...
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
22193d273a
fix(quic): ignore undefined cacertfile
2023-09-25 12:57:55 +02:00
8c93c79b44
chore: e5.3.0-alpha.2
2023-09-25 12:52:45 +02:00
7d58f6c61e
Merge pull request #11668 from JimMoen/saml-login-redirect
...
fix: saml login acs redirect to dashboard overview
2023-09-25 18:11:03 +08:00
b970a34ee0
chore: redact some audit logs from CLI
2023-09-25 18:06:39 +08:00
89e15e9134
Merge pull request #11665 from HJianBo/typo-fixes
...
Typo fixes
2023-09-25 17:23:40 +08:00
1cd9df6461
Merge pull request #11669 from lafirest/fix/dashboard_cli
...
fix(sso): refactor the `admins` CLI
2023-09-25 16:57:18 +08:00
e63d484632
fix(sso): move the config path of the SSO feature to `dashboard.sso`
2023-09-25 15:23:52 +08:00
e1f1c64ecd
Merge branch 'release-53' into 0925-sync-r53-to-master
2023-09-25 08:12:45 +02:00
e94192d1fa
fix(sso): refactor the `admins` CLI
...
1. revert the opensource version
2. allow delete the SSO user via CLI
2023-09-25 13:53:01 +08:00
4f4868a46c
fix(sso): Disable access to `logout` endpoint by the `API key`
2023-09-25 10:31:38 +08:00
30862a94c6
chore: typo fixes
2023-09-24 20:19:05 +08:00
ad4fadc2fa
fix: saml login acs redirect to dashboard overview
2023-09-23 17:29:02 +08:00
f8d06614c0
chore: fix dialyzer warnings
2023-09-23 07:34:04 +08:00
1dddccb448
fix(saml): cert files cleanup when destroy
2023-09-22 22:49:08 +08:00
80a6c1150d
fix(saml): saml login reply role `viewer` as default
2023-09-22 22:39:02 +08:00
cc3e4e4dc5
fix(saml): drop cert and key content and return path
2023-09-22 22:37:04 +08:00
6349cd3910
fix(saml): sp sign request
2023-09-22 21:57:50 +08:00
2a8f3f9eaa
fix: saml xml metedata format
2023-09-22 21:50:43 +08:00
a318ad486a
refactor: behavior login/2 use all http request
2023-09-22 21:50:43 +08:00
9181ec844f
chore: split out sso_saml_api module
2023-09-22 21:50:43 +08:00
df94426ee3
chore: make static_check happy
2023-09-22 21:50:42 +08:00
ec0894ca0b
chore: update esaml vsn
2023-09-22 21:50:42 +08:00
4a26f63bd6
chore: fix bugs
2023-09-22 21:50:41 +08:00
47badc3181
chore: make dialyzer happy
2023-09-22 21:49:12 +08:00
1c78c6bf6d
chore: fix 500 crashes when backend not existed
2023-09-22 21:49:12 +08:00
b4fb5196cb
fix(sso): SSO management API 500
2023-09-22 21:49:12 +08:00
bba5cc44a8
fix: keep same API path style
2023-09-22 21:49:12 +08:00
8300cd42d4
fix: acl url ignore auth check
2023-09-22 21:49:12 +08:00
44836ef5ee
chore: bump esaml vsn to v1.1.1
2023-09-22 21:49:12 +08:00
13666fa9f9
refactor: avoid dynamic call
2023-09-22 21:49:12 +08:00
c9e0d4fc30
feat: saml integration for dashboard sso
2023-09-22 21:49:10 +08:00
d9466eef63
chore: fix Dashboard RBAC license and rebar.config
2023-09-22 21:19:45 +08:00
fd932c9bf7
Merge pull request #11660 from zmstone/0922-fix-audit-eval-command
...
fix(audit): make emqx eval command auditable
2023-09-22 14:02:47 +02:00
a34ab19d93
fix(audit): make emqx eval command auditable
2023-09-22 12:05:45 +02:00
0eed01abee
fix: create ssl listener return 500 crash
2023-09-22 17:34:19 +08:00
13b5e4dbc9
Merge pull request #11658 from lafirest/fix/sso_misc
...
Fix/sso misc
2023-09-22 14:44:39 +08:00
7286d15ca6
chore(sso): adjust the schema of the SSO LDAP backend
2023-09-22 13:48:20 +08:00
9e55ae240a
feat(sso): add `role` into the result of login endpoints
2023-09-22 13:48:07 +08:00
681e57dee6
fix(RBAC): allow read-only users to logout
2023-09-22 11:06:24 +08:00
d6935b6a67
feat: add port scan diagnostics to mria waiting for tables checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-10944
Also updates ekka -> 0.15.15, mria -> 0.6.4
How to test
===========
1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.
Or, more easily:
1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.
Example output
==============
```
Check check_open_ports should get ok but got #{msg =>
"some ports are unreachable",
results =>
#{'emqx@172.100.239.4' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
4}],
status =>
bad_ports},
'emqx@172.100.239.5' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
5}],
status =>
bad_ports}}}
```
After one node is back:
```
Check check_open_ports should get ok but got #{msg =>
"some ports are unreachable",
results =>
#{'emqx@172.100.239.4' =>
#{ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
4}],
status => ok},
'emqx@172.100.239.5' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
5}],
status =>
bad_ports}}}
```
2023-09-21 14:29:01 -03:00
7cf60c5a91
chore: e5.3.0-alpha.1
2023-09-21 19:29:00 +02:00
acf4227fc6
test(session): fix quic testgroup in persistent session suite
...
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
76f614e9c5
Merge pull request #11653 from thalesmg/fix-dash-rbac-dialyzer-error-m-20230921
...
chore: fix dialyzer errors on ce version
2023-09-21 13:06:09 -03:00
85a8c174d9
chore: fix dialyzer errors on ce version
...
```
apps/emqx_dashboard/src/emqx_dashboard.erl
Line 225 Column 17: The pattern {'error', 'unauthorized_role'} can never match the type {'error','not_found' | 'token_timeout'} | {'ok',binary()}
```
2023-09-21 11:07:31 -03:00
b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
...
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log
2023-09-21 18:39:23 +08:00
0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
...
sync r52 to master
2023-09-21 12:31:24 +02:00
3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
...
quicer 0.0.200
2023-09-21 12:20:14 +02:00
dafd7c6085
chore: bump apps versions
2023-09-21 10:58:42 +02:00
21bb209fb1
feat: added support for auditing API and CLI activity in logs
2023-09-21 16:35:46 +08:00
105bebc250
chore: merge release-52 into master
2023-09-21 10:22:47 +02:00
a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently
2023-09-21 11:53:40 +04:00
9a128517d8
fix(sso): use the correct way to mark the API as authorization-free
2023-09-21 15:52:05 +08:00
dfc14a39bc
test: add emqx_management to telemetry SUITE
2023-09-21 14:15:06 +08:00
2283db62ec
fix(sso): add a API with authorization-free to list running backends
2023-09-21 11:52:30 +08:00
123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
...
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
9521e11d39
Merge pull request #11584 from zhongwencool/telemetry-error-on-windows
...
fix: telemetry report error on windows
2023-09-21 09:08:55 +08:00
b924fb618a
Merge pull request #11639 from lafirest/test/sso
...
test(sso): add test case for the integration of SSO with LDAP
2023-09-21 09:01:13 +08:00
1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify
2023-09-21 07:32:39 +08:00
a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
...
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy
2023-09-20 22:55:54 +04:00
21e82b9534
test(sessmem): make retry delivery testcase more involved
2023-09-20 22:55:54 +04:00
69889d14a3
fix(sessds): fix use of undefined types
2023-09-20 22:55:54 +04:00
c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
...
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
...
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
98706cd215
chore: ensure comments follow code style consistently
2023-09-20 22:55:25 +04:00
8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability
2023-09-20 22:55:25 +04:00
7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session
2023-09-20 22:55:25 +04:00
045d8b7f10
refactor(ds): reorder functions to improve readability
2023-09-20 22:55:24 +04:00
540ca6d60f
chore: drop few commented out and irrelevant pieces
2023-09-20 22:55:24 +04:00
9d145890cc
refactor(sessmem): pass log context as part of session event
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
e422f492ef
test(sessds): reuse and expand persistent session test suite
2023-09-20 22:55:24 +04:00
3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling
2023-09-20 22:55:23 +04:00
7326ef550b
fix(sessds): make existing parts of persistent session impl work
2023-09-20 22:55:22 +04:00
e713fc38aa
feat(broker): reflect persisted messages in publish result
...
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
f4953e719b
fix(cmproto): fix few typespecs
2023-09-20 22:53:45 +04:00
8af107e28d
test(ds): simplify cluster test setups
2023-09-20 22:53:45 +04:00
7c4f68dd3d
fix(session): make utility function names consistent
...
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem`
2023-09-20 22:53:44 +04:00
abeff0bc4f
chore(session): try to describe what happens after session takeover
2023-09-20 22:53:44 +04:00
45d44df11d
refactor(session): update eviction channel session logic
...
The changes partially reflect `emqx_channel` changes with respect to
in-memory session specific logic. The difference is that eviction
channel does not replay post-takeover, instead enqueues messages.
2023-09-20 22:53:44 +04:00
ab1c4c4222
refactor(session): drop `lookup/1` session API
...
Due to the fact it's not used for anything right now.
2023-09-20 22:53:44 +04:00
97881ff3ca
refactor(session): bring back common session timers
2023-09-20 22:53:43 +04:00
bf16417513
feat(session): introduce session implementation concept
2023-09-20 22:53:42 +04:00
780ca15298
chore: bump application versions
2023-09-20 22:52:14 +04:00
04731b7ef7
test(takeover): randomize messages for random natural ordering
2023-09-20 22:52:14 +04:00
f022c9b1a4
feat(emqx): add `emqx_inflight:fold/3` generic function
2023-09-20 22:52:14 +04:00
596ce157fd
refactor(exproto): make timer names equal to messages they send
...
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
57ae5b14f1
refactor(mqttsn): make timer names equal to messages they send
...
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
e4866adc2f
refactor(chan): make timer names equal to messages they send
...
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
cfb1bf1fa4
chore(emqx): drop remnants of former session persistence impl
...
1. It is not functional anyway.
2. It blocks `emqx_session` refactoring in a few places.
2023-09-20 22:52:13 +04:00
e43b87f18e
test(quic): update emqx_broker_SUITE
2023-09-20 20:20:18 +02:00
2965fa6fcb
Merge pull request #11613 from thalesmg/ds-keyspace-m-20230914
...
feat(ds): introduce keyspace concept
2023-09-20 14:21:27 -03:00
da626f6d29
feat(quic): bump to quicer 0.0.200 and emqtt 1.9.0
2023-09-20 16:43:15 +02:00
dd687d9582
fix: dialyzer warning
2023-09-20 22:41:26 +08:00
9e75ff88af
chore: apply suggestions from code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:15:39 +08:00
f789a7504f
chore: update apps/emqx/src/emqx_quic_connection.erl
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:00:44 +08:00
09aea47f40
chore: 5.2.1
2023-09-20 14:04:28 +02:00
21c37a3f14
fix(sso): adjust the schema of LDAP API && add more logs
2023-09-20 18:15:42 +08:00
2f1fa2e961
chore: unified slog message formatting to improve logging consistency
2023-09-20 18:13:00 +08:00
c26a18e949
fix: always return ok when remove local resource
2023-09-20 18:02:42 +08:00
b11c5d26ce
test(sso): add test case for the integration of SSO with LDAP
2023-09-20 16:25:19 +08:00
bc6edac45f
Merge pull request #11631 from lafirest/feat/dashboard_ldap
...
feat(dashboard): add SSO feature and integrate with LDAP
2023-09-20 11:40:42 +08:00
61311081d8
fix(sso): fix RBAC test errors && spellcheck
2023-09-20 10:42:17 +08:00
5ed5ac48ee
refactor: combine shard id and keyspace into a single value
2023-09-19 12:31:32 -03:00
5b00998050
fix(sso): fix invalid_dynamic_call
2023-09-19 22:47:48 +08:00
285e529766
fix(sso): fix sso errors found when manual test
2023-09-19 19:15:52 +08:00
65818740f3
Merge pull request #11622 from zmstone/0917-upgrade-gen_rpc-to-3.1.0
...
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-19 10:22:53 +02:00
a69c26b86d
chore: e5.2.1-alpha.3
2023-09-19 08:17:29 +02:00
2cddce5479
feat(dashboard): add SSO feature and integrate with LDAP
2023-09-19 14:10:29 +08:00
de1bc04843
chore: bump ekka to 0.15.14, mria to 0.6.3
...
Inclued ekka's updates:
- https://github.com/emqx/ekka/pull/214
- https://github.com/emqx/ekka/pull/215
- https://github.com/emqx/ekka/pull/216
- https://github.com/emqx/ekka/pull/217
mria's updates:
- https://github.com/emqx/mria/pull/160
- https://github.com/emqx/mria/pull/161
- https://github.com/emqx/mria/pull/162
2023-09-19 09:43:19 +08:00
a511088fd4
refactor: address review comments
2023-09-18 17:47:56 -03:00
9f8a7a96af
Merge pull request #11628 from zmstone/0918-upgrade-emqtt-to-1.8.7
...
0918 upgrade emqtt to 1.8.7
2023-09-18 21:56:04 +02:00
de93d71014
test: fix emqx_sahred_sub_SUITE:t_remote
...
prior to this fix, the peer ct node is actually node itself
2023-09-18 21:54:43 +02:00
99b60849b4
test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer
2023-09-18 20:44:38 +02:00
0b0e322473
Merge pull request #11627 from SergeTupchiy/EMQX-10962-fix_hstreamdb-bridge-on-stop
...
fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
2023-09-18 19:57:07 +03:00
d1e5b69eac
test: increase wait time
2023-09-18 17:33:09 +02:00
4f1ec795eb
test: delete special handling of emqtt race condition
...
the race should have been fixed in emqtt 1.8.7
2023-09-18 17:27:34 +02:00
46557c3462
fix(emqtt): upgrade from 1.8.6 to 1.8.7
2023-09-18 17:04:23 +02:00
80cf60e9be
fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
...
fixes: EMQX-10962
2023-09-18 17:48:05 +03:00
45f0d0dfce
Merge pull request #11623 from zmstone/0917-upgrade-esockd-to-5.9.7
...
0917 upgrade esockd to 5.9.7
2023-09-18 16:30:54 +02:00
c868264f65
Merge pull request #11617 from zhongwencool/rule-engine-console-args
...
fix: keep builtin_console actions not complain args
2023-09-18 20:40:02 +08:00
8b6d5d4ca9
test: add test for build_console's args
2023-09-18 16:48:38 +08:00
5490807b20
chore: update apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-18 16:11:39 +08:00
f66d9e76fe
fix(emqx_channel): aovid logging frame errors as exceptions
2023-09-18 09:04:00 +02:00
6fe846bf0e
Merge pull request #11610 from lafirest/feat/rbac
...
feat(dashboard): add RBAC feature for Dashboard
2023-09-18 14:41:47 +08:00
ca6547c05b
fix(rbac): change default role and update changes
2023-09-18 11:40:35 +08:00
abbee4bc47
fix(esockd): upgrade esockd from 5.9.6 to 5.9.7
2023-09-17 19:44:04 +02:00
985f0e6994
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-17 19:10:13 +02:00
a9682c2989
Merge pull request #11605 from zmstone/0913-lower-severity-level-for-CMD-override-warnings
...
0913 lower severity level for cmd override warnings
2023-09-16 11:32:07 +02:00
ac41f497b4
feat: add bytesize rule sql function
2023-09-15 16:04:21 +02:00
2aae3856a3
Merge pull request #11596 from thalesmg/fix-typespec-typos-m-20230912
...
fix: fix typespec typos
2023-09-15 09:24:42 -03:00
f0d2b2e6c3
chore: update emqx_eviction_agent version
2023-09-15 13:24:38 +03:00
4b97d3f57d
test(rbac): add test cases for RBAC && update changes
2023-09-15 18:16:28 +08:00
1959e15703
chore: simplify session eviction for node rebalance
2023-09-15 11:46:51 +02:00
32d838c85b
fix: format console action crash
2023-09-15 17:40:56 +08:00
640dd130fa
fix: don't urldecode twice clientid
2023-09-15 17:36:48 +08:00
d72348c8ed
fix: keep builtin_console actions not complain args
2023-09-15 16:38:04 +08:00
45ccc66474
fix(dashbaord): fix test cases to support RBAC
2023-09-15 15:49:37 +08:00
aace9215a4
Merge pull request #11608 from lafirest/feat/ldap_bind
...
feat(ldap): integrate authentication with LDAP bind operation
2023-09-15 14:06:39 +08:00
0846939760
fix(ldap): remove unused code and mark sensitive field
2023-09-15 10:13:07 +08:00
b30bcf32bd
feat(ds): introduce keyspace concept
...
Fixes https://emqx.atlassian.net/browse/EMQX-10579
This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.
We might want to tune the storage options differently for distinct sets of topics, the
keyspaces. The keyspace is composed by one or more shards.
- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
11c39c4b6a
chore: simplify session eviction for node rebalance
2023-09-14 18:28:25 +03:00
6be0e14e8c
fix: fix typespec typos
2023-09-14 10:21:05 -03:00
e41f7dd68c
Merge pull request #11598 from thalesmg/ds-fix-ps-router-m-20230912
...
fixes and improvements to persistent session router
2023-09-14 10:00:58 -03:00
5c31c5ce76
feat(dashboard): add RBAC feature for Dashboard
2023-09-14 19:28:37 +08:00
afbf13b8a2
feat(ldap): integrate authentication with LDAP bind operation
2023-09-14 14:10:43 +08:00
3eb46be2ae
fix(banned): fix typos
2023-09-14 14:01:59 +08:00
95f1312df7
chore(emqx_ctl): lower command override log from warning to info
...
prior to this change this warning message is always logged if a node
joins cluster.
the intention of this warning is to detect potential command name
clash (provided by different plugins), so arguably it should be
at debug level
2023-09-13 20:52:08 +02:00
803b69d878
refactor: use more consistent return types
...
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2023-09-13 10:49:45 -03:00
f55f6dc779
docs(ps_session): add notes explaining rationale about operation order and consistency
2023-09-13 10:12:27 -03:00
e081abb9e6
perf(ps_router): only check if there are any routes when deciding whether to persist message
2023-09-13 10:02:39 -03:00
cae9ae1fab
fix(ps_router): use `disc_copies` for storing persistent session routes
2023-09-13 10:02:39 -03:00
ea73d674b9
chore: bump version to e5.2.1-alpha.2
2023-09-13 14:55:17 +02:00
69d17f981d
Merge pull request #11595 from ieQu1/dev/dont-prevent-overlapping-partitions
...
fix(distribution): Set prevent_overlapping_partitions to false
2023-09-13 14:22:34 +02:00
e1456c1c04
feat(emqx_conf_schema): add node.default_bootstrap_batch_size config parameter
2023-09-13 10:11:14 +03:00
af015bc2bb
chore(ekka): bump version to 0.15.13
2023-09-13 10:11:14 +03:00
b41310af9c
fix: Add prevent_overlappiing_partitions to vm_args
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-12 20:01:56 +02:00
1a86c21e20
fix(distribution): Set prevent_overlapping_partitions to false
2023-09-12 17:41:51 +02:00
6ef296268e
chore: bump version to e5.2.1-alpha.1
2023-09-12 15:24:02 +02:00
59b94788bf
refactor: address review comments
2023-09-11 16:25:34 -03:00
64fcdb671d
test: fix a bunch of inter-suite interferences
2023-09-11 16:25:34 -03:00
f1294736b7
feat(ds): add filter for message persistence
...
Fixes https://emqx.atlassian.net/browse/EMQX-10520
2023-09-11 16:25:34 -03:00
fe4640922d
feat(ds): add delete callback
2023-09-11 16:25:33 -03:00
7805cc8c9b
fix(session): check if session is persistent
2023-09-11 16:25:33 -03:00
f490da71cd
chore: prepare v5.2.0-build.1
2023-09-11 16:09:17 -03:00
d0f1c75651
chore: bump apps vsns
2023-09-11 16:06:53 -03:00
223d47a6ac
Merge branch 'master' into sync-m-r52-20230911
2023-09-11 16:04:28 -03:00
063913f245
fix(session): remove recently added `iterators` field from `#session{}` record
...
Fixes https://emqx.atlassian.net/browse/EMQX-10945
This could lead to `badrecord` errors being raised if a takeover were to happen during a
rolling cluster upgrade, as the old nodes could receive a record with more fields than
expected.
2023-09-11 14:46:12 -03:00
887a74a6cb
fix: telemetry report error on windows
2023-09-11 16:48:30 +08:00
7010ac358c
chore: bump release version to v5.2.0
2023-09-08 16:31:52 +02:00
3d133801af
fix(router): use `ets:new/2` options compatible with OTP 24
...
While essentially keeping the same effect.
2023-09-08 18:04:29 +04:00
e089fda260
Merge pull request #11568 from thalesmg/republish-props-m-20230905
...
feat(republish): allow templating mqtt properties
2023-09-07 16:43:00 -03:00
f3bebb0ba2
Merge remote-tracking branch 'origin/master' into 0907-sync-master-to-release-52
2023-09-07 18:51:30 +02:00
68293231b8
test: add cases for more complex placeholders
2023-09-07 13:40:03 -03:00
014bc64d3b
chore: ensure general prop is a binary
2023-09-07 13:40:03 -03:00
02b8bbf76a
chore: drop templating support for `Subscription-Identifier`
...
This is highly-dependent on the session state, as is `Topic-Alias`.
2023-09-07 13:40:03 -03:00
47bd19dee4
chore: lower error log messages down to debug
2023-09-07 13:40:03 -03:00
080cb73da1
fix: handle badmap inside `emqx_placeholder:proc_tmpl`
2023-09-07 13:40:03 -03:00
db2ec55aa0
chore: bump release version to official e5.2.0
2023-09-07 09:40:15 +02:00
7c2f87fabe
test: merge broker and router boot modules
2023-09-06 21:36:16 +02:00
bb55b04d46
refactor: inline fn clauses, improve error logging
2023-09-06 13:54:40 -03:00
c9100c7591
refactor: improve logging information and descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-06 13:33:59 -03:00
78c5a779d7
feat(republish): allow templating mqtt properties
...
Fixes https://emqx.atlassian.net/browse/EMQX-10912
2023-09-06 09:54:02 -03:00
7fe704b6dd
Merge pull request #11563 from kjellwinblad/kjell/fix/release-52/EMQX-10775
...
fix: update gpb library to fix type error
2023-09-06 14:47:23 +02:00
18c6bfec97
chore: bump app vsns
2023-09-06 11:12:56 +02:00
e794143ae1
Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master
2023-09-06 09:08:22 +02:00
954b73a9aa
Merge pull request #11533 from thalesmg/ds-unsubscribe-m-20230828
...
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
2023-09-05 09:47:17 -03:00
01a128878f
chore(mria): Bump mria to 0.6.1
2023-09-05 11:08:32 +02:00
d26995a2aa
Merge pull request #11524 from keynslug/ft/EMQX-10713/unified-route-tab
...
feat(router): add routing storage schema with 2 tables
2023-09-05 12:38:00 +04:00
2b2e84b83c
fix: update gpb library to fix type error
...
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-09-05 09:52:17 +02:00
23d63f5e01
refactor: fork clientid type for `emqx_ds:session_id/0`
2023-09-04 15:38:25 -03:00
60ae3c15c8
refactor: disregard impossible case
2023-09-04 15:38:25 -03:00
2cfe5ba057
fix(plugins_api): fix response types and test
2023-09-04 14:37:25 -03:00
02c3d6a0a7
fix(data_import): rm duplicate import call and fix test
2023-09-04 14:19:59 -03:00
73d4248334
test(cth_cluster): always allocate listener ports
2023-09-04 11:22:26 -03:00
9f0f218387
test(router): simplify cluster-related testcase in helper testsuite
2023-09-04 18:05:59 +04:00
7d26b7bc1a
fix(router): emit clearer error when conflicting schemas in use
2023-09-04 16:37:38 +04:00
d33ad8a46f
feat: cluster purge (r5.2)
...
Port of https://github.com/emqx/emqx/pull/11447
targeting `release-52`
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-09-04 09:15:28 -03:00
893f69617a
chore(test): drop obsolete TODOs
2023-09-04 14:40:49 +04:00
f0a0c7d4b1
fix(cthsuite): avoid duplicate application env entries
...
Because `application:set_env/1` complains loudly if there are
duplicates.
2023-09-04 14:40:49 +04:00
5024304bf9
fix(router): wait for tables replicate before choosing schema vsn
2023-09-04 14:40:48 +04:00
24230a64df
chore: bump app versions
2023-09-04 11:39:21 +02:00
b2f57636bd
Merge remote-tracking branch 'upstream/release-51' into 0904-sync-release-51
2023-09-04 11:34:22 +02:00
52546ac50f
Merge pull request #11522 from kjellwinblad/kjell/fix/EMQX-10778
...
fix: bad error message when rule engine schema name is too long
2023-09-04 11:05:55 +02:00
240afecd69
Merge pull request #11487 from savonarola/0821-optimize-bcrypt
...
feat: reduce bcrypt rounds to a usable value
2023-09-04 10:12:40 +03:00
ce571d892e
chore: ping opentelemetry v1.3.0-emqx
2023-09-04 09:34:09 +08:00
ed9afe3345
refactor(emqx_connector): use `hocon_schema:override/2` to make pgsql 'username' field required
2023-09-01 19:56:07 +03:00
607705518b
test(emqx_bridge_cassandra): add connector test case for Cassandra configured without authentication
2023-09-01 19:56:07 +03:00
f0c75d97e1
chore: upgrade emqx_http_lib to 0.5.3
...
0.5.3 release includes a fix to parse <host>:<port> URLs using the default http scheme.
2023-09-01 19:56:07 +03:00
c3b3460173
fix(emqx_oracle): make username a required field in emqx_oracle_schema
2023-09-01 19:56:07 +03:00
4ab5f8374b
chore(schema): mark routing storage schema readOnly
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-01 19:58:31 +04:00
ebf13c90ea
test(router): make sort stable in e2e routing testcase
2023-09-01 19:48:56 +04:00
90156befb5
feat: improve bcrypt usability
...
* limit salt rounds to usable values
* update bcrypt library to enable concurrent bcrypt hash calculation
2023-09-01 17:36:11 +03:00
c54527857a
fix(emqx_bridge_sqlserver): use empty password by default as it is not a required field
2023-09-01 16:57:35 +03:00
1bf86250dd
fix(emqx_bridge_cassandra): allow cassandra bridge without username/password
...
Cassandra can be used without credentials, if it is configured with AllowAllAuthenticator (default).
2023-09-01 16:57:35 +03:00
545f1c84a6
fix(cth): do not allocate ports for `emqx` app by default
...
This causes tricky and impressively hard to track side effects down
the line. Namely, loading `emqx_schema` _after_ `emqx_conf_schema`
(as part of cluster node startup sequence) leads to a couple of
schema root rewrites, because `emqx_schema` defines similar config
roots yet slightly differently (e.g. `authorization`).
2023-09-01 17:23:34 +04:00
0a879bffd1
test(router): add e2e testcase for cluster-wide routing
...
Which verifies that both router storage schemas work correctly.
2023-09-01 13:49:12 +04:00
eb0385a28f
chore(router): separate test-only function exports
2023-09-01 13:49:11 +04:00
3d859453f4
Merge pull request #11554 from lafirest/chore/exhook_prop
...
chore: shutdown reason will never be a UTF8 atom
2023-09-01 16:53:26 +08:00
3bb65e6b05
Merge pull request #11542 from kjellwinblad/kjell/fix/EMQX-10775
...
fix: update gpb library to fix type error
2023-09-01 10:07:03 +02:00
53940754b7
chore: shutdown reason will never be a UTF8 atom
2023-09-01 14:17:33 +08:00
17fe406605
Merge pull request #11551 from paulozulato/refactor-connector-config-callback
...
refactor: remove BridgeName from connector_config callback
2023-08-31 17:05:49 -03:00
bf0f77b6dc
Merge pull request #11548 from thalesmg/fix-plugin-order-update-r52-20230830
...
fix(plugins): update plugin order on whole cluster
2023-08-31 16:18:28 -03:00
7b607c3300
refactor: remove BridgeName from connector_config callback
...
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.
Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
2023-08-31 11:01:49 -03:00
c20ba0572a
chore: bump to 5.2.0-alpha.4
2023-08-31 15:47:39 +02:00
876d539336
Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table
2023-08-31 15:46:33 +02:00
fb094e1d47
test(router): avoid testsuite collisions
2023-08-31 13:36:01 +04:00
727fd296ee
Merge pull request #11546 from paulozulato/fix-kafka-producer-unique-name
...
fix(kafka): avoid producer name collision between Kafka and AEH bridges
2023-08-30 18:08:54 -03:00
4dfbc859f9
fix(plugins): update plugin order on whole cluster
...
Fixes https://emqx.atlassian.net/browse/EMQX-10879
2023-08-30 16:39:52 -03:00
ee77976424
fix(kafka): avoid producer name collision between Kafka and AEH bridges
...
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-30 14:48:10 -03:00
2da604dfa4
test(mgmt): simplify flaky `t_verify_imported_mnesia_tab_on_cluster`
...
And also rest of the test setup in the suite.
2023-08-30 20:24:18 +04:00
b1defa29d7
feat(ruleeng): avoid storing whole rules in topic index
...
Because it doesn't really give any benefit, but wastes memory by
duplication.
2023-08-30 20:07:56 +04:00
6b9cb06334
fix(router): add / refine some comments and log messages
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-30 19:51:31 +04:00
922ca5e141
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
...
Fixes https://emqx.atlassian.net/browse/EMQX-9742
2023-08-30 11:14:28 -03:00
9c6dd30f44
feat(session): store iterator ids in session record
2023-08-30 11:13:55 -03:00
dfcede8794
fix: increment matched counter when bridge is unhealthy
...
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-30 10:52:53 -03:00
f8923ab419
Merge pull request #11539 from keynslug/ft/cth-desc-workdir
...
feat(cthsuite): add function to determine workdir of testrun
2023-08-30 17:15:55 +04:00
1cab687153
Merge pull request #11540 from thalesmg/fix-cert-path-utf8-r52-20230829
...
fix(bridge): validate bridge name before attempting to convert certificates
2023-08-30 09:45:55 -03:00
8b3a9072f9
Merge pull request #11532 from lafirest/fix/frame_parse
...
fix(frame): improve some error reasons when parsing invalid packet
2023-08-30 19:10:45 +08:00
d911f7fbea
fix: update gpb library to fix type error
...
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-08-30 11:12:33 +02:00
9fe4382d98
Merge pull request #11530 from JimMoen/chore-bump-hstreamdb-driver
...
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-30 14:37:17 +08:00
804443ba40
chore: fix typos
2023-08-30 11:22:09 +08:00
06fad14d07
Merge pull request #11536 from zhongwencool/comma_separated_binary
...
chore: ciphers should allow empty space(comma_separated_binary)
2023-08-30 10:20:06 +08:00
2ef60db147
Merge pull request #11481 from savonarola/0817-simplify-async-responses
...
chore(ft): refactor async reply mechanism
2023-08-30 01:20:09 +03:00
54ac4a8527
chore(ft): tidy up the code according to the review
2023-08-29 23:21:36 +03:00
0f297ffef4
fix(bridge): validate bridge name before attempting to convert certificates
...
Fixes https://emqx.atlassian.net/browse/EMQX-10865
2023-08-29 16:32:01 -03:00
0e770bdc95
test: switch `emqx_broker_SUITE` to use new cth tooling
2023-08-29 22:05:36 +04:00
3268093881
feat(cth): add module-level documenation
2023-08-29 22:05:36 +04:00
f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
...
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
063d6200c8
feat(router): enable using 2 tables to store routes
...
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.
Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
2023-08-29 17:41:59 +04:00
270fd107b2
feat(triesearch): allow reusing filter tokenization result
...
As a kind of micro-optimization.
2023-08-29 17:31:26 +04:00
501456efec
chore: more safe with iolist_to_binary
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-29 17:10:43 +08:00
4e7ba5d35f
chore: ciphers should allow empty space(comma_separated_binary)
2023-08-29 11:34:40 +08:00
6722722522
fix(frame): improve some error reasons when parsing invalid packet
2023-08-29 10:53:20 +08:00
4245a4d8c6
fix: authz clean-cache clientid always return not_found
2023-08-29 08:17:47 +08:00
cc3ba18734
fix: increment dropped message counter when bridge is unhealthy
...
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
2c89be04d1
Merge pull request #11527 from paulozulato/fix-kafka-header-template-handling
...
fix(kafka): fix template processing for header
2023-08-28 18:55:33 -03:00
0b86f04bae
fix(kafka): fix template processing for header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
279895b8fd
chore(ft): add read/write concurrency to emqx_ft_async_reply tabs
2023-08-28 22:46:30 +03:00
c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log
...
fix(emqx_trace): don't download empty trace log file
2023-08-28 22:17:00 +03:00
0072749143
fix(emqx_trace): don't download empty trace log file
...
Closes: EMQX-10274
2023-08-28 19:25:55 +03:00
5908b69353
chore: hide cluster purge from cli and api for now
2023-08-28 10:23:21 -03:00
f988de4ff4
feat(purge): clear delayed messages
2023-08-28 10:23:21 -03:00
9bded07834
feat: cluster purge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
dec21ffc95
Merge pull request #11362 from thalesmg/ds-handle-sub-20230725
...
feat(ds): open iterators when handling `SUBSCRIBE` packets
2023-08-28 10:21:28 -03:00
2d931a0512
test(evict): do not disable router module in cluster nodes
2023-08-28 14:07:46 +04:00
8d2ebdea7e
fix(router): generalize config option and make effects visible
2023-08-28 14:07:46 +04:00
e85789306b
chore(router): drop test-only `has_routes/1`
...
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
5d51687dbf
feat(router): add unified routing table config option
2023-08-28 14:07:45 +04:00
33e5e1ba57
feat(router): add unified routing table
2023-08-28 14:07:44 +04:00
166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
...
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
84e40fb6fe
test(trie): add more involved route add + delete case
...
That shows how current local trie implementation breaks because
of lack of refcounting.
2023-08-28 13:52:59 +04:00
dcb63440bc
refactor(mgmt): avoid dealing with router tab directly
...
Instead, contain all this behind `emqx_router` module interface.
2023-08-28 13:52:59 +04:00
9eccfa0909
refactor(router): isolate cleanup logic in router module
2023-08-28 13:52:59 +04:00
f3467f44b2
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-28 17:28:50 +08:00
4488e9e591
chore(ft): stop hook chain when doing actual handling
2023-08-28 11:57:44 +03:00
39a48179ea
chore(emqx_channel): use macros for reply construction
2023-08-28 11:57:44 +03:00
b8cacd2833
chore(ft): add tests for async reply registry
2023-08-28 11:57:44 +03:00
b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
...
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
053b7fb94a
test: add more test for conf_cli
2023-08-28 11:14:36 +08:00
d8be248a3d
fix: improve the suggest msg for update conf failed
2023-08-28 08:53:23 +08:00
b0d4a22aa8
chore(ft): refactor async reply mechanism
2023-08-27 21:54:23 +03:00
b874926de7
Merge pull request #11514 from zhongwencool/stop-load-application-before-reboot
...
fix: stop otel deps appication before reboot
2023-08-27 20:46:31 +08:00
2bb415f897
Merge pull request #11526 from paulozulato/fix-validator-return-msg
...
fix: flatten error message on resource validator
2023-08-25 17:27:31 -03:00
33a0048155
refactor: move logic to `ensure_iterator`
2023-08-25 15:24:53 -03:00
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
...
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
42877e282d
fix: flatten error message on resource validator
...
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
f15f59650d
test: rm obselete workaround code
2023-08-25 13:49:33 -03:00
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id
...
Fix Kafka/AEH ClientId uniqueness
2023-08-25 10:14:44 -03:00
d5cff533e3
chore(topicidx): drop TODO comment
2023-08-25 16:44:03 +04:00
ca59a87d47
chore(topicidx): refine example of wildcard compare
2023-08-25 16:44:03 +04:00
558402a68e
chore(topic_index): add topic validation
2023-08-25 16:44:03 +04:00
33ed53bb6a
refactor(topicidx): simplify `compare/3` further
...
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
2023-08-25 16:34:45 +04:00
fc37d235c7
refactor(topicidx): simplify seek instructions
...
Which also avoids comparing filter tail repeatedly when evaluating
it.
2023-08-25 15:24:06 +04:00
c65db82b07
fix: bad error message when rule engine schema name is too long
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-10778
2023-08-25 11:02:38 +02:00
c8af5a702c
test: remove connection_closed from test
2023-08-25 15:53:35 +08:00
fc1738188e
fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero
2023-08-25 15:18:34 +08:00
ba9cb0a1ed
Merge pull request #11512 from lafirest/fix/ldap_def_port
...
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-25 10:22:44 +08:00
8ab6eb5638
test: start emqx_opentelemetry in emqx_machine SUITE
2023-08-25 10:09:00 +08:00
cf45e80c71
feat(topicidx): iterate on trie search implementation
...
This improves matching performance and decreases GC pressure on
synthetic workloads.
2023-08-25 01:47:11 +04:00
535c7f8b43
fix(kafka): avoid ClientId collision between Kafka and Azure bridges
...
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-24 18:46:22 -03:00
d9a5a9ea2b
Merge pull request #11513 from thalesmg/kafka-fix-ts-template-r52-20230824
...
fix: use correct timestamp template for kafka and remove such fields from azure event hub producer
2023-08-24 16:26:52 -03:00
016ae0524f
fix(aeh_producer): remove timestamp template field
...
Fixes https://emqx.atlassian.net/browse/EMQX-10847
Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.
Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type
Example message consumed from AEH:
```
%{
"headers" => %{},
"key" => "",
"offset" => 4,
"topic" => "test0",
"ts" => 1692879703006,
"ts_type" => "append",
"value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```
Note the ts_type above is append.
Example message from a Kafka broker whose ts type is create:
```
%{
"headers" => %{},
"key" => "",
"offset" => 4,
"topic" => "test-topic-three-partitions",
"ts" => 1692881883668,
"ts_type" => "create",
"value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```
Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
2023-08-24 14:20:42 -03:00
0381ac0410
fix(kafka_producer): use correct timestamp template field
...
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
9fe5080705
build: bump versions before release
2023-08-24 16:46:27 +02:00
c215fe3736
fix: stop otel deps appication before reboot
2023-08-24 22:14:14 +08:00
60e6217496
fix(kafka): fix result handling when sending message with invalid header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
e5041de9cc
fix(gcp_consumer): handle 401 errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-10852
2023-08-24 09:03:34 -03:00
62423b0b12
refactor(topic_index): less special handling for leading $ words
2023-08-24 13:30:10 +02:00
a30d87e14f
refactor(topic_index): remove more unnecessary next calls
...
also avoid using records (setelement) for recursive return values
2023-08-24 12:25:04 +02:00
a1e6635614
refactor(topic_index): no forced ceiling entry in index table
2023-08-24 12:25:04 +02:00
f4c8c6be55
refactor(topic_index): optimize trie-search performance
2023-08-24 12:25:04 +02:00
6b152b3cb7
test: add more debug output
2023-08-24 12:25:04 +02:00
ae094e363c
chore: fix a typo in log message
2023-08-24 12:25:04 +02:00
6085442958
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-24 16:50:38 +08:00
2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics
...
Prometheus metrics
2023-08-24 12:06:32 +08:00
8d2be27ade
chore: add changlog for 11466
2023-08-24 06:46:50 +08:00
84dd3b83bb
fix: hocon converter not working when make_serializable is true(check_plain)
2023-08-24 06:46:47 +08:00
f72ba6819f
Merge pull request #11498 from zhongwencool/fix-otel-stop-error-log
...
fix: stop otel deps application in terminate crashed
2023-08-24 06:40:24 +08:00
d287b736ec
chore: add changelog for 11497
2023-08-23 17:02:30 +08:00
58be7f709b
fix: stop otel deps application in terminate crashed
2023-08-23 16:45:41 +08:00
6bb15541c7
feat: rename olp metrics to overload_protection metrics
2023-08-23 15:25:54 +08:00
b9a97923c4
feat: add some authz/connection missing metrics
2023-08-23 14:47:53 +08:00
141676185d
feat: add authentication metrics to prometheus
2023-08-23 14:15:16 +08:00
4928c69758
feat: add authorization metrics to prometheus
2023-08-23 14:11:59 +08:00
df9f8da178
feat: add overload_protection metrics to prometheus
2023-08-23 14:07:37 +08:00
188a81913e
test: add prometheus api test
2023-08-23 11:23:43 +08:00
ae10415fc3
feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security
2023-08-23 09:03:22 +08:00
61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators
...
fix(kinesis): set validators for static constraints
2023-08-22 18:28:35 -03:00
20b46091f9
fix(kinesis): set validators for static constraints
...
Fixes https://emqx.atlassian.net/browse/EMQX-10833
2023-08-22 14:42:44 -03:00
f007b44426
fix(data_import): rm duplicate import call and fix test
2023-08-22 13:29:15 -03:00
ee2897e5de
test(refactor): move test to integration tests dir
2023-08-22 13:29:15 -03:00
c1f49abad2
test: fix inter-suite flakiness
2023-08-22 13:29:15 -03:00
dad27091be
test: rm custom option
2023-08-22 13:29:15 -03:00
dbfacae283
fix: reinstate transactions
2023-08-22 13:29:15 -03:00
65085d012b
refactor: rename fn
2023-08-22 13:29:15 -03:00
3344bfb0bd
refactor: rm `emqx_ds_replay`
2023-08-22 13:29:15 -03:00
c46b8de938
test: remove unused things, refactor some functions
2023-08-22 13:29:15 -03:00
e8c73b06e1
docs: add comment about future test failure
2023-08-22 13:29:15 -03:00
6de0bbe76a
test(refactor): always allocate listeners for emqx app
2023-08-22 13:29:15 -03:00
a15405a800
test: fix assertions
2023-08-22 13:29:15 -03:00
c74abe79d0
refactor: reduce arity
2023-08-22 13:29:15 -03:00
021755b82b
refactor: rm iterators from DS `#session{}` record
2023-08-22 13:29:15 -03:00
e4e88ebf36
test: add scenario for node stopping midway during subscribe
2023-08-22 13:29:15 -03:00
3239f5ac5b
feat: rm unnecessary transactions, use separate table for iterator references
2023-08-22 13:29:15 -03:00
8eab389ae1
perf: avoid unnecessary transaction
2023-08-22 13:29:15 -03:00
c28c6d1b7e
fix: ensure iterator is opened
2023-08-22 13:29:15 -03:00
33ddbe80ad
refactor: remove persistence leakeage from emqx_cm level
2023-08-22 13:29:15 -03:00
e8d7bb9a67
refactor: rename module
2023-08-22 13:29:15 -03:00
9463e271c0
feat(ds): open iterators when handling `SUBSCRIBE` packets
...
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
ca697a4e14
fix: rename `emqx_ds{,_replay}:replay_id()`
2023-08-22 13:29:15 -03:00
45e2e687e5
fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication
...
Fixes https://emqx.atlassian.net/browse/EMQX-10843
2023-08-22 12:42:48 -03:00
3ffbb7decf
fix: HTTP API /api/v5/publish schema
...
The schema for the /api/v5/publish HTTP API endpoint was incorrect.
For 400 (Bad Request) error it cannot return a list but the incorrect
schema declared that the response could include a list.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10837
https://github.com/emqx/emqx/issues/11488
2023-08-22 15:21:20 +02:00
7b8a599d17
Merge pull request #11486 from lafirest/fix/ldap_autn_wrong_pw
...
fix(ldap): Return correct reason code when password is wrong
2023-08-22 18:53:17 +08:00
f8f39bf223
Merge pull request #11490 from lafirest/fix/absent_pw
...
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 18:52:58 +08:00
e20b804e6b
chore: update changes && test cases
2023-08-22 17:41:08 +08:00
184e03387a
fix(ldap): fix status detection and auto reconnecting errors
2023-08-22 17:06:00 +08:00
663d03112e
Merge pull request #11476 from Kinplemelon/kinple/for-upgrade-dashboard
...
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-22 14:28:40 +08:00
b0c5b447a4
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 11:49:32 +08:00
13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support
...
Hstreamdb tls support
2023-08-22 10:40:54 +08:00
037b0a7599
fix(ldap): Return correct reason code when password is wrong
2023-08-22 09:42:04 +08:00
4e4c93a758
Merge pull request #11469 from paulozulato/feat-redis-auth-username
...
Add support for AUTH with username on Redis
2023-08-21 16:44:21 -03:00
74d4a8325e
Merge pull request #11470 from thalesmg/gcp-consumer-rm-worker-pool-size-r52-20230817
...
fix(gcp_consumer): remove unused resource_opts fields
2023-08-21 14:49:59 -03:00
bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816
...
fix(bridge): make dryrun health check timeout more malleable
2023-08-21 14:49:46 -03:00
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
...
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
4163f06611
Merge pull request #11480 from thalesmg/rule-funcs-func-clause-r52-20230818
...
fix(rule_engine): capture function_clause errors
2023-08-21 11:16:25 -03:00
443d44aaa3
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-21 11:49:57 +08:00
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0
2023-08-21 10:40:51 +08:00
649647190e
test: new batch mechanism of hstreamdb_erl
2023-08-21 10:40:51 +08:00
5527edf442
chore: bump hstreamdb_erl to `0.4.3+v0.16.1`
2023-08-21 10:40:36 +08:00
ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
...
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7
...
chore: upgrade Kafka client wolff to 1.7.7
2023-08-20 10:14:13 +02:00
05fdf52d8e
fix(redis): set bridge as unhealthy on authentication error
2023-08-18 14:51:03 -03:00
fc35c74eab
fix(redis): use eredis_secret to avoid password leaking
2023-08-18 14:51:03 -03:00
2c458b62f5
feat(redis): add support for `username` on AUTH command
...
Fixes https://emqx.atlassian.net/browse/EMQX-9911
2023-08-18 14:50:06 -03:00
66dec69d09
fix(rule_engine): capture function_clause errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-10798
2023-08-18 13:51:19 -03:00
befc260555
fix(gcp_consumer): remove unused resource_opts fields
...
Fixes https://emqx.atlassian.net/browse/EMQX-10789
2023-08-17 17:02:02 -03:00
e48bfb52bb
Merge branch 'release-52' into sync-r52-master-20230817
2023-08-17 17:01:04 -03:00
afffdbbaa1
fix(kinesis): replace default payload template
...
Fixes https://emqx.atlassian.net/browse/EMQX-10766
2023-08-17 17:00:14 -03:00
ieQu1