yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,510 Commits 393 Branches 684 Tags 74 MiB
dependabot/github_actions/dot-github/actions/package-macos/actions-package-macos-b9d24f3e7e
Commit Graph

10450 Commits

Author SHA1 Message Date
Dennis Zhuang
8bc874c261 feat: supports async mode for greptimedb data bridge 2023-12-04 20:30:33 +08:00
Dennis Zhuang
90d0081df4 feat: upgrade greptimedb and client 2023-12-04 20:30:33 +08:00
Andrew Mayorov
130a5a5442
fix(ds): pass topics to emqx_topic:words/1 before feeding LTS tree 
So that empty levels in topics will be properly mapped into `''` atoms.
 2023-12-04 13:39:01 +03:00
Andrew Mayorov
92c4b29a4c
feat(topic): match empty topic levels more loosely 
So that the result of `emqx_topic:tokens/1` would be perfectly
matchable with the result of `emqx_topic:words/1` of a topic filter
with empty levels.
 2023-12-04 13:38:55 +03:00
Andrew Mayorov
b5c485cd55
test(sessds): add empty level topic subscription testcase 2023-12-04 13:38:42 +03:00
Serge Tupchii
e455433694 test(emqx_gateway): fix test failures caused by grpc-erl upgrade 2023-12-04 12:17:58 +02:00
JimMoen
cb992fcf55
Merge pull request #12059 from JimMoen/1130-fix-vm-time-warp 
fix: vm.args use `multi_time_warp`
 2023-12-04 11:56:08 +08:00
Andrew Mayorov
627c58b07d
fix(sessds): handle expire_awaiting_rel common timer with no-op 
Otherwise, the channel crashes when the timer is triggered.
 2023-12-02 15:00:30 +03:00
Andrew Mayorov
47bc747323
fix(sessds): drop everything related to session on CleanStart 2023-12-02 15:00:30 +03:00
Andrew Mayorov
ee9a98f0a4
chore: bump emqx_gateway_mqttsn to 0.1.7 2023-12-02 15:00:29 +03:00
Andrew Mayorov
fd26e690b8
refactor(sessds): move parts of message processing to replayer 
To simplify the processing flow, reducing the number of back-and-forth
between the session and the replayer.
 2023-12-02 15:00:29 +03:00
Andrew Mayorov
29ec73847a
fix(utils): make flattermap/2 results less variative 2023-12-02 14:38:04 +03:00
Andrew Mayorov
6255ee0833
fix(sessmem): avoid defining unsafe defaults in #session 2023-12-02 14:38:04 +03:00
Andrew Mayorov
b5ad0f9815
fix(test): subscribe with QoS 2 to receive QoS 2 message 2023-12-02 14:38:04 +03:00
Andrew Mayorov
1948719eec
test(sessds): unskip testcase related to QoS2 handling 2023-12-02 14:38:04 +03:00
Andrew Mayorov
3265d2f2aa
fix(sessds): respect subscription options when publishing 2023-12-02 14:38:03 +03:00
Andrew Mayorov
b5f39f89e3
feat(utils): add flattermap/2 as slightly more generic flatmap/2 2023-12-02 14:36:21 +03:00
Andrew Mayorov
508346f095
refactor(topicidx): split persistent term stuff off gbt-based index 2023-12-02 14:36:21 +03:00
Andrew Mayorov
88103c5f0e
refactor(session): pass config to SessionImpl:open/3 as well 
* Anticipate that connection info may change during session takeover.
* Avoid persisting session conf as part of persistent session state.
 2023-12-02 14:36:21 +03:00
Andrew Mayorov
6eba082992
Merge pull request #12075 from keynslug/fix/EMQX-11474/function-clause 
fix(channel): expect there's no session yet on disconnect
 2023-12-02 14:32:50 +03:00
Thales Macedo Garitezi
dc5e3b939c refactor(resource_manager): use macros and better differentiate status from state 
Internally in `emqx_resource_manager`, there seems to be many points where the
`gen_statem` states are conflated with resource status, since their names coincide.  While
that works for now, introducing a new `gen_statem` state, an internal state, shouldn't
necessarily imply a new, externally facing resource status.

Here we also introduce the usage of some macros to avoid the pitfalls of making a typo in
a state/status name.
 2023-12-01 18:23:05 -03:00
Zaiming (Stone) Shi
55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all 
test: replace 'slave' and 'ct_slave' with 'peer'
 2023-12-01 21:35:40 +01:00
Thales Macedo Garitezi
29ae45c39d fix(connector): don't start buffer workers for the connector itself 
Fixes https://emqx.atlassian.net/browse/EMQX-11448
 2023-12-01 17:20:38 -03:00
Zaiming (Stone) Shi
6e30718818 test: allow join cluster later 2023-12-01 20:33:39 +01:00
ieQu1
3d2e95fe3e
Merge pull request #12081 from ieQu1/dev/gen-rpc-3.3.0 
chore(gen_rpc): Bump version to 3.3.0
 2023-12-01 17:57:33 +01:00
ieQu1
476c300ecf
Merge pull request #12070 from ieQu1/dev/ds-min-max-batch-size 
Various minor fixes
 2023-12-01 17:57:15 +01:00
ieQu1
d3b7478d70 chore(gen_rpc): Bump version to 3.3.0 2023-12-01 17:10:12 +01:00
Zaiming (Stone) Shi
ab7d57e1c5 test: delete debug level logging 2023-12-01 17:07:14 +01:00
Ivan Dyachkov
0e27d031a8 chore: 5.4.0-alpha.1 2023-12-01 16:54:03 +01:00
Zaiming (Stone) Shi
6a9cc20d4d test: wait for all nodes clustered 2023-12-01 16:37:39 +01:00
Zaiming (Stone) Shi
956b25cc4c
Merge pull request #12071 from zmstone/1201-sync-release-54 
1201 sync release 54
 2023-12-01 14:38:47 +01:00
Thales Macedo Garitezi
cb60880bab
Merge pull request #12067 from thalesmg/fix-kafka-check-client-m-20231130 
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
 2023-12-01 09:55:12 -03:00
Andrew Mayorov
6b17920fea
fix(channel): expect there's no session yet on disconnect 2023-12-01 15:29:05 +03:00
Andrew Mayorov
cce474ce98
fix(session): include session term in stacktrace 2023-12-01 15:29:05 +03:00
Zaiming (Stone) Shi
1a78e7ae79 fix(connector): ensure webhook bridge convert to http connector 2023-12-01 11:56:37 +01:00
Zaiming (Stone) Shi
90b7486624 fix(redis): ensure schema namespace 2023-12-01 11:37:06 +01:00
ieQu1
e1ec560639 test(emqx): Fix flaky emqx_takeover_SUITE:t_takeover testcase 
This testcase uses QoS1, so it must account for possible duplication
of the messages.
 2023-12-01 10:06:18 +01:00
Zaiming (Stone) Shi
60d70b22dc chore: bump app vsn for emqx_auth_redis 2023-12-01 08:37:37 +01:00
Zaiming (Stone) Shi
7f4d91d490 Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54 2023-12-01 08:32:22 +01:00
ieQu1
e238602533 fix(ds): Update README 2023-12-01 08:27:05 +01:00
ieQu1
0ae618d010 fix(ds): Use emqx_rpc for calls that work with large binaries 2023-12-01 08:27:05 +01:00
ieQu1
0e625d814a feat(sessds): Make batch size configurable 
This change affects flow control. It allows to configure maximum size
of a batch, as well as fetch threshold.
 2023-12-01 08:27:05 +01:00
ieQu1
4717e56fb6 fix(sessds): Schedule poll immediately upon receiving an ack 
This commit affects the flow control, and improves the throughput by
removing a delay between freeing up space in the in-flight window and
polling new messages.
 2023-12-01 08:27:05 +01:00
ieQu1
38800c0260 refactor(sessds): Store timers in the session 2023-12-01 08:27:05 +01:00
Zaiming (Stone) Shi
22f7cc1622 test: replace 'slave' and 'ct_slave' with 'peer' 2023-12-01 08:07:09 +01:00
JimMoen
ee25d9bd9e
fix: vm.args use multi_time_warp 
See also:
https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
 2023-12-01 14:52:31 +08:00
ieQu1
69f1ca43c3 fix(sessds): Create a timer even if it's present in the map 2023-12-01 06:11:13 +01:00
ieQu1
1897e5c31b refactor(session): Use common naming conventions 2023-12-01 06:11:08 +01:00
ieQu1
d9e7544070 refactor(sessds): Introduce macros for the timers 2023-12-01 06:11:08 +01:00
ieQu1
0d245acdc1
Merge pull request #12066 from keynslug/fix/sessds-tc-flaky 
test(sessds): wait client disconnect propagates to broker
 2023-12-01 00:13:38 +01:00
Thales Macedo Garitezi
113a4ad4b1
Merge pull request #12057 from thalesmg/ds-session-gc-m-20231128 
feat(ds): add session gc process
 2023-11-30 17:51:52 -03:00
Thales Macedo Garitezi
0388e1c1c4 fix(kafka_producer): add resource_opts to connector schema, and check for client connectivity 
Fixes https://emqx.atlassian.net/browse/EMQX-11494
 2023-11-30 17:51:32 -03:00
Ivan Dyachkov
9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Thales Macedo Garitezi
684d637fed test(bridge_api): workaround strange config syncing problem 
For some unknown reason, this test has difficulties in syncing the config correctly
between the nodes, while the equivalent in bridge_v2_api_SUITE doesn't.
 2023-11-30 19:51:35 +01:00
Ivan Dyachkov
ec10c51073 Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53 2023-11-30 19:51:12 +01:00
Zaiming (Stone) Shi
dcb1c0680b
Merge pull request #12053 from zmstone/1129-prepare-for-otp26 
1129 prepare for OTP 26
 2023-11-30 19:38:40 +01:00
Andrew Mayorov
ec471c0557
test(sessds): wait client disconnect propagates to broker 2023-11-30 20:14:57 +03:00
Serge Tupchii
f54bda7063 fix(emqx_mgmt_api_data_backup): add count and hasnext meta paging params 2023-11-30 19:10:12 +02:00
Zaiming (Stone) Shi
81e75cf068 test: fix bulk-kick test case flakyness 2023-11-30 17:30:25 +01:00
Zaiming (Stone) Shi
5427ebc5f1 fix: log error when failed to install plugin 
the error return is a map with details.
prior to this change only the reason is returned in the api response,
now the error map is logged at error level to help troubleshooting
 2023-11-30 16:49:30 +01:00
Ivan Dyachkov
6dd92c382f chore: 5.3.2 release 2023-11-30 16:18:44 +01:00
Thales Macedo Garitezi
cf6cb3e4ad chore: set low importance to config 2023-11-30 11:55:57 -03:00
Thales Macedo Garitezi
1ab009f081 refactor: rename supervisor 2023-11-30 11:55:48 -03:00
Zaiming (Stone) Shi
eed253af82 test: implement a new node restart helper func 2023-11-30 15:11:38 +01:00
Thales Macedo Garitezi
880f5e8f89 feat(ds): add session gc process 
Fixes https://emqx.atlassian.net/browse/EMQX-9744
 2023-11-30 09:38:59 -03:00
Zaiming (Stone) Shi
cf72c04fdd test: fix peer node stop and plugin SUITE typo 2023-11-30 09:53:36 +01:00
Zaiming (Stone) Shi
f2db4cc7fc chore: upgrade to gen_rpc 3.2.2 2023-11-30 09:42:44 +01:00
Zaiming (Stone) Shi
640b0df319 test: do not add -master erl flag for peer nodes 2023-11-30 08:49:57 +01:00
zhongwencool
773bd8ecb2
Merge pull request #12047 from thalesmg/test-more-flaky-tests-r53-20231128 
test(flaky): more adjustments
 2023-11-30 09:33:55 +08:00
Zaiming (Stone) Shi
4ecfe2be30 test: use peer module for slave and ct_slave 2023-11-29 21:45:53 +01:00
Thales Macedo Garitezi
62b763a8f8 test(gcp_pubsub_consumer): even more adjustments 2023-11-29 15:12:47 -03:00
Ivan Dyachkov
7f5433f6dd chore: 5.3.2-rc.1 2023-11-29 18:20:15 +01:00
Thales Macedo Garitezi
fdafe2493e
Merge pull request #12041 from thalesmg/ds-estimate-last-alive-m-20231128 
fix(ds_session): take conservative estimate of `last_alive_at` when bumping
 2023-11-29 13:43:59 -03:00
Zaiming (Stone) Shi
6f35f25163 chore: upgrade esockd to 5.9.8 for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi
1a563b4f65 chore: fix 0.0 match for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi
14644988e0 chore: change triple-quotes to single-quotes 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi
e6eb97e104 chore: upgrade dependency brod_gssapi to work with OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi
41194cacc8
Merge pull request #12030 from zhongwencool/redis-connector-v2-540 
Redis connector v2 540
 2023-11-29 16:13:09 +01:00
Kjell Winblad
a750fbbc52
Merge pull request #12051 from HJianBo/fix-http-bridge-bugs 
Fix http bridge bugs
 2023-11-29 15:48:51 +01:00
Zaiming (Stone) Shi
89cdfbca63 fix(emqx_vm): trim new-line in otp version string 2023-11-29 15:32:25 +01:00
Andrew Mayorov
ccef91437d
Merge pull request #12029 from keynslug/ft/EMQX-11049/qos2 
feat(sessds): provide QoS2 message replay support
 2023-11-29 14:10:16 +03:00
Serge Tupchii
009a15c7d0 fix(emqx_mgmt_api_data_backup): validate empty file in upload API 
fixes EMQX-11488
 2023-11-29 12:12:42 +02:00
Andrew Mayorov
d8691f1d64
refactor(sessds): rename marker → committed offset 
For better clarity.
 2023-11-29 13:04:42 +03:00
zhongwencool
0aec2f7605 feat: redis bridge v2 2023-11-29 16:06:46 +08:00
zhongwencool
8ec3857f4d
Merge pull request #12046 from savonarola/1128-fix-redis-54 
[5.4] Fix Redis connector
 2023-11-29 16:06:23 +08:00
William Yang
e90e843ff3
Merge pull request #12040 from qzhuyan/dev/william/quic-0.0.300 
feat(quic): bump to quicer-0.0.303
 2023-11-29 08:48:47 +01:00
JianBo He
d2e5f302a8 chore(http): add description for bridges v1 2023-11-29 13:47:59 +08:00
JianBo He
72bc046063 chore: avoid unnecessary default values being filled when querying via the v1 api. 
see: https://emqx.atlassian.net/browse/EMQX-11482
 2023-11-29 12:01:59 +08:00
JianBo He
d7bf8e97d2 chore: add tests case 2023-11-29 11:18:51 +08:00
Serge Tupchii
b0d4ba183e test: use emqx_cth_suite in emqx_coap_api_SUITE 
`emqx_coap_api_SUITE` was failing with `emqx_authn_schema_not_injected`
(missing authn schema mods in persistent_term) if it was run standalone.
 2023-11-28 20:02:21 +02:00
Serge Tupchii
3530ed66d5 fix(emqx_gateway_coap): take subscribe options from uri_query 
Closes #12031
 2023-11-28 20:02:21 +02:00
Thales Macedo Garitezi
095e7c4ecb test(flaky): more adjustments 2023-11-28 13:41:31 -03:00
Thales Macedo Garitezi
f6f61f4353
Merge pull request #12036 from thalesmg/test-more-flaky-tests-r54-20231127 
test: attempting to stabilize more flaky tests
 2023-11-28 13:40:49 -03:00
Ilya Averyanov
e5508105c7 fix(redis): start and load eredis app 2023-11-28 18:51:53 +03:00
Ilya Averyanov
55218e2df2 fix(redis): start and load eredis app 2023-11-28 18:51:19 +03:00
Zaiming (Stone) Shi
d93fdd8e9b
Merge pull request #12039 from zmstone/1128-sync-release-54 
Sync release-54 to master
 2023-11-28 15:52:49 +01:00
William Yang
64f6220cf5 feat(quic): bump to quicer-0.0.303 and emqtt 1.9.7 2023-11-28 14:43:47 +01:00
ieQu1
8a065ba6d0
Merge pull request #12027 from ieQu1/dev/ds-meta-persist-shard 
fix(ds): Persist shard table
 2023-11-28 14:33:57 +01:00
Thales Macedo Garitezi
f5c4fb5860 fix(ds_session): take conservative estimate of last_alive_at when bumping 
Addresses https://github.com/emqx/emqx/pull/12024#discussion_r1407432154
 2023-11-28 09:58:03 -03:00
Thales Macedo Garitezi
bd40b5c553 test(flaky): more adjustments 2023-11-28 09:48:40 -03:00
Ivan Dyachkov
2e11ab6a16 chore: 5.3.2-alpha.2 2023-11-28 13:46:28 +01:00
ieQu1
05e47254e2 fix(ds): Fixes related to the shards table 2023-11-28 12:59:09 +01:00
Zaiming (Stone) Shi
6c85e62d26 fix(schema): add namespaces 2023-11-28 12:07:42 +01:00
Zaiming (Stone) Shi
1b1cea24f3 Merge remote-tracking branch 'origin/release-54' into 1128-sync-release-54 2023-11-28 11:43:30 +01:00
Andrew Mayorov
34c9c022d0
chore(replayer): add comment describing what "until" means 2023-11-28 11:47:54 +03:00
JianBo He
bd6e9503e6 fix(http): compose the url and path in correctly format 2023-11-28 16:15:46 +08:00
zhongwencool
6e6e68c902
Merge pull request #12037 from zhongwencool/license-watermark-check 
fix: validate error when set license's watermark to 100%
 2023-11-28 16:04:39 +08:00
zhongwencool
29bcdb9a4a fix: validate error when set license's watermark to 100% 2023-11-28 13:16:16 +08:00
JianBo He
891ecc179d chore: fix flaky tests 2023-11-28 10:34:44 +08:00
JianBo He
c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
JianBo He
cdb90ebe6b feat: rename webhook bridge to http bridge 2023-11-28 09:53:46 +08:00
JianBo He
dc99651690 test(bridge): ensure almost test cases passed 2023-11-28 09:53:46 +08:00
JianBo He
8954450c0b chore: fix compile warnings 2023-11-28 09:53:46 +08:00
JianBo He
96af7a74e8 feat: impl the http bridge v2 2023-11-28 09:53:46 +08:00
JianBo He
f9a1e747fd chore(http): break the bridge confs to connector and action parts 2023-11-28 09:53:46 +08:00
SergeTupchiy
3b1ae0f3df
Merge pull request #12017 from SergeTupchiy/EMQX-11319-data-backup-http-api 
EMQX-11319 data backup http api
 2023-11-27 19:23:54 +02:00
Thales Macedo Garitezi
e3fbf6c958 test: attempting to stabilize more flaky tests 2023-11-27 14:12:48 -03:00
Thales Macedo Garitezi
26e59f9508
Merge pull request #12024 from thalesmg/ds-session-expiry-m-20231124 
feat(ds): session expiry
 2023-11-27 14:11:38 -03:00
Stefan Strigler
ac370fed61
Merge pull request #12035 from sstrigler/EMQX-11382-a-new-field-should-be-added-to-the-interface-of-the-connector-to-return-the-actions-associated-with-it 
feat(emqx_connector): add field 'actions' in API response
 2023-11-27 17:39:44 +01:00
Zaiming (Stone) Shi
d933d0b9e0
Merge pull request #12016 from zmstone/1122-load-license-file 
1122 load license file
 2023-11-27 16:43:51 +01:00
Stefan Strigler
70978df5bb fix: use parse_id 2023-11-27 16:32:15 +01:00
Serge Tupchii
89fdbf3e73 fix(emqx_mgmt_cli): remove duplicated word in the log message 2023-11-27 17:30:58 +02:00
Serge Tupchii
64ee29af81 feat(emqx_management): implement data backup API 2023-11-27 17:30:58 +02:00
Andrew Mayorov
86685bdce2
feat(sessds): use integer tags for pubrange types 2023-11-27 18:27:15 +03:00
Andrew Mayorov
bb05281adb
refactor(sessds): add dedicated #ds_pubrange.tracks field 
This slightly simplifies the replayer code.
 2023-11-27 18:24:57 +03:00
Thales Macedo Garitezi
d88deb9ceb feat(ds): add session timer to bump last alive at timestamp 2023-11-27 12:19:52 -03:00
Stefan Strigler
048f4724a9 feat(emqx_connector): add field 'actions' in API response 
Also unify schemas, use emqx_connector_schema for the generic parts.
 2023-11-27 16:09:17 +01:00
Thales Macedo Garitezi
09c4e40511 refactor(ds): rename disconnected_at to last_alive_at, add more assertions 2023-11-27 11:48:44 -03:00
Thales Macedo Garitezi
f2dbddc315 test: attempting to stabilize more flaky tests 2023-11-27 11:36:32 -03:00
Thales Macedo Garitezi
6acdc6e432 feat: add /actions/:id/metrics/, /actions/:id/metrics/reset APIs 
Fixes https://emqx.atlassian.net/browse/EMQX-11381
 2023-11-27 11:33:11 -03:00
Andrew Mayorov
923898eadf
chore(chan): leave a TODO note for PUBREC handler 2023-11-27 09:55:46 +03:00
Andrew Mayorov
41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs 2023-11-27 09:55:46 +03:00
Andrew Mayorov
46475fac66
feat(sessds): provide QoS2 message replay support 2023-11-27 09:55:45 +03:00
Andrew Mayorov
ce59cb71bb
chore: bump emqtt to 1.9.6 2023-11-27 09:55:45 +03:00
zhongwencool
b5a00ec6b2
Merge pull request #12023 from emqx/master 
chore: sync master to release-54
 2023-11-27 09:31:47 +08:00
ieQu1
bd7a84fe3e revert(ds): Don't duplicate the clean start in session_ds 2023-11-26 19:18:59 +01:00
Kjell Winblad
c85004b7ef chore: remove obsolete TODO 2023-11-24 21:22:27 +01:00
Kjell Winblad
c6c1d886f0 fix: make pgsql action schema properly 2023-11-24 21:19:26 +01:00
Thales Macedo Garitezi
839f9dbedb feat(ds): session expiry 
Fixes https://emqx.atlassian.net/browse/EMQX-11048
 2023-11-24 16:31:54 -03:00
Kjell Winblad
66945dcc5c fix: address more comments from @thalesmg 2023-11-24 20:31:29 +01:00
Kjell Winblad
30e248061f
fix: problems found by @thalesmg in code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-24 20:07:37 +01:00
Kjell Winblad
f070d80b1a fix: swagger examples for PostgreSQL, Matrix and Timescale 2023-11-24 18:50:52 +01:00
Kjell Winblad
d03674a505 fix: duplicate key in example 2023-11-24 17:59:56 +01:00
Kjell Winblad
fc7bedb81a fix: remove duplicated entry 2023-11-24 17:37:58 +01:00
Kjell Winblad
e920160805 fix: add enable and description fields to PostgreSQL connector 2023-11-24 17:33:39 +01:00
Kjell Winblad
3dca83c854 fix: all missing descriptions 2023-11-24 17:33:39 +01:00
Kjell Winblad
c5e281b84b fix: emqx_auth_postgresql test suites 2023-11-24 17:33:39 +01:00
Kjell Winblad
6ef9c6fe4a fix: ops 2023-11-24 17:33:39 +01:00
Kjell Winblad
f7296d549f fix: elvis problem 2023-11-24 17:33:39 +01:00
Kjell Winblad
64c015cf6f fix: unify the schema modules in emqx_bridge_pgsql 2023-11-24 17:33:39 +01:00
Kjell Winblad
b8f510d956 fix: add env variables about action info 2023-11-24 17:33:39 +01:00
Kjell Winblad
2e3028a8f8 fix(emqx_postgresql): fix lifecycle test 2023-11-24 17:33:39 +01:00
Kjell Winblad
dbe73c70b1 fix: dialyzer problem 2023-11-24 17:33:38 +01:00
Kjell Winblad
d5b62eead0 feat: split pgsql, matrix and timescale into connector action 
This commit splits the bridges pgsql, matrix and timescale into
connector and action.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
 2023-11-24 17:33:35 +01:00
Stefan Strigler
f8f8cf9f30
Merge pull request #11963 from sstrigler/EMQX-11156-bridge-v-2-mongo-db-support 
EMQX 11156 bridge v2 mongo db support
 2023-11-24 16:49:57 +01:00
zhongwencool
8f548f4cbb
Merge pull request #12020 from thalesmg/sync-r53-m-20231124 
chore: sync `release-53` to `master`
 2023-11-24 22:54:53 +08:00
Stefan Strigler
c1ef773e77 fix: check for sane state after regular shutdown 2023-11-24 15:09:35 +01:00
Stefan Strigler
4e077c951b feat(emqx_bridge_mongodb): port mongodb to shared connector and actions 2023-11-24 15:09:35 +01:00
Thales Macedo Garitezi
261fe8a831 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 10:10:09 -03:00
ieQu1
2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start 
Discard sessions when client connects with clean start = true
 2023-11-24 13:35:39 +01:00
Thales Macedo Garitezi
e95ec5b150 test: fix another flaky test 2023-11-24 09:24:21 -03:00
Thales Macedo Garitezi
f8fd95c683 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 09:22:24 -03:00
Thales Macedo Garitezi
6a3c438194
Merge pull request #12009 from thalesmg/fix-kafka-get-bridges-r53-20231123 
fix(bridge_api): don't mangle configs, use correct type as argument
 2023-11-24 08:56:24 -03:00
Stefan Strigler
9ebbc9bbea refactor(emqx_bridge): use more simplistic function to validate connector 2023-11-24 10:07:11 +01:00
Stefan Strigler
d2901afd1b fix(emqx_bridge_kafka): match example in api schema 2023-11-24 10:07:11 +01:00
ieQu1
e616e0746a feat(ds): Implement a function for dumping persistent session state 2023-11-24 04:39:46 +01:00
ieQu1
8dfcb69e52 feat(ds): Discard session when client connects with CleanStart=1 2023-11-24 04:17:28 +01:00
ieQu1
e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0 
Support QoS0 messages in persistent_session_ds
 2023-11-23 23:43:18 +01:00
ieQu1
449bafc27e fix(ds): LTS trie handles empty topic levels 2023-11-23 22:16:38 +01:00
ieQu1
a158f25a40 fix(ds): Fix return type of emqx_persistent_session_ds:publish 2023-11-23 22:16:38 +01:00
ieQu1
c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1
1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message 
feat(ds): Don't store #message record in the DB
 2023-11-23 18:56:45 +01:00
Stefan Strigler
1c11a02a76
Merge pull request #12010 from sstrigler/EMQX-11415-e-5-3-2-api-should-return-404-when-the-built-in-database-authentication-doesnt-exist 
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
 2023-11-23 18:08:48 +01:00
ieQu1
c69b82455e feat(ds): Don't store #message record in the DB 2023-11-23 17:56:35 +01:00
Zaiming (Stone) Shi
14077ec43b feat(license): allow setting 'default' license key 2023-11-23 17:52:17 +01:00
Zaiming (Stone) Shi
caaf8113fc feat(license): support loading license from file 2023-11-23 17:52:17 +01:00
Thales Macedo Garitezi
4597e1c5b9
Merge pull request #12011 from thalesmg/fix-kafka-get-bridges-m-20231123 
fix(bridge_api): don't mangle configs, use correct type as argument
 2023-11-23 13:15:39 -03:00
ieQu1
1ced8786fd feat(ds): Make session poll interval configurable 2023-11-23 17:10:03 +01:00
ieQu1
82e74d0201 feat(ds): Add a flag that forces all sessions to become durable 2023-11-23 17:10:03 +01:00
Stefan Strigler
8ba116d378 fix(emqx_auth): check authenticator exists in /authenticator/:id/users 2023-11-23 16:15:03 +01:00
Ilya Averyanov
ce77a12cd3
Merge pull request #12012 from savonarola/1123-fix-callback-types 
fix(ds): impose more flexible types for ds implementations
 2023-11-23 17:03:56 +02:00
Thales Macedo Garitezi
fdfa3213cc fix(bridge_api): don't mangle configs, use correct type as argument 
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
 2023-11-23 11:41:26 -03:00
Zaiming (Stone) Shi
7375bc5f9b fix(license): allow CRLF in license keys 2023-11-23 15:39:18 +01:00
Zaiming (Stone) Shi
48b69bd60c chore: delete a TODO from perfectionism 2023-11-23 15:39:18 +01:00
Ilya Averyanov
29d42506d5 fix(ds): impose more flexible types for ds implementations 2023-11-23 17:16:11 +03:00
Thales Macedo Garitezi
8e8d6d2192
Merge pull request #12006 from thalesmg/test-debug-flaky-test-r53-20231122 
test(gcp_pubsub_consumer): fix flaky test
 2023-11-23 10:57:22 -03:00
Thales Macedo Garitezi
c89ec0b1f7 fix(bridge_api): don't mangle configs, use correct type as argument 
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
 2023-11-23 10:25:36 -03:00
Thales Macedo Garitezi
f3693e5dbc fix(gcp_pubsub_producer): add missing references to api specs 2023-11-23 09:02:52 -03:00
Zaiming (Stone) Shi
209077593d
Merge pull request #11970 from zmstone/1108-add-v2-schema-dump 
1108 add v2 schema dump
 2023-11-22 23:42:54 +01:00
Thales Macedo Garitezi
db83457d13 test: fix flaky test 
The cause was that the call `sys:terminate/2` was timing out...

`exit/2` doens't always work:

```
 2023-11-22 19:14:40.974
killed async workers

Error: -22T19:14:40.974563+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15908.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 28, reductions: 1822; neighbours:
Error: -22T19:14:40.998051+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15908.7>.
2023-11-22T19:15:41.088752+00:00 [critical] Run stage failed: error:{badmatch,{timeout,#{expected_remaining => 1,mailbox => {messages,[]},msgs_so_far => []}}}, Stacktrace: [{emqx_bridge_gcp_pubsub_consumer_SUITE,'-t_async_worker_death_mid_pull/1-fun-17-',3,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1576}]},{emqx_bridge_gcp_pubsub_consumer_SUITE,t_async_worker_death_mid_pull,1,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1505}]}], Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-11-22_19.14.27/snabbkaffe/1700680540975786370.log", mfa: undefined
Error: -22T19:15:46.095702+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15934.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 1471; neighbours:
Error: -22T19:15:46.095192+00:00 [error] Supervisor: {local,ehttpc_sup}. Context: shutdown_error. Reason: killed. Offender: id={ehttpc_pool_sup,<<98,114,105,100,103,101,58,103,99,112,95,112,117,98,115,117,98,95,99,111,110,115,117,109,101,114,58,116,95,97,115,121,110,99,95,119,111,114,107,101,114,95,100,101,97,116,104,95,109,105,100,95,112,117,108,108,45,53,55,54,52,54,48,55,53,50,51,48,51,52,50,50,55,53,49>>},pid=<0.15903.7>.
Error: -22T19:15:46.095470+00:00 [error] Supervisor: {<0.15906.7>,ehttpc_worker_sup}. Context: shutdown_error. Reason: killed. Offender: id={worker,1},pid=<0.15924.7>.
Error: -22T19:15:46.096762+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15934.7>.
Warning: 2T19:15:46.098278+00:00 [warning] msg: remove_local_resource_failed, mfa: emqx_resource:remove_local/1(362), error: {error,timeout}, resource_id: <<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>
Error: -22T19:15:46.149090+00:00 [error] Generic server <0.15904.7> terminating. Reason: killed. Last message: {'EXIT',<0.15903.7>,killed}. State: {state,<<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>,1,random}.
Error: -22T19:15:46.149525+00:00 [error] crasher: initial call: ehttpc_pool:init/1, pid: <0.15904.7>, registered_name: [], exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,909}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.15903.7>,ehttpc_sup,<0.15731.7>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 3428; neighbours:
```
 2023-11-22 17:41:32 -03:00
Zaiming (Stone) Shi
d9f964a44f test: fix test cases after schema type namespace change 2023-11-22 16:58:05 +01:00
Thales Macedo Garitezi
fc849f0c05 ci(test): add info to help diagnose flaky test 2023-11-22 12:36:10 -03:00
Thales Macedo Garitezi
6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Zaiming (Stone) Shi
e065217e9a
Merge pull request #11997 from zmstone/1121-do-not-allow-special-char-as-prefix-for-resource-name 
1121 do not allow special char as prefix for resource name
 2023-11-22 15:46:11 +01:00
Thales Macedo Garitezi
274a254624
Merge pull request #12002 from thalesmg/multiple-action-infos-per-app-m-20231122 
feat(actions): allow multiple action info modules per application
 2023-11-22 10:42:56 -03:00
Thales Macedo Garitezi
bb549cdf8b feat(actions): allow multiple action info modules per application 2023-11-22 09:33:01 -03:00
Thales Macedo Garitezi
38d3a1d7d0 feat(actions): allow multiple action info modules per application 2023-11-22 09:30:37 -03:00
Zaiming (Stone) Shi
db33bc616a feat(schema): Add v2 scheam JSON dump 2023-11-22 13:12:35 +01:00
Zaiming (Stone) Shi
1b2c052646 docs: add type namespaces 2023-11-22 13:12:35 +01:00
Serge Tupchii
d9f95cdc56 feat: integrate OpenTelemetry log handler 2023-11-22 13:59:30 +02:00
Ilya Averyanov
a1b9a14fa1 feat(ds): allow fdb implementation for durable storage 2023-11-22 14:49:38 +03:00
Zaiming (Stone) Shi
3261a12140 fix(emqx_resource): do not allow leading _ or - as resource name 2023-11-22 10:58:54 +01:00
ieQu1
ee191803ea
Merge pull request #11990 from ieQu1/dev/ds-shards 
Implement a prototype of durable message sharding
 2023-11-22 00:57:54 +01:00
Zaiming (Stone) Shi
869e73d637
Merge pull request #11966 from thalesmg/fix-kafka-parameters-r53-20231117 
fix(kafka): don't return `parameters` from `/bridges` API
 2023-11-22 00:36:39 +01:00
Zaiming (Stone) Shi
4fb2d3a503
Merge pull request #11996 from zmstone/1121-prepare-for-e5.3.2-alpha.1 
chore: bump release version to e5.3.2-alpha.1
 2023-11-21 22:01:05 +01:00
Thales Macedo Garitezi
39791511fc chore: remove obsolete workaround 2023-11-21 17:38:13 -03:00
Thales Macedo Garitezi
01895fd660
Merge pull request #11995 from thalesmg/fix-action-api-bad-name-r53-20231121 
fix(actions_api): don't crash on validation errors
 2023-11-21 17:27:38 -03:00
Zaiming (Stone) Shi
5ebd954b16 chore: bump release version to e5.3.2-alpha.1 2023-11-21 21:19:05 +01:00
Zaiming (Stone) Shi
8fbdcab118
Merge pull request #11987 from zmstone/1120-do-not-crash-on-einval-after-check-cache 
fix(emqx_connection): handle socket activation error return
 2023-11-21 20:31:56 +01:00
Thales Macedo Garitezi
3a8c332805 fix(actions_api): don't crash on validation errors 
Fixes https://emqx.atlassian.net/browse/EMQX-11394
 2023-11-21 16:27:54 -03:00
ieQu1
3165b4f645 fix(ds): Abort application startup when rocksdb is not avialable 2023-11-21 20:27:07 +01:00
Thales Macedo Garitezi
11ec1a30a0 test(flaky): fix flaky pulsar test 2023-11-21 16:00:19 -03:00
Thales Macedo Garitezi
b3dffa4390 fix(kafka): don't return parameters from /bridges API 
Fixes https://emqx.atlassian.net/browse/EMQX-11412
 2023-11-21 16:00:15 -03:00
ieQu1
3d823beb11 fix(ds): Apply review remarks 2023-11-21 19:37:01 +01:00
ieQu1
4d47490734 chore(ds): Rebase configuration 2023-11-21 19:37:01 +01:00
ieQu1
f5c71e8068 refactor(ds): Add a wrapper to the store batch API 2023-11-21 19:37:01 +01:00
ieQu1
2a1f7d946a feat(ds): Shard messages by publisher client ID 2023-11-21 19:37:01 +01:00
ieQu1
62542e5844 feat(ds): Metadata storage for the replication layer 2023-11-21 19:37:01 +01:00
Kjell Winblad
b0d670aaa9
Merge pull request #11992 from kjellwinblad/kjell/backport/action_upgrade_downgrade_hooks 
fix(action): upgrade and downgrade strategy
 2023-11-21 19:14:37 +01:00
Thales Macedo Garitezi
9e1796ec4f feat(gcp_pubsub_producer): migrate GCP PubSub producer to actions 
Fixes https://emqx.atlassian.net/browse/EMQX-11157
 2023-11-21 14:22:42 -03:00
Zaiming (Stone) Shi
fa91bacdfb
Merge pull request #11975 from zmstone/1119-fix-socket-close-race-condition 
fix(emqx_channel): do not log stale sock_close event as error
 2023-11-21 16:47:03 +01:00
Kjell Winblad
fa7151f255 fix: port emqx_utils_maps:rename function from master 
The emqx_utils_maps:rename function is needed by action
upgrade/downgrade hoos.
 2023-11-21 15:18:22 +01:00
Thales Macedo Garitezi
6c9a8461f7
Merge pull request #11989 from thalesmg/confluent-action-m-20231120 
feat: add confluent connector/action
 2023-11-21 11:01:40 -03:00
Kjell Winblad
cd72dc11dd fix: missing emqx_action_info module mapping 2023-11-21 14:22:29 +01:00
Thales Macedo Garitezi
068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120 
sync `release-53` to `master`
 2023-11-21 09:02:51 -03:00
Thales Macedo Garitezi
759468efcf
Merge pull request #11986 from thalesmg/split-resource-opts-actions-r53-20231120 
refactor: split `resource_opts` fields between connector and actions
 2023-11-21 09:01:29 -03:00
Thales Macedo Garitezi
75ee3ced68 feat: add confluent connector/action 
Fixes https://emqx.atlassian.net/browse/EMQX-11044

See also: https://emqx.atlassian.net/wiki/spaces/EMQX/pages/712179934/Confluent+Integration
 2023-11-21 08:59:42 -03:00
Kjell Winblad
6030bf6fa5 fix(action): upgrade and downgrade strategy 
This commit adds upgrade and downgrade hooks that are called when
upgrading from a bridge V1 to connector and action or the other way
around. The automatic translation is used if the callback is not
defined.

NOTE: Backported from master
 2023-11-21 11:18:21 +01:00
Ilya Averyanov
c6fd1e4c75
Merge pull request #11971 from savonarola/1116-no-auth-rebalance-status 
feat(rebalance): improve rebalance usability
 2023-11-21 12:10:18 +02:00
Andrew Mayorov
cd48c283db
Merge pull request #11988 from keynslug/ft/EMQX-11222/basic-config 
feat(schema): introduce separate root for new session persistence
 2023-11-21 15:02:21 +07:00
JianBo He
9716b051d9 fix(ocpp): fix bad webscoket schema for HTTP API 2023-11-21 11:24:23 +08:00
Thales Macedo Garitezi
6f8630304d
Merge pull request #11980 from kjellwinblad/kjell/fix_fixup_callbacks/EMQX-11428 
fix: bridge to action upgrade fixup hook should run after upgrade
 2023-11-20 16:19:43 -03:00
Kjell Winblad
7fb5ade832 fix(kafka_producer): hocon renames kafka field 2023-11-20 19:20:13 +01:00
Kjell Winblad
d214ae8772
fix: problems found by @thalesmg in code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-20 18:59:57 +01:00
Andrew Mayorov
110a5a4896
feat(schema): introduce separate root for new session persistence 
With some knobs to choose a storage backend. Support only builtin
RocksDB-based backend with minimal configuration for now.
 2023-11-20 23:49:14 +07:00
Zaiming (Stone) Shi
b02711af79 refactor(emqx_ws_connection): rename cache to buffer for limiter 2023-11-20 17:47:20 +01:00
Andrew Mayorov
8e107ffe45
Merge pull request #11958 from keynslug/ft/EMQX-9745/preserve-acks 
feat(sessds): preserve acks / replays in session state
 2023-11-20 23:40:22 +07:00
Kjell Winblad
edbfe090c9 feat: add custom callback for creating connector from bridge V1 config 2023-11-20 17:23:51 +01:00
Zaiming (Stone) Shi
ec19247271 refactor: rename limiter buffer related messages and var names 2023-11-20 16:55:26 +01:00
Andrew Mayorov
9684e79ee0
fix(sessds): ensure dup flag is on for replayed messages 2023-11-20 22:54:40 +07:00
Zaiming (Stone) Shi
8ec3b1db5d fix(emqx_connection): handle socket activation error return 2023-11-20 16:51:39 +01:00
Kjell Winblad
3aa8044475 fix(action): upgrade and downgrade strategy 
This commit fixes the upgrade and downgrade strategy when upgrading
from a bridge V1 to connector and action or the other way around so that
the custom callbacks get the complete unchanged input instead of the
result of the automatic translation. The automatic translation is used
if the callback is not defined.
 2023-11-20 16:40:27 +01:00
Thales Macedo Garitezi
f40f6bc5dd refactor: split resource_opts fields between connector and actions 2023-11-20 12:37:36 -03:00
Stefan Strigler
1f1d9e58c6 fix(emqx_connector): don't crash in API on delete with active channels 2023-11-20 16:23:46 +01:00
Thales Macedo Garitezi
53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Andrew Mayorov
ef7cfd0202
feat(sessds): add field to ds_pubrange for forward-compat 2023-11-20 19:56:55 +07:00
Thales Macedo Garitezi
b42d4e5ab0
Merge pull request #11950 from thalesmg/fix-bridge-v1-examples-r53-20231114 
fix(bridges_v1): avoid merging action examples for non-v1 bridges
 2023-11-20 09:12:23 -03:00
Thales Macedo Garitezi
926078b82c
Merge pull request #11960 from thalesmg/fix-kafka-producer-channel-status-r53-20231116 
fix(kafka_producer): make status `connecting` while the client fails to connect
 2023-11-20 09:12:05 -03:00
Kjell Winblad
79a764f117 fix: bridge to action upgrade fix up hook should run after upgrade 
This commit changes how the `emqx_action_info` callback
`bridge_v1_to_action_fixup/1` works. It is now called after the
automatic upgrade instead of before. Since the full Bridge V1 config
might be needed to do the fixup, it is provided in a special field
`<<"__bridge_v1_conf__">>`. The `<<"__bridge_v1_conf__">>` field is
removed after the callback is called and can thus be ignored if it is
not needed.
 2023-11-20 12:06:21 +01:00
Stefan Strigler
84ff7b0b38 feat(emqx_bridge): action_info with dynamic lookup 
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
 2023-11-20 10:22:37 +01:00
Andrew Mayorov
d7974d835f
fix(sharesub): anticipate messages w/o redispatch header 
For instance, `emqx_session_mem` will push almost all of the messages
that are still in the state to `emqx_shared_sub:redispatch/1`. After
this commit, the session will no longer crash during channel terminate.
 2023-11-20 15:53:24 +07:00
Andrew Mayorov
7081f1951f
refactor(sessds): use ds_pubrange record as is 
Instead of converting it into almost similar runtime representation.
 2023-11-20 15:40:43 +07:00
Andrew Mayorov
a5ff4144fe
test(sessds): add complex testcase for session replay 2023-11-20 15:40:43 +07:00
Andrew Mayorov
1246d714c5
feat(sessds): preserve acks / ranges in mnesia for replays 2023-11-20 13:33:48 +07:00
Andrew Mayorov
5b40304d1f
chore(sessds): simplify subscriptions handling 
There's currently no point in storing parsed topic filters in the
subscriptions table.
 2023-11-20 13:33:48 +07:00
Andrew Mayorov
648b6ac63e
chore(sessds): rename iterators -> subscriptions 
Also try to make clearer the difference between 2 flavors of topic
filter representation in use.
 2023-11-20 13:33:48 +07:00
Zaiming (Stone) Shi
e73bf716ae fix(emqx_channel): do not log stale sock_close event as error 
In some cases, EMQX may decide to close socket and mark connection
at 'disconnected' state, for example, when DISCONNECTE packet is
received, or, when failed to write data to socket.
However, by the time EMQX decided to close the socket, the socket
might have already been closed by peer, and the `tcp_closed` envet
is already delivered to the process mailbox -- causing EMQX to
handle sock_close event at 'disconnected' state.
 2023-11-19 22:27:25 +01:00
Ilya Averyanov
e93e9ed108 feat(rebalance): improve rebalance usability 
* make availability API endpoint public
* allow connections during wait_health_check interval
* make availability status calculation more consistent and lightweight
* refactor test to get rid of some mocks and to use cth
 2023-11-18 21:29:16 +03:00
Thales Macedo Garitezi
eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Kjell Winblad
86c126ffcd feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade 
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
 2023-11-17 16:10:08 -03:00
Kjell Winblad
9feba802e9 chore: add convenience function for creating action schemas 2023-11-17 16:10:08 -03:00
Ilya Averyanov
58437cd35a fix(mongodb): fix deadlock while stopping mongodb resource 2023-11-17 12:34:36 +03:00
zhongwencool
b2e168d508 feat: add prometheus.push_gateway.enable config 2023-11-17 09:48:39 +08:00
zhongwencool
c4eb9f86e7 chore: add refactor prometheus refactor changelog 2023-11-17 08:23:51 +08:00
zhongwencool
b343653769 chore: support list(tuple()) for schema_with_examples/3 2023-11-17 08:23:51 +08:00
zhongwencool
0c2610fa61 feat: support prometheus.enable_basic_auth 2023-11-17 08:23:51 +08:00
zhongwencool
c376a5db29 refactor: prometheus config 2023-11-17 08:23:51 +08:00
Thales Macedo Garitezi
b92821188b fix(kafka_producer): make status connecting while the client fails to connect 
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
 2023-11-16 14:50:23 -03:00
Ilya Averyanov
4c54efd8b1 fix(auth): fix deadlock while stopping mongodb resource 2023-11-16 19:29:08 +03:00
Thales Macedo Garitezi
36b5d58957 test: reorganize test suite a bit 2023-11-16 13:16:02 -03:00
Thales Macedo Garitezi
90571b7d8e test: fix noise about undefined unofficial callbacks 2023-11-16 13:16:02 -03:00
Andrew Mayorov
656d575e2d
ci(router): fix flaky testcase 2023-11-16 17:53:20 +07:00
Andrew Mayorov
d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default 
feat(router): switch to v2 routing store by default
 2023-11-16 15:56:26 +07:00
Andrew Mayorov
893e90b372
fix(kafka): use safe publish in consumer 
Routing with v2 schema is actually more strict with respect to input to
`emqx_router` module routines. This causes Kafka consumer bridge to
crash when it tries to publish a message to a topic that looks like a
topic filter.
 2023-11-15 22:37:00 +07:00
Andrew Mayorov
98f947f4f3
ci(router): fix flaky testcase 2023-11-15 22:28:52 +07:00
Andrew Mayorov
8919b08207
fix(utils): rename emqx_utils_stream:take/2 to consume/2 
Which is more neutral and harder to confuse with a destructive `take` in
collections.
 2023-11-15 17:20:40 +07:00
Ivan Dyachkov
65ba381fd8 Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53 2023-11-15 09:25:03 +01:00
Thales Macedo Garitezi
f5545bbdf3 fix(schema_types): add missing secret() type 2023-11-14 16:08:13 -03:00
Thales Macedo Garitezi
5275a01727 fix(syskeeper): refactor action to new action info scheme 2023-11-14 19:49:28 +01:00
Thales Macedo Garitezi
d2cf29fae4 fix(schema): use hoconsc:union/1 2023-11-14 19:49:08 +01:00
Thales Macedo Garitezi
423fb32171 test: fix noise about undefined unofficial callbacks 2023-11-14 19:48:39 +01:00
Ivan Dyachkov
7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Thales Macedo Garitezi
2f1d88d414 fix(bridges_v1): avoid merging action examples for non-v1 bridges 
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
 2023-11-14 13:56:50 -03:00
Zaiming (Stone) Shi
88637f81d1
Merge pull request #11936 from zmstone/1112-readable-types 
refactor(schema): keep type converters close
 2023-11-14 17:56:07 +01:00
ieQu1
124c0e2dba fix(ds): Respect receive_maximum from the connection info 2023-11-14 16:06:19 +01:00
Ivan Dyachkov
bbf00c2f71
Merge pull request #11946 from id/1114-sync-r53 
sync release-53 to master
 2023-11-14 13:38:24 +01:00
Thales Macedo Garitezi
01a2a3b1c0
Merge pull request #11930 from thalesmg/ds-session-discard-m-20231110 
feat(ds): implement session discard
 2023-11-14 09:30:34 -03:00
Thales Macedo Garitezi
d6e9bbb95c fix(connector): validate connector name before converting ssl certs 
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
 2023-11-14 09:29:59 -03:00
Andrew Mayorov
6812ee9d0f
fix(mgmt): hide route selection behind router interface 
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
 2023-11-14 18:36:25 +07:00
Ivan Dyachkov
28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
Ivan Dyachkov
1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Andrew Mayorov
d1c3b1c659
fix(bridges): handle secrets more consistently 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-14 16:05:54 +07:00
Andrew Mayorov
d4dd4508c9
feat(greptimedb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov
5d620465e2
feat(influx): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov
7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov
7817502b8b
feat(rocketmq): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov
2449d54b1f
feat(pulsar): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov
e2b7b33d14
feat(kinesis): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov
93eaf0caee
feat(dynamo): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov
7250059c60
feat(sqlserver): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov
06861e377f
feat(ldap): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov
fda395014c
test(kafka): verify file-based secrets work 2023-11-14 16:05:25 +07:00
Andrew Mayorov
aa458b65d6
test(kafka): simplify consumer testsuite matrix setup 2023-11-14 16:05:25 +07:00
Andrew Mayorov
18cd98def6
chore(test): fix formatting quirks 2023-11-14 16:05:25 +07:00
Andrew Mayorov
c300eb41a7
feat(kafka): accept wrapped secrets as passwords 2023-11-14 16:05:25 +07:00
Andrew Mayorov
9c5856029f
feat(cassa): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov
8b4ac8eb4f
feat(oracle): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov
34aeeab041
feat(rabbitmq): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov
a69a78d024
feat(clickhouse): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov
fc340a276e
feat(mongo): accept wrapped secrets as passwords 
Also test authorization with mongo in bridge / auth test suites.
 2023-11-14 16:05:24 +07:00
Andrew Mayorov
f827df2821
test(mysql): slightly simplify bridge testsuite 2023-11-14 16:05:23 +07:00
Andrew Mayorov
36e57a479d
feat(mysql): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov
ad72ab7a8c
feat(redis): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov
4385b2f020
feat(pgsql): accept wrapped secrets as passwords 
That are coming from `emqx_schema_secret`. Also adapt pgsql-related
connectors.
 2023-11-14 16:05:23 +07:00
Andrew Mayorov
2b36b48f98
feat(connector): use emqx_schema_secret as password field schema 2023-11-14 16:05:23 +07:00
Ivan Dyachkov
a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad
d682e6e23c test: fix test mock by calling exported function 2023-11-14 09:20:46 +01:00
Kjell Winblad
3bea3496af test: fix test case mock that broke due to removal 2023-11-14 09:20:46 +01:00
Kjell Winblad
093c8b0c6e docs: add missing copyright headers 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca3e5eab11 refactor: remove emqx_bridge_v2_enterprise.erl 2023-11-14 09:20:46 +01:00
Kjell Winblad
e93b71d8d5 fix: problems found by @thalesmg in code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-14 09:20:46 +01:00
Kjell Winblad
49fdfef8c3 fix: azure event hub names are defined in one place 2023-11-14 09:20:46 +01:00
Kjell Winblad
5e8e407017 refactor: action schema retrival after PR feedback 2023-11-14 09:20:46 +01:00
Kjell Winblad
ab078647a5 chore: fix formatting problem 2023-11-14 09:20:46 +01:00
Kjell Winblad
bdab421885 fix(bridge_v2): name and copyright header 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca2cdbc08d fix: bapi static check error 2023-11-14 09:20:46 +01:00
Kjell Winblad
7839f0cbc8 chore(bridge_v2): make dialyzer and bpapi happy 2023-11-14 09:20:46 +01:00
Kjell Winblad
d26a1b9afb fix(bridge_v1): no hard coded downgrade and upgrade type functions 2023-11-14 09:20:46 +01:00
Kjell Winblad
a6aa81b548 fix(emqx_bridge_v2): dialyzer found bad format parameter error 2023-11-14 09:20:46 +01:00
Kjell Winblad
3c778121a5 fix: bridge V1 type lookup issue 2023-11-14 09:20:46 +01:00
Kjell Winblad
77aaff137a test(emqx_bridge_v2_SUITE): fix incorrect assumption about return type 
This commit fixes an incorrect assumption about the return type of
emqx_bridge_v2:lookup/2 in emqx_bridge_v2_SUITE:t_load_config_success/1.
 2023-11-14 09:20:46 +01:00
Kjell Winblad
14e305e2a6 fix(emqx_bridge_v2): xref warning 2023-11-14 09:20:46 +01:00
Kjell Winblad
9eaee8f333 refactor(emqx_bridge_v2): make independent of Kafka 
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
 2023-11-14 09:20:46 +01:00
Kjell Winblad
cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Kjell Winblad
99031f0dae refactor(bridge_v2): prefix compatibility functions with bridge_v1 2023-11-14 09:20:46 +01:00
Kjell Winblad
729e7df0d5 docs(bridge_v2): add specs for CRUD functions 2023-11-14 09:20:46 +01:00
Ivan Dyachkov
a9cf8fe78c chore: 5.3.1 2023-11-14 08:44:41 +01:00
Zaiming (Stone) Shi
ab1f3b6b2f fix(emqx_conf_schema_types): address review comments 2023-11-13 21:55:03 +01:00
Stefan Strigler
fc2891d6de
Merge pull request #11937 from sstrigler/EMQX-11318-crash-in-get-bridges-v-2-if-a-broken-bridge-is-configured 
Emqx 11318 crash in get bridges v 2 if a broken bridge is configured
 2023-11-13 17:28:56 +01:00
Zaiming (Stone) Shi
4c5d64abc2 refactor(schema): keep type converters close 2023-11-13 15:26:30 +01:00
Stefan Strigler
17b18849de test(emqx_bridge): test broken config 2023-11-13 14:52:06 +01:00
Zaiming (Stone) Shi
9b9908780c
Merge pull request #11926 from kjellwinblad/kjell/fix_bridge_v2_to_action_cluster_hocon/EMQX-11353 
fix: reflect bridge V1 upgreade in cluster.hocon
 2023-11-13 14:18:57 +01:00
Stefan Strigler
0b3645057d fix(emqx_bridge): don't try to handle what we don't understand 2023-11-13 14:08:54 +01:00
Stefan Strigler
0d33284672 fix(emqx_bridge): don't crash uninstallling broken action 2023-11-13 14:08:54 +01:00
Stefan Strigler
4fe1d332b3 fix(emqx_bridge): don't crash if there's no status field 2023-11-13 14:08:54 +01:00
Thales Macedo Garitezi
45dad2ed3a feat(ds): implement session discard 
Fixes https://emqx.atlassian.net/browse/EMQX-9739

Fixes some issues to ensure the session is discarded when the client connects with
`clean_start = true`, and added some cleanup to subscriptions/routes/iterators/streams.

> There is an API that session garbage collector can use to perform cleaning

We already have `emqx_session:destroy/1`, which could serve as an API for a periodic
session GC to use.
 2023-11-13 09:06:22 -03:00
Kjell Winblad
26fc9ed89d
fix: log warning message after suggestion from @zmstone 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-11-13 12:53:09 +01:00
Andrew Mayorov
03843c6071
feat(router): switch to v2 routing store by default 2023-11-13 17:43:30 +07:00
Kjell Winblad
5838568d5d fix: hard coded call to schema module 
The user can configure the configuration loader and schema module so we
should not hard code a call to the default schema module.
 2023-11-13 11:41:40 +01:00
Zaiming (Stone) Shi
518b02fc70
Merge pull request #11931 from zmstone/1110-delete-stale-type-converters 
1110 delete stale type converters
 2023-11-13 10:27:51 +01:00
Zaiming (Stone) Shi
65efa2672e test(schema): delete stale reference to emqx_schema:file() type 2023-11-13 09:28:26 +01:00
Zaiming (Stone) Shi
2fda91ec59 refactor: delete default type converter for dashboard schema json 
prior to this commit, dashboard scheam json type conversion
has a 'fallback' type for all unknown types, which is 'string'

this commit removes this fallback and add all possible types
to the function clauses.

also, some of the old types which are no longer in use
are deleted from both swagger spec converter and dashboard
sechema converter
 2023-11-12 20:08:53 +01:00
JimMoen
952ab08991
fix: make dialyzer happy 2023-11-12 21:44:24 +08:00
JimMoen
20bf9ed26d
fix(gw_gbt32960): mountpoint should not with / suffix 2023-11-12 21:44:23 +08:00
JimMoen
867d5aaea6
chore: rm dbg funcs in SUITE 2023-11-12 21:44:23 +08:00
JimMoen
70fcfdddcb
test(gw_jt808): fix test auth server bootstrap 2023-11-12 21:44:23 +08:00
JimMoen
dc64068e41
test(gw_jt808): frame parser and channel SUITE 2023-11-12 21:44:23 +08:00
JimMoen
4d493292fb
feat(gw_jt808): port jt808 gateway from 4.4 2023-11-12 21:44:18 +08:00
JimMoen
526faa0362
fix(gw_gbt32960): clean frame parser state acc data 2023-11-12 21:43:12 +08:00
JimMoen
d30f0aa4a0
fix(gw_gbt32960): parse state atom key error 2023-11-12 21:43:11 +08:00
JimMoen
a4da1d260a
docs(gw_jt808): data exchange guide cn 2023-11-12 21:43:11 +08:00
JimMoen
17edeead6e
fix: erlang_ls complains gateway hrl file can not be found 2023-11-12 21:43:11 +08:00
JimMoen
a218ff6dd8
fix(gw_gbt32960): undefined type emqx_gbt32960_frame:frame() 2023-11-12 21:43:11 +08:00
JimMoen
aef5b17ed5
fix(gw_lwm2m): lwm2m channel behaviour 2023-11-12 21:43:11 +08:00
JimMoen
caade4a7e5
chore: emacs mode entry comment for non-erl files 2023-11-12 21:43:11 +08:00
JimMoen
25e277bdc6
chore: fix schema desc and README.md typo 2023-11-12 21:43:10 +08:00
JimMoen
523bb17fc8
test: mark proper types gateway connection modules need be fixed 2023-11-12 21:43:10 +08:00
Zaiming (Stone) Shi
b7d2c38213 refactor: delete stale type converters 
Some of the types are refactored to use typerefl alias
or hoconsc:enum, no need to keep the special function
clauses to translate them to swagger spec
 2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi
a32cd20758 refactor(emqx_authn_jwt_schema): add alias for verify_claims 
verifiy_claims is a list after converted, but input is a map
 2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi
021f7e6b49 refactor(schema): comma_separated_list is list(string()) not list(any()) 2023-11-12 11:37:58 +01:00
Zaiming (Stone) Shi
06e440260f refactor(schema): add args to map alias 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi
8be718b22f refactor(emqx_schema): delete unused type bar_separated_list 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi
ff5916e33f docs: fix bad html tag 2023-11-12 10:31:37 +01:00
Xinyu Liu
7bb3ca5519
Merge pull request #11909 from emqx/bugfix-plugins 
fix: emqx failed to start if plugin dir is removed by the user
 2023-11-11 10:51:26 +08:00
Thales Macedo Garitezi
b4e45f0189
Merge pull request #11923 from thalesmg/sync-m-to-r54-20231109 
chore: sync `master` to `release-54`
 2023-11-10 13:04:55 -03:00
Thales Macedo Garitezi
088de9476c fix(ds): use integer for tag values 
Follow up to https://github.com/emqx/emqx/pull/11906#discussion_r1389115973
 2023-11-10 11:26:32 -03:00
ieQu1
5170267fef
Merge pull request #11924 from ieQu1/dev/emqx-ds-behavior 
feat(ds): Introduce emqx_ds behavior
 2023-11-10 15:23:28 +01:00
Kjell Winblad
6ebf0d1bdd fix: reflect bridge V1 upgrade in override configurations 
Fixes:
https://emqx.atlassian.net/browse/EMQX-11353
 2023-11-10 14:52:29 +01:00
Zaiming (Stone) Shi
101990b8fc test: fix schema tests after types change 2023-11-10 14:23:43 +01:00
Zaiming (Stone) Shi
e653c6b4e0 chore: bump app versions 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
855b3c5b29 test: ensure atom exists 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
f1de0aa176 fix(schema): add namespace to authn schemas 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
86110824eb feat: upgrade hocon to 0.40.0 which supports union type display name 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
8a4fba431e refactor(emqx_limiter_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
d603de10e6 refactor(schema): add namespace/0 to all schema modules 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
7b59d46854 fix(schema): add namesapce to sso schema 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
6b3aaf5b24 refactor(emqx_bridge_gcp_pubsub): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
b24b66081a refactor(authn/authz_http_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
ieQu1
2ade6da7a6 fix(ds): Remove redundand type 2023-11-10 12:53:30 +01:00
ieQu1
feef23fc08 feat(ds): Pass store_batch through RPC 2023-11-10 12:16:39 +01:00
ieQu1
8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
William Yang
0fc4bcaf0d
Merge pull request #11912 from qzhuyan/test/william/quic-multistream-flaky 
test(quic): fix flaky
 2023-11-10 11:13:35 +01:00
JianBo He
d1623f491c
Merge pull request #11925 from lafirest/docs/gbt 
chore(gbt32960): add EN documentation of the data exchange guide
 2023-11-10 16:43:49 +08:00
Zaiming (Stone) Shi
3e2ce57cb1 refactor(emqx_bridge_greptimedb): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
46fffa401e refactor(emqx_prometheus_schema): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
ae24b45d1b fix(schema): add type namespce to emqx_dashboard_sso_ldap 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
a2700771c9 refactor(emqx_s3_schema): make type readable 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
b3df8604fc refactor(emqx_conf_schema): use hocon enum type for log levels 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
df96798c53 refactor(http_bridge): use more readable type for pool_type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
40c2ca134d refactor(schema): delete unused type cipher() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
7a0a0bee98 refactor(schema): change file() type to string() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
af3dfc91b1 refactor(schema): change log_lvel to enum type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi
3d20e566d8 docs: add namespace to authn and authz schema 2023-11-10 08:45:20 +01:00
firest
0299e9033a chore(gbt32960): add EN documentation of the data exchange guide 2023-11-10 15:22:28 +08:00
Zaiming (Stone) Shi
9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109 
fix(actions_api): change bridge v2 occurrences to action
 2023-11-10 07:35:35 +01:00
firest
447c933ff8 fix(syskeeper): homogenize the Syskeeper bridge and update license data 2023-11-10 10:42:53 +08:00
ieQu1
38d004e06e
Merge pull request #11913 from ieQu1/dev/fix-hooks-comment 
chore(hooks): Fix comment
 2023-11-10 03:15:54 +01:00
firest
548e39e799 refactor(sysk): refactor the syskeeper to v2 style 2023-11-10 09:52:38 +08:00
firest
abc7233a77 test(sysk): add test suite for syskeeper 2023-11-10 09:52:38 +08:00
firest
e93216fa62 chore(sysk): fix spellchecks & update change & more logs 2023-11-10 09:52:32 +08:00
firest
ce83079c6b feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend 2023-11-10 09:51:43 +08:00
JianBo He
6500d21d98
Merge pull request #11885 from HJianBo/ocpp-gw 
feat: Port OCPP gateway from v4
 2023-11-10 09:32:40 +08:00
ieQu1
ced7be61f0
Merge pull request #11917 from ieQu1/dev/fix-packet-id-to-seqno-trans 
fix(ds): Fix packet id -> sequence number translation
 2023-11-09 22:57:16 +01:00
Thales Macedo Garitezi
7a237f6a3a Merge branch 'master' into sync-m-to-r54-20231109 2023-11-09 18:49:52 -03:00
Thales Macedo Garitezi
371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
ieQu1
5aa9d026df fix(ds): Apply review remarks 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-09 21:16:59 +01:00
ieQu1
b812db1e3c fix(ds): Fix packet id -> sequence number translation 2023-11-09 20:12:39 +01:00
Thales Macedo Garitezi
2d539463cd test: fix flaky test 
```
 Testing lib.emqx.emqx_cm_SUITE: *** FAILED test case 24 of 31 ***
%%% emqx_cm_SUITE ==> t_stepdown_session_takeover_begin_timeout: FAILED
%%% emqx_cm_SUITE ==>
Failure/Error: ?assertEqual([<0.6635.8>,<0.6520.8>], lists : sort ( emqx_cm : lookup_channels ( ClientId ) ))
  expected: [<0.6635.8>,<0.6520.8>]
       got: [<0.6520.8>,<0.6635.8>]
      line: 309
```
 2023-11-09 15:32:52 -03:00
Thales Macedo Garitezi
ee87f90b0d
Merge pull request #11906 from thalesmg/ds-avoid-records-m-20231108 
chore(ds): avoid using records in persistence / rpc
 2023-11-09 14:25:27 -03:00
Thales Macedo Garitezi
7977ae95c0 fix(actions_api): change bridge v2 occurrences to action 
Fixes https://emqx.atlassian.net/browse/EMQX-11338
 2023-11-09 14:20:08 -03:00
Thales Macedo Garitezi
eabd09051a feat(actions_api): add /action_types API 2023-11-09 13:54:57 -03:00
ieQu1
42a9f0eb6b
Merge pull request #11914 from ieQu1/dev/speed-up-poll 
fix(ds): Speed up polling for the new messages
 2023-11-09 16:40:06 +01:00
Thales Macedo Garitezi
42536bb5f7 chore(ds): avoid using records in persistence / rpc 
Fixes https://emqx.atlassian.net/browse/EMQX-11279
 2023-11-09 12:00:25 -03:00
ieQu1
cf4a46a78b fix(ds): Speed up polling for the new messages 
Poll immediately if the previous poll returned non-empty result
 2023-11-09 14:22:32 +01:00
ieQu1
3350bda446 chore(hooks): Fix comment 2023-11-09 14:05:13 +01:00
firest
2b97800c8b fix(limiter): simplify the descriptions for fields of the limiter 2023-11-09 18:00:09 +08:00
William Yang
f7a2359329 test(quic): fix flaky 2023-11-09 10:56:16 +01:00
JianBo He
d55f1e0813 chore: fix mix compiling failures 2023-11-09 16:54:41 +08:00
Shawn
e0f4a2014d fix: emqx failed to start if plugin dir is removed by the user 2023-11-09 16:01:20 +08:00
Ivan Dyachkov
0bf5deaed9 chore: merge master into release-54 2023-11-09 08:32:26 +01:00
JianBo He
d6104b37cf test: fix flaky tests 2023-11-09 15:01:12 +08:00
JianBo He
4b18631d49 chore: fix gateway failed test cases 2023-11-09 13:44:20 +08:00
JianBo He
2d958beae9 chore: try to fix spellcheck 2023-11-09 09:51:08 +08:00
Zaiming (Stone) Shi
42fa1289a1
Merge pull request #11905 from thalesmg/tag-e531-alpha5-r53-20231108 
chore: prepare to tag `e5.3.1-alpha.5`
 2023-11-08 23:39:33 +01:00
Zaiming (Stone) Shi
f95058a3e7
Merge pull request #11897 from zmstone/1107-fix-cluster-conf-sync-wait-loop 
1107 fix cluster conf sync wait loop
 2023-11-08 23:37:09 +01:00
Zaiming (Stone) Shi
1e3500ffd2
fix(emqx_conf_app): ensure log data fields 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-08 21:19:22 +01:00
Thales Macedo Garitezi
8540566eba chore: prepare to tag e5.3.1-alpha.5 2023-11-08 13:02:19 -03:00
Zaiming (Stone) Shi
f9e9748cec fix(cluster-rpc): boot from local config if table loaded 
When EMQX boots up, it tries to get latest config from peer (core type)
nodes, if none of the nodes are replying, the node will decide
to boot with local config (and replay the committed changes) if
the commit table is loaded from disk locally (an indication of the
data being latest), otherwise it will sleep for 1-2 seconds and
retry.

This lead to a race condition, e.g. in a two nodes cluster:

1. node1 boots up
2. node2 boots up and copy mnesia table from node1
3. node1 restart before node2 can sync cluster.hocon from it
4. node1 boots up and copy mnesia table from node2

Now that both node1 and node2 has the mnesia `load_node` pointing
to each other (i.e. not a local disk load).

Prior to this fix, the nodes would wait for each other in a dead loop.

This commit fixes the issue by allowing node to boot
with local config if it does not have a lagging.
 2023-11-08 15:06:47 +01:00
ieQu1
788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session 
Refactor emqx_durable storage application and introduce learned topic structure storage
 2023-11-08 14:27:40 +01:00
Thales Macedo Garitezi
f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
 2023-11-08 09:53:42 -03:00
Ilya Averyanov
d061d64c70 fix(plugins): fix backward compatibility 2023-11-08 12:36:09 +03:00
Ivan Dyachkov
0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
ieQu1
a1cdbaa76d fix(ds): Address code review remarks 2023-11-08 04:31:27 +01:00
ieQu1
c030188eb7 chore(ds): Add rebar.config file to app/emqx_durable_storage 2023-11-08 03:04:19 +01:00
ieQu1
7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
ieQu1
74cb43f8b1 fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
Andrew Mayorov
54951c273f feat(ds): mix safe cutoff into keymapper filter during iteration 2023-11-08 03:04:19 +01:00
Andrew Mayorov
7a94db25c3 fix(ds): don't iterate over incomplete epoch in bitmask lts storage 2023-11-08 03:04:19 +01:00
Andrew Mayorov
46d8301bc0 feat(emqx): expose timestamp function in emqx_message 
So that the code that relies on it would not need to guess clock source
and precision.
 2023-11-08 03:04:19 +01:00
ieQu1
38b7f516bc fix(ds): Change the directory where message database is stored 2023-11-08 03:04:19 +01:00
ieQu1
8e5dda40be fix(ds): Fix static checks 2023-11-08 03:04:19 +01:00
ieQu1
99329e1243 refactor(ds): Address review remarks 2023-11-08 03:04:19 +01:00
ieQu1
2de79dd9ac feat(ds): Replay QoS1 messages 2023-11-08 03:04:19 +01:00
ieQu1
e745e42093 test(ds): Explore full range of keys when testing ratchet function 2023-11-08 03:04:19 +01:00
ieQu1
87689890ff chore(ds): Fix linter and compilation warnings 2023-11-08 03:04:19 +01:00
ieQu1
465e8a90dd revert(ds): Remove change from the old protocol file 2023-11-08 03:04:19 +01:00
ieQu1
ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
ieQu1
164ae9e94a feat(ds): LTS bitfield storage passes all tests 2023-11-08 03:04:19 +01:00
ieQu1
56b6b176c2 fix(ds): LTS shall keeps the concrete topic indexes 2023-11-08 03:04:19 +01:00
ieQu1
7428e7037b feat(ds): Bitfield + Learned Topic Structure 2023-11-08 03:04:19 +01:00
ieQu1
ac91dbc58f feat(ds): Restore LTS trie from a dump 2023-11-08 03:04:19 +01:00
ieQu1
f1ab7c8a7c feat(ds): Add persist callback to LTS trie 2023-11-08 03:04:19 +01:00
ieQu1
c149e0e2df fix(ds): Pass topic filter to emqx_ds:make_iterator call 2023-11-08 03:04:19 +01:00
ieQu1
51a6f623fd refactor(ds): Split out bitfield keymapper to a different module 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
903b3863d1 chore(ps_ds): make persistent session module use new emqx_ds APIs 2023-11-08 03:04:19 +01:00
ieQu1
2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1
6d65707d41 refactor(ds): Implement drop_db function 2023-11-08 03:04:19 +01:00
ieQu1
c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1
59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1
7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
ieQu1
c91df2f5cd refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
ieQu1
f1454bb57e feat(ds): learned topic structure 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
7ab57824dc chore(ds): change return type of storage_layer:next/{1,2} 
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
 2023-11-08 03:04:19 +01:00
Ilya Averyanov
75704513d8 fix(plugins): fix backward compatibility 2023-11-07 19:06:37 +03:00
Thales Macedo Garitezi
c4c9d7ceb3 chore: prepara to tag e5.3.1-alpha.4 2023-11-07 13:04:46 -03:00
Thales Macedo Garitezi
b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106 
fix(resource): take error from action/connector before attempting query
 2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi
7dcdbc9e51 fix(resource): take error from action/connector before attempting query 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
 2023-11-07 10:04:04 -03:00
Stefan Strigler
fa740a1927 docs: fix inconsistencies for bridge and connector operations 
It's only `start` allowed for now. Also fixed some wordings.
 2023-11-07 13:58:23 +01:00
Zaiming (Stone) Shi
f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config 
Rename connector channel related configs in bridge_v2 to 'parameters'
 2023-11-07 13:41:55 +01:00
Thales Macedo Garitezi
9742dae30d
Merge pull request #11889 from thalesmg/rename-bridgev2-to-action-apis-r53-20231106 
chore: rename `bridges_v2` -> `actions` in the public facing APIs
 2023-11-07 09:29:06 -03:00
JianBo He
d76ed585b0 chore: correct the application vsn 2023-11-07 11:11:20 +08:00
JianBo He
7bd5579980 chore: fix the documentation generation and example conf checking 2023-11-07 11:09:39 +08:00
JianBo He
4ef156d69e chore: fix dialyzer warnings 2023-11-07 09:04:50 +08:00
JianBo He
0d9e0bd3fe chore: fix lot of running bugs 2023-11-07 08:52:10 +08:00
Zaiming (Stone) Shi
bb8f80f256 test: add test case to pin bridge schema json version 0.1.0 2023-11-06 21:59:36 +01:00
Zaiming (Stone) Shi
9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi
2b8cf50a1d chore: rename bridges_v2 -> actions in the public facing APIs 
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
 2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi
6af6309301 chore: upgrade to hocon 0.39.19 2023-11-06 19:16:19 +01:00
Thales Macedo Garitezi
f17b762596 chore: don't disable rule that references non-existent bridge 
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
 2023-11-06 10:09:14 -03:00
JianBo He
6f51b9f842 chore: ensure elvis pass 2023-11-06 12:31:54 +08:00
JianBo He
8871738756 chore: ensure emqx_gateway_ocpp tests pass 2023-11-06 12:26:58 +08:00
JianBo He
bea0acd929 fix(ocpp): ensure ocpp gateway options works 2023-11-06 11:56:50 +08:00
Zaiming (Stone) Shi
5881e34d4e
Merge pull request #11874 from zmstone/1103-validate-bridge-type-and-name-at-entry 
fix(bridge): do not allow dot in bridge name
 2023-11-03 23:04:10 +01:00
Thales Macedo Garitezi
0ff4465c78
Merge pull request #11875 from thalesmg/fix-kafka-connecting-r53-20231103 
fix(kafka_producer): don't return `disconnected` when there are connections issues while starting the bridge
 2023-11-03 17:56:00 -03:00
Thales Macedo Garitezi
a064eecb68
Merge pull request #11879 from thalesmg/fix-rule-api-downgrade-bridge-r53-20231103 
fix(rules_api): downgrade bridge id
 2023-11-03 17:26:09 -03:00
Zaiming (Stone) Shi
21a167e1a5
Merge pull request #11873 from zmstone/1103-unhide-nodelay-field 
fix(kafka): do not hide nodelay field
 2023-11-03 20:49:29 +01:00
Zaiming (Stone) Shi
600747b7e5 fix(bridge): do not allow dot in bridge name 
also validate name at the API entry
 2023-11-03 20:44:57 +01:00
Thales Macedo Garitezi
0ee74cd8ae fix(rules_api): downgrade bridge id 
Fixes https://emqx.atlassian.net/browse/EMQX-11312
 2023-11-03 15:57:13 -03:00
Thales Macedo Garitezi
4265ef66cc fix(kafka_producer): don't return disconnected when there are connection issues while starting the bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixex https://emqx.atlassian.net/browse/EMQX-11298

We don't enforce the connection to be up when starting/creating the bridge, otherwise the
status will be `disconnected` for a possibly transient reason such as network issues or
Kafka broker restart.

Same applies for Azure Event Hub Producer bridge, as they share the same module.
 2023-11-03 15:49:36 -03:00
Kjell Winblad
dd8085d3a4 test(emqx_bridge_v2_api_SUITE): fix broken test case 2023-11-03 17:49:25 +01:00