Commit Graph

10340 Commits

Author SHA1 Message Date
JianBo He cdb90ebe6b feat: rename webhook bridge to http bridge 2023-11-28 09:53:46 +08:00
JianBo He dc99651690 test(bridge): ensure almost test cases passed 2023-11-28 09:53:46 +08:00
JianBo He 8954450c0b chore: fix compile warnings 2023-11-28 09:53:46 +08:00
JianBo He 96af7a74e8 feat: impl the http bridge v2 2023-11-28 09:53:46 +08:00
JianBo He f9a1e747fd chore(http): break the bridge confs to connector and action parts 2023-11-28 09:53:46 +08:00
SergeTupchiy 3b1ae0f3df
Merge pull request #12017 from SergeTupchiy/EMQX-11319-data-backup-http-api
EMQX-11319 data backup http api
2023-11-27 19:23:54 +02:00
Thales Macedo Garitezi e3fbf6c958 test: attempting to stabilize more flaky tests 2023-11-27 14:12:48 -03:00
Thales Macedo Garitezi 26e59f9508
Merge pull request #12024 from thalesmg/ds-session-expiry-m-20231124
feat(ds): session expiry
2023-11-27 14:11:38 -03:00
Stefan Strigler ac370fed61
Merge pull request #12035 from sstrigler/EMQX-11382-a-new-field-should-be-added-to-the-interface-of-the-connector-to-return-the-actions-associated-with-it
feat(emqx_connector): add field 'actions' in API response
2023-11-27 17:39:44 +01:00
Zaiming (Stone) Shi d933d0b9e0
Merge pull request #12016 from zmstone/1122-load-license-file
1122 load license file
2023-11-27 16:43:51 +01:00
Stefan Strigler 70978df5bb fix: use parse_id 2023-11-27 16:32:15 +01:00
Serge Tupchii 89fdbf3e73 fix(emqx_mgmt_cli): remove duplicated word in the log message 2023-11-27 17:30:58 +02:00
Serge Tupchii 64ee29af81 feat(emqx_management): implement data backup API 2023-11-27 17:30:58 +02:00
Andrew Mayorov 86685bdce2
feat(sessds): use integer tags for pubrange types 2023-11-27 18:27:15 +03:00
Andrew Mayorov bb05281adb
refactor(sessds): add dedicated `#ds_pubrange.tracks` field
This slightly simplifies the replayer code.
2023-11-27 18:24:57 +03:00
Thales Macedo Garitezi d88deb9ceb feat(ds): add session timer to bump last alive at timestamp 2023-11-27 12:19:52 -03:00
Stefan Strigler 048f4724a9 feat(emqx_connector): add field 'actions' in API response
Also unify schemas, use emqx_connector_schema for the generic parts.
2023-11-27 16:09:17 +01:00
Thales Macedo Garitezi 09c4e40511 refactor(ds): rename `disconnected_at` to `last_alive_at`, add more assertions 2023-11-27 11:48:44 -03:00
Thales Macedo Garitezi f2dbddc315 test: attempting to stabilize more flaky tests 2023-11-27 11:36:32 -03:00
Thales Macedo Garitezi 6acdc6e432 feat: add `/actions/:id/metrics/`, `/actions/:id/metrics/reset` APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11381
2023-11-27 11:33:11 -03:00
Andrew Mayorov 923898eadf
chore(chan): leave a TODO note for PUBREC handler 2023-11-27 09:55:46 +03:00
Andrew Mayorov 41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs 2023-11-27 09:55:46 +03:00
Andrew Mayorov 46475fac66
feat(sessds): provide QoS2 message replay support 2023-11-27 09:55:45 +03:00
Andrew Mayorov ce59cb71bb
chore: bump emqtt to 1.9.6 2023-11-27 09:55:45 +03:00
zhongwencool b5a00ec6b2
Merge pull request #12023 from emqx/master
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
ieQu1 bd7a84fe3e revert(ds): Don't duplicate the clean start in session_ds 2023-11-26 19:18:59 +01:00
Kjell Winblad c85004b7ef chore: remove obsolete TODO 2023-11-24 21:22:27 +01:00
Kjell Winblad c6c1d886f0 fix: make pgsql action schema properly 2023-11-24 21:19:26 +01:00
Thales Macedo Garitezi 839f9dbedb feat(ds): session expiry
Fixes https://emqx.atlassian.net/browse/EMQX-11048
2023-11-24 16:31:54 -03:00
Kjell Winblad 66945dcc5c fix: address more comments from @thalesmg 2023-11-24 20:31:29 +01:00
Kjell Winblad 30e248061f
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-24 20:07:37 +01:00
Kjell Winblad f070d80b1a fix: swagger examples for PostgreSQL, Matrix and Timescale 2023-11-24 18:50:52 +01:00
Kjell Winblad d03674a505 fix: duplicate key in example 2023-11-24 17:59:56 +01:00
Kjell Winblad fc7bedb81a fix: remove duplicated entry 2023-11-24 17:37:58 +01:00
Kjell Winblad e920160805 fix: add enable and description fields to PostgreSQL connector 2023-11-24 17:33:39 +01:00
Kjell Winblad 3dca83c854 fix: all missing descriptions 2023-11-24 17:33:39 +01:00
Kjell Winblad c5e281b84b fix: emqx_auth_postgresql test suites 2023-11-24 17:33:39 +01:00
Kjell Winblad 6ef9c6fe4a fix: ops 2023-11-24 17:33:39 +01:00
Kjell Winblad f7296d549f fix: elvis problem 2023-11-24 17:33:39 +01:00
Kjell Winblad 64c015cf6f fix: unify the schema modules in emqx_bridge_pgsql 2023-11-24 17:33:39 +01:00
Kjell Winblad b8f510d956 fix: add env variables about action info 2023-11-24 17:33:39 +01:00
Kjell Winblad 2e3028a8f8 fix(emqx_postgresql): fix lifecycle test 2023-11-24 17:33:39 +01:00
Kjell Winblad dbe73c70b1 fix: dialyzer problem 2023-11-24 17:33:38 +01:00
Kjell Winblad d5b62eead0 feat: split pgsql, matrix and timescale into connector action
This commit splits the bridges pgsql, matrix and timescale into
connector and action.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
2023-11-24 17:33:35 +01:00
Stefan Strigler f8f8cf9f30
Merge pull request #11963 from sstrigler/EMQX-11156-bridge-v-2-mongo-db-support
EMQX 11156 bridge v2 mongo db support
2023-11-24 16:49:57 +01:00
zhongwencool 8f548f4cbb
Merge pull request #12020 from thalesmg/sync-r53-m-20231124
chore: sync `release-53` to `master`
2023-11-24 22:54:53 +08:00
Stefan Strigler c1ef773e77 fix: check for sane state after regular shutdown 2023-11-24 15:09:35 +01:00
Stefan Strigler 4e077c951b feat(emqx_bridge_mongodb): port mongodb to shared connector and actions 2023-11-24 15:09:35 +01:00
Thales Macedo Garitezi 261fe8a831 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 10:10:09 -03:00
ieQu1 2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start
Discard sessions when client connects with clean start = true
2023-11-24 13:35:39 +01:00
Thales Macedo Garitezi e95ec5b150 test: fix another flaky test 2023-11-24 09:24:21 -03:00
Thales Macedo Garitezi f8fd95c683 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 09:22:24 -03:00
Thales Macedo Garitezi 6a3c438194
Merge pull request #12009 from thalesmg/fix-kafka-get-bridges-r53-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-24 08:56:24 -03:00
Stefan Strigler 9ebbc9bbea refactor(emqx_bridge): use more simplistic function to validate connector 2023-11-24 10:07:11 +01:00
Stefan Strigler d2901afd1b fix(emqx_bridge_kafka): match example in api schema 2023-11-24 10:07:11 +01:00
ieQu1 e616e0746a feat(ds): Implement a function for dumping persistent session state 2023-11-24 04:39:46 +01:00
ieQu1 8dfcb69e52 feat(ds): Discard session when client connects with CleanStart=1 2023-11-24 04:17:28 +01:00
ieQu1 e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0
Support QoS0 messages in persistent_session_ds
2023-11-23 23:43:18 +01:00
ieQu1 449bafc27e fix(ds): LTS trie handles empty topic levels 2023-11-23 22:16:38 +01:00
ieQu1 a158f25a40 fix(ds): Fix return type of emqx_persistent_session_ds:publish 2023-11-23 22:16:38 +01:00
ieQu1 c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1 1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message
feat(ds): Don't store #message record in the DB
2023-11-23 18:56:45 +01:00
Stefan Strigler 1c11a02a76
Merge pull request #12010 from sstrigler/EMQX-11415-e-5-3-2-api-should-return-404-when-the-built-in-database-authentication-doesnt-exist
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
2023-11-23 18:08:48 +01:00
ieQu1 c69b82455e feat(ds): Don't store #message record in the DB 2023-11-23 17:56:35 +01:00
Zaiming (Stone) Shi 14077ec43b feat(license): allow setting 'default' license key 2023-11-23 17:52:17 +01:00
Zaiming (Stone) Shi caaf8113fc feat(license): support loading license from file 2023-11-23 17:52:17 +01:00
Thales Macedo Garitezi 4597e1c5b9
Merge pull request #12011 from thalesmg/fix-kafka-get-bridges-m-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-23 13:15:39 -03:00
ieQu1 1ced8786fd feat(ds): Make session poll interval configurable 2023-11-23 17:10:03 +01:00
ieQu1 82e74d0201 feat(ds): Add a flag that forces all sessions to become durable 2023-11-23 17:10:03 +01:00
Stefan Strigler 8ba116d378 fix(emqx_auth): check authenticator exists in /authenticator/:id/users 2023-11-23 16:15:03 +01:00
Ilya Averyanov ce77a12cd3
Merge pull request #12012 from savonarola/1123-fix-callback-types
fix(ds): impose more flexible types for ds implementations
2023-11-23 17:03:56 +02:00
Thales Macedo Garitezi fdfa3213cc fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 11:41:26 -03:00
Zaiming (Stone) Shi 7375bc5f9b fix(license): allow CRLF in license keys 2023-11-23 15:39:18 +01:00
Zaiming (Stone) Shi 48b69bd60c chore: delete a TODO from perfectionism 2023-11-23 15:39:18 +01:00
Ilya Averyanov 29d42506d5 fix(ds): impose more flexible types for ds implementations 2023-11-23 17:16:11 +03:00
Thales Macedo Garitezi 8e8d6d2192
Merge pull request #12006 from thalesmg/test-debug-flaky-test-r53-20231122
test(gcp_pubsub_consumer): fix flaky test
2023-11-23 10:57:22 -03:00
Thales Macedo Garitezi c89ec0b1f7 fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 10:25:36 -03:00
Thales Macedo Garitezi f3693e5dbc fix(gcp_pubsub_producer): add missing references to api specs 2023-11-23 09:02:52 -03:00
Zaiming (Stone) Shi 209077593d
Merge pull request #11970 from zmstone/1108-add-v2-schema-dump
1108 add v2 schema dump
2023-11-22 23:42:54 +01:00
Thales Macedo Garitezi db83457d13 test: fix flaky test
The cause was that the call `sys:terminate/2` was timing out...

`exit/2` doens't always work:

```
 2023-11-22 19:14:40.974
killed async workers

Error: -22T19:14:40.974563+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15908.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 28, reductions: 1822; neighbours:
Error: -22T19:14:40.998051+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15908.7>.
2023-11-22T19:15:41.088752+00:00 [critical] Run stage failed: error:{badmatch,{timeout,#{expected_remaining => 1,mailbox => {messages,[]},msgs_so_far => []}}}, Stacktrace: [{emqx_bridge_gcp_pubsub_consumer_SUITE,'-t_async_worker_death_mid_pull/1-fun-17-',3,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1576}]},{emqx_bridge_gcp_pubsub_consumer_SUITE,t_async_worker_death_mid_pull,1,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1505}]}], Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-11-22_19.14.27/snabbkaffe/1700680540975786370.log", mfa: undefined
Error: -22T19:15:46.095702+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15934.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 1471; neighbours:
Error: -22T19:15:46.095192+00:00 [error] Supervisor: {local,ehttpc_sup}. Context: shutdown_error. Reason: killed. Offender: id={ehttpc_pool_sup,<<98,114,105,100,103,101,58,103,99,112,95,112,117,98,115,117,98,95,99,111,110,115,117,109,101,114,58,116,95,97,115,121,110,99,95,119,111,114,107,101,114,95,100,101,97,116,104,95,109,105,100,95,112,117,108,108,45,53,55,54,52,54,48,55,53,50,51,48,51,52,50,50,55,53,49>>},pid=<0.15903.7>.
Error: -22T19:15:46.095470+00:00 [error] Supervisor: {<0.15906.7>,ehttpc_worker_sup}. Context: shutdown_error. Reason: killed. Offender: id={worker,1},pid=<0.15924.7>.
Error: -22T19:15:46.096762+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15934.7>.
Warning: 2T19:15:46.098278+00:00 [warning] msg: remove_local_resource_failed, mfa: emqx_resource:remove_local/1(362), error: {error,timeout}, resource_id: <<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>
Error: -22T19:15:46.149090+00:00 [error] Generic server <0.15904.7> terminating. Reason: killed. Last message: {'EXIT',<0.15903.7>,killed}. State: {state,<<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>,1,random}.
Error: -22T19:15:46.149525+00:00 [error] crasher: initial call: ehttpc_pool:init/1, pid: <0.15904.7>, registered_name: [], exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,909}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.15903.7>,ehttpc_sup,<0.15731.7>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 3428; neighbours:
```
2023-11-22 17:41:32 -03:00
Zaiming (Stone) Shi d9f964a44f test: fix test cases after schema type namespace change 2023-11-22 16:58:05 +01:00
Thales Macedo Garitezi fc849f0c05 ci(test): add info to help diagnose flaky test 2023-11-22 12:36:10 -03:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Zaiming (Stone) Shi e065217e9a
Merge pull request #11997 from zmstone/1121-do-not-allow-special-char-as-prefix-for-resource-name
1121 do not allow special char as prefix for resource name
2023-11-22 15:46:11 +01:00
Thales Macedo Garitezi 274a254624
Merge pull request #12002 from thalesmg/multiple-action-infos-per-app-m-20231122
feat(actions): allow multiple action info modules per application
2023-11-22 10:42:56 -03:00
Thales Macedo Garitezi bb549cdf8b feat(actions): allow multiple action info modules per application 2023-11-22 09:33:01 -03:00
Thales Macedo Garitezi 38d3a1d7d0 feat(actions): allow multiple action info modules per application 2023-11-22 09:30:37 -03:00
Zaiming (Stone) Shi db33bc616a feat(schema): Add v2 scheam JSON dump 2023-11-22 13:12:35 +01:00
Zaiming (Stone) Shi 1b2c052646 docs: add type namespaces 2023-11-22 13:12:35 +01:00
Serge Tupchii d9f95cdc56 feat: integrate OpenTelemetry log handler 2023-11-22 13:59:30 +02:00
Ilya Averyanov a1b9a14fa1 feat(ds): allow fdb implementation for durable storage 2023-11-22 14:49:38 +03:00
Zaiming (Stone) Shi 3261a12140 fix(emqx_resource): do not allow leading _ or - as resource name 2023-11-22 10:58:54 +01:00
ieQu1 ee191803ea
Merge pull request #11990 from ieQu1/dev/ds-shards
Implement a prototype of durable message sharding
2023-11-22 00:57:54 +01:00
Zaiming (Stone) Shi 869e73d637
Merge pull request #11966 from thalesmg/fix-kafka-parameters-r53-20231117
fix(kafka): don't return `parameters` from `/bridges` API
2023-11-22 00:36:39 +01:00
Zaiming (Stone) Shi 4fb2d3a503
Merge pull request #11996 from zmstone/1121-prepare-for-e5.3.2-alpha.1
chore: bump release version to e5.3.2-alpha.1
2023-11-21 22:01:05 +01:00
Thales Macedo Garitezi 39791511fc chore: remove obsolete workaround 2023-11-21 17:38:13 -03:00
Thales Macedo Garitezi 01895fd660
Merge pull request #11995 from thalesmg/fix-action-api-bad-name-r53-20231121
fix(actions_api): don't crash on validation errors
2023-11-21 17:27:38 -03:00
Zaiming (Stone) Shi 5ebd954b16 chore: bump release version to e5.3.2-alpha.1 2023-11-21 21:19:05 +01:00
Zaiming (Stone) Shi 8fbdcab118
Merge pull request #11987 from zmstone/1120-do-not-crash-on-einval-after-check-cache
fix(emqx_connection): handle socket activation error return
2023-11-21 20:31:56 +01:00
Thales Macedo Garitezi 3a8c332805 fix(actions_api): don't crash on validation errors
Fixes https://emqx.atlassian.net/browse/EMQX-11394
2023-11-21 16:27:54 -03:00
ieQu1 3165b4f645 fix(ds): Abort application startup when rocksdb is not avialable 2023-11-21 20:27:07 +01:00
Thales Macedo Garitezi 11ec1a30a0 test(flaky): fix flaky pulsar test 2023-11-21 16:00:19 -03:00
Thales Macedo Garitezi b3dffa4390 fix(kafka): don't return `parameters` from `/bridges` API
Fixes https://emqx.atlassian.net/browse/EMQX-11412
2023-11-21 16:00:15 -03:00
ieQu1 3d823beb11 fix(ds): Apply review remarks 2023-11-21 19:37:01 +01:00
ieQu1 4d47490734 chore(ds): Rebase configuration 2023-11-21 19:37:01 +01:00
ieQu1 f5c71e8068 refactor(ds): Add a wrapper to the store batch API 2023-11-21 19:37:01 +01:00
ieQu1 2a1f7d946a feat(ds): Shard messages by publisher client ID 2023-11-21 19:37:01 +01:00
ieQu1 62542e5844 feat(ds): Metadata storage for the replication layer 2023-11-21 19:37:01 +01:00
Kjell Winblad b0d670aaa9
Merge pull request #11992 from kjellwinblad/kjell/backport/action_upgrade_downgrade_hooks
fix(action): upgrade and downgrade strategy
2023-11-21 19:14:37 +01:00
Thales Macedo Garitezi 9e1796ec4f feat(gcp_pubsub_producer): migrate GCP PubSub producer to actions
Fixes https://emqx.atlassian.net/browse/EMQX-11157
2023-11-21 14:22:42 -03:00
Zaiming (Stone) Shi fa91bacdfb
Merge pull request #11975 from zmstone/1119-fix-socket-close-race-condition
fix(emqx_channel): do not log stale sock_close event as error
2023-11-21 16:47:03 +01:00
Kjell Winblad fa7151f255 fix: port emqx_utils_maps:rename function from master
The emqx_utils_maps:rename function is needed by action
upgrade/downgrade hoos.
2023-11-21 15:18:22 +01:00
Thales Macedo Garitezi 6c9a8461f7
Merge pull request #11989 from thalesmg/confluent-action-m-20231120
feat: add confluent connector/action
2023-11-21 11:01:40 -03:00
Kjell Winblad cd72dc11dd fix: missing emqx_action_info module mapping 2023-11-21 14:22:29 +01:00
Thales Macedo Garitezi 068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120
sync `release-53` to `master`
2023-11-21 09:02:51 -03:00
Thales Macedo Garitezi 759468efcf
Merge pull request #11986 from thalesmg/split-resource-opts-actions-r53-20231120
refactor: split `resource_opts` fields between connector and actions
2023-11-21 09:01:29 -03:00
Thales Macedo Garitezi 75ee3ced68 feat: add confluent connector/action
Fixes https://emqx.atlassian.net/browse/EMQX-11044

See also: https://emqx.atlassian.net/wiki/spaces/EMQX/pages/712179934/Confluent+Integration
2023-11-21 08:59:42 -03:00
Kjell Winblad 6030bf6fa5 fix(action): upgrade and downgrade strategy
This commit adds upgrade and downgrade hooks that are called when
upgrading from a bridge V1 to connector and action or the other way
around. The automatic translation is used if the callback is not
defined.

NOTE: Backported from master
2023-11-21 11:18:21 +01:00
Ilya Averyanov c6fd1e4c75
Merge pull request #11971 from savonarola/1116-no-auth-rebalance-status
feat(rebalance): improve rebalance usability
2023-11-21 12:10:18 +02:00
Andrew Mayorov cd48c283db
Merge pull request #11988 from keynslug/ft/EMQX-11222/basic-config
feat(schema): introduce separate root for new session persistence
2023-11-21 15:02:21 +07:00
JianBo He 9716b051d9 fix(ocpp): fix bad webscoket schema for HTTP API 2023-11-21 11:24:23 +08:00
Thales Macedo Garitezi 6f8630304d
Merge pull request #11980 from kjellwinblad/kjell/fix_fixup_callbacks/EMQX-11428
fix: bridge to action upgrade fixup hook should run after upgrade
2023-11-20 16:19:43 -03:00
Kjell Winblad 7fb5ade832 fix(kafka_producer): hocon renames kafka field 2023-11-20 19:20:13 +01:00
Kjell Winblad d214ae8772
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-20 18:59:57 +01:00
Andrew Mayorov 110a5a4896
feat(schema): introduce separate root for new session persistence
With some knobs to choose a storage backend. Support only builtin
RocksDB-based backend with minimal configuration for now.
2023-11-20 23:49:14 +07:00
Zaiming (Stone) Shi b02711af79 refactor(emqx_ws_connection): rename cache to buffer for limiter 2023-11-20 17:47:20 +01:00
Andrew Mayorov 8e107ffe45
Merge pull request #11958 from keynslug/ft/EMQX-9745/preserve-acks
feat(sessds): preserve acks / replays in session state
2023-11-20 23:40:22 +07:00
Kjell Winblad edbfe090c9 feat: add custom callback for creating connector from bridge V1 config 2023-11-20 17:23:51 +01:00
Zaiming (Stone) Shi ec19247271 refactor: rename limiter buffer related messages and var names 2023-11-20 16:55:26 +01:00
Andrew Mayorov 9684e79ee0
fix(sessds): ensure dup flag is on for replayed messages 2023-11-20 22:54:40 +07:00
Zaiming (Stone) Shi 8ec3b1db5d fix(emqx_connection): handle socket activation error return 2023-11-20 16:51:39 +01:00
Kjell Winblad 3aa8044475 fix(action): upgrade and downgrade strategy
This commit fixes the upgrade and downgrade strategy when upgrading
from a bridge V1 to connector and action or the other way around so that
the custom callbacks get the complete unchanged input instead of the
result of the automatic translation. The automatic translation is used
if the callback is not defined.
2023-11-20 16:40:27 +01:00
Thales Macedo Garitezi f40f6bc5dd refactor: split `resource_opts` fields between connector and actions 2023-11-20 12:37:36 -03:00
Stefan Strigler 1f1d9e58c6 fix(emqx_connector): don't crash in API on delete with active channels 2023-11-20 16:23:46 +01:00
Thales Macedo Garitezi 53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Andrew Mayorov ef7cfd0202
feat(sessds): add field to `ds_pubrange` for forward-compat 2023-11-20 19:56:55 +07:00
Thales Macedo Garitezi b42d4e5ab0
Merge pull request #11950 from thalesmg/fix-bridge-v1-examples-r53-20231114
fix(bridges_v1): avoid merging action examples for non-v1 bridges
2023-11-20 09:12:23 -03:00
Thales Macedo Garitezi 926078b82c
Merge pull request #11960 from thalesmg/fix-kafka-producer-channel-status-r53-20231116
fix(kafka_producer): make status `connecting` while the client fails to connect
2023-11-20 09:12:05 -03:00
Kjell Winblad 79a764f117 fix: bridge to action upgrade fix up hook should run after upgrade
This commit changes how the `emqx_action_info` callback
`bridge_v1_to_action_fixup/1` works. It is now called after the
automatic upgrade instead of before. Since the full Bridge V1 config
might be needed to do the fixup, it is provided in a special field
`<<"__bridge_v1_conf__">>`. The `<<"__bridge_v1_conf__">>` field is
removed after the callback is called and can thus be ignored if it is
not needed.
2023-11-20 12:06:21 +01:00
Stefan Strigler 84ff7b0b38 feat(emqx_bridge): action_info with dynamic lookup
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
Andrew Mayorov d7974d835f
fix(sharesub): anticipate messages w/o redispatch header
For instance, `emqx_session_mem` will push almost all of the messages
that are still in the state to `emqx_shared_sub:redispatch/1`. After
this commit, the session will no longer crash during channel terminate.
2023-11-20 15:53:24 +07:00
Andrew Mayorov 7081f1951f
refactor(sessds): use `ds_pubrange` record as is
Instead of converting it into almost similar runtime representation.
2023-11-20 15:40:43 +07:00
Andrew Mayorov a5ff4144fe
test(sessds): add complex testcase for session replay 2023-11-20 15:40:43 +07:00
Andrew Mayorov 1246d714c5
feat(sessds): preserve acks / ranges in mnesia for replays 2023-11-20 13:33:48 +07:00
Andrew Mayorov 5b40304d1f
chore(sessds): simplify subscriptions handling
There's currently no point in storing parsed topic filters in the
subscriptions table.
2023-11-20 13:33:48 +07:00
Andrew Mayorov 648b6ac63e
chore(sessds): rename iterators -> subscriptions
Also try to make clearer the difference between 2 flavors of topic
filter representation in use.
2023-11-20 13:33:48 +07:00
Zaiming (Stone) Shi e73bf716ae fix(emqx_channel): do not log stale sock_close event as error
In some cases, EMQX may decide to close socket and mark connection
at 'disconnected' state, for example, when DISCONNECTE packet is
received, or, when failed to write data to socket.
However, by the time EMQX decided to close the socket, the socket
might have already been closed by peer, and the `tcp_closed` envet
is already delivered to the process mailbox -- causing EMQX to
handle sock_close event at 'disconnected' state.
2023-11-19 22:27:25 +01:00
Ilya Averyanov e93e9ed108 feat(rebalance): improve rebalance usability
* make availability API endpoint public
* allow connections during wait_health_check interval
* make availability status calculation more consistent and lightweight
* refactor test to get rid of some mocks and to use cth
2023-11-18 21:29:16 +03:00
Thales Macedo Garitezi eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Kjell Winblad 86c126ffcd feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
2023-11-17 16:10:08 -03:00
Kjell Winblad 9feba802e9 chore: add convenience function for creating action schemas 2023-11-17 16:10:08 -03:00
Ilya Averyanov 58437cd35a fix(mongodb): fix deadlock while stopping mongodb resource 2023-11-17 12:34:36 +03:00
zhongwencool b2e168d508 feat: add prometheus.push_gateway.enable config 2023-11-17 09:48:39 +08:00
zhongwencool c4eb9f86e7 chore: add refactor prometheus refactor changelog 2023-11-17 08:23:51 +08:00
zhongwencool b343653769 chore: support list(tuple()) for schema_with_examples/3 2023-11-17 08:23:51 +08:00
zhongwencool 0c2610fa61 feat: support prometheus.enable_basic_auth 2023-11-17 08:23:51 +08:00
zhongwencool c376a5db29 refactor: prometheus config 2023-11-17 08:23:51 +08:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Ilya Averyanov 4c54efd8b1 fix(auth): fix deadlock while stopping mongodb resource 2023-11-16 19:29:08 +03:00
Thales Macedo Garitezi 36b5d58957 test: reorganize test suite a bit 2023-11-16 13:16:02 -03:00
Thales Macedo Garitezi 90571b7d8e test: fix noise about undefined unofficial callbacks 2023-11-16 13:16:02 -03:00
Andrew Mayorov 656d575e2d
ci(router): fix flaky testcase 2023-11-16 17:53:20 +07:00
Andrew Mayorov d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default
feat(router): switch to v2 routing store by default
2023-11-16 15:56:26 +07:00
Andrew Mayorov 893e90b372
fix(kafka): use safe publish in consumer
Routing with v2 schema is actually more strict with respect to input to
`emqx_router` module routines. This causes Kafka consumer bridge to
crash when it tries to publish a message to a topic that looks like a
topic filter.
2023-11-15 22:37:00 +07:00
Andrew Mayorov 98f947f4f3
ci(router): fix flaky testcase 2023-11-15 22:28:52 +07:00
Andrew Mayorov 8919b08207
fix(utils): rename `emqx_utils_stream:take/2` to `consume/2`
Which is more neutral and harder to confuse with a destructive `take` in
collections.
2023-11-15 17:20:40 +07:00
Ivan Dyachkov 65ba381fd8 Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53 2023-11-15 09:25:03 +01:00
Thales Macedo Garitezi f5545bbdf3 fix(schema_types): add missing `secret()` type 2023-11-14 16:08:13 -03:00
Thales Macedo Garitezi 5275a01727 fix(syskeeper): refactor action to new action info scheme 2023-11-14 19:49:28 +01:00
Thales Macedo Garitezi d2cf29fae4 fix(schema): use `hoconsc:union/1` 2023-11-14 19:49:08 +01:00
Thales Macedo Garitezi 423fb32171 test: fix noise about undefined unofficial callbacks 2023-11-14 19:48:39 +01:00
Ivan Dyachkov 7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Thales Macedo Garitezi 2f1d88d414 fix(bridges_v1): avoid merging action examples for non-v1 bridges
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
2023-11-14 13:56:50 -03:00
Zaiming (Stone) Shi 88637f81d1
Merge pull request #11936 from zmstone/1112-readable-types
refactor(schema): keep type converters close
2023-11-14 17:56:07 +01:00
ieQu1 124c0e2dba fix(ds): Respect receive_maximum from the connection info 2023-11-14 16:06:19 +01:00
Ivan Dyachkov bbf00c2f71
Merge pull request #11946 from id/1114-sync-r53
sync release-53 to master
2023-11-14 13:38:24 +01:00
Thales Macedo Garitezi 01a2a3b1c0
Merge pull request #11930 from thalesmg/ds-session-discard-m-20231110
feat(ds): implement session discard
2023-11-14 09:30:34 -03:00
Thales Macedo Garitezi d6e9bbb95c fix(connector): validate connector name before converting ssl certs
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
2023-11-14 09:29:59 -03:00
Andrew Mayorov 6812ee9d0f
fix(mgmt): hide route selection behind router interface
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
2023-11-14 18:36:25 +07:00
Ivan Dyachkov 28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
Ivan Dyachkov 1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Andrew Mayorov d1c3b1c659
fix(bridges): handle secrets more consistently
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 16:05:54 +07:00
Andrew Mayorov d4dd4508c9
feat(greptimedb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 5d620465e2
feat(influx): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 7817502b8b
feat(rocketmq): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 2449d54b1f
feat(pulsar): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov e2b7b33d14
feat(kinesis): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 93eaf0caee
feat(dynamo): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 7250059c60
feat(sqlserver): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 06861e377f
feat(ldap): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov fda395014c
test(kafka): verify file-based secrets work 2023-11-14 16:05:25 +07:00
Andrew Mayorov aa458b65d6
test(kafka): simplify consumer testsuite matrix setup 2023-11-14 16:05:25 +07:00
Andrew Mayorov 18cd98def6
chore(test): fix formatting quirks 2023-11-14 16:05:25 +07:00
Andrew Mayorov c300eb41a7
feat(kafka): accept wrapped secrets as passwords 2023-11-14 16:05:25 +07:00
Andrew Mayorov 9c5856029f
feat(cassa): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov 8b4ac8eb4f
feat(oracle): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov 34aeeab041
feat(rabbitmq): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov a69a78d024
feat(clickhouse): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov fc340a276e
feat(mongo): accept wrapped secrets as passwords
Also test authorization with mongo in bridge / auth test suites.
2023-11-14 16:05:24 +07:00
Andrew Mayorov f827df2821
test(mysql): slightly simplify bridge testsuite 2023-11-14 16:05:23 +07:00
Andrew Mayorov 36e57a479d
feat(mysql): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov ad72ab7a8c
feat(redis): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov 4385b2f020
feat(pgsql): accept wrapped secrets as passwords
That are coming from `emqx_schema_secret`. Also adapt pgsql-related
connectors.
2023-11-14 16:05:23 +07:00
Andrew Mayorov 2b36b48f98
feat(connector): use `emqx_schema_secret` as password field schema 2023-11-14 16:05:23 +07:00
Ivan Dyachkov a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad d682e6e23c test: fix test mock by calling exported function 2023-11-14 09:20:46 +01:00
Kjell Winblad 3bea3496af test: fix test case mock that broke due to removal 2023-11-14 09:20:46 +01:00
Kjell Winblad 093c8b0c6e docs: add missing copyright headers 2023-11-14 09:20:46 +01:00
Kjell Winblad ca3e5eab11 refactor: remove emqx_bridge_v2_enterprise.erl 2023-11-14 09:20:46 +01:00
Kjell Winblad e93b71d8d5 fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 09:20:46 +01:00
Kjell Winblad 49fdfef8c3 fix: azure event hub names are defined in one place 2023-11-14 09:20:46 +01:00
Kjell Winblad 5e8e407017 refactor: action schema retrival after PR feedback 2023-11-14 09:20:46 +01:00
Kjell Winblad ab078647a5 chore: fix formatting problem 2023-11-14 09:20:46 +01:00
Kjell Winblad bdab421885 fix(bridge_v2): name and copyright header 2023-11-14 09:20:46 +01:00
Kjell Winblad ca2cdbc08d fix: bapi static check error 2023-11-14 09:20:46 +01:00
Kjell Winblad 7839f0cbc8 chore(bridge_v2): make dialyzer and bpapi happy 2023-11-14 09:20:46 +01:00
Kjell Winblad d26a1b9afb fix(bridge_v1): no hard coded downgrade and upgrade type functions 2023-11-14 09:20:46 +01:00
Kjell Winblad a6aa81b548 fix(emqx_bridge_v2): dialyzer found bad format parameter error 2023-11-14 09:20:46 +01:00
Kjell Winblad 3c778121a5 fix: bridge V1 type lookup issue 2023-11-14 09:20:46 +01:00
Kjell Winblad 77aaff137a test(emqx_bridge_v2_SUITE): fix incorrect assumption about return type
This commit fixes an incorrect assumption about the return type of
emqx_bridge_v2:lookup/2 in emqx_bridge_v2_SUITE:t_load_config_success/1.
2023-11-14 09:20:46 +01:00
Kjell Winblad 14e305e2a6 fix(emqx_bridge_v2): xref warning 2023-11-14 09:20:46 +01:00
Kjell Winblad 9eaee8f333 refactor(emqx_bridge_v2): make independent of Kafka
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
2023-11-14 09:20:46 +01:00
Kjell Winblad cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Kjell Winblad 99031f0dae refactor(bridge_v2): prefix compatibility functions with bridge_v1 2023-11-14 09:20:46 +01:00
Kjell Winblad 729e7df0d5 docs(bridge_v2): add specs for CRUD functions 2023-11-14 09:20:46 +01:00
Ivan Dyachkov a9cf8fe78c chore: 5.3.1 2023-11-14 08:44:41 +01:00
Zaiming (Stone) Shi ab1f3b6b2f fix(emqx_conf_schema_types): address review comments 2023-11-13 21:55:03 +01:00
Stefan Strigler fc2891d6de
Merge pull request #11937 from sstrigler/EMQX-11318-crash-in-get-bridges-v-2-if-a-broken-bridge-is-configured
Emqx 11318 crash in get bridges v 2 if a broken bridge is configured
2023-11-13 17:28:56 +01:00
Zaiming (Stone) Shi 4c5d64abc2 refactor(schema): keep type converters close 2023-11-13 15:26:30 +01:00
Stefan Strigler 17b18849de test(emqx_bridge): test broken config 2023-11-13 14:52:06 +01:00
Zaiming (Stone) Shi 9b9908780c
Merge pull request #11926 from kjellwinblad/kjell/fix_bridge_v2_to_action_cluster_hocon/EMQX-11353
fix: reflect bridge V1 upgreade in cluster.hocon
2023-11-13 14:18:57 +01:00
Stefan Strigler 0b3645057d fix(emqx_bridge): don't try to handle what we don't understand 2023-11-13 14:08:54 +01:00
Stefan Strigler 0d33284672 fix(emqx_bridge): don't crash uninstallling broken action 2023-11-13 14:08:54 +01:00
Stefan Strigler 4fe1d332b3 fix(emqx_bridge): don't crash if there's no status field 2023-11-13 14:08:54 +01:00
Thales Macedo Garitezi 45dad2ed3a feat(ds): implement session discard
Fixes https://emqx.atlassian.net/browse/EMQX-9739

Fixes some issues to ensure the session is discarded when the client connects with
`clean_start = true`, and added some cleanup to subscriptions/routes/iterators/streams.

> There is an API that session garbage collector can use to perform cleaning

We already have `emqx_session:destroy/1`, which could serve as an API for a periodic
session GC to use.
2023-11-13 09:06:22 -03:00
Kjell Winblad 26fc9ed89d
fix: log warning message after suggestion from @zmstone
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-11-13 12:53:09 +01:00
Andrew Mayorov 03843c6071
feat(router): switch to v2 routing store by default 2023-11-13 17:43:30 +07:00
Kjell Winblad 5838568d5d fix: hard coded call to schema module
The user can configure the configuration loader and schema module so we
should not hard code a call to the default schema module.
2023-11-13 11:41:40 +01:00
Zaiming (Stone) Shi 518b02fc70
Merge pull request #11931 from zmstone/1110-delete-stale-type-converters
1110 delete stale type converters
2023-11-13 10:27:51 +01:00
Zaiming (Stone) Shi 65efa2672e test(schema): delete stale reference to emqx_schema:file() type 2023-11-13 09:28:26 +01:00
Zaiming (Stone) Shi 2fda91ec59 refactor: delete default type converter for dashboard schema json
prior to this commit, dashboard scheam json type conversion
has a 'fallback' type for all unknown types, which is 'string'

this commit removes this fallback and add all possible types
to the function clauses.

also, some of the old types which are no longer in use
are deleted from both swagger spec converter and dashboard
sechema converter
2023-11-12 20:08:53 +01:00
JimMoen 952ab08991
fix: make dialyzer happy 2023-11-12 21:44:24 +08:00
JimMoen 20bf9ed26d
fix(gw_gbt32960): mountpoint should not with `/` suffix 2023-11-12 21:44:23 +08:00
JimMoen 867d5aaea6
chore: rm dbg funcs in SUITE 2023-11-12 21:44:23 +08:00
JimMoen 70fcfdddcb
test(gw_jt808): fix test auth server bootstrap 2023-11-12 21:44:23 +08:00
JimMoen dc64068e41
test(gw_jt808): frame parser and channel SUITE 2023-11-12 21:44:23 +08:00
JimMoen 4d493292fb
feat(gw_jt808): port jt808 gateway from 4.4 2023-11-12 21:44:18 +08:00
JimMoen 526faa0362
fix(gw_gbt32960): clean frame parser state acc data 2023-11-12 21:43:12 +08:00
JimMoen d30f0aa4a0
fix(gw_gbt32960): parse state atom key error 2023-11-12 21:43:11 +08:00
JimMoen a4da1d260a
docs(gw_jt808): data exchange guide cn 2023-11-12 21:43:11 +08:00
JimMoen 17edeead6e
fix: erlang_ls complains gateway hrl file can not be found 2023-11-12 21:43:11 +08:00
JimMoen a218ff6dd8
fix(gw_gbt32960): undefined type `emqx_gbt32960_frame:frame()` 2023-11-12 21:43:11 +08:00
JimMoen aef5b17ed5
fix(gw_lwm2m): lwm2m channel behaviour 2023-11-12 21:43:11 +08:00
JimMoen caade4a7e5
chore: emacs mode entry comment for non-erl files 2023-11-12 21:43:11 +08:00
JimMoen 25e277bdc6
chore: fix schema desc and README.md typo 2023-11-12 21:43:10 +08:00
JimMoen 523bb17fc8
test: mark proper types gateway connection modules need be fixed 2023-11-12 21:43:10 +08:00
Zaiming (Stone) Shi b7d2c38213 refactor: delete stale type converters
Some of the types are refactored to use typerefl alias
or hoconsc:enum, no need to keep the special function
clauses to translate them to swagger spec
2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi a32cd20758 refactor(emqx_authn_jwt_schema): add alias for verify_claims
verifiy_claims is a list after converted, but input is a map
2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi 021f7e6b49 refactor(schema): comma_separated_list is list(string()) not list(any()) 2023-11-12 11:37:58 +01:00
Zaiming (Stone) Shi 06e440260f refactor(schema): add args to map alias 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi 8be718b22f refactor(emqx_schema): delete unused type bar_separated_list 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi ff5916e33f docs: fix bad html tag 2023-11-12 10:31:37 +01:00
Xinyu Liu 7bb3ca5519
Merge pull request #11909 from emqx/bugfix-plugins
fix: emqx failed to start if plugin dir is removed by the user
2023-11-11 10:51:26 +08:00
Thales Macedo Garitezi b4e45f0189
Merge pull request #11923 from thalesmg/sync-m-to-r54-20231109
chore: sync `master` to `release-54`
2023-11-10 13:04:55 -03:00
Thales Macedo Garitezi 088de9476c fix(ds): use integer for tag values
Follow up to https://github.com/emqx/emqx/pull/11906#discussion_r1389115973
2023-11-10 11:26:32 -03:00
ieQu1 5170267fef
Merge pull request #11924 from ieQu1/dev/emqx-ds-behavior
feat(ds): Introduce emqx_ds behavior
2023-11-10 15:23:28 +01:00
Kjell Winblad 6ebf0d1bdd fix: reflect bridge V1 upgrade in override configurations
Fixes:
https://emqx.atlassian.net/browse/EMQX-11353
2023-11-10 14:52:29 +01:00
Zaiming (Stone) Shi 101990b8fc test: fix schema tests after types change 2023-11-10 14:23:43 +01:00
Zaiming (Stone) Shi e653c6b4e0 chore: bump app versions 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 855b3c5b29 test: ensure atom exists 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi f1de0aa176 fix(schema): add namespace to authn schemas 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 86110824eb feat: upgrade hocon to 0.40.0 which supports union type display name 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 8a4fba431e refactor(emqx_limiter_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi d603de10e6 refactor(schema): add namespace/0 to all schema modules 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 7b59d46854 fix(schema): add namesapce to sso schema 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 6b3aaf5b24 refactor(emqx_bridge_gcp_pubsub): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi b24b66081a refactor(authn/authz_http_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
ieQu1 2ade6da7a6 fix(ds): Remove redundand type 2023-11-10 12:53:30 +01:00
ieQu1 feef23fc08 feat(ds): Pass store_batch through RPC 2023-11-10 12:16:39 +01:00
ieQu1 8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
William Yang 0fc4bcaf0d
Merge pull request #11912 from qzhuyan/test/william/quic-multistream-flaky
test(quic): fix flaky
2023-11-10 11:13:35 +01:00
JianBo He d1623f491c
Merge pull request #11925 from lafirest/docs/gbt
chore(gbt32960): add EN documentation of the data exchange guide
2023-11-10 16:43:49 +08:00
Zaiming (Stone) Shi 3e2ce57cb1 refactor(emqx_bridge_greptimedb): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 46fffa401e refactor(emqx_prometheus_schema): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi ae24b45d1b fix(schema): add type namespce to emqx_dashboard_sso_ldap 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi a2700771c9 refactor(emqx_s3_schema): make type readable 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi b3df8604fc refactor(emqx_conf_schema): use hocon enum type for log levels 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi df96798c53 refactor(http_bridge): use more readable type for pool_type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 40c2ca134d refactor(schema): delete unused type cipher() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 7a0a0bee98 refactor(schema): change file() type to string() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi af3dfc91b1 refactor(schema): change log_lvel to enum type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 3d20e566d8 docs: add namespace to authn and authz schema 2023-11-10 08:45:20 +01:00
firest 0299e9033a chore(gbt32960): add EN documentation of the data exchange guide 2023-11-10 15:22:28 +08:00
Zaiming (Stone) Shi 9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109
fix(actions_api): change bridge v2 occurrences to action
2023-11-10 07:35:35 +01:00
firest 447c933ff8 fix(syskeeper): homogenize the Syskeeper bridge and update license data 2023-11-10 10:42:53 +08:00
ieQu1 38d004e06e
Merge pull request #11913 from ieQu1/dev/fix-hooks-comment
chore(hooks): Fix comment
2023-11-10 03:15:54 +01:00
firest 548e39e799 refactor(sysk): refactor the syskeeper to v2 style 2023-11-10 09:52:38 +08:00
firest abc7233a77 test(sysk): add test suite for syskeeper 2023-11-10 09:52:38 +08:00
firest e93216fa62 chore(sysk): fix spellchecks & update change & more logs 2023-11-10 09:52:32 +08:00
firest ce83079c6b feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend 2023-11-10 09:51:43 +08:00
JianBo He 6500d21d98
Merge pull request #11885 from HJianBo/ocpp-gw
feat: Port OCPP gateway from v4
2023-11-10 09:32:40 +08:00
ieQu1 ced7be61f0
Merge pull request #11917 from ieQu1/dev/fix-packet-id-to-seqno-trans
fix(ds): Fix packet id -> sequence number translation
2023-11-09 22:57:16 +01:00
Thales Macedo Garitezi 7a237f6a3a Merge branch 'master' into sync-m-to-r54-20231109 2023-11-09 18:49:52 -03:00
Thales Macedo Garitezi 371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
ieQu1 5aa9d026df fix(ds): Apply review remarks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-09 21:16:59 +01:00
ieQu1 b812db1e3c fix(ds): Fix packet id -> sequence number translation 2023-11-09 20:12:39 +01:00
Thales Macedo Garitezi 2d539463cd test: fix flaky test
```
 Testing lib.emqx.emqx_cm_SUITE: *** FAILED test case 24 of 31 ***
%%% emqx_cm_SUITE ==> t_stepdown_session_takeover_begin_timeout: FAILED
%%% emqx_cm_SUITE ==>
Failure/Error: ?assertEqual([<0.6635.8>,<0.6520.8>], lists : sort ( emqx_cm : lookup_channels ( ClientId ) ))
  expected: [<0.6635.8>,<0.6520.8>]
       got: [<0.6520.8>,<0.6635.8>]
      line: 309
```
2023-11-09 15:32:52 -03:00
Thales Macedo Garitezi ee87f90b0d
Merge pull request #11906 from thalesmg/ds-avoid-records-m-20231108
chore(ds): avoid using records in persistence / rpc
2023-11-09 14:25:27 -03:00
Thales Macedo Garitezi 7977ae95c0 fix(actions_api): change bridge v2 occurrences to action
Fixes https://emqx.atlassian.net/browse/EMQX-11338
2023-11-09 14:20:08 -03:00
Thales Macedo Garitezi eabd09051a feat(actions_api): add `/action_types` API 2023-11-09 13:54:57 -03:00
ieQu1 42a9f0eb6b
Merge pull request #11914 from ieQu1/dev/speed-up-poll
fix(ds): Speed up polling for the new messages
2023-11-09 16:40:06 +01:00
Thales Macedo Garitezi 42536bb5f7 chore(ds): avoid using records in persistence / rpc
Fixes https://emqx.atlassian.net/browse/EMQX-11279
2023-11-09 12:00:25 -03:00
ieQu1 cf4a46a78b fix(ds): Speed up polling for the new messages
Poll immediately if the previous poll returned non-empty result
2023-11-09 14:22:32 +01:00
ieQu1 3350bda446 chore(hooks): Fix comment 2023-11-09 14:05:13 +01:00
firest 2b97800c8b fix(limiter): simplify the descriptions for fields of the limiter 2023-11-09 18:00:09 +08:00
William Yang f7a2359329 test(quic): fix flaky 2023-11-09 10:56:16 +01:00
JianBo He d55f1e0813 chore: fix mix compiling failures 2023-11-09 16:54:41 +08:00
Shawn e0f4a2014d fix: emqx failed to start if plugin dir is removed by the user 2023-11-09 16:01:20 +08:00
Ivan Dyachkov 0bf5deaed9 chore: merge master into release-54 2023-11-09 08:32:26 +01:00
JianBo He d6104b37cf test: fix flaky tests 2023-11-09 15:01:12 +08:00
JianBo He 4b18631d49 chore: fix gateway failed test cases 2023-11-09 13:44:20 +08:00
JianBo He 2d958beae9 chore: try to fix spellcheck 2023-11-09 09:51:08 +08:00
Zaiming (Stone) Shi 42fa1289a1
Merge pull request #11905 from thalesmg/tag-e531-alpha5-r53-20231108
chore: prepare to tag `e5.3.1-alpha.5`
2023-11-08 23:39:33 +01:00
Zaiming (Stone) Shi f95058a3e7
Merge pull request #11897 from zmstone/1107-fix-cluster-conf-sync-wait-loop
1107 fix cluster conf sync wait loop
2023-11-08 23:37:09 +01:00
Zaiming (Stone) Shi 1e3500ffd2
fix(emqx_conf_app): ensure log data fields
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-08 21:19:22 +01:00
Thales Macedo Garitezi 8540566eba chore: prepare to tag `e5.3.1-alpha.5` 2023-11-08 13:02:19 -03:00
Zaiming (Stone) Shi f9e9748cec fix(cluster-rpc): boot from local config if table loaded
When EMQX boots up, it tries to get latest config from peer (core type)
nodes, if none of the nodes are replying, the node will decide
to boot with local config (and replay the committed changes) if
the commit table is loaded from disk locally (an indication of the
data being latest), otherwise it will sleep for 1-2 seconds and
retry.

This lead to a race condition, e.g. in a two nodes cluster:

1. node1 boots up
2. node2 boots up and copy mnesia table from node1
3. node1 restart before node2 can sync cluster.hocon from it
4. node1 boots up and copy mnesia table from node2

Now that both node1 and node2 has the mnesia `load_node` pointing
to each other (i.e. not a local disk load).

Prior to this fix, the nodes would wait for each other in a dead loop.

This commit fixes the issue by allowing node to boot
with local config if it does not have a lagging.
2023-11-08 15:06:47 +01:00
ieQu1 788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session
Refactor emqx_durable storage application and introduce learned topic structure storage
2023-11-08 14:27:40 +01:00
Thales Macedo Garitezi f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-08 09:53:42 -03:00
Ilya Averyanov d061d64c70 fix(plugins): fix backward compatibility 2023-11-08 12:36:09 +03:00
Ivan Dyachkov 0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
ieQu1 a1cdbaa76d fix(ds): Address code review remarks 2023-11-08 04:31:27 +01:00
ieQu1 c030188eb7 chore(ds): Add rebar.config file to app/emqx_durable_storage 2023-11-08 03:04:19 +01:00
ieQu1 7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
ieQu1 74cb43f8b1 fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
Andrew Mayorov 54951c273f feat(ds): mix safe cutoff into keymapper filter during iteration 2023-11-08 03:04:19 +01:00
Andrew Mayorov 7a94db25c3 fix(ds): don't iterate over incomplete epoch in bitmask lts storage 2023-11-08 03:04:19 +01:00
Andrew Mayorov 46d8301bc0 feat(emqx): expose timestamp function in `emqx_message`
So that the code that relies on it would not need to guess clock source
and precision.
2023-11-08 03:04:19 +01:00
ieQu1 38b7f516bc fix(ds): Change the directory where message database is stored 2023-11-08 03:04:19 +01:00
ieQu1 8e5dda40be fix(ds): Fix static checks 2023-11-08 03:04:19 +01:00
ieQu1 99329e1243 refactor(ds): Address review remarks 2023-11-08 03:04:19 +01:00
ieQu1 2de79dd9ac feat(ds): Replay QoS1 messages 2023-11-08 03:04:19 +01:00
ieQu1 e745e42093 test(ds): Explore full range of keys when testing ratchet function 2023-11-08 03:04:19 +01:00
ieQu1 87689890ff chore(ds): Fix linter and compilation warnings 2023-11-08 03:04:19 +01:00
ieQu1 465e8a90dd revert(ds): Remove change from the old protocol file 2023-11-08 03:04:19 +01:00
ieQu1 ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
ieQu1 164ae9e94a feat(ds): LTS bitfield storage passes all tests 2023-11-08 03:04:19 +01:00
ieQu1 56b6b176c2 fix(ds): LTS shall keeps the concrete topic indexes 2023-11-08 03:04:19 +01:00
ieQu1 7428e7037b feat(ds): Bitfield + Learned Topic Structure 2023-11-08 03:04:19 +01:00
ieQu1 ac91dbc58f feat(ds): Restore LTS trie from a dump 2023-11-08 03:04:19 +01:00
ieQu1 f1ab7c8a7c feat(ds): Add persist callback to LTS trie 2023-11-08 03:04:19 +01:00
ieQu1 c149e0e2df fix(ds): Pass topic filter to emqx_ds:make_iterator call 2023-11-08 03:04:19 +01:00
ieQu1 51a6f623fd refactor(ds): Split out bitfield keymapper to a different module 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 903b3863d1 chore(ps_ds): make persistent session module use new `emqx_ds` APIs 2023-11-08 03:04:19 +01:00
ieQu1 2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1 6d65707d41 refactor(ds): Implement drop_db function 2023-11-08 03:04:19 +01:00
ieQu1 c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1 59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1 7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
ieQu1 c91df2f5cd refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
ieQu1 f1454bb57e feat(ds): learned topic structure 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 7ab57824dc chore(ds): change return type of `storage_layer:next/{1,2}`
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
Ilya Averyanov 75704513d8 fix(plugins): fix backward compatibility 2023-11-07 19:06:37 +03:00
Thales Macedo Garitezi c4c9d7ceb3 chore: prepara to tag `e5.3.1-alpha.4` 2023-11-07 13:04:46 -03:00
Thales Macedo Garitezi b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi 7dcdbc9e51 fix(resource): take error from action/connector before attempting query
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
Stefan Strigler fa740a1927 docs: fix inconsistencies for bridge and connector operations
It's only `start` allowed for now. Also fixed some wordings.
2023-11-07 13:58:23 +01:00
Zaiming (Stone) Shi f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config
Rename connector channel related configs in bridge_v2 to 'parameters'
2023-11-07 13:41:55 +01:00
Thales Macedo Garitezi 9742dae30d
Merge pull request #11889 from thalesmg/rename-bridgev2-to-action-apis-r53-20231106
chore: rename `bridges_v2` -> `actions` in the public facing APIs
2023-11-07 09:29:06 -03:00
JianBo He d76ed585b0 chore: correct the application vsn 2023-11-07 11:11:20 +08:00
JianBo He 7bd5579980 chore: fix the documentation generation and example conf checking 2023-11-07 11:09:39 +08:00
JianBo He 4ef156d69e chore: fix dialyzer warnings 2023-11-07 09:04:50 +08:00
JianBo He 0d9e0bd3fe chore: fix lot of running bugs 2023-11-07 08:52:10 +08:00
Zaiming (Stone) Shi bb8f80f256 test: add test case to pin bridge schema json version 0.1.0 2023-11-06 21:59:36 +01:00
Zaiming (Stone) Shi 9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi 2b8cf50a1d chore: rename `bridges_v2` -> `actions` in the public facing APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi 6af6309301 chore: upgrade to hocon 0.39.19 2023-11-06 19:16:19 +01:00
Thales Macedo Garitezi f17b762596 chore: don't disable rule that references non-existent bridge
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
2023-11-06 10:09:14 -03:00
JianBo He 6f51b9f842 chore: ensure elvis pass 2023-11-06 12:31:54 +08:00
JianBo He 8871738756 chore: ensure emqx_gateway_ocpp tests pass 2023-11-06 12:26:58 +08:00
JianBo He bea0acd929 fix(ocpp): ensure ocpp gateway options works 2023-11-06 11:56:50 +08:00
Zaiming (Stone) Shi 5881e34d4e
Merge pull request #11874 from zmstone/1103-validate-bridge-type-and-name-at-entry
fix(bridge): do not allow dot in bridge name
2023-11-03 23:04:10 +01:00
Thales Macedo Garitezi 0ff4465c78
Merge pull request #11875 from thalesmg/fix-kafka-connecting-r53-20231103
fix(kafka_producer): don't return `disconnected` when there are connections issues while starting the bridge
2023-11-03 17:56:00 -03:00
Thales Macedo Garitezi a064eecb68
Merge pull request #11879 from thalesmg/fix-rule-api-downgrade-bridge-r53-20231103
fix(rules_api): downgrade bridge id
2023-11-03 17:26:09 -03:00
Zaiming (Stone) Shi 21a167e1a5
Merge pull request #11873 from zmstone/1103-unhide-nodelay-field
fix(kafka): do not hide nodelay field
2023-11-03 20:49:29 +01:00
Zaiming (Stone) Shi 600747b7e5 fix(bridge): do not allow dot in bridge name
also validate name at the API entry
2023-11-03 20:44:57 +01:00
Thales Macedo Garitezi 0ee74cd8ae fix(rules_api): downgrade bridge id
Fixes https://emqx.atlassian.net/browse/EMQX-11312
2023-11-03 15:57:13 -03:00
Thales Macedo Garitezi 4265ef66cc fix(kafka_producer): don't return `disconnected` when there are connection issues while starting the bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixex https://emqx.atlassian.net/browse/EMQX-11298

We don't enforce the connection to be up when starting/creating the bridge, otherwise the
status will be `disconnected` for a possibly transient reason such as network issues or
Kafka broker restart.

Same applies for Azure Event Hub Producer bridge, as they share the same module.
2023-11-03 15:49:36 -03:00
Kjell Winblad dd8085d3a4 test(emqx_bridge_v2_api_SUITE): fix broken test case 2023-11-03 17:49:25 +01:00
Stefan Strigler 8a1b370a5e
Merge pull request #11876 from sstrigler/EMQX-11317-creating-a-bridge-for-a-non-existent-or-non-matching-connector-results-in-a-crash
handle errors from pre/post_config_update
2023-11-03 16:26:21 +01:00
Thales Macedo Garitezi 953d483c24
Merge pull request #11864 from thalesmg/test-enable-rule-check-deps-r53-20231101
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 11:01:38 -03:00
Stefan Strigler 45bfcea6ae
Merge pull request #11866 from sstrigler/EMQX-11294-e-5-3-1-swaggers-several-misdescriptions-of-ap-is
EMQX 11294 swaggers several misdescriptions of APIs
2023-11-03 14:55:46 +01:00
Stefan Strigler dc11008993 fix(emqx_bridge): handle errors from pre/post_config_update 2023-11-03 14:11:25 +01:00
Zaiming (Stone) Shi fcecd3e9d4 fix(kafka): do not hide nodelay field
otherwise:
it's returned in the GET response,
but it's not allowed in the POST request
2023-11-03 13:35:52 +01:00
Thales Macedo Garitezi c84e4a4187 fix(rule_engine): don't enable a rule that references non-existent bridge 2023-11-03 09:24:53 -03:00
Kjell Winblad 65ba0b9de4
Merge pull request #11869 from kjellwinblad/kjell/shared_con/start_feedback/EMQX-11304
fix(bridge_v2): start operation should return an error when unsuccessful
2023-11-03 13:16:32 +01:00
Zaiming (Stone) Shi 4dd054b0a2 test(emqx_bridge_kafka_impl_consumer_SUITE): fix flaky 2023-11-03 11:48:50 +01:00
Stefan Strigler dd11a30383 docs(emqx_bridge_azure_event_hub): remove timestamp from message template example 2023-11-03 11:18:54 +01:00
Stefan Strigler a89b0c8cf4 fix(emqx_bridge_azure_event_hub): make ssl mandatory (and give an example) 2023-11-03 11:18:38 +01:00
Kjell Winblad ebb5997a8c test(emqx_bridge_v2_api_SUITE): fix cluster test 2023-11-03 11:11:27 +01:00
JianBo He 7cab269e0b feat: port the ocpp gateway from version 4 2023-11-03 16:51:21 +08:00
Ivan Dyachkov fdbf439df8
Merge pull request #11859 from cisiqo/patch-1
fix content type returned by Stomp
2023-11-03 08:21:08 +01:00
Zaiming (Stone) Shi afe6f79a66 feat(bridge): add description field to bridge and connector 2023-11-03 06:53:42 +01:00
Kjell Winblad d14d0fbcb1 fix(bridge_v2 operations): better error message 2023-11-03 06:35:42 +01:00
Kjell Winblad 36640263ba fix(bridge_v2): start operation should return an error when unsuccessful
The bridge V2 HTTP API start operation should return a 400 error if the
start is unsuccessful.

The bridge V1 HTTP API compatibility layer for Bridge V2 should return a
400 error if the start or restart operation is unsuccessful.

This commit fixes the above and adds tests that checks this for the V2
HTTP API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11304
2023-11-02 18:28:22 +01:00
Andrew Mayorov 910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder
feat(tpl): split `emqx_placeholder` into a couple of modules
2023-11-02 22:50:05 +07:00
Andrew Mayorov 729c6edff6
chore(repub): simplify error handling in MQTT props templating 2023-11-02 20:13:32 +07:00
Andrew Mayorov f1847fe494
chore(tpl): drop unnecessary binding 2023-11-02 20:13:32 +07:00
Andrew Mayorov e521a9f5fc
fix(utils): denote `emqx_jsonish` follows access module behaviour
Defined in `emqx_template`.
2023-11-02 20:13:32 +07:00
Andrew Mayorov b5b6c3f8cc
fix(tpl): ensure full backward compat in basic connectors 2023-11-02 20:13:32 +07:00
Andrew Mayorov 02c1bd70b6
feat(tpl): factor out loose json concept into a separate module
Which is called `emqx_jsonish`. Also introduce an _access module_
abstraction to extract information from such data during rendering.
2023-11-02 20:13:31 +07:00
chengshq fbd27eda6a fix(Stomp): content type returned 2023-11-02 11:21:27 +00:00
Andrew Mayorov 69cfa740ea
fix(ruleeng): ensure full backward compatibility 2023-11-02 17:11:12 +07:00
Andrew Mayorov 75cc663786
chore(ruleeng): streamline application dependencies 2023-11-02 17:11:12 +07:00
Andrew Mayorov 8e4585d64f
chore: move template modules to `emqx_utils`
Even though most of the time these modules will be used by
connectors, there are exceptions (namely, `emqx_rule_engine`).
Besides, they are general enough to land there, more so given
that `emqx_placeholder` is already there.
2023-11-02 17:11:12 +07:00
Andrew Mayorov 343b679741
feat(tpl): make escaping mechanism more foolproof
Treat "${$}" as literal "$". This allows to template express
strings, for example, of the form "${some_var_value}" where
`some_var_value` is interpolated from bindings.
2023-11-02 17:11:11 +07:00
Andrew Mayorov f689d6c233
fix(tpl): ensure backward compat in `emqx_rule_engine`
Missing bindings in string templates will be rendered as "undefined",
as before. Rendering still assumes that missing binding with implicit
default (`undefined`) is an error.

This will also restore complete backward compat in `emqx_prometheus`.
2023-11-02 17:11:11 +07:00
Andrew Mayorov 1fcdfe991c
fix(tpl): add few missing `@doc`s + correct typespecs 2023-11-02 17:11:11 +07:00
Andrew Mayorov a9693eada7
fix(tpl): rename `trivial` -> `is_const`
This is clearer. Former naming was a bit misleading.
2023-11-02 17:11:11 +07:00
Andrew Mayorov 7bb995f0c6
feat(tpl): support `:n` SQL parameters 2023-11-02 17:11:11 +07:00
Andrew Mayorov dfb7faf6a8
fix(tpl): ensure rough backward compat in `emqx_prometheus`
But also emit warning when job name template renders with errors.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 49fba40ee7
fix(tpl): ensure backward compat with authz / authn templates
This commit leans heavy into discouraging the former approach where
only part of placeholders were interpolated, depending on `placeholders`
option.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 49f5325c67
feat(tpl): unify validations / errors var representations 2023-11-02 17:11:10 +07:00
Andrew Mayorov b812f9af5a
feat(tpl): use `emqx_connector_template` in `emqx_rule_engine` app 2023-11-02 17:11:10 +07:00
Andrew Mayorov e1bca5844f
feat(tpl): use `emqx_connector_template` in `emqx_prometheus` app 2023-11-02 17:11:10 +07:00
Andrew Mayorov 0538a77700
feat(tpl): use `emqx_connector_template` in `emqx_authn`, `emqx_authz`
This slightly changes semantics: now the attempt to create authenticator
with illegal bindings in templates will fail, instead of treating them
as literals. The runtime behaviour on the other hand should be the same.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 35902dc72d
feat(tpl): switch basic connectors to `emqx_connector_template`
Also avoid `filename:join/2` in HTTP connector since it's both OS specific
and an overkill.
2023-11-02 17:11:09 +07:00
Andrew Mayorov 28d55d72ca
feat(tpl): split `emqx_placeholder` into a couple of modules
Located under `emqx_connector` for now. The APIs of the new modules
are slightly different from the old ones. The new ones are more
explicit in terms of error handling. Also copy the according testsuite
from `emqx_plugin_libs` and enrich it for better coverage.
2023-11-02 17:11:09 +07:00
Zaiming (Stone) Shi 73dd2f0ffd chore: bump version to e5.3.1-alpha.3 2023-11-02 10:36:41 +01:00
Zaiming (Stone) Shi f5818a86e3
Merge pull request #11860 from thalesmg/fix-delete-shared-bridgev2-r53-20231101
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-02 10:30:47 +01:00
Zaiming (Stone) Shi a44f8eb583
Merge pull request #11861 from zmstone/1101-do-not-repeat-print-shell-warning
fix(shell): do not repeat print shell warnings
2023-11-02 10:28:56 +01:00
Zaiming (Stone) Shi 34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293
fix(bridge_v2 API): optional cascading delete operation
2023-11-02 10:28:43 +01:00
Kjell Winblad ec2d339355
Merge pull request #11857 from kjellwinblad/kjell/shared_con/EMQX-11270
fix(bridge_v2): channels should not be removed when status is connecting
2023-11-01 16:46:47 +01:00
Ivan Dyachkov 5b964ef415
Merge pull request #11858 from id/1101-emqx-11288-fix-kafka-replaq-dir-conflict
emqx 11288 fix kafka replaq dir conflict
2023-11-01 16:45:02 +01:00
Thales Macedo Garitezi 4bea65bf97 fix(bridge_v2_api): don't disable schema check 2023-11-01 12:44:40 -03:00
Ivan Dyachkov 2dd5061643 fix(kafka): kafka bridge replaq dir conflict 2023-11-01 15:44:28 +01:00
Kjell Winblad 357b664c8d fix(bridge_v2): more fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:54 +01:00
Kjell Winblad b06d05eaac test(bridge_v2): fix test case after new API 2023-11-01 15:27:54 +01:00
Kjell Winblad 96d6c6db49 test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change 2023-11-01 15:27:53 +01:00
Kjell Winblad edb1d37e67 chore(bridge_v2): make fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:53 +01:00
Kjell Winblad 95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
Kjell Winblad 1e935e9eb4 fix(bridge_v2 API): optional cascading delete operation
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
2023-11-01 15:03:18 +01:00
Zaiming (Stone) Shi c38725facd fix(shell): do not repeat print shell warnings 2023-11-01 14:44:18 +01:00
lafirest 17544dc410
Merge pull request #11852 from lafirest/feat/gbt_gw
feat(gbt32960): Port the GBT32960 gateway from v4
2023-11-01 21:03:18 +08:00
Thales Macedo Garitezi c03979a48a fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1 2023-11-01 09:51:26 -03:00
firest 6b50fdcf9e chore(gbt32960): update changes 2023-11-01 19:35:44 +08:00
firest 30a72f557f fix(gateway): improve gateway schema modules
1. enhances the gateway name as an enum
2. make the schema more flexible and extensible without some hardcode
2023-11-01 18:55:36 +08:00
Kjell Winblad 10b2894b6e chore(bridge_v2_api): remove misplaced handling code 2023-11-01 11:31:17 +01:00
Zaiming (Stone) Shi ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched
perf: no need to format event message if no bridge matched
2023-11-01 10:22:06 +01:00
Ivan Dyachkov aa8a6f2e26
Merge pull request #11853 from thalesmg/test-bridge-v1-compat-layer-r53-20231030
test(bridges): add bridge v1 compatibility layer test suite, and other fixes
2023-11-01 09:51:10 +01:00
Stefan Strigler 29683072a1 fix(emqx_connector): remove `stop` and `restart` operations 2023-11-01 09:18:50 +01:00
firest 5e314d4ef1 test(gbt32960): add test suites 2023-11-01 13:17:01 +08:00
Thales Macedo Garitezi 45a39d97c6 fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1
Fixes https://emqx.atlassian.net/browse/EMQX-11291
2023-10-31 16:39:47 -03:00
Andrew Mayorov 7092c75597
Merge pull request #11809 from keynslug/ft/EMQX-10808/file-secrets
feat(mqttbridge): support file-sourced secrets as passwords
2023-11-01 00:29:30 +07:00
Thales Macedo Garitezi b420b53075 fix(bridges_v2): check bridge name length before attempting atom conversion
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
Thales Macedo Garitezi 8eb822d898 test(bridges): add bridge v1 compatibility layer test suite 2023-10-31 13:21:01 -03:00
Zaiming (Stone) Shi db5cdb6742 perf: no need to format event message if no bridge matched 2023-10-31 16:48:10 +01:00
Stefan Strigler 3e9f48d10f fix: remove unneeded clause 2023-10-31 16:05:31 +01:00
Stefan Strigler 2a4eaf41ba docs: fix examples for PUT operations on bridge_v2 and connectors
Examples would show `type` and `name` properties in the request body, which is
not accepted by the schema.

Also fixes some minor inconsistencies in the example names of connectors and
bridges.
2023-10-31 15:10:11 +01:00
Thales Macedo Garitezi 84e78f5d2e fix(bridge_api): fix response status code for some operations 2023-10-31 10:40:56 -03:00
Thales Macedo Garitezi c90ca1ea53 fix(bridge_v1): always delete connector when deleting v1 bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11287
2023-10-31 10:40:06 -03:00
Andrew Mayorov d278486416
fix(secret): dedicate a specific loader module for file secrets
To make code employing `emqx_secret` easier to follow.
2023-10-31 19:26:04 +07:00
firest b35cd57034 fix(gbt32960): make elvis happy 2023-10-31 19:04:21 +08:00
firest 3ec811e828 feat(gbt32960): Port the GBT32960 gateway from v4 2023-10-31 18:25:43 +08:00
Ivan Dyachkov b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Stefan Strigler 39f49d425c test(emqx_bridge): fix test for bridge_v2_type_to_connector_type 2023-10-31 08:34:05 +01:00
Stefan Strigler 63606922f2 refactor(emqx_bridge): only one place to generate same error 2023-10-31 08:34:05 +01:00
Stefan Strigler b256241650 test(emqx_bridge_azure_event_hub): fix legacy v1 bridge API tests 2023-10-31 08:34:05 +01:00
Stefan Strigler 176bbe88bc fix: use `_producer` for AEH bridge type 2023-10-31 08:34:03 +01:00
Kjell Winblad aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import
EMQX-11272 fix kafka bridge v1 import
2023-10-31 07:58:54 +01:00
JianBo He 21e4f918aa
Merge pull request #11833 from JimMoen/fix-shared-sub-topic-or-subscription-searching
Fix shared sub topic or subscription searching
2023-10-31 08:16:36 +08:00
Kjell Winblad 9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271
fix: crash when listing non bridge_v1 compatible bridge_v2
2023-10-30 20:29:36 +01:00
Zaiming (Stone) Shi c07cf9051e
Merge pull request #11844 from zmstone/1030-downgrade-bridge-type-for-old-api
1030 downgrade bridge type for old api
2023-10-30 19:47:24 +01:00
Zaiming (Stone) Shi 124d79a1ca
Merge pull request #11843 from thalesmg/test-kafka-producer-nits-r53-20231030
test(kafka_producer): minor adjustments to test suite
2023-10-30 18:27:10 +01:00
Zaiming (Stone) Shi 350e8433ab fix(bridge_v2): downgrade bridge type for GET api results 2023-10-30 18:17:35 +01:00
Serge Tupchii 0562585c8f fix(emqx_mgmt_data_backup): implement importer modules ordering
`emqx_bridge_v2` depends on `emqx_connector`, so connectors must be imported first.
2023-10-30 19:14:58 +02:00
Serge Tupchii bc8c291820 fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2 2023-10-30 19:14:58 +02:00
Serge Tupchii d94193ac15 fix(emqx_connector_schema): add kafka alias for kafka_producer 2023-10-30 19:14:58 +02:00
Serge Tupchii 0935bb6225 fix(emqx_connector): fix badarity error 2023-10-30 17:52:39 +02:00
Serge Tupchii d1cd5dd817 fix(emqx_mgmt_data_backup): upgrade raw conf before validating and importing 2023-10-30 17:52:39 +02:00
Kjell Winblad 8fb426e57f fix: crash when listing non bridge_v1 compatible bridge_v2
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
2023-10-30 16:32:04 +01:00
Thales Macedo Garitezi a60b96c5fd test(kafka_producer): minor adjustments to test suite
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
Ilya Averyanov 3f6c09b195
Merge pull request #11780 from savonarola/1017-fix-pbkdf2-validation
fix(authn): fix pbkdf2 option validation
2023-10-30 16:37:37 +02:00
Stefan Strigler 5a6e55e7af fix(emqx_bridge_kafka): schema fixes for kafka_producer in bridge_v2 2023-10-30 14:59:06 +01:00
Zaiming (Stone) Shi f463eff02d chore: e5.3.1-alpha.2 2023-10-30 14:49:08 +01:00
Zaiming (Stone) Shi 0656b6be3c refactor(emqx_config): use dynamic callback to upgrade raw config 2023-10-30 14:49:08 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
zhongwencool e896957803
Merge pull request #11824 from zhongwencool/audit-log-reviewed
refactor: audit log review refactor
2023-10-30 17:51:15 +08:00
JimMoen 3a09fdc495
refactor: check match topic before do subscriptions query 2023-10-30 16:45:29 +08:00
JimMoen e9de7316b6
test: shared-sub topics/subscription api 2023-10-30 14:42:59 +08:00
JimMoen 814e22feb3
fix: topics/subscripton mgmt api searching 2023-10-30 14:42:47 +08:00
zhongwencool 0634ff61c0 fix: dialyzer warning 2023-10-30 09:57:30 +08:00
JimMoen a2015f37ae
Merge pull request #10976 from JimMoen/fix-shared-sub-unsub 2023-10-27 21:22:27 +08:00
lafirest e63602c4b0
Merge pull request #11811 from lafirest/fix/api_rbac
fix(rbac): for compatibility with old data schema, extend the existing field as an extra
2023-10-27 17:56:54 +08:00
JimMoen d563121284
refactor: move ?REDISPATCH_TO macro to emqx_mqtt.hrl 2023-10-27 17:54:14 +08:00
firest 4ba34f8f3e chore(rbac): fix CI errors & update change 2023-10-27 17:17:04 +08:00
JimMoen 3b5cc912e7
fix: add `redispatch_to` header to all msgs when deliver shared-sub
- to find correct SubOpts for shared-sub dispatch
- use previous key `redispatch_to` to ensure rolling upgrade compatibility
2023-10-27 16:15:23 +08:00
zhongwencool 22223dc536 fix: return 400 when audit log is disabled 2023-10-27 16:11:30 +08:00
zhongwencool 995948f0e8 refactor: remove seq from audit record 2023-10-27 14:07:19 +08:00