Commit Graph

9956 Commits

Author SHA1 Message Date
Ilya Averyanov 55218e2df2 fix(redis): start and load eredis app 2023-11-28 18:51:19 +03:00
Zaiming (Stone) Shi d93fdd8e9b
Merge pull request #12039 from zmstone/1128-sync-release-54
Sync release-54 to master
2023-11-28 15:52:49 +01:00
William Yang 64f6220cf5 feat(quic): bump to quicer-0.0.303 and emqtt 1.9.7 2023-11-28 14:43:47 +01:00
ieQu1 8a065ba6d0
Merge pull request #12027 from ieQu1/dev/ds-meta-persist-shard
fix(ds): Persist shard table
2023-11-28 14:33:57 +01:00
Thales Macedo Garitezi f5c4fb5860 fix(ds_session): take conservative estimate of `last_alive_at` when bumping
Addresses https://github.com/emqx/emqx/pull/12024#discussion_r1407432154
2023-11-28 09:58:03 -03:00
Thales Macedo Garitezi bd40b5c553 test(flaky): more adjustments 2023-11-28 09:48:40 -03:00
Ivan Dyachkov 2e11ab6a16 chore: 5.3.2-alpha.2 2023-11-28 13:46:28 +01:00
ieQu1 05e47254e2 fix(ds): Fixes related to the shards table 2023-11-28 12:59:09 +01:00
Zaiming (Stone) Shi 6c85e62d26 fix(schema): add namespaces 2023-11-28 12:07:42 +01:00
Zaiming (Stone) Shi 1b1cea24f3 Merge remote-tracking branch 'origin/release-54' into 1128-sync-release-54 2023-11-28 11:43:30 +01:00
Andrew Mayorov 34c9c022d0
chore(replayer): add comment describing what "until" means 2023-11-28 11:47:54 +03:00
JianBo He bd6e9503e6 fix(http): compose the url and path in correctly format 2023-11-28 16:15:46 +08:00
zhongwencool 6e6e68c902
Merge pull request #12037 from zhongwencool/license-watermark-check
fix: validate error when set license's watermark to 100%
2023-11-28 16:04:39 +08:00
zhongwencool 29bcdb9a4a fix: validate error when set license's watermark to 100% 2023-11-28 13:16:16 +08:00
JianBo He 891ecc179d chore: fix flaky tests 2023-11-28 10:34:44 +08:00
JianBo He c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
JianBo He cdb90ebe6b feat: rename webhook bridge to http bridge 2023-11-28 09:53:46 +08:00
JianBo He dc99651690 test(bridge): ensure almost test cases passed 2023-11-28 09:53:46 +08:00
JianBo He 8954450c0b chore: fix compile warnings 2023-11-28 09:53:46 +08:00
JianBo He 96af7a74e8 feat: impl the http bridge v2 2023-11-28 09:53:46 +08:00
JianBo He f9a1e747fd chore(http): break the bridge confs to connector and action parts 2023-11-28 09:53:46 +08:00
SergeTupchiy 3b1ae0f3df
Merge pull request #12017 from SergeTupchiy/EMQX-11319-data-backup-http-api
EMQX-11319 data backup http api
2023-11-27 19:23:54 +02:00
Thales Macedo Garitezi e3fbf6c958 test: attempting to stabilize more flaky tests 2023-11-27 14:12:48 -03:00
Thales Macedo Garitezi 26e59f9508
Merge pull request #12024 from thalesmg/ds-session-expiry-m-20231124
feat(ds): session expiry
2023-11-27 14:11:38 -03:00
Stefan Strigler ac370fed61
Merge pull request #12035 from sstrigler/EMQX-11382-a-new-field-should-be-added-to-the-interface-of-the-connector-to-return-the-actions-associated-with-it
feat(emqx_connector): add field 'actions' in API response
2023-11-27 17:39:44 +01:00
Zaiming (Stone) Shi d933d0b9e0
Merge pull request #12016 from zmstone/1122-load-license-file
1122 load license file
2023-11-27 16:43:51 +01:00
Stefan Strigler 70978df5bb fix: use parse_id 2023-11-27 16:32:15 +01:00
Serge Tupchii 89fdbf3e73 fix(emqx_mgmt_cli): remove duplicated word in the log message 2023-11-27 17:30:58 +02:00
Serge Tupchii 64ee29af81 feat(emqx_management): implement data backup API 2023-11-27 17:30:58 +02:00
Andrew Mayorov 86685bdce2
feat(sessds): use integer tags for pubrange types 2023-11-27 18:27:15 +03:00
Andrew Mayorov bb05281adb
refactor(sessds): add dedicated `#ds_pubrange.tracks` field
This slightly simplifies the replayer code.
2023-11-27 18:24:57 +03:00
Thales Macedo Garitezi d88deb9ceb feat(ds): add session timer to bump last alive at timestamp 2023-11-27 12:19:52 -03:00
Stefan Strigler 048f4724a9 feat(emqx_connector): add field 'actions' in API response
Also unify schemas, use emqx_connector_schema for the generic parts.
2023-11-27 16:09:17 +01:00
Thales Macedo Garitezi 09c4e40511 refactor(ds): rename `disconnected_at` to `last_alive_at`, add more assertions 2023-11-27 11:48:44 -03:00
Thales Macedo Garitezi f2dbddc315 test: attempting to stabilize more flaky tests 2023-11-27 11:36:32 -03:00
Thales Macedo Garitezi 6acdc6e432 feat: add `/actions/:id/metrics/`, `/actions/:id/metrics/reset` APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11381
2023-11-27 11:33:11 -03:00
Andrew Mayorov 923898eadf
chore(chan): leave a TODO note for PUBREC handler 2023-11-27 09:55:46 +03:00
Andrew Mayorov 41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs 2023-11-27 09:55:46 +03:00
Andrew Mayorov 46475fac66
feat(sessds): provide QoS2 message replay support 2023-11-27 09:55:45 +03:00
Andrew Mayorov ce59cb71bb
chore: bump emqtt to 1.9.6 2023-11-27 09:55:45 +03:00
zhongwencool b5a00ec6b2
Merge pull request #12023 from emqx/master
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
ieQu1 bd7a84fe3e revert(ds): Don't duplicate the clean start in session_ds 2023-11-26 19:18:59 +01:00
Kjell Winblad c85004b7ef chore: remove obsolete TODO 2023-11-24 21:22:27 +01:00
Kjell Winblad c6c1d886f0 fix: make pgsql action schema properly 2023-11-24 21:19:26 +01:00
Thales Macedo Garitezi 839f9dbedb feat(ds): session expiry
Fixes https://emqx.atlassian.net/browse/EMQX-11048
2023-11-24 16:31:54 -03:00
Kjell Winblad 66945dcc5c fix: address more comments from @thalesmg 2023-11-24 20:31:29 +01:00
Kjell Winblad 30e248061f
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-24 20:07:37 +01:00
Kjell Winblad f070d80b1a fix: swagger examples for PostgreSQL, Matrix and Timescale 2023-11-24 18:50:52 +01:00
Kjell Winblad d03674a505 fix: duplicate key in example 2023-11-24 17:59:56 +01:00
Kjell Winblad fc7bedb81a fix: remove duplicated entry 2023-11-24 17:37:58 +01:00
Kjell Winblad e920160805 fix: add enable and description fields to PostgreSQL connector 2023-11-24 17:33:39 +01:00
Kjell Winblad 3dca83c854 fix: all missing descriptions 2023-11-24 17:33:39 +01:00
Kjell Winblad c5e281b84b fix: emqx_auth_postgresql test suites 2023-11-24 17:33:39 +01:00
Kjell Winblad 6ef9c6fe4a fix: ops 2023-11-24 17:33:39 +01:00
Kjell Winblad f7296d549f fix: elvis problem 2023-11-24 17:33:39 +01:00
Kjell Winblad 64c015cf6f fix: unify the schema modules in emqx_bridge_pgsql 2023-11-24 17:33:39 +01:00
Kjell Winblad b8f510d956 fix: add env variables about action info 2023-11-24 17:33:39 +01:00
Kjell Winblad 2e3028a8f8 fix(emqx_postgresql): fix lifecycle test 2023-11-24 17:33:39 +01:00
Kjell Winblad dbe73c70b1 fix: dialyzer problem 2023-11-24 17:33:38 +01:00
Kjell Winblad d5b62eead0 feat: split pgsql, matrix and timescale into connector action
This commit splits the bridges pgsql, matrix and timescale into
connector and action.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
2023-11-24 17:33:35 +01:00
Stefan Strigler f8f8cf9f30
Merge pull request #11963 from sstrigler/EMQX-11156-bridge-v-2-mongo-db-support
EMQX 11156 bridge v2 mongo db support
2023-11-24 16:49:57 +01:00
zhongwencool 8f548f4cbb
Merge pull request #12020 from thalesmg/sync-r53-m-20231124
chore: sync `release-53` to `master`
2023-11-24 22:54:53 +08:00
Stefan Strigler c1ef773e77 fix: check for sane state after regular shutdown 2023-11-24 15:09:35 +01:00
Stefan Strigler 4e077c951b feat(emqx_bridge_mongodb): port mongodb to shared connector and actions 2023-11-24 15:09:35 +01:00
Thales Macedo Garitezi 261fe8a831 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 10:10:09 -03:00
ieQu1 2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start
Discard sessions when client connects with clean start = true
2023-11-24 13:35:39 +01:00
Thales Macedo Garitezi e95ec5b150 test: fix another flaky test 2023-11-24 09:24:21 -03:00
Thales Macedo Garitezi f8fd95c683 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 09:22:24 -03:00
Thales Macedo Garitezi 6a3c438194
Merge pull request #12009 from thalesmg/fix-kafka-get-bridges-r53-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-24 08:56:24 -03:00
Stefan Strigler 9ebbc9bbea refactor(emqx_bridge): use more simplistic function to validate connector 2023-11-24 10:07:11 +01:00
Stefan Strigler d2901afd1b fix(emqx_bridge_kafka): match example in api schema 2023-11-24 10:07:11 +01:00
ieQu1 e616e0746a feat(ds): Implement a function for dumping persistent session state 2023-11-24 04:39:46 +01:00
ieQu1 8dfcb69e52 feat(ds): Discard session when client connects with CleanStart=1 2023-11-24 04:17:28 +01:00
ieQu1 e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0
Support QoS0 messages in persistent_session_ds
2023-11-23 23:43:18 +01:00
ieQu1 449bafc27e fix(ds): LTS trie handles empty topic levels 2023-11-23 22:16:38 +01:00
ieQu1 a158f25a40 fix(ds): Fix return type of emqx_persistent_session_ds:publish 2023-11-23 22:16:38 +01:00
ieQu1 c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1 1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message
feat(ds): Don't store #message record in the DB
2023-11-23 18:56:45 +01:00
Stefan Strigler 1c11a02a76
Merge pull request #12010 from sstrigler/EMQX-11415-e-5-3-2-api-should-return-404-when-the-built-in-database-authentication-doesnt-exist
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
2023-11-23 18:08:48 +01:00
ieQu1 c69b82455e feat(ds): Don't store #message record in the DB 2023-11-23 17:56:35 +01:00
Zaiming (Stone) Shi 14077ec43b feat(license): allow setting 'default' license key 2023-11-23 17:52:17 +01:00
Zaiming (Stone) Shi caaf8113fc feat(license): support loading license from file 2023-11-23 17:52:17 +01:00
Thales Macedo Garitezi 4597e1c5b9
Merge pull request #12011 from thalesmg/fix-kafka-get-bridges-m-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-23 13:15:39 -03:00
ieQu1 1ced8786fd feat(ds): Make session poll interval configurable 2023-11-23 17:10:03 +01:00
ieQu1 82e74d0201 feat(ds): Add a flag that forces all sessions to become durable 2023-11-23 17:10:03 +01:00
Stefan Strigler 8ba116d378 fix(emqx_auth): check authenticator exists in /authenticator/:id/users 2023-11-23 16:15:03 +01:00
Ilya Averyanov ce77a12cd3
Merge pull request #12012 from savonarola/1123-fix-callback-types
fix(ds): impose more flexible types for ds implementations
2023-11-23 17:03:56 +02:00
Thales Macedo Garitezi fdfa3213cc fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 11:41:26 -03:00
Zaiming (Stone) Shi 7375bc5f9b fix(license): allow CRLF in license keys 2023-11-23 15:39:18 +01:00
Zaiming (Stone) Shi 48b69bd60c chore: delete a TODO from perfectionism 2023-11-23 15:39:18 +01:00
Ilya Averyanov 29d42506d5 fix(ds): impose more flexible types for ds implementations 2023-11-23 17:16:11 +03:00
Thales Macedo Garitezi 8e8d6d2192
Merge pull request #12006 from thalesmg/test-debug-flaky-test-r53-20231122
test(gcp_pubsub_consumer): fix flaky test
2023-11-23 10:57:22 -03:00
Thales Macedo Garitezi c89ec0b1f7 fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 10:25:36 -03:00
Thales Macedo Garitezi f3693e5dbc fix(gcp_pubsub_producer): add missing references to api specs 2023-11-23 09:02:52 -03:00
Zaiming (Stone) Shi 209077593d
Merge pull request #11970 from zmstone/1108-add-v2-schema-dump
1108 add v2 schema dump
2023-11-22 23:42:54 +01:00
Thales Macedo Garitezi db83457d13 test: fix flaky test
The cause was that the call `sys:terminate/2` was timing out...

`exit/2` doens't always work:

```
 2023-11-22 19:14:40.974
killed async workers

Error: -22T19:14:40.974563+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15908.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 28, reductions: 1822; neighbours:
Error: -22T19:14:40.998051+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15908.7>.
2023-11-22T19:15:41.088752+00:00 [critical] Run stage failed: error:{badmatch,{timeout,#{expected_remaining => 1,mailbox => {messages,[]},msgs_so_far => []}}}, Stacktrace: [{emqx_bridge_gcp_pubsub_consumer_SUITE,'-t_async_worker_death_mid_pull/1-fun-17-',3,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1576}]},{emqx_bridge_gcp_pubsub_consumer_SUITE,t_async_worker_death_mid_pull,1,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1505}]}], Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-11-22_19.14.27/snabbkaffe/1700680540975786370.log", mfa: undefined
Error: -22T19:15:46.095702+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15934.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 1471; neighbours:
Error: -22T19:15:46.095192+00:00 [error] Supervisor: {local,ehttpc_sup}. Context: shutdown_error. Reason: killed. Offender: id={ehttpc_pool_sup,<<98,114,105,100,103,101,58,103,99,112,95,112,117,98,115,117,98,95,99,111,110,115,117,109,101,114,58,116,95,97,115,121,110,99,95,119,111,114,107,101,114,95,100,101,97,116,104,95,109,105,100,95,112,117,108,108,45,53,55,54,52,54,48,55,53,50,51,48,51,52,50,50,55,53,49>>},pid=<0.15903.7>.
Error: -22T19:15:46.095470+00:00 [error] Supervisor: {<0.15906.7>,ehttpc_worker_sup}. Context: shutdown_error. Reason: killed. Offender: id={worker,1},pid=<0.15924.7>.
Error: -22T19:15:46.096762+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15934.7>.
Warning: 2T19:15:46.098278+00:00 [warning] msg: remove_local_resource_failed, mfa: emqx_resource:remove_local/1(362), error: {error,timeout}, resource_id: <<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>
Error: -22T19:15:46.149090+00:00 [error] Generic server <0.15904.7> terminating. Reason: killed. Last message: {'EXIT',<0.15903.7>,killed}. State: {state,<<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>,1,random}.
Error: -22T19:15:46.149525+00:00 [error] crasher: initial call: ehttpc_pool:init/1, pid: <0.15904.7>, registered_name: [], exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,909}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.15903.7>,ehttpc_sup,<0.15731.7>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 3428; neighbours:
```
2023-11-22 17:41:32 -03:00
Zaiming (Stone) Shi d9f964a44f test: fix test cases after schema type namespace change 2023-11-22 16:58:05 +01:00
Thales Macedo Garitezi fc849f0c05 ci(test): add info to help diagnose flaky test 2023-11-22 12:36:10 -03:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Zaiming (Stone) Shi e065217e9a
Merge pull request #11997 from zmstone/1121-do-not-allow-special-char-as-prefix-for-resource-name
1121 do not allow special char as prefix for resource name
2023-11-22 15:46:11 +01:00