Commit Graph

9406 Commits

Author SHA1 Message Date
Andrew Mayorov a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently 2023-09-21 11:53:40 +04:00
firest 9a128517d8 fix(sso): use the correct way to mark the API as authorization-free 2023-09-21 15:52:05 +08:00
zhongwencool dfc14a39bc test: add emqx_management to telemetry SUITE 2023-09-21 14:15:06 +08:00
firest 2283db62ec fix(sso): add a API with authorization-free to list running backends 2023-09-21 11:52:30 +08:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool 9521e11d39
Merge pull request #11584 from zhongwencool/telemetry-error-on-windows
fix: telemetry report error on windows
2023-09-21 09:08:55 +08:00
JianBo He b924fb618a
Merge pull request #11639 from lafirest/test/sso
test(sso): add test case for the integration of SSO with LDAP
2023-09-21 09:01:13 +08:00
zhongwencool 1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify 2023-09-21 07:32:39 +08:00
Andrew Mayorov a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
Andrew Mayorov 3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy 2023-09-20 22:55:54 +04:00
Andrew Mayorov 21e82b9534
test(sessmem): make retry delivery testcase more involved 2023-09-20 22:55:54 +04:00
Andrew Mayorov 69889d14a3
fix(sessds): fix use of undefined types 2023-09-20 22:55:54 +04:00
Andrew Mayorov c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
Andrew Mayorov 9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
Andrew Mayorov 98706cd215
chore: ensure comments follow code style consistently 2023-09-20 22:55:25 +04:00
Andrew Mayorov 8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability 2023-09-20 22:55:25 +04:00
Andrew Mayorov 7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov 045d8b7f10
refactor(ds): reorder functions to improve readability 2023-09-20 22:55:24 +04:00
Andrew Mayorov 540ca6d60f
chore: drop few commented out and irrelevant pieces 2023-09-20 22:55:24 +04:00
Andrew Mayorov 9d145890cc
refactor(sessmem): pass log context as part of session event
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov e422f492ef
test(sessds): reuse and expand persistent session test suite 2023-09-20 22:55:24 +04:00
Andrew Mayorov 3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling 2023-09-20 22:55:23 +04:00
Andrew Mayorov 7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Andrew Mayorov e713fc38aa
feat(broker): reflect persisted messages in publish result
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
Andrew Mayorov f4953e719b
fix(cmproto): fix few typespecs 2023-09-20 22:53:45 +04:00
Andrew Mayorov 8af107e28d
test(ds): simplify cluster test setups 2023-09-20 22:53:45 +04:00
Andrew Mayorov 7c4f68dd3d
fix(session): make utility function names consistent
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
Andrew Mayorov 2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem` 2023-09-20 22:53:44 +04:00
Andrew Mayorov abeff0bc4f
chore(session): try to describe what happens after session takeover 2023-09-20 22:53:44 +04:00
Andrew Mayorov 45d44df11d
refactor(session): update eviction channel session logic
The changes partially reflect `emqx_channel` changes with respect to
in-memory session specific logic. The difference is that eviction
channel does not replay post-takeover, instead enqueues messages.
2023-09-20 22:53:44 +04:00
Andrew Mayorov ab1c4c4222
refactor(session): drop `lookup/1` session API
Due to the fact it's not used for anything right now.
2023-09-20 22:53:44 +04:00
Andrew Mayorov 97881ff3ca
refactor(session): bring back common session timers 2023-09-20 22:53:43 +04:00
Andrew Mayorov bf16417513
feat(session): introduce session implementation concept 2023-09-20 22:53:42 +04:00
Andrew Mayorov 780ca15298
chore: bump application versions 2023-09-20 22:52:14 +04:00
Andrew Mayorov 04731b7ef7
test(takeover): randomize messages for random natural ordering 2023-09-20 22:52:14 +04:00
Andrew Mayorov f022c9b1a4
feat(emqx): add `emqx_inflight:fold/3` generic function 2023-09-20 22:52:14 +04:00
Andrew Mayorov 596ce157fd
refactor(exproto): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov 57ae5b14f1
refactor(mqttsn): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov e4866adc2f
refactor(chan): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov cfb1bf1fa4
chore(emqx): drop remnants of former session persistence impl
1. It is not functional anyway.
2. It blocks `emqx_session` refactoring in a few places.
2023-09-20 22:52:13 +04:00
William Yang e43b87f18e test(quic): update emqx_broker_SUITE 2023-09-20 20:20:18 +02:00
Thales Macedo Garitezi 2965fa6fcb
Merge pull request #11613 from thalesmg/ds-keyspace-m-20230914
feat(ds): introduce keyspace concept
2023-09-20 14:21:27 -03:00
William Yang da626f6d29 feat(quic): bump to quicer 0.0.200 and emqtt 1.9.0 2023-09-20 16:43:15 +02:00
zhongwencool dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool 9e75ff88af
chore: apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:15:39 +08:00
zhongwencool f789a7504f
chore: update apps/emqx/src/emqx_quic_connection.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:00:44 +08:00
Ivan Dyachkov 09aea47f40 chore: 5.2.1 2023-09-20 14:04:28 +02:00
firest 21c37a3f14 fix(sso): adjust the schema of LDAP API && add more logs 2023-09-20 18:15:42 +08:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
zhongwencool c26a18e949 fix: always return ok when remove local resource 2023-09-20 18:02:42 +08:00
firest b11c5d26ce test(sso): add test case for the integration of SSO with LDAP 2023-09-20 16:25:19 +08:00
lafirest bc6edac45f
Merge pull request #11631 from lafirest/feat/dashboard_ldap
feat(dashboard): add SSO feature and integrate with LDAP
2023-09-20 11:40:42 +08:00
firest 61311081d8 fix(sso): fix RBAC test errors && spellcheck 2023-09-20 10:42:17 +08:00
Thales Macedo Garitezi 5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
firest 5b00998050 fix(sso): fix invalid_dynamic_call 2023-09-19 22:47:48 +08:00
firest 285e529766 fix(sso): fix sso errors found when manual test 2023-09-19 19:15:52 +08:00
Zaiming (Stone) Shi 65818740f3
Merge pull request #11622 from zmstone/0917-upgrade-gen_rpc-to-3.1.0
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-19 10:22:53 +02:00
Ivan Dyachkov a69c26b86d chore: e5.2.1-alpha.3 2023-09-19 08:17:29 +02:00
firest 2cddce5479 feat(dashboard): add SSO feature and integrate with LDAP 2023-09-19 14:10:29 +08:00
JianBo He de1bc04843 chore: bump ekka to 0.15.14, mria to 0.6.3
Inclued ekka's updates:
- https://github.com/emqx/ekka/pull/214
- https://github.com/emqx/ekka/pull/215
- https://github.com/emqx/ekka/pull/216
- https://github.com/emqx/ekka/pull/217

mria's updates:
- https://github.com/emqx/mria/pull/160
- https://github.com/emqx/mria/pull/161
- https://github.com/emqx/mria/pull/162
2023-09-19 09:43:19 +08:00
Thales Macedo Garitezi a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Zaiming (Stone) Shi 9f8a7a96af
Merge pull request #11628 from zmstone/0918-upgrade-emqtt-to-1.8.7
0918 upgrade emqtt to 1.8.7
2023-09-18 21:56:04 +02:00
Zaiming (Stone) Shi de93d71014 test: fix emqx_sahred_sub_SUITE:t_remote
prior to this fix, the peer ct node is actually node itself
2023-09-18 21:54:43 +02:00
Zaiming (Stone) Shi 99b60849b4 test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer 2023-09-18 20:44:38 +02:00
SergeTupchiy 0b0e322473
Merge pull request #11627 from SergeTupchiy/EMQX-10962-fix_hstreamdb-bridge-on-stop
fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
2023-09-18 19:57:07 +03:00
Zaiming (Stone) Shi d1e5b69eac test: increase wait time 2023-09-18 17:33:09 +02:00
Zaiming (Stone) Shi 4f1ec795eb test: delete special handling of emqtt race condition
the race should have been fixed in emqtt 1.8.7
2023-09-18 17:27:34 +02:00
Zaiming (Stone) Shi 46557c3462 fix(emqtt): upgrade from 1.8.6 to 1.8.7 2023-09-18 17:04:23 +02:00
Serge Tupchii 80cf60e9be fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
fixes: EMQX-10962
2023-09-18 17:48:05 +03:00
Zaiming (Stone) Shi 45f0d0dfce
Merge pull request #11623 from zmstone/0917-upgrade-esockd-to-5.9.7
0917 upgrade esockd to 5.9.7
2023-09-18 16:30:54 +02:00
zhongwencool c868264f65
Merge pull request #11617 from zhongwencool/rule-engine-console-args
fix: keep builtin_console actions not complain args
2023-09-18 20:40:02 +08:00
zhongwencool 8b6d5d4ca9 test: add test for build_console's args 2023-09-18 16:48:38 +08:00
zhongwencool 5490807b20
chore: update apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-18 16:11:39 +08:00
Zaiming (Stone) Shi f66d9e76fe fix(emqx_channel): aovid logging frame errors as exceptions 2023-09-18 09:04:00 +02:00
lafirest 6fe846bf0e
Merge pull request #11610 from lafirest/feat/rbac
feat(dashboard): add RBAC feature for Dashboard
2023-09-18 14:41:47 +08:00
firest ca6547c05b fix(rbac): change default role and update changes 2023-09-18 11:40:35 +08:00
Zaiming (Stone) Shi abbee4bc47 fix(esockd): upgrade esockd from 5.9.6 to 5.9.7 2023-09-17 19:44:04 +02:00
Zaiming (Stone) Shi 985f0e6994 chore(gen_rpc): upgrade from 2.8.1 to 3.1.0 2023-09-17 19:10:13 +02:00
Zaiming (Stone) Shi a9682c2989
Merge pull request #11605 from zmstone/0913-lower-severity-level-for-CMD-override-warnings
0913 lower severity level for cmd override warnings
2023-09-16 11:32:07 +02:00
Zaiming (Stone) Shi ac41f497b4 feat: add bytesize rule sql function 2023-09-15 16:04:21 +02:00
Thales Macedo Garitezi 2aae3856a3
Merge pull request #11596 from thalesmg/fix-typespec-typos-m-20230912
fix: fix typespec typos
2023-09-15 09:24:42 -03:00
Ilya Averyanov f0d2b2e6c3 chore: update emqx_eviction_agent version 2023-09-15 13:24:38 +03:00
firest 4b97d3f57d test(rbac): add test cases for RBAC && update changes 2023-09-15 18:16:28 +08:00
Ilya Averyanov 1959e15703 chore: simplify session eviction for node rebalance 2023-09-15 11:46:51 +02:00
zhongwencool 32d838c85b fix: format console action crash 2023-09-15 17:40:56 +08:00
zhongwencool 640dd130fa fix: don't urldecode twice clientid 2023-09-15 17:36:48 +08:00
zhongwencool d72348c8ed fix: keep builtin_console actions not complain args 2023-09-15 16:38:04 +08:00
firest 45ccc66474 fix(dashbaord): fix test cases to support RBAC 2023-09-15 15:49:37 +08:00
lafirest aace9215a4
Merge pull request #11608 from lafirest/feat/ldap_bind
feat(ldap): integrate authentication with LDAP bind operation
2023-09-15 14:06:39 +08:00
firest 0846939760 fix(ldap): remove unused code and mark sensitive field 2023-09-15 10:13:07 +08:00
Thales Macedo Garitezi b30bcf32bd feat(ds): introduce keyspace concept
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
Ilya Averyanov 11c39c4b6a chore: simplify session eviction for node rebalance 2023-09-14 18:28:25 +03:00
Thales Macedo Garitezi 6be0e14e8c fix: fix typespec typos 2023-09-14 10:21:05 -03:00
Thales Macedo Garitezi e41f7dd68c
Merge pull request #11598 from thalesmg/ds-fix-ps-router-m-20230912
fixes and improvements to persistent session router
2023-09-14 10:00:58 -03:00
firest 5c31c5ce76 feat(dashboard): add RBAC feature for Dashboard 2023-09-14 19:28:37 +08:00
firest afbf13b8a2 feat(ldap): integrate authentication with LDAP bind operation 2023-09-14 14:10:43 +08:00
firest 3eb46be2ae fix(banned): fix typos 2023-09-14 14:01:59 +08:00
Zaiming (Stone) Shi 95f1312df7 chore(emqx_ctl): lower command override log from warning to info
prior to this change this warning message is always logged if a node
joins cluster.

the intention of this warning is to detect potential command name
clash (provided by different plugins), so arguably it should be
at debug level
2023-09-13 20:52:08 +02:00
Thales Macedo Garitezi 803b69d878
refactor: use more consistent return types
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2023-09-13 10:49:45 -03:00
Thales Macedo Garitezi f55f6dc779 docs(ps_session): add notes explaining rationale about operation order and consistency 2023-09-13 10:12:27 -03:00
Thales Macedo Garitezi e081abb9e6 perf(ps_router): only check if there are any routes when deciding whether to persist message 2023-09-13 10:02:39 -03:00
Thales Macedo Garitezi cae9ae1fab fix(ps_router): use `disc_copies` for storing persistent session routes 2023-09-13 10:02:39 -03:00
William Yang ea73d674b9 chore: bump version to e5.2.1-alpha.2 2023-09-13 14:55:17 +02:00
ieQu1 69d17f981d
Merge pull request #11595 from ieQu1/dev/dont-prevent-overlapping-partitions
fix(distribution): Set prevent_overlapping_partitions to false
2023-09-13 14:22:34 +02:00
Serge Tupchii e1456c1c04 feat(emqx_conf_schema): add node.default_bootstrap_batch_size config parameter 2023-09-13 10:11:14 +03:00
Serge Tupchii af015bc2bb chore(ekka): bump version to 0.15.13 2023-09-13 10:11:14 +03:00
ieQu1 b41310af9c
fix: Add prevent_overlappiing_partitions to vm_args
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-12 20:01:56 +02:00
ieQu1 1a86c21e20 fix(distribution): Set prevent_overlapping_partitions to false 2023-09-12 17:41:51 +02:00
William Yang 6ef296268e chore: bump version to e5.2.1-alpha.1 2023-09-12 15:24:02 +02:00
Thales Macedo Garitezi 59b94788bf refactor: address review comments 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi 64fcdb671d test: fix a bunch of inter-suite interferences 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi f1294736b7 feat(ds): add filter for message persistence
Fixes https://emqx.atlassian.net/browse/EMQX-10520
2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi fe4640922d feat(ds): add delete callback 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi 7805cc8c9b fix(session): check if session is persistent 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi f490da71cd chore: prepare v5.2.0-build.1 2023-09-11 16:09:17 -03:00
Thales Macedo Garitezi d0f1c75651 chore: bump apps vsns 2023-09-11 16:06:53 -03:00
Thales Macedo Garitezi 223d47a6ac Merge branch 'master' into sync-m-r52-20230911 2023-09-11 16:04:28 -03:00
Thales Macedo Garitezi 063913f245 fix(session): remove recently added `iterators` field from `#session{}` record
Fixes https://emqx.atlassian.net/browse/EMQX-10945

This could lead to `badrecord` errors being raised if a takeover were to happen during a
rolling cluster upgrade, as the old nodes could receive a record with more fields than
expected.
2023-09-11 14:46:12 -03:00
zhongwencool 887a74a6cb fix: telemetry report error on windows 2023-09-11 16:48:30 +08:00
Zaiming (Stone) Shi 7010ac358c chore: bump release version to v5.2.0 2023-09-08 16:31:52 +02:00
Andrew Mayorov 3d133801af
fix(router): use `ets:new/2` options compatible with OTP 24
While essentially keeping the same effect.
2023-09-08 18:04:29 +04:00
Thales Macedo Garitezi e089fda260
Merge pull request #11568 from thalesmg/republish-props-m-20230905
feat(republish): allow templating mqtt properties
2023-09-07 16:43:00 -03:00
Zaiming (Stone) Shi f3bebb0ba2 Merge remote-tracking branch 'origin/master' into 0907-sync-master-to-release-52 2023-09-07 18:51:30 +02:00
Thales Macedo Garitezi 68293231b8 test: add cases for more complex placeholders 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 014bc64d3b chore: ensure general prop is a binary 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 02b8bbf76a chore: drop templating support for `Subscription-Identifier`
This is highly-dependent on the session state, as is `Topic-Alias`.
2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 47bd19dee4 chore: lower error log messages down to debug 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 080cb73da1 fix: handle badmap inside `emqx_placeholder:proc_tmpl` 2023-09-07 13:40:03 -03:00
Zaiming (Stone) Shi db2ec55aa0 chore: bump release version to official e5.2.0 2023-09-07 09:40:15 +02:00
Zaiming (Stone) Shi 7c2f87fabe test: merge broker and router boot modules 2023-09-06 21:36:16 +02:00
Thales Macedo Garitezi bb55b04d46 refactor: inline fn clauses, improve error logging 2023-09-06 13:54:40 -03:00
Thales Macedo Garitezi c9100c7591
refactor: improve logging information and descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-06 13:33:59 -03:00
Thales Macedo Garitezi 78c5a779d7 feat(republish): allow templating mqtt properties
Fixes https://emqx.atlassian.net/browse/EMQX-10912
2023-09-06 09:54:02 -03:00
Kjell Winblad 7fe704b6dd
Merge pull request #11563 from kjellwinblad/kjell/fix/release-52/EMQX-10775
fix: update gpb library to fix type error
2023-09-06 14:47:23 +02:00
Zaiming (Stone) Shi 18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Zaiming (Stone) Shi e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
Thales Macedo Garitezi 954b73a9aa
Merge pull request #11533 from thalesmg/ds-unsubscribe-m-20230828
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
2023-09-05 09:47:17 -03:00
ieQu1 01a128878f chore(mria): Bump mria to 0.6.1 2023-09-05 11:08:32 +02:00
Andrew Mayorov d26995a2aa
Merge pull request #11524 from keynslug/ft/EMQX-10713/unified-route-tab
feat(router): add routing storage schema with 2 tables
2023-09-05 12:38:00 +04:00
Kjell Winblad 2b2e84b83c fix: update gpb library to fix type error
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-09-05 09:52:17 +02:00
Thales Macedo Garitezi 23d63f5e01 refactor: fork clientid type for `emqx_ds:session_id/0` 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 60ae3c15c8 refactor: disregard impossible case 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 2cfe5ba057 fix(plugins_api): fix response types and test 2023-09-04 14:37:25 -03:00
Thales Macedo Garitezi 02c3d6a0a7 fix(data_import): rm duplicate import call and fix test 2023-09-04 14:19:59 -03:00
Thales Macedo Garitezi 73d4248334 test(cth_cluster): always allocate listener ports 2023-09-04 11:22:26 -03:00
Andrew Mayorov 9f0f218387
test(router): simplify cluster-related testcase in helper testsuite 2023-09-04 18:05:59 +04:00
Andrew Mayorov 7d26b7bc1a
fix(router): emit clearer error when conflicting schemas in use 2023-09-04 16:37:38 +04:00
Thales Macedo Garitezi d33ad8a46f feat: cluster purge (r5.2)
Port of https://github.com/emqx/emqx/pull/11447

targeting `release-52`

Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-09-04 09:15:28 -03:00
Andrew Mayorov 893f69617a
chore(test): drop obsolete TODOs 2023-09-04 14:40:49 +04:00
Andrew Mayorov f0a0c7d4b1
fix(cthsuite): avoid duplicate application env entries
Because `application:set_env/1` complains loudly if there are
duplicates.
2023-09-04 14:40:49 +04:00
Andrew Mayorov 5024304bf9
fix(router): wait for tables replicate before choosing schema vsn 2023-09-04 14:40:48 +04:00
Ivan Dyachkov 24230a64df chore: bump app versions 2023-09-04 11:39:21 +02:00
Ivan Dyachkov b2f57636bd Merge remote-tracking branch 'upstream/release-51' into 0904-sync-release-51 2023-09-04 11:34:22 +02:00
Kjell Winblad 52546ac50f
Merge pull request #11522 from kjellwinblad/kjell/fix/EMQX-10778
fix: bad error message when rule engine schema name is too long
2023-09-04 11:05:55 +02:00
Ilya Averyanov 240afecd69
Merge pull request #11487 from savonarola/0821-optimize-bcrypt
feat: reduce bcrypt rounds to a usable value
2023-09-04 10:12:40 +03:00
zhongwencool ce571d892e chore: ping opentelemetry v1.3.0-emqx 2023-09-04 09:34:09 +08:00
Serge Tupchii ed9afe3345 refactor(emqx_connector): use `hocon_schema:override/2` to make pgsql 'username' field required 2023-09-01 19:56:07 +03:00
Serge Tupchii 607705518b test(emqx_bridge_cassandra): add connector test case for Cassandra configured without authentication 2023-09-01 19:56:07 +03:00
Serge Tupchii f0c75d97e1 chore: upgrade emqx_http_lib to 0.5.3
0.5.3 release includes a fix to parse <host>:<port> URLs using the default http scheme.
2023-09-01 19:56:07 +03:00
Serge Tupchii c3b3460173 fix(emqx_oracle): make username a required field in emqx_oracle_schema 2023-09-01 19:56:07 +03:00
Andrew Mayorov 4ab5f8374b
chore(schema): mark routing storage schema readOnly
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-01 19:58:31 +04:00
Andrew Mayorov ebf13c90ea
test(router): make sort stable in e2e routing testcase 2023-09-01 19:48:56 +04:00
Ilya Averyanov 90156befb5 feat: improve bcrypt usability
* limit salt rounds to usable values
* update bcrypt library to enable concurrent bcrypt hash calculation
2023-09-01 17:36:11 +03:00
Serge Tupchii c54527857a fix(emqx_bridge_sqlserver): use empty password by default as it is not a required field 2023-09-01 16:57:35 +03:00
Serge Tupchii 1bf86250dd fix(emqx_bridge_cassandra): allow cassandra bridge without username/password
Cassandra can be used without credentials, if it is configured with AllowAllAuthenticator (default).
2023-09-01 16:57:35 +03:00
Andrew Mayorov 545f1c84a6
fix(cth): do not allocate ports for `emqx` app by default
This causes tricky and impressively hard to track side effects down
the line. Namely, loading `emqx_schema` _after_ `emqx_conf_schema`
(as part of cluster node startup sequence) leads to a couple of
schema root rewrites, because `emqx_schema` defines similar config
roots yet slightly differently (e.g. `authorization`).
2023-09-01 17:23:34 +04:00
Andrew Mayorov 0a879bffd1
test(router): add e2e testcase for cluster-wide routing
Which verifies that both router storage schemas work correctly.
2023-09-01 13:49:12 +04:00
Andrew Mayorov eb0385a28f
chore(router): separate test-only function exports 2023-09-01 13:49:11 +04:00
zhongwencool 3d859453f4
Merge pull request #11554 from lafirest/chore/exhook_prop
chore: shutdown reason will never be a UTF8 atom
2023-09-01 16:53:26 +08:00
Kjell Winblad 3bb65e6b05
Merge pull request #11542 from kjellwinblad/kjell/fix/EMQX-10775
fix: update gpb library to fix type error
2023-09-01 10:07:03 +02:00
firest 53940754b7 chore: shutdown reason will never be a UTF8 atom 2023-09-01 14:17:33 +08:00
Paulo Zulato 17fe406605
Merge pull request #11551 from paulozulato/refactor-connector-config-callback
refactor: remove BridgeName from connector_config callback
2023-08-31 17:05:49 -03:00
Thales Macedo Garitezi bf0f77b6dc
Merge pull request #11548 from thalesmg/fix-plugin-order-update-r52-20230830
fix(plugins): update plugin order on whole cluster
2023-08-31 16:18:28 -03:00
Paulo Zulato 7b607c3300 refactor: remove BridgeName from connector_config callback
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.

Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
2023-08-31 11:01:49 -03:00
Zaiming (Stone) Shi c20ba0572a chore: bump to 5.2.0-alpha.4 2023-08-31 15:47:39 +02:00
Zaiming (Stone) Shi 876d539336 Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table 2023-08-31 15:46:33 +02:00
Andrew Mayorov fb094e1d47
test(router): avoid testsuite collisions 2023-08-31 13:36:01 +04:00
Paulo Zulato 727fd296ee
Merge pull request #11546 from paulozulato/fix-kafka-producer-unique-name
fix(kafka): avoid producer name collision between Kafka and AEH bridges
2023-08-30 18:08:54 -03:00
Thales Macedo Garitezi 4dfbc859f9 fix(plugins): update plugin order on whole cluster
Fixes https://emqx.atlassian.net/browse/EMQX-10879
2023-08-30 16:39:52 -03:00
Paulo Zulato ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-30 14:48:10 -03:00
Andrew Mayorov 2da604dfa4
test(mgmt): simplify flaky `t_verify_imported_mnesia_tab_on_cluster`
And also rest of the test setup in the suite.
2023-08-30 20:24:18 +04:00
Andrew Mayorov b1defa29d7
feat(ruleeng): avoid storing whole rules in topic index
Because it doesn't really give any benefit, but wastes memory by
duplication.
2023-08-30 20:07:56 +04:00
Andrew Mayorov 6b9cb06334
fix(router): add / refine some comments and log messages
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-30 19:51:31 +04:00
Thales Macedo Garitezi 922ca5e141 feat(ds): close iterators when handling `UNSUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9742
2023-08-30 11:14:28 -03:00
Thales Macedo Garitezi 9c6dd30f44 feat(session): store iterator ids in session record 2023-08-30 11:13:55 -03:00
Paulo Zulato dfcede8794 fix: increment matched counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-30 10:52:53 -03:00
Andrew Mayorov f8923ab419
Merge pull request #11539 from keynslug/ft/cth-desc-workdir
feat(cthsuite): add function to determine workdir of testrun
2023-08-30 17:15:55 +04:00
Thales Macedo Garitezi 1cab687153
Merge pull request #11540 from thalesmg/fix-cert-path-utf8-r52-20230829
fix(bridge): validate bridge name before attempting to convert certificates
2023-08-30 09:45:55 -03:00
lafirest 8b3a9072f9
Merge pull request #11532 from lafirest/fix/frame_parse
fix(frame): improve some error reasons when parsing invalid packet
2023-08-30 19:10:45 +08:00
Kjell Winblad d911f7fbea fix: update gpb library to fix type error
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-08-30 11:12:33 +02:00
JianBo He 9fe4382d98
Merge pull request #11530 from JimMoen/chore-bump-hstreamdb-driver
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-30 14:37:17 +08:00
firest 804443ba40 chore: fix typos 2023-08-30 11:22:09 +08:00
JianBo He 06fad14d07
Merge pull request #11536 from zhongwencool/comma_separated_binary
chore: ciphers should allow empty space(comma_separated_binary)
2023-08-30 10:20:06 +08:00
Ilya Averyanov 2ef60db147
Merge pull request #11481 from savonarola/0817-simplify-async-responses
chore(ft): refactor async reply mechanism
2023-08-30 01:20:09 +03:00
Ilya Averyanov 54ac4a8527 chore(ft): tidy up the code according to the review 2023-08-29 23:21:36 +03:00
Thales Macedo Garitezi 0f297ffef4 fix(bridge): validate bridge name before attempting to convert certificates
Fixes https://emqx.atlassian.net/browse/EMQX-10865
2023-08-29 16:32:01 -03:00
Andrew Mayorov 0e770bdc95
test: switch `emqx_broker_SUITE` to use new cth tooling 2023-08-29 22:05:36 +04:00
Andrew Mayorov 3268093881
feat(cth): add module-level documenation 2023-08-29 22:05:36 +04:00
Andrew Mayorov f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
Andrew Mayorov 063d6200c8
feat(router): enable using 2 tables to store routes
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.

Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
2023-08-29 17:41:59 +04:00
Andrew Mayorov 270fd107b2
feat(triesearch): allow reusing filter tokenization result
As a kind of micro-optimization.
2023-08-29 17:31:26 +04:00
zhongwencool 501456efec
chore: more safe with iolist_to_binary
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-29 17:10:43 +08:00
zhongwencool 4e7ba5d35f chore: ciphers should allow empty space(comma_separated_binary) 2023-08-29 11:34:40 +08:00
firest 6722722522 fix(frame): improve some error reasons when parsing invalid packet 2023-08-29 10:53:20 +08:00
zhongwencool 4245a4d8c6 fix: authz clean-cache clientid always return not_found 2023-08-29 08:17:47 +08:00
Paulo Zulato cc3ba18734 fix: increment dropped message counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
Paulo Zulato 2c89be04d1
Merge pull request #11527 from paulozulato/fix-kafka-header-template-handling
fix(kafka): fix template processing for header
2023-08-28 18:55:33 -03:00
Paulo Zulato 0b86f04bae fix(kafka): fix template processing for header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
Ilya Averyanov 279895b8fd chore(ft): add read/write concurrency to emqx_ft_async_reply tabs 2023-08-28 22:46:30 +03:00
SergeTupchiy c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log
fix(emqx_trace): don't download empty trace log file
2023-08-28 22:17:00 +03:00
Serge Tupchii 0072749143 fix(emqx_trace): don't download empty trace log file
Closes: EMQX-10274
2023-08-28 19:25:55 +03:00
Thales Macedo Garitezi 5908b69353 chore: hide cluster purge from cli and api for now 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi f988de4ff4 feat(purge): clear delayed messages 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi 9bded07834 feat: cluster purge
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
Thales Macedo Garitezi dec21ffc95
Merge pull request #11362 from thalesmg/ds-handle-sub-20230725
feat(ds): open iterators when handling `SUBSCRIBE` packets
2023-08-28 10:21:28 -03:00
Andrew Mayorov 2d931a0512
test(evict): do not disable router module in cluster nodes 2023-08-28 14:07:46 +04:00
Andrew Mayorov 8d2ebdea7e
fix(router): generalize config option and make effects visible 2023-08-28 14:07:46 +04:00
Andrew Mayorov e85789306b
chore(router): drop test-only `has_routes/1`
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
Andrew Mayorov 5d51687dbf
feat(router): add unified routing table config option 2023-08-28 14:07:45 +04:00
Andrew Mayorov 33e5e1ba57
feat(router): add unified routing table 2023-08-28 14:07:44 +04:00
Andrew Mayorov 166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 84e40fb6fe
test(trie): add more involved route add + delete case
That shows how current local trie implementation breaks because
of lack of refcounting.
2023-08-28 13:52:59 +04:00
Andrew Mayorov dcb63440bc
refactor(mgmt): avoid dealing with router tab directly
Instead, contain all this behind `emqx_router` module interface.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 9eccfa0909
refactor(router): isolate cleanup logic in router module 2023-08-28 13:52:59 +04:00
JimMoen f3467f44b2
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1` 2023-08-28 17:28:50 +08:00
Ilya Averyanov 4488e9e591 chore(ft): stop hook chain when doing actual handling 2023-08-28 11:57:44 +03:00
Ilya Averyanov 39a48179ea chore(emqx_channel): use macros for reply construction 2023-08-28 11:57:44 +03:00
Ilya Averyanov b8cacd2833 chore(ft): add tests for async reply registry 2023-08-28 11:57:44 +03:00
Andrew Mayorov b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
zhongwencool 053b7fb94a test: add more test for conf_cli 2023-08-28 11:14:36 +08:00
zhongwencool d8be248a3d fix: improve the suggest msg for update conf failed 2023-08-28 08:53:23 +08:00
Ilya Averyanov b0d4a22aa8 chore(ft): refactor async reply mechanism 2023-08-27 21:54:23 +03:00
zhongwencool b874926de7
Merge pull request #11514 from zhongwencool/stop-load-application-before-reboot
fix: stop otel deps appication before reboot
2023-08-27 20:46:31 +08:00
Paulo Zulato 2bb415f897
Merge pull request #11526 from paulozulato/fix-validator-return-msg
fix: flatten error message on resource validator
2023-08-25 17:27:31 -03:00
Thales Macedo Garitezi 33a0048155 refactor: move logic to `ensure_iterator` 2023-08-25 15:24:53 -03:00
Paulo Zulato 84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
Paulo Zulato 42877e282d fix: flatten error message on resource validator
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
Thales Macedo Garitezi f15f59650d test: rm obselete workaround code 2023-08-25 13:49:33 -03:00
Paulo Zulato cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id
Fix Kafka/AEH ClientId uniqueness
2023-08-25 10:14:44 -03:00
Andrew Mayorov d5cff533e3
chore(topicidx): drop TODO comment 2023-08-25 16:44:03 +04:00
Andrew Mayorov ca59a87d47
chore(topicidx): refine example of wildcard compare 2023-08-25 16:44:03 +04:00
Zaiming (Stone) Shi 558402a68e
chore(topic_index): add topic validation 2023-08-25 16:44:03 +04:00
Andrew Mayorov 33ed53bb6a
refactor(topicidx): simplify `compare/3` further
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
2023-08-25 16:34:45 +04:00
Andrew Mayorov fc37d235c7
refactor(topicidx): simplify seek instructions
Which also avoids comparing filter tail repeatedly when evaluating
it.
2023-08-25 15:24:06 +04:00
Kjell Winblad c65db82b07 fix: bad error message when rule engine schema name is too long
Fixes:
https://emqx.atlassian.net/browse/EMQX-10778
2023-08-25 11:02:38 +02:00
zhongwencool c8af5a702c test: remove connection_closed from test 2023-08-25 15:53:35 +08:00
zhongwencool fc1738188e fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero 2023-08-25 15:18:34 +08:00
lafirest ba9cb0a1ed
Merge pull request #11512 from lafirest/fix/ldap_def_port
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-25 10:22:44 +08:00
zhongwencool 8ab6eb5638 test: start emqx_opentelemetry in emqx_machine SUITE 2023-08-25 10:09:00 +08:00
Andrew Mayorov cf45e80c71
feat(topicidx): iterate on trie search implementation
This improves matching performance and decreases GC pressure on
synthetic workloads.
2023-08-25 01:47:11 +04:00
Paulo Zulato 535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi d9a5a9ea2b
Merge pull request #11513 from thalesmg/kafka-fix-ts-template-r52-20230824
fix: use correct timestamp template for kafka and remove such fields from azure event hub producer
2023-08-24 16:26:52 -03:00
Thales Macedo Garitezi 016ae0524f fix(aeh_producer): remove timestamp template field
Fixes https://emqx.atlassian.net/browse/EMQX-10847

Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.

Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type

Example message consumed from AEH:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test0",
  "ts" => 1692879703006,
  "ts_type" => "append",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Note the ts_type above is append.

Example message from a Kafka broker whose ts type is create:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test-topic-three-partitions",
  "ts" => 1692881883668,
  "ts_type" => "create",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
2023-08-24 14:20:42 -03:00
Thales Macedo Garitezi 0381ac0410 fix(kafka_producer): use correct timestamp template field
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
Kjell Winblad 9fe5080705 build: bump versions before release 2023-08-24 16:46:27 +02:00
zhongwencool c215fe3736 fix: stop otel deps appication before reboot 2023-08-24 22:14:14 +08:00
Paulo Zulato 60e6217496 fix(kafka): fix result handling when sending message with invalid header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi e5041de9cc fix(gcp_consumer): handle 401 errors
Fixes https://emqx.atlassian.net/browse/EMQX-10852
2023-08-24 09:03:34 -03:00
Zaiming (Stone) Shi 62423b0b12 refactor(topic_index): less special handling for leading $ words 2023-08-24 13:30:10 +02:00
Zaiming (Stone) Shi a30d87e14f refactor(topic_index): remove more unnecessary next calls
also avoid using records (setelement) for recursive return values
2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi a1e6635614 refactor(topic_index): no forced ceiling entry in index table 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi f4c8c6be55 refactor(topic_index): optimize trie-search performance 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi 6b152b3cb7 test: add more debug output 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi ae094e363c chore: fix a typo in log message 2023-08-24 12:25:04 +02:00
firest 6085442958 fix(ldap): expose request_timeout to prevent infinite hang 2023-08-24 16:50:38 +08:00
zhongwencool 2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics
Prometheus metrics
2023-08-24 12:06:32 +08:00
zhongwencool 8d2be27ade chore: add changlog for 11466 2023-08-24 06:46:50 +08:00
zhongwencool 84dd3b83bb fix: hocon converter not working when make_serializable is true(check_plain) 2023-08-24 06:46:47 +08:00
zhongwencool f72ba6819f
Merge pull request #11498 from zhongwencool/fix-otel-stop-error-log
fix: stop otel deps application in terminate crashed
2023-08-24 06:40:24 +08:00
zhongwencool d287b736ec chore: add changelog for 11497 2023-08-23 17:02:30 +08:00
zhongwencool 58be7f709b fix: stop otel deps application in terminate crashed 2023-08-23 16:45:41 +08:00
zhongwencool 6bb15541c7 feat: rename olp metrics to overload_protection metrics 2023-08-23 15:25:54 +08:00
zhongwencool b9a97923c4 feat: add some authz/connection missing metrics 2023-08-23 14:47:53 +08:00
zhongwencool 141676185d feat: add authentication metrics to prometheus 2023-08-23 14:15:16 +08:00
zhongwencool 4928c69758 feat: add authorization metrics to prometheus 2023-08-23 14:11:59 +08:00
zhongwencool df9f8da178 feat: add overload_protection metrics to prometheus 2023-08-23 14:07:37 +08:00
zhongwencool 188a81913e test: add prometheus api test 2023-08-23 11:23:43 +08:00
zhongwencool ae10415fc3 feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security 2023-08-23 09:03:22 +08:00
Paulo Zulato 61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators
fix(kinesis): set validators for static constraints
2023-08-22 18:28:35 -03:00
Paulo Zulato 20b46091f9 fix(kinesis): set validators for static constraints
Fixes https://emqx.atlassian.net/browse/EMQX-10833
2023-08-22 14:42:44 -03:00
Thales Macedo Garitezi f007b44426 fix(data_import): rm duplicate import call and fix test 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi ee2897e5de test(refactor): move test to integration tests dir 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c1f49abad2 test: fix inter-suite flakiness 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi dad27091be test: rm custom option 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi dbfacae283 fix: reinstate transactions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 65085d012b refactor: rename fn 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 3344bfb0bd refactor: rm `emqx_ds_replay` 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c46b8de938 test: remove unused things, refactor some functions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8c73b06e1 docs: add comment about future test failure 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 6de0bbe76a test(refactor): always allocate listeners for emqx app 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi a15405a800 test: fix assertions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c74abe79d0 refactor: reduce arity 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 021755b82b refactor: rm iterators from DS `#session{}` record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 3239f5ac5b feat: rm unnecessary transactions, use separate table for iterator references 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 8eab389ae1 perf: avoid unnecessary transaction 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c28c6d1b7e fix: ensure iterator is opened 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 33ddbe80ad refactor: remove persistence leakeage from emqx_cm level 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8d7bb9a67 refactor: rename module 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 9463e271c0 feat(ds): open iterators when handling `SUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi ca697a4e14 fix: rename `emqx_ds{,_replay}:replay_id()` 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 45e2e687e5 fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication
Fixes https://emqx.atlassian.net/browse/EMQX-10843
2023-08-22 12:42:48 -03:00
Kjell Winblad 3ffbb7decf fix: HTTP API /api/v5/publish schema
The schema for the /api/v5/publish HTTP API endpoint was incorrect.
For 400 (Bad Request) error it cannot return a list but the incorrect
schema declared that the response could include a list.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10837
https://github.com/emqx/emqx/issues/11488
2023-08-22 15:21:20 +02:00
lafirest 7b8a599d17
Merge pull request #11486 from lafirest/fix/ldap_autn_wrong_pw
fix(ldap): Return correct reason code when password is wrong
2023-08-22 18:53:17 +08:00
lafirest f8f39bf223
Merge pull request #11490 from lafirest/fix/absent_pw
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 18:52:58 +08:00
firest e20b804e6b chore: update changes && test cases 2023-08-22 17:41:08 +08:00
firest 184e03387a fix(ldap): fix status detection and auto reconnecting errors 2023-08-22 17:06:00 +08:00
Kinple 663d03112e
Merge pull request #11476 from Kinplemelon/kinple/for-upgrade-dashboard
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-22 14:28:40 +08:00
firest b0c5b447a4 fix(authn): quickly return when the password is absent in password-based authentication 2023-08-22 11:49:32 +08:00
JimMoen 13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support
Hstreamdb tls support
2023-08-22 10:40:54 +08:00
firest 037b0a7599 fix(ldap): Return correct reason code when password is wrong 2023-08-22 09:42:04 +08:00
Paulo Zulato 4e4c93a758
Merge pull request #11469 from paulozulato/feat-redis-auth-username
Add support for AUTH with username on Redis
2023-08-21 16:44:21 -03:00
Thales Macedo Garitezi 74d4a8325e
Merge pull request #11470 from thalesmg/gcp-consumer-rm-worker-pool-size-r52-20230817
fix(gcp_consumer): remove unused resource_opts fields
2023-08-21 14:49:59 -03:00
Thales Macedo Garitezi bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816
fix(bridge): make dryrun health check timeout more malleable
2023-08-21 14:49:46 -03:00
Ivan Dyachkov 999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
Thales Macedo Garitezi 4163f06611
Merge pull request #11480 from thalesmg/rule-funcs-func-clause-r52-20230818
fix(rule_engine): capture function_clause errors
2023-08-21 11:16:25 -03:00
Kinplemelon 443d44aaa3 fix: set the `endpoint` in kinesis bridge as a required field 2023-08-21 11:49:57 +08:00
JimMoen f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
JimMoen 649647190e
test: new batch mechanism of hstreamdb_erl 2023-08-21 10:40:51 +08:00
JimMoen 5527edf442
chore: bump hstreamdb_erl to `0.4.3+v0.16.1` 2023-08-21 10:40:36 +08:00
Ilya Averyanov ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
Zaiming (Stone) Shi 01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7
chore: upgrade Kafka client wolff to 1.7.7
2023-08-20 10:14:13 +02:00
Paulo Zulato 05fdf52d8e fix(redis): set bridge as unhealthy on authentication error 2023-08-18 14:51:03 -03:00
Paulo Zulato fc35c74eab fix(redis): use eredis_secret to avoid password leaking 2023-08-18 14:51:03 -03:00
Paulo Zulato 2c458b62f5 feat(redis): add support for `username` on AUTH command
Fixes https://emqx.atlassian.net/browse/EMQX-9911
2023-08-18 14:50:06 -03:00
Thales Macedo Garitezi 66dec69d09 fix(rule_engine): capture function_clause errors
Fixes https://emqx.atlassian.net/browse/EMQX-10798
2023-08-18 13:51:19 -03:00
Thales Macedo Garitezi befc260555 fix(gcp_consumer): remove unused resource_opts fields
Fixes https://emqx.atlassian.net/browse/EMQX-10789
2023-08-17 17:02:02 -03:00
Thales Macedo Garitezi e48bfb52bb Merge branch 'release-52' into sync-r52-master-20230817 2023-08-17 17:01:04 -03:00
Paulo Zulato afffdbbaa1 fix(kinesis): replace default payload template
Fixes https://emqx.atlassian.net/browse/EMQX-10766
2023-08-17 17:00:14 -03:00
Ivan Dyachkov e2b9d905d7
Merge pull request #11465 from id/0718-e5.2.0-alpha.3 2023-08-17 20:59:13 +02:00
Andrew Mayorov 75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-17 20:43:09 +04:00
Ivan Dyachkov a0c2fe8cc1 chore: e5.2.0-alpha.3 2023-08-17 17:55:56 +02:00
Thales Macedo Garitezi ebecbd1545 fix(bridge): make dryrun health check timeout more malleable
Fixes https://emqx.atlassian.net/browse/EMQX-10773

- Makes the timeout for probing a bridge more malleable to account for differences between
  each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
  slowness/throttling.
2023-08-17 09:21:19 -03:00
Thales Macedo Garitezi e3572eb758
Merge pull request #11459 from thalesmg/kafka-health-check-interval-r52-20230816
feat(kafka): add option to configure health check interval
2023-08-17 09:04:30 -03:00
Thales Macedo Garitezi 029b461a13
Merge pull request #11448 from thalesmg/gcp-consumer-403-20230815
fix(gcp_consumer): handle 403 responses
2023-08-17 09:04:00 -03:00
ieQu1 7466024bca
Merge pull request #11458 from ieQu1/fix-excluded-apps-230916
fix(emqx_machine_boot): Fix excluded_apps
2023-08-17 11:36:23 +02:00
Ivan Dyachkov 7469222a17 Merge remote-tracking branch 'upstream/master' into 0817-e5.2.0-code-freeze 2023-08-17 08:16:04 +02:00
lafirest 5e448f5a02
Merge pull request #11446 from lafirest/refactor/calendar
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-17 07:40:44 +08:00
Andrew Mayorov 5d79823891
perf(topicidx): preserve next key on the stack to reuse later
This should further optimize the number of `ets:next/2` calls
required for a single match query.
2023-08-17 01:13:19 +04:00
Paulo Zulato 740680d54b
Merge pull request #11444 from paulozulato/fix-kinesis-msg-access-denied
fix(kinesis): return error message on access denied
2023-08-16 18:02:46 -03:00
Thales Macedo Garitezi f03733ba6b
Merge pull request #11460 from thalesmg/integration-tests-dir-20230816
ci: add `integration_test` dir to `emqx` application
2023-08-16 17:47:14 -03:00
Ilya Averyanov d17bcc94f6 chore(auth): update emqx_gcp_device tests 2023-08-16 22:37:42 +03:00
Paulo Zulato a64386ef82 fix(kinesis): return error message on access denied
Fixes https://emqx.atlassian.net/browse/EMQX-10764
2023-08-16 16:04:23 -03:00
Ilya Averyanov 3943830eae chore(auth): remove unnecessary atom conversions 2023-08-16 20:50:35 +03:00
Ilya Averyanov 0dae040047 chore(auth): restore injected schema key position 2023-08-16 20:50:35 +03:00
Ilya Averyanov db31e5f0d9 chore(auth): tidy up the code 2023-08-16 20:50:35 +03:00
Ilya Averyanov 67e06b3171 chore(auth): make schema injection be more universal 2023-08-16 20:41:05 +03:00
Ilya Averyanov abcdf18ca4 chore(auth): update ldap authn test 2023-08-16 20:41:05 +03:00
Ilya Averyanov f00f4568dc chore(auth): simplify schema injection 2023-08-16 20:41:05 +03:00