Commit Graph

3982 Commits

Author SHA1 Message Date
Andrew Mayorov 0b3f5f7c37
feat(syncer): allow to turn syncer pool on/off through config 2024-01-12 11:57:24 +01:00
Andrew Mayorov 38e13f2337
fix(syncer): start syncer pool before broker pool
As the latter depends on the former.
2024-01-12 11:57:24 +01:00
Andrew Mayorov a1ccf85c66
test(routesync): verify that syncer preserves consistency
Under a highly concurrent load. Be aware that this testcase is not
deterministic.
2024-01-12 11:57:24 +01:00
Andrew Mayorov 5aeff20f8b
fix(routesync): ensure causal relationships are preserved
At the cost of strict FIFO semantics though.
2024-01-12 11:57:23 +01:00
Andrew Mayorov 54f8b47455
feat(routing): add route sync process pool
Dedicated to synchronizing local state updates with the global view
of the routing state.
2024-01-12 11:57:18 +01:00
JianBo He 9e85b53c39
Merge pull request #12296 from HJianBo/fix-delayed-message-http-api-500
fix(delayed): fix http 500 error
2024-01-12 09:37:41 +08:00
Ilya Averyanov 7b0b2a0527 fix(conn): avoid storing incomplete channel info 2024-01-11 18:08:13 +03:00
JianBo He 596607d549 chore: update bpapi.versions 2024-01-11 14:56:02 +08:00
Serge Tupchii 965ce5d446 perf: use mria:match_delete/2 to cleanup routes
This must be much more network efficient since both Mria and Mnesia
need only to replicate one op, e.g.: `{MatchPattern, clear_table}`
instead of replicating one per each key to be deleted.
2024-01-10 16:23:58 +02:00
Serge Tupchii febaaefc38 chore: upgrade ekka to 0.18.1 (mria 0.8.1)
Mria 0.8.1 release implements `mria:match_delete/2` function
2024-01-10 16:23:58 +02:00
Zaiming (Stone) Shi 7c8a36fc06
Merge pull request #12289 from zmstone/0108-support-acl-cache-masks
0108 support acl cache excludes
2024-01-10 13:40:29 +00:00
Zaiming (Stone) Shi 9e8a67fd68 feat: support authz cache exclusion config
now one can configure a list of topic-filters to avoid
caching ACL check results

for example

authorization.cache.excludes = ["nocache/#"]

this means ACL check results for topics having 'nocache/' prefix
will not be cached
2024-01-10 13:52:00 +01:00
Andrew Mayorov ff2fb95e5d
fix(conf): bump to hocon 0.40.4
Which includes a bugfix for undefined sensitive values obfuscation.
2024-01-10 13:09:08 +01:00
Zaiming (Stone) Shi 98e94d8619 docs: fix typo in comments 2024-01-09 19:16:55 +01:00
William Yang 2201dec2f4 feat(quic): 3s cutoff time for fast_close 2024-01-09 17:29:28 +01:00
William Yang 2483a4ecff feat(quic): support listener config reload 2024-01-09 17:29:25 +01:00
William Yang 973f40041d chore: bump 0.0.311 2024-01-09 09:35:03 +01:00
Ivan Dyachkov 06117c3a33 Merge remote-tracking branch 'upstream/release-54' into 0105-sync-r54 2024-01-05 14:20:38 +01:00
Ivan Dyachkov d04ac08d1d chore: release 5.4.1 2024-01-05 11:48:28 +01:00
Ivan Dyachkov d4c83cf463 chore: tag 5.4.1-alpha.1 2024-01-02 14:37:45 +01:00
Andrew Mayorov a87df28dfc
Merge pull request #12243 from keynslug/fix/EMQX-11483/route-async-repl
fix(router): rely on local sync state updates to propagate routes
2023-12-29 16:31:00 +01:00
Andrew Mayorov 5c91984ad7
chore: leave comment describing need for tighter synchronization 2023-12-29 14:04:48 +01:00
JimMoen 5e100f52b8
style: erlfmt all `rebar.config` files and `bin/nodetool` 2023-12-29 09:08:03 +08:00
JimMoen a3f33cc8bc
chore: fix typos in code and suite 2023-12-29 09:07:51 +08:00
JimMoen ed10827343
docs: README spell 2023-12-29 09:07:25 +08:00
Andrew Mayorov b9627c420f
fix(router): rely on local sync state updates to propagate routes
Instead of (potentially lagging) global table state itself. Local
state should be the only source of truth anyway. Moreover,
synchronize parts of local state update logic through the broker
pool to ensure consistency.

The problem was that `emqx_router:has_route/2` check may observe a
stale route, which deletion is not yet replicated from the core node
to the local replicant node.

For example:
1. The only one subscriber per a given topic A unsubscribes from
   a replicant node.
2. The route to A is deleted by the emqx_broker.
3. Mria makes RPC to a core node, it succeeds and returns.
4. The client resubscribes or another client subscribes to the same
   topic A.
5. emqx_broker tries to add a route again:
   `emqx_router:do_add_route(Topic)`.
6. `emqx_router` checks if the route is present.
7. The stale route is present because deletion not replicated yet,
   so no route is being added.
8. Route deletion is replicated locally but it's too late:
   we already have a local subscriber without a route.

Co-Authored-By: Serge Tupchii <serge.tupchii@protonmail.com>
2023-12-28 19:16:31 +01:00
Andrew Mayorov 331bfaa535
fix(broker): avoid reclaiming per-topic counter twice 2023-12-28 17:43:50 +01:00
Andrew Mayorov a8cd609ff4
chore(broker): simplify subscribe code path 2023-12-28 17:43:42 +01:00
Zaiming (Stone) Shi 4669c4d552 fix(bridge/mqtt): ensure short clientid
Some mqtt brokers do not allow long client IDs.
To make it compatible with this limitation, this commit
tries to limit the number of bytes under 23 with a best-effort
attempt to derive it from the bridge name.
2023-12-28 11:53:27 +01:00
Zaiming (Stone) Shi e421e9ce56 Merge remote-tracking branch 'origin/release-54' into 1222-sync-e5.4.0-build.2-to-master 2023-12-23 11:22:05 +01:00
Zaiming (Stone) Shi f36b842d94 chore: prepare for e5.4.0 retag 2023-12-23 11:18:19 +01:00
Zaiming (Stone) Shi 86a018d921 chore: bump app versions 2023-12-22 21:28:26 +01:00
Zaiming (Stone) Shi 9fdac4af0c Merge remote-tracking branch 'origin/master' into 1222-sync-e5.4.0-build.2-to-master 2023-12-22 21:27:27 +01:00
Zaiming (Stone) Shi ac6e4cf2cc fix: config update for ft 2023-12-22 19:01:14 +01:00
Zaiming (Stone) Shi 733196aa83 chore: prepare for e5.4.0-build.2 2023-12-22 18:59:46 +01:00
Zaiming (Stone) Shi 23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi 891fd972bd Merge remote-tracking branch 'origin/release-54' into 1221-prepare-bpapi-on-otp-26 2023-12-22 15:19:25 +01:00
Zaiming (Stone) Shi 5b5d4a6f68 chore: bump release version to e5.4.0-build.1 2023-12-22 14:00:12 +01:00
Zaiming (Stone) Shi 20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi 322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Zaiming (Stone) Shi ecc31217fd Merge remote-tracking branch 'origin/master' into 1221-prepare-bpapi-on-otp-26 2023-12-22 12:48:06 +01:00
Zaiming (Stone) Shi 17d3037a13 chore: add 5.4.bpapi 2023-12-22 12:47:41 +01:00
ieQu1 ec0693861a fix(static_checks): Prepare for OTP26 2023-12-22 12:02:30 +01:00
Thales Macedo Garitezi 035f5f977e Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231221 2023-12-21 17:53:05 -03:00
Zaiming (Stone) Shi 36d2929534 chore: prepare for release version 5.4.0 2023-12-21 19:51:12 +01:00
Thales Macedo Garitezi 18b69809da Merge branch 'release-54' into sync-r54-m-20231221 2023-12-21 10:05:13 -03:00
Thales Macedo Garitezi a43f49f8fa chore: prepare to tag `5.4.0-rc.1` 2023-12-20 12:43:08 -03:00
Andrew Mayorov 76c89ad372
test(gw-authz): switch to `emqx_cth_suite` + simplify testsuite
Also separate DTLS related MQTT-SN test config, so that it doesn't
leak into this test suite.
2023-12-20 14:37:49 +01:00
Andrew Mayorov 3d679f7e26
test(cth-tls): add test helpers for TLS related activities
Currently, only certificate issuing and related utilities are there.
2023-12-20 11:46:07 +01:00
Andrew Mayorov 78d3f49fe4
fix(gw): use more conservative set of DTLS options 2023-12-20 11:46:04 +01:00
Andrew Mayorov d3b32b64e1
Merge pull request #12201 from keynslug/feat/EMQX-11527/tls-hot-update
feat(listen): support hot config update of MQTT listeners
2023-12-20 09:45:12 +01:00
JianBo He cc15e3a03b
Merge pull request #12199 from zhongwencool/configs-put-bridge-v1-api
fix: support bridge v1 conf hocon-format in the put configs API.
2023-12-20 09:32:32 +08:00
Andrew Mayorov a6c26ce992
test(listen): stabilize flaky testcase 2023-12-19 23:36:52 +01:00
Andrew Mayorov baf46c9aa2
test(listen): also change CA on update and verify clients notice 2023-12-19 22:48:38 +01:00
Andrew Mayorov 7e4049620d
fix(listen): ensure limiter server state consistent with updates 2023-12-19 16:06:27 +01:00
Andrew Mayorov 4796f85dff
test(listen): verify SSL options of WSS listener are hot updateable 2023-12-19 16:06:27 +01:00
Andrew Mayorov 036048b9ab
feat(listen): support hot update of WS/WSS options as well
However, currently any transport option changes will cause listener to
restart, which slightly impacts the availability of the broker.
2023-12-19 16:06:27 +01:00
Andrew Mayorov 68bbbec556
chore: bump to esockd 5.11.1
With a typespec fix.
2023-12-19 16:06:27 +01:00
Andrew Mayorov fb2d4544b6
test(listen): verify that SSL options are hot-updateable 2023-12-19 16:06:26 +01:00
Andrew Mayorov da0f0f947e
feat(listen): support hot config update of esockd-based listeners 2023-12-19 16:06:26 +01:00
Zaiming (Stone) Shi 2b6176aa60 fix: upgrade to gen_rpc 3.3.1 which does not log cast args only arity 2023-12-19 12:46:42 +01:00
Andrew Mayorov cafd384466
chore: upgrade to esockd 5.11.0 2023-12-19 11:15:09 +01:00
zhongwencool a46b415c77 fix: support bridge v1 conf hocon-format in the put configs API. 2023-12-19 16:40:45 +08:00
Thales Macedo Garitezi cf9331a95f Merge branch 'release-54' into sync-r54-m-20231218 2023-12-18 17:21:08 -03:00
Andrew Mayorov 78f7d01b1c
test(listen): simplify test suite setup / teardown 2023-12-18 20:52:57 +01:00
Zaiming (Stone) Shi ef2cf71bea chore: bump version to 5.4.0-alpha.2 2023-12-18 13:51:21 +01:00
Zaiming (Stone) Shi a9963e043b refactor(authz): improve logging
Move authz result logging to common place.

Prior to this change, the final result is not logged when
fallback to the default authorization.no_match config value.

Aso, if the result is provided by a hook callback,
it's also not logged.

After this change, only the final result is logged.
The authz chain resutls can be traced (or logged at debug level).
2023-12-17 22:32:26 +01:00
Serge Tupchii 24db03a7df chore(emqx_opentelemetry): remove unused `event` functionality 2023-12-15 14:58:45 +02:00
Zaiming (Stone) Shi cf33b3b4b1 fix(emqx.app.src): add dependen apps in app.src 2023-12-15 10:40:33 +01:00
zhongwencool d9e179b364 fix: use microsecond precision as the primary key for audit logs to prevent duplication 2023-12-15 09:07:04 +08:00
Zaiming (Stone) Shi 406a2d7a52 test: fix shared sub test case flaky 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi fa85c3061b test: fix emqx_ocsp_cache_SUITE error message match pattern 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi 19051f639b test: ensure verify_none to ssl client opts as default value 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi 778662de5d test(crl_cache): fix ssl options 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi dca72e0052 test: fix mtls options 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi 06bfd878a0 test: fix compile warnings in test suites 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi a0833282f2 fix(dialyzer): fix dialyzer warnings for emqx app standalone 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi b7531839cd fix(dialyzer): delete stale code 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi bcae3d67c7
Merge pull request #12169 from ieQu1/dev/bpapi-separate-for-otp26
fix(static_checks): Prepare for OTP26
2023-12-14 20:32:45 +01:00
Zaiming (Stone) Shi c1f2287b86 Merge remote-tracking branch 'origin/release-54' 2023-12-14 15:26:49 +01:00
ieQu1 02e2231734 fix(static_checks): Prepare for OTP26 2023-12-14 15:22:09 +01:00
Thales Macedo Garitezi dbc8141930
Merge pull request #12125 from thalesmg/ds-cache-m-20231206
chore(ds): return DS key from `next` and add `update_iterator` callback
2023-12-14 10:23:25 -03:00
firest f1bde41bf6 fix(sso): Correctly handle wrapped passwords when updating 2023-12-14 10:07:04 +08:00
firest d278d3afb5 feat(ds): implemented the replayer scheduler 2023-12-13 23:52:02 +08:00
zhongwencool 1807df157c fix: license watermark percent support float 2023-12-12 17:27:31 +08:00
JimMoen 6552e0523a
Merge pull request #12141 from JimMoen/fix-shared-sub-api-internal-error
Fix shared sub api internal error
2023-12-11 21:43:09 +08:00
Zaiming (Stone) Shi b6c11be159
Merge pull request #12138 from zmstone/1208-fix-dialyzer
fix(dialyzer): batch 3
2023-12-11 12:52:56 +01:00
Zaiming (Stone) Shi d560366c14 test: fix some compile warnings 2023-12-11 09:43:13 +01:00
JimMoen 8c25371ff3
chore: bump gitignore for emqx standalone common test SUITES 2023-12-11 15:37:36 +08:00
JimMoen 68da627b4d
feat(channel): add peerport field in ClientInfo 2023-12-11 14:11:18 +08:00
Zaiming (Stone) Shi 50f4aba5cd fix(dialyzer): batch 3 2023-12-09 15:50:09 +01:00
Thales Macedo Garitezi 66d043becd feat(ds): introduce `update_iterator` callback 2023-12-08 15:04:18 -03:00
Andrew Mayorov 76970c268e
fix(sessds): stop overwriting QoS0-only pubrange checkpoints
Through making `StreamRef` part of a record key. Otherwise with
QoS0-only streams the seqno doesn't change, thus ranges over
different streams will have the same `FirstSeqno`.
2023-12-08 14:33:47 +01:00
Andrew Mayorov 24710d0f56
fix(sessds): avoid accumulating QoS0-only ranges in memory 2023-12-08 14:31:18 +01:00
Thales Macedo Garitezi f37c86afb9 Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231207 2023-12-07 13:30:04 -03:00
Thales Macedo Garitezi 2a6d72878f chore(ds): return DS message key along with batch 2023-12-07 11:36:08 -03:00
William Yang 499f4d4bc5 chore(quicer): dialyzer 2023-12-07 13:52:48 +01:00
William Yang 583645d97b chore: bump emqtt 1.10.1 and quicer 0.0.308 2023-12-07 13:52:44 +01:00
ieQu1 9b612cb0e7
Merge pull request #12119 from ieQu1/EMQX-10342
Make stream renew timer configurable
2023-12-07 10:42:23 +01:00
Zaiming (Stone) Shi 7560016f0e fix(hocon): upgrade hocon from 0.40.0 to 0.40.1
fixed type spec, exported some public types from emqx_schema module
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 6b553e37c0 chore: pin ekka 0.17.0 previously 0.15.16
change summary:
1. otp 26
2. type spec fixes
3. dropped UDP mcast clustering
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Zaiming (Stone) Shi 71f9838fd7 chore: upgrade to emqtt 1.10.0 2023-12-06 19:18:15 +01:00
Zaiming (Stone) Shi 83206daa20 refactor(emqx_cm_locker): delete dead code 2023-12-06 19:18:15 +01:00
William Yang 64401cb6cc
Merge pull request #12042 from qzhuyan/dev/william/rel54/esockd-robustnes
fix: bump to esockd 8.9.9  improve robustness
2023-12-06 17:39:38 +01:00
Andrew Mayorov 074dc8ce11
test(emqx): switch rest of test suite to use `emqx_cth_suite` 2023-12-06 16:47:04 +01:00
ieQu1 371ec34992 feat(sessds): Make stream renew interval configurable 2023-12-06 16:04:53 +01:00
ieQu1 c43b3eb535 fix(sessds): Add debug logs for the session garbage collection 2023-12-06 15:37:23 +01:00
Andrew Mayorov c2c9de69b4
test(emqx): switch `emqx_crl_cache_SUITE` to use `emqx_cth_suite` 2023-12-06 14:34:41 +01:00
Andrew Mayorov f06a1f10ef
test(emqx): switch most of testsuites to use `emqx_cth_suite`
For increased robustness and isolation.
2023-12-06 14:34:41 +01:00
Andrew Mayorov d643330cb9
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-06 14:34:40 +01:00
Andrew Mayorov 7d80511b3a
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-06 14:34:40 +01:00
Thales Macedo Garitezi 8a5b0d252d
Merge pull request #12110 from thalesmg/test-plugin-flaky-r54-20231205
test(cth_peer): use an exclusive current dir for each peer
2023-12-06 10:16:13 -03:00
William Yang c4956d99d1 fix: bump to esockd 8.9.9 improve listener robustness 2023-12-06 13:55:13 +01:00
Thales Macedo Garitezi b82189fb4d test(cth_peer): use an exclusive current dir for each peer 2023-12-06 09:35:13 -03:00
Andrew Mayorov 43813e29c2
chore(emqx): drop legacy session persistence impl leftovers 2023-12-06 10:53:57 +01:00
SergeTupchiy 28ff53e99c
Merge pull request #11984 from SergeTupchiy/EMQX-10535-openetelmetry-tracing-new
Integrate OpenTelmetry tracing
2023-12-06 11:45:41 +02:00
Andrew Mayorov 3798060543
Merge pull request #12109 from keynslug/test/emqx-cth-suite
test(emqx): switch select test suites to use `emqx_cth_suite`
2023-12-06 09:48:53 +01:00
Andrew Mayorov 83bea2254d
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-05 19:16:37 +01:00
Andrew Mayorov 6f5228e991
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-05 19:16:37 +01:00
Serge Tupchii 938508b270 refactor(emqx_opentelemetry): pass only channel info to trace functions 2023-12-05 19:01:08 +02:00
Thales Macedo Garitezi deb3fcd606 Merge remote-tracking branch 'origin/master' into sync-m-r54-20231205 2023-12-05 13:25:45 -03:00
Serge Tupchii 85441fda0f test(emqx_opentelemetry): add trace test suite 2023-12-05 18:18:31 +02:00
Serge Tupchii 8d3f98eff2 fix(emqx_opentelemetry): fix dialyzer warnings 2023-12-05 18:18:31 +02:00
Serge Tupchii 7fdc650448 feat: integrate OpenTelemetry traces 2023-12-05 18:11:00 +02:00
Andrew Mayorov 286d483a3a
Merge pull request #12055 from keynslug/ft/EMQX-11474/subopts
fix(sessds): respect subscription options when publishing
2023-12-05 15:45:25 +01:00
Andrew Mayorov 92c4b29a4c
feat(topic): match empty topic levels more loosely
So that the result of `emqx_topic:tokens/1` would be perfectly
matchable with the result of `emqx_topic:words/1` of a topic filter
with empty levels.
2023-12-04 13:38:55 +03:00
Andrew Mayorov b5c485cd55
test(sessds): add empty level topic subscription testcase 2023-12-04 13:38:42 +03:00
JimMoen cb992fcf55
Merge pull request #12059 from JimMoen/1130-fix-vm-time-warp
fix: vm.args use `multi_time_warp`
2023-12-04 11:56:08 +08:00
Andrew Mayorov 627c58b07d
fix(sessds): handle `expire_awaiting_rel` common timer with no-op
Otherwise, the channel crashes when the timer is triggered.
2023-12-02 15:00:30 +03:00
Andrew Mayorov 47bc747323
fix(sessds): drop everything related to session on CleanStart 2023-12-02 15:00:30 +03:00
Andrew Mayorov fd26e690b8
refactor(sessds): move parts of message processing to replayer
To simplify the processing flow, reducing the number of back-and-forth
between the session and the replayer.
2023-12-02 15:00:29 +03:00
Andrew Mayorov 6255ee0833
fix(sessmem): avoid defining unsafe defaults in `#session` 2023-12-02 14:38:04 +03:00
Andrew Mayorov b5ad0f9815
fix(test): subscribe with QoS 2 to receive QoS 2 message 2023-12-02 14:38:04 +03:00
Andrew Mayorov 1948719eec
test(sessds): unskip testcase related to QoS2 handling 2023-12-02 14:38:04 +03:00
Andrew Mayorov 3265d2f2aa
fix(sessds): respect subscription options when publishing 2023-12-02 14:38:03 +03:00
Andrew Mayorov 508346f095
refactor(topicidx): split persistent term stuff off gbt-based index 2023-12-02 14:36:21 +03:00
Andrew Mayorov 88103c5f0e
refactor(session): pass config to `SessionImpl:open/3` as well
* Anticipate that connection info may change during session takeover.
* Avoid persisting session conf as part of persistent session state.
2023-12-02 14:36:21 +03:00
Andrew Mayorov 6eba082992
Merge pull request #12075 from keynslug/fix/EMQX-11474/function-clause
fix(channel): expect there's no session yet on disconnect
2023-12-02 14:32:50 +03:00
Zaiming (Stone) Shi 55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 21:35:40 +01:00
Zaiming (Stone) Shi 6e30718818 test: allow join cluster later 2023-12-01 20:33:39 +01:00
ieQu1 3d2e95fe3e
Merge pull request #12081 from ieQu1/dev/gen-rpc-3.3.0
chore(gen_rpc): Bump version to 3.3.0
2023-12-01 17:57:33 +01:00
ieQu1 476c300ecf
Merge pull request #12070 from ieQu1/dev/ds-min-max-batch-size
Various minor fixes
2023-12-01 17:57:15 +01:00
ieQu1 d3b7478d70 chore(gen_rpc): Bump version to 3.3.0 2023-12-01 17:10:12 +01:00
Zaiming (Stone) Shi ab7d57e1c5 test: delete debug level logging 2023-12-01 17:07:14 +01:00
Ivan Dyachkov 0e27d031a8 chore: 5.4.0-alpha.1 2023-12-01 16:54:03 +01:00
Zaiming (Stone) Shi 6a9cc20d4d test: wait for all nodes clustered 2023-12-01 16:37:39 +01:00
Andrew Mayorov 6b17920fea
fix(channel): expect there's no session yet on disconnect 2023-12-01 15:29:05 +03:00
Andrew Mayorov cce474ce98
fix(session): include session term in stacktrace 2023-12-01 15:29:05 +03:00
ieQu1 e1ec560639 test(emqx): Fix flaky emqx_takeover_SUITE:t_takeover testcase
This testcase uses QoS1, so it must account for possible duplication
of the messages.
2023-12-01 10:06:18 +01:00
Zaiming (Stone) Shi 7f4d91d490 Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54 2023-12-01 08:32:22 +01:00
ieQu1 0e625d814a feat(sessds): Make batch size configurable
This change affects flow control. It allows to configure maximum size
of a batch, as well as fetch threshold.
2023-12-01 08:27:05 +01:00
ieQu1 4717e56fb6 fix(sessds): Schedule poll immediately upon receiving an ack
This commit affects the flow control, and improves the throughput by
removing a delay between freeing up space in the in-flight window and
polling new messages.
2023-12-01 08:27:05 +01:00
ieQu1 38800c0260 refactor(sessds): Store timers in the session 2023-12-01 08:27:05 +01:00
Zaiming (Stone) Shi 22f7cc1622 test: replace 'slave' and 'ct_slave' with 'peer' 2023-12-01 08:07:09 +01:00
JimMoen ee25d9bd9e
fix: vm.args use `multi_time_warp`
See also:
https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
2023-12-01 14:52:31 +08:00
ieQu1 69f1ca43c3 fix(sessds): Create a timer even if it's present in the map 2023-12-01 06:11:13 +01:00
ieQu1 1897e5c31b refactor(session): Use common naming conventions 2023-12-01 06:11:08 +01:00
ieQu1 d9e7544070 refactor(sessds): Introduce macros for the timers 2023-12-01 06:11:08 +01:00
ieQu1 0d245acdc1
Merge pull request #12066 from keynslug/fix/sessds-tc-flaky
test(sessds): wait client disconnect propagates to broker
2023-12-01 00:13:38 +01:00
Thales Macedo Garitezi 113a4ad4b1
Merge pull request #12057 from thalesmg/ds-session-gc-m-20231128
feat(ds): add session gc process
2023-11-30 17:51:52 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Thales Macedo Garitezi 684d637fed test(bridge_api): workaround strange config syncing problem
For some unknown reason, this test has difficulties in syncing the config correctly
between the nodes, while the equivalent in bridge_v2_api_SUITE doesn't.
2023-11-30 19:51:35 +01:00
Ivan Dyachkov ec10c51073 Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53 2023-11-30 19:51:12 +01:00
Zaiming (Stone) Shi dcb1c0680b
Merge pull request #12053 from zmstone/1129-prepare-for-otp26
1129 prepare for OTP 26
2023-11-30 19:38:40 +01:00
Andrew Mayorov ec471c0557
test(sessds): wait client disconnect propagates to broker 2023-11-30 20:14:57 +03:00
Ivan Dyachkov 6dd92c382f chore: 5.3.2 release 2023-11-30 16:18:44 +01:00
Thales Macedo Garitezi cf6cb3e4ad chore: set low importance to config 2023-11-30 11:55:57 -03:00
Thales Macedo Garitezi 1ab009f081 refactor: rename supervisor 2023-11-30 11:55:48 -03:00
Zaiming (Stone) Shi eed253af82 test: implement a new node restart helper func 2023-11-30 15:11:38 +01:00
Thales Macedo Garitezi 880f5e8f89 feat(ds): add session gc process
Fixes https://emqx.atlassian.net/browse/EMQX-9744
2023-11-30 09:38:59 -03:00
Zaiming (Stone) Shi cf72c04fdd test: fix peer node stop and plugin SUITE typo 2023-11-30 09:53:36 +01:00
Zaiming (Stone) Shi f2db4cc7fc chore: upgrade to gen_rpc 3.2.2 2023-11-30 09:42:44 +01:00
Zaiming (Stone) Shi 640b0df319 test: do not add -master erl flag for peer nodes 2023-11-30 08:49:57 +01:00
Zaiming (Stone) Shi 4ecfe2be30 test: use peer module for slave and ct_slave 2023-11-29 21:45:53 +01:00
Ivan Dyachkov 7f5433f6dd chore: 5.3.2-rc.1 2023-11-29 18:20:15 +01:00
Thales Macedo Garitezi fdafe2493e
Merge pull request #12041 from thalesmg/ds-estimate-last-alive-m-20231128
fix(ds_session): take conservative estimate of `last_alive_at` when bumping
2023-11-29 13:43:59 -03:00
Zaiming (Stone) Shi 6f35f25163 chore: upgrade esockd to 5.9.8 for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi 1a563b4f65 chore: fix 0.0 match for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi 89cdfbca63 fix(emqx_vm): trim new-line in otp version string 2023-11-29 15:32:25 +01:00
Andrew Mayorov ccef91437d
Merge pull request #12029 from keynslug/ft/EMQX-11049/qos2
feat(sessds): provide QoS2 message replay support
2023-11-29 14:10:16 +03:00
Andrew Mayorov d8691f1d64
refactor(sessds): rename marker → committed offset
For better clarity.
2023-11-29 13:04:42 +03:00
William Yang 64f6220cf5 feat(quic): bump to quicer-0.0.303 and emqtt 1.9.7 2023-11-28 14:43:47 +01:00
Thales Macedo Garitezi f5c4fb5860 fix(ds_session): take conservative estimate of `last_alive_at` when bumping
Addresses https://github.com/emqx/emqx/pull/12024#discussion_r1407432154
2023-11-28 09:58:03 -03:00
Ivan Dyachkov 2e11ab6a16 chore: 5.3.2-alpha.2 2023-11-28 13:46:28 +01:00
Zaiming (Stone) Shi 1b1cea24f3 Merge remote-tracking branch 'origin/release-54' into 1128-sync-release-54 2023-11-28 11:43:30 +01:00
Andrew Mayorov 34c9c022d0
chore(replayer): add comment describing what "until" means 2023-11-28 11:47:54 +03:00
JianBo He c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
JianBo He dc99651690 test(bridge): ensure almost test cases passed 2023-11-28 09:53:46 +08:00
Serge Tupchii 64ee29af81 feat(emqx_management): implement data backup API 2023-11-27 17:30:58 +02:00
Andrew Mayorov 86685bdce2
feat(sessds): use integer tags for pubrange types 2023-11-27 18:27:15 +03:00
Andrew Mayorov bb05281adb
refactor(sessds): add dedicated `#ds_pubrange.tracks` field
This slightly simplifies the replayer code.
2023-11-27 18:24:57 +03:00
Thales Macedo Garitezi d88deb9ceb feat(ds): add session timer to bump last alive at timestamp 2023-11-27 12:19:52 -03:00
Thales Macedo Garitezi 09c4e40511 refactor(ds): rename `disconnected_at` to `last_alive_at`, add more assertions 2023-11-27 11:48:44 -03:00
Andrew Mayorov 923898eadf
chore(chan): leave a TODO note for PUBREC handler 2023-11-27 09:55:46 +03:00
Andrew Mayorov 41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs 2023-11-27 09:55:46 +03:00
Andrew Mayorov 46475fac66
feat(sessds): provide QoS2 message replay support 2023-11-27 09:55:45 +03:00
Andrew Mayorov ce59cb71bb
chore: bump emqtt to 1.9.6 2023-11-27 09:55:45 +03:00
zhongwencool b5a00ec6b2
Merge pull request #12023 from emqx/master
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
ieQu1 bd7a84fe3e revert(ds): Don't duplicate the clean start in session_ds 2023-11-26 19:18:59 +01:00
Thales Macedo Garitezi 839f9dbedb feat(ds): session expiry
Fixes https://emqx.atlassian.net/browse/EMQX-11048
2023-11-24 16:31:54 -03:00
ieQu1 2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start
Discard sessions when client connects with clean start = true
2023-11-24 13:35:39 +01:00
ieQu1 e616e0746a feat(ds): Implement a function for dumping persistent session state 2023-11-24 04:39:46 +01:00
ieQu1 8dfcb69e52 feat(ds): Discard session when client connects with CleanStart=1 2023-11-24 04:17:28 +01:00
ieQu1 e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0
Support QoS0 messages in persistent_session_ds
2023-11-23 23:43:18 +01:00
ieQu1 a158f25a40 fix(ds): Fix return type of emqx_persistent_session_ds:publish 2023-11-23 22:16:38 +01:00
ieQu1 c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1 1ced8786fd feat(ds): Make session poll interval configurable 2023-11-23 17:10:03 +01:00
ieQu1 82e74d0201 feat(ds): Add a flag that forces all sessions to become durable 2023-11-23 17:10:03 +01:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Ilya Averyanov a1b9a14fa1 feat(ds): allow fdb implementation for durable storage 2023-11-22 14:49:38 +03:00
ieQu1 ee191803ea
Merge pull request #11990 from ieQu1/dev/ds-shards
Implement a prototype of durable message sharding
2023-11-22 00:57:54 +01:00
Zaiming (Stone) Shi 5ebd954b16 chore: bump release version to e5.3.2-alpha.1 2023-11-21 21:19:05 +01:00
Zaiming (Stone) Shi 8fbdcab118
Merge pull request #11987 from zmstone/1120-do-not-crash-on-einval-after-check-cache
fix(emqx_connection): handle socket activation error return
2023-11-21 20:31:56 +01:00
ieQu1 3165b4f645 fix(ds): Abort application startup when rocksdb is not avialable 2023-11-21 20:27:07 +01:00
ieQu1 4d47490734 chore(ds): Rebase configuration 2023-11-21 19:37:01 +01:00
ieQu1 2a1f7d946a feat(ds): Shard messages by publisher client ID 2023-11-21 19:37:01 +01:00
ieQu1 62542e5844 feat(ds): Metadata storage for the replication layer 2023-11-21 19:37:01 +01:00
Zaiming (Stone) Shi fa91bacdfb
Merge pull request #11975 from zmstone/1119-fix-socket-close-race-condition
fix(emqx_channel): do not log stale sock_close event as error
2023-11-21 16:47:03 +01:00
Thales Macedo Garitezi 068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120
sync `release-53` to `master`
2023-11-21 09:02:51 -03:00
Ilya Averyanov c6fd1e4c75
Merge pull request #11971 from savonarola/1116-no-auth-rebalance-status
feat(rebalance): improve rebalance usability
2023-11-21 12:10:18 +02:00
Andrew Mayorov 110a5a4896
feat(schema): introduce separate root for new session persistence
With some knobs to choose a storage backend. Support only builtin
RocksDB-based backend with minimal configuration for now.
2023-11-20 23:49:14 +07:00
Zaiming (Stone) Shi b02711af79 refactor(emqx_ws_connection): rename cache to buffer for limiter 2023-11-20 17:47:20 +01:00
Zaiming (Stone) Shi ec19247271 refactor: rename limiter buffer related messages and var names 2023-11-20 16:55:26 +01:00
Andrew Mayorov 9684e79ee0
fix(sessds): ensure dup flag is on for replayed messages 2023-11-20 22:54:40 +07:00
Zaiming (Stone) Shi 8ec3b1db5d fix(emqx_connection): handle socket activation error return 2023-11-20 16:51:39 +01:00
Thales Macedo Garitezi 53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Andrew Mayorov ef7cfd0202
feat(sessds): add field to `ds_pubrange` for forward-compat 2023-11-20 19:56:55 +07:00
Andrew Mayorov d7974d835f
fix(sharesub): anticipate messages w/o redispatch header
For instance, `emqx_session_mem` will push almost all of the messages
that are still in the state to `emqx_shared_sub:redispatch/1`. After
this commit, the session will no longer crash during channel terminate.
2023-11-20 15:53:24 +07:00
Andrew Mayorov 7081f1951f
refactor(sessds): use `ds_pubrange` record as is
Instead of converting it into almost similar runtime representation.
2023-11-20 15:40:43 +07:00
Andrew Mayorov a5ff4144fe
test(sessds): add complex testcase for session replay 2023-11-20 15:40:43 +07:00
Andrew Mayorov 1246d714c5
feat(sessds): preserve acks / ranges in mnesia for replays 2023-11-20 13:33:48 +07:00
Andrew Mayorov 5b40304d1f
chore(sessds): simplify subscriptions handling
There's currently no point in storing parsed topic filters in the
subscriptions table.
2023-11-20 13:33:48 +07:00
Andrew Mayorov 648b6ac63e
chore(sessds): rename iterators -> subscriptions
Also try to make clearer the difference between 2 flavors of topic
filter representation in use.
2023-11-20 13:33:48 +07:00
Zaiming (Stone) Shi e73bf716ae fix(emqx_channel): do not log stale sock_close event as error
In some cases, EMQX may decide to close socket and mark connection
at 'disconnected' state, for example, when DISCONNECTE packet is
received, or, when failed to write data to socket.
However, by the time EMQX decided to close the socket, the socket
might have already been closed by peer, and the `tcp_closed` envet
is already delivered to the process mailbox -- causing EMQX to
handle sock_close event at 'disconnected' state.
2023-11-19 22:27:25 +01:00
Ilya Averyanov e93e9ed108 feat(rebalance): improve rebalance usability
* make availability API endpoint public
* allow connections during wait_health_check interval
* make availability status calculation more consistent and lightweight
* refactor test to get rid of some mocks and to use cth
2023-11-18 21:29:16 +03:00
Thales Macedo Garitezi 90571b7d8e test: fix noise about undefined unofficial callbacks 2023-11-16 13:16:02 -03:00
Andrew Mayorov 656d575e2d
ci(router): fix flaky testcase 2023-11-16 17:53:20 +07:00
Andrew Mayorov d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default
feat(router): switch to v2 routing store by default
2023-11-16 15:56:26 +07:00
Andrew Mayorov 98f947f4f3
ci(router): fix flaky testcase 2023-11-15 22:28:52 +07:00
Andrew Mayorov 8919b08207
fix(utils): rename `emqx_utils_stream:take/2` to `consume/2`
Which is more neutral and harder to confuse with a destructive `take` in
collections.
2023-11-15 17:20:40 +07:00
Thales Macedo Garitezi 423fb32171 test: fix noise about undefined unofficial callbacks 2023-11-14 19:48:39 +01:00
Ivan Dyachkov 7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
ieQu1 124c0e2dba fix(ds): Respect receive_maximum from the connection info 2023-11-14 16:06:19 +01:00
Ivan Dyachkov bbf00c2f71
Merge pull request #11946 from id/1114-sync-r53
sync release-53 to master
2023-11-14 13:38:24 +01:00
Thales Macedo Garitezi 01a2a3b1c0
Merge pull request #11930 from thalesmg/ds-session-discard-m-20231110
feat(ds): implement session discard
2023-11-14 09:30:34 -03:00
Andrew Mayorov 6812ee9d0f
fix(mgmt): hide route selection behind router interface
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
2023-11-14 18:36:25 +07:00
Ivan Dyachkov 28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
Ivan Dyachkov 1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Ivan Dyachkov a9cf8fe78c chore: 5.3.1 2023-11-14 08:44:41 +01:00
Zaiming (Stone) Shi 9b9908780c
Merge pull request #11926 from kjellwinblad/kjell/fix_bridge_v2_to_action_cluster_hocon/EMQX-11353
fix: reflect bridge V1 upgreade in cluster.hocon
2023-11-13 14:18:57 +01:00
Thales Macedo Garitezi 45dad2ed3a feat(ds): implement session discard
Fixes https://emqx.atlassian.net/browse/EMQX-9739

Fixes some issues to ensure the session is discarded when the client connects with
`clean_start = true`, and added some cleanup to subscriptions/routes/iterators/streams.

> There is an API that session garbage collector can use to perform cleaning

We already have `emqx_session:destroy/1`, which could serve as an API for a periodic
session GC to use.
2023-11-13 09:06:22 -03:00
Kjell Winblad 26fc9ed89d
fix: log warning message after suggestion from @zmstone
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-11-13 12:53:09 +01:00
Andrew Mayorov 03843c6071
feat(router): switch to v2 routing store by default 2023-11-13 17:43:30 +07:00
Kjell Winblad 5838568d5d fix: hard coded call to schema module
The user can configure the configuration loader and schema module so we
should not hard code a call to the default schema module.
2023-11-13 11:41:40 +01:00
Zaiming (Stone) Shi 65efa2672e test(schema): delete stale reference to emqx_schema:file() type 2023-11-13 09:28:26 +01:00
JimMoen 867d5aaea6
chore: rm dbg funcs in SUITE 2023-11-12 21:44:23 +08:00
JimMoen 523bb17fc8
test: mark proper types gateway connection modules need be fixed 2023-11-12 21:43:10 +08:00
Zaiming (Stone) Shi 021f7e6b49 refactor(schema): comma_separated_list is list(string()) not list(any()) 2023-11-12 11:37:58 +01:00
Zaiming (Stone) Shi 8be718b22f refactor(emqx_schema): delete unused type bar_separated_list 2023-11-12 11:29:33 +01:00
ieQu1 5170267fef
Merge pull request #11924 from ieQu1/dev/emqx-ds-behavior
feat(ds): Introduce emqx_ds behavior
2023-11-10 15:23:28 +01:00
Kjell Winblad 6ebf0d1bdd fix: reflect bridge V1 upgrade in override configurations
Fixes:
https://emqx.atlassian.net/browse/EMQX-11353
2023-11-10 14:52:29 +01:00
Zaiming (Stone) Shi 86110824eb feat: upgrade hocon to 0.40.0 which supports union type display name 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 8a4fba431e refactor(emqx_limiter_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
ieQu1 8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
ieQu1 38d004e06e
Merge pull request #11913 from ieQu1/dev/fix-hooks-comment
chore(hooks): Fix comment
2023-11-10 03:15:54 +01:00
ieQu1 ced7be61f0
Merge pull request #11917 from ieQu1/dev/fix-packet-id-to-seqno-trans
fix(ds): Fix packet id -> sequence number translation
2023-11-09 22:57:16 +01:00
Thales Macedo Garitezi 7a237f6a3a Merge branch 'master' into sync-m-to-r54-20231109 2023-11-09 18:49:52 -03:00
Thales Macedo Garitezi 371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
ieQu1 5aa9d026df fix(ds): Apply review remarks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-09 21:16:59 +01:00
ieQu1 b812db1e3c fix(ds): Fix packet id -> sequence number translation 2023-11-09 20:12:39 +01:00
Thales Macedo Garitezi 2d539463cd test: fix flaky test
```
 Testing lib.emqx.emqx_cm_SUITE: *** FAILED test case 24 of 31 ***
%%% emqx_cm_SUITE ==> t_stepdown_session_takeover_begin_timeout: FAILED
%%% emqx_cm_SUITE ==>
Failure/Error: ?assertEqual([<0.6635.8>,<0.6520.8>], lists : sort ( emqx_cm : lookup_channels ( ClientId ) ))
  expected: [<0.6635.8>,<0.6520.8>]
       got: [<0.6520.8>,<0.6635.8>]
      line: 309
```
2023-11-09 15:32:52 -03:00
Thales Macedo Garitezi ee87f90b0d
Merge pull request #11906 from thalesmg/ds-avoid-records-m-20231108
chore(ds): avoid using records in persistence / rpc
2023-11-09 14:25:27 -03:00
ieQu1 42a9f0eb6b
Merge pull request #11914 from ieQu1/dev/speed-up-poll
fix(ds): Speed up polling for the new messages
2023-11-09 16:40:06 +01:00
Thales Macedo Garitezi 42536bb5f7 chore(ds): avoid using records in persistence / rpc
Fixes https://emqx.atlassian.net/browse/EMQX-11279
2023-11-09 12:00:25 -03:00
ieQu1 cf4a46a78b fix(ds): Speed up polling for the new messages
Poll immediately if the previous poll returned non-empty result
2023-11-09 14:22:32 +01:00
ieQu1 3350bda446 chore(hooks): Fix comment 2023-11-09 14:05:13 +01:00
firest 2b97800c8b fix(limiter): simplify the descriptions for fields of the limiter 2023-11-09 18:00:09 +08:00
William Yang f7a2359329 test(quic): fix flaky 2023-11-09 10:56:16 +01:00
Ivan Dyachkov 0bf5deaed9 chore: merge master into release-54 2023-11-09 08:32:26 +01:00
Thales Macedo Garitezi 8540566eba chore: prepare to tag `e5.3.1-alpha.5` 2023-11-08 13:02:19 -03:00
ieQu1 788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session
Refactor emqx_durable storage application and introduce learned topic structure storage
2023-11-08 14:27:40 +01:00
Ilya Averyanov d061d64c70 fix(plugins): fix backward compatibility 2023-11-08 12:36:09 +03:00
Ivan Dyachkov 0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
ieQu1 a1cdbaa76d fix(ds): Address code review remarks 2023-11-08 04:31:27 +01:00
ieQu1 7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
ieQu1 74cb43f8b1 fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
Andrew Mayorov 46d8301bc0 feat(emqx): expose timestamp function in `emqx_message`
So that the code that relies on it would not need to guess clock source
and precision.
2023-11-08 03:04:19 +01:00
ieQu1 8e5dda40be fix(ds): Fix static checks 2023-11-08 03:04:19 +01:00
ieQu1 99329e1243 refactor(ds): Address review remarks 2023-11-08 03:04:19 +01:00
ieQu1 2de79dd9ac feat(ds): Replay QoS1 messages 2023-11-08 03:04:19 +01:00
ieQu1 465e8a90dd revert(ds): Remove change from the old protocol file 2023-11-08 03:04:19 +01:00
ieQu1 ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 903b3863d1 chore(ps_ds): make persistent session module use new `emqx_ds` APIs 2023-11-08 03:04:19 +01:00
ieQu1 2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1 c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1 59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1 7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
ieQu1 c91df2f5cd refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 7ab57824dc chore(ds): change return type of `storage_layer:next/{1,2}`
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
Ilya Averyanov 75704513d8 fix(plugins): fix backward compatibility 2023-11-07 19:06:37 +03:00
Thales Macedo Garitezi c4c9d7ceb3 chore: prepara to tag `e5.3.1-alpha.4` 2023-11-07 13:04:46 -03:00
Zaiming (Stone) Shi 6af6309301 chore: upgrade to hocon 0.39.19 2023-11-06 19:16:19 +01:00
Zaiming (Stone) Shi afe6f79a66 feat(bridge): add description field to bridge and connector 2023-11-03 06:53:42 +01:00
Andrew Mayorov 910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder
feat(tpl): split `emqx_placeholder` into a couple of modules
2023-11-02 22:50:05 +07:00
Andrew Mayorov 0538a77700
feat(tpl): use `emqx_connector_template` in `emqx_authn`, `emqx_authz`
This slightly changes semantics: now the attempt to create authenticator
with illegal bindings in templates will fail, instead of treating them
as literals. The runtime behaviour on the other hand should be the same.
2023-11-02 17:11:10 +07:00
Zaiming (Stone) Shi 73dd2f0ffd chore: bump version to e5.3.1-alpha.3 2023-11-02 10:36:41 +01:00
Andrew Mayorov 7092c75597
Merge pull request #11809 from keynslug/ft/EMQX-10808/file-secrets
feat(mqttbridge): support file-sourced secrets as passwords
2023-11-01 00:29:30 +07:00
Thales Macedo Garitezi b420b53075 fix(bridges_v2): check bridge name length before attempting atom conversion
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
Andrew Mayorov d278486416
fix(secret): dedicate a specific loader module for file secrets
To make code employing `emqx_secret` easier to follow.
2023-10-31 19:26:04 +07:00
Ivan Dyachkov b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
JianBo He 21e4f918aa
Merge pull request #11833 from JimMoen/fix-shared-sub-topic-or-subscription-searching
Fix shared sub topic or subscription searching
2023-10-31 08:16:36 +08:00
Thales Macedo Garitezi a60b96c5fd test(kafka_producer): minor adjustments to test suite
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
Zaiming (Stone) Shi f463eff02d chore: e5.3.1-alpha.2 2023-10-30 14:49:08 +01:00
Zaiming (Stone) Shi 0656b6be3c refactor(emqx_config): use dynamic callback to upgrade raw config 2023-10-30 14:49:08 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
zhongwencool e896957803
Merge pull request #11824 from zhongwencool/audit-log-reviewed
refactor: audit log review refactor
2023-10-30 17:51:15 +08:00
JimMoen 814e22feb3
fix: topics/subscripton mgmt api searching 2023-10-30 14:42:47 +08:00
zhongwencool 0634ff61c0 fix: dialyzer warning 2023-10-30 09:57:30 +08:00
JimMoen a2015f37ae
Merge pull request #10976 from JimMoen/fix-shared-sub-unsub 2023-10-27 21:22:27 +08:00
JimMoen d563121284
refactor: move ?REDISPATCH_TO macro to emqx_mqtt.hrl 2023-10-27 17:54:14 +08:00
JimMoen 3b5cc912e7
fix: add `redispatch_to` header to all msgs when deliver shared-sub
- to find correct SubOpts for shared-sub dispatch
- use previous key `redispatch_to` to ensure rolling upgrade compatibility
2023-10-27 16:15:23 +08:00
zhongwencool 995948f0e8 refactor: remove seq from audit record 2023-10-27 14:07:19 +08:00
JimMoen afec6fa2f6
fix: TopicFilter may modified by `client.subscribe` hook 2023-10-27 09:23:44 +08:00
JimMoen 802a36c670
fix: find SubOpts by shared_record, not deliver topic 2023-10-27 09:23:44 +08:00
JimMoen 0ca725ff25
fix: deprecated `share` in `subopts` 2023-10-27 09:23:43 +08:00
JimMoen b5411da770
refactor: subscribe process to fix shared-sub 2023-10-27 09:23:43 +08:00
Andrew Mayorov 44b4205561
fix(secret): do not treat missing file secrets as config error
They are intended to be used mostly in the context of resources, which
have their own feedback mechanism: statuses, retries, etc.

Also turn the error into a throw exception, so that it can be
interpreted as a regular error condition, for example by the resource
manager.
2023-10-26 14:37:14 +07:00
Zaiming (Stone) Shi 76a8b0efd4 test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked 2023-10-25 16:08:31 +02:00
firest 7a8a5926ab fix(rbac): adjust the role names 2023-10-24 19:01:12 +08:00
Andrew Mayorov 1c2f9321d1
feat(emqx): add file-sourced generic secrets
These secrets follow the same `emqx_secret` convention of 0-arity
functions. Also provide a simple HOCON schema module for use in
application schemas.
2023-10-24 16:24:51 +07:00
Ilya Averyanov edde661da3 fix(authn): fix pbkdf2 option validation 2023-10-23 10:26:11 +03:00
William Yang a9b42af8b3
Merge pull request #11796 from qzhuyan/fix/william/gen_rpc-driver
fix(gen_rpc): ensure client/server use same driver
2023-10-23 08:24:17 +02:00
JimMoen d467289bb2
fix: avoid duplicated apikey from data import 2023-10-20 21:13:04 +08:00
William Yang dd5807b797 fix(gen_rpc): ensure client/server use same driver
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
zhongwencool ef692596f7 fix: don't crash when 401 and 403 unauthorize 2023-10-20 15:22:30 +08:00
JimMoen f381961108 fix: macro EMQX_RELEASE_EDITION when `emqx` run as standalnoe app 2023-10-20 14:47:24 +08:00
zhongwencool befc484544 chore: add changelog entry and fix xref warning 2023-10-20 14:47:24 +08:00
zhongwencool 141061c1e2 feat: add max_filter_size and ignore_high_frequency_request config 2023-10-20 14:47:24 +08:00
zhongwencool 926c804314 feat: add /audit http api to filter audit log 2023-10-20 14:47:24 +08:00
lafirest 74442b0d31
Merge pull request #11766 from lafirest/feat/rbac
feat(api_key): add RBAC feature for the API key
2023-10-20 09:24:41 +08:00
firest 26ec860d96 feat(backup): add migration mechanism when import backup data 2023-10-17 16:23:32 +08:00
firest db3915d472 fix(rbac): update changes && fix CI errors 2023-10-17 13:44:37 +08:00
firest e095de7367 feat(api_key): add RBAC feature for the API key 2023-10-16 14:10:01 +08:00
Zaiming (Stone) Shi 6eb3bb7cff Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53 2023-10-14 10:16:38 +02:00
Andrew Mayorov 083e2da347
chore(bpapicheck): make some failures more user-friendly 2023-10-13 15:42:08 +07:00
Andrew Mayorov 0b9ac24c1e
fix(bpapicheck): stop depending on git index 2023-10-13 15:40:49 +07:00
William Yang a69c53455f fix: bump to quicer 0.0.202
bring in urgent fixs from msquic 2.2.3
2023-10-11 13:12:32 +02:00
Zaiming (Stone) Shi a2e86c67db
Merge pull request #11733 from keynslug/fix/EMQX-10827/sesson-takeover
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
2023-10-10 19:03:29 +02:00
Ivan Dyachkov bfb2218392 chore: fix release version 2023-10-10 14:50:49 +02:00
Zaiming (Stone) Shi 143673b703 chore: bump release version to 5.3.1-alpha.1 2023-10-10 14:35:46 +02:00
Andrew Mayorov 522302fee1
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
Which was accidentally broken in bf164175.
2023-10-10 01:34:10 +07:00
Zaiming (Stone) Shi 1e93d2f1fc chore: upgrade to ekka 0.15.16 gen_rpc 3.2.0
* ekka 0.15.16 supports 'inet6_tls' as ekka.proto_dist
* gen_rpc 3.2.0 supports true | false as gen_rpc.ipv6_only
2023-10-09 20:03:21 +02:00
Ilya Averyanov 5dff36474d chore(auth): get rid of hardcoded schema modules in auth 2023-10-05 13:41:50 +03:00
Ilya Averyanov c2c56ba481 chore(auth): update tests 2023-10-05 13:41:50 +03:00
Ilya Averyanov 1eb75b43c4 chore(auth): split emqx_authn and emqx_authz apps 2023-10-05 13:41:50 +03:00
Zaiming (Stone) Shi ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
0928 sync release 53
2023-10-02 21:57:32 +02:00
Thales Macedo Garitezi 2358d67908 refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
Zaiming (Stone) Shi bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Zaiming (Stone) Shi ed5b456d62 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 18:24:44 +02:00
Zaiming (Stone) Shi a852400fb3 chore: add 5.3.bpapi 2023-09-29 16:34:04 +02:00
Zaiming (Stone) Shi fe01aaff4f chore: prepare for e5.3.0 promotion 2023-09-29 13:32:57 +02:00
Zaiming (Stone) Shi ce5bd0a3ce Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 11:36:32 +02:00
Zaiming (Stone) Shi c64e599e81 docs: document how to retrieve peercert 2023-09-29 10:33:57 +02:00
Zaiming (Stone) Shi 1177a32310 chore: bump version to 5.3.0-rc.2 2023-09-29 09:49:27 +02:00
Zaiming (Stone) Shi b267fc2588 chore: bump release version to 5.3.0 2023-09-28 21:22:33 +02:00
Zaiming (Stone) Shi 1a13b2ac56 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-28 13:53:03 +02:00
Zaiming (Stone) Shi 7ddcd71412 docs(ssl_dist.conf): update doc link 2023-09-28 11:16:18 +02:00
ieQu1 b77e5e880a
Merge pull request #11697 from ieQu1/dev/gen-rpc-3.1.1
chore(gen_rpc): Bump version to 3.1.1
2023-09-28 10:03:36 +02:00
JianBo He 6d2adfc259 chore(audit): emit the message field 2023-09-28 11:04:42 +08:00
JianBo He 859b122cdd chore: format codes 2023-09-28 10:25:37 +08:00
JianBo He dd9938114c chore(audit): add from field 2023-09-28 10:25:37 +08:00
lafirest b0d86eecd6
Merge pull request #11691 from lafirest/fix/sso_ssl
fix(sso): support for SSL update && ensure update is atomic
2023-09-28 10:22:34 +08:00
ieQu1 0aa3ccdd65 feat(gen_rpc): Add schema for the TLS versions and cipher suites 2023-09-28 00:36:59 +02:00
Zaiming (Stone) Shi afdda107af fix(logger): json format log encode binary list as string array 2023-09-27 23:40:01 +02:00
Zaiming (Stone) Shi 6f7a4344dc fix: do not gc sso saml SP singing keys 2023-09-27 23:19:39 +02:00
Zaiming (Stone) Shi c8cbbff044 fix(logger): no need for special handling of empty string
when formating json logs, there is no need to handle empty strings
special, already covered by unicode handling
2023-09-27 21:26:47 +02:00
ieQu1 3c37f19105 chore(gen_rpc): Bump version to 3.1.1 2023-09-27 21:09:00 +02:00
Zaiming (Stone) Shi 6e8c73258f test: refactor emqx_rpc unit tests 2023-09-27 12:58:06 +02:00
Andrew Mayorov 1d0e789e4d
Merge pull request #11679 from keynslug/ft/EMQX-10942/custom-session-timers
feat(session): add custom session timers mechanism
2023-09-26 19:09:53 +03:00
Thales Macedo Garitezi aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
Andrew Mayorov bce2142df8
test(session): add custom session timers testcase 2023-09-26 16:22:06 +03:00
Zaiming (Stone) Shi 4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
Thales Macedo Garitezi fdcd73e20c fix(file_logger): change file logger type depending on rotation size
Fixes https://emqx.atlassian.net/browse/EMQX-11036

From `logger_disk_log_h:open_disk_log`:

```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
    case filelib:ensure_dir(File) of
        ok ->
            Size =
                if Type==halt -> MaxNoBytes;
                   Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
                end,
            Opts = [{name,   Name},
                    {file,   File},
                    {size,   Size},
                    {type,   Type},
                    {linkto, self()},
                    {repair, false},
                    {format, external},
                    {notify, true},
                    {quiet,  true},
                    {mode,   read_write}],
            case disk_log:open(Opts) of
```

Affects all file loggers (audit included):

```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
                                                                             {badarg,
                                                                              [{size,
                                                                                {infinity,
                                                                                 10}},
                                                                               {type,
                                                                                wrap},
                                                                               {linkto,
                                                                                <0.1952.0>},
                                                                               {repair,
                                                                                false},
                                                                               {format,
                                                                                external},
                                                                               {notify,
                                                                                true},
                                                                               {quiet,
                                                                                true},
                                                                               {mode,
                                                                                read_write}]}}
```
2023-09-26 09:04:44 -03:00
Ilya Averyanov 14983ec14a chore(hooks): validate hookpoints and document hook callbacks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
Thales Macedo Garitezi ff7f37ccf5 test(cth): allow defining schema to load for app 2023-09-25 13:22:41 -03:00
Andrew Mayorov b1f144ab8b
feat(session): add custom session timers mechanism
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
lafirest 1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
Zaiming (Stone) Shi 5e6996dc05 refactor: log formatter format mfa+line as m:f/a(line)
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 6a557980e4 refactor(logger): ensure JSON log field order
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 5f45ba50ff refactor: delete log formatter config for audit log handler
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 1fed38c248 fix(logger): write 'json' format logs as JSON 2023-09-25 16:55:22 +02:00
William Yang 9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify 2023-09-25 15:48:01 +02:00
firest cfdb25b213 fix(sso): updates the SSO backend when the `[dashboard]` has updated 2023-09-25 20:30:47 +08:00
William Yang 36d3a3a524 fix: bump to quicer 0.0.201
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
William Yang 22193d273a fix(quic): ignore undefined cacertfile 2023-09-25 12:57:55 +02:00
Ivan Dyachkov 8c93c79b44 chore: e5.3.0-alpha.2 2023-09-25 12:52:45 +02:00
Ivan Dyachkov e1f1c64ecd Merge branch 'release-53' into 0925-sync-r53-to-master 2023-09-25 08:12:45 +02:00
JianBo He df94426ee3
chore: make static_check happy 2023-09-22 21:50:42 +08:00
Thales Macedo Garitezi d6935b6a67 feat: add port scan diagnostics to mria waiting for tables checks
Fixes https://emqx.atlassian.net/browse/EMQX-10944

Also updates ekka -> 0.15.15, mria -> 0.6.4

How to test
===========

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.

Or, more easily:

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.

Example output
==============

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status =>
                                                                 bad_ports},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```

After one node is back:

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status => ok},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```
2023-09-21 14:29:01 -03:00
Ivan Dyachkov 7cf60c5a91 chore: e5.3.0-alpha.1 2023-09-21 19:29:00 +02:00
Andrew Mayorov acf4227fc6
test(session): fix quic testgroup in persistent session suite
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
Andrew Mayorov b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov 0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
sync r52 to master
2023-09-21 12:31:24 +02:00
William Yang 3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
quicer 0.0.200
2023-09-21 12:20:14 +02:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
Ivan Dyachkov 105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
Andrew Mayorov a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently 2023-09-21 11:53:40 +04:00
zhongwencool 1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify 2023-09-21 07:32:39 +08:00
Andrew Mayorov a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
Andrew Mayorov 3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy 2023-09-20 22:55:54 +04:00
Andrew Mayorov 21e82b9534
test(sessmem): make retry delivery testcase more involved 2023-09-20 22:55:54 +04:00
Andrew Mayorov 69889d14a3
fix(sessds): fix use of undefined types 2023-09-20 22:55:54 +04:00
Andrew Mayorov 9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
Andrew Mayorov 98706cd215
chore: ensure comments follow code style consistently 2023-09-20 22:55:25 +04:00
Andrew Mayorov 8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability 2023-09-20 22:55:25 +04:00
Andrew Mayorov 7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov 540ca6d60f
chore: drop few commented out and irrelevant pieces 2023-09-20 22:55:24 +04:00
Andrew Mayorov 9d145890cc
refactor(sessmem): pass log context as part of session event
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov e422f492ef
test(sessds): reuse and expand persistent session test suite 2023-09-20 22:55:24 +04:00
Andrew Mayorov 3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling 2023-09-20 22:55:23 +04:00
Andrew Mayorov 7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Andrew Mayorov e713fc38aa
feat(broker): reflect persisted messages in publish result
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
Andrew Mayorov f4953e719b
fix(cmproto): fix few typespecs 2023-09-20 22:53:45 +04:00
Andrew Mayorov 8af107e28d
test(ds): simplify cluster test setups 2023-09-20 22:53:45 +04:00
Andrew Mayorov 7c4f68dd3d
fix(session): make utility function names consistent
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
Andrew Mayorov 2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem` 2023-09-20 22:53:44 +04:00
Andrew Mayorov abeff0bc4f
chore(session): try to describe what happens after session takeover 2023-09-20 22:53:44 +04:00
Andrew Mayorov 45d44df11d
refactor(session): update eviction channel session logic
The changes partially reflect `emqx_channel` changes with respect to
in-memory session specific logic. The difference is that eviction
channel does not replay post-takeover, instead enqueues messages.
2023-09-20 22:53:44 +04:00
Andrew Mayorov ab1c4c4222
refactor(session): drop `lookup/1` session API
Due to the fact it's not used for anything right now.
2023-09-20 22:53:44 +04:00
Andrew Mayorov 97881ff3ca
refactor(session): bring back common session timers 2023-09-20 22:53:43 +04:00
Andrew Mayorov bf16417513
feat(session): introduce session implementation concept 2023-09-20 22:53:42 +04:00
Andrew Mayorov 04731b7ef7
test(takeover): randomize messages for random natural ordering 2023-09-20 22:52:14 +04:00
Andrew Mayorov f022c9b1a4
feat(emqx): add `emqx_inflight:fold/3` generic function 2023-09-20 22:52:14 +04:00
Andrew Mayorov e4866adc2f
refactor(chan): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov cfb1bf1fa4
chore(emqx): drop remnants of former session persistence impl
1. It is not functional anyway.
2. It blocks `emqx_session` refactoring in a few places.
2023-09-20 22:52:13 +04:00
William Yang e43b87f18e test(quic): update emqx_broker_SUITE 2023-09-20 20:20:18 +02:00
Thales Macedo Garitezi 2965fa6fcb
Merge pull request #11613 from thalesmg/ds-keyspace-m-20230914
feat(ds): introduce keyspace concept
2023-09-20 14:21:27 -03:00
William Yang da626f6d29 feat(quic): bump to quicer 0.0.200 and emqtt 1.9.0 2023-09-20 16:43:15 +02:00
zhongwencool f789a7504f
chore: update apps/emqx/src/emqx_quic_connection.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:00:44 +08:00
Ivan Dyachkov 09aea47f40 chore: 5.2.1 2023-09-20 14:04:28 +02:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Thales Macedo Garitezi 5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
Zaiming (Stone) Shi 65818740f3
Merge pull request #11622 from zmstone/0917-upgrade-gen_rpc-to-3.1.0
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-19 10:22:53 +02:00
Ivan Dyachkov a69c26b86d chore: e5.2.1-alpha.3 2023-09-19 08:17:29 +02:00
JianBo He de1bc04843 chore: bump ekka to 0.15.14, mria to 0.6.3
Inclued ekka's updates:
- https://github.com/emqx/ekka/pull/214
- https://github.com/emqx/ekka/pull/215
- https://github.com/emqx/ekka/pull/216
- https://github.com/emqx/ekka/pull/217

mria's updates:
- https://github.com/emqx/mria/pull/160
- https://github.com/emqx/mria/pull/161
- https://github.com/emqx/mria/pull/162
2023-09-19 09:43:19 +08:00
Thales Macedo Garitezi a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Zaiming (Stone) Shi de93d71014 test: fix emqx_sahred_sub_SUITE:t_remote
prior to this fix, the peer ct node is actually node itself
2023-09-18 21:54:43 +02:00
Zaiming (Stone) Shi 99b60849b4 test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer 2023-09-18 20:44:38 +02:00
Zaiming (Stone) Shi d1e5b69eac test: increase wait time 2023-09-18 17:33:09 +02:00
Zaiming (Stone) Shi 4f1ec795eb test: delete special handling of emqtt race condition
the race should have been fixed in emqtt 1.8.7
2023-09-18 17:27:34 +02:00
Zaiming (Stone) Shi 46557c3462 fix(emqtt): upgrade from 1.8.6 to 1.8.7 2023-09-18 17:04:23 +02:00
Zaiming (Stone) Shi f66d9e76fe fix(emqx_channel): aovid logging frame errors as exceptions 2023-09-18 09:04:00 +02:00
Zaiming (Stone) Shi abbee4bc47 fix(esockd): upgrade esockd from 5.9.6 to 5.9.7 2023-09-17 19:44:04 +02:00
Zaiming (Stone) Shi 985f0e6994 chore(gen_rpc): upgrade from 2.8.1 to 3.1.0 2023-09-17 19:10:13 +02:00
Thales Macedo Garitezi 2aae3856a3
Merge pull request #11596 from thalesmg/fix-typespec-typos-m-20230912
fix: fix typespec typos
2023-09-15 09:24:42 -03:00
Ilya Averyanov 1959e15703 chore: simplify session eviction for node rebalance 2023-09-15 11:46:51 +02:00
Thales Macedo Garitezi b30bcf32bd feat(ds): introduce keyspace concept
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
Ilya Averyanov 11c39c4b6a chore: simplify session eviction for node rebalance 2023-09-14 18:28:25 +03:00
Thales Macedo Garitezi 6be0e14e8c fix: fix typespec typos 2023-09-14 10:21:05 -03:00
Thales Macedo Garitezi e41f7dd68c
Merge pull request #11598 from thalesmg/ds-fix-ps-router-m-20230912
fixes and improvements to persistent session router
2023-09-14 10:00:58 -03:00
firest 3eb46be2ae fix(banned): fix typos 2023-09-14 14:01:59 +08:00
Thales Macedo Garitezi 803b69d878
refactor: use more consistent return types
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2023-09-13 10:49:45 -03:00
Thales Macedo Garitezi f55f6dc779 docs(ps_session): add notes explaining rationale about operation order and consistency 2023-09-13 10:12:27 -03:00
Thales Macedo Garitezi e081abb9e6 perf(ps_router): only check if there are any routes when deciding whether to persist message 2023-09-13 10:02:39 -03:00
Thales Macedo Garitezi cae9ae1fab fix(ps_router): use `disc_copies` for storing persistent session routes 2023-09-13 10:02:39 -03:00
William Yang ea73d674b9 chore: bump version to e5.2.1-alpha.2 2023-09-13 14:55:17 +02:00
Serge Tupchii af015bc2bb chore(ekka): bump version to 0.15.13 2023-09-13 10:11:14 +03:00
William Yang 6ef296268e chore: bump version to e5.2.1-alpha.1 2023-09-12 15:24:02 +02:00
Thales Macedo Garitezi 59b94788bf refactor: address review comments 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi 64fcdb671d test: fix a bunch of inter-suite interferences 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi f1294736b7 feat(ds): add filter for message persistence
Fixes https://emqx.atlassian.net/browse/EMQX-10520
2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi 7805cc8c9b fix(session): check if session is persistent 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi f490da71cd chore: prepare v5.2.0-build.1 2023-09-11 16:09:17 -03:00
Thales Macedo Garitezi d0f1c75651 chore: bump apps vsns 2023-09-11 16:06:53 -03:00
Thales Macedo Garitezi 223d47a6ac Merge branch 'master' into sync-m-r52-20230911 2023-09-11 16:04:28 -03:00
Thales Macedo Garitezi 063913f245 fix(session): remove recently added `iterators` field from `#session{}` record
Fixes https://emqx.atlassian.net/browse/EMQX-10945

This could lead to `badrecord` errors being raised if a takeover were to happen during a
rolling cluster upgrade, as the old nodes could receive a record with more fields than
expected.
2023-09-11 14:46:12 -03:00
Zaiming (Stone) Shi 7010ac358c chore: bump release version to v5.2.0 2023-09-08 16:31:52 +02:00
Andrew Mayorov 3d133801af
fix(router): use `ets:new/2` options compatible with OTP 24
While essentially keeping the same effect.
2023-09-08 18:04:29 +04:00
Zaiming (Stone) Shi f3bebb0ba2 Merge remote-tracking branch 'origin/master' into 0907-sync-master-to-release-52 2023-09-07 18:51:30 +02:00
Zaiming (Stone) Shi db2ec55aa0 chore: bump release version to official e5.2.0 2023-09-07 09:40:15 +02:00
Zaiming (Stone) Shi 7c2f87fabe test: merge broker and router boot modules 2023-09-06 21:36:16 +02:00
Zaiming (Stone) Shi e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
Thales Macedo Garitezi 954b73a9aa
Merge pull request #11533 from thalesmg/ds-unsubscribe-m-20230828
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
2023-09-05 09:47:17 -03:00
ieQu1 01a128878f chore(mria): Bump mria to 0.6.1 2023-09-05 11:08:32 +02:00
Andrew Mayorov d26995a2aa
Merge pull request #11524 from keynslug/ft/EMQX-10713/unified-route-tab
feat(router): add routing storage schema with 2 tables
2023-09-05 12:38:00 +04:00
Thales Macedo Garitezi 23d63f5e01 refactor: fork clientid type for `emqx_ds:session_id/0` 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 60ae3c15c8 refactor: disregard impossible case 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 73d4248334 test(cth_cluster): always allocate listener ports 2023-09-04 11:22:26 -03:00
Andrew Mayorov 9f0f218387
test(router): simplify cluster-related testcase in helper testsuite 2023-09-04 18:05:59 +04:00
Andrew Mayorov 7d26b7bc1a
fix(router): emit clearer error when conflicting schemas in use 2023-09-04 16:37:38 +04:00
Thales Macedo Garitezi d33ad8a46f feat: cluster purge (r5.2)
Port of https://github.com/emqx/emqx/pull/11447

targeting `release-52`

Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-09-04 09:15:28 -03:00
Andrew Mayorov 893f69617a
chore(test): drop obsolete TODOs 2023-09-04 14:40:49 +04:00
Andrew Mayorov f0a0c7d4b1
fix(cthsuite): avoid duplicate application env entries
Because `application:set_env/1` complains loudly if there are
duplicates.
2023-09-04 14:40:49 +04:00
Andrew Mayorov 5024304bf9
fix(router): wait for tables replicate before choosing schema vsn 2023-09-04 14:40:48 +04:00
Ivan Dyachkov 24230a64df chore: bump app versions 2023-09-04 11:39:21 +02:00
Ivan Dyachkov b2f57636bd Merge remote-tracking branch 'upstream/release-51' into 0904-sync-release-51 2023-09-04 11:34:22 +02:00
Serge Tupchii f0c75d97e1 chore: upgrade emqx_http_lib to 0.5.3
0.5.3 release includes a fix to parse <host>:<port> URLs using the default http scheme.
2023-09-01 19:56:07 +03:00
Andrew Mayorov 4ab5f8374b
chore(schema): mark routing storage schema readOnly
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-01 19:58:31 +04:00
Andrew Mayorov ebf13c90ea
test(router): make sort stable in e2e routing testcase 2023-09-01 19:48:56 +04:00
Andrew Mayorov 545f1c84a6
fix(cth): do not allocate ports for `emqx` app by default
This causes tricky and impressively hard to track side effects down
the line. Namely, loading `emqx_schema` _after_ `emqx_conf_schema`
(as part of cluster node startup sequence) leads to a couple of
schema root rewrites, because `emqx_schema` defines similar config
roots yet slightly differently (e.g. `authorization`).
2023-09-01 17:23:34 +04:00
Andrew Mayorov 0a879bffd1
test(router): add e2e testcase for cluster-wide routing
Which verifies that both router storage schemas work correctly.
2023-09-01 13:49:12 +04:00
Andrew Mayorov eb0385a28f
chore(router): separate test-only function exports 2023-09-01 13:49:11 +04:00
Zaiming (Stone) Shi c20ba0572a chore: bump to 5.2.0-alpha.4 2023-08-31 15:47:39 +02:00
Zaiming (Stone) Shi 876d539336 Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table 2023-08-31 15:46:33 +02:00
Andrew Mayorov fb094e1d47
test(router): avoid testsuite collisions 2023-08-31 13:36:01 +04:00
Thales Macedo Garitezi 4dfbc859f9 fix(plugins): update plugin order on whole cluster
Fixes https://emqx.atlassian.net/browse/EMQX-10879
2023-08-30 16:39:52 -03:00
Andrew Mayorov 6b9cb06334
fix(router): add / refine some comments and log messages
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-30 19:51:31 +04:00
Thales Macedo Garitezi 922ca5e141 feat(ds): close iterators when handling `UNSUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9742
2023-08-30 11:14:28 -03:00
Thales Macedo Garitezi 9c6dd30f44 feat(session): store iterator ids in session record 2023-08-30 11:13:55 -03:00
Andrew Mayorov f8923ab419
Merge pull request #11539 from keynslug/ft/cth-desc-workdir
feat(cthsuite): add function to determine workdir of testrun
2023-08-30 17:15:55 +04:00
lafirest 8b3a9072f9
Merge pull request #11532 from lafirest/fix/frame_parse
fix(frame): improve some error reasons when parsing invalid packet
2023-08-30 19:10:45 +08:00
firest 804443ba40 chore: fix typos 2023-08-30 11:22:09 +08:00
Ilya Averyanov 2ef60db147
Merge pull request #11481 from savonarola/0817-simplify-async-responses
chore(ft): refactor async reply mechanism
2023-08-30 01:20:09 +03:00
Ilya Averyanov 54ac4a8527 chore(ft): tidy up the code according to the review 2023-08-29 23:21:36 +03:00
Andrew Mayorov 0e770bdc95
test: switch `emqx_broker_SUITE` to use new cth tooling 2023-08-29 22:05:36 +04:00
Andrew Mayorov 3268093881
feat(cth): add module-level documenation 2023-08-29 22:05:36 +04:00
Andrew Mayorov f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
Andrew Mayorov 063d6200c8
feat(router): enable using 2 tables to store routes
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.

Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
2023-08-29 17:41:59 +04:00
Andrew Mayorov 270fd107b2
feat(triesearch): allow reusing filter tokenization result
As a kind of micro-optimization.
2023-08-29 17:31:26 +04:00
zhongwencool 4e7ba5d35f chore: ciphers should allow empty space(comma_separated_binary) 2023-08-29 11:34:40 +08:00
firest 6722722522 fix(frame): improve some error reasons when parsing invalid packet 2023-08-29 10:53:20 +08:00
SergeTupchiy c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log
fix(emqx_trace): don't download empty trace log file
2023-08-28 22:17:00 +03:00
Serge Tupchii 0072749143 fix(emqx_trace): don't download empty trace log file
Closes: EMQX-10274
2023-08-28 19:25:55 +03:00
Thales Macedo Garitezi f988de4ff4 feat(purge): clear delayed messages 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi 9bded07834 feat: cluster purge
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
Andrew Mayorov 8d2ebdea7e
fix(router): generalize config option and make effects visible 2023-08-28 14:07:46 +04:00
Andrew Mayorov e85789306b
chore(router): drop test-only `has_routes/1`
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
Andrew Mayorov 5d51687dbf
feat(router): add unified routing table config option 2023-08-28 14:07:45 +04:00
Andrew Mayorov 33e5e1ba57
feat(router): add unified routing table 2023-08-28 14:07:44 +04:00
Andrew Mayorov 166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 84e40fb6fe
test(trie): add more involved route add + delete case
That shows how current local trie implementation breaks because
of lack of refcounting.
2023-08-28 13:52:59 +04:00
Andrew Mayorov dcb63440bc
refactor(mgmt): avoid dealing with router tab directly
Instead, contain all this behind `emqx_router` module interface.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 9eccfa0909
refactor(router): isolate cleanup logic in router module 2023-08-28 13:52:59 +04:00
Ilya Averyanov 39a48179ea chore(emqx_channel): use macros for reply construction 2023-08-28 11:57:44 +03:00
Ilya Averyanov b8cacd2833 chore(ft): add tests for async reply registry 2023-08-28 11:57:44 +03:00
Andrew Mayorov b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
Ilya Averyanov b0d4a22aa8 chore(ft): refactor async reply mechanism 2023-08-27 21:54:23 +03:00
Thales Macedo Garitezi 33a0048155 refactor: move logic to `ensure_iterator` 2023-08-25 15:24:53 -03:00
Thales Macedo Garitezi f15f59650d test: rm obselete workaround code 2023-08-25 13:49:33 -03:00
Andrew Mayorov d5cff533e3
chore(topicidx): drop TODO comment 2023-08-25 16:44:03 +04:00
Andrew Mayorov ca59a87d47
chore(topicidx): refine example of wildcard compare 2023-08-25 16:44:03 +04:00
Zaiming (Stone) Shi 558402a68e
chore(topic_index): add topic validation 2023-08-25 16:44:03 +04:00
Andrew Mayorov 33ed53bb6a
refactor(topicidx): simplify `compare/3` further
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
2023-08-25 16:34:45 +04:00
Andrew Mayorov fc37d235c7
refactor(topicidx): simplify seek instructions
Which also avoids comparing filter tail repeatedly when evaluating
it.
2023-08-25 15:24:06 +04:00
zhongwencool c8af5a702c test: remove connection_closed from test 2023-08-25 15:53:35 +08:00
zhongwencool fc1738188e fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero 2023-08-25 15:18:34 +08:00
Andrew Mayorov cf45e80c71
feat(topicidx): iterate on trie search implementation
This improves matching performance and decreases GC pressure on
synthetic workloads.
2023-08-25 01:47:11 +04:00
Kjell Winblad 9fe5080705 build: bump versions before release 2023-08-24 16:46:27 +02:00
Zaiming (Stone) Shi 62423b0b12 refactor(topic_index): less special handling for leading $ words 2023-08-24 13:30:10 +02:00
Zaiming (Stone) Shi a30d87e14f refactor(topic_index): remove more unnecessary next calls
also avoid using records (setelement) for recursive return values
2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi a1e6635614 refactor(topic_index): no forced ceiling entry in index table 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi f4c8c6be55 refactor(topic_index): optimize trie-search performance 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi 6b152b3cb7 test: add more debug output 2023-08-24 12:25:04 +02:00
zhongwencool 2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics
Prometheus metrics
2023-08-24 12:06:32 +08:00
zhongwencool 8d2be27ade chore: add changlog for 11466 2023-08-24 06:46:50 +08:00
zhongwencool 84dd3b83bb fix: hocon converter not working when make_serializable is true(check_plain) 2023-08-24 06:46:47 +08:00
zhongwencool d287b736ec chore: add changelog for 11497 2023-08-23 17:02:30 +08:00
zhongwencool 6bb15541c7 feat: rename olp metrics to overload_protection metrics 2023-08-23 15:25:54 +08:00
zhongwencool b9a97923c4 feat: add some authz/connection missing metrics 2023-08-23 14:47:53 +08:00
zhongwencool ae10415fc3 feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security 2023-08-23 09:03:22 +08:00
Thales Macedo Garitezi ee2897e5de test(refactor): move test to integration tests dir 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c1f49abad2 test: fix inter-suite flakiness 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi dad27091be test: rm custom option 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 65085d012b refactor: rename fn 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c46b8de938 test: remove unused things, refactor some functions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8c73b06e1 docs: add comment about future test failure 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 6de0bbe76a test(refactor): always allocate listeners for emqx app 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi a15405a800 test: fix assertions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c74abe79d0 refactor: reduce arity 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 021755b82b refactor: rm iterators from DS `#session{}` record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c28c6d1b7e fix: ensure iterator is opened 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 33ddbe80ad refactor: remove persistence leakeage from emqx_cm level 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8d7bb9a67 refactor: rename module 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 9463e271c0 feat(ds): open iterators when handling `SUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
Ilya Averyanov ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
Ivan Dyachkov e2b9d905d7
Merge pull request #11465 from id/0718-e5.2.0-alpha.3 2023-08-17 20:59:13 +02:00
Andrew Mayorov 75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-17 20:43:09 +04:00
Ivan Dyachkov a0c2fe8cc1 chore: e5.2.0-alpha.3 2023-08-17 17:55:56 +02:00
lafirest 5e448f5a02
Merge pull request #11446 from lafirest/refactor/calendar
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-17 07:40:44 +08:00
Andrew Mayorov 5d79823891
perf(topicidx): preserve next key on the stack to reuse later
This should further optimize the number of `ets:next/2` calls
required for a single match query.
2023-08-17 01:13:19 +04:00
Thales Macedo Garitezi f03733ba6b
Merge pull request #11460 from thalesmg/integration-tests-dir-20230816
ci: add `integration_test` dir to `emqx` application
2023-08-16 17:47:14 -03:00
Ilya Averyanov 3943830eae chore(auth): remove unnecessary atom conversions 2023-08-16 20:50:35 +03:00
Ilya Averyanov 0dae040047 chore(auth): restore injected schema key position 2023-08-16 20:50:35 +03:00
Ilya Averyanov db31e5f0d9 chore(auth): tidy up the code 2023-08-16 20:50:35 +03:00
Ilya Averyanov 67e06b3171 chore(auth): make schema injection be more universal 2023-08-16 20:41:05 +03:00
Ilya Averyanov f00f4568dc chore(auth): simplify schema injection 2023-08-16 20:41:05 +03:00
Ilya Averyanov 8a3e8ee528 chore(auth): add config propagation tests for authn 2023-08-16 20:41:05 +03:00
Ilya Averyanov 128aa19d47 chore(auth): improve emqx_config_handler to replace adhoc config updates 2023-08-16 20:41:05 +03:00
Ilya Averyanov 8213aa42c9 chore(auth): move authn leftovers to the authn app 2023-08-16 18:52:14 +03:00
Ilya Averyanov ca8c1e3ef8 chore(auth): removed direct usages of authn from emqx core app 2023-08-16 18:52:14 +03:00
Ilya Averyanov ce2b159022 chore(auth): make pre_hook_authenticate be part of emqx_access_control domain 2023-08-16 18:52:14 +03:00
zhongwencool e6f0dead9e
Merge pull request #11445 from zhongwencool/remove-os-mon-from-windows
fix: remove os_mon application in Windows release
2023-08-16 23:30:24 +08:00
Thales Macedo Garitezi a6a023eff7 ci: add `integration_test` dir to `emqx` application
This gives us a place where to put tests that exercise multiple umbrella application,
which is more closely related to `emqx`, but in a way that doesn't affect the standalone
app tests.
2023-08-16 11:48:47 -03:00
Andrew Mayorov c84ee66d80
Merge pull request #11450 from keynslug/fix/dirty-testsuite-flapping
fix(test): switch `emqx_flapping_SUITE` to cth tooling
2023-08-16 17:20:31 +04:00
zhongwencool 78356d2625 chore: start os_mon in handle_continue/2 2023-08-16 20:54:53 +08:00
zhongwencool 8b23ee86b3 fix: hide cpu_status if os_check is not supported 2023-08-16 20:49:28 +08:00
zhongwencool b817e03c08 fix: start os_mon application temporary 2023-08-16 20:49:27 +08:00
zhongwencool 848eb7e3c3 fix: remove os_mon application in Windows release 2023-08-16 20:49:27 +08:00
zhongwencool b733adca06 fix: allow empty cacertfile pem 2023-08-16 20:15:21 +08:00
zhongwencool 1af0407c48 fix: don't crash when debug huge payload 2023-08-16 15:35:09 +08:00
firest b08102269a refactor(calendar): refactor datetime-related code and remove redundant 2023-08-16 10:55:46 +08:00
Andrew Mayorov 0b21b87296
fix(test): switch `emqx_flapping_SUITE` to cth tooling 2023-08-15 21:29:42 +04:00
Andrew Mayorov 5560e6ed4f
fix(cthsuite): also prevent `emqx` from loading default conf 2023-08-15 21:25:39 +04:00
Thales Macedo Garitezi 5d707c8b7d
Merge pull request #11403 from thalesmg/gcp-produ-attr-20230728
feat(gcp_producer): add support for defining message attributes and ordering key
2023-08-15 13:22:31 -03:00
Zaiming (Stone) Shi fa9e9f3348
Merge pull request #11442 from zmstone/0814-sync-release-52-to-master
0814 sync release 52 to master
2023-08-15 16:21:55 +02:00
Andrew Mayorov e39bbf4c49
chore(topicidx): add more descriptive comments and specs
To (hopefully) better illustrate what is happening there.
2023-08-15 16:55:48 +04:00
Thales Macedo Garitezi 82b8538041 feat(gcp_producer): add support for defining message attributes and ordering key
Fixes https://emqx.atlassian.net/browse/EMQX-10652
2023-08-14 10:33:17 -03:00
lafirest 1bda8020f5
Merge pull request #11436 from lafirest/feat/banned_clear
feat(banned): add a new API used to clear all banned data
2023-08-14 20:48:43 +08:00
Andrew Mayorov 9a249e4b01
test(topicidx): increase test coverage 2023-08-14 15:36:58 +04:00
Andrew Mayorov fd0986071c
perf(topicidx): implement fast-forwarding prefixes
This should give less `ets:next/2` calls in general and much less
when index has relatively small number of long non-wildcard topics.
2023-08-14 15:36:58 +04:00
Andrew Mayorov 0c7bdbdab4
test(topicidx): add property test
Co-Authored-By: JianBo He <heeejianbo@gmail.com>
2023-08-14 15:36:57 +04:00
Andrew Mayorov 48a50c9137
fix(topicidx): fix missing matches when 'a/b' and 'a/b/#' both exist
Thanks to @HJianBo for spotting this issue. The approach to fix it
is different though: we try to keep the "recurrency branch factor"
to a minimum, instead introducing new condition for the case when
filter does not match, but iteration with `ets:next/2` is not yet
finished for the prefix.

Co-Authored-By: JianBo He <heeejianbo@gmail.com>
2023-08-14 15:36:57 +04:00
Andrew Mayorov 6a13406363
fix(topicidx): use custom topic words to keep required ordering
Otherwise, topic with empty tokens (e.g. `a/b///c`) would have
some of their tokens ordered before `#` / `+`, because empty
token was represented as empty atom (`''`).
2023-08-14 15:36:57 +04:00
Andrew Mayorov 28bcb394d1
fix(topicidx): allow to return matches unique by record id 2023-08-14 15:36:57 +04:00
Andrew Mayorov 8feda315f6
feat(index): add topic index facility
Somewhat similar to `emqx_trie` in design and logic, yet built on
top of a single, potentially pre-existing table.
2023-08-14 15:36:54 +04:00
Zaiming (Stone) Shi 970ff28200 Merge remote-tracking branch 'origin/master' into release-52 2023-08-14 11:50:40 +02:00
zhongwencool 41180170f4 fix: update error msg 2023-08-14 15:45:57 +08:00
zhongwencool e10f9e5e9b feat: change mqtt.max_packet_size type from string to bytesize 2023-08-14 10:10:46 +08:00
lafirest b8d3dcf59a
Merge pull request #11424 from lafirest/fix/epoch_maximum_val
fix(datetime): make sure the epoch is not larger than the maximum supported value
2023-08-11 20:37:56 +08:00
firest 749c2d075f chore: update changes && bump app version 2023-08-11 17:57:27 +08:00
firest bc417a6764 fix(datetime): make sure the epoch is not larger than the maximum supported value 2023-08-11 17:57:21 +08:00
firest 2fc0468e0c feat(banned): add a new API used to clear all banned data 2023-08-11 17:25:49 +08:00
Ivan Dyachkov 7852af896c chore: e5.2.0-alpha.1 2023-08-11 11:16:30 +02:00
Ivan Dyachkov bab05b6613
Merge pull request #11433 from id/0811-ci-upgrade-push-helm-action 2023-08-11 11:03:00 +02:00
Ivan Dyachkov b6084b0001 chore: bump apps/emqx version 2023-08-11 09:19:51 +02:00
lafirest 4eaced241d
Merge pull request #11423 from lafirest/chore/limiter_utils
fix(limiter): simplify the code of the limiter
2023-08-11 15:17:42 +08:00
Ivan Dyachkov 801c93dd46 chore: v5.1.5-build.3 2023-08-10 12:56:28 +02:00
Ivan Dyachkov 6ead09b28d chore: merge 'upstream/master' 2023-08-10 12:56:00 +02:00
firest 6dbddfb089 fix(limiter): simplify the code of the limiter
move runtime code out from the schema
2023-08-10 18:20:05 +08:00
zhongwencool aa798e531e
Merge pull request #11409 from zhongwencool/opentelemetry-metrics
feat: support opentelemetry metrics
2023-08-10 14:35:25 +08:00
zhongwencool 5a4dd3a5e5 fix: truncate large payload 2023-08-10 09:35:12 +08:00
zhongwencool d5fe891961 fix: don't be kill when send large payload when log is debug 2023-08-10 09:22:30 +08:00
Thales Macedo Garitezi bb8ce68c2e docs: Generate changelog for v5.1.5-patch.2 2023-08-09 16:38:28 -03:00
Thales Macedo Garitezi 95df958f10 docs: Generate changelog for v5.1.5-patch.1 2023-08-09 13:18:09 -03:00
Thales Macedo Garitezi cce0d144f9 Merge branch 'master' into release-v515-patch1-20230809 2023-08-09 13:08:43 -03:00
zhongwencool 3933227636 feat: support opentelemetry metrics 2023-08-09 20:23:40 +08:00
lafirest bef8eddad3
Merge pull request #11405 from lafirest/fix/kal_kialo
fix(calendar): make date parse error reason more sense
2023-08-09 14:42:08 +08:00
Thales Macedo Garitezi fdfdb105f0 docs: Generate changelog for v5.1.5 2023-08-08 10:33:34 -03:00
Thales Macedo Garitezi a1ffd0263d fix(cacerts): add `cacerts` back as deprecated to allow upgrading from older versions
See https://github.com/emqx/emqx/issues/11380
2023-08-08 10:13:13 -03:00
firest 9f38f5f26a fix(calendar): make date parse error reason more sense 2023-08-08 14:28:38 +08:00
lafirest 2b03436552
Merge pull request #11392 from lafirest/feat/ldap_authz
feat(ldap-authz): integrate the LDAP authorization
2023-08-07 11:12:05 +08:00
zhongwencool 665695a977
Merge pull request #11388 from SergeTupchiy/EMQX-10703-fix-replicant-crash-when-core-terminates-abnormally
fix: increase emqx_router_sup restart intensity
2023-08-06 14:40:32 +08:00
Serge Tupchii 2b7798608d fix(emqx_router_helper): don't cleanup down node on a replicant
The cleanup on a replicant node is redundant, as Mria would delegate this delete op
to a core node (via RPC), and the core node is expected to receive the same
`nodedown` message and process it.
2023-08-04 16:55:36 +03:00
SergeTupchiy 2044326734
Merge pull request #11390 from SergeTupchiy/EMQX-10661-improve-perf-high-latency-cluster-network
EMQX-10661 improve performance on high latency cluster network
2023-08-04 16:41:08 +03:00
firest b24a9d5343 chore(ldap-authz): update apps version && changes 2023-08-04 18:14:09 +08:00