Commit Graph

3661 Commits

Author SHA1 Message Date
ieQu1 f5b9bd30aa fix(sessds): Apply review remarks 2024-01-26 17:49:33 +01:00
ieQu1 39857626ce test(sessds): Fix failing tests 2024-01-26 17:49:33 +01:00
ieQu1 e7b03cdc59 test(sessds): Create a property-based test for the session state 2024-01-26 17:49:33 +01:00
ieQu1 893656f092 fix(sessds): Fix session garbage collection after the refactoring 2024-01-26 17:49:33 +01:00
ieQu1 963df8f941 fix(sessds): Delete the routes when the session expires 2024-01-26 17:49:33 +01:00
ieQu1 cff6c15e13 fix(sessds): Store the QoS as the MSB of the packet ID 2024-01-26 17:49:33 +01:00
ieQu1 978a3bfef3 refactor(sessds): Simplify representation of QoS tracks 2024-01-26 17:49:33 +01:00
ieQu1 1b4f69b44d refactor(sessds): Simplify data structure of ds_state pmap datatype 2024-01-26 17:49:33 +01:00
ieQu1 4f4831fe7f refactor(sessds): Factor out stream scheduler into its own module 2024-01-26 17:49:33 +01:00
ieQu1 82ef34998a feat(sessds): Index streams by a unique subid 2024-01-26 17:49:33 +01:00
ieQu1 3fb2064ea4 test(sessds): Add property-based tests for seqno generator 2024-01-26 17:49:33 +01:00
ieQu1 8e8d3af096 fix(sessds): Refactor emqx_persistent_session_ds to use CRUD module 2024-01-26 17:49:33 +01:00
ieQu1 2d08aa88d8 refactor(ds): Create a CRUD module for the persistent session 2024-01-26 17:49:33 +01:00
Zaiming (Stone) Shi 2a40152721 Merge remote-tracking branch 'origin/release-55' into sync-release-55 2024-01-26 16:51:13 +01:00
William Yang 933c00c7ad
Merge pull request #12392 from qzhuyan/perf/william/ws-no-utf8-validate
perf: new ws listener option to disable UTF-8 validation
2024-01-26 16:01:04 +01:00
ieQu1 0933dc818e chore(ekka): Bump version to 0.18.4
Don't escalate the errors in etcd node discovery to the node level.

Fixes: #12255
2024-01-26 14:25:16 +01:00
William Yang 726302ef6a perf: new ws listener option to disable UTF-8 validation 2024-01-26 09:55:17 +01:00
JianBo He e29a80d568
Merge pull request #12388 from qzhuyan/dev/william/per-listn-conn-cnt
feat: quic per listener conn count
2024-01-26 09:01:39 +08:00
Thales Macedo Garitezi 8e31afe6c2 fix(ds): don't make data dir part of the schema
The data directory was ending up being persisted in the database schema.  This led to
issues when opening the DB on different nodes.
2024-01-25 14:44:06 -03:00
Thales Macedo Garitezi b5567d9d9f
Merge pull request #12389 from thalesmg/ci-peer-cover-m-20240124
ci: start/flush cover when using peer
2024-01-25 10:00:50 -03:00
zhongwencool 4f8accc31b
Merge branch 'master' into sync-r55-m-20240124 2024-01-25 16:09:05 +08:00
Thales Macedo Garitezi 846ad42a65 fix(ds): don't use env var for data dir default value
Fixes https://github.com/emqx/emqx/pull/12380
2024-01-25 16:01:44 +08:00
Thales Macedo Garitezi 024ffe2909 test(ds): reduce flakiness 2024-01-25 16:01:44 +08:00
zhongwencool 186e1591df chore: document api-key path api only support bearerAuth 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi 878c9ee8b1 fix(ds): do not count persistent session-only routed messages as dropped
Fixes https://emqx.atlassian.net/browse/EMQX-11539
2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi d323fc7c27 feat(ps): add message gc 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi 4a0fd756ae feat(ds): add `list_generations` and `drop_generation` APIs 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi fb39e1eacc feat(ds): allow customizing the data directory
The storage expectations for the RocksDB DB may be different from our usual data
directory.  Also, it may consume a lot more storage than other data.

This allows customizing the data directory for the builtin DS storage backend.

Note: if the cluster was already initialized using a directory path, changing that config
will have no effect.  This path is currently persisted in mnesia and used when reopening
the DB.
2024-01-25 16:01:42 +08:00
Serge Tupchii a8c6280a5e test(emqx_cm_SUITE): use one helper function: `emqx_pool:flush_async_tasks/1` 2024-01-25 16:01:42 +08:00
Serge Tupchii f52cc93d9d perf(emqx_cm): use a dedicated pool for channel cleanup
This is to isolate channels cleanup from other async tasks (like routes cleanup),
as channels cleanup can be quite slow under high network latency conditions.

Fixes: EMQX-11743
2024-01-25 16:01:42 +08:00
JianBo He 52a7b45d97 fix: upgrade ekka to 0.18.3 to ensure the leader correct 2024-01-25 14:47:59 +08:00
Thales Macedo Garitezi ae387d1812 ci: start/flush cover when using peer
`cover` is not automatically started by `peer`.  Without starting/flushing it, we don't
get coverage data from peer nodes.
2024-01-24 18:02:47 -03:00
William Yang 9fb6688801 fix(quic): Per listener conn count 2024-01-24 21:36:25 +01:00
ieQu1 7b5f2948fe test(ds): Fix flaky testcase 2024-01-24 19:33:30 +01:00
ieQu1 eee221f1d0 feat(ds): Make egress batching configurable 2024-01-24 19:33:30 +01:00
ieQu1 137535a821 feat(ds): Introduce egress process for the builtin backend 2024-01-24 19:33:30 +01:00
Thales Macedo Garitezi 5547a40ceb fix(ds): don't use env var for data dir default value
Fixes https://github.com/emqx/emqx/pull/12380
2024-01-24 11:39:14 -03:00
Thales Macedo Garitezi 77329209a2
Merge pull request #12368 from thalesmg/ds-inherit-wildcards-only-lts-m-20240122
perf(ds): inherit only LTS paths containing wildcards when adding a new generation
2024-01-23 17:14:51 -03:00
Thales Macedo Garitezi eecd7e084c test(ds): reduce flakiness 2024-01-23 09:47:03 -03:00
zhongwencool 94692162b5
Merge pull request #12370 from zhongwencool/api-key-auth
chore: document api-key path api only support bearerAuth
2024-01-23 14:22:35 +08:00
zhongwencool 0dbaaa5d94 chore: document api-key path api only support bearerAuth 2024-01-23 10:31:43 +08:00
Thales Macedo Garitezi 609ba7e332 fix(ds): do not count persistent session-only routed messages as dropped
Fixes https://emqx.atlassian.net/browse/EMQX-11539
2024-01-22 16:58:45 -03:00
Thales Macedo Garitezi d122340c13
Merge pull request #12338 from thalesmg/ds-message-gc-20240115
feat(ds): add message GC
2024-01-22 16:57:26 -03:00
Thales Macedo Garitezi 7035b4c8b3 feat(ps): add message gc 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi 75b08b525b feat(ds): add `list_generations` and `drop_generation` APIs 2024-01-22 14:53:17 -03:00
JimMoen 6b064dd8eb
fix(prom_stats): missing metric key
- follow https://github.com/emqx/emqx/pull/11497
2024-01-22 10:39:53 +08:00
JimMoen 57f3efde63
feat(prom_auth): cluster metrics with different format-mode 2024-01-21 00:14:48 +08:00
Thales Macedo Garitezi 57074015c6 feat(ds): allow customizing the data directory
The storage expectations for the RocksDB DB may be different from our usual data
directory.  Also, it may consume a lot more storage than other data.

This allows customizing the data directory for the builtin DS storage backend.

Note: if the cluster was already initialized using a directory path, changing that config
will have no effect.  This path is currently persisted in mnesia and used when reopening
the DB.
2024-01-19 13:07:24 -03:00
Zaiming (Stone) Shi 7d14a13b7b Merge remote-tracking branch 'origin/release-55' 2024-01-18 14:36:30 +01:00
Thales Macedo Garitezi e219c3eda1
Merge pull request #12166 from kjellwinblad/kjell/mqtt_conn_action_3/EMQX-11489
feat: refactor MQTT bridge to source, action, and connector
2024-01-18 09:28:14 -03:00
Zaiming (Stone) Shi 556092b7d0 feat(authz/prometheus): add authz cache_miss counter 2024-01-18 09:01:17 +01:00
Zaiming (Stone) Shi 85b6a3454c fix(authz): use binary() type instead of string() for cache.excludes 2024-01-18 09:01:17 +01:00
Zaiming (Stone) Shi 4fa906731d
Merge pull request #12333 from zmstone/0116-add-tags-for-action-and-connectors
feat: add 'tags' field for action and connector
2024-01-16 18:56:57 +00:00
Serge Tupchii 80e82db282 test(emqx_cm_SUITE): use one helper function: `emqx_pool:flush_async_tasks/1` 2024-01-16 19:42:37 +02:00
Serge Tupchii b472b56883 perf(emqx_cm): use a dedicated pool for channel cleanup
This is to isolate channels cleanup from other async tasks (like routes cleanup),
as channels cleanup can be quite slow under high network latency conditions.

Fixes: EMQX-11743
2024-01-16 19:08:02 +02:00
Zaiming (Stone) Shi 01d52e37c4
fix: resource tag type should be binary string
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-01-16 17:35:22 +01:00
Thales Macedo Garitezi 8f304d3456 test(bridge_v2_api): refactor suite to use CT matrix 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 6511693b2e refactor(action_api): prepare for `/sources` HTTP API 2024-01-16 10:42:52 -03:00
Zaiming (Stone) Shi 996a851cf6 chore: format username as string in log formatter 2024-01-16 14:42:10 +01:00
Zaiming (Stone) Shi 541525c50f fix(authz): fix authz result logs
prior to this fix, it's always the default authz result logged at
warning level
2024-01-16 14:40:53 +01:00
Zaiming (Stone) Shi 1fe1a62fe2 test: fix already exported function warning 2024-01-16 14:00:34 +01:00
Zaiming (Stone) Shi ab66986f16 feat: add 'tags' field for action and connector 2024-01-16 13:57:48 +01:00
JianBo He b6d0365027
Merge pull request #12267 from HJianBo/new-timeout-param-for-invite
feat(cluster): supports inviting nodes to join the cluster in an asynchronous manner
2024-01-16 10:11:20 +08:00
Zaiming (Stone) Shi 58a54adbb4 Merge remote-tracking branch 'origin/master' into release-55 2024-01-15 17:58:48 +01:00
William Yang d2d3ddb72a test(quic): listener port updates 2024-01-15 11:42:45 +01:00
William Yang c77837a9ea feat(quic): support reload with new binding port 2024-01-15 11:33:25 +01:00
JianBo He 4c40e754f4 chore: forward the async invite to leader node 2024-01-15 11:24:05 +08:00
Zaiming (Stone) Shi 61417f26d4 chore: bump versions to 5.5.x 2024-01-12 13:54:35 +01:00
Andrew Mayorov b4eac25678
chore(broker): fix code style issue
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 13:54:23 +01:00
Andrew Mayorov 2f98f1faaf
fix(route-sync): ensure batch sync preserve idemopotency 2024-01-12 11:57:26 +01:00
Andrew Mayorov e21a3497c7
chore(route-sync): turn TODO into comment 2024-01-12 11:57:26 +01:00
Andrew Mayorov 8f4758d9d4
feat(route-sync): use the smallest possible min sync delay 2024-01-12 11:57:26 +01:00
Andrew Mayorov 2ac6cddf19
fix(route-sync): handle batch sync errors gracefully 2024-01-12 11:57:25 +01:00
Andrew Mayorov a28fc7bfa8
feat(route-sync): do not run empty batches 2024-01-12 11:57:25 +01:00
Andrew Mayorov 2dffd44985
feat(route-sync): allow to enable syncer pool only on cores/replicants
So we would able to roll it out and test more gradually.
2024-01-12 11:57:25 +01:00
Andrew Mayorov 884f784c1c
refactor(router): don't `emqx_broker_helper:monitor/1` in batches
As per out current understanding, this doesn't changes much in terms of
observability, since other nodes call `ekka:monitor(membership)` anyway,
so they will observe nodedowns without explicitly writing an entry into
the `?ROUTING_NODE` table.
2024-01-12 11:57:25 +01:00
Andrew Mayorov d6f731c4fc
fix(route-sync): use public function as mria activity target 2024-01-12 11:57:25 +01:00
Andrew Mayorov 7d037cfe91
chore(route-sync): clarify why wait indefinitely for reply 2024-01-12 11:57:24 +01:00
Andrew Mayorov 498b7a922d
chore(syncer): fix code style issues
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 11:57:24 +01:00
Andrew Mayorov 0b3f5f7c37
feat(syncer): allow to turn syncer pool on/off through config 2024-01-12 11:57:24 +01:00
Andrew Mayorov 38e13f2337
fix(syncer): start syncer pool before broker pool
As the latter depends on the former.
2024-01-12 11:57:24 +01:00
Andrew Mayorov a1ccf85c66
test(routesync): verify that syncer preserves consistency
Under a highly concurrent load. Be aware that this testcase is not
deterministic.
2024-01-12 11:57:24 +01:00
Andrew Mayorov 5aeff20f8b
fix(routesync): ensure causal relationships are preserved
At the cost of strict FIFO semantics though.
2024-01-12 11:57:23 +01:00
Andrew Mayorov 54f8b47455
feat(routing): add route sync process pool
Dedicated to synchronizing local state updates with the global view
of the routing state.
2024-01-12 11:57:18 +01:00
JianBo He 9e85b53c39
Merge pull request #12296 from HJianBo/fix-delayed-message-http-api-500
fix(delayed): fix http 500 error
2024-01-12 09:37:41 +08:00
Ilya Averyanov 7b0b2a0527 fix(conn): avoid storing incomplete channel info 2024-01-11 18:08:13 +03:00
JianBo He 596607d549 chore: update bpapi.versions 2024-01-11 14:56:02 +08:00
Serge Tupchii 965ce5d446 perf: use mria:match_delete/2 to cleanup routes
This must be much more network efficient since both Mria and Mnesia
need only to replicate one op, e.g.: `{MatchPattern, clear_table}`
instead of replicating one per each key to be deleted.
2024-01-10 16:23:58 +02:00
Serge Tupchii febaaefc38 chore: upgrade ekka to 0.18.1 (mria 0.8.1)
Mria 0.8.1 release implements `mria:match_delete/2` function
2024-01-10 16:23:58 +02:00
Zaiming (Stone) Shi 7c8a36fc06
Merge pull request #12289 from zmstone/0108-support-acl-cache-masks
0108 support acl cache excludes
2024-01-10 13:40:29 +00:00
Zaiming (Stone) Shi 9e8a67fd68 feat: support authz cache exclusion config
now one can configure a list of topic-filters to avoid
caching ACL check results

for example

authorization.cache.excludes = ["nocache/#"]

this means ACL check results for topics having 'nocache/' prefix
will not be cached
2024-01-10 13:52:00 +01:00
Andrew Mayorov ff2fb95e5d
fix(conf): bump to hocon 0.40.4
Which includes a bugfix for undefined sensitive values obfuscation.
2024-01-10 13:09:08 +01:00
Zaiming (Stone) Shi 98e94d8619 docs: fix typo in comments 2024-01-09 19:16:55 +01:00
William Yang 2201dec2f4 feat(quic): 3s cutoff time for fast_close 2024-01-09 17:29:28 +01:00
William Yang 2483a4ecff feat(quic): support listener config reload 2024-01-09 17:29:25 +01:00
William Yang 973f40041d chore: bump 0.0.311 2024-01-09 09:35:03 +01:00
Ivan Dyachkov 06117c3a33 Merge remote-tracking branch 'upstream/release-54' into 0105-sync-r54 2024-01-05 14:20:38 +01:00
Ivan Dyachkov d04ac08d1d chore: release 5.4.1 2024-01-05 11:48:28 +01:00
Ivan Dyachkov d4c83cf463 chore: tag 5.4.1-alpha.1 2024-01-02 14:37:45 +01:00
Andrew Mayorov a87df28dfc
Merge pull request #12243 from keynslug/fix/EMQX-11483/route-async-repl
fix(router): rely on local sync state updates to propagate routes
2023-12-29 16:31:00 +01:00
Andrew Mayorov 5c91984ad7
chore: leave comment describing need for tighter synchronization 2023-12-29 14:04:48 +01:00
JimMoen 5e100f52b8
style: erlfmt all `rebar.config` files and `bin/nodetool` 2023-12-29 09:08:03 +08:00
JimMoen a3f33cc8bc
chore: fix typos in code and suite 2023-12-29 09:07:51 +08:00
JimMoen ed10827343
docs: README spell 2023-12-29 09:07:25 +08:00
Andrew Mayorov b9627c420f
fix(router): rely on local sync state updates to propagate routes
Instead of (potentially lagging) global table state itself. Local
state should be the only source of truth anyway. Moreover,
synchronize parts of local state update logic through the broker
pool to ensure consistency.

The problem was that `emqx_router:has_route/2` check may observe a
stale route, which deletion is not yet replicated from the core node
to the local replicant node.

For example:
1. The only one subscriber per a given topic A unsubscribes from
   a replicant node.
2. The route to A is deleted by the emqx_broker.
3. Mria makes RPC to a core node, it succeeds and returns.
4. The client resubscribes or another client subscribes to the same
   topic A.
5. emqx_broker tries to add a route again:
   `emqx_router:do_add_route(Topic)`.
6. `emqx_router` checks if the route is present.
7. The stale route is present because deletion not replicated yet,
   so no route is being added.
8. Route deletion is replicated locally but it's too late:
   we already have a local subscriber without a route.

Co-Authored-By: Serge Tupchii <serge.tupchii@protonmail.com>
2023-12-28 19:16:31 +01:00
Andrew Mayorov 331bfaa535
fix(broker): avoid reclaiming per-topic counter twice 2023-12-28 17:43:50 +01:00
Andrew Mayorov a8cd609ff4
chore(broker): simplify subscribe code path 2023-12-28 17:43:42 +01:00
Zaiming (Stone) Shi 4669c4d552 fix(bridge/mqtt): ensure short clientid
Some mqtt brokers do not allow long client IDs.
To make it compatible with this limitation, this commit
tries to limit the number of bytes under 23 with a best-effort
attempt to derive it from the bridge name.
2023-12-28 11:53:27 +01:00
Zaiming (Stone) Shi e421e9ce56 Merge remote-tracking branch 'origin/release-54' into 1222-sync-e5.4.0-build.2-to-master 2023-12-23 11:22:05 +01:00
Zaiming (Stone) Shi f36b842d94 chore: prepare for e5.4.0 retag 2023-12-23 11:18:19 +01:00
Zaiming (Stone) Shi 86a018d921 chore: bump app versions 2023-12-22 21:28:26 +01:00
Zaiming (Stone) Shi 9fdac4af0c Merge remote-tracking branch 'origin/master' into 1222-sync-e5.4.0-build.2-to-master 2023-12-22 21:27:27 +01:00
Zaiming (Stone) Shi ac6e4cf2cc fix: config update for ft 2023-12-22 19:01:14 +01:00
Zaiming (Stone) Shi 733196aa83 chore: prepare for e5.4.0-build.2 2023-12-22 18:59:46 +01:00
Zaiming (Stone) Shi 23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi 891fd972bd Merge remote-tracking branch 'origin/release-54' into 1221-prepare-bpapi-on-otp-26 2023-12-22 15:19:25 +01:00
Zaiming (Stone) Shi 5b5d4a6f68 chore: bump release version to e5.4.0-build.1 2023-12-22 14:00:12 +01:00
Zaiming (Stone) Shi 20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi 322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Zaiming (Stone) Shi ecc31217fd Merge remote-tracking branch 'origin/master' into 1221-prepare-bpapi-on-otp-26 2023-12-22 12:48:06 +01:00
Zaiming (Stone) Shi 17d3037a13 chore: add 5.4.bpapi 2023-12-22 12:47:41 +01:00
ieQu1 ec0693861a fix(static_checks): Prepare for OTP26 2023-12-22 12:02:30 +01:00
Thales Macedo Garitezi 035f5f977e Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231221 2023-12-21 17:53:05 -03:00
Zaiming (Stone) Shi 36d2929534 chore: prepare for release version 5.4.0 2023-12-21 19:51:12 +01:00
Thales Macedo Garitezi 18b69809da Merge branch 'release-54' into sync-r54-m-20231221 2023-12-21 10:05:13 -03:00
Thales Macedo Garitezi a43f49f8fa chore: prepare to tag `5.4.0-rc.1` 2023-12-20 12:43:08 -03:00
Andrew Mayorov 76c89ad372
test(gw-authz): switch to `emqx_cth_suite` + simplify testsuite
Also separate DTLS related MQTT-SN test config, so that it doesn't
leak into this test suite.
2023-12-20 14:37:49 +01:00
Andrew Mayorov 3d679f7e26
test(cth-tls): add test helpers for TLS related activities
Currently, only certificate issuing and related utilities are there.
2023-12-20 11:46:07 +01:00
Andrew Mayorov 78d3f49fe4
fix(gw): use more conservative set of DTLS options 2023-12-20 11:46:04 +01:00
Andrew Mayorov d3b32b64e1
Merge pull request #12201 from keynslug/feat/EMQX-11527/tls-hot-update
feat(listen): support hot config update of MQTT listeners
2023-12-20 09:45:12 +01:00
JianBo He cc15e3a03b
Merge pull request #12199 from zhongwencool/configs-put-bridge-v1-api
fix: support bridge v1 conf hocon-format in the put configs API.
2023-12-20 09:32:32 +08:00
Andrew Mayorov a6c26ce992
test(listen): stabilize flaky testcase 2023-12-19 23:36:52 +01:00
Andrew Mayorov baf46c9aa2
test(listen): also change CA on update and verify clients notice 2023-12-19 22:48:38 +01:00
Andrew Mayorov 7e4049620d
fix(listen): ensure limiter server state consistent with updates 2023-12-19 16:06:27 +01:00
Andrew Mayorov 4796f85dff
test(listen): verify SSL options of WSS listener are hot updateable 2023-12-19 16:06:27 +01:00
Andrew Mayorov 036048b9ab
feat(listen): support hot update of WS/WSS options as well
However, currently any transport option changes will cause listener to
restart, which slightly impacts the availability of the broker.
2023-12-19 16:06:27 +01:00
Andrew Mayorov 68bbbec556
chore: bump to esockd 5.11.1
With a typespec fix.
2023-12-19 16:06:27 +01:00
Andrew Mayorov fb2d4544b6
test(listen): verify that SSL options are hot-updateable 2023-12-19 16:06:26 +01:00
Andrew Mayorov da0f0f947e
feat(listen): support hot config update of esockd-based listeners 2023-12-19 16:06:26 +01:00
Zaiming (Stone) Shi 2b6176aa60 fix: upgrade to gen_rpc 3.3.1 which does not log cast args only arity 2023-12-19 12:46:42 +01:00
Andrew Mayorov cafd384466
chore: upgrade to esockd 5.11.0 2023-12-19 11:15:09 +01:00
zhongwencool a46b415c77 fix: support bridge v1 conf hocon-format in the put configs API. 2023-12-19 16:40:45 +08:00
Thales Macedo Garitezi cf9331a95f Merge branch 'release-54' into sync-r54-m-20231218 2023-12-18 17:21:08 -03:00
Andrew Mayorov 78f7d01b1c
test(listen): simplify test suite setup / teardown 2023-12-18 20:52:57 +01:00
Zaiming (Stone) Shi ef2cf71bea chore: bump version to 5.4.0-alpha.2 2023-12-18 13:51:21 +01:00
Zaiming (Stone) Shi a9963e043b refactor(authz): improve logging
Move authz result logging to common place.

Prior to this change, the final result is not logged when
fallback to the default authorization.no_match config value.

Aso, if the result is provided by a hook callback,
it's also not logged.

After this change, only the final result is logged.
The authz chain resutls can be traced (or logged at debug level).
2023-12-17 22:32:26 +01:00
Serge Tupchii 24db03a7df chore(emqx_opentelemetry): remove unused `event` functionality 2023-12-15 14:58:45 +02:00
Zaiming (Stone) Shi cf33b3b4b1 fix(emqx.app.src): add dependen apps in app.src 2023-12-15 10:40:33 +01:00
zhongwencool d9e179b364 fix: use microsecond precision as the primary key for audit logs to prevent duplication 2023-12-15 09:07:04 +08:00
Zaiming (Stone) Shi 406a2d7a52 test: fix shared sub test case flaky 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi fa85c3061b test: fix emqx_ocsp_cache_SUITE error message match pattern 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi 19051f639b test: ensure verify_none to ssl client opts as default value 2023-12-14 22:21:31 +01:00
Zaiming (Stone) Shi 778662de5d test(crl_cache): fix ssl options 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi dca72e0052 test: fix mtls options 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi 06bfd878a0 test: fix compile warnings in test suites 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi a0833282f2 fix(dialyzer): fix dialyzer warnings for emqx app standalone 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi b7531839cd fix(dialyzer): delete stale code 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi bcae3d67c7
Merge pull request #12169 from ieQu1/dev/bpapi-separate-for-otp26
fix(static_checks): Prepare for OTP26
2023-12-14 20:32:45 +01:00
Zaiming (Stone) Shi c1f2287b86 Merge remote-tracking branch 'origin/release-54' 2023-12-14 15:26:49 +01:00
ieQu1 02e2231734 fix(static_checks): Prepare for OTP26 2023-12-14 15:22:09 +01:00
Thales Macedo Garitezi dbc8141930
Merge pull request #12125 from thalesmg/ds-cache-m-20231206
chore(ds): return DS key from `next` and add `update_iterator` callback
2023-12-14 10:23:25 -03:00
firest f1bde41bf6 fix(sso): Correctly handle wrapped passwords when updating 2023-12-14 10:07:04 +08:00
firest d278d3afb5 feat(ds): implemented the replayer scheduler 2023-12-13 23:52:02 +08:00
zhongwencool 1807df157c fix: license watermark percent support float 2023-12-12 17:27:31 +08:00
JimMoen 6552e0523a
Merge pull request #12141 from JimMoen/fix-shared-sub-api-internal-error
Fix shared sub api internal error
2023-12-11 21:43:09 +08:00
Zaiming (Stone) Shi b6c11be159
Merge pull request #12138 from zmstone/1208-fix-dialyzer
fix(dialyzer): batch 3
2023-12-11 12:52:56 +01:00
Zaiming (Stone) Shi d560366c14 test: fix some compile warnings 2023-12-11 09:43:13 +01:00
JimMoen 8c25371ff3
chore: bump gitignore for emqx standalone common test SUITES 2023-12-11 15:37:36 +08:00
JimMoen 68da627b4d
feat(channel): add peerport field in ClientInfo 2023-12-11 14:11:18 +08:00
Zaiming (Stone) Shi 50f4aba5cd fix(dialyzer): batch 3 2023-12-09 15:50:09 +01:00
Thales Macedo Garitezi 66d043becd feat(ds): introduce `update_iterator` callback 2023-12-08 15:04:18 -03:00
Andrew Mayorov 76970c268e
fix(sessds): stop overwriting QoS0-only pubrange checkpoints
Through making `StreamRef` part of a record key. Otherwise with
QoS0-only streams the seqno doesn't change, thus ranges over
different streams will have the same `FirstSeqno`.
2023-12-08 14:33:47 +01:00
Andrew Mayorov 24710d0f56
fix(sessds): avoid accumulating QoS0-only ranges in memory 2023-12-08 14:31:18 +01:00
Thales Macedo Garitezi f37c86afb9 Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231207 2023-12-07 13:30:04 -03:00
Thales Macedo Garitezi 2a6d72878f chore(ds): return DS message key along with batch 2023-12-07 11:36:08 -03:00
William Yang 499f4d4bc5 chore(quicer): dialyzer 2023-12-07 13:52:48 +01:00
William Yang 583645d97b chore: bump emqtt 1.10.1 and quicer 0.0.308 2023-12-07 13:52:44 +01:00
ieQu1 9b612cb0e7
Merge pull request #12119 from ieQu1/EMQX-10342
Make stream renew timer configurable
2023-12-07 10:42:23 +01:00
Zaiming (Stone) Shi 7560016f0e fix(hocon): upgrade hocon from 0.40.0 to 0.40.1
fixed type spec, exported some public types from emqx_schema module
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 6b553e37c0 chore: pin ekka 0.17.0 previously 0.15.16
change summary:
1. otp 26
2. type spec fixes
3. dropped UDP mcast clustering
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Zaiming (Stone) Shi 71f9838fd7 chore: upgrade to emqtt 1.10.0 2023-12-06 19:18:15 +01:00
Zaiming (Stone) Shi 83206daa20 refactor(emqx_cm_locker): delete dead code 2023-12-06 19:18:15 +01:00
William Yang 64401cb6cc
Merge pull request #12042 from qzhuyan/dev/william/rel54/esockd-robustnes
fix: bump to esockd 8.9.9  improve robustness
2023-12-06 17:39:38 +01:00
Andrew Mayorov 074dc8ce11
test(emqx): switch rest of test suite to use `emqx_cth_suite` 2023-12-06 16:47:04 +01:00
ieQu1 371ec34992 feat(sessds): Make stream renew interval configurable 2023-12-06 16:04:53 +01:00
ieQu1 c43b3eb535 fix(sessds): Add debug logs for the session garbage collection 2023-12-06 15:37:23 +01:00
Andrew Mayorov c2c9de69b4
test(emqx): switch `emqx_crl_cache_SUITE` to use `emqx_cth_suite` 2023-12-06 14:34:41 +01:00
Andrew Mayorov f06a1f10ef
test(emqx): switch most of testsuites to use `emqx_cth_suite`
For increased robustness and isolation.
2023-12-06 14:34:41 +01:00
Andrew Mayorov d643330cb9
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-06 14:34:40 +01:00
Andrew Mayorov 7d80511b3a
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-06 14:34:40 +01:00
Thales Macedo Garitezi 8a5b0d252d
Merge pull request #12110 from thalesmg/test-plugin-flaky-r54-20231205
test(cth_peer): use an exclusive current dir for each peer
2023-12-06 10:16:13 -03:00
William Yang c4956d99d1 fix: bump to esockd 8.9.9 improve listener robustness 2023-12-06 13:55:13 +01:00
Thales Macedo Garitezi b82189fb4d test(cth_peer): use an exclusive current dir for each peer 2023-12-06 09:35:13 -03:00
Andrew Mayorov 43813e29c2
chore(emqx): drop legacy session persistence impl leftovers 2023-12-06 10:53:57 +01:00
SergeTupchiy 28ff53e99c
Merge pull request #11984 from SergeTupchiy/EMQX-10535-openetelmetry-tracing-new
Integrate OpenTelmetry tracing
2023-12-06 11:45:41 +02:00
Andrew Mayorov 3798060543
Merge pull request #12109 from keynslug/test/emqx-cth-suite
test(emqx): switch select test suites to use `emqx_cth_suite`
2023-12-06 09:48:53 +01:00
Andrew Mayorov 83bea2254d
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-05 19:16:37 +01:00
Andrew Mayorov 6f5228e991
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-05 19:16:37 +01:00
Serge Tupchii 938508b270 refactor(emqx_opentelemetry): pass only channel info to trace functions 2023-12-05 19:01:08 +02:00
Thales Macedo Garitezi deb3fcd606 Merge remote-tracking branch 'origin/master' into sync-m-r54-20231205 2023-12-05 13:25:45 -03:00
Serge Tupchii 85441fda0f test(emqx_opentelemetry): add trace test suite 2023-12-05 18:18:31 +02:00
Serge Tupchii 8d3f98eff2 fix(emqx_opentelemetry): fix dialyzer warnings 2023-12-05 18:18:31 +02:00
Serge Tupchii 7fdc650448 feat: integrate OpenTelemetry traces 2023-12-05 18:11:00 +02:00
Andrew Mayorov 286d483a3a
Merge pull request #12055 from keynslug/ft/EMQX-11474/subopts
fix(sessds): respect subscription options when publishing
2023-12-05 15:45:25 +01:00
Andrew Mayorov 92c4b29a4c
feat(topic): match empty topic levels more loosely
So that the result of `emqx_topic:tokens/1` would be perfectly
matchable with the result of `emqx_topic:words/1` of a topic filter
with empty levels.
2023-12-04 13:38:55 +03:00
Andrew Mayorov b5c485cd55
test(sessds): add empty level topic subscription testcase 2023-12-04 13:38:42 +03:00
JimMoen cb992fcf55
Merge pull request #12059 from JimMoen/1130-fix-vm-time-warp
fix: vm.args use `multi_time_warp`
2023-12-04 11:56:08 +08:00
Andrew Mayorov 627c58b07d
fix(sessds): handle `expire_awaiting_rel` common timer with no-op
Otherwise, the channel crashes when the timer is triggered.
2023-12-02 15:00:30 +03:00
Andrew Mayorov 47bc747323
fix(sessds): drop everything related to session on CleanStart 2023-12-02 15:00:30 +03:00
Andrew Mayorov fd26e690b8
refactor(sessds): move parts of message processing to replayer
To simplify the processing flow, reducing the number of back-and-forth
between the session and the replayer.
2023-12-02 15:00:29 +03:00
Andrew Mayorov 6255ee0833
fix(sessmem): avoid defining unsafe defaults in `#session` 2023-12-02 14:38:04 +03:00
Andrew Mayorov b5ad0f9815
fix(test): subscribe with QoS 2 to receive QoS 2 message 2023-12-02 14:38:04 +03:00
Andrew Mayorov 1948719eec
test(sessds): unskip testcase related to QoS2 handling 2023-12-02 14:38:04 +03:00
Andrew Mayorov 3265d2f2aa
fix(sessds): respect subscription options when publishing 2023-12-02 14:38:03 +03:00
Andrew Mayorov 508346f095
refactor(topicidx): split persistent term stuff off gbt-based index 2023-12-02 14:36:21 +03:00
Andrew Mayorov 88103c5f0e
refactor(session): pass config to `SessionImpl:open/3` as well
* Anticipate that connection info may change during session takeover.
* Avoid persisting session conf as part of persistent session state.
2023-12-02 14:36:21 +03:00
Andrew Mayorov 6eba082992
Merge pull request #12075 from keynslug/fix/EMQX-11474/function-clause
fix(channel): expect there's no session yet on disconnect
2023-12-02 14:32:50 +03:00
Zaiming (Stone) Shi 55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 21:35:40 +01:00
Zaiming (Stone) Shi 6e30718818 test: allow join cluster later 2023-12-01 20:33:39 +01:00
ieQu1 3d2e95fe3e
Merge pull request #12081 from ieQu1/dev/gen-rpc-3.3.0
chore(gen_rpc): Bump version to 3.3.0
2023-12-01 17:57:33 +01:00
ieQu1 476c300ecf
Merge pull request #12070 from ieQu1/dev/ds-min-max-batch-size
Various minor fixes
2023-12-01 17:57:15 +01:00
ieQu1 d3b7478d70 chore(gen_rpc): Bump version to 3.3.0 2023-12-01 17:10:12 +01:00
Zaiming (Stone) Shi ab7d57e1c5 test: delete debug level logging 2023-12-01 17:07:14 +01:00
Ivan Dyachkov 0e27d031a8 chore: 5.4.0-alpha.1 2023-12-01 16:54:03 +01:00
Zaiming (Stone) Shi 6a9cc20d4d test: wait for all nodes clustered 2023-12-01 16:37:39 +01:00
Andrew Mayorov 6b17920fea
fix(channel): expect there's no session yet on disconnect 2023-12-01 15:29:05 +03:00
Andrew Mayorov cce474ce98
fix(session): include session term in stacktrace 2023-12-01 15:29:05 +03:00
ieQu1 e1ec560639 test(emqx): Fix flaky emqx_takeover_SUITE:t_takeover testcase
This testcase uses QoS1, so it must account for possible duplication
of the messages.
2023-12-01 10:06:18 +01:00
Zaiming (Stone) Shi 7f4d91d490 Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54 2023-12-01 08:32:22 +01:00
ieQu1 0e625d814a feat(sessds): Make batch size configurable
This change affects flow control. It allows to configure maximum size
of a batch, as well as fetch threshold.
2023-12-01 08:27:05 +01:00
ieQu1 4717e56fb6 fix(sessds): Schedule poll immediately upon receiving an ack
This commit affects the flow control, and improves the throughput by
removing a delay between freeing up space in the in-flight window and
polling new messages.
2023-12-01 08:27:05 +01:00
ieQu1 38800c0260 refactor(sessds): Store timers in the session 2023-12-01 08:27:05 +01:00
Zaiming (Stone) Shi 22f7cc1622 test: replace 'slave' and 'ct_slave' with 'peer' 2023-12-01 08:07:09 +01:00
JimMoen ee25d9bd9e
fix: vm.args use `multi_time_warp`
See also:
https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
2023-12-01 14:52:31 +08:00
ieQu1 69f1ca43c3 fix(sessds): Create a timer even if it's present in the map 2023-12-01 06:11:13 +01:00
ieQu1 1897e5c31b refactor(session): Use common naming conventions 2023-12-01 06:11:08 +01:00
ieQu1 d9e7544070 refactor(sessds): Introduce macros for the timers 2023-12-01 06:11:08 +01:00
ieQu1 0d245acdc1
Merge pull request #12066 from keynslug/fix/sessds-tc-flaky
test(sessds): wait client disconnect propagates to broker
2023-12-01 00:13:38 +01:00
Thales Macedo Garitezi 113a4ad4b1
Merge pull request #12057 from thalesmg/ds-session-gc-m-20231128
feat(ds): add session gc process
2023-11-30 17:51:52 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Thales Macedo Garitezi 684d637fed test(bridge_api): workaround strange config syncing problem
For some unknown reason, this test has difficulties in syncing the config correctly
between the nodes, while the equivalent in bridge_v2_api_SUITE doesn't.
2023-11-30 19:51:35 +01:00
Ivan Dyachkov ec10c51073 Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53 2023-11-30 19:51:12 +01:00
Zaiming (Stone) Shi dcb1c0680b
Merge pull request #12053 from zmstone/1129-prepare-for-otp26
1129 prepare for OTP 26
2023-11-30 19:38:40 +01:00
Andrew Mayorov ec471c0557
test(sessds): wait client disconnect propagates to broker 2023-11-30 20:14:57 +03:00
Ivan Dyachkov 6dd92c382f chore: 5.3.2 release 2023-11-30 16:18:44 +01:00
Thales Macedo Garitezi cf6cb3e4ad chore: set low importance to config 2023-11-30 11:55:57 -03:00
Thales Macedo Garitezi 1ab009f081 refactor: rename supervisor 2023-11-30 11:55:48 -03:00
Zaiming (Stone) Shi eed253af82 test: implement a new node restart helper func 2023-11-30 15:11:38 +01:00
Thales Macedo Garitezi 880f5e8f89 feat(ds): add session gc process
Fixes https://emqx.atlassian.net/browse/EMQX-9744
2023-11-30 09:38:59 -03:00