Commit Graph

3982 Commits

Author SHA1 Message Date
Kjell Winblad ef9f8a8fdf fix: unreachable clause found by dialyzer 2024-04-18 10:28:37 +02:00
Kjell Winblad 6c0ee8bb01 test(emqx_trace_SUITE): fix failure due to new field 2024-04-18 10:28:37 +02:00
Kjell Winblad 968dc2ccda fix(json trace format): use best_effort_json instead of new encoder 2024-04-18 10:28:37 +02:00
Kjell Winblad aa39835242 feat(emqx_cli): add parameter for trace format (plain or json) 2024-04-18 10:28:29 +02:00
Kjell Winblad 6fd7a06c5d fix: problems reported by dialyzer 2024-04-18 10:20:18 +02:00
Kjell Winblad a6558740e8 feat(emqx_trace): add JSON trace log entry formatter
This commit makes it possible to select the JSON trace log entry
formatter when crating a trace pattern. This will make it easier for the
dashboard and automatic tests to parse the log entries.

Fixes: EMQX-12025 (partly)
2024-04-18 10:20:11 +02:00
Kjell Winblad 2a2fadfbff
Merge pull request #12827 from kjellwinblad/kjell/emqx_rule_tracing/EMQX-11966
Add rule ID tracing support
2024-04-18 09:06:31 +02:00
Ivan Dyachkov e93bd314bc chore: 5.6.1 2024-04-18 08:03:56 +02:00
zmstone 5a4bfff9e5 refactor: add template_str type 2024-04-17 20:34:33 +02:00
Kjell Winblad 6cf29ba688 fix: clean up traces to make them easier to parse and understand 2024-04-17 18:20:17 +02:00
zmstone 51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
ieQu1 124c5047d0
feat(sessds): Add API for getting session data from the cold storage 2024-04-17 12:31:54 +02:00
ieQu1 e439a2e0f2
fix(sessds): Save protocol name and version in the session metadata 2024-04-17 12:31:54 +02:00
ieQu1 197a4c30be
fix(sessds): Strip unneccessary data from the durable session state 2024-04-17 01:21:52 +02:00
ieQu1 87ffaf89e5
refactor(sessds_state): Use macros for map keys 2024-04-17 01:21:52 +02:00
ieQu1 113a990482
feat(sessds): Support max subscriptions 2024-04-17 01:21:51 +02:00
ieQu1 6c897c26ae
fix(sessds): Commit session on unsubscribe 2024-04-17 01:21:51 +02:00
ieQu1 b30ddc206e
fix(sessds): Immutable subscriptions
This commit fixes two issues:
- Behavior of overlapping subscriptions has been aligned with the
in-memory session.

- Fixed handling of replays when subscription changes (either by
client or EMQX configuration)
2024-04-17 01:21:51 +02:00
ieQu1 3e0c649e8e
feat(sessds): Store awaiting rel 2024-04-17 01:21:51 +02:00
Zaiming (Stone) Shi 4d38a8fd44
Merge pull request #12872 from zmstone/0412-support-variform-for-client_attrs
0412 use variform for client_attrs
2024-04-16 13:29:56 +02:00
ieQu1 c645cfa5d6
fix(sessds): Graceful handling of shared subscription error 2024-04-16 12:37:01 +02:00
Ivan Dyachkov f3bb28c6af chore: 5.6.1-rc.2 2024-04-16 09:47:32 +02:00
Ivan Dyachkov a0ffe5e7ae chore: 5.6.1-rc.1 2024-04-15 22:05:05 +02:00
Ivan Dyachkov 3ef160eed2 Merge remote-tracking branch 'upstream/release-56' into 0415-sync-release-56 2024-04-15 08:09:03 +02:00
zmstone b76b6fbe63 feat(variform): initialize client_attrs with variform
Moved regular expression extraction as a variform function.
2024-04-14 10:13:24 +02:00
zmstone da5b01aa46 refactor(client_attr): allow more than one initial extraction 2024-04-13 01:00:25 +02:00
Kjell Winblad 6777f04780 fix: iolist_to_binrary -> unicode:characters_to_binary
It is not always safe to use iolist_to_binrary on the output of an
io_lib:format call with the ~ts placeholder.
2024-04-12 15:21:25 +02:00
Kjell Winblad 9998940aa2 fix(trace): several improvements thanks to comments from @zmstone 2024-04-11 16:48:43 +02:00
Ivan Dyachkov 3b7cade671 chore: 5.6.1-beta.1 2024-04-11 13:49:40 +02:00
Serge Tupchii 55179ccfed chore: update ekka to 0.19.3
Included updates:
  - https://github.com/emqx/mria/pull/178
2024-04-09 19:04:38 +03:00
Kjell Winblad 7bcd553786 test: move test case with rule_engine dep from emqx to emqx_rule_engine 2024-04-09 14:08:46 +02:00
Kjell Winblad 3898950017 test: fix test by adding dependency 2024-04-09 11:38:35 +02:00
JimMoen 03a9c46ca7
fix(sys_topic): format shared topics 2024-04-09 17:06:36 +08:00
JimMoen 47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api 2024-04-09 17:06:32 +08:00
JianBo He e11c4a9c83
Merge pull request #12826 from emqx/import-source-bridges
fix: source bridges missing after restore the backup files
2024-04-08 16:02:15 +08:00
Kjell Winblad 5479932190 feat(apply rule test): make option to stop action after render work
This commit makes the apply rule HTTP API option to stop an action work
for the HTTP action, and adds infrastructure that makes it easy to add
this functionality to other actions.
2024-04-06 17:21:12 +02:00
Kjell Winblad ef705c2285 feat: add apply rule API, clientid/ruleid tracing for rule and connector
This commit adds:

* Support for forwarding the rule id and client id to the connector so
  that events such as template rendered successfully can be traced.
* HTTP API for for applying/activating a rule with the given context
2024-04-06 17:20:47 +02:00
Andrew Mayorov 70396e9766
Merge pull request #12825 from keynslug/feat/EMQX-12110/repl-meta-api
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 22:32:03 +02:00
Andrew Mayorov ad52f7838e
feat(dsrepl): add APIs to manage DB replication sites 2024-04-04 21:22:01 +02:00
Kjell Winblad 59a442cdb5 feat(rule trace): add support for ruleid as a trace type 2024-04-04 14:55:32 +02:00
Shawn 319ec50c0d fix: source bridges missing after restore the backup files 2024-04-03 18:26:51 +08:00
ieQu1 b379f331de
fix(sessds): Handle errors when storing messages 2024-04-02 16:47:41 +02:00
ieQu1 b9ad241658
feat(sessds): Add metrics for the number of persisted messages 2024-04-02 16:45:52 +02:00
ieQu1 d8204021dc
refactor(metrics): Move metrics worker to emqx_utils application 2024-04-02 16:25:04 +02:00
Andrew Mayorov 2d074df209
Merge pull request #12797 from keynslug/fix/dsrepl-error-handling
fix(dsrepl): handle RPC errors gracefully when storage is down
2024-04-02 13:40:31 +02:00
JimMoen 5759ba5162
chore: bump app version 2024-04-02 17:09:22 +08:00
JimMoen 50bceee9ab
fix(stats): `'subscribers.count'` contains shared-subscriber 2024-04-02 16:56:40 +08:00
JimMoen 0f4b148294
refactor: uniform shared_sub table macros 2024-04-02 16:56:39 +08:00
Serge Tupchii 3eda182e9a fix: prevent a node from discovering and re-joining the same cluster after it has (manually) left it. 2024-03-28 18:09:27 +02:00
Andrew Mayorov 35c43eb8a0
feat(sessds): handle recoverable errors in stream scheduler 2024-03-28 15:17:01 +01:00
Thales Macedo Garitezi 8fb4ef9fe3 test: fix flaky test 2024-03-28 10:53:44 -03:00
Ivan Dyachkov 9b4d960888 chore: prepare for 5.7.x releases 2024-03-28 13:52:48 +01:00
ieQu1 b1855f95c1
fix(bpapi): Add exceptions for experimental features 2024-03-28 12:07:45 +01:00
ieQu1 8c6d8bdd12
fix(bpapi): Add exceptions for experimental features 2024-03-28 12:03:41 +01:00
ieQu1 02b72d79fc
fix(bpapi): Fix comparison of releases during compatibility check 2024-03-28 11:21:30 +01:00
Ivan Dyachkov db9efb9317 chore: bump apps versions 2024-03-28 10:19:09 +01:00
Ivan Dyachkov f4446ec680 Merge remote-tracking branch 'upstream/release-56' into 0328-sync-release-56 2024-03-28 09:59:54 +01:00
Ivan Dyachkov ea917d6b2b
Merge pull request #12795 from id/0327-cut-5.6.0
chore: 5.6.0
2024-03-28 09:30:08 +01:00
ieQu1 4b3c29ded8
fix(bpapi): Add 5.5 dump 2024-03-27 17:39:38 +01:00
Zaiming (Stone) Shi beb9152d50
Merge pull request #12750 from zmstone/0316-introduce-client-attrs
0316 introduce client_attrs
2024-03-27 16:19:47 +01:00
ieQu1 5ed20e15cd
fix(bpapi): Add 5.5 bpapi file 2024-03-27 13:45:00 +01:00
Ivan Dyachkov 33d9619271 chore: 5.6.0 2024-03-27 11:17:13 +01:00
Serge Tupchii a3b730ff10 fix(emqx_session_mem): keep msg inflight insert timestamp in Inflight after retry 2024-03-27 11:31:51 +02:00
JianBo He 4f9daa0d4a chore: bump version to 5.6.0-rc.6 2024-03-27 14:04:27 +08:00
JianBo He d82b6d216d chore: remove the duplicated time field
Fixes https://emqx.atlassian.net/browse/EMQX-12104
2024-03-27 12:09:21 +08:00
Ivan Dyachkov f127bf53b3 chore: 5.6.0-rc.5 2024-03-26 17:52:57 +01:00
ieQu1 291cf689e2
Merge pull request #12786 from ieQu1/dev/check-core-emqx-vsn
fix(mria): Strictly check compatibility of the replicant with core
2024-03-26 17:49:37 +01:00
ieQu1 ba43da1b8e
fix(mria): Strictly check compatibility of the replicant with core 2024-03-26 16:32:01 +01:00
zmstone b1c9bb63d7 chore: delete dead code
hocon schema validator never validates 'undefined'
2024-03-26 13:20:48 +01:00
zmstone c42e980442 feat(log): configurable time format
now logs can be configured to use 'epoch' or 'rfc3339' time format
2024-03-26 13:09:32 +01:00
Thales Macedo Garitezi cc37030265
Merge pull request #12770 from thalesmg/mv-metrics-m-20240321
feat(message validation): implement metrics
2024-03-26 08:59:05 -03:00
zhongwencool 56a40d61ae fix: bpapi undef abort on old node 2024-03-26 13:06:56 +08:00
Ivan Dyachkov 2ae973821f chore: 5.6.0-rc.4 2024-03-25 20:00:10 +01:00
Zaiming (Stone) Shi b06d3932af
Merge pull request #12768 from keynslug/fix/EMQX-12068/routing-schema-conflict
fix(routing): add heuristic for routing schema in empty cluster
2024-03-25 19:50:51 +01:00
Thales Macedo Garitezi 0f426e6e77 feat(ds): make durable sessions handle will messages
Fixes https://emqx.atlassian.net/browse/EMQX-10431
2024-03-25 14:18:25 -03:00
Thales Macedo Garitezi 8c1a1d21a7 feat(session): pass will message down to session when creating/opening 2024-03-25 14:18:23 -03:00
ieQu1 8b963d5960
Merge pull request #12739 from ieQu1/dev/zone-durability
feat(sessds): Add zone overrides for session durability settings
2024-03-25 17:32:53 +01:00
zmstone b8b0b809b4 chore: add a warning log if conflict is detected 2024-03-25 17:07:56 +01:00
zmstone 9411d6078e docs: remove unnecessary restart instruction 2024-03-25 16:50:51 +01:00
zmstone d38545010a fix: case_clause 2024-03-25 16:43:24 +01:00
Ivan Dyachkov c93145cb97 fix: function clause 2024-03-25 16:40:53 +01:00
zmstone 3aff9eb2a4 fix(route schema): allow boot if all peer nodes agree on one version 2024-03-25 16:33:27 +01:00
Ivan Dyachkov fe3cc25855 chore: update instructions for routing schema conflict 2024-03-25 15:22:43 +01:00
Thales Macedo Garitezi 59ff43ec87 feat(message validation): implement metrics
Fixes https://emqx.atlassian.net/browse/EMQX-12071
2024-03-25 10:55:01 -03:00
Andrew Mayorov 849fe0c2c8
feat(routing): add schema conflict resolution procedure
In the log message printed when a schema conflict in cluster routing
is detected.
2024-03-25 13:43:46 +01:00
JimMoen 5f7f9e43f9
fix(stats): `'subscribers.count'` contains shared-subscriber 2024-03-25 15:12:57 +08:00
zhongwencool 2926518b3d
Merge pull request #12764 from zhongwencool/fix-bpapi-undef
fix: bpapi undef on old node
2024-03-25 13:37:37 +08:00
zmstone 22838f027a fix: mountpoint template render should not replace unknown as undefined
For backward compatibility, the unknown vars used in mountpoint
is kept unchanged.
e.g. '${unknown}/foo/bar' should be rendered as '${unknown}/foo/bar'
but not 'undefined/foo/bar'
2024-03-23 10:16:05 +01:00
zmstone 3136ec5958 feat: allow mountpoint to use client_attrs 2024-03-23 10:16:05 +01:00
zmstone 8e8fc6a3d1 test: fix config test 2024-03-23 10:16:05 +01:00
zmstone 5e9814d171 fix: add debug level logging for invalid client attributes 2024-03-23 10:16:05 +01:00
zmstone 8254b801ae feat: support initialize client attribute from user property 2024-03-23 10:16:02 +01:00
zmstone 2fd0a2cd4d feat: support extracting initial client attrs from clientinfo 2024-03-23 10:16:02 +01:00
zmstone 9ec99fef4a feat: allow client_attr used in authz rules 2024-03-23 10:16:02 +01:00
zmstone e5816f5a13 refactor: rename attr to client_attr
client_attr is unique enough for all contexts
so the name can be unified from external responses
to internal template rendering, and rule-engine template rendering
2024-03-23 10:16:02 +01:00
zmstone cc4805b1ac feat: extract attrs field from http and jwt auth response 2024-03-23 10:16:02 +01:00
zmstone f9d51484cc feat: introuce attrs field in clientinfo
the 'attrs' field is to be used as extra attributes extracted
from various client properties
for example, HTTP auth result can return an extra client_attributes
field in the result which will get merged into the original
attrs field in clientinfo
2024-03-23 10:16:02 +01:00
Andrew Mayorov cbc84900b2
chore: update bpapi versions dump 2024-03-22 16:24:52 +01:00
Ivan Dyachkov 9986de49d1 chore: make dyalizer happy 2024-03-22 16:16:49 +01:00
Ivan Dyachkov be1886fb91 chore: make dyalizer happy and update changelog 2024-03-22 15:56:44 +01:00
Andrew Mayorov b81a11b790
chore(routing): mark function that is now an RPC target 2024-03-22 15:14:53 +01:00
Andrew Mayorov ba276d807f
fix(routing): add heuristic for routing schema in empty cluster
When the cluster is empty but comprised of different versions of EMQX,
try to additionally ask the cluster nodes for the routing schema in use,
to be able to make more informed decisions.
2024-03-22 15:14:32 +01:00
JimMoen 42faffc320
refactor: uniform shared_sub table macros 2024-03-22 18:03:13 +08:00
zhongwencool 45eedbb9c6 fix: bpapi undef on old node 2024-03-22 17:02:09 +08:00
Serge Tupchii 5d77e1b01e fix: return not_implemented error for mqueue/inflight msgs queries 2024-03-22 10:02:31 +02:00
Thales Macedo Garitezi e17f663fa5
Merge pull request #12749 from thalesmg/mv-followup1-m-20240320
Follow up features and fixes for message validation - part 1
2024-03-21 16:03:49 -03:00
Ivan Dyachkov 93c80bc912 chore: 5.6.0-rc.3 2024-03-21 18:10:59 +01:00
Thales Macedo Garitezi 00aa7b5621 feat: create new `message.validation_failed` hookpoint and rule engine event 2024-03-21 13:46:27 -03:00
Thales Macedo Garitezi 7d7c6685d4
Merge pull request #12753 from thalesmg/test-flaky-retry-m-20240321
test: attempt to stabilize flaky test
2024-03-21 13:43:51 -03:00
Andrew Mayorov 7d6fde5960
fix(topics-api): respond with correct totals to paged queries 2024-03-21 16:02:16 +01:00
Thales Macedo Garitezi 64399b6861 test: attempt to stabilize flaky test 2024-03-21 11:04:14 -03:00
ieQu1 cada944350
feat(sessds): Expose relevant durable session info in the REST API 2024-03-21 10:37:04 +01:00
Ivan Dyachkov 49e0488498 chore: 5.6.0-rc.2 2024-03-20 17:11:37 +01:00
Zaiming (Stone) Shi bede5a5b85
Merge pull request #12746 from zmstone/0320-add-throttle-log-for-auth-failure
feat: add authentication_failure throttled log
2024-03-20 17:06:36 +01:00
SergeTupchiy 27c1f1a4e4
Merge pull request #12732 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API-updates
EMQX-11861 client mqueue inflight api updates
2024-03-20 16:27:44 +02:00
Andrew Mayorov 7257fe526b
fix(ci): add `ra` to emqx app dependencies as well 2024-03-20 14:46:53 +01:00
zmstone 2c81b56377 feat: add authentication_failure throttled log
also included changes:

1. add 'username' to log metadata
2. add 'tag' to authz warning logs
2024-03-20 14:42:01 +01:00
Serge Tupchii cb5fdb3c79 fix: rework In-flight / Mqueue API
- use timestamp 'position' to find the next chunk of data
- add 'start' position to response meta
- sort In-flight messages by insertion time
- sort Mqueue messages by priority
2024-03-20 15:25:33 +02:00
Andrew Mayorov e55e1dd1b2
chore: whitelist `ra` to make RPCs w/o BPAPIs 2024-03-20 13:20:25 +01:00
Andrew Mayorov a8baff61ec
docs(dsrepl): describe briefly what `n_sites` is for 2024-03-20 13:20:25 +01:00
Andrew Mayorov efac5c6197
test(ds): stabilize `t_message_gc` testcase 2024-03-20 13:20:25 +01:00
Andrew Mayorov 1b647035d0
chore(dsrepl): make dialyzer a bit happier 2024-03-20 13:20:23 +01:00
Andrew Mayorov 74881e8706
feat(dsrepl): make storage layer unaware of granularity of time
Storage also becomes a bit more pure, depending on the upper layer to
provide the timestamps, which also makes it possible to handle more
operations idempotently.
2024-03-20 13:20:23 +01:00
ieQu1 783f769a98 fix(mgmt): Fix bulk kickout of durable sessions 2024-03-20 12:52:44 +01:00
ieQu1 0547b32727
feat(sessds): Add zone overrides for session durability settings 2024-03-19 22:51:06 +01:00
ieQu1 611e4f6710
Merge pull request #12740 from ieQu1/dev/kick-durable-sessions
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 21:30:40 +01:00
Andrew Mayorov e2db038e7c
Merge pull request #12734 from keynslug/fix/EMQX-12030/topics-api
fix(api-topics): expose persistent session topics in APIs
2024-03-19 21:08:23 +01:00
Andrew Mayorov 5cc0246351
feat(dsrepl): allow to tune select ra options 2024-03-19 20:21:55 +01:00
Andrew Mayorov 54b5adf868
feat(dsrepl): allocate shards predictably
To ensure strictly optimal and fair shard allocation across
cluster. Before this commit it was quite easy to end up with
an allocation significantly skewed towards some node, because
of the nature of randomness and relatively small number of
shards.
2024-03-19 20:21:55 +01:00
Andrew Mayorov 5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
Andrew Mayorov 146f082fdc
feat(dsrepl): implement raft-based replication
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
Ivan Dyachkov 923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
sync release-56
2024-03-19 19:09:41 +01:00
Andrew Mayorov f453bdf055
chore(topic-api): describe `emqx_persistent_session_ds_router:stream/1` 2024-03-19 18:51:06 +01:00
Andrew Mayorov 027b237f07
test(ds): stabilize `t_session_replay_retry` testcase 2024-03-19 18:33:38 +01:00
ieQu1 236685f714 fix(sessds): Channel must destroy sessions that are kicked 2024-03-19 17:49:04 +01:00
Ivan Dyachkov f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
feat: implement message validation
2024-03-19 11:12:28 -03:00
Andrew Mayorov e1b3263177
fix(sessds): make dialyzer a tad happier 2024-03-19 13:11:32 +01:00
Andrew Mayorov 985f1c4879
fix(api-topics): expose persistent session topics in APIs 2024-03-19 11:01:26 +01:00
Thales Macedo Garitezi f84a996671 feat: implement message validation
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
JimMoen 889eb48368
fix(shared-sub): update stats when unsub shared topic filter 2024-03-16 20:43:05 +08:00
Ivan Dyachkov e28992d586 chore: 5.6.0-rc.1 2024-03-15 17:08:00 +01:00
Ivan Dyachkov ea76dac760 chore: fix xref 2024-03-15 17:08:00 +01:00
ieQu1 b6cc9177a6 feat(sessds): Store peername in the persistent session state 2024-03-15 13:48:53 +01:00
SergeTupchiy e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client
Refactor emqx mgmt call client
2024-03-15 10:01:49 +02:00
ieQu1 69a9431471
Merge pull request #12665 from ieQu1/dev/sessds-hot-conf-disable
fix(sessds): Prevent hot update of session_persistence.enable config
2024-03-14 13:20:52 +01:00
Zaiming (Stone) Shi cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node
fix: do not crash on replicant node
2024-03-14 09:42:27 +01:00
Serge Tupchii 685f29d3f2 chore: upgrade hocon to 0.42.1
hocon 0.42.1 allows to use "b" or "B" (byte) unit in bytesize fields.
2024-03-13 19:30:28 +02:00
zmstone ccd973d13e fix: do not crash on replicant node 2024-03-13 16:42:39 +01:00
Serge Tupchii 65be76aa06 refactor(emqx_mgmt): add call_client timeout and improve RPC error handling 2024-03-13 10:56:02 +02:00
JimMoen 0edeff4786
test: fix flaky 2024-03-12 15:21:17 +08:00
JimMoen bc9084db79
test: cpu utilization value 2024-03-12 14:14:46 +08:00
JimMoen 0297c7b83b
refactor: conditional enclosed in parentheses 2024-03-12 14:14:45 +08:00
JimMoen 207f38c42a
fix(vm): cpu usage/idle handled by single worker 2024-03-12 14:14:44 +08:00
Andrew Mayorov f7e3afde16
test(ds): avoid introducing new macros 2024-03-07 16:49:20 +01:00
Andrew Mayorov 69427dc42d
test(ds): add tests for error mapping and replay recovery 2024-03-07 12:59:58 +01:00
Andrew Mayorov e7e8771277
fix(sessds): set replay retry timer if initial `replay/3` fails 2024-03-07 12:59:58 +01:00
Andrew Mayorov b604c3dbd4
refactor(sessds): make replay error handling a bit more clear
Also leave a forgotten TODO.
2024-03-07 12:59:57 +01:00
Andrew Mayorov 3f3e33b2cb
fix(sessds): untangle pull and replay retry timers
And restore the convention that timer handler always manages only
its own timers.
2024-03-07 12:59:57 +01:00
Andrew Mayorov b39c710ec2
fix(ds): tidy up few typespecs 2024-03-07 12:59:57 +01:00
Andrew Mayorov 1cf672e78d
feat(sessds): handle recoverable errors during replay 2024-03-07 12:59:57 +01:00
SergeTupchiy 1f38813cb9
Merge pull request #12561 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API
feat: add client mqueue/inflight messages API
2024-03-07 13:20:31 +02:00
ieQu1 c62776edaf fix(sessds): Prevent hot update of session_persistence.enable config 2024-03-07 12:19:53 +01:00
Serge Tupchii a2e761681e feat: add client mqueue/inflight messages API 2024-03-07 12:06:49 +02:00
zmstone f57f2fa1b7 chore: bump app version numbers 2024-03-06 19:37:06 +01:00
zmstone e99546e009 Merge remote-tracking branch 'origin/release-56' into sync-5.5.1 2024-03-06 17:27:54 +01:00
JianBo He 1737df05c7
Merge pull request #12641 from zmstone/0304-improve-text-log-formatter
0304 improve text log formatter
2024-03-06 09:12:18 +08:00
Kjell Winblad 8cf681ad3b
Merge pull request #12639 from kjellwinblad/kjell/fix_flaky_test_case/emqx_broker_SUITE.connected_client_count_group.quic.t_connected_client_count_transient_takeover
test(emqx_broker_SUITE): fix flaky test case
2024-03-05 18:02:07 +01:00
Thales Macedo Garitezi c5489fee90 Merge remote-tracking branch 'origin/release-56' into sync-r56-m-20240304 2024-03-05 09:08:59 -03:00
Kjell Winblad c8e42cf6b1 test(emqx_broker_SUITE): fix flaky test case 2024-03-05 10:14:07 +01:00
zmstone cb0066d639 chore: add tag for logs from MQTT connection modules 2024-03-05 08:39:53 +01:00
Zaiming (Stone) Shi 00b21cf9c5
Merge pull request #12634 from zmstone/0304-breaking-no-escape-for-triple-quote-string-characters
chore: upgrade to hocon 0.42.0
2024-03-05 08:32:50 +01:00
Thales Macedo Garitezi 61328eb91a chore: prepare e5.6.0-alpha.2 2024-03-04 15:37:58 -03:00
zmstone 3e3194fd21 chore: upgrade to hocon 0.42.0 2024-03-04 19:12:42 +01:00
zmstone 6c9eb16a95 refactor(logger): reorder log fields
tag > clientid > msg > peername > username > topic > [other fields]
2024-03-04 18:10:19 +01:00
firest 6d9b2a95c3 Merge remote-tracking branch 'origin/release-55' 2024-03-04 20:47:20 +08:00
Thales Macedo Garitezi 0dd9990f24 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 09:44:30 -03:00
ieQu1 2e792da934
Merge pull request #12562 from ieQu1/dev/new-ds-schema
Refactor schema for the durable storage
2024-03-04 10:33:27 +01:00
Ivan Dyachkov 9f61239704 Merge remote-tracking branch 'upstream/release-56' into 0304-sync-release-56 2024-03-04 08:00:39 +01:00
Ivan Dyachkov a20478fff0 chore: update scripts and workflows for 5.6.x 2024-03-01 09:01:34 +01:00
Zaiming (Stone) Shi 66fbcdcefa test: fix a compile warning 2024-02-29 16:13:52 +01:00
William Yang b82973b36b
Merge pull request #11868 from qzhuyan/fix/william/takeover-pub-willmsg
fix: willmsg not published in takeover
2024-02-29 13:14:18 +01:00
zhongwencool 21e0ecfcce
Merge pull request #12583 from zhongwencool/wait-for-listener-stop
chore: wait_for cowboy listener release sock when stop_listener
2024-02-29 15:03:22 +08:00
zhongwencool 49b508544e
chore: port not released msg
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-02-29 10:49:29 +08:00
Zaiming (Stone) Shi 704eceb7c7
Merge pull request #12599 from zmstone/0227-refactor-log-config-template
docs: add log docs
2024-02-28 17:58:14 +01:00
zhongwencool 660a1ce99d chore: use gen_tcp:recv timeout to wait socket closed 2024-02-28 16:18:32 +08:00
zhongwencool 4307aecefd chore: wait_for cowboy listener release sock when stop_listener 2024-02-28 14:24:15 +08:00
Zaiming (Stone) Shi ef99037335 docs: add log docs 2024-02-27 19:24:56 +01:00
Andrew Mayorov 56eefe34d5
feat(cth-cluster): use workdir as cwd on each node 2024-02-27 16:41:53 +01:00
Andrew Mayorov 3a008c8b4c
refactor: bump ekka to 0.19.0 w/o mnesia boot phase 2024-02-27 16:41:50 +01:00
JimMoen c6b1102b2b
fix(mgmt_client): `nl` not allowed for shared-sub 2024-02-27 17:51:39 +08:00
Zaiming (Stone) Shi 1567c5beb0
Merge pull request #12580 from zmstone/0224-delete-stale-macro
chore: delete stale macro
2024-02-27 09:58:06 +01:00
JimMoen a76415c4f6
fix(mgmt): sub/unsub a share subscription to the client via http api
- `/clients/:clientid/subscribe`
- `/clients/:clientid/subscribe/bulk`
- `/clients/:clientid/unsubscribe`
- `/clients/:clientid/unsubscribe/bulk`
2024-02-27 16:00:45 +08:00
William Yang 6c7b7741d5 feat(quic): mqtt 5.0 graceful shutdown in takeover 2024-02-26 23:05:49 +01:00
William Yang c8f9ffdc1f chore: update some comments 2024-02-26 22:24:16 +01:00
William Yang 88fc67b369 chore(willmsg): remove unreachable code 2024-02-26 22:24:16 +01:00
William Yang d5247cb567 refactor: update notes for willmsg 2024-02-26 22:24:16 +01:00
William Yang 975c7429e5 chore: clean in tests 2024-02-26 22:24:13 +01:00
Zaiming (Stone) Shi bb6a95bae8
Merge pull request #12590 from zmstone/0226-remove-mfa-from-logs
refactor: delete mfa from log metadata
2024-02-26 21:37:57 +01:00
Thales Macedo Garitezi 9ca5684b34
Merge pull request #12591 from thalesmg/dev-cls-m-20240226
test: make `clear_screen` safer
2024-02-26 13:40:16 -03:00
Serge Tupchii 5d4c85cd84 chore: list throttled log events in the change-log 2024-02-26 15:09:38 +02:00
Serge Tupchii 5b48b06d4a fix: trace throttled log events 2024-02-26 15:09:38 +02:00
Thales Macedo Garitezi c6195fcf50 test: make `clear_screen` safer
Useful when iterating on the tests in a loop, to get rid of all the garbaged printed
before the test itself beings.

Only actually does anything if the environment variable `CLEAR_SCREEN` is set to `true`
and only clears the screen the screen the first time it's encountered, so it's harmless
otherwise.
2024-02-26 09:31:06 -03:00
Zaiming (Stone) Shi 00c2aeb1ab refactor: delete mfa from log metadata 2024-02-26 12:15:00 +01:00
Zaiming (Stone) Shi 94a01d23c1 chore: delete stale macro
EMQX_API_VERSION is no longer in use since 5.2.0
2024-02-24 09:06:25 +01:00
Zaiming (Stone) Shi a3e81c5039 chore: prepare for release 5.5.1-rc.2 2024-02-24 08:59:51 +01:00
ieQu1 fe4c7cd2dc
fix(ds): Apply review remarks, and hide certain fields 2024-02-23 18:07:56 +01:00
ieQu1 e126393cbf
fix(ds): Fix schema checker warnings for DS schema 2024-02-23 15:39:52 +01:00
ieQu1 c18fc6a4bb
fix(sessds): Remove deprecated configuration parameters 2024-02-23 15:08:13 +01:00
ieQu1 91ddbbcc3f
fix(sessds): Replace min- and max- batch size with batch_size 2024-02-23 15:08:13 +01:00
ieQu1 94b0ab983d
docs(ds): Add descriptions for the builtin's egress config 2024-02-23 15:08:13 +01:00
ieQu1 17ab3c6362
chore(ds_schema): Use atoms for record and field names 2024-02-23 15:08:13 +01:00
ieQu1 24337ecec7
feat(sessds): Move config schema to a separate root 2024-02-23 15:08:13 +01:00
ieQu1 8907e5afb3
chore(sessds): Remove deprecated schema 2024-02-23 15:08:11 +01:00
Zaiming (Stone) Shi 5af01c041b
Merge pull request #12559 from zmstone/0221-refactor-use-atom-fileds
refactor: use atoms for root config fields
2024-02-23 14:38:19 +01:00
Thales Macedo Garitezi 15f919e60f
Merge pull request #12564 from thalesmg/bw-support-batch-list-resp-m-20240221
feat(resource): allow `on_batch_query{,_async}` to return a list of individual results
2024-02-23 09:37:42 -03:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
William Yang 2ff33f98ef chore(willmsg): add come comments 2024-02-22 17:12:19 +01:00
William Yang e5a3574d89 fix: maybe send willmsg 2024-02-22 17:12:16 +01:00
Zaiming (Stone) Shi 88b1d9ba88 refactor: use atoms for root config fields and types 2024-02-22 16:51:40 +01:00
William Yang 6311b582ec test(willmsg): session taken over before willmsg delay /session expire 2024-02-22 16:39:27 +01:00
William Yang 5397402396 test(willmsg): test will delay and session expires
will delay > session expire
will delay < session expire

timer triggered events are handled in seq, exclude the case of
(will delay == session expire)
2024-02-22 16:39:27 +01:00
William Yang dd62280e04 fix: handle delayed willmsg, part 1 2024-02-22 16:39:27 +01:00
William Yang b76c701b1c test(takeover): add back the delay when takeover 2024-02-22 16:39:27 +01:00
William Yang 6243cf0b0c fix(kick): defer willmsg publish when conn terminates
because kick means shutdown connection AND delete session
2024-02-22 16:39:27 +01:00
William Yang 9da4896f57 fix(mqtt): ensure publish willmsg when session expires 2024-02-22 16:39:24 +01:00
William Yang 07eec31a8a fix: willmsg not published in takeover 2024-02-22 16:35:59 +01:00
zhongwencool ccf607ee2c fix: bpapi announce's bad node name 2024-02-22 14:55:11 +08:00
Thales Macedo Garitezi 985a3e3062 test: remove unused var warning 2024-02-21 18:14:46 -03:00
SergeTupchiy 8e47503f7d
Merge pull request #12560 from SergeTupchiy/EMQX-11530-log-throttling-followup-fixes
fix: disable log throttling if primary log level is debug
2024-02-21 22:02:32 +02:00
Thales Macedo Garitezi f31e9e6a15
Merge pull request #12555 from thalesmg/ds-move-count-to-mgmt-m-20240221
refactor(mgmt): move persistent session counting function to mgmt module
2024-02-21 16:36:29 -03:00
Serge Tupchii ceb208f1eb fix: disable log throttling if primary log level is debug 2024-02-21 21:19:15 +02:00
Zaiming (Stone) Shi 415d27a4c9
Merge pull request #12544 from zmstone/0220-delete-non-prod-code
0220 delete non prod code in emqx_resource.erl
2024-02-21 14:08:08 +01:00
Thales Macedo Garitezi 25cdc2a04b chore: bump version to 5.5.1-rc.1 2024-02-21 09:57:05 -03:00
Thales Macedo Garitezi 03138f8345 refactor(mgmt): move persistent session counting function to mgmt module
Follow up to https://github.com/emqx/emqx/pull/12500#discussion_r1496634087
2024-02-21 09:12:56 -03:00
Thales Macedo Garitezi 529211b9ac
Merge pull request #12500 from thalesmg/ds-list-client-api-m-20240209
feat(ds): list disconnected persistent sessions in clients API
2024-02-21 09:02:27 -03:00
Zaiming (Stone) Shi de93d1c0a3 chore: add bpapi.versions 2024-02-21 10:53:22 +01:00
SergeTupchiy f7a7b7c8e0
Merge pull request #12545 from SergeTupchiy/fix-typo-tls_gc
chore(emqx_tls_certfile_gc): fix typo
2024-02-21 10:11:13 +02:00
Thales Macedo Garitezi 65cfece0a3 chore: delete 5.5.bpapi2
> This particular warning won't be seen in the real life. Also, we need to start storing
bpapi files for each minor release, since the plan to use hot upgrade has been abandoned.

> ...The original procedure for BPAPI life cycle was designed under the assumption that
we'll support hot upgrade between minor releases, so we'll only ever need to keep one
BPAPI file per major release, and rewrite it whenever a new minor version is
released. We'll need to revisit it.

https://emqxio.slack.com/archives/C04H4RHKKEY/p1706862980546309
2024-02-20 17:35:28 -03:00
Thales Macedo Garitezi 74dbbf6d9d Merge branch 'release-55' into sync-r55-m-20240220 2024-02-20 16:33:34 -03:00
Serge Tupchii dd9f96211b chore(emqx_tls_certfile_gc): fix typo 2024-02-20 21:19:36 +02:00
Thales Macedo Garitezi 3a4c7f60e2 feat(ds): list disconnected persistent sessions in clients API
Fixes https://emqx.atlassian.net/browse/EMQX-11540

Note that not all information provided by disconnected in-memory sessions is available to
disconnected persistent sessions, nor does all of them make sense.
2024-02-20 14:52:55 -03:00
Serge Tupchii 9bd0d1ba1b feat: enable log throttling for potentially flooding log events 2024-02-20 14:20:21 +02:00
Serge Tupchii 98ba300f7c feat: implement log throttling 2024-02-20 13:23:17 +02:00
Thales Macedo Garitezi bee7fa630b chore: bump app vsns 2024-02-19 09:05:13 -03:00
Thales Macedo Garitezi dbb6725a47 fix(config): apply config upgrade to deprecated configs
Fixes https://emqx.atlassian.net/browse/EMQX-11845

Fixes https://github.com/emqx/emqx/issues/12452
2024-02-19 09:02:43 -03:00
Zaiming (Stone) Shi 8b0e15e402 refactor: rename 'hint' to 'cause' for MQTT fram parse failure reason
'reason' is maybe the wrapping field's name, so it was not used.
'hint' however, per our logging convention, is usually a free text
description for human to read.
change to 'cause' here because the field is always an atom and
it's use as shutdown counter in esockd_connection_sup
2024-02-19 09:38:35 +01:00
Zaiming (Stone) Shi 668d3be390 test: for better test coverage 2024-02-18 19:23:35 +01:00
Zaiming (Stone) Shi 22dabcb3ea feat(mqtt): add more logging context for frame_too_large error 2024-02-18 18:33:12 +01:00
Zaiming (Stone) Shi 52cfeee2b1
Merge pull request #12526 from zmstone/0216-fix-flaky-tests
test(listeners): try to stop all listeners after each test
2024-02-16 15:33:04 +01:00