Commit Graph

1274 Commits

Author SHA1 Message Date
Kjell Winblad 6e99f01ecd
Merge pull request #12863 from kjellwinblad/kjell/new_trace_log_formatter/EMQX-12025
feat(emqx_trace): add JSON trace log entry formatter
2024-04-22 06:25:18 +02:00
ieQu1 bac5100635
Merge pull request #12874 from ieQu1/dev/EMQX-12030-subscriptions-api
API for durable subscriptions
2024-04-19 17:11:53 +02:00
ieQu1 180130d684
feat(sessds): List persistent subscriptions in the REST API 2024-04-19 01:20:29 +02:00
Kjell Winblad 285bfa9367 fix: improve rendering of action_template_rendered trace 2024-04-18 17:00:47 +02:00
Kjell Winblad aa388adba9 fix(json trace format): format client_ids and rule_ids as lists 2024-04-18 11:27:39 +02:00
Kjell Winblad 10957e7d79 fix: change name of the default trace log format from plain to text 2024-04-18 10:42:51 +02:00
Kjell Winblad aa39835242 feat(emqx_cli): add parameter for trace format (plain or json) 2024-04-18 10:28:29 +02:00
Kjell Winblad a6558740e8 feat(emqx_trace): add JSON trace log entry formatter
This commit makes it possible to select the JSON trace log entry
formatter when crating a trace pattern. This will make it easier for the
dashboard and automatic tests to parse the log entries.

Fixes: EMQX-12025 (partly)
2024-04-18 10:20:11 +02:00
Kjell Winblad 2a2fadfbff
Merge pull request #12827 from kjellwinblad/kjell/emqx_rule_tracing/EMQX-11966
Add rule ID tracing support
2024-04-18 09:06:31 +02:00
ieQu1 38a2e8add9
fix(sessds): Return the number of subscriptions for offline sessions 2024-04-17 12:31:54 +02:00
ieQu1 e439a2e0f2
fix(sessds): Save protocol name and version in the session metadata 2024-04-17 12:31:54 +02:00
ieQu1 197a4c30be
fix(sessds): Strip unneccessary data from the durable session state 2024-04-17 01:21:52 +02:00
ieQu1 6c83bbe10b
feat(mgmt): Filter subscriptions by durability 2024-04-17 01:21:51 +02:00
Ivan Dyachkov 3ef160eed2 Merge remote-tracking branch 'upstream/release-56' into 0415-sync-release-56 2024-04-15 08:09:03 +02:00
Kjell Winblad ed5409fb6a docs(trace): add emqx_ctl documentation for the new ruleid trace 2024-04-12 11:54:14 +02:00
JianBo He 91ffc95f29
Merge pull request #12855 from JimMoen/fix-share-queue-format
fix(mgmt): $queue shared topics format in mgmt topics api
2024-04-10 10:52:48 +08:00
Serge Tupchii fae9005f87 test(emqx_mgmt_cli): test that replicants do not join a left core node 2024-04-09 19:06:13 +03:00
JimMoen 47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api 2024-04-09 17:06:32 +08:00
JimMoen 6cb00cc8c7
Merge pull request #12844 from JimMoen/EMQX-12012/cpu-use-idle-two-decimal-places
fix: cpu usage and idle use two decimal places
2024-04-09 10:11:15 +08:00
Thales Macedo Garitezi ba96edb061 fix(clients api): use alternative base64 function for OTP 25
Fixes https://github.com/emqx/emqx/pull/12798#discussion_r1555524603
2024-04-08 10:23:10 -03:00
JianBo He e11c4a9c83
Merge pull request #12826 from emqx/import-source-bridges
fix: source bridges missing after restore the backup files
2024-04-08 16:02:15 +08:00
JimMoen 282cbb18be
fix: cpu usage and idle use two decimal places
- prometheus
- opentelemetry
2024-04-08 14:14:09 +08:00
Shawn e89dc32c90 ci: run emqx_management both with ee and ce profile 2024-04-07 18:33:52 +08:00
Shawn 1c81c79a2c chore: add testcase for importing retained msgs and sources 2024-04-07 17:24:26 +08:00
ieQu1 2504b8126b
feat(ds): Pass mgmt_ds REST API calls to the application 2024-04-05 15:22:06 +02:00
ieQu1 46261440cb
feat(ds): Add a CLI for managing DB replicas 2024-04-05 15:22:06 +02:00
ieQu1 a62db08676
feat(ds): Add REST API for durable storage 2024-04-05 15:22:06 +02:00
Kjell Winblad 59a442cdb5 feat(rule trace): add support for ruleid as a trace type 2024-04-04 14:55:32 +02:00
Shawn 319ec50c0d fix: source bridges missing after restore the backup files 2024-04-03 18:26:51 +08:00
Thales Macedo Garitezi 2097e854fc feat(client mgmt api): add cursor-based list API
Fixes https://emqx.atlassian.net/browse/EMQX-12028
2024-04-02 10:55:28 -03:00
JimMoen 1a4cfc2a2d
fix(api_schema): removed metrics schema in api spec
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622).
2024-04-02 16:56:36 +08:00
Serge Tupchii ceb04ba06d fix(emqx_mgmt): do not attempt to get a stacktrace of a remote client connection process 2024-04-01 16:42:12 +03:00
Serge Tupchii 42af1f9d63 fix: handle internal timeout errors in client Mqueue/Inflight APIs 2024-03-29 23:03:35 +02:00
Serge Tupchii f5a820cb10 fix(emqx_mgmt): catch OOM shutdown exits properly when calling a client conn process
The exit reason is expected to include gen_server `Location`:
  `{{shutdown, OOMInfo}, Location}`.
2024-03-29 13:09:08 +02:00
SergeTupchiy 2e528d1dd8
Merge pull request #12802 from SergeTupchiy/EMQX-11826-prevent-left-node-from-rejoining-5.6.1
prevent a left node from rejoining the same cluster
2024-03-28 19:49:18 +02:00
Serge Tupchii 3eda182e9a fix: prevent a node from discovering and re-joining the same cluster after it has (manually) left it. 2024-03-28 18:09:27 +02:00
Serge Tupchii 1d1f595e6f fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid 2024-03-28 12:54:51 +02:00
Ivan Dyachkov db9efb9317 chore: bump apps versions 2024-03-28 10:19:09 +01:00
Ivan Dyachkov f4446ec680 Merge remote-tracking branch 'upstream/release-56' into 0328-sync-release-56 2024-03-28 09:59:54 +01:00
Serge Tupchii ed0b5ac939 fix(emqx_mgmt): handle client shutdown errors in call_client
The client process may be shutdown normally during/shortly before the call.
2024-03-22 16:47:02 +02:00
Serge Tupchii 5d77e1b01e fix: return not_implemented error for mqueue/inflight msgs queries 2024-03-22 10:02:31 +02:00
JimMoen 9b085b5797
fix(api_schema): removed metrics schema in api spec
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622).
2024-03-22 10:57:14 +08:00
Serge Tupchii 40eccb10d6 fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid 2024-03-21 17:36:39 +02:00
Andrew Mayorov 7d6fde5960
fix(topics-api): respond with correct totals to paged queries 2024-03-21 16:02:16 +01:00
ieQu1 cada944350
feat(sessds): Expose relevant durable session info in the REST API 2024-03-21 10:37:04 +01:00
SergeTupchiy 27c1f1a4e4
Merge pull request #12732 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API-updates
EMQX-11861 client mqueue inflight api updates
2024-03-20 16:27:44 +02:00
Serge Tupchii cb5fdb3c79 fix: rework In-flight / Mqueue API
- use timestamp 'position' to find the next chunk of data
- add 'start' position to response meta
- sort In-flight messages by insertion time
- sort Mqueue messages by priority
2024-03-20 15:25:33 +02:00
ieQu1 783f769a98 fix(mgmt): Fix bulk kickout of durable sessions 2024-03-20 12:52:44 +01:00
ieQu1 611e4f6710
Merge pull request #12740 from ieQu1/dev/kick-durable-sessions
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 21:30:40 +01:00
Andrew Mayorov e2db038e7c
Merge pull request #12734 from keynslug/fix/EMQX-12030/topics-api
fix(api-topics): expose persistent session topics in APIs
2024-03-19 21:08:23 +01:00
ieQu1 236685f714 fix(sessds): Channel must destroy sessions that are kicked 2024-03-19 17:49:04 +01:00
Ivan Dyachkov f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Andrew Mayorov 10ea367030
test(client-api): attempt to get rid of inter-testcase side effects 2024-03-19 14:56:50 +01:00
Andrew Mayorov 3129a8fa42
test(client-api): fix snabbkaffe crash noise in the CT logs 2024-03-19 14:55:28 +01:00
JianBo He c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
zhongwencool 111cbbbe3c fix: don't crash when ban ip error 2024-03-19 19:10:36 +08:00
Andrew Mayorov 985f1c4879
fix(api-topics): expose persistent session topics in APIs 2024-03-19 11:01:26 +01:00
Serge Tupchii b3d44125f6 feat: support multiple clientid / username Qs params in "/clients" API
Additionally, add an option to specify which Client info fields to return in the response.
2024-03-18 18:00:26 +02:00
ieQu1 a93f747afa fix(sessds): Return peername of the disconnected client in the REST 2024-03-15 13:48:53 +01:00
SergeTupchiy e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client
Refactor emqx mgmt call client
2024-03-15 10:01:49 +02:00
zmstone d8bdb3c9aa fix(api/clients): drop expired sessions from durable storage 2024-03-14 20:57:53 +01:00
Zaiming (Stone) Shi cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node
fix: do not crash on replicant node
2024-03-14 09:42:27 +01:00
zmstone ccd973d13e fix: do not crash on replicant node 2024-03-13 16:42:39 +01:00
Serge Tupchii e514c4a2ac fix(emqx_mgmt_api_clients): check that max_payload_bytes is higher than 0 2024-03-13 14:00:30 +02:00
Serge Tupchii 65be76aa06 refactor(emqx_mgmt): add call_client timeout and improve RPC error handling 2024-03-13 10:56:02 +02:00
Serge Tupchii 8be02327b2 refactor(emqx_mgmt): avoid call_client RPC to all runing nodes if global cm_registry is enabled 2024-03-12 20:58:53 +02:00
JimMoen 207f38c42a
fix(vm): cpu usage/idle handled by single worker 2024-03-12 14:14:44 +08:00
Serge Tupchii 980a4c3a4d fix(emqx_mgmt_api): use emqx_utils:bin_to_hexstr/2 2024-03-11 10:40:19 +02:00
Serge Tupchii f376aa8072 refactor(emqx_mgmt_api_clients): remove unnecessary function 2024-03-11 10:39:09 +02:00
zmstone 2cc1c563df fix(mgmt): avoid using base64:decode/2 and encode/2
they are not available in otp 25
2024-03-08 16:46:54 +01:00
SergeTupchiy 1f38813cb9
Merge pull request #12561 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API
feat: add client mqueue/inflight messages API
2024-03-07 13:20:31 +02:00
Serge Tupchii a2e761681e feat: add client mqueue/inflight messages API 2024-03-07 12:06:49 +02:00
zmstone f57f2fa1b7 chore: bump app version numbers 2024-03-06 19:37:06 +01:00
Thales Macedo Garitezi 0dd9990f24 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 09:44:30 -03:00
JimMoen 9143d5994d
feat: refactor MS SQL Server bridge to connector and action 2024-03-04 17:13:21 +08:00
Andrew Mayorov c3b044a37b
fix(api-topics): respond with `count` when it's cheap to compute 2024-02-28 11:28:33 +01:00
Andrew Mayorov bd578a799e
fix(api-topics): avoid doing full scans over router tables 2024-02-28 10:52:44 +01:00
Andrew Mayorov 9bb0868628
test(api-topics): add testcase verifying paged queries work 2024-02-28 10:52:41 +01:00
JimMoen 2ded751237
test: `nl` not allowed for shared-sub 2024-02-27 18:24:30 +08:00
JimMoen c6b1102b2b
fix(mgmt_client): `nl` not allowed for shared-sub 2024-02-27 17:51:39 +08:00
JimMoen c1da7449fe
test(mgmt_api): bulk/unbulk sub/unsub shared topic filter 2024-02-27 16:01:12 +08:00
JimMoen a76415c4f6
fix(mgmt): sub/unsub a share subscription to the client via http api
- `/clients/:clientid/subscribe`
- `/clients/:clientid/subscribe/bulk`
- `/clients/:clientid/unsubscribe`
- `/clients/:clientid/unsubscribe/bulk`
2024-02-27 16:00:45 +08:00
Zaiming (Stone) Shi 5af01c041b
Merge pull request #12559 from zmstone/0221-refactor-use-atom-fileds
refactor: use atoms for root config fields
2024-02-23 14:38:19 +01:00
zhongwencool fbf4afc3c5 fix(api): more detail error when don't allow update readonly key 2024-02-23 16:43:15 +08:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Zaiming (Stone) Shi 88b1d9ba88 refactor: use atoms for root config fields and types 2024-02-22 16:51:40 +01:00
firest f0aeaadc48 fix(api_key): enhanced bootstrap files for REST API keys 2024-02-22 18:40:24 +08:00
Thales Macedo Garitezi 03138f8345 refactor(mgmt): move persistent session counting function to mgmt module
Follow up to https://github.com/emqx/emqx/pull/12500#discussion_r1496634087
2024-02-21 09:12:56 -03:00
Thales Macedo Garitezi 529211b9ac
Merge pull request #12500 from thalesmg/ds-list-client-api-m-20240209
feat(ds): list disconnected persistent sessions in clients API
2024-02-21 09:02:27 -03:00
Thales Macedo Garitezi d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
Thales Macedo Garitezi 3a4c7f60e2 feat(ds): list disconnected persistent sessions in clients API
Fixes https://emqx.atlassian.net/browse/EMQX-11540

Note that not all information provided by disconnected in-memory sessions is available to
disconnected persistent sessions, nor does all of them make sense.
2024-02-20 14:52:55 -03:00
Zaiming (Stone) Shi 2b46cbab7a
Merge pull request #12517 from zmstone/0213-hocon-multiline-string-with-indentation
0213 hocon multiline string with indentation
2024-02-16 15:12:46 +01:00
Zaiming (Stone) Shi f57f617ba3 refactor(schema): ensure roots/0 and namespace/0 for all schema modules 2024-02-16 11:35:32 +01:00
Andrew Mayorov 6514659733
test(mgmt): update and simplify testsuites setup 2024-02-15 11:04:58 +01:00
Andrew Mayorov b6d77c164e
feat(emqx-mgmt): manage mria tables explicitly during startup 2024-02-13 20:36:22 +01:00
Ilya Averyanov 90fd2b26d3 feat(banned): allow ban by clientid/username regexps, peerhost cidrs 2024-02-10 17:59:22 +03:00
Andrew Mayorov 0515c5528f test: add facility for starting `emqx_dashboard` with `emqx_cth_suite` 2024-02-09 14:29:04 -03:00
Zaiming (Stone) Shi b1a05c7b59
Merge pull request #12326 from zmstone/0111-unregister-session-with-timestamp
0111 unregister session with timestamp
2024-02-02 17:00:23 +01:00
ieQu1 3d2ac97c61 feat(ds): Add a CLI interface to inspect status of DS databases 2024-02-02 10:11:01 +01:00
Zaiming (Stone) Shi f0569d8ae8 refactor: use mria:async_dirty to group dirty ops 2024-02-02 09:26:43 +01:00
ieQu1 98d1094d73 feat(sessds): Expose subscriptions in the REST API 2024-02-01 16:47:57 +01:00
Zaiming (Stone) Shi 87a2368e37 feat(metrics): add cluster_session guage 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 509ab6f35a feat(api): add /sessions_count api to count sessions 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 562a2736ae feat: add `broker` root to hot-config schema 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 55fdc83aa5 Merge remote-tracking branch 'origin/release-55' into 0130-syn-release-55-to-master 2024-01-30 12:06:43 +01:00
JianBo He 4190682a16 chore: fix the data type and example value for cluster invitation result 2024-01-30 15:29:46 +08:00
zhongwencool 186e1591df chore: document api-key path api only support bearerAuth 2024-01-25 16:01:43 +08:00
JianBo He 52a7b45d97 fix: upgrade ekka to 0.18.3 to ensure the leader correct 2024-01-25 14:47:59 +08:00
Thales Macedo Garitezi 3207f0ea80 fix(machine_boot): inject connector/bridge apps as dependencies to `emqx_connector` app
Fixes https://emqx.atlassian.net/browse/EMQX-11771

For the same reasons as we inject `emqx_bridge_*` applications as dependencies to
`emqx_bridge` when starting the node.  Already configured connectors are started when
`emqx_connector` application starts, and may lead to crashes and noise in the logs.  One
example is to configure a mongodb bridge and restart the node.
2024-01-22 11:13:34 -03:00
Thales Macedo Garitezi 862283ff7c test: fix expected connector name after name convention generation changed 2024-01-16 10:42:52 -03:00
JianBo He 4c40e754f4 chore: forward the async invite to leader node 2024-01-15 11:24:05 +08:00
JianBo He 93ef6766ef chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-13 15:29:58 +08:00
JianBo He d2991dae03 chore: make dialyzer happy 2024-01-13 15:29:58 +08:00
JianBo He 81209e0ded chore: update changes 2024-01-13 15:29:58 +08:00
JianBo He 944137ad45 chore: format error message 2024-01-13 15:29:58 +08:00
JianBo He 6ff4c560e4 feat: support invite node in async mananer 2024-01-13 15:29:58 +08:00
JianBo He a08f56db42 feat(cluster): expose the timeout parameter to invite node 2024-01-13 15:29:58 +08:00
lafirest c026d3f18a
Merge pull request #12300 from lafirest/fix/clients_api
fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT
2024-01-11 19:10:54 +08:00
firest aca9942fd4 fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT 2024-01-11 16:53:02 +08:00
Zaiming (Stone) Shi 7c8a36fc06
Merge pull request #12289 from zmstone/0108-support-acl-cache-masks
0108 support acl cache excludes
2024-01-10 13:40:29 +00:00
Zaiming (Stone) Shi 6fcb2fdb05 test: fix test case flakyness 2024-01-10 13:52:00 +01:00
JianBo He 6ab8e31db0 test: fix failed tests 2024-01-10 09:53:22 +08:00
JianBo He da0307faa9 chore: update apps/emqx_management/src/emqx_mgmt_api_clients.erl
Co-authored-by: William Yang <mscame@gmail.com>
2024-01-10 09:52:27 +08:00
JianBo He f45298d14a chore: improve the error message if the parameter invalid 2024-01-10 09:52:27 +08:00
Ivan Dyachkov 06117c3a33 Merge remote-tracking branch 'upstream/release-54' into 0105-sync-r54 2024-01-05 14:20:38 +01:00
zhongwencool 00e8a36dbe fix: the API prompt garbled characters for updating the configuration 2024-01-03 15:34:45 +08:00
JianBo He 57c0a7c5ef
Merge pull request #12235 from zhongwencool/ft-config-update-api
fix: ft config update api return raw config
2024-01-03 14:07:22 +08:00
JimMoen a3f33cc8bc
chore: fix typos in code and suite 2023-12-29 09:07:51 +08:00
zhongwencool e645fa4874 fix: ft config update api return raw config 2023-12-27 21:13:47 +08:00
zhongwencool 11feab983f fix: clean rpc_commit when force_leave cluster 2023-12-26 15:25:58 +08:00
Zaiming (Stone) Shi 86a018d921 chore: bump app versions 2023-12-22 21:28:26 +01:00
Zaiming (Stone) Shi 9fdac4af0c Merge remote-tracking branch 'origin/master' into 1222-sync-e5.4.0-build.2-to-master 2023-12-22 21:27:27 +01:00
Zaiming (Stone) Shi ac6e4cf2cc fix: config update for ft 2023-12-22 19:01:14 +01:00
Zaiming (Stone) Shi 23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi 20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi 322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Thales Macedo Garitezi 035f5f977e Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231221 2023-12-21 17:53:05 -03:00
zhongwencool 37930f1d3c fix: emqx conf ctl load failed with connectors/actions 2023-12-21 22:09:53 +08:00
Thales Macedo Garitezi 18b69809da Merge branch 'release-54' into sync-r54-m-20231221 2023-12-21 10:05:13 -03:00
zhongwencool 5d1a412d0c fix: redis resource_opts not working 2023-12-21 18:29:49 +08:00
zhongwencool a4fe4ceb9e chore: add example for clients api 2023-12-20 14:53:05 +08:00
zhongwencool a46b415c77 fix: support bridge v1 conf hocon-format in the put configs API. 2023-12-19 16:40:45 +08:00
Zaiming (Stone) Shi 35504bd323 refactor: move ntoa (ip address formatting) code to emqx_utils 2023-12-17 21:03:16 +01:00
zhongwencool 96d21d4dbe fix: ensure config update error text is readable 2023-12-13 17:07:28 +08:00
JimMoen 7b60928288
fix(api): endpoint `/v5/topics` with invalid topic filter 2023-12-11 15:36:50 +08:00
Zaiming (Stone) Shi 7e3ed05f87
Merge pull request #12115 from zmstone/1206-fix-dialyzer
1206 fix dialyzer
2023-12-07 12:24:47 +01:00
zhongwencool 253ea6093a
Merge pull request #12111 from zhongwencool/fix-token-not-found
fix: use sync_transaction when update admin/token
2023-12-07 09:56:20 +08:00
Zaiming (Stone) Shi 423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Thales Macedo Garitezi b82189fb4d test(cth_peer): use an exclusive current dir for each peer 2023-12-06 09:35:13 -03:00
zhongwencool b015d08cba fix: use sync_transation when update admin/token 2023-12-06 09:04:27 +08:00