Commit Graph

11786 Commits

Author SHA1 Message Date
zmstone f2fa35efdf Merge remote-tracking branch 'origin/master' into 0424-merge-latest-master-to-release-57 2024-04-24 22:02:56 +02:00
Serge Tupchii 02a0faa1d6 refactor: avoid evaluating Data more than once in SLOG_THROTTE macro 2024-04-24 16:26:34 +03:00
zmstone b66d7a688e Merge remote-tracking branch 'origin/release-57' into release-57 2024-04-24 13:09:06 +02:00
zhongwencool 4f12d45436 chore: provided more specific error for wrong import method 2024-04-24 15:37:35 +08:00
zhongwencool 712008bc1d
Merge pull request #12909 from zhongwencool/esockd-bump-version
fix: bump escokd to 5.11.2 to handle udp_error/closed message
2024-04-24 09:15:39 +08:00
ieQu1 5804b70c1f
Merge pull request #12918 from ieQu1/dev/ds-readme1
docs(ds): Update README with CLI and REST API endpoints
2024-04-23 17:02:01 +02:00
ieQu1 e4c3283c9c docs(ds): Update README with CLI and REST API endpoints 2024-04-23 16:28:35 +02:00
Kjell Winblad e1eed30b5d
Merge pull request #12912 from kjellwinblad/kjell/rule_trigger_time/EMQX-12025
feat(rule tracing): add rule trigger time meta data field
2024-04-23 15:00:05 +02:00
lafirest 0d1c13661f
Merge pull request #12895 from lafirest/fix/dyndb
fix(dynamo): Added missing keys for DynamoDB
2024-04-22 23:03:00 +08:00
lafirest a81ca359fc
Merge pull request #12899 from lafirest/fix/rocmq
feat(rocketmq): add support for namespace and key dispatch strategy
2024-04-22 21:20:44 +08:00
Kjell Winblad e9d498dde2 feat(rule tracing): add rule trigger time meta data field
Fixes:
https://emqx.atlassian.net/browse/EMQX-12025
2024-04-22 13:51:48 +02:00
zhongwencool 6049b4966f fix: bump escokd to 5.11.2 to handle udp_error/closed message 2024-04-22 16:41:09 +08:00
ieQu1 f9eda1883f
Merge pull request #12903 from ieQu1/dev/ds-egress-flush-condition
fix(ds): Fix egress flush condition
2024-04-22 10:17:22 +02:00
JianBo He c5c4bb987f
Merge pull request #12902 from HJianBo/stomp
Pass the Content-type of MQTT message to the Stomp message
2024-04-22 16:02:03 +08:00
Ivan Dyachkov b723e87e54 Merge remote-tracking branch 'upstream/release-56' into 0422-sync-release-56 2024-04-22 08:12:05 +02:00
firest d5cdc07eab feat(rocketmq): add support for namespace and key dispatch strategy 2024-04-22 13:14:11 +08:00
firest 46f339dfab chore: updte change log 2024-04-22 13:12:01 +08:00
firest a1e85e3c59 fix(dynamo): Added missing keys for DynamoDB 2024-04-22 13:09:34 +08:00
Kjell Winblad 6e99f01ecd
Merge pull request #12863 from kjellwinblad/kjell/new_trace_log_formatter/EMQX-12025
feat(emqx_trace): add JSON trace log entry formatter
2024-04-22 06:25:18 +02:00
JianBo He d85df14b85
Merge pull request #12892 from HJianBo/fix-gateway-related-issues
fix(ocpp): avoid an error log in handling downstream messages
2024-04-22 11:25:26 +08:00
JianBo He 69bdcd2f24
Merge pull request #12898 from emqx/iotdb-1.3.0
feat: support iotdb 1.3.0
2024-04-22 10:56:06 +08:00
ieQu1 4c76a2574d
fix(ds): Fix egress flush condition 2024-04-21 21:51:31 +02:00
Zaiming (Stone) Shi 307cd79be2
Merge pull request #12897 from zmstone/0418-support-dynamic-license
feat(license): add business-critical customer type
2024-04-20 09:03:58 +02:00
JianBo He 5520e54147 chore: add tests 2024-04-20 08:59:36 +08:00
JianBo He aaf7953968 chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-04-20 08:36:59 +08:00
JianBo He 95f3e49edb fix(stomp): pass the Content-Type from the MQTT message 2024-04-20 08:33:55 +08:00
Andrew Mayorov 43f8346c00
fix(dssnap): ensure idempotent write of empty chunks 2024-04-19 18:52:33 +02:00
ieQu1 bac5100635
Merge pull request #12874 from ieQu1/dev/EMQX-12030-subscriptions-api
API for durable subscriptions
2024-04-19 17:11:53 +02:00
Kjell Winblad 2890bc2619 fix(tracing): remove internal extra field from the trace config
This commit removes the internal extra field from the trace config
structure exposed to the user via the HTTP API.
2024-04-19 17:05:34 +02:00
Kjell Winblad 9e46c18443 fix(emqx_trace_SUITE:t_base_create_delete): broken test case 2024-04-19 15:41:50 +02:00
zmstone ec83fbe3dc feat(license): add business critical customer type 2024-04-19 14:19:39 +02:00
ieQu1 ede7246882
fix(sessds): Avoid double-enriching transient messages 2024-04-19 13:39:04 +02:00
Ilia Averianov b7a4536f47
Merge pull request #12871 from savonarola/0412-fix-rebalance-app-dependencies
fix(rebalance): fix start order of rebalance applications
2024-04-19 13:27:41 +03:00
Zaiming (Stone) Shi a8b1224225
Merge pull request #12893 from zmstone/0416-add-is-template-flag-to-dashboard-schema
0416 add `is_template` flag to dashboard schema
2024-04-19 10:41:43 +02:00
Kjell Winblad c163956d08 fix(trace formatter): remove record field to enable rolling upgrade 2024-04-19 09:52:20 +02:00
zhongwencool b163a87386 feat: support batch_size on iotdb 2024-04-19 14:52:56 +08:00
zhongwencool c0521fd250 feat: support iotdb 1.3.0 2024-04-19 08:54:18 +08:00
ieQu1 f1e6565ddd
refactor(sessds): Move all subscription logic to the subs module 2024-04-19 01:20:29 +02:00
ieQu1 d12966db5b
test: Avoid dumping raw snabbkaffe traces to the console 2024-04-19 01:20:29 +02:00
ieQu1 180130d684
feat(sessds): List persistent subscriptions in the REST API 2024-04-19 01:20:29 +02:00
Kjell Winblad 09b414f368 test: add necessary application to test suites 2024-04-18 17:39:52 +02:00
Kjell Winblad 285bfa9367 fix: improve rendering of action_template_rendered trace 2024-04-18 17:00:47 +02:00
Kjell Winblad 7be18730e8 test(emqx_rule_engine_api_rule_apply_SUITE): remove unnecessary code 2024-04-18 15:16:18 +02:00
Kjell Winblad f5b0439724 fix(emqx_rule_engine_api_rule_apply_SUITE): flaky test case 2024-04-18 14:28:28 +02:00
Kjell Winblad 8f1486f6d3 fix: clean up trace messages to make it easier to interpret
This commit removes some redundant trace messages and renames some to
make it easier to interpret what is happening for the user.
2024-04-18 14:28:01 +02:00
zmstone 5b38d592f0 feat(http): add `is_template` as HTTP headers field property
is_template was designed to be type property.
however for HTTP headers, it's a map() type,
instead of creating a new type for it, it's easier to just
add it as a field property.
2024-04-18 13:16:29 +02:00
zmstone ede4eeae9f fix(http_bridge): path is template field 2024-04-18 13:04:56 +02:00
zmstone ca56e7e8d7 fix(kafka): headers are template fields 2024-04-18 13:04:36 +02:00
Kjell Winblad 3232ab5ea3 fix(rule tracing): unset trace meta data in try-after-end
We wrap the reset of the process trace meta data in the after clause of
a try-after-end expression to be sure we never get any lingering
incorrect meta data.
2024-04-18 11:28:50 +02:00
Kjell Winblad aa388adba9 fix(json trace format): format client_ids and rule_ids as lists 2024-04-18 11:27:39 +02:00
Kjell Winblad 10957e7d79 fix: change name of the default trace log format from plain to text 2024-04-18 10:42:51 +02:00
Kjell Winblad ef9f8a8fdf fix: unreachable clause found by dialyzer 2024-04-18 10:28:37 +02:00
Kjell Winblad 6c0ee8bb01 test(emqx_trace_SUITE): fix failure due to new field 2024-04-18 10:28:37 +02:00
Kjell Winblad 968dc2ccda fix(json trace format): use best_effort_json instead of new encoder 2024-04-18 10:28:37 +02:00
Kjell Winblad aa39835242 feat(emqx_cli): add parameter for trace format (plain or json) 2024-04-18 10:28:29 +02:00
Kjell Winblad 6fd7a06c5d fix: problems reported by dialyzer 2024-04-18 10:20:18 +02:00
Kjell Winblad a6558740e8 feat(emqx_trace): add JSON trace log entry formatter
This commit makes it possible to select the JSON trace log entry
formatter when crating a trace pattern. This will make it easier for the
dashboard and automatic tests to parse the log entries.

Fixes: EMQX-12025 (partly)
2024-04-18 10:20:11 +02:00
zmstone ab763fe665 test: fix test case flakyness 2024-04-18 09:32:05 +02:00
Kjell Winblad 2a2fadfbff
Merge pull request #12827 from kjellwinblad/kjell/emqx_rule_tracing/EMQX-11966
Add rule ID tracing support
2024-04-18 09:06:31 +02:00
JianBo He d7ebecddb4 fix(ocpp): return correct current_connections number of listenrs http api 2024-04-18 14:54:12 +08:00
Ivan Dyachkov e93bd314bc chore: 5.6.1 2024-04-18 08:03:56 +02:00
JianBo He 75632bb2cd chore(gw): update listener examples 2024-04-18 13:58:31 +08:00
Kjell Winblad b16b9d8fcc refactor: made code more readable thanks to suggestion from @zmstone 2024-04-18 06:51:30 +02:00
zmstone 55941000c0 test: make test case more stable, less flaky 2024-04-17 20:42:03 +02:00
zmstone 6ab2b004ed fix(resource_manager): update cache after channel add 2024-04-17 20:34:33 +02:00
zmstone c96ae8dd23 fix: return 503 if bridge bpapi call timeout 2024-04-17 20:34:33 +02:00
zmstone 5a4bfff9e5 refactor: add template_str type 2024-04-17 20:34:33 +02:00
zmstone 5c014f4c29 test: fix test cases 2024-04-17 20:34:33 +02:00
Kjell Winblad 95891db29a test: clean up created resources in the end of test case 2024-04-17 18:43:19 +02:00
Kjell Winblad 6cf29ba688 fix: clean up traces to make them easier to parse and understand 2024-04-17 18:20:17 +02:00
Kjell Winblad cf56050759 feat: avoid mixing request with and without the stop_after_render flag
Previously a batch of requests that was sent to a connector could
contain both requests with the stop_after_rendering flag and requests
without this flag. When this happened a warning message was generated and
the stop_after_render flags for the batch would be ignored. This commit
fixes so that a mixed batch is never created so there is no longer any
need for a warning message or ignoring flags.
2024-04-17 16:21:21 +02:00
zmstone d49e98bc4b test: fix dashboard schema validation 2024-04-17 14:09:48 +02:00
zmstone f9f14f9758 refactor(emqx_conf): raise exception at higher level for more context 2024-04-17 13:10:55 +02:00
zmstone 51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
JianBo He 2468243dfd chore: update changes 2024-04-17 18:52:48 +08:00
ieQu1 124c5047d0
feat(sessds): Add API for getting session data from the cold storage 2024-04-17 12:31:54 +02:00
ieQu1 38a2e8add9
fix(sessds): Return the number of subscriptions for offline sessions 2024-04-17 12:31:54 +02:00
ieQu1 e439a2e0f2
fix(sessds): Save protocol name and version in the session metadata 2024-04-17 12:31:54 +02:00
JianBo He 18196ec19c fix(ocpp): avoid an error log in handling dnsteeam messages 2024-04-17 17:48:19 +08:00
JianBo He c61c3157c6
Merge pull request #12888 from HJianBo/data-backup-for-license
fix(license): license file is not taking effect after importing backup files
2024-04-17 16:31:16 +08:00
JimMoen 1dfd9115cd
Merge pull request #12880 from JimMoen/EMQX-12147/influx-write-syntax-tag-type
fix(influx): literal number values in tag set
2024-04-17 10:23:57 +08:00
ieQu1 197a4c30be
fix(sessds): Strip unneccessary data from the durable session state 2024-04-17 01:21:52 +02:00
ieQu1 93bb840365
docs(ds): Update README 2024-04-17 01:21:52 +02:00
ieQu1 87ffaf89e5
refactor(sessds_state): Use macros for map keys 2024-04-17 01:21:52 +02:00
ieQu1 113a990482
feat(sessds): Support max subscriptions 2024-04-17 01:21:51 +02:00
ieQu1 6c897c26ae
fix(sessds): Commit session on unsubscribe 2024-04-17 01:21:51 +02:00
ieQu1 6c83bbe10b
feat(mgmt): Filter subscriptions by durability 2024-04-17 01:21:51 +02:00
ieQu1 b30ddc206e
fix(sessds): Immutable subscriptions
This commit fixes two issues:
- Behavior of overlapping subscriptions has been aligned with the
in-memory session.

- Fixed handling of replays when subscription changes (either by
client or EMQX configuration)
2024-04-17 01:21:51 +02:00
ieQu1 3e0c649e8e
feat(sessds): Store awaiting rel 2024-04-17 01:21:51 +02:00
zmstone df458b98d7 refactor(dashboard_schema): no need to translate labels
the trans_label implementation was ugly, it compares an anonymous
function to check if the label should be translated.

since we have stopped generating i18n message ids for dashboard
schema, this entire function is now stale, so this function
is deleted.
2024-04-16 17:54:34 +02:00
zmstone 2bd72aab44 chore: bump dashboard schema version to 0.2.0 2024-04-16 16:58:45 +02:00
Zaiming (Stone) Shi 4d38a8fd44
Merge pull request #12872 from zmstone/0412-support-variform-for-client_attrs
0412 use variform for client_attrs
2024-04-16 13:29:56 +02:00
JianBo He 6b8111c066 fix(license): license file is not taking effect after importing backup files 2024-04-16 19:07:17 +08:00
ieQu1 c645cfa5d6
fix(sessds): Graceful handling of shared subscription error 2024-04-16 12:37:01 +02:00
Ivan Dyachkov f3bb28c6af chore: 5.6.1-rc.2 2024-04-16 09:47:32 +02:00
Andrew Mayorov 5d7b2e2ce6
fix(dsrepl): attempt leadership transfer on terminate
In addition to on removal. The reasoning is basically the same: try to
avoid situations when log entries are replicated (or will be considered
replicated when the new leader is elected) but the leader terminates
before replying to the client.

To be clear: this is a stupid solution. Something much more robust is
needed.
2024-04-15 22:05:24 +02:00
Ivan Dyachkov a0ffe5e7ae chore: 5.6.1-rc.1 2024-04-15 22:05:05 +02:00
Kjell Winblad f481871792 fix(rocketmq action): we need one producer group per channel and topic
We need one producer group per channel and topic because we can have
several topics per channel due to templating.
2024-04-15 20:52:08 +02:00
Kjell Winblad 1fe92bddd0 fix(rocketmq action): make sure that topic template is respected 2024-04-15 20:37:23 +02:00
Kjell Winblad 7f9a311988 fix(rocketmq action): all actions used only one topic
This commit makes sure that a set of rocketmq workers are started for
each topic. Before this commit all actions for a rocketmq connector used
the same workers which all were bound to a single topic so all messages
got delivered to that topic regardless of the action configuration.

We should have automatic tests to check that the messages actually go to
different topics but this needs to go into another PR since we have to
deliver the fix fast and the rocketmq library does not support reading
from topics.

Fixes:
https://emqx.atlassian.net/browse/EEC-1006
2024-04-15 20:08:56 +02:00
zmstone f80d078de3 feat(variform): Add more functions 2024-04-15 16:56:52 +02:00
Andrew Mayorov 89f42f1171
fix(dsrepl): make placeholder shard process permanent under supervisor 2024-04-15 16:43:52 +02:00
Andrew Mayorov c4d1360b96
fix(dsrepl): trigger election for new ra servers unconditionallly
Otherwise we might end up in a situation when there's no member online
yet at the time of the election trigger, and the election will never
happen.
2024-04-15 16:42:29 +02:00
JimMoen 084e920c6e
test(influx): literal values or variable in tag set 2024-04-15 17:47:53 +08:00
JimMoen 8ed397d4fa
fix(influx): literal number values in tag set 2024-04-15 17:47:52 +08:00
Ivan Dyachkov 3ef160eed2 Merge remote-tracking branch 'upstream/release-56' into 0415-sync-release-56 2024-04-15 08:09:03 +02:00
zmstone b76b6fbe63 feat(variform): initialize client_attrs with variform
Moved regular expression extraction as a variform function.
2024-04-14 10:13:24 +02:00
zmstone da5b01aa46 refactor(client_attr): allow more than one initial extraction 2024-04-13 01:00:25 +02:00
zmstone 2577224bc7 fix(swagger): do not generate dummy descriptions 2024-04-13 01:00:25 +02:00
Zaiming (Stone) Shi 06b07460e8
Merge pull request #12866 from zmstone/0411-refactor-resove-TODOs
refactor: resolve some old TODOs
2024-04-13 00:10:47 +02:00
Kjell Winblad b4198185bc fix(http connector): redact sensitive information from headers 2024-04-12 15:27:03 +02:00
Kjell Winblad 6777f04780 fix: iolist_to_binrary -> unicode:characters_to_binary
It is not always safe to use iolist_to_binrary on the output of an
io_lib:format call with the ~ts placeholder.
2024-04-12 15:21:25 +02:00
Ilya Averyanov 500d4fedda fix(rebalance): fix start order of rebalance applications 2024-04-12 15:58:57 +03:00
Kjell Winblad ed5409fb6a docs(trace): add emqx_ctl documentation for the new ruleid trace 2024-04-12 11:54:14 +02:00
Kjell Winblad f444c6fc32 fix: pass stop_action_after_render=true in trace meta data
Even if there is no trace we still need to pass
stop_action_after_render=true in the trace meta data so that the action
will be stopped.
2024-04-12 11:38:01 +02:00
Kjell Winblad aa950f97fa test: fix tests with missing application 2024-04-12 11:23:19 +02:00
Kjell Winblad 9998940aa2 fix(trace): several improvements thanks to comments from @zmstone 2024-04-11 16:48:43 +02:00
Ivan Dyachkov b27fc0da26 test(emqx_machine): ensure node is down before testing open ports 2024-04-11 15:24:41 +02:00
zmstone dfd13b4ab5 test: fix dashboard schema json test cases 2024-04-11 15:12:01 +02:00
zmstone afc87ddc9e refactor: do not generate i18n msgid in dashboard schema json
dashboard has its own mind
now i18n is only used to generate docs:
1. runtime swagger spec at /api-docs
2. build-time config schema
both result in the resolved i18n text, but not msgid
2024-04-11 14:09:22 +02:00
Ivan Dyachkov 3b7cade671 chore: 5.6.1-beta.1 2024-04-11 13:49:40 +02:00
zmstone eac25194e5 refactor: resolve some old TODOs
move dashboard schema generation code to the right module
2024-04-11 13:41:34 +02:00
Zaiming (Stone) Shi e7a4210943
Merge pull request #12858 from zmstone/0410-fix-variform-number-handling
fix(variform): allow numbers to be numbers
2024-04-11 11:24:55 +02:00
zmstone 834bddadad test: delete flaky test for now 2024-04-11 09:39:17 +02:00
zmstone e6330dddec fix(variform): allow numbers to be numbers 2024-04-10 11:57:45 +02:00
Zaiming (Stone) Shi 2fea651d1d
Merge pull request #12851 from zmstone/0327-feat-add-emqx_variform
emqx_variform for string substitution and transform
2024-04-10 10:18:40 +02:00
JianBo He 91ffc95f29
Merge pull request #12855 from JimMoen/fix-share-queue-format
fix(mgmt): $queue shared topics format in mgmt topics api
2024-04-10 10:52:48 +08:00
JianBo He 5f8986209e
Merge pull request #12842 from lafirest/fix/iotdb
fix(iotdb): correctly handle undefined value of bool type
2024-04-10 09:49:16 +08:00
Serge Tupchii fae9005f87 test(emqx_mgmt_cli): test that replicants do not join a left core node 2024-04-09 19:06:13 +03:00
Serge Tupchii 55179ccfed chore: update ekka to 0.19.3
Included updates:
  - https://github.com/emqx/mria/pull/178
2024-04-09 19:04:38 +03:00
Kjell Winblad 31142df5cf fix: default value of stop_action_after_template_rendering to true
This commit changes the default value for the
stop_action_after_template_rendering option of the apply rule HTTP API
endpoint so that it is true instead of false.
2024-04-09 16:20:37 +02:00
Kjell Winblad 7bcd553786 test: move test case with rule_engine dep from emqx to emqx_rule_engine 2024-04-09 14:08:46 +02:00
Kjell Winblad 3898950017 test: fix test by adding dependency 2024-04-09 11:38:35 +02:00
Kjell Winblad 958748cf7f test: fix inter test suite problem 2024-04-09 11:10:39 +02:00
JimMoen 03a9c46ca7
fix(sys_topic): format shared topics 2024-04-09 17:06:36 +08:00
JimMoen 47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api 2024-04-09 17:06:32 +08:00
ieQu1 8bbfa9ca8b
Merge pull request #12839 from keynslug/feat/EMQX-12110/rebalancing-impl
feat(dsrepl): implement membership changes and rebalancing
2024-04-09 10:46:31 +02:00
zmstone 53b78086ed chore: fix xref checks 2024-04-09 09:34:05 +02:00
zhongwencool 8381132919 chore: bump emqx_conf to 0.1.36 2024-04-09 14:44:11 +08:00
zhongwencool a1495689c0 fix: clean self node's cluster commit when leave cluster 2024-04-09 14:13:25 +08:00
JimMoen 6cb00cc8c7
Merge pull request #12844 from JimMoen/EMQX-12012/cpu-use-idle-two-decimal-places
fix: cpu usage and idle use two decimal places
2024-04-09 10:11:15 +08:00
Andrew Mayorov d12e907209
fix(dsrepl): correctly handle ra membership change command results
Before this change, results similar to `{error, {no_more_servers_to_try,
[{error, nodedown}, {error, not_member}]}}` were considered retryable
failures, which is incorrect.
2024-04-08 22:44:34 +02:00
Andrew Mayorov 3223797ae5
fix(dsrepl): attempt leadership transfer before server removal
This should make it much less likely to hit weird edge cases that lead
to duplicate Raft log entries because of client retries upon receiving
`shutdown` from the leader being removed.
2024-04-08 22:43:58 +02:00
Andrew Mayorov 1e95bd4da6
test(dsrepl): test unresponsive nodes removal / node restarts 2024-04-08 21:27:56 +02:00
zmstone 41677eb785 refactor: make elvis happy 2024-04-08 21:25:58 +02:00
zmstone bf12efac6d fix(variform): add basic tests 2024-04-08 21:08:43 +02:00
Andrew Mayorov 7a836317ac
fix(dsrepl): trigger unfinished shard transition upon startup
Also provide a trivial API to trigger them by hand.
2024-04-08 16:12:42 +02:00
Andrew Mayorov 75bb7f5cdc
fix(dsrepl): retry only `{add, Site}` crashed membership transitions
To minimize the potential negative impact of removal transitions that
crash for some unknown and unusual reasons.
2024-04-08 16:04:33 +02:00
Kjell Winblad 9628a00a82 docs(emqx_rule_api apply rule): fix doc strings 2024-04-08 15:34:29 +02:00
Thales Macedo Garitezi ba96edb061 fix(clients api): use alternative base64 function for OTP 25
Fixes https://github.com/emqx/emqx/pull/12798#discussion_r1555524603
2024-04-08 10:23:10 -03:00