yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,425 Commits 393 Branches 684 Tags 74 MiB
28f6eb0200
Commit Graph

11944 Commits

Author SHA1 Message Date
Kjell Winblad
7ad354f412 feat: add stop after render and after render trace to clickhouse action 2024-04-25 11:51:46 +02:00
Kjell Winblad
32c27f1711 feat: add stop after render and after render trace to kafka action 2024-04-25 11:51:46 +02:00
Kjell Winblad
b02ed4e6ec feat: add stop after render and after render trace to pgsql action 2024-04-25 11:51:46 +02:00
Kjell Winblad
1f676ce035 feat: add stop after render and after render trace to mqtt action 2024-04-25 11:51:46 +02:00
Kjell Winblad
5ca90ccced fix: improve structure of log trace entries for HTTP action 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12025
 2024-04-25 11:51:46 +02:00
zmstone
a67773e973 Merge remote-tracking branch 'origin/master' into release-57 2024-04-25 10:56:52 +02:00
zhongwencool
2cbf4dc789
Merge pull request #12923 from zhongwencool/authn-mnesia 
chore: provided more specific error for wrong import method
 2024-04-25 10:28:09 +08:00
zhongwencool
2a2da3766e
chore: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2024-04-25 09:16:53 +08:00
zmstone
f2fa35efdf Merge remote-tracking branch 'origin/master' into 0424-merge-latest-master-to-release-57 2024-04-24 22:02:56 +02:00
Serge Tupchii
02a0faa1d6 refactor: avoid evaluating Data more than once in SLOG_THROTTE macro 2024-04-24 16:26:34 +03:00
zmstone
b66d7a688e Merge remote-tracking branch 'origin/release-57' into release-57 2024-04-24 13:09:06 +02:00
zhongwencool
4f12d45436 chore: provided more specific error for wrong import method 2024-04-24 15:37:35 +08:00
zhongwencool
712008bc1d
Merge pull request #12909 from zhongwencool/esockd-bump-version 
fix: bump escokd to 5.11.2 to handle udp_error/closed message
 2024-04-24 09:15:39 +08:00
ieQu1
5804b70c1f
Merge pull request #12918 from ieQu1/dev/ds-readme1 
docs(ds): Update README with CLI and REST API endpoints
 2024-04-23 17:02:01 +02:00
ieQu1
e4c3283c9c docs(ds): Update README with CLI and REST API endpoints 2024-04-23 16:28:35 +02:00
Kjell Winblad
e1eed30b5d
Merge pull request #12912 from kjellwinblad/kjell/rule_trigger_time/EMQX-12025 
feat(rule tracing): add rule trigger time meta data field
 2024-04-23 15:00:05 +02:00
lafirest
0d1c13661f
Merge pull request #12895 from lafirest/fix/dyndb 
fix(dynamo): Added missing keys for DynamoDB
 2024-04-22 23:03:00 +08:00
lafirest
a81ca359fc
Merge pull request #12899 from lafirest/fix/rocmq 
feat(rocketmq): add support for namespace and key dispatch strategy
 2024-04-22 21:20:44 +08:00
Kjell Winblad
e9d498dde2 feat(rule tracing): add rule trigger time meta data field 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12025
 2024-04-22 13:51:48 +02:00
zhongwencool
6049b4966f fix: bump escokd to 5.11.2 to handle udp_error/closed message 2024-04-22 16:41:09 +08:00
ieQu1
f9eda1883f
Merge pull request #12903 from ieQu1/dev/ds-egress-flush-condition 
fix(ds): Fix egress flush condition
 2024-04-22 10:17:22 +02:00
JianBo He
c5c4bb987f
Merge pull request #12902 from HJianBo/stomp 
Pass the Content-type of MQTT message to the Stomp message
 2024-04-22 16:02:03 +08:00
Ivan Dyachkov
b723e87e54 Merge remote-tracking branch 'upstream/release-56' into 0422-sync-release-56 2024-04-22 08:12:05 +02:00
firest
d5cdc07eab feat(rocketmq): add support for namespace and key dispatch strategy 2024-04-22 13:14:11 +08:00
firest
46f339dfab chore: updte change log 2024-04-22 13:12:01 +08:00
firest
a1e85e3c59 fix(dynamo): Added missing keys for DynamoDB 2024-04-22 13:09:34 +08:00
Kjell Winblad
6e99f01ecd
Merge pull request #12863 from kjellwinblad/kjell/new_trace_log_formatter/EMQX-12025 
feat(emqx_trace): add JSON trace log entry formatter
 2024-04-22 06:25:18 +02:00
JianBo He
d85df14b85
Merge pull request #12892 from HJianBo/fix-gateway-related-issues 
fix(ocpp): avoid an error log in handling downstream messages
 2024-04-22 11:25:26 +08:00
JianBo He
69bdcd2f24
Merge pull request #12898 from emqx/iotdb-1.3.0 
feat: support iotdb 1.3.0
 2024-04-22 10:56:06 +08:00
ieQu1
4c76a2574d
fix(ds): Fix egress flush condition 2024-04-21 21:51:31 +02:00
Zaiming (Stone) Shi
307cd79be2
Merge pull request #12897 from zmstone/0418-support-dynamic-license 
feat(license): add business-critical customer type
 2024-04-20 09:03:58 +02:00
JianBo He
5520e54147 chore: add tests 2024-04-20 08:59:36 +08:00
JianBo He
aaf7953968 chore: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2024-04-20 08:36:59 +08:00
JianBo He
95f3e49edb fix(stomp): pass the Content-Type from the MQTT message 2024-04-20 08:33:55 +08:00
Andrew Mayorov
43f8346c00
fix(dssnap): ensure idempotent write of empty chunks 2024-04-19 18:52:33 +02:00
ieQu1
bac5100635
Merge pull request #12874 from ieQu1/dev/EMQX-12030-subscriptions-api 
API for durable subscriptions
 2024-04-19 17:11:53 +02:00
Kjell Winblad
2890bc2619 fix(tracing): remove internal extra field from the trace config 
This commit removes the internal extra field from the trace config
structure exposed to the user via the HTTP API.
 2024-04-19 17:05:34 +02:00
Kjell Winblad
9e46c18443 fix(emqx_trace_SUITE:t_base_create_delete): broken test case 2024-04-19 15:41:50 +02:00
zmstone
ec83fbe3dc feat(license): add business critical customer type 2024-04-19 14:19:39 +02:00
ieQu1
ede7246882
fix(sessds): Avoid double-enriching transient messages 2024-04-19 13:39:04 +02:00
Ilia Averianov
b7a4536f47
Merge pull request #12871 from savonarola/0412-fix-rebalance-app-dependencies 
fix(rebalance): fix start order of rebalance applications
 2024-04-19 13:27:41 +03:00
Zaiming (Stone) Shi
a8b1224225
Merge pull request #12893 from zmstone/0416-add-is-template-flag-to-dashboard-schema 
0416 add `is_template` flag to dashboard schema
 2024-04-19 10:41:43 +02:00
Kjell Winblad
c163956d08 fix(trace formatter): remove record field to enable rolling upgrade 2024-04-19 09:52:20 +02:00
zhongwencool
b163a87386 feat: support batch_size on iotdb 2024-04-19 14:52:56 +08:00
zhongwencool
c0521fd250 feat: support iotdb 1.3.0 2024-04-19 08:54:18 +08:00
ieQu1
f1e6565ddd
refactor(sessds): Move all subscription logic to the subs module 2024-04-19 01:20:29 +02:00
ieQu1
d12966db5b
test: Avoid dumping raw snabbkaffe traces to the console 2024-04-19 01:20:29 +02:00
ieQu1
180130d684
feat(sessds): List persistent subscriptions in the REST API 2024-04-19 01:20:29 +02:00
Kjell Winblad
09b414f368 test: add necessary application to test suites 2024-04-18 17:39:52 +02:00
Kjell Winblad
285bfa9367 fix: improve rendering of action_template_rendered trace 2024-04-18 17:00:47 +02:00
Kjell Winblad
7be18730e8 test(emqx_rule_engine_api_rule_apply_SUITE): remove unnecessary code 2024-04-18 15:16:18 +02:00
Kjell Winblad
f5b0439724 fix(emqx_rule_engine_api_rule_apply_SUITE): flaky test case 2024-04-18 14:28:28 +02:00
Kjell Winblad
8f1486f6d3 fix: clean up trace messages to make it easier to interpret 
This commit removes some redundant trace messages and renames some to
make it easier to interpret what is happening for the user.
 2024-04-18 14:28:01 +02:00
zmstone
5b38d592f0 feat(http): add is_template as HTTP headers field property 
is_template was designed to be type property.
however for HTTP headers, it's a map() type,
instead of creating a new type for it, it's easier to just
add it as a field property.
 2024-04-18 13:16:29 +02:00
zmstone
ede4eeae9f fix(http_bridge): path is template field 2024-04-18 13:04:56 +02:00
zmstone
ca56e7e8d7 fix(kafka): headers are template fields 2024-04-18 13:04:36 +02:00
Kjell Winblad
3232ab5ea3 fix(rule tracing): unset trace meta data in try-after-end 
We wrap the reset of the process trace meta data in the after clause of
a try-after-end expression to be sure we never get any lingering
incorrect meta data.
 2024-04-18 11:28:50 +02:00
Kjell Winblad
aa388adba9 fix(json trace format): format client_ids and rule_ids as lists 2024-04-18 11:27:39 +02:00
Kjell Winblad
10957e7d79 fix: change name of the default trace log format from plain to text 2024-04-18 10:42:51 +02:00
Kjell Winblad
ef9f8a8fdf fix: unreachable clause found by dialyzer 2024-04-18 10:28:37 +02:00
Kjell Winblad
6c0ee8bb01 test(emqx_trace_SUITE): fix failure due to new field 2024-04-18 10:28:37 +02:00
Kjell Winblad
968dc2ccda fix(json trace format): use best_effort_json instead of new encoder 2024-04-18 10:28:37 +02:00
Kjell Winblad
aa39835242 feat(emqx_cli): add parameter for trace format (plain or json) 2024-04-18 10:28:29 +02:00
Kjell Winblad
6fd7a06c5d fix: problems reported by dialyzer 2024-04-18 10:20:18 +02:00
Kjell Winblad
a6558740e8 feat(emqx_trace): add JSON trace log entry formatter 
This commit makes it possible to select the JSON trace log entry
formatter when crating a trace pattern. This will make it easier for the
dashboard and automatic tests to parse the log entries.

Fixes: EMQX-12025 (partly)
 2024-04-18 10:20:11 +02:00
zmstone
ab763fe665 test: fix test case flakyness 2024-04-18 09:32:05 +02:00
Kjell Winblad
2a2fadfbff
Merge pull request #12827 from kjellwinblad/kjell/emqx_rule_tracing/EMQX-11966 
Add rule ID tracing support
 2024-04-18 09:06:31 +02:00
JianBo He
d7ebecddb4 fix(ocpp): return correct current_connections number of listenrs http api 2024-04-18 14:54:12 +08:00
Ivan Dyachkov
e93bd314bc chore: 5.6.1 2024-04-18 08:03:56 +02:00
JianBo He
75632bb2cd chore(gw): update listener examples 2024-04-18 13:58:31 +08:00
Kjell Winblad
b16b9d8fcc refactor: made code more readable thanks to suggestion from @zmstone 2024-04-18 06:51:30 +02:00
zmstone
55941000c0 test: make test case more stable, less flaky 2024-04-17 20:42:03 +02:00
zmstone
6ab2b004ed fix(resource_manager): update cache after channel add 2024-04-17 20:34:33 +02:00
zmstone
c96ae8dd23 fix: return 503 if bridge bpapi call timeout 2024-04-17 20:34:33 +02:00
zmstone
5a4bfff9e5 refactor: add template_str type 2024-04-17 20:34:33 +02:00
zmstone
5c014f4c29 test: fix test cases 2024-04-17 20:34:33 +02:00
Kjell Winblad
95891db29a test: clean up created resources in the end of test case 2024-04-17 18:43:19 +02:00
Kjell Winblad
6cf29ba688 fix: clean up traces to make them easier to parse and understand 2024-04-17 18:20:17 +02:00
Kjell Winblad
cf56050759 feat: avoid mixing request with and without the stop_after_render flag 
Previously a batch of requests that was sent to a connector could
contain both requests with the stop_after_rendering flag and requests
without this flag. When this happened a warning message was generated and
the stop_after_render flags for the batch would be ignored. This commit
fixes so that a mixed batch is never created so there is no longer any
need for a warning message or ignoring flags.
 2024-04-17 16:21:21 +02:00
zmstone
d49e98bc4b test: fix dashboard schema validation 2024-04-17 14:09:48 +02:00
zmstone
f9f14f9758 refactor(emqx_conf): raise exception at higher level for more context 2024-04-17 13:10:55 +02:00
zmstone
51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
JianBo He
2468243dfd chore: update changes 2024-04-17 18:52:48 +08:00
ieQu1
124c5047d0
feat(sessds): Add API for getting session data from the cold storage 2024-04-17 12:31:54 +02:00
ieQu1
38a2e8add9
fix(sessds): Return the number of subscriptions for offline sessions 2024-04-17 12:31:54 +02:00
ieQu1
e439a2e0f2
fix(sessds): Save protocol name and version in the session metadata 2024-04-17 12:31:54 +02:00
JianBo He
18196ec19c fix(ocpp): avoid an error log in handling dnsteeam messages 2024-04-17 17:48:19 +08:00
JianBo He
c61c3157c6
Merge pull request #12888 from HJianBo/data-backup-for-license 
fix(license): license file is not taking effect after importing backup files
 2024-04-17 16:31:16 +08:00
JimMoen
1dfd9115cd
Merge pull request #12880 from JimMoen/EMQX-12147/influx-write-syntax-tag-type 
fix(influx): literal number values in tag set
 2024-04-17 10:23:57 +08:00
ieQu1
197a4c30be
fix(sessds): Strip unneccessary data from the durable session state 2024-04-17 01:21:52 +02:00
ieQu1
93bb840365
docs(ds): Update README 2024-04-17 01:21:52 +02:00
ieQu1
87ffaf89e5
refactor(sessds_state): Use macros for map keys 2024-04-17 01:21:52 +02:00
ieQu1
113a990482
feat(sessds): Support max subscriptions 2024-04-17 01:21:51 +02:00
ieQu1
6c897c26ae
fix(sessds): Commit session on unsubscribe 2024-04-17 01:21:51 +02:00
ieQu1
6c83bbe10b
feat(mgmt): Filter subscriptions by durability 2024-04-17 01:21:51 +02:00
ieQu1
b30ddc206e
fix(sessds): Immutable subscriptions 
This commit fixes two issues:
- Behavior of overlapping subscriptions has been aligned with the
in-memory session.

- Fixed handling of replays when subscription changes (either by
client or EMQX configuration)
 2024-04-17 01:21:51 +02:00
ieQu1
3e0c649e8e
feat(sessds): Store awaiting rel 2024-04-17 01:21:51 +02:00
zmstone
df458b98d7 refactor(dashboard_schema): no need to translate labels 
the trans_label implementation was ugly, it compares an anonymous
function to check if the label should be translated.

since we have stopped generating i18n message ids for dashboard
schema, this entire function is now stale, so this function
is deleted.
 2024-04-16 17:54:34 +02:00
zmstone
2bd72aab44 chore: bump dashboard schema version to 0.2.0 2024-04-16 16:58:45 +02:00
Zaiming (Stone) Shi
4d38a8fd44
Merge pull request #12872 from zmstone/0412-support-variform-for-client_attrs 
0412 use variform for client_attrs
 2024-04-16 13:29:56 +02:00
JianBo He
6b8111c066 fix(license): license file is not taking effect after importing backup files 2024-04-16 19:07:17 +08:00
ieQu1
c645cfa5d6
fix(sessds): Graceful handling of shared subscription error 2024-04-16 12:37:01 +02:00
Ivan Dyachkov
f3bb28c6af chore: 5.6.1-rc.2 2024-04-16 09:47:32 +02:00
Andrew Mayorov
5d7b2e2ce6
fix(dsrepl): attempt leadership transfer on terminate 
In addition to on removal. The reasoning is basically the same: try to
avoid situations when log entries are replicated (or will be considered
replicated when the new leader is elected) but the leader terminates
before replying to the client.

To be clear: this is a stupid solution. Something much more robust is
needed.
 2024-04-15 22:05:24 +02:00
Ivan Dyachkov
a0ffe5e7ae chore: 5.6.1-rc.1 2024-04-15 22:05:05 +02:00
Kjell Winblad
f481871792 fix(rocketmq action): we need one producer group per channel and topic 
We need one producer group per channel and topic because we can have
several topics per channel due to templating.
 2024-04-15 20:52:08 +02:00
Kjell Winblad
1fe92bddd0 fix(rocketmq action): make sure that topic template is respected 2024-04-15 20:37:23 +02:00
Kjell Winblad
7f9a311988 fix(rocketmq action): all actions used only one topic 
This commit makes sure that a set of rocketmq workers are started for
each topic. Before this commit all actions for a rocketmq connector used
the same workers which all were bound to a single topic so all messages
got delivered to that topic regardless of the action configuration.

We should have automatic tests to check that the messages actually go to
different topics but this needs to go into another PR since we have to
deliver the fix fast and the rocketmq library does not support reading
from topics.

Fixes:
https://emqx.atlassian.net/browse/EEC-1006
 2024-04-15 20:08:56 +02:00
zmstone
f80d078de3 feat(variform): Add more functions 2024-04-15 16:56:52 +02:00
Andrew Mayorov
89f42f1171
fix(dsrepl): make placeholder shard process permanent under supervisor 2024-04-15 16:43:52 +02:00
Andrew Mayorov
c4d1360b96
fix(dsrepl): trigger election for new ra servers unconditionallly 
Otherwise we might end up in a situation when there's no member online
yet at the time of the election trigger, and the election will never
happen.
 2024-04-15 16:42:29 +02:00
JimMoen
084e920c6e
test(influx): literal values or variable in tag set 2024-04-15 17:47:53 +08:00
JimMoen
8ed397d4fa
fix(influx): literal number values in tag set 2024-04-15 17:47:52 +08:00
Ivan Dyachkov
3ef160eed2 Merge remote-tracking branch 'upstream/release-56' into 0415-sync-release-56 2024-04-15 08:09:03 +02:00
zmstone
b76b6fbe63 feat(variform): initialize client_attrs with variform 
Moved regular expression extraction as a variform function.
 2024-04-14 10:13:24 +02:00
zmstone
da5b01aa46 refactor(client_attr): allow more than one initial extraction 2024-04-13 01:00:25 +02:00
zmstone
2577224bc7 fix(swagger): do not generate dummy descriptions 2024-04-13 01:00:25 +02:00
Zaiming (Stone) Shi
06b07460e8
Merge pull request #12866 from zmstone/0411-refactor-resove-TODOs 
refactor: resolve some old TODOs
 2024-04-13 00:10:47 +02:00
Kjell Winblad
b4198185bc fix(http connector): redact sensitive information from headers 2024-04-12 15:27:03 +02:00
Kjell Winblad
6777f04780 fix: iolist_to_binrary -> unicode:characters_to_binary 
It is not always safe to use iolist_to_binrary on the output of an
io_lib:format call with the ~ts placeholder.
 2024-04-12 15:21:25 +02:00
Ilya Averyanov
500d4fedda fix(rebalance): fix start order of rebalance applications 2024-04-12 15:58:57 +03:00
Kjell Winblad
ed5409fb6a docs(trace): add emqx_ctl documentation for the new ruleid trace 2024-04-12 11:54:14 +02:00
Kjell Winblad
f444c6fc32 fix: pass stop_action_after_render=true in trace meta data 
Even if there is no trace we still need to pass
stop_action_after_render=true in the trace meta data so that the action
will be stopped.
 2024-04-12 11:38:01 +02:00
Kjell Winblad
aa950f97fa test: fix tests with missing application 2024-04-12 11:23:19 +02:00
Kjell Winblad
9998940aa2 fix(trace): several improvements thanks to comments from @zmstone 2024-04-11 16:48:43 +02:00
Ivan Dyachkov
b27fc0da26 test(emqx_machine): ensure node is down before testing open ports 2024-04-11 15:24:41 +02:00
zmstone
dfd13b4ab5 test: fix dashboard schema json test cases 2024-04-11 15:12:01 +02:00
zmstone
afc87ddc9e refactor: do not generate i18n msgid in dashboard schema json 
dashboard has its own mind
now i18n is only used to generate docs:
1. runtime swagger spec at /api-docs
2. build-time config schema
both result in the resolved i18n text, but not msgid
 2024-04-11 14:09:22 +02:00
Ivan Dyachkov
3b7cade671 chore: 5.6.1-beta.1 2024-04-11 13:49:40 +02:00
zmstone
eac25194e5 refactor: resolve some old TODOs 
move dashboard schema generation code to the right module
 2024-04-11 13:41:34 +02:00
Zaiming (Stone) Shi
e7a4210943
Merge pull request #12858 from zmstone/0410-fix-variform-number-handling 
fix(variform): allow numbers to be numbers
 2024-04-11 11:24:55 +02:00
zmstone
834bddadad test: delete flaky test for now 2024-04-11 09:39:17 +02:00
zmstone
e6330dddec fix(variform): allow numbers to be numbers 2024-04-10 11:57:45 +02:00
Zaiming (Stone) Shi
2fea651d1d
Merge pull request #12851 from zmstone/0327-feat-add-emqx_variform 
emqx_variform for string substitution and transform
 2024-04-10 10:18:40 +02:00
JianBo He
91ffc95f29
Merge pull request #12855 from JimMoen/fix-share-queue-format 
fix(mgmt): $queue shared topics format in mgmt topics api
 2024-04-10 10:52:48 +08:00
JianBo He
5f8986209e
Merge pull request #12842 from lafirest/fix/iotdb 
fix(iotdb): correctly handle undefined value of bool type
 2024-04-10 09:49:16 +08:00
Serge Tupchii
fae9005f87 test(emqx_mgmt_cli): test that replicants do not join a left core node 2024-04-09 19:06:13 +03:00
Serge Tupchii
55179ccfed chore: update ekka to 0.19.3 
Included updates:
  - https://github.com/emqx/mria/pull/178
 2024-04-09 19:04:38 +03:00
Kjell Winblad
31142df5cf fix: default value of stop_action_after_template_rendering to true 
This commit changes the default value for the
stop_action_after_template_rendering option of the apply rule HTTP API
endpoint so that it is true instead of false.
 2024-04-09 16:20:37 +02:00
Kjell Winblad
7bcd553786 test: move test case with rule_engine dep from emqx to emqx_rule_engine 2024-04-09 14:08:46 +02:00
Kjell Winblad
3898950017 test: fix test by adding dependency 2024-04-09 11:38:35 +02:00
Kjell Winblad
958748cf7f test: fix inter test suite problem 2024-04-09 11:10:39 +02:00
JimMoen
03a9c46ca7
fix(sys_topic): format shared topics 2024-04-09 17:06:36 +08:00
JimMoen
47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api 2024-04-09 17:06:32 +08:00
ieQu1
8bbfa9ca8b
Merge pull request #12839 from keynslug/feat/EMQX-12110/rebalancing-impl 
feat(dsrepl): implement membership changes and rebalancing
 2024-04-09 10:46:31 +02:00
zmstone
53b78086ed chore: fix xref checks 2024-04-09 09:34:05 +02:00
zhongwencool
8381132919 chore: bump emqx_conf to 0.1.36 2024-04-09 14:44:11 +08:00
zhongwencool
a1495689c0 fix: clean self node's cluster commit when leave cluster 2024-04-09 14:13:25 +08:00
JimMoen
6cb00cc8c7
Merge pull request #12844 from JimMoen/EMQX-12012/cpu-use-idle-two-decimal-places 
fix: cpu usage and idle use two decimal places
 2024-04-09 10:11:15 +08:00
Andrew Mayorov
d12e907209
fix(dsrepl): correctly handle ra membership change command results 
Before this change, results similar to `{error, {no_more_servers_to_try,
[{error, nodedown}, {error, not_member}]}}` were considered retryable
failures, which is incorrect.
 2024-04-08 22:44:34 +02:00
Andrew Mayorov
3223797ae5
fix(dsrepl): attempt leadership transfer before server removal 
This should make it much less likely to hit weird edge cases that lead
to duplicate Raft log entries because of client retries upon receiving
`shutdown` from the leader being removed.
 2024-04-08 22:43:58 +02:00
Andrew Mayorov
1e95bd4da6
test(dsrepl): test unresponsive nodes removal / node restarts 2024-04-08 21:27:56 +02:00
zmstone
41677eb785 refactor: make elvis happy 2024-04-08 21:25:58 +02:00
zmstone
bf12efac6d fix(variform): add basic tests 2024-04-08 21:08:43 +02:00
Andrew Mayorov
7a836317ac
fix(dsrepl): trigger unfinished shard transition upon startup 
Also provide a trivial API to trigger them by hand.
 2024-04-08 16:12:42 +02:00
Andrew Mayorov
75bb7f5cdc
fix(dsrepl): retry only {add, Site} crashed membership transitions 
To minimize the potential negative impact of removal transitions that
crash for some unknown and unusual reasons.
 2024-04-08 16:04:33 +02:00
Kjell Winblad
9628a00a82 docs(emqx_rule_api apply rule): fix doc strings 2024-04-08 15:34:29 +02:00
Thales Macedo Garitezi
ba96edb061 fix(clients api): use alternative base64 function for OTP 25 
Fixes https://github.com/emqx/emqx/pull/12798#discussion_r1555524603
 2024-04-08 10:23:10 -03:00
Kjell Winblad
600526a0e4 test(emqx_bridge_http_SUITE): test case after name change 2024-04-08 14:54:33 +02:00
Kjell Winblad
02ee873094 docs(emqx_rule_api_schema): fix type spec 2024-04-08 13:42:15 +02:00
Andrew Mayorov
4c0cc079c2
fix(dsrepl): apply unnecessary rebalancing transitions cleanly 2024-04-08 13:25:45 +02:00
Andrew Mayorov
dcde30c38a
test(dsrepl): add two more testcases for rebalancing 2024-04-08 13:22:31 +02:00
Kjell Winblad
b57725f996 test(resource_SUITE): do test case fixes needed due to rule tracing work 2024-04-08 12:41:35 +02:00
Kjell Winblad
79440064fe style: fix problems reported by elvis 2024-04-08 11:03:55 +02:00
JianBo He
e11c4a9c83
Merge pull request #12826 from emqx/import-source-bridges 
fix: source bridges missing after restore the backup files
 2024-04-08 16:02:15 +08:00
Zaiming (Stone) Shi
3e87d4bf9f
Merge pull request #12772 from killme2008/feature/upgrade-greptimedb-ingester-ci 
feat: update greptimedb client lib and ci version
 2024-04-08 09:02:03 +02:00
JimMoen
282cbb18be
fix: cpu usage and idle use two decimal places 
- prometheus
- opentelemetry
 2024-04-08 14:14:09 +08:00
Andrew Mayorov
2ace9bb893
chore(dsrepl): sprinkle few comments and typespecs for exports 2024-04-07 22:51:56 +02:00
Andrew Mayorov
ecaad348a7
chore(dsrepl): update few outdated comments / TODOs 2024-04-07 22:51:56 +02:00
Andrew Mayorov
6293efb995
fix(dsrepl): retry crashed membership transitions 2024-04-07 22:51:56 +02:00
Andrew Mayorov
826ce5806d
fix(dsrepl): ensure that new member UID matches server's UID 
Before that change, UIDs supplied in the `ra:add_member/3` were not
the same as those servers were using. This haven't caused any issues
for some reason, but it's better to ensure that UIDs are the same.
 2024-04-07 22:31:24 +02:00
Shawn
e89dc32c90 ci: run emqx_management both with ee and ce profile 2024-04-07 18:33:52 +08:00
firest
f40e47365c fix(iotdb): correctly handle undefined value of bool type 2024-04-07 17:26:15 +08:00
Shawn
1c81c79a2c chore: add testcase for importing retained msgs and sources 2024-04-07 17:24:26 +08:00
Shawn
9d1a69aaa9 fix: cannot import retained messages 2024-04-07 17:03:14 +08:00
Kjell Winblad
5479932190 feat(apply rule test): make option to stop action after render work 
This commit makes the apply rule HTTP API option to stop an action work
for the HTTP action, and adds infrastructure that makes it easy to add
this functionality to other actions.
 2024-04-06 17:21:12 +02:00
Kjell Winblad
ef705c2285 feat: add apply rule API, clientid/ruleid tracing for rule and connector 
This commit adds:

* Support for forwarding the rule id and client id to the connector so
  that events such as template rendered successfully can be traced.
* HTTP API for for applying/activating a rule with the given context
 2024-04-06 17:20:47 +02:00
Thales Macedo Garitezi
04ba2aaf8a
Merge pull request #12830 from thalesmg/async-channel-hc-m-20240404 
feat(resource): non-blocking channel health checks
 2024-04-05 14:24:09 -03:00
Andrew Mayorov
556ffc78c9
feat(dsrepl): implement membership changes and rebalancing 2024-04-05 18:57:28 +02:00
Andrew Mayorov
d6058b7f51
feat(dsrepl): allow to subscribe to DB metadata changes 
Currently, only shard metadata changes are announced to the
subscribers.
 2024-04-05 17:40:55 +02:00
Andrew Mayorov
a07295d3bc
fix(ds): address shards in the supervisor properly 2024-04-05 17:40:38 +02:00
ieQu1
3f7b14c861
Merge pull request #12833 from ieQu1/dev/ds-cluster-api 
feat(ds): Add REST API for durable storage
 2024-04-05 17:33:20 +02:00
ieQu1
2504b8126b
feat(ds): Pass mgmt_ds REST API calls to the application 2024-04-05 15:22:06 +02:00
ieQu1
46261440cb
feat(ds): Add a CLI for managing DB replicas 2024-04-05 15:22:06 +02:00
ieQu1
a62db08676
feat(ds): Add REST API for durable storage 2024-04-05 15:22:06 +02:00
ieQu1
d09787d1a6
fix(ds): Fix return types in replication_layer_meta 2024-04-05 15:22:06 +02:00
Ivan Dyachkov
be47fe49ad chore: bump ecql version to 0.7.0 
PR: https://github.com/emqx/ecql/pull/13
No functional changes, just switch gen_fsm to gen_statem.
 2024-04-05 13:31:33 +02:00
Andrew Mayorov
70396e9766
Merge pull request #12825 from keynslug/feat/EMQX-12110/repl-meta-api 
feat(dsrepl): add APIs to manage DB replication sites
 2024-04-04 22:32:03 +02:00
Andrew Mayorov
df6c5b35fe
feat(dsrepl): add more primitive operations to modify DB sites 2024-04-04 21:22:49 +02:00
Andrew Mayorov
bb8ffee18c
feat(dsrepl): add API to get current DB replication sites 2024-04-04 21:22:02 +02:00
Andrew Mayorov
ad52f7838e
feat(dsrepl): add APIs to manage DB replication sites 2024-04-04 21:22:01 +02:00
Thales Macedo Garitezi
60cad74286 feat(resource): non-blocking channel health checks 
Fixes https://emqx.atlassian.net/browse/EMQX-12015

Continuation of https://github.com/emqx/emqx/pull/12812
 2024-04-04 16:13:30 -03:00
Thales Macedo Garitezi
8d58b40f33
Merge pull request #12831 from thalesmg/ds-checkpoint-clean-m-20240404 
feat(ds): clear all checkpoints when (re)starting storage layer
 2024-04-04 15:41:50 -03:00
Thales Macedo Garitezi
217b35bce5
Merge pull request #12798 from thalesmg/ds-client-api-v2-m-20240327 
feat(client mgmt api): add cursor-based list API
 2024-04-04 15:10:49 -03:00
Thales Macedo Garitezi
c57c36adb2 feat(ds): clear all checkpoints when (re)starting storage layer 
Fixes https://emqx.atlassian.net/browse/EMQX-12143
 2024-04-04 14:05:52 -03:00
Kjell Winblad
59a442cdb5 feat(rule trace): add support for ruleid as a trace type 2024-04-04 14:55:32 +02:00
Thales Macedo Garitezi
069cd4fbb4
Merge pull request #12812 from thalesmg/async-res-manager-health-check-m-20240328 
feat(resource manager): perform non-blocking health checks
 2024-04-04 09:07:02 -03:00
zmstone
0e79b543cf refactor: move variform to emqx_utils 2024-04-04 11:10:56 +02:00
ieQu1
f37ed3a40a fix(ds): Limit the number of retries in egress to 0 2024-04-03 16:38:49 +02:00
Shawn
319ec50c0d fix: source bridges missing after restore the backup files 2024-04-03 18:26:51 +08:00
ieQu1
2bbfada7af
fix(ds): Make async batches truly async 2024-04-03 11:57:47 +02:00
ieQu1
92ca90c0ca
fix(ds): Improve egress logging 2024-04-03 11:57:47 +02:00
ieQu1
ae5935e7f7
test(ds): Attempt to stabilize metrics_worker tests in CI 2024-04-02 19:14:10 +02:00
ieQu1
4382971443
fix(ds): Preserve errors in the egress 2024-04-02 16:47:43 +02:00
ieQu1
94ca7ad0f8
feat(ds): Report counters for LTS storage layout 2024-04-02 16:47:43 +02:00
ieQu1
f14c253dea
fix(prometheus): Don't add DS metrics when feature is disabled 2024-04-02 16:47:43 +02:00
ieQu1
b379f331de
fix(sessds): Handle errors when storing messages 2024-04-02 16:47:41 +02:00
ieQu1
f41e538526
feat(sessds): Observe next time 2024-04-02 16:45:52 +02:00
ieQu1
b9ad241658
feat(sessds): Add metrics for the number of persisted messages 2024-04-02 16:45:52 +02:00
ieQu1
75b092bf0e
fix(ds): Actually retry sending batch 2024-04-02 16:45:49 +02:00
ieQu1
0de255cac8
feat(ds): Report egress flush time 2024-04-02 16:25:04 +02:00
ieQu1
044f3d4ef5
fix(ds): Don't reverse entries in the atomic batch 2024-04-02 16:25:04 +02:00
ieQu1
606f2a88cd
feat(ds): Add egress metrics 2024-04-02 16:25:04 +02:00
ieQu1
c9de336234
feat(ds): Add metrics worker to the builtin db supervision tree 2024-04-02 16:25:04 +02:00
ieQu1
d8204021dc
refactor(metrics): Move metrics worker to emqx_utils application 2024-04-02 16:25:04 +02:00
Thales Macedo Garitezi
2097e854fc feat(client mgmt api): add cursor-based list API 
Fixes https://emqx.atlassian.net/browse/EMQX-12028
 2024-04-02 10:55:28 -03:00
Andrew Mayorov
778e897f1f
chore(dsrepl): describe snapshot ownership and few shortcomings 2024-04-02 13:48:51 +02:00
Andrew Mayorov
c666c65c6a
test(ds): factor out storage iteration into helper module 2024-04-02 13:48:51 +02:00
Andrew Mayorov
7cebf598a8
chore(dsrepl): simplify snapshot transfer code a bit 
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-04-02 13:48:51 +02:00
Andrew Mayorov
e029b8f996
test(dsrepl): wait for whole cluster readiness 
To minimize the chance of flaky tests due to the shards not being
completely online.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-04-02 13:48:50 +02:00
Andrew Mayorov
e8b06a6a9f
chore(dsrepl): mark few more BPAPI targets as obsolete 2024-04-02 13:48:50 +02:00
Andrew Mayorov
d31cd0c728
feat(ds): ensure LTS state ids are deterministic 2024-04-02 13:48:50 +02:00
Andrew Mayorov
2cd357a5bd
fix(ds): ensure store batch is idempotent wrt generations 2024-04-02 13:48:50 +02:00
Andrew Mayorov
77a022bd93
feat(dsrepl): transfer storage snapshot during ra snapshot recovery 2024-04-02 13:48:49 +02:00
Andrew Mayorov
b8b9b7739b
chore(ds): slightly simplify working with storage generations 2024-04-02 13:48:08 +02:00
Andrew Mayorov
2d074df209
Merge pull request #12797 from keynslug/fix/dsrepl-error-handling 
fix(dsrepl): handle RPC errors gracefully when storage is down
 2024-04-02 13:40:31 +02:00
JimMoen
5759ba5162
chore: bump app version 2024-04-02 17:09:22 +08:00
JimMoen
50bceee9ab
fix(stats): 'subscribers.count' contains shared-subscriber 2024-04-02 16:56:40 +08:00
JimMoen
0f4b148294
refactor: uniform shared_sub table macros 2024-04-02 16:56:39 +08:00
JimMoen
1a4cfc2a2d
fix(api_schema): removed metrics schema in api spec 
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622).
 2024-04-02 16:56:36 +08:00
Thales Macedo Garitezi
bade09b56e feat(resource manager): perform non-blocking resource health checks 
Fixes https://emqx.atlassian.net/browse/EMQX-12015

This introduces only _resource_ non-blocking health checks.  _Channel_ non-blocking health
checks may be introduced later.
 2024-04-01 14:46:15 -03:00
Serge Tupchii
c62410ff75 refactor: remove already bound variable 2024-04-01 17:03:50 +03:00
Serge Tupchii
ceb04ba06d fix(emqx_mgmt): do not attempt to get a stacktrace of a remote client connection process 2024-04-01 16:42:12 +03:00
Serge Tupchii
42af1f9d63 fix: handle internal timeout errors in client Mqueue/Inflight APIs 2024-03-29 23:03:35 +02:00
Serge Tupchii
f5a820cb10 fix(emqx_mgmt): catch OOM shutdown exits properly when calling a client conn process 
The exit reason is expected to include gen_server `Location`:
  `{{shutdown, OOMInfo}, Location}`.
 2024-03-29 13:09:08 +02:00
zmstone
bfca3ebc71 feat(variform): support array syntax '[' and ']' 2024-03-28 19:34:57 +01:00
zmstone
5f26e4ed5e feat(variform): implement variform engine 2024-03-28 19:34:57 +01:00
SergeTupchiy
2e528d1dd8
Merge pull request #12802 from SergeTupchiy/EMQX-11826-prevent-left-node-from-rejoining-5.6.1 
prevent a left node from rejoining the same cluster
 2024-03-28 19:49:18 +02:00
zmstone
ad95473aae refactor: move string functions to emqx_variform 2024-03-28 18:03:37 +01:00
Serge Tupchii
3eda182e9a fix: prevent a node from discovering and re-joining the same cluster after it has (manually) left it. 2024-03-28 18:09:27 +02:00
zmstone
9bf65a415b feat(variform): add a variable transformer 2024-03-28 16:11:26 +01:00
Andrew Mayorov
35c43eb8a0
feat(sessds): handle recoverable errors in stream scheduler 2024-03-28 15:17:01 +01:00
Andrew Mayorov
fa66a640c3
fix(dsrepl): handle RPC errors gracefully when storage is down 2024-03-28 15:17:01 +01:00
SergeTupchiy
63c017a72f
Merge pull request #12804 from SergeTupchiy/EMQX-12058-improve-force-shutdown-error-reason-5.6.1 
chore: rename `message_queue_too_long` error reason to `mailbox_overflow` (5.6.1 port)
 2024-03-28 16:14:51 +02:00
Thales Macedo Garitezi
8fb4ef9fe3 test: fix flaky test 2024-03-28 10:53:44 -03:00
SergeTupchiy
93c87fcb25
Merge pull request #12803 from SergeTupchiy/EMQX-11808-remove-uploaded-invalid-backups-5.6.1 
fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid (5.6.1 port)
 2024-03-28 15:37:10 +02:00
Ivan Dyachkov
9b4d960888 chore: prepare for 5.7.x releases 2024-03-28 13:52:48 +01:00
Thales Macedo Garitezi
04bf763890 fix(kafka-based bridges): avoid trying to get raw config for replayq dir 
Fixes https://emqx.atlassian.net/browse/EMQX-12049
 2024-03-28 09:13:34 -03:00
ieQu1
b1855f95c1
fix(bpapi): Add exceptions for experimental features 2024-03-28 12:07:45 +01:00
ieQu1
8c6d8bdd12
fix(bpapi): Add exceptions for experimental features 2024-03-28 12:03:41 +01:00
Serge Tupchii
906a77d167 chore: rename message_queue_too_long error reason to mailbox_overflow 
`mailbox_overflow` is consistent with the corresponding config parameter:
 'force_shutdown.max_mailbox_size'
 2024-03-28 12:56:00 +02:00
Serge Tupchii
1d1f595e6f fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid 2024-03-28 12:54:51 +02:00
ieQu1
02b72d79fc
fix(bpapi): Fix comparison of releases during compatibility check 2024-03-28 11:21:30 +01:00
Ivan Dyachkov
db9efb9317 chore: bump apps versions 2024-03-28 10:19:09 +01:00
Ivan Dyachkov
f4446ec680 Merge remote-tracking branch 'upstream/release-56' into 0328-sync-release-56 2024-03-28 09:59:54 +01:00
Ivan Dyachkov
ea917d6b2b
Merge pull request #12795 from id/0327-cut-5.6.0 
chore: 5.6.0
 2024-03-28 09:30:08 +01:00
Ilya Averyanov
11bc1aeea1 fix: update hstreamdb driver 2024-03-28 10:30:48 +03:00
JimMoen
cf29f97440
revert: "test: comment out hstreamdb:stop_client" 
This reverts commits:
3536b32ad2b126c27a1b0bd91e615069230931d7..
d63f10928d
 2024-03-28 14:32:18 +08:00
JimMoen
34b5cd7bfd
chore: reuse hstream_erl 0.5.18+v0.18.1 
- change ezstd dep vsn for bulid
 2024-03-28 14:31:10 +08:00
JianBo He
d63f10928d test: comment out hstreamdb:stop_client 2024-03-28 13:25:01 +08:00
JianBo He
3536b32ad2 ci: print the failed stacktrace 2024-03-28 11:17:26 +08:00
JianBo He
d7917d05c6 chore: upgrade hstreamdb to 0.5.22+v0.18.1 2024-03-28 10:11:53 +08:00
ieQu1
a279a399db
ci: Override ezstd dependency 2024-03-27 20:54:46 +01:00
ieQu1
4b3c29ded8
fix(bpapi): Add 5.5 dump 2024-03-27 17:39:38 +01:00
Zaiming (Stone) Shi
beb9152d50
Merge pull request #12750 from zmstone/0316-introduce-client-attrs 
0316 introduce client_attrs
 2024-03-27 16:19:47 +01:00
ieQu1
5ed20e15cd
fix(bpapi): Add 5.5 bpapi file 2024-03-27 13:45:00 +01:00
Thales Macedo Garitezi
df0c67fe50
Merge pull request #12787 from thalesmg/mv-api-examples-m-20240326 
docs(message validation): add swagger API examples
 2024-03-27 09:06:58 -03:00
Ivan Dyachkov
33d9619271 chore: 5.6.0 2024-03-27 11:17:13 +01:00
Serge Tupchii
a3b730ff10 fix(emqx_session_mem): keep msg inflight insert timestamp in Inflight after retry 2024-03-27 11:31:51 +02:00
JianBo He
4f9daa0d4a chore: bump version to 5.6.0-rc.6 2024-03-27 14:04:27 +08:00
JianBo He
d82b6d216d chore: remove the duplicated time field 
Fixes https://emqx.atlassian.net/browse/EMQX-12104
 2024-03-27 12:09:21 +08:00
Ivan Dyachkov
f127bf53b3 chore: 5.6.0-rc.5 2024-03-26 17:52:57 +01:00
ieQu1
291cf689e2
Merge pull request #12786 from ieQu1/dev/check-core-emqx-vsn 
fix(mria): Strictly check compatibility of the replicant with core
 2024-03-26 17:49:37 +01:00
ieQu1
ba43da1b8e
fix(mria): Strictly check compatibility of the replicant with core 2024-03-26 16:32:01 +01:00
Thales Macedo Garitezi
da4a04c411
Merge pull request #12777 from thalesmg/test-flaky-conf-copy-m-20240325 
test: attempt to fix flaky test
 2024-03-26 11:52:12 -03:00
zmstone
78c5f8177c test: fix test case 2024-03-26 14:51:21 +01:00
Thales Macedo Garitezi
10a43db742 docs(message validation): add swagger API examples 
Fixes https://emqx.atlassian.net/browse/EMQX-12102
 2024-03-26 09:39:05 -03:00
zmstone
067f018337 test: fix test cases 2024-03-26 13:36:12 +01:00
zmstone
b1c9bb63d7 chore: delete dead code 
hocon schema validator never validates 'undefined'
 2024-03-26 13:20:48 +01:00
zmstone
c42e980442 feat(log): configurable time format 
now logs can be configured to use 'epoch' or 'rfc3339' time format
 2024-03-26 13:09:32 +01:00
Thales Macedo Garitezi
cc37030265
Merge pull request #12770 from thalesmg/mv-metrics-m-20240321 
feat(message validation): implement metrics
 2024-03-26 08:59:05 -03:00
zhongwencool
56a40d61ae fix: bpapi undef abort on old node 2024-03-26 13:06:56 +08:00
Ivan Dyachkov
2ae973821f chore: 5.6.0-rc.4 2024-03-25 20:00:10 +01:00
Zaiming (Stone) Shi
b06d3932af
Merge pull request #12768 from keynslug/fix/EMQX-12068/routing-schema-conflict 
fix(routing): add heuristic for routing schema in empty cluster
 2024-03-25 19:50:51 +01:00
Thales Macedo Garitezi
6259f95aab test: attempt to fix flaky test 
```
   expected: {ok,<<...>>}
       got: {error,enoent}
      line: 187
   comment: #{node => 'emqx_conf_app_SUITE9@127.0.0.1'}
```
 2024-03-25 14:36:22 -03:00
Thales Macedo Garitezi
0f426e6e77 feat(ds): make durable sessions handle will messages 
Fixes https://emqx.atlassian.net/browse/EMQX-10431
 2024-03-25 14:18:25 -03:00
Thales Macedo Garitezi
8c1a1d21a7 feat(session): pass will message down to session when creating/opening 2024-03-25 14:18:23 -03:00
Dennis Zhuang
64eeeb7494 chore: verify status code 2024-03-25 10:05:39 -07:00
ieQu1
8b963d5960
Merge pull request #12739 from ieQu1/dev/zone-durability 
feat(sessds): Add zone overrides for session durability settings
 2024-03-25 17:32:53 +01:00
zmstone
b8b0b809b4 chore: add a warning log if conflict is detected 2024-03-25 17:07:56 +01:00
zmstone
9411d6078e docs: remove unnecessary restart instruction 2024-03-25 16:50:51 +01:00
zmstone
d38545010a fix: case_clause 2024-03-25 16:43:24 +01:00
Ivan Dyachkov
c93145cb97 fix: function clause 2024-03-25 16:40:53 +01:00
zmstone
3aff9eb2a4 fix(route schema): allow boot if all peer nodes agree on one version 2024-03-25 16:33:27 +01:00
Ivan Dyachkov
fe3cc25855 chore: update instructions for routing schema conflict 2024-03-25 15:22:43 +01:00
Thales Macedo Garitezi
59ff43ec87 feat(message validation): implement metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-12071
 2024-03-25 10:55:01 -03:00
Kjell Winblad
94c6e5bb38
Merge pull request #12727 from kjellwinblad/kjell/emqx_connector_info/EMQX-11427 
refactor: add emqx_connector_info behavior
 2024-03-25 14:31:40 +01:00
Andrew Mayorov
849fe0c2c8
feat(routing): add schema conflict resolution procedure 
In the log message printed when a schema conflict in cluster routing
is detected.
 2024-03-25 13:43:46 +01:00
Thales Macedo Garitezi
f393d75a1e
Merge pull request #12771 from thalesmg/test-another-flaky-monitor-m-20240322 
test: attempt to fix flaky test
 2024-03-25 09:19:20 -03:00
Kjell Winblad
e39fa5eaa9 docs(es_connector_info): add copyright header 2024-03-25 10:44:33 +01:00