Commit Graph

11317 Commits

Author SHA1 Message Date
Andrew Mayorov 5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
Andrew Mayorov be793e4735
fix(dsrepl): reassign timestamp at the time of submission
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
Andrew Mayorov 146f082fdc
feat(dsrepl): implement raft-based replication
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
Ivan Dyachkov 923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
sync release-56
2024-03-19 19:09:41 +01:00
Ivan Dyachkov f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
feat: implement message validation
2024-03-19 11:12:28 -03:00
Thales Macedo Garitezi c478e375f6 refactor: change failure log configuration schema 2024-03-19 10:04:18 -03:00
JianBo He c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
Thales Macedo Garitezi 2eac54bf54 chore: rm `move` API
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
SergeTupchiy 92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
JianBo He 24c04e715d
Merge pull request #12715 from zhongwencool/fix-source-replace-error
fix: cant replace source conf
2024-03-19 19:32:47 +08:00
zhongwencool 111cbbbe3c fix: don't crash when ban ip error 2024-03-19 19:10:36 +08:00
zhongwencool d7c01a4fe0 fix: cant replace source conf 2024-03-19 18:53:51 +08:00
zhongwencool 9a44c6712a fix: otel metrics create view crash 2024-03-19 12:20:49 +08:00
Thales Macedo Garitezi e8a8416e44 feat(message_validation): impose restrictions on validation name 2024-03-18 15:57:39 -03:00
Thales Macedo Garitezi 74c03377f2 feat(message_validation_api): implement `reorder` API 2024-03-18 14:30:52 -03:00
Thales Macedo Garitezi bcb7fe96d5 refactor: remove ambiguity from api
In preparation for the "reorder" API
2024-03-18 13:20:05 -03:00
Thales Macedo Garitezi 96ffc9b174 refactor: export `schema_check` fn from schema registry 2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi cde87bce66 chore: add missing traces 2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi f84a996671 feat: implement message validation
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
Serge Tupchii 8f8b023429 refactor: simplify match spec holder construction 2024-03-18 18:00:26 +02:00
Serge Tupchii b3d44125f6 feat: support multiple clientid / username Qs params in "/clients" API
Additionally, add an option to specify which Client info fields to return in the response.
2024-03-18 18:00:26 +02:00
JianBo He 2a611e4508 chore: fix the error api desc 2024-03-18 18:27:28 +08:00
JianBo He 485d8dbbdb feat: add the `get /source_types` endpoint to return all supported source 2024-03-18 18:11:06 +08:00
JimMoen 889eb48368
fix(shared-sub): update stats when unsub shared topic filter 2024-03-16 20:43:05 +08:00
JimMoen 59e280dc54
test: cluster consistented metrics and label for nodes running/stopped
- `emqx_cluster_sessions_count`
- `emqx_cluster_sessions_max`
- `emqx_subscriptions_shared_count`
- `emqx_subscriptions_shared_max`
- `emqx_cluster_nodes_running`
- `emqx_cluster_nodes_stopped`
2024-03-16 20:43:04 +08:00
JimMoen 55a8488a4f
test: fields in `/monitor_current`
- `retained_msg_count`
- `shared_subscriptions`
2024-03-16 20:43:03 +08:00
JimMoen 02b3292025
fix: cluster nodes running/stopped with node name as label 2024-03-16 20:39:34 +08:00
JimMoen 0e25eb287b
fix(prom): cluster session count/max is cluster consistented 2024-03-16 20:39:33 +08:00
JimMoen 62f8f889e1
refactor: func rename to fix typo 2024-03-16 20:39:32 +08:00
JimMoen e56bee618b
fix: `shared_subscriptions` is cluster consistented 2024-03-16 20:39:31 +08:00
Ivan Dyachkov 3f93780d8c
Merge pull request #12717 from id/0315-prep-5.6.0-rc.1
chore: 5.6.0-rc.1
2024-03-15 21:28:31 +01:00
Ivan Dyachkov e28992d586 chore: 5.6.0-rc.1 2024-03-15 17:08:00 +01:00
Ivan Dyachkov ea76dac760 chore: fix xref 2024-03-15 17:08:00 +01:00
Kjell Winblad 1fecd01405
Merge pull request #12708 from kjellwinblad/kjell/dynamo_bridge/update_error/EMQX-11984
fix: DynamoDB connector status check takes too long
2024-03-15 16:20:19 +01:00
ieQu1 c22735b3f5
Merge pull request #12707 from ieQu1/dev/store-ds-session-ip
Store peer name in the durable session metadata.
2024-03-15 16:15:29 +01:00
ieQu1 a93f747afa fix(sessds): Return peername of the disconnected client in the REST 2024-03-15 13:48:53 +01:00
ieQu1 b6cc9177a6 feat(sessds): Store peername in the persistent session state 2024-03-15 13:48:53 +01:00
SergeTupchiy e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client
Refactor emqx mgmt call client
2024-03-15 10:01:49 +02:00
Thales Macedo Garitezi 11ae04d810 test(ds): rm unused var warning 2024-03-14 17:16:24 -03:00
zmstone d8bdb3c9aa fix(api/clients): drop expired sessions from durable storage 2024-03-14 20:57:53 +01:00
Kjell Winblad 4e9f2c8f5d fix: DynamoDB connector status check takes too long
The DynamoDB connector status checks takes very long when the server is
unavailable which makes the resource manager blocked for a long time.
This causes calls to update the Bridge config with the Bridge V1 API fail
due to a timeout when it calls the resource manager to remove the
channel.

A better fix would be to change the resource manager so that the status
check cannot block it for a long time. However, this is more complicated
so it needs to be done in a later commit. A new ticket has been created
for this task https://emqx.atlassian.net/browse/EMQX-12015 .

Fixes:
https://emqx.atlassian.net/browse/EMQX-11984
2024-03-14 17:21:55 +01:00
Thales Macedo Garitezi 2ebc8dcc55 fix(ds): use `infinity` timeout when storing batches 2024-03-14 10:17:18 -03:00
Thales Macedo Garitezi e340ab7f1b
Merge pull request #12696 from thalesmg/fix-default-enable-r56-20240313
fix(bridges): fix default value for `enable` when attempting operations
2024-03-14 09:50:40 -03:00
ieQu1 69a9431471
Merge pull request #12665 from ieQu1/dev/sessds-hot-conf-disable
fix(sessds): Prevent hot update of session_persistence.enable config
2024-03-14 13:20:52 +01:00
Thales Macedo Garitezi d11949ac51
Merge pull request #12694 from thalesmg/fix-kconsu-tm-validation-r56-20240313
fix(kafka_consumer): validate topic mapping in v2 schema
2024-03-14 09:19:41 -03:00
Kjell Winblad 93903eb04b
Merge pull request #12683 from kjellwinblad/kjell/kinesis2/fix/parameter_restrictions/EMQX-11983
fix(Amazon Kinesis Action): batch size restriction
2024-03-14 10:55:44 +01:00
Zaiming (Stone) Shi cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node
fix: do not crash on replicant node
2024-03-14 09:42:27 +01:00
lafirest afb7075f82
Merge pull request #12695 from lafirest/fix/opents
perf(opents): Improve the message processing efficiency of opentsdb
2024-03-14 09:32:42 +08:00
JianBo He 45fe0787ca
Merge pull request #12689 from lafirest/fix/ld_log
fix(ldap): lower the log level of LDAB to `debug`
2024-03-14 09:21:44 +08:00
lafirest 4f97743652
Merge pull request #12687 from lafirest/fix/iotdb
fix(iotdb): Fix function_clause when the convert value is null
2024-03-14 08:52:18 +08:00
lafirest 35fb6ee656
Merge pull request #12688 from lafirest/fix/tdengine
fix(tdengine): enhanced health check result, make it more sense
2024-03-14 08:50:04 +08:00
Serge Tupchii 685f29d3f2 chore: upgrade hocon to 0.42.1
hocon 0.42.1 allows to use "b" or "B" (byte) unit in bytesize fields.
2024-03-13 19:30:28 +02:00
Thales Macedo Garitezi 4ac6d0716a
Merge pull request #12698 from thalesmg/test-flaky-conf-r56-20240313
test(conf): fix flaky config sync testcases
2024-03-13 13:47:23 -03:00
zmstone ccd973d13e fix: do not crash on replicant node 2024-03-13 16:42:39 +01:00
Kjell Winblad 9d21372ff0
Merge pull request #12678 from kjellwinblad/kjell/dynamo_bridge/fix/error_log/EMQX-11934
fix: return error reason from dynamo connector status check
2024-03-13 15:27:08 +01:00
Andrew Mayorov 92088a5905 test(conf): fix flaky config sync testcases
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-13 11:26:49 -03:00
Thales Macedo Garitezi b587ebac0c fix(bridges): fix default value for `enable` when attempting operations
Fixes https://emqx.atlassian.net/browse/EMQX-11999

Although the frontend doesn't send `enable` when creating a bridge/action/connector, the
default value in the schema is `true`, so when attempting to fetch it from the raw config
we should adhere to that default.
2024-03-13 11:25:22 -03:00
firest a58ee801b2 fix(tdengine): enhanced health check result, make it more sense 2024-03-13 21:30:30 +08:00
firest b156e55430 perf(opents): Improve the message processing efficiency of opentsdb 2024-03-13 21:06:52 +08:00
Thales Macedo Garitezi a852695950 fix(kafka_consumer): validate topic mapping in v2 schema
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
Serge Tupchii e514c4a2ac fix(emqx_mgmt_api_clients): check that max_payload_bytes is higher than 0 2024-03-13 14:00:30 +02:00
firest e9febf231f fix(ldap): lower the log level of LDAB to `debug` 2024-03-13 19:20:40 +08:00
Kjell Winblad 7908e2d591
Merge pull request #12682 from kjellwinblad/kjell/hstream_bridge/fix/parameter_restrictions/EMQX-11939
fix(HStreamDB bridge/action): restrict configuration parameters
2024-03-13 11:40:53 +01:00
Kjell Winblad 9d999920be
Merge pull request #12681 from kjellwinblad/kjell/rocketmq_conn/remove_secret_from_logs/EMQX-11987
fix: redact secrets from RocketMQ debug log
2024-03-13 11:39:47 +01:00
JimMoen d723a5edf7
Merge pull request #12670 from JimMoen/feat-shared-sub-count-monitor-current
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
2024-03-13 17:09:30 +08:00
Serge Tupchii 65be76aa06 refactor(emqx_mgmt): add call_client timeout and improve RPC error handling 2024-03-13 10:56:02 +02:00
firest 4b3c6f8330 fix(iotdb): Fix function_clause when the convert value is null 2024-03-13 11:28:07 +08:00
Serge Tupchii 8be02327b2 refactor(emqx_mgmt): avoid call_client RPC to all runing nodes if global cm_registry is enabled 2024-03-12 20:58:53 +02:00
Kjell Winblad 4cd434dbe3 docs(kinesis action): add description for action resource_opts 2024-03-12 18:38:39 +01:00
Thales Macedo Garitezi 1edf284fed test: trigger hocon validators when checking schema 2024-03-12 14:32:16 -03:00
Kjell Winblad c90f4f5794 fix(DynamoDB connector): fix error in status check when no workers 2024-03-12 17:22:22 +01:00
Kjell Winblad 7e73b79b75 fix(Amazon Kinesis Action): batch size restriction
Make sure that the Amazon Kinesis action has the same batch size
restriction as the Amazon Kinesis bridge.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11983
2024-03-12 12:52:54 +01:00
Kjell Winblad da87abd9c8 fix(HStreamDB bridge/action): restrict configuration parameters
This commit restricts a few HStreamDB bridge/action parameters from
being any integer to being integers greater than 0. Lower values than 1
for these parameters resulted in runtime errors.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11939
2024-03-12 11:01:15 +01:00
Kjell Winblad fc4603c078 fix: redact secrets from RocketMQ debug log
Fixes:
https://emqx.atlassian.net/browse/EMQX-11987
2024-03-12 10:32:25 +01:00
Kjell Winblad 91514f3ace refactor: clean up over complicated code
This commit cleans up overly complicated code and handles
the case when the worker pool is empty.

Thank you @thalesmg for suggesting this change.

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-12 10:22:29 +01:00
Kjell Winblad 1d1cdc1009 fix: return error reason from dynamo connector status check
Fixes:
https://emqx.atlassian.net/browse/EMQX-11934
2024-03-12 10:22:08 +01:00
JimMoen 0edeff4786
test: fix flaky 2024-03-12 15:21:17 +08:00
JimMoen bc9084db79
test: cpu utilization value 2024-03-12 14:14:46 +08:00
JimMoen 0297c7b83b
refactor: conditional enclosed in parentheses 2024-03-12 14:14:45 +08:00
JimMoen 207f38c42a
fix(vm): cpu usage/idle handled by single worker 2024-03-12 14:14:44 +08:00
JianBo He f24a76e770
Merge pull request #12668 from emqx/leap-year-bug-sql-fun
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-12 10:48:31 +08:00
Andrew Mayorov 552968e5fe
test(conf): fix flaky config sync testcases
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-11 13:03:13 +01:00
Andrew Mayorov d725206bcb
Merge pull request #12624 from keynslug/fix/EMQX-11901/ds-error-class
feat(sessds): handle recoverable errors during replay
2024-03-11 12:41:45 +01:00
SergeTupchiy b673ad3e5c
Merge pull request #12673 from zmstone/0308-fix-base64-compatibility-on-otp25
fix(mgmt): avoid using base64:decode/2 and encode/2
2024-03-11 13:23:37 +02:00
JimMoen a82d4e1f7a
fix: `retained_msg_count` and `shared_subscriptions` api schema 2024-03-11 17:44:11 +08:00
JimMoen e7574a70fb
feat(api): field `shared_subscriptions` in endpoint `/monitor_current` 2024-03-11 17:44:10 +08:00
Serge Tupchii 980a4c3a4d fix(emqx_mgmt_api): use emqx_utils:bin_to_hexstr/2 2024-03-11 10:40:19 +02:00
Serge Tupchii f376aa8072 refactor(emqx_mgmt_api_clients): remove unnecessary function 2024-03-11 10:39:09 +02:00
Zaiming (Stone) Shi 68682c4231
Merge pull request #12671 from kjellwinblad/kjell/rule_engine/fix/unescape_rules_from_dashboard/EMQX-11847
feat(rule engine SQL): add an `unescape` function
2024-03-08 17:57:31 +01:00
zmstone 2cc1c563df fix(mgmt): avoid using base64:decode/2 and encode/2
they are not available in otp 25
2024-03-08 16:46:54 +01:00
Thales Macedo Garitezi d92fd3e42d
Merge pull request #12667 from thalesmg/ds-delete-m-20240307
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
2024-03-08 10:57:51 -03:00
Thales Macedo Garitezi 79d7821222
Merge pull request #12655 from thalesmg/fix-kconsu-status-r56-20240306
fix(kafka_consumer): check client connectivity
2024-03-08 10:53:44 -03:00
Thales Macedo Garitezi 6af01b916e feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-08 09:56:46 -03:00
Thales Macedo Garitezi 5dfd864c1a
Merge pull request #12666 from thalesmg/test-flaky-machine-m-20240307
test(machine): attempt to stabilize flaky test
2024-03-08 08:57:35 -03:00
Kjell Winblad 5a6f96212d feat(rule engine SQL): add an `unescape` function
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:

- Standard C escape sequences:
  - `\n` for newline (LF)
  - `\t` for horizontal tab (HT)
  - `\r` for carriage return (CR)
  - `\b` for backspace (BS)
  - `\f` for formfeed (FF)
  - `\v` for vertical tab (VT)
  - `\'` for single quote (')
  - `\"` for double quote (")
  - `\\` for backslash (\)
  - `\?` for question mark (?)
  - `\a` for alert (bell, BEL)

- Hexadecimal escape codes:
  - `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
    a-f), allowing for the encoding of arbitrary utf32 characters.

If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.

Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
Kjell Winblad 060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943
fix(rule_engine): support non-byte sized input to bin2hexstr
2024-03-08 09:41:03 +01:00
Kjell Winblad 6054499607
Merge pull request #12657 from kjellwinblad/kjell/rule_engine/fix/fun_calls_as_array_items/EMQX-11953
fix(rule SQL): allow expressions as array items
2024-03-08 09:39:46 +01:00
Shawn 29111a2192 fix: dialyzer problems 2024-03-08 14:25:19 +08:00
Shawn 163d095dca fix: port the changes for date_to_unix_ts SQL fun from 4.4 2024-03-08 10:48:08 +08:00