Andrew Mayorov
5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
...
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
Andrew Mayorov
be793e4735
fix(dsrepl): reassign timestamp at the time of submission
...
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
Andrew Mayorov
146f082fdc
feat(dsrepl): implement raft-based replication
...
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
Ivan Dyachkov
923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
...
sync release-56
2024-03-19 19:09:41 +01:00
Ivan Dyachkov
f2dc940436
Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56
2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi
a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
...
feat: implement message validation
2024-03-19 11:12:28 -03:00
Thales Macedo Garitezi
c478e375f6
refactor: change failure log configuration schema
2024-03-19 10:04:18 -03:00
JianBo He
c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
...
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
Thales Macedo Garitezi
2eac54bf54
chore: rm `move` API
...
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
SergeTupchiy
92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
...
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
JianBo He
24c04e715d
Merge pull request #12715 from zhongwencool/fix-source-replace-error
...
fix: cant replace source conf
2024-03-19 19:32:47 +08:00
zhongwencool
111cbbbe3c
fix: don't crash when ban ip error
2024-03-19 19:10:36 +08:00
zhongwencool
d7c01a4fe0
fix: cant replace source conf
2024-03-19 18:53:51 +08:00
zhongwencool
9a44c6712a
fix: otel metrics create view crash
2024-03-19 12:20:49 +08:00
Thales Macedo Garitezi
e8a8416e44
feat(message_validation): impose restrictions on validation name
2024-03-18 15:57:39 -03:00
Thales Macedo Garitezi
74c03377f2
feat(message_validation_api): implement `reorder` API
2024-03-18 14:30:52 -03:00
Thales Macedo Garitezi
bcb7fe96d5
refactor: remove ambiguity from api
...
In preparation for the "reorder" API
2024-03-18 13:20:05 -03:00
Thales Macedo Garitezi
96ffc9b174
refactor: export `schema_check` fn from schema registry
2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi
cde87bce66
chore: add missing traces
2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi
f84a996671
feat: implement message validation
...
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
Serge Tupchii
8f8b023429
refactor: simplify match spec holder construction
2024-03-18 18:00:26 +02:00
Serge Tupchii
b3d44125f6
feat: support multiple clientid / username Qs params in "/clients" API
...
Additionally, add an option to specify which Client info fields to return in the response.
2024-03-18 18:00:26 +02:00
JianBo He
2a611e4508
chore: fix the error api desc
2024-03-18 18:27:28 +08:00
JianBo He
485d8dbbdb
feat: add the `get /source_types` endpoint to return all supported source
2024-03-18 18:11:06 +08:00
JimMoen
889eb48368
fix(shared-sub): update stats when unsub shared topic filter
2024-03-16 20:43:05 +08:00
JimMoen
59e280dc54
test: cluster consistented metrics and label for nodes running/stopped
...
- `emqx_cluster_sessions_count`
- `emqx_cluster_sessions_max`
- `emqx_subscriptions_shared_count`
- `emqx_subscriptions_shared_max`
- `emqx_cluster_nodes_running`
- `emqx_cluster_nodes_stopped`
2024-03-16 20:43:04 +08:00
JimMoen
55a8488a4f
test: fields in `/monitor_current`
...
- `retained_msg_count`
- `shared_subscriptions`
2024-03-16 20:43:03 +08:00
JimMoen
02b3292025
fix: cluster nodes running/stopped with node name as label
2024-03-16 20:39:34 +08:00
JimMoen
0e25eb287b
fix(prom): cluster session count/max is cluster consistented
2024-03-16 20:39:33 +08:00
JimMoen
62f8f889e1
refactor: func rename to fix typo
2024-03-16 20:39:32 +08:00
JimMoen
e56bee618b
fix: `shared_subscriptions` is cluster consistented
2024-03-16 20:39:31 +08:00
Ivan Dyachkov
3f93780d8c
Merge pull request #12717 from id/0315-prep-5.6.0-rc.1
...
chore: 5.6.0-rc.1
2024-03-15 21:28:31 +01:00
Ivan Dyachkov
e28992d586
chore: 5.6.0-rc.1
2024-03-15 17:08:00 +01:00
Ivan Dyachkov
ea76dac760
chore: fix xref
2024-03-15 17:08:00 +01:00
Kjell Winblad
1fecd01405
Merge pull request #12708 from kjellwinblad/kjell/dynamo_bridge/update_error/EMQX-11984
...
fix: DynamoDB connector status check takes too long
2024-03-15 16:20:19 +01:00
ieQu1
c22735b3f5
Merge pull request #12707 from ieQu1/dev/store-ds-session-ip
...
Store peer name in the durable session metadata.
2024-03-15 16:15:29 +01:00
ieQu1
a93f747afa
fix(sessds): Return peername of the disconnected client in the REST
2024-03-15 13:48:53 +01:00
ieQu1
b6cc9177a6
feat(sessds): Store peername in the persistent session state
2024-03-15 13:48:53 +01:00
SergeTupchiy
e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client
...
Refactor emqx mgmt call client
2024-03-15 10:01:49 +02:00
Thales Macedo Garitezi
11ae04d810
test(ds): rm unused var warning
2024-03-14 17:16:24 -03:00
zmstone
d8bdb3c9aa
fix(api/clients): drop expired sessions from durable storage
2024-03-14 20:57:53 +01:00
Kjell Winblad
4e9f2c8f5d
fix: DynamoDB connector status check takes too long
...
The DynamoDB connector status checks takes very long when the server is
unavailable which makes the resource manager blocked for a long time.
This causes calls to update the Bridge config with the Bridge V1 API fail
due to a timeout when it calls the resource manager to remove the
channel.
A better fix would be to change the resource manager so that the status
check cannot block it for a long time. However, this is more complicated
so it needs to be done in a later commit. A new ticket has been created
for this task https://emqx.atlassian.net/browse/EMQX-12015 .
Fixes:
https://emqx.atlassian.net/browse/EMQX-11984
2024-03-14 17:21:55 +01:00
Thales Macedo Garitezi
2ebc8dcc55
fix(ds): use `infinity` timeout when storing batches
2024-03-14 10:17:18 -03:00
Thales Macedo Garitezi
e340ab7f1b
Merge pull request #12696 from thalesmg/fix-default-enable-r56-20240313
...
fix(bridges): fix default value for `enable` when attempting operations
2024-03-14 09:50:40 -03:00
ieQu1
69a9431471
Merge pull request #12665 from ieQu1/dev/sessds-hot-conf-disable
...
fix(sessds): Prevent hot update of session_persistence.enable config
2024-03-14 13:20:52 +01:00
Thales Macedo Garitezi
d11949ac51
Merge pull request #12694 from thalesmg/fix-kconsu-tm-validation-r56-20240313
...
fix(kafka_consumer): validate topic mapping in v2 schema
2024-03-14 09:19:41 -03:00
Kjell Winblad
93903eb04b
Merge pull request #12683 from kjellwinblad/kjell/kinesis2/fix/parameter_restrictions/EMQX-11983
...
fix(Amazon Kinesis Action): batch size restriction
2024-03-14 10:55:44 +01:00
Zaiming (Stone) Shi
cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node
...
fix: do not crash on replicant node
2024-03-14 09:42:27 +01:00
lafirest
afb7075f82
Merge pull request #12695 from lafirest/fix/opents
...
perf(opents): Improve the message processing efficiency of opentsdb
2024-03-14 09:32:42 +08:00
JianBo He
45fe0787ca
Merge pull request #12689 from lafirest/fix/ld_log
...
fix(ldap): lower the log level of LDAB to `debug`
2024-03-14 09:21:44 +08:00
lafirest
4f97743652
Merge pull request #12687 from lafirest/fix/iotdb
...
fix(iotdb): Fix function_clause when the convert value is null
2024-03-14 08:52:18 +08:00
lafirest
35fb6ee656
Merge pull request #12688 from lafirest/fix/tdengine
...
fix(tdengine): enhanced health check result, make it more sense
2024-03-14 08:50:04 +08:00
Serge Tupchii
685f29d3f2
chore: upgrade hocon to 0.42.1
...
hocon 0.42.1 allows to use "b" or "B" (byte) unit in bytesize fields.
2024-03-13 19:30:28 +02:00
Thales Macedo Garitezi
4ac6d0716a
Merge pull request #12698 from thalesmg/test-flaky-conf-r56-20240313
...
test(conf): fix flaky config sync testcases
2024-03-13 13:47:23 -03:00
zmstone
ccd973d13e
fix: do not crash on replicant node
2024-03-13 16:42:39 +01:00
Kjell Winblad
9d21372ff0
Merge pull request #12678 from kjellwinblad/kjell/dynamo_bridge/fix/error_log/EMQX-11934
...
fix: return error reason from dynamo connector status check
2024-03-13 15:27:08 +01:00
Andrew Mayorov
92088a5905
test(conf): fix flaky config sync testcases
...
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-13 11:26:49 -03:00
Thales Macedo Garitezi
b587ebac0c
fix(bridges): fix default value for `enable` when attempting operations
...
Fixes https://emqx.atlassian.net/browse/EMQX-11999
Although the frontend doesn't send `enable` when creating a bridge/action/connector, the
default value in the schema is `true`, so when attempting to fetch it from the raw config
we should adhere to that default.
2024-03-13 11:25:22 -03:00
firest
a58ee801b2
fix(tdengine): enhanced health check result, make it more sense
2024-03-13 21:30:30 +08:00
firest
b156e55430
perf(opents): Improve the message processing efficiency of opentsdb
2024-03-13 21:06:52 +08:00
Thales Macedo Garitezi
a852695950
fix(kafka_consumer): validate topic mapping in v2 schema
...
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
Serge Tupchii
e514c4a2ac
fix(emqx_mgmt_api_clients): check that max_payload_bytes is higher than 0
2024-03-13 14:00:30 +02:00
firest
e9febf231f
fix(ldap): lower the log level of LDAB to `debug`
2024-03-13 19:20:40 +08:00
Kjell Winblad
7908e2d591
Merge pull request #12682 from kjellwinblad/kjell/hstream_bridge/fix/parameter_restrictions/EMQX-11939
...
fix(HStreamDB bridge/action): restrict configuration parameters
2024-03-13 11:40:53 +01:00
Kjell Winblad
9d999920be
Merge pull request #12681 from kjellwinblad/kjell/rocketmq_conn/remove_secret_from_logs/EMQX-11987
...
fix: redact secrets from RocketMQ debug log
2024-03-13 11:39:47 +01:00
JimMoen
d723a5edf7
Merge pull request #12670 from JimMoen/feat-shared-sub-count-monitor-current
...
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
2024-03-13 17:09:30 +08:00
Serge Tupchii
65be76aa06
refactor(emqx_mgmt): add call_client timeout and improve RPC error handling
2024-03-13 10:56:02 +02:00
firest
4b3c6f8330
fix(iotdb): Fix function_clause when the convert value is null
2024-03-13 11:28:07 +08:00
Serge Tupchii
8be02327b2
refactor(emqx_mgmt): avoid call_client RPC to all runing nodes if global cm_registry is enabled
2024-03-12 20:58:53 +02:00
Kjell Winblad
4cd434dbe3
docs(kinesis action): add description for action resource_opts
2024-03-12 18:38:39 +01:00
Thales Macedo Garitezi
1edf284fed
test: trigger hocon validators when checking schema
2024-03-12 14:32:16 -03:00
Kjell Winblad
c90f4f5794
fix(DynamoDB connector): fix error in status check when no workers
2024-03-12 17:22:22 +01:00
Kjell Winblad
7e73b79b75
fix(Amazon Kinesis Action): batch size restriction
...
Make sure that the Amazon Kinesis action has the same batch size
restriction as the Amazon Kinesis bridge.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11983
2024-03-12 12:52:54 +01:00
Kjell Winblad
da87abd9c8
fix(HStreamDB bridge/action): restrict configuration parameters
...
This commit restricts a few HStreamDB bridge/action parameters from
being any integer to being integers greater than 0. Lower values than 1
for these parameters resulted in runtime errors.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11939
2024-03-12 11:01:15 +01:00
Kjell Winblad
fc4603c078
fix: redact secrets from RocketMQ debug log
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11987
2024-03-12 10:32:25 +01:00
Kjell Winblad
91514f3ace
refactor: clean up over complicated code
...
This commit cleans up overly complicated code and handles
the case when the worker pool is empty.
Thank you @thalesmg for suggesting this change.
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-12 10:22:29 +01:00
Kjell Winblad
1d1cdc1009
fix: return error reason from dynamo connector status check
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11934
2024-03-12 10:22:08 +01:00
JimMoen
0edeff4786
test: fix flaky
2024-03-12 15:21:17 +08:00
JimMoen
bc9084db79
test: cpu utilization value
2024-03-12 14:14:46 +08:00
JimMoen
0297c7b83b
refactor: conditional enclosed in parentheses
2024-03-12 14:14:45 +08:00
JimMoen
207f38c42a
fix(vm): cpu usage/idle handled by single worker
2024-03-12 14:14:44 +08:00
JianBo He
f24a76e770
Merge pull request #12668 from emqx/leap-year-bug-sql-fun
...
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-12 10:48:31 +08:00
Andrew Mayorov
552968e5fe
test(conf): fix flaky config sync testcases
...
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-11 13:03:13 +01:00
Andrew Mayorov
d725206bcb
Merge pull request #12624 from keynslug/fix/EMQX-11901/ds-error-class
...
feat(sessds): handle recoverable errors during replay
2024-03-11 12:41:45 +01:00
SergeTupchiy
b673ad3e5c
Merge pull request #12673 from zmstone/0308-fix-base64-compatibility-on-otp25
...
fix(mgmt): avoid using base64:decode/2 and encode/2
2024-03-11 13:23:37 +02:00
JimMoen
a82d4e1f7a
fix: `retained_msg_count` and `shared_subscriptions` api schema
2024-03-11 17:44:11 +08:00
JimMoen
e7574a70fb
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
2024-03-11 17:44:10 +08:00
Serge Tupchii
980a4c3a4d
fix(emqx_mgmt_api): use emqx_utils:bin_to_hexstr/2
2024-03-11 10:40:19 +02:00
Serge Tupchii
f376aa8072
refactor(emqx_mgmt_api_clients): remove unnecessary function
2024-03-11 10:39:09 +02:00
Zaiming (Stone) Shi
68682c4231
Merge pull request #12671 from kjellwinblad/kjell/rule_engine/fix/unescape_rules_from_dashboard/EMQX-11847
...
feat(rule engine SQL): add an `unescape` function
2024-03-08 17:57:31 +01:00
zmstone
2cc1c563df
fix(mgmt): avoid using base64:decode/2 and encode/2
...
they are not available in otp 25
2024-03-08 16:46:54 +01:00
Thales Macedo Garitezi
d92fd3e42d
Merge pull request #12667 from thalesmg/ds-delete-m-20240307
...
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
2024-03-08 10:57:51 -03:00
Thales Macedo Garitezi
79d7821222
Merge pull request #12655 from thalesmg/fix-kconsu-status-r56-20240306
...
fix(kafka_consumer): check client connectivity
2024-03-08 10:53:44 -03:00
Thales Macedo Garitezi
6af01b916e
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
...
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-08 09:56:46 -03:00
Thales Macedo Garitezi
5dfd864c1a
Merge pull request #12666 from thalesmg/test-flaky-machine-m-20240307
...
test(machine): attempt to stabilize flaky test
2024-03-08 08:57:35 -03:00
Kjell Winblad
5a6f96212d
feat(rule engine SQL): add an `unescape` function
...
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:
- Standard C escape sequences:
- `\n` for newline (LF)
- `\t` for horizontal tab (HT)
- `\r` for carriage return (CR)
- `\b` for backspace (BS)
- `\f` for formfeed (FF)
- `\v` for vertical tab (VT)
- `\'` for single quote (')
- `\"` for double quote (")
- `\\` for backslash (\)
- `\?` for question mark (?)
- `\a` for alert (bell, BEL)
- Hexadecimal escape codes:
- `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
a-f), allowing for the encoding of arbitrary utf32 characters.
If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.
Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
Kjell Winblad
060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943
...
fix(rule_engine): support non-byte sized input to bin2hexstr
2024-03-08 09:41:03 +01:00
Kjell Winblad
6054499607
Merge pull request #12657 from kjellwinblad/kjell/rule_engine/fix/fun_calls_as_array_items/EMQX-11953
...
fix(rule SQL): allow expressions as array items
2024-03-08 09:39:46 +01:00
Shawn
29111a2192
fix: dialyzer problems
2024-03-08 14:25:19 +08:00
Shawn
163d095dca
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-08 10:48:08 +08:00