yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,804 Commits 393 Branches 684 Tags 74 MiB
2d074df209
Commit Graph

11446 Commits

Author SHA1 Message Date
Kjell Winblad
af3a604354 refactor: add emqx_connector_info behavior 
This commit adds the behavior `emqx_connector_info`. The
`emqx_connector_info` behavior should be implement when creating a new
connector to provide information about the connector (such as connector
schema etc) to the `emqx_connetor` application.

The connector in the `emqx_bridge_dynamo` application has also been
refactored to use the new behavior (as a test to see that the behavior
is working as intended).

Fixes:
https://emqx.atlassian.net/browse/EMQX-11427
 2024-03-18 13:50:23 +01:00
JianBo He
2a611e4508 chore: fix the error api desc 2024-03-18 18:27:28 +08:00
JianBo He
485d8dbbdb feat: add the get /source_types endpoint to return all supported source 2024-03-18 18:11:06 +08:00
JimMoen
889eb48368
fix(shared-sub): update stats when unsub shared topic filter 2024-03-16 20:43:05 +08:00
JimMoen
59e280dc54
test: cluster consistented metrics and label for nodes running/stopped 
- `emqx_cluster_sessions_count`
- `emqx_cluster_sessions_max`
- `emqx_subscriptions_shared_count`
- `emqx_subscriptions_shared_max`
- `emqx_cluster_nodes_running`
- `emqx_cluster_nodes_stopped`
 2024-03-16 20:43:04 +08:00
JimMoen
55a8488a4f
test: fields in /monitor_current 
- `retained_msg_count`
- `shared_subscriptions`
 2024-03-16 20:43:03 +08:00
JimMoen
02b3292025
fix: cluster nodes running/stopped with node name as label 2024-03-16 20:39:34 +08:00
JimMoen
0e25eb287b
fix(prom): cluster session count/max is cluster consistented 2024-03-16 20:39:33 +08:00
JimMoen
62f8f889e1
refactor: func rename to fix typo 2024-03-16 20:39:32 +08:00
JimMoen
e56bee618b
fix: shared_subscriptions is cluster consistented 2024-03-16 20:39:31 +08:00
Ivan Dyachkov
3f93780d8c
Merge pull request #12717 from id/0315-prep-5.6.0-rc.1 
chore: 5.6.0-rc.1
 2024-03-15 21:28:31 +01:00
Ivan Dyachkov
e28992d586 chore: 5.6.0-rc.1 2024-03-15 17:08:00 +01:00
Ivan Dyachkov
ea76dac760 chore: fix xref 2024-03-15 17:08:00 +01:00
Kjell Winblad
1fecd01405
Merge pull request #12708 from kjellwinblad/kjell/dynamo_bridge/update_error/EMQX-11984 
fix: DynamoDB connector status check takes too long
 2024-03-15 16:20:19 +01:00
ieQu1
c22735b3f5
Merge pull request #12707 from ieQu1/dev/store-ds-session-ip 
Store peer name in the durable session metadata.
 2024-03-15 16:15:29 +01:00
ieQu1
a93f747afa fix(sessds): Return peername of the disconnected client in the REST 2024-03-15 13:48:53 +01:00
ieQu1
b6cc9177a6 feat(sessds): Store peername in the persistent session state 2024-03-15 13:48:53 +01:00
SergeTupchiy
e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client 
Refactor emqx mgmt call client
 2024-03-15 10:01:49 +02:00
Thales Macedo Garitezi
11ae04d810 test(ds): rm unused var warning 2024-03-14 17:16:24 -03:00
zmstone
d8bdb3c9aa fix(api/clients): drop expired sessions from durable storage 2024-03-14 20:57:53 +01:00
Kjell Winblad
4e9f2c8f5d fix: DynamoDB connector status check takes too long 
The DynamoDB connector status checks takes very long when the server is
unavailable which makes the resource manager blocked for a long time.
This causes calls to update the Bridge config with the Bridge V1 API fail
due to a timeout when it calls the resource manager to remove the
channel.

A better fix would be to change the resource manager so that the status
check cannot block it for a long time. However, this is more complicated
so it needs to be done in a later commit. A new ticket has been created
for this task https://emqx.atlassian.net/browse/EMQX-12015 .

Fixes:
https://emqx.atlassian.net/browse/EMQX-11984
 2024-03-14 17:21:55 +01:00
Thales Macedo Garitezi
2ebc8dcc55 fix(ds): use infinity timeout when storing batches 2024-03-14 10:17:18 -03:00
Thales Macedo Garitezi
e340ab7f1b
Merge pull request #12696 from thalesmg/fix-default-enable-r56-20240313 
fix(bridges): fix default value for `enable` when attempting operations
 2024-03-14 09:50:40 -03:00
ieQu1
69a9431471
Merge pull request #12665 from ieQu1/dev/sessds-hot-conf-disable 
fix(sessds): Prevent hot update of session_persistence.enable config
 2024-03-14 13:20:52 +01:00
Thales Macedo Garitezi
d11949ac51
Merge pull request #12694 from thalesmg/fix-kconsu-tm-validation-r56-20240313 
fix(kafka_consumer): validate topic mapping in v2 schema
 2024-03-14 09:19:41 -03:00
Kjell Winblad
93903eb04b
Merge pull request #12683 from kjellwinblad/kjell/kinesis2/fix/parameter_restrictions/EMQX-11983 
fix(Amazon Kinesis Action): batch size restriction
 2024-03-14 10:55:44 +01:00
Zaiming (Stone) Shi
cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node 
fix: do not crash on replicant node
 2024-03-14 09:42:27 +01:00
lafirest
afb7075f82
Merge pull request #12695 from lafirest/fix/opents 
perf(opents): Improve the message processing efficiency of opentsdb
 2024-03-14 09:32:42 +08:00
JianBo He
45fe0787ca
Merge pull request #12689 from lafirest/fix/ld_log 
fix(ldap): lower the log level of LDAB to `debug`
 2024-03-14 09:21:44 +08:00
lafirest
4f97743652
Merge pull request #12687 from lafirest/fix/iotdb 
fix(iotdb): Fix function_clause when the convert value is null
 2024-03-14 08:52:18 +08:00
lafirest
35fb6ee656
Merge pull request #12688 from lafirest/fix/tdengine 
fix(tdengine): enhanced health check result, make it more sense
 2024-03-14 08:50:04 +08:00
Serge Tupchii
685f29d3f2 chore: upgrade hocon to 0.42.1 
hocon 0.42.1 allows to use "b" or "B" (byte) unit in bytesize fields.
 2024-03-13 19:30:28 +02:00
Thales Macedo Garitezi
4ac6d0716a
Merge pull request #12698 from thalesmg/test-flaky-conf-r56-20240313 
test(conf): fix flaky config sync testcases
 2024-03-13 13:47:23 -03:00
zmstone
ccd973d13e fix: do not crash on replicant node 2024-03-13 16:42:39 +01:00
Kjell Winblad
9d21372ff0
Merge pull request #12678 from kjellwinblad/kjell/dynamo_bridge/fix/error_log/EMQX-11934 
fix: return error reason from dynamo connector status check
 2024-03-13 15:27:08 +01:00
Andrew Mayorov
92088a5905 test(conf): fix flaky config sync testcases 
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
 2024-03-13 11:26:49 -03:00
Thales Macedo Garitezi
b587ebac0c fix(bridges): fix default value for enable when attempting operations 
Fixes https://emqx.atlassian.net/browse/EMQX-11999

Although the frontend doesn't send `enable` when creating a bridge/action/connector, the
default value in the schema is `true`, so when attempting to fetch it from the raw config
we should adhere to that default.
 2024-03-13 11:25:22 -03:00
firest
a58ee801b2 fix(tdengine): enhanced health check result, make it more sense 2024-03-13 21:30:30 +08:00
firest
b156e55430 perf(opents): Improve the message processing efficiency of opentsdb 2024-03-13 21:06:52 +08:00
Thales Macedo Garitezi
a852695950 fix(kafka_consumer): validate topic mapping in v2 schema 
Fixes https://emqx.atlassian.net/browse/EMQX-12008
 2024-03-13 09:57:46 -03:00
Serge Tupchii
e514c4a2ac fix(emqx_mgmt_api_clients): check that max_payload_bytes is higher than 0 2024-03-13 14:00:30 +02:00
firest
e9febf231f fix(ldap): lower the log level of LDAB to debug 2024-03-13 19:20:40 +08:00
Kjell Winblad
7908e2d591
Merge pull request #12682 from kjellwinblad/kjell/hstream_bridge/fix/parameter_restrictions/EMQX-11939 
fix(HStreamDB bridge/action): restrict configuration parameters
 2024-03-13 11:40:53 +01:00
Kjell Winblad
9d999920be
Merge pull request #12681 from kjellwinblad/kjell/rocketmq_conn/remove_secret_from_logs/EMQX-11987 
fix: redact secrets from RocketMQ debug log
 2024-03-13 11:39:47 +01:00
JimMoen
d723a5edf7
Merge pull request #12670 from JimMoen/feat-shared-sub-count-monitor-current 
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
 2024-03-13 17:09:30 +08:00
Serge Tupchii
65be76aa06 refactor(emqx_mgmt): add call_client timeout and improve RPC error handling 2024-03-13 10:56:02 +02:00
firest
4b3c6f8330 fix(iotdb): Fix function_clause when the convert value is null 2024-03-13 11:28:07 +08:00
Serge Tupchii
8be02327b2 refactor(emqx_mgmt): avoid call_client RPC to all runing nodes if global cm_registry is enabled 2024-03-12 20:58:53 +02:00
Kjell Winblad
4cd434dbe3 docs(kinesis action): add description for action resource_opts 2024-03-12 18:38:39 +01:00
Thales Macedo Garitezi
1edf284fed test: trigger hocon validators when checking schema 2024-03-12 14:32:16 -03:00
Kjell Winblad
c90f4f5794 fix(DynamoDB connector): fix error in status check when no workers 2024-03-12 17:22:22 +01:00
Kjell Winblad
7e73b79b75 fix(Amazon Kinesis Action): batch size restriction 
Make sure that the Amazon Kinesis action has the same batch size
restriction as the Amazon Kinesis bridge.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11983
 2024-03-12 12:52:54 +01:00
Kjell Winblad
da87abd9c8 fix(HStreamDB bridge/action): restrict configuration parameters 
This commit restricts a few HStreamDB bridge/action parameters from
being any integer to being integers greater than 0. Lower values than 1
for these parameters resulted in runtime errors.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11939
 2024-03-12 11:01:15 +01:00
Kjell Winblad
fc4603c078 fix: redact secrets from RocketMQ debug log 
Fixes:
https://emqx.atlassian.net/browse/EMQX-11987
 2024-03-12 10:32:25 +01:00
Kjell Winblad
91514f3ace refactor: clean up over complicated code 
This commit cleans up overly complicated code and handles
the case when the worker pool is empty.

Thank you @thalesmg for suggesting this change.

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-03-12 10:22:29 +01:00
Kjell Winblad
1d1cdc1009 fix: return error reason from dynamo connector status check 
Fixes:
https://emqx.atlassian.net/browse/EMQX-11934
 2024-03-12 10:22:08 +01:00
JimMoen
0edeff4786
test: fix flaky 2024-03-12 15:21:17 +08:00
JimMoen
bc9084db79
test: cpu utilization value 2024-03-12 14:14:46 +08:00
JimMoen
0297c7b83b
refactor: conditional enclosed in parentheses 2024-03-12 14:14:45 +08:00
JimMoen
207f38c42a
fix(vm): cpu usage/idle handled by single worker 2024-03-12 14:14:44 +08:00
JianBo He
f24a76e770
Merge pull request #12668 from emqx/leap-year-bug-sql-fun 
fix: port the changes for date_to_unix_ts SQL fun from 4.4
 2024-03-12 10:48:31 +08:00
Andrew Mayorov
552968e5fe
test(conf): fix flaky config sync testcases 
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
 2024-03-11 13:03:13 +01:00
Andrew Mayorov
d725206bcb
Merge pull request #12624 from keynslug/fix/EMQX-11901/ds-error-class 
feat(sessds): handle recoverable errors during replay
 2024-03-11 12:41:45 +01:00
SergeTupchiy
b673ad3e5c
Merge pull request #12673 from zmstone/0308-fix-base64-compatibility-on-otp25 
fix(mgmt): avoid using base64:decode/2 and encode/2
 2024-03-11 13:23:37 +02:00
JimMoen
a82d4e1f7a
fix: retained_msg_count and shared_subscriptions api schema 2024-03-11 17:44:11 +08:00
JimMoen
e7574a70fb
feat(api): field shared_subscriptions in endpoint /monitor_current 2024-03-11 17:44:10 +08:00
Serge Tupchii
980a4c3a4d fix(emqx_mgmt_api): use emqx_utils:bin_to_hexstr/2 2024-03-11 10:40:19 +02:00
Serge Tupchii
f376aa8072 refactor(emqx_mgmt_api_clients): remove unnecessary function 2024-03-11 10:39:09 +02:00
Zaiming (Stone) Shi
68682c4231
Merge pull request #12671 from kjellwinblad/kjell/rule_engine/fix/unescape_rules_from_dashboard/EMQX-11847 
feat(rule engine SQL): add an `unescape` function
 2024-03-08 17:57:31 +01:00
zmstone
2cc1c563df fix(mgmt): avoid using base64:decode/2 and encode/2 
they are not available in otp 25
 2024-03-08 16:46:54 +01:00
Thales Macedo Garitezi
d92fd3e42d
Merge pull request #12667 from thalesmg/ds-delete-m-20240307 
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
 2024-03-08 10:57:51 -03:00
Thales Macedo Garitezi
79d7821222
Merge pull request #12655 from thalesmg/fix-kconsu-status-r56-20240306 
fix(kafka_consumer): check client connectivity
 2024-03-08 10:53:44 -03:00
Thales Macedo Garitezi
6af01b916e feat(ds): implement get_delete_streams, make_delete_iterator and delete_next callbacks for builtin storage 
Part of https://emqx.atlassian.net/browse/EMQX-11841
 2024-03-08 09:56:46 -03:00
Thales Macedo Garitezi
5dfd864c1a
Merge pull request #12666 from thalesmg/test-flaky-machine-m-20240307 
test(machine): attempt to stabilize flaky test
 2024-03-08 08:57:35 -03:00
Kjell Winblad
5a6f96212d feat(rule engine SQL): add an unescape function 
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:

- Standard C escape sequences:
  - `\n` for newline (LF)
  - `\t` for horizontal tab (HT)
  - `\r` for carriage return (CR)
  - `\b` for backspace (BS)
  - `\f` for formfeed (FF)
  - `\v` for vertical tab (VT)
  - `\'` for single quote (')
  - `\"` for double quote (")
  - `\\` for backslash (\)
  - `\?` for question mark (?)
  - `\a` for alert (bell, BEL)

- Hexadecimal escape codes:
  - `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
    a-f), allowing for the encoding of arbitrary utf32 characters.

If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.

Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
 2024-03-08 12:05:06 +01:00
Kjell Winblad
060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943 
fix(rule_engine): support non-byte sized input to bin2hexstr
 2024-03-08 09:41:03 +01:00
Kjell Winblad
6054499607
Merge pull request #12657 from kjellwinblad/kjell/rule_engine/fix/fun_calls_as_array_items/EMQX-11953 
fix(rule SQL): allow expressions as array items
 2024-03-08 09:39:46 +01:00
Shawn
29111a2192 fix: dialyzer problems 2024-03-08 14:25:19 +08:00
Shawn
163d095dca fix: port the changes for date_to_unix_ts SQL fun from 4.4 2024-03-08 10:48:08 +08:00
Thales Macedo Garitezi
963e0de0c3 fix(kafka_consumer): check client connectivity 
Fixes https://emqx.atlassian.net/browse/EMQX-11945
 2024-03-07 14:22:03 -03:00
Andrew Mayorov
f7e3afde16
test(ds): avoid introducing new macros 2024-03-07 16:49:20 +01:00
Thales Macedo Garitezi
b52c9c0062 test(machine): attempt to stabilize flaky test 2024-03-07 10:23:02 -03:00
Andrew Mayorov
69427dc42d
test(ds): add tests for error mapping and replay recovery 2024-03-07 12:59:58 +01:00
Andrew Mayorov
e7e8771277
fix(sessds): set replay retry timer if initial replay/3 fails 2024-03-07 12:59:58 +01:00
Andrew Mayorov
09905d78cd
chore(ds): make error handling slightly simpler 
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-03-07 12:59:57 +01:00
Andrew Mayorov
b604c3dbd4
refactor(sessds): make replay error handling a bit more clear 
Also leave a forgotten TODO.
 2024-03-07 12:59:57 +01:00
Andrew Mayorov
3f3e33b2cb
fix(sessds): untangle pull and replay retry timers 
And restore the convention that timer handler always manages only
its own timers.
 2024-03-07 12:59:57 +01:00
Andrew Mayorov
b39c710ec2
fix(ds): tidy up few typespecs 2024-03-07 12:59:57 +01:00
Andrew Mayorov
1cf672e78d
feat(sessds): handle recoverable errors during replay 2024-03-07 12:59:57 +01:00
Andrew Mayorov
2146d9e1fe
feat(ds): introduce error classes in critical API functions 
For now, only recoverable / unrecoverable errors are introduced.
 2024-03-07 12:59:57 +01:00
SergeTupchiy
1f38813cb9
Merge pull request #12561 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API 
feat: add client mqueue/inflight messages API
 2024-03-07 13:20:31 +02:00
ieQu1
c62776edaf fix(sessds): Prevent hot update of session_persistence.enable config 2024-03-07 12:19:53 +01:00
Serge Tupchii
a2e761681e feat: add client mqueue/inflight messages API 2024-03-07 12:06:49 +02:00
zmstone
6a2731f2da Merge remote-tracking branch 'origin/master' into release-56 2024-03-07 09:36:37 +01:00
Zaiming (Stone) Shi
98034cb4dd
Merge pull request #12660 from zmstone/sync-5.5.1 
Sync 5.5.1
 2024-03-07 09:36:08 +01:00
Thales Macedo Garitezi
c62dd56a4a
Merge pull request #12661 from thalesmg/ds-atomic-store-m-20240305 
feat(ds): add atomic store API
 2024-03-06 17:48:28 -03:00
zmstone
f57f2fa1b7 chore: bump app version numbers 2024-03-06 19:37:06 +01:00
Thales Macedo Garitezi
df3ebc5c56
Merge pull request #12659 from thalesmg/test-flaky-pulsar-r56-20240306 
test(pulsar): fix flaky test
 2024-03-06 15:29:01 -03:00
Thales Macedo Garitezi
b421363661
Merge pull request #12656 from thalesmg/fix-gprodu-status-r56-20240306 
fix(gcp_pubsub_producer): check for topic existence when creating action
 2024-03-06 15:28:51 -03:00
Thales Macedo Garitezi
5d87d400f4 feat(ds): add atomic store API 
Part of https://emqx.atlassian.net/browse/EMQX-11841
 2024-03-06 15:24:14 -03:00
Thales Macedo Garitezi
dc16e59f2c fix(gcp_pubsub_producer): check for topic existence when creating action 
Fixes https://emqx.atlassian.net/browse/EMQX-11949
 2024-03-06 13:43:50 -03:00
zmstone
e99546e009 Merge remote-tracking branch 'origin/release-56' into sync-5.5.1 2024-03-06 17:27:54 +01:00
Thales Macedo Garitezi
eacd803a37 test(pulsar): fix flaky test 2024-03-06 12:07:02 -03:00
Thales Macedo Garitezi
fd7eacb953 test(elastic_search): fix flaky test 2024-03-06 11:38:22 -03:00
Kjell Winblad
c206ee37e0 fix(rule SQL): allow expressions as array items 
Fixes:
https://github.com/emqx/emqx/issues/12465
https://emqx.atlassian.net/browse/EMQX-11953
 2024-03-06 15:38:19 +01:00
Kjell Winblad
69114bc6c2 fix(rule_engine): support non-byte sized input to bin2hexstr 
The rule engine subbits function can return a bitstring which size is
not divisible by 8. Therefore, it makes sense that the rule engine
function bin2hexstr can handle such bitstrings as well. This is fixed by
this commit.

Fixes:
https://github.com/emqx/emqx/issues/12586
https://emqx.atlassian.net/browse/EMQX-11943
 2024-03-06 15:26:49 +01:00
Kjell Winblad
78d5f76f70
Merge pull request #12652 from kjellwinblad/kjell/rule_engine/fix/subbits/EMQX-11942 
fix: add subbits/4 and subits/5 rule_engine functions
 2024-03-06 15:20:30 +01:00
Zaiming (Stone) Shi
fc8b5d4522
Merge pull request #12646 from zmstone/0304-rule-engin-func-fix-zone-shift-in-date-string-parse 
fix(rule_func): time zone shift at wrong precision
 2024-03-06 13:22:30 +01:00
JianBo He
f1e9da9048
Merge pull request #12604 from thalesmg/test-gcp-consu-tm-m-20240227 
test(gcp_pubsub_consumer): add test case for updating topic when there is a topic mapping
 2024-03-06 09:13:41 +08:00
JianBo He
1737df05c7
Merge pull request #12641 from zmstone/0304-improve-text-log-formatter 
0304 improve text log formatter
 2024-03-06 09:12:18 +08:00
Kjell Winblad
8cf681ad3b
Merge pull request #12639 from kjellwinblad/kjell/fix_flaky_test_case/emqx_broker_SUITE.connected_client_count_group.quic.t_connected_client_count_transient_takeover 
test(emqx_broker_SUITE): fix flaky test case
 2024-03-05 18:02:07 +01:00
Ilya Averyanov
3285edc004
Merge pull request #12650 from savonarola/0305-refine-retainer-readme 
chore(retainer): actualize README
 2024-03-05 18:57:02 +02:00
Thales Macedo Garitezi
3ce8dcaa3f
Merge pull request #12643 from thalesmg/test-flaky-elasticsearch-m-20240304 
test(elastic_search): fix flaky test
 2024-03-05 13:53:42 -03:00
Thales Macedo Garitezi
24cb45a643 test(elastic_search): fix flaky test 2024-03-05 11:29:49 -03:00
Thales Macedo Garitezi
69dcc69761
Merge pull request #12640 from thalesmg/sync-r56-m-20240304 
sync `release-56` to `master`
 2024-03-05 10:44:28 -03:00
Kjell Winblad
365d054e01 fix: add subbits/4 and subits/5 rule_engine functions 
The documentation for the family of subbits functions says that the
fifth and sixth parameters are optional (since they only make sense when
the forth parameter is 'integer'). However, before this commit
`subbits/4` and `subbits/5` did not exist.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11942
https://github.com/emqx/emqx/issues/12587
 2024-03-05 13:27:07 +01:00
Thales Macedo Garitezi
676df7eb30 test(gcp_pubsub_consumer): add test case for updating topic when there is a topic mapping 
Checks that, if a migrated bridge originally has a `topic_mapping` and is later updated
with V2 APIs (without topic mapping in the input), then the new V2 `topic` field prevails.
 2024-03-05 09:12:24 -03:00
Thales Macedo Garitezi
a840925a50
Merge pull request #12642 from thalesmg/ds-fix-drop-typespec-m-20240304 
fix(ds): fix `drop_generation` typespec
 2024-03-05 09:11:08 -03:00
Thales Macedo Garitezi
c5489fee90 Merge remote-tracking branch 'origin/release-56' into sync-r56-m-20240304 2024-03-05 09:08:59 -03:00
Ilya Averyanov
ce50aed930 chore(retainer): actualize README 2024-03-05 13:08:36 +03:00
Kjell Winblad
c8e42cf6b1 test(emqx_broker_SUITE): fix flaky test case 2024-03-05 10:14:07 +01:00
zmstone
cb0066d639 chore: add tag for logs from MQTT connection modules 2024-03-05 08:39:53 +01:00
Zaiming (Stone) Shi
00b21cf9c5
Merge pull request #12634 from zmstone/0304-breaking-no-escape-for-triple-quote-string-characters 
chore: upgrade to hocon 0.42.0
 2024-03-05 08:32:50 +01:00
lafirest
c2dcb507cf
Merge pull request #12637 from lafirest/merge-55 
sync release-55 to master
 2024-03-05 08:47:30 +08:00
Serge Tupchii
e725064a2a fix(emqx_conf_schema): use timeout_duration_s() type for log.throttling.time_window field 2024-03-04 22:41:28 +02:00
zmstone
9929025820 fix(rule_func): time zone shift at wrong precision 2024-03-04 21:25:49 +01:00
Thales Macedo Garitezi
61328eb91a chore: prepare e5.6.0-alpha.2 2024-03-04 15:37:58 -03:00
zmstone
3e3194fd21 chore: upgrade to hocon 0.42.0 2024-03-04 19:12:42 +01:00
Thales Macedo Garitezi
06334798a5 fix(ds): fix drop_generation typespec 
This typespec fix will be used downstream by other backends.
 2024-03-04 14:15:59 -03:00
zmstone
6c9eb16a95 refactor(logger): reorder log fields 
tag > clientid > msg > peername > username > topic > [other fields]
 2024-03-04 18:10:19 +01:00
Thales Macedo Garitezi
577ef41e45 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 12:48:06 -03:00
Ivan Dyachkov
cdcab23c54
Merge pull request #12615 from id/0229-prepare-5.5.1-release 
chore: prepare 5.5.1 release
 2024-03-04 16:34:13 +01:00
lafirest
79f0720d14
Merge pull request #12635 from lafirest/fix/r551 
fix(http): fix that sensitive headers may be printed in log when querying
 2024-03-04 22:32:14 +08:00
Ivan Dyachkov
7780b4f1d7 chore: prepare 5.5.1 release 2024-03-04 15:15:10 +01:00
firest
6d9b2a95c3 Merge remote-tracking branch 'origin/release-55' 2024-03-04 20:47:20 +08:00
Thales Macedo Garitezi
0dd9990f24 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 09:44:30 -03:00
Thales Macedo Garitezi
8b66ea7852
Merge pull request #12625 from thalesmg/test-retainer-extensions-m-20240301 
test(retainer): extend test suite for usage by other backends
 2024-03-04 09:40:32 -03:00
firest
c5eb09a72f fix(http): fix that sensitive headers may be printed in log when querying 2024-03-04 20:36:01 +08:00
JimMoen
3e7311ee31
Merge pull request #12619 from JimMoen/EMQX-11468/refactor/sqlserver 
feat: refactor MS SQL Server bridge to connector and action
 2024-03-04 18:05:14 +08:00
ieQu1
2e792da934
Merge pull request #12562 from ieQu1/dev/new-ds-schema 
Refactor schema for the durable storage
 2024-03-04 10:33:27 +01:00
JimMoen
4623b73f47
fix: sqlserver connector auto reconnect 2024-03-04 17:13:25 +08:00
JimMoen
2e3003e0f1
test: sqlserver bridge v2 2024-03-04 17:13:24 +08:00
JimMoen
1d58092176
fix: bridge v2 on_query matched ChannelId 2024-03-04 17:13:23 +08:00
JimMoen
00d50479f5
fix: compatible with bridge v1 2024-03-04 17:13:22 +08:00
JimMoen
9143d5994d
feat: refactor MS SQL Server bridge to connector and action 2024-03-04 17:13:21 +08:00
JimMoen
0a33f9d027
fix(calendar): leap year time to unix timestamp 2024-03-04 15:37:49 +08:00
Ivan Dyachkov
9f61239704 Merge remote-tracking branch 'upstream/release-56' into 0304-sync-release-56 2024-03-04 08:00:39 +01:00
Zaiming (Stone) Shi
e9a5670cce
Merge pull request #12581 from zmstone/0223-add-json-schema 
feat: schema registry supports json schema
 2024-03-02 10:09:31 +01:00
Thales Macedo Garitezi
08ef2c7b8b test(retainer): extend test suite for usage by other backends 
Part of https://emqx.atlassian.net/browse/EMQX-11922
 2024-03-01 17:27:15 -03:00
Ivan Dyachkov
a20478fff0 chore: update scripts and workflows for 5.6.x 2024-03-01 09:01:34 +01:00
Zaiming (Stone) Shi
fa275bf0a2 refactor(schema_registry): use one func for both encode and decode check 2024-03-01 07:20:59 +01:00
firest
1c1c4e497d chore: bump version && update change 2024-03-01 12:42:19 +08:00
firest
5a3a34cce7 fix(redact): enhanced the redact for sensitive headers 2024-03-01 12:30:26 +08:00
firest
0670272188 fix: Revert "fix: redact all headers from logs" 
This reverts commit d8032f47ca.
 2024-03-01 10:32:57 +08:00
zhongwencool
d56fb22208
Merge pull request #12595 from thalesmg/kafka-consumer-source-m-20240223 
feat: migrate kafka consumer bridge to source + connector
 2024-03-01 10:11:16 +08:00
Zaiming (Stone) Shi
b7e5ea2941 feat(schema_registry): add JSON schema 2024-02-29 21:03:52 +01:00
Thales Macedo Garitezi
5d94a910b9 test: reduce flakiness 2024-02-29 13:59:43 -03:00
Zaiming (Stone) Shi
21cf600242 chore: address previous review comments 2024-02-29 16:13:52 +01:00
Zaiming (Stone) Shi
66fbcdcefa test: fix a compile warning 2024-02-29 16:13:52 +01:00
Zaiming (Stone) Shi
5afe000ada
Merge pull request #12612 from zmstone/0226-fix-schema-registry-replication 
refactor: change schema registry SERD_TAB to ets
 2024-02-29 14:58:41 +01:00
Ilya Averyanov
5bb769e659
Merge pull request #12616 from savonarola/0229-ds-interface 
feat(ds): export types
 2024-02-29 15:42:42 +02:00
Thales Macedo Garitezi
16b3dc1166 refactor: use individual keys for subscriber id resources 2024-02-29 10:39:37 -03:00
zhongwencool
bc9de20024
Merge pull request #12540 from zhongwencool/bridge-pulsar-v2 
feat: pulsar bridge v2
 2024-02-29 21:25:11 +08:00
Thales Macedo Garitezi
0aa9a872a3 feat: check brod_sup for client id 2024-02-29 09:53:23 -03:00
William Yang
b82973b36b
Merge pull request #11868 from qzhuyan/fix/william/takeover-pub-willmsg 
fix: willmsg not published in takeover
 2024-02-29 13:14:18 +01:00
Ilya Averyanov
b706caf294 feat(ds): export types 2024-02-29 14:27:18 +03:00
Ilya Averyanov
f87c17b540
Merge pull request #12611 from savonarola/0228-ds-interface 
feat(ds): update delete/count interface
 2024-02-29 10:31:51 +02:00
Ilya Averyanov
d4519bda82
Merge pull request #12589 from savonarola/0222-retainer-interface 
Make retainer pluggable
 2024-02-29 10:31:15 +02:00
zhongwencool
3814203fa2 test(pulsar): add pulsar action v2 testcase 2024-02-29 16:23:37 +08:00
zhongwencool
e9e1daf962 chore: update some pulsar's desc 2024-02-29 16:23:37 +08:00
zhongwencool
650f9659a4 fix: create pulsar producer when on_add_channel 2024-02-29 16:23:37 +08:00
zhongwencool
7f1b4cef27 feat: pulsar bridge v2 2024-02-29 16:23:37 +08:00
zhongwencool
21e0ecfcce
Merge pull request #12583 from zhongwencool/wait-for-listener-stop 
chore: wait_for cowboy listener release sock when stop_listener
 2024-02-29 15:03:22 +08:00
lafirest
d2dda2535b
Merge pull request #12602 from lafirest/fix/iotdb_ping 
fix(iotdb): use the `ping` API to check the status of the IOTDB connections
 2024-02-29 12:26:10 +08:00
firest
0bebd66f05 fix(iotdb): make dialyzer happy 2024-02-29 10:51:12 +08:00
zhongwencool
49b508544e
chore: port not released msg 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2024-02-29 10:49:29 +08:00
Zaiming (Stone) Shi
10c1245125 refactor: change schema registry SERD_TAB to ets 
Prior to this change, it's a mria/mnesia table
which may cause nodes to overwrite each other.
 2024-02-28 23:13:07 +01:00
Ilya Averyanov
d5ae0e5c53 feat(ds): update delete/count interface 2024-02-28 22:51:24 +03:00
Zaiming (Stone) Shi
704eceb7c7
Merge pull request #12599 from zmstone/0227-refactor-log-config-template 
docs: add log docs
 2024-02-28 17:58:14 +01:00
Andrew Mayorov
ef4ae92da4
Merge pull request #12523 from keynslug/ft/bump-ekka-0.19.0 
refactor: bump ekka to 0.19.0 w/o mnesia boot phase
 2024-02-28 17:46:29 +01:00
Ilya Averyanov
dfdf7455d3 feat(retainer): make additional implementations pluggable 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-02-28 18:43:17 +03:00
Ilya Averyanov
d94066a10a chore(retainer): refactor types; remove hotwiring to builtin config 2024-02-28 18:43:16 +03:00
Ilya Averyanov
aa2ee9c409 chore(retainer): add backend creation to the behaviour 2024-02-28 18:43:16 +03:00
lafirest
d3fb72733e
Merge pull request #12610 from lafirest/fix/ldap_bind_health 
fix(ldap): fixed that the connection to the LDAP connector could be disconnected after a period of time
 2024-02-28 23:03:10 +08:00
lafirest
eaef753c36
Merge pull request #12608 from lafirest/fix/iotdb 
fix(iotdb): fix function clause error when there is no `payload` field
 2024-02-28 22:00:54 +08:00
Andrew Mayorov
20c4029b33
test(cluster-rpc): unload any mocks in end_per_suite/1 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-02-28 14:52:37 +01:00
firest
b9f644c355 fix(ldap): fixed that the connection to the LDAP connector could be disconnected after a period of time 2024-02-28 21:46:03 +08:00
firest
ce2552712f chore: bump version && update change 2024-02-28 18:32:08 +08:00
Andrew Mayorov
c3b044a37b
fix(api-topics): respond with count when it's cheap to compute 2024-02-28 11:28:33 +01:00
Andrew Mayorov
bd578a799e
fix(api-topics): avoid doing full scans over router tables 2024-02-28 10:52:44 +01:00
Andrew Mayorov
9bb0868628
test(api-topics): add testcase verifying paged queries work 2024-02-28 10:52:41 +01:00
firest
01207ef97b fix(iotdb): fix function clause error when there is no payload field 2024-02-28 16:54:01 +08:00
zhongwencool
660a1ce99d chore: use gen_tcp:recv timeout to wait socket closed 2024-02-28 16:18:32 +08:00
JimMoen
6fd04e33f5
fix(prom): skip cert info for disabled ssl/wss/quic listeners 2024-02-28 15:39:33 +08:00
JimMoen
5f125047f5
refactor: remove unnecessary lists fold for cerfile 2024-02-28 14:53:20 +08:00
JimMoen
812e8ef314
fix: try-catch for unknow reason to inhibit api crash 2024-02-28 14:53:19 +08:00
zhongwencool
4307aecefd chore: wait_for cowboy listener release sock when stop_listener 2024-02-28 14:24:15 +08:00
firest
c3a2cf6220 chore: update change && bump version 2024-02-28 13:18:48 +08:00
JimMoen
042a84c30f
fix(prom): api crash when tls cert file non existed 2024-02-28 11:35:55 +08:00
zhongwencool
aa7def425d fix: get grpc_timeout from hstreamdb connector config failed 2024-02-28 08:45:06 +08:00
Andrew Mayorov
28d664bae2
test(pulsar): simplify the test suite 2024-02-27 23:51:54 +01:00
Andrew Mayorov
367ffa8e80
feat(utils-fs): add function to compute relpaths naively 2024-02-27 23:41:50 +01:00
Thales Macedo Garitezi
6b9844ae82 feat: migrate kafka consumer bridge to source + connector 
Fixes https://emqx.atlassian.net/browse/EMQX-11848
 2024-02-27 17:52:33 -03:00
Thales Macedo Garitezi
787dcd0e86
Merge pull request #12577 from thalesmg/gcp-service-json-binary-m-20240223 
feat(gcp_pubsub_bridges): make service account json a binary
 2024-02-27 15:42:42 -03:00
Zaiming (Stone) Shi
ef99037335 docs: add log docs 2024-02-27 19:24:56 +01:00
Andrew Mayorov
6ea9d2a6d9
test(conf): simplify confsync test suite 2024-02-27 16:41:54 +01:00
Andrew Mayorov
ae79516fd2
test(cluster-rpc): ensure testsuite is side-effects-free 2024-02-27 16:41:54 +01:00
Andrew Mayorov
56eefe34d5
feat(cth-cluster): use workdir as cwd on each node 2024-02-27 16:41:53 +01:00
Andrew Mayorov
d60ff1e616
test(plugins): avoid using same workdir for different nodes 2024-02-27 16:41:53 +01:00
Andrew Mayorov
3a008c8b4c
refactor: bump ekka to 0.19.0 w/o mnesia boot phase 2024-02-27 16:41:50 +01:00
Thales Macedo Garitezi
8a204f2ac1
Merge pull request #12596 from thalesmg/fix-header-logging-r55-20240226 
fix: redact all headers from logs
 2024-02-27 11:19:51 -03:00
firest
77239da7ed fix(iotdb): use the ping API to check the status of the IOTDB connections 2024-02-27 22:07:41 +08:00
lafirest
02de92a53d
Merge pull request #12601 from lafirest/fix/eldap_log 
fix(ldap): fix that logs of eldap will never be logged
 2024-02-27 21:50:05 +08:00
JimMoen
2ded751237
test: nl not allowed for shared-sub 2024-02-27 18:24:30 +08:00
zhongwencool
54c542c795 chore: rename _probe_ to t_probe_ 2024-02-27 17:57:15 +08:00
zhongwencool
c67f2130f2 fix: check connector and bridge_v2 with the right schema 2024-02-27 17:57:10 +08:00
JimMoen
c6b1102b2b
fix(mgmt_client): nl not allowed for shared-sub 2024-02-27 17:51:39 +08:00
firest
f3237a1bf7 chore: bump emqx_ldap version 2024-02-27 17:48:34 +08:00
firest
5c69500c52 fix(ldap): fix that logs of eldap will never be logged 2024-02-27 17:48:22 +08:00
lafirest
9ae7ac04a2
Merge pull request #12597 from lafirest/fix/ldap_log 
fix(ldap): fix that logs of eldap will never be logged
 2024-02-27 17:40:44 +08:00
Zaiming (Stone) Shi
1567c5beb0
Merge pull request #12580 from zmstone/0224-delete-stale-macro 
chore: delete stale macro
 2024-02-27 09:58:06 +01:00
Zaiming (Stone) Shi
9c9a2b8494
Merge pull request #12592 from zmstone/0226-duration_s-min-value 
chore: for dashboad add a 'minimum' attribute for duration_s type
 2024-02-27 09:57:33 +01:00
firest
da13ec736c chore: bump emqx_ldap version 2024-02-27 16:08:47 +08:00
JimMoen
c1da7449fe
test(mgmt_api): bulk/unbulk sub/unsub shared topic filter 2024-02-27 16:01:12 +08:00
JimMoen
a76415c4f6
fix(mgmt): sub/unsub a share subscription to the client via http api 
- `/clients/:clientid/subscribe`
- `/clients/:clientid/subscribe/bulk`
- `/clients/:clientid/unsubscribe`
- `/clients/:clientid/unsubscribe/bulk`
 2024-02-27 16:00:45 +08:00
firest
6e73cfa2cf fix(ldap): fix that logs of eldap will never be logged 2024-02-27 15:47:46 +08:00
William Yang
6c7b7741d5 feat(quic): mqtt 5.0 graceful shutdown in takeover 2024-02-26 23:05:49 +01:00
William Yang
c8f9ffdc1f chore: update some comments 2024-02-26 22:24:16 +01:00
William Yang
88fc67b369 chore(willmsg): remove unreachable code 2024-02-26 22:24:16 +01:00
William Yang
d5247cb567 refactor: update notes for willmsg 2024-02-26 22:24:16 +01:00
William Yang
975c7429e5 chore: clean in tests 2024-02-26 22:24:13 +01:00
Thales Macedo Garitezi
d8032f47ca fix: redact all headers from logs 
Fixes https://emqx.atlassian.net/browse/EMQX-11904

Since headers are usually used for authentication and the headers used for that are very
flexible, we redact all headers from logs to avoid leaking anything.
 2024-02-26 18:04:12 -03:00
Zaiming (Stone) Shi
bb6a95bae8
Merge pull request #12590 from zmstone/0226-remove-mfa-from-logs 
refactor: delete mfa from log metadata
 2024-02-26 21:37:57 +01:00
Ilya Averyanov
5326cc80ed
Merge pull request #12594 from savonarola/0226-add-ds-delete-interface 
chore(ds): add delete callbacks
 2024-02-26 20:07:21 +02:00
Thales Macedo Garitezi
9ca5684b34
Merge pull request #12591 from thalesmg/dev-cls-m-20240226 
test: make `clear_screen` safer
 2024-02-26 13:40:16 -03:00
Ilya Averyanov
b010d34640 chore(ds): add delete callbacks 2024-02-26 17:35:13 +03:00
Serge Tupchii
5d4c85cd84 chore: list throttled log events in the change-log 2024-02-26 15:09:38 +02:00
Serge Tupchii
5b48b06d4a fix: trace throttled log events 2024-02-26 15:09:38 +02:00
Zaiming (Stone) Shi
652e977b3c chore: for dashboad add a 'minimum' attribute for duration_s type 2024-02-26 13:51:07 +01:00
Thales Macedo Garitezi
c6195fcf50 test: make clear_screen safer 
Useful when iterating on the tests in a loop, to get rid of all the garbaged printed
before the test itself beings.

Only actually does anything if the environment variable `CLEAR_SCREEN` is set to `true`
and only clears the screen the screen the first time it's encountered, so it's harmless
otherwise.
 2024-02-26 09:31:06 -03:00
Thales Macedo Garitezi
f2c372d9ff feat(gcp_pubsub_bridges): make service account json a binary 
Fixes https://emqx.atlassian.net/browse/EMQX-11384

Today, service_account_json config field is an embedded object (map()).

This requires user to embed a JSON object into the config file instead of embedding it as
a string.

We should support binary() type as input, but keep supporting map() for backward
compatibility.
 2024-02-26 09:21:07 -03:00
Zaiming (Stone) Shi
00c2aeb1ab refactor: delete mfa from log metadata 2024-02-26 12:15:00 +01:00
Zaiming (Stone) Shi
de5e4491f7
Merge pull request #12576 from zmstone/0223-hide-bridges-root-from-docs 
docs: stop generating schema doc for 'bridges' root
 2024-02-24 09:08:05 +01:00
Zaiming (Stone) Shi
94a01d23c1 chore: delete stale macro 
EMQX_API_VERSION is no longer in use since 5.2.0
 2024-02-24 09:06:25 +01:00
Zaiming (Stone) Shi
a3e81c5039 chore: prepare for release 5.5.1-rc.2 2024-02-24 08:59:51 +01:00
ieQu1
fe4c7cd2dc
fix(ds): Apply review remarks, and hide certain fields 2024-02-23 18:07:56 +01:00
ieQu1
e126393cbf
fix(ds): Fix schema checker warnings for DS schema 2024-02-23 15:39:52 +01:00
ieQu1
c18fc6a4bb
fix(sessds): Remove deprecated configuration parameters 2024-02-23 15:08:13 +01:00
ieQu1
91ddbbcc3f
fix(sessds): Replace min- and max- batch size with batch_size 2024-02-23 15:08:13 +01:00
ieQu1
94b0ab983d
docs(ds): Add descriptions for the builtin's egress config 2024-02-23 15:08:13 +01:00