yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,732 Commits 393 Branches 684 Tags 74 MiB
fda365a87b
Commit Graph

12180 Commits

Author SHA1 Message Date
Kjell Winblad
413ad60bdb fix(cassandra): format rendered trace in a better way 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12393
 2024-05-16 11:35:12 +02:00
Kjell Winblad
9fd8e930be fix(best_effort_json): only do tuple list to map without losing pairs 2024-05-16 11:19:48 +02:00
Ilya Averyanov
e1ce6377f3 chore(streams): add stream methods, optimize streams 2024-05-16 12:04:08 +03:00
Ilia Averianov
1f7e358e1e
Merge pull request #13006 from savonarola/0509-fix-retainer-banned 
fix(banned): verify delayed/taken over/retained messages against clientid_re ban rules
 2024-05-16 11:37:15 +03:00
zmstone
2acde5a4e4 fix(authn/http): log meaningful error message if http header is missing 2024-05-16 11:36:52 +03:00
zmstone
4fb484d4cf chore(emqx_schema): change atom array to enum array for alarm.actions 2024-05-16 11:36:52 +03:00
Ilya Averyanov
daf2e5a444 chore(auth_http): unify http request generation 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-05-16 11:36:52 +03:00
Kjell Winblad
cf5b464777
Merge pull request #12937 from kjellwinblad/kjell/add_use_legacy_protocol_option_to_mongodb_auth/EMQX-12245 
test(mongodb authn, authz): add test cases for use_legacy_protocol
 2024-05-16 10:10:17 +02:00
Kjell Winblad
72b0c63ed2
Merge pull request #13043 from kjellwinblad/kjell/kafka_incorrect_query_type/EMQX-12363 
fix(kafka trace): do not include query type in trace entry
 2024-05-16 10:08:16 +02:00
ieQu1
deffb01544
fix(ds): Change the default number of shards to 12 2024-05-16 08:06:07 +02:00
Ilya Averyanov
552b62236c chore(retainer): move filter/foreach to emqx_utils_stream 2024-05-15 19:09:06 +03:00
Ilya Averyanov
0b39aaadbd chore(retainer): get rid of qlc usage 2024-05-15 19:09:06 +03:00
Thales Macedo Garitezi
2eb1bb2b13
Merge pull request #13032 from kjellwinblad/kjell/no_ctx_message_validation_failed/EMQX-12354 
fix: add handling of message_validation_failed ctx for rule testing
 2024-05-15 11:20:18 -03:00
Kjell Winblad
246bce85ec fix: make mysql action rendered trace include parameters 
When doing non-batch inserts with the MySQL action, prepared statements
are used. We therefore need to include the parameters to the prepared
statement when tracing the rendered templates.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12335
 2024-05-15 14:49:33 +02:00
Kjell Winblad
c72682d81b fix: remove query mode from redis action trace as it only supports sync 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12336
 2024-05-15 14:32:15 +02:00
Ilya Averyanov
78a87ab5a6 fix(banned): verify delayed/taken over/retained messages against clientid_re rules 2024-05-15 14:05:38 +03:00
Kjell Winblad
8cbfc4c0cd fix: evaluate lazy values in normal log formatters when level is debug 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12380
 2024-05-15 11:47:21 +02:00
lafirest
faf3492e58
Merge pull request #13039 from lafirest/fix/rocket_key 
fix(rocketmq): fix a typo and a RocketMQ encoding error
 2024-05-15 15:28:35 +08:00
Andrew Mayorov
3e1e4bab17
Merge pull request #13047 from keynslug/fix/EMQX-12366/api-ds-leave 
feat(api-ds): provide more information on nonexistent site leave
 2024-05-14 23:21:07 +02:00
Thales Macedo Garitezi
bcbeadd4af test: stabilize flaky test 2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi
f062517ccb test: stabilize flaky test 2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi
b0aa7f25aa test: add more context to help debug flaky test 
Ex: https://github.com/emqx/emqx/actions/runs/9084293110/job/24967912514?pr=13050#step:6:18330
 2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi
fd949240c0 feat(schema validation): implement backup restore 
Fixes https://emqx.atlassian.net/browse/EMQX-12346
 2024-05-14 15:58:19 -03:00
Thales Macedo Garitezi
bc5071590b
Merge pull request #13033 from thalesmg/fix-ds-more-client-api-fields-r57-20240513 
fix(ds clients mgmt api): add more fields to disconnected durable sessions
 2024-05-14 15:29:40 -03:00
Thales Macedo Garitezi
7023e6ad96
Merge pull request #13017 from thalesmg/fix-ds-subs-pages-r57-20240510 
fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page
 2024-05-14 15:29:29 -03:00
Thales Macedo Garitezi
2a0ac34656 Merge branch 'release-57' into sync-r57-m-20240514 2024-05-14 15:03:04 -03:00
Kjell Winblad
e811f4a392 fix: add handling of message_validation_failed ctx for rule testing 
Before this commit, the backend could not handle the test context for
message_validation_failed events that could be sent from the dashboard
since no handling code for this contexts existed in the backend. This
has now been fixed by adding such handling code.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12354
 2024-05-14 13:07:32 -03:00
Thales Macedo Garitezi
1730a41337
Merge pull request #13046 from thalesmg/refactor-mv-rename-sv-r57-20240514 
refactor: rename `message_validation` to `schema_validation`
 2024-05-14 13:03:12 -03:00
Kjell Winblad
adbc392265
Merge pull request #13027 from kjellwinblad/kjell/fix_trace_json_report_formatting/EMQX-12344 
fix: JSON trace formatter should handle report style log entries
 2024-05-14 18:02:59 +02:00
firest
437a0c0d55 fix(rocketmq): fix a typo and a RocketMQ encoding error 2024-05-14 23:01:31 +08:00
Andrew Mayorov
2f90d15ce6
fix(ds-api): adapt CLI to API changes 2024-05-14 16:55:17 +02:00
Andrew Mayorov
35e360fcbe
feat(api-ds): provide more information on nonexistent site leave 2024-05-14 15:57:41 +02:00
Thales Macedo Garitezi
257dae5211 refactor: rename message_validation to schema_validation 
Request from Product team.

Fixes https://emqx.atlassian.net/browse/EMQX-12383
 2024-05-14 10:32:27 -03:00
lafirest
dc391d2403
Merge pull request #13009 from lafirest/fix/rate_log 
feat(limiter): lift the log level of rate limiter to `warning`
 2024-05-14 21:11:26 +08:00
Thales Macedo Garitezi
15acd86b31
Merge pull request #13034 from thalesmg/test-flaky-mq-r57-20240513 
test(client mgmt api): attempt to fix flaky test
 2024-05-14 09:01:29 -03:00
Thales Macedo Garitezi
e8d2c48d3c
Merge pull request #13018 from thalesmg/fix-postgres-log-spam-r57-20240510 
fix(postgres): reduce log spamming when connection goes down
 2024-05-14 09:00:36 -03:00
Kjell Winblad
f75c7a5cea fix(trace log entry error): better structure and naming 2024-05-14 13:38:41 +02:00
Andrew Mayorov
74ed0f578a
Merge pull request #13042 from keynslug/fix/EMQX-12204/api-ns 
fix(s3-aggreg): disambiguate action schema namespace
 2024-05-14 13:12:34 +02:00
Kjell Winblad
121084b2ff fix(kafka trace): do not include query type in trace entry 
For Kafka that has its internal buffering the on_query_async callback is
used both for sync and async queries (for sync queries the caller waits
for a response directly after calling on_query_async). Therefore, before
this commit, the trace included incorrect query type information. This
has been fixed by removing the query type information from the trace.
This should be okay since the query type is not essential information
and can be derived from the configuration.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12363
 2024-05-14 12:24:30 +02:00
ieQu1
525e4dac95
Merge pull request #13036 from ieQu1/dev/reduce-log-spam 
tests: Reduce log spam in the failed test suites
 2024-05-14 10:53:30 +02:00
ieQu1
f663373c57
Merge pull request #13037 from ieQu1/dev/merge-release57-240513 
Sync release-57 to the master
 2024-05-14 10:50:30 +02:00
Andrew Mayorov
c10a48f4be
fix(s3-aggreg): disambiguate action schema namespace 
Otherwise schemas for `s3` and `s3_aggregated_upload` collide when
projected into OpenAPI schema.
 2024-05-14 10:49:21 +02:00
zmstone
93232d4253 fix(authn/http): log meaningful error message if http header is missing 2024-05-14 10:22:07 +02:00
Andrew Mayorov
98a93662c1
Merge pull request #13035 from keynslug/fix/EMQX-12291/pers-sys-msg 
fix(sessds): persist $SYS messages as well
 2024-05-14 10:05:49 +02:00
firest
7a7bd84cba fix(limiter): improved log content 2024-05-14 15:34:13 +08:00
Ilya Averyanov
bca3782d73 fix(auth_http): fix query encoding 
* ignore authenticator if JSON format is set up for requests, but non-utf8 data is going to be sent
* use application/json format by default
* fix encoding of query part of the requests
 2024-05-14 10:32:53 +03:00
ieQu1
25c6ac3d16
Merge remote-tracking branch 'origin/release-57' 2024-05-13 22:05:29 +02:00
ieQu1
ac3f5a083d
test: Reduce log spam in the failed test suites 2024-05-13 22:00:33 +02:00
ieQu1
8506ca7919
Merge pull request #12998 from ieQu1/dev/improve-latency 
Use leader's clock when calculating LTS cutoff timestamp
 2024-05-13 21:54:06 +02:00
Andrew Mayorov
66b7ac4c45
fix(sessds): persist $SYS messages as well 
Otherwise, persistent sessions will not be able to receive $SYS messages
whatsoever.
 2024-05-13 21:27:55 +02:00
Thales Macedo Garitezi
a2a5fd1f8e test(client mgmt api): attempt to fix flaky test 
```
%%% emqx_mgmt_api_clients_SUITE ==> msgs_plain_encoding.t_mqueue_messages: FAILED
%%% emqx_mgmt_api_clients_SUITE ==>
Failure/Error: ?assertEqual(79, Count)
  expected: 79
       got: 100
      line: 1303
```
 2024-05-13 15:53:21 -03:00
ieQu1
3da3a36863
test(ds): Add generation in the replication suite 2024-05-13 19:51:04 +02:00
ieQu1
9f7ef9f34f
fix(ds): Apply review remarks 2024-05-13 19:35:24 +02:00
Thales Macedo Garitezi
e7c47f4ec0 fix(ds clients mgmt api): add more fields to disconnected durable sessions 
Fixes https://emqx.atlassian.net/browse/EMQX-12369
 2024-05-13 14:25:02 -03:00
Kjell Winblad
1c34a84dd2
Merge pull request #13028 from kjellwinblad/kjell/swagger_example_missing_ruleid_paramter/EMQX-12325 
fix: include ruleid parameter in swagger example for apply rule endpoint
 2024-05-13 17:35:31 +02:00
Thales Macedo Garitezi
81473519bd
Merge pull request #13031 from thalesmg/fix-clientsv2-sec-r57-20240513 
fix(client mgmt api): require authn for endpoint
 2024-05-13 12:17:31 -03:00
Thales Macedo Garitezi
39899b917d fix(client mgmt api): require authn for endpoint 
Fixes https://emqx.atlassian.net/browse/EMQX-12361
 2024-05-13 10:54:21 -03:00
Thales Macedo Garitezi
6b032faebc fix(client mgmt api): return expired durable sessions with is_expired: true 
Fixes https://emqx.atlassian.net/browse/EMQX-12274
 2024-05-13 10:51:24 -03:00
Thales Macedo Garitezi
0dee2d67b1 fix: always check if first query has more rows 2024-05-13 09:10:18 -03:00
Kjell Winblad
c8690a8273 fix: include ruleid parameter in swagger example for apply rule endpoint 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12325
 2024-05-13 12:28:17 +02:00
Kjell Winblad
921d82f083 fix: JSON trace formatter should handle report style log entries 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12344
 2024-05-13 12:11:11 +02:00
zmstone
7faf73e0f4 chore(emqx_schema): change atom array to enum array for alarm.actions 2024-05-13 10:54:33 +02:00
zmstone
290ebe2fc5 fix: deny subscribing to +/# by default ACL 
Prior to this change, EMQX default ACL has a deny rule to reject
subscribing to `#`.
For completeness, the default ACL should also deny `+/#` because
they are essentially equivalent.
 2024-05-13 09:26:42 +02:00
JianBo He
54f6bd181b
Merge pull request #13010 from JimMoen/fix-jt808-reg-failed 
fix(gw): jt808 REG_ACK failed due to faulty clientinfo
 2024-05-13 14:19:18 +08:00
JimMoen
2f212f34a3
Merge pull request #12989 from JimMoen/fix-plugin-config-file-backup 
fix(plugin): add a backup for the plugin config file
 2024-05-11 16:38:36 +08:00
JimMoen
a5ecceb755
chore: apply suggestions from code review, thanks @thalesmg 2024-05-11 11:36:40 +08:00
Thales Macedo Garitezi
e25fe62cbc fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page 
Fixes https://emqx.atlassian.net/browse/EMQX-12294
 2024-05-10 14:48:46 -03:00
Kjell Winblad
c6df069c5a
Merge pull request #13012 from kjellwinblad/kjell/fix_incorrect_config_crash/EMQX-12315 
fix: listener crash if access_rules config option is incorrect
 2024-05-10 17:46:40 +02:00
Thales Macedo Garitezi
f690db9849 fix(postgres): reduce log spamming when connection goes down 
Fixes https://emqx.atlassian.net/browse/EMQX-12334

See also: https://github.com/emqx/epgsql/pull/10
 2024-05-10 12:15:01 -03:00
JimMoen
3c7e8a3bd5
test(gw): jt808 register failed 2024-05-10 23:09:22 +08:00
JimMoen
f8fd4d15be
fix(gw): process jt808 client deregister packet 2024-05-10 23:00:59 +08:00
Zaiming (Stone) Shi
c491b83857
Merge pull request #13014 from zmstone/0510-add-is_template-for-kinesis-partition-key 
fix(kinesis): partition_key should be of 'template' type
 2024-05-10 16:11:35 +02:00
Kjell Winblad
b13fa37771 fix: dialyzer warning 2024-05-10 15:51:28 +02:00
Thales Macedo Garitezi
6be4e6f631 Merge branch 'release-57' into sync-r57-m-20240510 2024-05-10 10:23:28 -03:00
Thales Macedo Garitezi
0eaef18391
Merge pull request #13005 from thalesmg/fix-aggreg-sup-tree-r57-20240509 
fix(aggregator): refactor supervision tree
 2024-05-10 10:22:33 -03:00
Kjell Winblad
2bed5894e3 test: add test cases for listeners access_rules validation and split 2024-05-10 14:21:43 +02:00
Thales Macedo Garitezi
3a29696a48 fix(aggregator): refactor supervision tree 
Instead of using a aggregator supervisor with a fixed local name, we should allow
specifying different names so each action app will spawn its own aggregator supervisor.
 2024-05-10 09:06:12 -03:00
zmstone
47df968d4a fix(kinesis): partition_key should be of 'template' type 2024-05-10 13:57:38 +02:00
firest
1fefb421bc feat(limiter): lift the log level of rate limiter to warning 2024-05-10 18:47:55 +08:00
Kjell Winblad
0aeb2cd77f fix: listener crash if access_rules config option is incorrect 
Previously when changing the access_rules configuration option of a
listener to something that was not a valid rule, the listener would
crash. This has now been fixed by the addition of a configuration
validator that checks the access_rules field.

Additionally, a configuration option converter has been added to the
access_rules field so that one can specify several rules in a single
string by using "," (comma) as separator.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12315
 2024-05-10 12:27:04 +02:00
JimMoen
1a6f0c9234
chore: bump gw_jt808 app vsn 2024-05-10 17:13:23 +08:00
JimMoen
b07242515f
fix(gw): jt808 REG_ACK failed due to faulty clientinfo 2024-05-10 17:12:28 +08:00
Ivan Dyachkov
900536712d
Merge pull request #12997 from thalesmg/sync-r57-m-20240508 
sync `release-57` to `master`
 2024-05-10 10:41:30 +02:00
Kjell Winblad
8c0b55a89e
Merge pull request #12992 from kjellwinblad/kjell/fix_rule_unescape_function/EMQX-12313 
fix: make rule engine unescape convert \a to the terminal alarm char
 2024-05-10 09:48:33 +02:00
lafirest
da7b11020a
Merge pull request #13001 from lafirest/fix/sysk_reconn 
fix(sysk): fix that the syskeeper forwarding never reconnecting
 2024-05-10 11:24:28 +08:00
firest
1f3b640a3d chore: update change 2024-05-10 07:29:14 +08:00
Thales Macedo Garitezi
7b80a9aa44 fix(aggregator): namespace aggregator ids with action type 
Otherwise, actions of different types but same names will clash when starting the
aggregator supervision tree.
 2024-05-09 16:02:11 -03:00
Andrew Mayorov
57dda70d2f
Merge pull request #12994 from keynslug/fix/sessds/log-levels 
fix(sessds): use milder log level for regular error conditions
 2024-05-09 18:34:54 +02:00
lafirest
23a1b18459
Merge pull request #12983 from emqx/feat/authn_hook 
feat(events): add new hook && event `client.check_authn_complete`
 2024-05-09 22:36:52 +08:00
Thales Macedo Garitezi
401f0fa84b Merge branch 'release-57' into sync-r57-m-20240508 2024-05-09 09:13:30 -03:00
firest
2bc014db69 fix(events): call client.check_authn_complete even if authentication fails 2024-05-09 18:15:32 +08:00
firest
e3a59c4037 fix(sysk): fix that the syskeeper forwarding never reconnecting 2024-05-09 16:27:58 +08:00
Ilia Averianov
a1aedee446
Merge pull request #12996 from savonarola/0508-fix-cursor-usage 
fix(retainer): fix qlc cursor cleanup
 2024-05-09 08:04:10 +03:00
firest
d2ca4e9f11 chore: update change 2024-05-09 11:04:56 +08:00
ieQu1
07aa708894
test(ds): Refactor replication suite 2024-05-09 03:56:56 +02:00
zhongwencool
107d00adee
Merge pull request #12944 from zhongwencool/mqtt-utf8-strict-mode 
fix: non-utf8 clientid connect event case crash if strict_mod=false
 2024-05-09 09:16:10 +08:00
ieQu1
63e51fca66
test(ds): Use streams to fill the storage 2024-05-09 02:46:57 +02:00
ieQu1
a0a3977043
feat(ds): Assign latest timestamp deterministically 2024-05-08 23:17:57 +02:00
ieQu1
2236af84ba
feat(ds): two-stage storage commit on the storage level 2024-05-08 23:17:57 +02:00
ieQu1
f250f00f3f
chore: Bump snabbkaffe version to 1.0.10 2024-05-08 23:17:57 +02:00
ieQu1
1ddbbca90e
feat(ds): Allow incremental update of the LTS trie 2024-05-08 23:17:57 +02:00
ieQu1
68ca891f41
test(ds): Use streams to create traffic 2024-05-08 23:17:57 +02:00
Zaiming (Stone) Shi
018d79b498
Merge pull request #12993 from zmstone/0508-fix-update-listener-zone-from-dashboard 
0508 fix update listener zone from dashboard
 2024-05-08 22:03:10 +02:00
Thales Macedo Garitezi
55f27281b3
Merge pull request #12991 from thalesmg/refactor-mv-message-type-r57-20240508 
refactor(message validation): rename `message_name` to `message_type`
 2024-05-08 16:58:28 -03:00
Thales Macedo Garitezi
d3905b6336
Merge pull request #12990 from thalesmg/fix-mv-conf-update-r57-20240508 
fix(message validation): don't call `emqx_conf:update` during boot
 2024-05-08 16:58:15 -03:00
Ilya Averyanov
57287f0722 fix(retainer): fix qlc cursor cleanup 2024-05-08 21:38:37 +03:00
Thales Macedo Garitezi
8d0574abf0
Merge pull request #12979 from thalesmg/fix-ps-monitor-topic-count-r57-20240506 
fix(monitor api): count persistent routes and subscriptions
 2024-05-08 15:09:16 -03:00
Andrew Mayorov
d84c180ccb
feat(dsrepl): avoid contacting unreachable ra servers 
Assuming estabilished Erlang distribution channel is a reliable way to
tell whether a remote node is reachable.
 2024-05-08 18:12:13 +02:00
Andrew Mayorov
3e956e8711
fix(sessds): use milder log level for regular error conditions 
Especially when such events are emitted in (potentially) tight loops,
i.e. pulling messages from iterators pointing to remote shards.
 2024-05-08 17:40:08 +02:00
zmstone
9edbad5459 fix(listener_api): do not allow update listener with unknown zone name 2024-05-08 16:37:55 +02:00
Thales Macedo Garitezi
d3bc32dc35 fix(message validation): don't call emqx_conf:update during boot 
`load/0` and `unload/0` shouldn't call cluster operations.
 2024-05-08 10:18:29 -03:00
Kjell Winblad
916168c755 fix: make rule engine unescape convert \a to the terminal alarm char 
The rule engine unescape function should convert the escape sequence \a
to the alarm bell symbol (ASCII 7). This bug was created as it was
assumed that Erlang convert $\a to 7 but Erlang does not handle the \a
escape sequence and instead convert it to 97 (ASCII code for a).

Fixes:
https://emqx.atlassian.net/browse/EMQX-12313
 2024-05-08 15:18:13 +02:00
Thales Macedo Garitezi
5172606032 refactor(message validation): rename message_name to message_type 
On Product's behest, to make naming consistent with protobuf nomenclature.

https://protobuf.dev/programming-guides/proto3/#adding-types
 2024-05-08 09:13:40 -03:00
zmstone
3a2ff34433 chore: add zone in listener config example 2024-05-08 11:43:10 +02:00
JimMoen
25be9c2890
fix(plugin): use binary NameVsn to re-build plugin avsc serde 2024-05-08 15:43:54 +08:00
JimMoen
68c601ad72
fix(plugin): add a backup for the plugin config file 2024-05-08 15:43:53 +08:00
JianBo He
4403b4f5ce
Merge pull request #12976 from HJianBo/fix-duplicated-disconnect 
Fix duplicated disconnect event
 2024-05-08 14:43:19 +08:00
Kjell Winblad
72f68afbca
Merge pull request #12981 from kjellwinblad/kjell/fixup_trace_all_http_and_error 
Make formatting of action result trace entries better
 2024-05-08 06:10:03 +02:00
Thales Macedo Garitezi
0c7b221c42 fix(clients mgmt api): set durable flag for DS sessions and subscriptions 2024-05-07 14:55:50 -03:00
Kjell Winblad
d5324e295f test: do cleanup in emqx_common_test_helpers:on_exit function 
Thanks @thalesmg for the suggestion.
 2024-05-07 18:18:18 +02:00
Thales Macedo Garitezi
02c58b67f0
Merge pull request #12986 from thalesmg/fix-mv-name-length-r57-20240507 
fix(message validation): limit the length of validation name
 2024-05-07 11:39:17 -03:00
Thales Macedo Garitezi
2989793f4c
Merge pull request #12980 from thalesmg/fix-ds-get-offline-client-stuff-r57-20240506 
fix(mgmt clients api): hold channel info after client disconnects for display in API
 2024-05-07 11:18:00 -03:00
Thales Macedo Garitezi
d4d2b4c460
Merge pull request #12985 from thalesmg/fix-mv-cluster-metrics-r57-20240507 
fix(message validation): apply config CRUD operations on whole cluster
 2024-05-07 11:10:58 -03:00
Kjell Winblad
09ee7ec0e2 fix(rule tracing): make sure that recoverable errors are traced 2024-05-07 15:49:38 +02:00
Thales Macedo Garitezi
96bb6e7d88 fix(message validation): limit the length of validation name 
Fixes https://emqx.atlassian.net/browse/EMQX-12299
 2024-05-07 10:04:56 -03:00
Thales Macedo Garitezi
06f16d4ee0 fix(message validation): apply config CRUD operations on whole cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-12298
 2024-05-07 09:45:50 -03:00
Thales Macedo Garitezi
6d6eb42fa3 fix(mgmt clients api): hold channel info after client disconnects for display in api 
Fixes https://emqx.atlassian.net/browse/EMQX-12266
 2024-05-07 09:22:33 -03:00
Serge Tupchii
b869488d35 fix(emqx_license): don't crash if an imported backup file has no license config 2024-05-07 13:48:04 +03:00
zhongwencool
a0bf06caba chore: add test case for non-utf8 topic 2024-05-07 17:02:45 +08:00
firest
f641d0b2b7 feat(events): add new hook && event client.check_authn_complete 2024-05-07 16:24:31 +08:00
Kjell Winblad
feecc36607 fix(rule tracing): clean up error tuple in the action_failed trace 2024-05-07 09:13:55 +02:00
JianBo He
1642b06bf9 test: add tests 2024-05-07 10:30:53 +08:00
Thales Macedo Garitezi
28355a4cfc fix(monitor api): count persistent routes and subscriptions 
Fixes https://emqx.atlassian.net/browse/EMQX-12267
 2024-05-06 16:51:31 -03:00
Zaiming (Stone) Shi
93e31d088a
Merge pull request #12978 from JimMoen/fix-typo 
chore: fix typo
 2024-05-06 21:28:51 +02:00
Kjell Winblad
ca88f5731b fix(rule tracing): format result traces in a more structured way 2024-05-06 17:33:59 +02:00
Thales Macedo Garitezi
cd4aaf663b docs: briefly describe delivery callbacks 2024-05-06 10:00:10 -03:00
Thales Macedo Garitezi
e6b1dfb095 refactor: improve error 2024-05-06 09:53:49 -03:00
Thales Macedo Garitezi
793cc4df2c fix: add fully qualified calls to enable hot beam reloads 2024-05-06 09:52:31 -03:00
Thales Macedo Garitezi
edcb750d9f fix: remove unused dependency 2024-05-06 09:51:32 -03:00
Thales Macedo Garitezi
36542a4b1d docs: bump copyright years 2024-05-06 09:50:12 -03:00
Ilia Averianov
e19222fc0d
Merge pull request #12971 from savonarola/0502-fix-url-parse 
fix(auth,http): improve URI handling
 2024-05-06 13:41:10 +03:00
JimMoen
1df2d837ee
chore: fix typo 2024-05-06 17:37:32 +08:00
ieQu1
3642bcd1b6
docs(ds): Fix comment for the builtin DS metrics 2024-05-06 11:21:32 +02:00
ieQu1
b2a633aca1
fix(ds): Use leader's clock for computing LTS safe cutoff time 2024-05-06 11:21:32 +02:00
ieQu1
1ff2e02fd9
feat(ds): Pass current time to the storage layer via argument 2024-05-06 11:21:32 +02:00
ieQu1
8ac9700aab
feat(ds): Add an API for DB-global variables 2024-05-06 11:21:32 +02:00
ieQu1
86d45522e3
fix(dsrepl): Don't reverse elements of batches 2024-05-06 11:21:32 +02:00
ieQu1
bcfa7b2209
fix(ds): Destroy LTS tries when the generation is dropped 2024-05-06 11:21:32 +02:00
ieQu1
9999ccd36c
feat(ds): Ignore safe cutoff time for streams without varying levels 2024-05-06 11:21:32 +02:00
Ilya Averyanov
31026d51f7 chore(auth,http): cache REs for parsing URIs 2024-05-06 11:46:18 +03:00
JianBo He
b320b20c7b fix(mqtt): disconnected event should not be sent twice 2024-05-06 16:17:02 +08:00
JianBo He
cabe2ae100 chore: fix dialyzer warning 2024-05-06 14:58:27 +08:00
JianBo He
d2b6e41cd1 chore(stomp): parse \n as heartbeat frame 2024-05-06 14:39:41 +08:00
Ilya Averyanov
3b655f56cb fix(auth,http): improve URI handling 2024-05-04 09:47:13 +03:00
Zaiming (Stone) Shi
e7f0c83406
Merge pull request #12970 from zmstone/0503-refactor-dashboard-listener 
refactor: simplify https listener config for dashboard
 2024-05-04 08:38:29 +02:00
Zaiming (Stone) Shi
c5d8659f35
Merge pull request #12972 from zmstone/0503-unhide-default-dashboard-username-password 
chore: unhide dashboard default_password config
 2024-05-04 08:36:32 +02:00
Thales Macedo Garitezi
32493e395c refactor: avoid passing record down to template rendering function 2024-05-03 17:43:30 -03:00
Thales Macedo Garitezi
157e2c2535 refactor: make api more generic 2024-05-03 17:25:02 -03:00
Thales Macedo Garitezi
a6508e2fca refactor: use better names 
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
 2024-05-03 17:04:09 -03:00
Thales Macedo Garitezi
d61f4078e2 refactor: extract blob aggregator logic to separate application 
This will allow other bridges to share the same aggregation logic as S3 aggregated action.
 2024-05-03 14:35:58 -03:00
zmstone
4d3156b5ed chore: unhide dashboard default_username and default_password config 2024-05-03 18:35:49 +02:00
Thales Macedo Garitezi
3818b75188
Merge pull request #12959 from thalesmg/kprodu-connector-hc-m-20240429 
feat(kafka producer): add health check topic option
 2024-05-03 12:48:48 -03:00
zmstone
3c5c76fcfc refactor: simplify https listener config for dashboard 2024-05-03 16:45:40 +02:00
Kjell Winblad
004dc80fb2
Merge pull request #12958 from kjellwinblad/kjell/fixup_trace 
fix: rule trace formatting, republish and console stop after rendering
 2024-05-03 16:24:17 +02:00
Zaiming (Stone) Shi
52357742aa
Merge pull request #12962 from zmstone/0502-fix-ssl-client-opts-to-allow-wildcard-cert-by-default 
fix(ssl-clients): allow wildcard certificates by default
 2024-05-03 15:19:37 +02:00
Kjell Winblad
5bfe31b691 fix: issues found during PR review (thanks @thalesmg and @zmstone) 
* Simpler handling of true and false in best effort JSON formatter
* inet:ntoa/1 to format IP addresses
* Made a record for lazy formatted trace values and formatter to improve
  maintainability
* Added callback to format return value from connector
  * Extended test case to check that the format return value callback
    works
* Added handling of "lazy" trace entry data to the text formatter. Do we
  need to handle this data in the normal log formatters as well?
 2024-05-03 15:04:57 +02:00
Kjell Winblad
e32745bca6 test(apply rule trace): add test case for republish and console print 2024-05-03 15:02:37 +02:00
Kjell Winblad
b9cd75eef0 test(emqx_mgmt_api_trace_SUITE): fix bad test cases 2024-05-03 15:02:37 +02:00
Kjell Winblad
76ccef7ce3 fix(trace api): trace should be ready after create 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12276
 2024-05-03 15:02:37 +02:00
Kjell Winblad
9576efb716 fix(rule apply test): do not crash when the rule is not found 2024-05-03 15:02:37 +02:00
Kjell Winblad
3ba5cb7858 fix(apply rule): make sure we return something JSON compatible 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12275
 2024-05-03 15:02:37 +02:00
Kjell Winblad
ea7633c484 fix: rule trace formatting, republish and console stop after rendering 
* Better rule trace formatting for many trace entries
* The republish and console actions have got working stop after
  rendering functionality
 2024-05-03 15:02:36 +02:00
zmstone
37d66e90fb fix(ssl-clients): allow wildcard certificates by default 2024-05-03 12:52:42 +02:00
Ilia Averianov
3ed385201c
Merge pull request #12947 from savonarola/0426-auth-timeout 
feat(authn): add connection expire based on authn data
 2024-05-03 11:32:18 +03:00
Thales Macedo Garitezi
6f3da6b131 feat(kafka producer): add health check topic option 
Fixes https://emqx.atlassian.net/browse/EMQX-12241

This allows more accurate health checking for Kafka Producers.  Without a topic, it's not
possible to actually probe the connection to partition leaders, so the connector might not
be reported as `disconnected` without testing a concrete topic.
 2024-05-02 17:07:44 -03:00
Thales Macedo Garitezi
c71f73924b
Merge pull request #12961 from thalesmg/kconsu-custom-group-id-m-20240430 
feat(kafka consumer): allow custom group id
 2024-05-02 14:37:42 -03:00
Ivan Dyachkov
cc577e636d Merge remote-tracking branch 'upstream/release-57' into 0502-sync-release-57 2024-05-02 17:06:18 +02:00
Ivan Dyachkov
3064a1cbae
Merge pull request #12957 from id/0430-adjust-build-scripts 
adjust build scripts
 2024-05-02 17:05:06 +02:00
Thales Macedo Garitezi
eb113fa578 fix: add non-empty validator 2024-05-02 11:19:00 -03:00
William Yang
8f780ae8bc
Merge pull request #12955 from qzhuyan/port/william/5/partial-chain 
port: TLS partial chain
 2024-05-02 15:34:25 +02:00
Thales Macedo Garitezi
607bf0fe44
Merge pull request #12952 from thalesmg/fix-dont-list-mv-rule-event-r57-20240429 
fix(rule events): don't list validation failure event in CE API
 2024-05-02 09:20:53 -03:00
Thales Macedo Garitezi
3942b371d7 feat(kafka consumer): allow custom group id 
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
 2024-05-02 08:59:40 -03:00
Ivan Dyachkov
035b8480b0 chore(emqx_conf): extract data dirs to sync to a macro 2024-05-02 10:25:53 +02:00
William Yang
1a4a4bb3a5 chore: fix nit 2024-05-02 10:13:57 +02:00
Thales Macedo Garitezi
14ef0b1e51 feat(kafka consumer): allow custom group id 
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
 2024-05-01 13:53:29 -03:00
Thales Macedo Garitezi
42cb17360e Merge branch 'release-57' into sync-r57-m-20240430 2024-04-30 14:42:22 -03:00
Thales Macedo Garitezi
7a44746492 fix(rule events): don't list validation failures in CE 
Fixes https://emqx.atlassian.net/browse/EMQX-12284
 2024-04-30 14:41:09 -03:00
Thales Macedo Garitezi
874f1f2428
Merge pull request #12950 from thalesmg/fix-mv-duplicated-topics-r57-20240429 
fix(message validation): validate duplicated topics
 2024-04-30 13:34:58 -03:00
Thales Macedo Garitezi
dd6566f3c5
Merge pull request #12948 from thalesmg/fix-http-bridge-header-update-obfuscate-r57-20240429 
fix(http connector): deobfuscate sensitive headers
 2024-04-30 13:34:15 -03:00
Ilya Averyanov
e4154dd472 feat(authn): use correct time resolution for setting channel expire in JWT authn 2024-04-30 19:01:16 +03:00
William Yang
fb30207ef3 chore: fix test 2024-04-30 16:41:46 +02:00
William Yang
337c230e79 feat(partial_chain): gateway support 2024-04-30 16:41:26 +02:00
Ilya Averyanov
80d724c504 feat(authn): add connection expire based on authn data 2024-04-30 17:04:55 +03:00
Andrew Mayorov
279169105b
Merge pull request #12934 from keynslug/feat/EMQX-12204/aggreg-s3-bridge 
feat(s3-bridge): implement aggregated upload action
 2024-04-30 13:50:19 +02:00
zmstone
1974ec15ec fix(client_attrs): fix client_attrs extraction loop 2024-04-30 12:12:35 +02:00
Zaiming (Stone) Shi
a88862ebe0
Merge pull request #12951 from zmstone/0429-fix-description-schema 
fix(schema): description should be in binary() type
 2024-04-30 12:03:55 +02:00
Andrew Mayorov
6d3add3646
fix(s3-aggreg): do not handle {error, closed} on buffer write 
Because it's not really something `file:write/2` is supposed to return.
 2024-04-30 10:48:16 +02:00
Andrew Mayorov
4bea938ef2
fix(s3-csv): clarify naming of CSV container concepts 
Co-Authored-By: Ilya Averyanov <av@rubybox.dev>
 2024-04-30 10:48:16 +02:00
Andrew Mayorov
83366cbed0
fix(s3-aggreg): ensure action works in Rule SQL contexts 2024-04-30 10:48:16 +02:00
Andrew Mayorov
a1a313d992
fix(s3-aggeg): apply CSV column order setting consistently 
Otherwise, columns that are part of column order could appear and
disappear from consecutive uploads, depending on if they are part
of the very first buffered event or not.
 2024-04-30 10:48:16 +02:00
Andrew Mayorov
f6e5eea4f7
feat(s3-aggreg): handle delivery shutdowns gracefully 2024-04-30 10:48:15 +02:00
Andrew Mayorov
339036045d
feat(s3-aggreg): support custom and default S3 object HTTP headers 
I.e. configured container decides default `Content-Type` header.
 2024-04-30 10:48:15 +02:00
Andrew Mayorov
5b15b2d641
docs(s3-bridge): document aggregated upload action schema 2024-04-30 10:48:15 +02:00
Andrew Mayorov
ccbcc0c4e3
feat(s3-bridge): implement aggregated upload action 2024-04-30 10:48:15 +02:00
Andrew Mayorov
b91ff97170
feat(s3): separate streaming upload logic into dedicated module 
And use it in `emqx_s3_uploader`, while also turning it into a simple
gen_server.
 2024-04-30 10:48:12 +02:00
Kjell Winblad
7b638a5829 style: fix spelling mistake (thanks @savonarola) 2024-04-30 09:28:12 +02:00
William Yang
70ffd77f99 chore(TLS-chain-test): update for OTP 26 2024-04-30 09:05:51 +02:00
zmstone
c8d6976b14 feat: add conditions to variform expressions 
- refactored `coalesce` function to allow lazy evaluation
- added `iif(Cond, IfExpr, EleseExpr)` to allow simple conditions
 2024-04-30 07:28:27 +02:00
William Yang
650cf4b27e test(partial_chain): update tcs for OTP-25 2024-04-30 06:25:03 +02:00
William Yang
43ad665dcf fix(test): tls_verify_partial_chain 2024-04-30 06:25:03 +02:00
William Yang
a29a43e5fc fix(listener): remove partial_chain in wss opts 2024-04-30 06:25:03 +02:00
William Yang
4e9c1ec0c9 chore: happy elvis 2024-04-30 06:25:03 +02:00
William Yang
8eb463c58d feat(tls): update schema for TLS keyusage 2024-04-30 06:25:03 +02:00
William Yang
90430fa66d fix(tls): undefined keyusage 2024-04-30 06:25:03 +02:00
William Yang
eb1ab9adfe test(tls): verify peer keyusage 2024-04-30 06:25:03 +02:00
William Yang
8bc3a86f63 feat(config): partial_chain 2024-04-30 06:25:03 +02:00
William Yang
fa4357ce89 test: port listener tls partial_chain 2024-04-30 06:25:03 +02:00
William Yang
0b95a08d32 feat(tls): port partial_chain, part 1 2024-04-30 06:25:03 +02:00
zmstone
9917293fd0 fix(schema): description should be in binary() type 2024-04-29 21:44:55 +02:00
Thales Macedo Garitezi
ffedce014f fix(message validation): validate duplicated topics 
Fixes https://emqx.atlassian.net/browse/EMQX-12254
 2024-04-29 15:44:03 -03:00
Thales Macedo Garitezi
a847389159 fix(http connector): deobfuscate sensitive headers 
Fixes https://emqx.atlassian.net/browse/EMQX-12213
 2024-04-29 15:20:46 -03:00
Thales Macedo Garitezi
5cf92dcb73 refactor: use spawn_link instead of spawn_monitor 
This should cover the case when the resource manager is brutally killed.
 2024-04-29 15:20:32 -03:00
Thales Macedo Garitezi
475077c798 fix(resource): account for ongoing channel health checks, update data and reply immediately when receiving an update 2024-04-29 15:20:32 -03:00
Thales Macedo Garitezi
79526d539a fix(resource manager): clean up any running health checks when terminating 
Fixes https://github.com/emqx/emqx/pull/12812#discussion_r1555564254
 2024-04-29 15:20:32 -03:00
zmstone
7a05a4754f docs: expose zone config in schema doc 2024-04-29 13:57:18 +02:00
Zaiming (Stone) Shi
a41652ec31
Merge pull request #12940 from zmstone/0427-catch-hocon-syntax-error 
feat(mgmt): add ignore_readonly to  configs API
 2024-04-29 13:22:12 +02:00
zmstone
1db932df21 chore(mgmt): PUT /configs?ignore_readonly=true, lower log to info level 2024-04-29 10:20:47 +02:00
zhongwencool
ef9a63ae17 fix: non-utf8 clientid connect event case crash if strict_mod=false 2024-04-29 14:43:29 +08:00
zmstone
07cbdc6e90 feat(mgmt): add ignore_readonly qeury-string to PUT /configs API 2024-04-28 14:45:53 +02:00
zmstone
10625eacac chore: upgrade to hocon-0.42.2 
hocon pretty-print quotes more string values
if a string has '.' or '-', or if it starts with a digit 0-9,
then it's quoted.

see details here: https://github.com/emqx/hocon/pull/293
 2024-04-28 14:05:30 +02:00
firest
d6c203b4fd fix(dynamo): fixed the keys checking for Dynamo 2024-04-28 16:59:01 +08:00
zmstone
5f4215b333 fix(mgmt): avoid 500 error when hocon syntax error 2024-04-27 21:12:42 +02:00
JimMoen
43ac4f5dfe
fix: make bpapi check happy 2024-04-26 21:17:01 +08:00
JimMoen
5ff4e76904
refactor: rename plugins config api functions 2024-04-26 20:51:20 +08:00
Kjell Winblad
db48a977bb test(mongodb authn, authz): add test cases for use_legacy_protocol 
This commit adds test cases that check that the authn and authz modules
for MongoDB support the use_legacy_protocol configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12245
 2024-04-26 12:41:50 +02:00
JimMoen
00cab33fde
fix: plugin's internal config api 2024-04-26 18:10:55 +08:00
Ilya Averyanov
aaf57ecfbc chore(authz): improve and clarify types 2024-04-26 12:09:18 +03:00
JimMoen
11389bc086
fix: i18n file renamed 2024-04-26 11:12:24 +08:00
JimMoen
28e8131984
refactor: avoid make when do serde 2024-04-26 10:37:15 +08:00
JimMoen
e5bd747b32
refactor: read avsc file when make serde 2024-04-26 10:25:19 +08:00
JimMoen
670ddae57c
chore: fix typo 2024-04-26 10:25:18 +08:00
JimMoen
c884dfb451
test: make eunit happy 2024-04-26 10:25:17 +08:00
JimMoen
1869f6fd0a
fix: enusre plugin installed when do config operation 2024-04-26 10:25:16 +08:00
JimMoen
b0aa3bb70f
fix(plugin): get plugin config api 2024-04-26 10:25:15 +08:00
JimMoen
f343cd2021
fix: get plugin config in json 2024-04-26 10:25:14 +08:00
JimMoen
d2ecccc2ff
fix: call json encoder/decoder for plugin config 2024-04-26 10:25:12 +08:00
JimMoen
c180b6a417
fix(api): plugin api docs 2024-04-26 10:25:11 +08:00
JimMoen
1f00ce789f
fix(plugin): gen_server call timeout infinity 2024-04-26 10:25:10 +08:00
JimMoen
c0429ca333
fix(plugin): refine schema serde log 2024-04-26 10:25:09 +08:00
JimMoen
27d1f91cac
refactor: refine function name 2024-04-26 10:25:08 +08:00