Commit Graph

12500 Commits

Author SHA1 Message Date
Andrew Mayorov e6c5c1b598
chore(dsrepl): provide more information in rebalancing log messages 2024-05-22 17:24:08 +02:00
Andrew Mayorov c355c9ad50
fix(dsrepl): properly handle transaction abort during forget site 2024-05-22 17:22:55 +02:00
Kjell Winblad cff8b97e8a fix: handle channel updated during health check
This commit fixes an issue found by CI test case
emqx_bridge_influxdb_SUITE:t_start_stop and others. While the channel
health check process is running, the channel could be removed or updated
which could cause a crash in the resource manager or non up-to-date
alarms being triggered.
2024-05-22 12:15:04 +02:00
Kjell Winblad 39d758c4d6 fix: do not return configs for channels from emqx_resource_manager 2024-05-22 12:15:04 +02:00
Kjell Winblad 917474f694 fix: action config update would sometimes not be reflected in connector
Before this commit the following happened sometimes:

1. action status is connected
2. action config is updated to something that should change the status to
   disconnected
3. action status is still connected and the old config is being used by
   the connector even though the config has been correctly updated.

The reason for this bug is that the post_config_hook runs before the
global EMQX config is updated. The post config hook is adding the new
config to the connector. Since the new config causes the action to get
status disconnected, the adding of the action to the connector is
retried when the health check runs but this time the config will be
loaded from the global config which could cause the old config to be
loaded (this happens when the global config has not had time to get
updated).

The above problem has been fixed in this commit by only reading action
configs from the global config when the connector starts/restarts and
instead store the latest configs for the actions in the connector.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12376
2024-05-22 12:15:04 +02:00
firest f3e8037e0f fix(rocketmq): fix namespace error for RocketMQ 2024-05-22 16:29:38 +08:00
Andrew Mayorov a3cd3e31b1
Merge pull request #13084 from keynslug/fix/EMQX-12422/part-size
fix(s3-bridge): report inconsistent `min/max_part_size` option
2024-05-22 09:48:09 +02:00
JimMoen e5f7aa9817
refactor: plguin functions and types rename 2024-05-22 11:51:54 +08:00
JimMoen 33aa61daea
fix: use hocon format as plugin config 2024-05-22 06:37:58 +08:00
JimMoen 14f2a68799
fix: bpapi spec type 2024-05-22 06:01:57 +08:00
Thales Macedo Garitezi 31a35f2a15
Merge pull request #13076 from thalesmg/fix-ds-session-count-client-api-r57-20240520
fix(client mgmt api): cache disconnected durable session count for `/clients` api
2024-05-21 15:34:21 -03:00
JimMoen 5abd23af5a
test: plugin refactor 2024-05-22 02:09:37 +08:00
JimMoen 87b3b214b9
fix: make static_check happy 2024-05-22 02:09:36 +08:00
JimMoen 3ce091e9d7
fix: ensure plugin tarball and extracted 2024-05-22 02:09:35 +08:00
JimMoen 1abc8cf502
fix: ensure plugin's config on boot and join cluster 2024-05-22 02:09:34 +08:00
JimMoen 677352e498
fix: ensure installed and plugin state on boot 2024-05-22 02:09:33 +08:00
JimMoen 2076681e76
fix: put plugin config with binary namevsn key 2024-05-22 02:09:32 +08:00
JimMoen 140b7ce51e
fix(plugin): schema content only provided in enterprise edition 2024-05-22 02:09:31 +08:00
JimMoen a7f2f95318
fix: ensure avro file 2024-05-22 02:09:30 +08:00
JimMoen df7dcb2764
fix: do not let plugin start failed lead emqx start failed 2024-05-22 02:09:29 +08:00
JimMoen e0e4517d9e
fix: ensure plugin config on boot 2024-05-22 02:09:28 +08:00
JimMoen e8d4e88118
fix(plugin): serde not found on new-joined nodes 2024-05-22 02:09:26 +08:00
JimMoen 4ad0743f61
Merge pull request #13081 from JimMoen/fix-typo
chore: fix typos
2024-05-22 02:05:20 +08:00
Andrew Mayorov 28f6eb0200
fix(s3-bridge): report inconsistent `min/max_part_size` option 2024-05-21 18:49:01 +02:00
SergeTupchiy 528d6b7ef1
Merge pull request #13082 from SergeTupchiy/routes-fixes
Minor routes fixes
2024-05-21 19:11:47 +03:00
Serge Tupchii 38e57e321f fix(ds_router): return correct type from lookup_routes/1, fix fun specs 2024-05-21 16:31:31 +03:00
Serge Tupchii 0db6b8dc54 fix(emqx_router): return correct type from lookup_routes/1 (v2 schema) 2024-05-21 16:28:59 +03:00
ieQu1 acb19a06cf Merge remote-tracking branch 'origin/release-57' 2024-05-21 15:04:46 +02:00
Thales Macedo Garitezi 7c5cb1acc5 fix(monitor current api): expose `disconnected_durable_sessions` count
Fixes https://emqx.atlassian.net/browse/EMQX-12423
2024-05-21 10:03:09 -03:00
JimMoen bb3c66638c
chore: fix typos 2024-05-21 17:45:20 +08:00
JianBo He 4f341bef8b
Merge pull request #12920 from cisiqo/patch-6
fix: STOMP heartbeat
2024-05-21 16:57:58 +08:00
Thales Macedo Garitezi 8b4a1c3d75 fix(client mgmt api): cache disconnected durable session count for `/clients` api
Fixes https://emqx.atlassian.net/browse/EMQX-12396
2024-05-20 11:48:08 -03:00
Kjell Winblad 8279d8c787
Merge pull request #13053 from kjellwinblad/kjell/fix_rule_trace_issues/EMQX-12327/EMQX-12335/EMQX-12336
Fix small action trace issues
2024-05-20 16:25:38 +02:00
Thales Macedo Garitezi a5d973b681 docs: hide `/clients_v2` from swagger api docs
Since it's not yet ready for production, we'll hide it from the API docs.
2024-05-20 10:00:43 -03:00
Kjell Winblad 3c87bcde46 fix(rule trace): restore logger metadata to its previous value 2024-05-20 10:57:15 +02:00
zmstone 6940930c3e chore: bump emqx_auth_http app vsn 2024-05-19 09:30:39 +02:00
zmstone 331d44a78a chore: improve Kafka producer health-check error logs
The Kafka producer lib wolff returns detailed error logs
about which host:port had issue, EMQX should log them to help
troubleshooting
2024-05-18 19:12:51 +02:00
Zaiming (Stone) Shi c6ee37bacd
Merge pull request #13068 from zmstone/0517-fix-dynamic-license-usage-for-dashboard
fix(license): show dynamic max_connections for bc
2024-05-18 09:49:18 +02:00
Andrew Mayorov fc3f525a67
Merge pull request #13066 from keynslug/fix/s3-bridge/desc-html
chore(s3-bridge): decrease default aggregation settings
2024-05-17 19:43:19 +02:00
zmstone b3afff9638 fix(license): show dynamic max_connections for bc 2024-05-17 17:44:21 +02:00
Thales Macedo Garitezi 388e6c8262 fix(dssubs): introduce separate gauge for subscriptions from durable sessions
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-17 10:17:16 -03:00
Kjell Winblad 5ce095f30e fix(rule trace): only include stop_after_rendering when value is true 2024-05-17 14:53:14 +02:00
Andrew Mayorov 385ac268d7
chore(s3-bridge): decrease default aggreg interval to 30 minutes 2024-05-17 14:49:18 +02:00
Andrew Mayorov c348f759af
chore(s3-bridge): decrease default aggreg limit to 100K records 2024-05-17 14:19:59 +02:00
Kjell Winblad 2e6db85578 fix(rule trace): rename rule_trigger_time(s) and cleaups
This commit renames trace fields rule_trigger_time and
rule_trigger_times to rule_trigger_ts and makes sure that the value for
rule_trigger_ts will always be a list of timestamps.
2024-05-17 14:13:36 +02:00
Andrew Mayorov 4d1db9f847
Merge pull request #13059 from keynslug/feat/EMQX-12204/single-action-schema
feat(s3-bridge): meld 2 separate actions into a single one
2024-05-17 13:27:40 +02:00
Kjell Winblad 2209b26fa5 fix(rule trace): do not leak trace meta data when tracing action result
Fixes:
https://emqx.atlassian.net/browse/EMQX-12391
2024-05-17 10:05:33 +02:00
ieQu1 5ba71ae6f7
Merge pull request #13062 from ieQu1/dev/ds-better-naming
fix(sessds): Rename last_alive_update_interval -> heartbeat_interval
2024-05-17 09:55:40 +02:00
Andrew Mayorov 39c2433afb
fix(s3-bridge): update config typespecs 2024-05-17 09:47:16 +02:00
Andrew Mayorov 86f99959b0
Merge pull request #13054 from keynslug/fix/EMQX-12365/node-leave
fix(dsrepl): anticipate and handle nodes leaving the cluster
2024-05-17 09:43:15 +02:00
Andrew Mayorov 5b2c2b336c
feat(s3-bridge): meld 2 separate actions into a single one
Discriminated through `mode` parameter. Also rename "simple" action to
a "direct" upload mode.
2024-05-17 09:34:21 +02:00
zhongwencool b4eb910cb1 chore: improved InfluxDB connection error messaging 2024-05-17 10:38:11 +08:00
zhongwencool 1ac7aa151e
Merge pull request #13048 from thalesmg/sv-backup-r57-20240514
feat(schema validation): implement backup restore
2024-05-17 09:47:04 +08:00
ieQu1 6f30c1c541
fix(sessds): Rename last_alive_update_interval -> heartbeat_interval 2024-05-17 01:20:10 +02:00
ieQu1 73f17249e9
fix(sessds): Rename session_persistence to durable_sessions 2024-05-16 21:31:32 +02:00
ieQu1 ee6e7174cf
fix(sessds): Rename the durable messages DB to `messages` 2024-05-16 21:31:32 +02:00
Andrew Mayorov 5157e61418
fix(dsrepl): verify if shards already allocated first 2024-05-16 18:56:54 +02:00
Andrew Mayorov 0119728d45
feat(dsrepl): also reflect pending transitions in ds status 2024-05-16 18:56:21 +02:00
Andrew Mayorov 26c4a4f597
feat(dsrepl): reflect conflicts and inconsistencies in ds status 2024-05-16 18:32:08 +02:00
Andrew Mayorov 7e86e3e61c
fix(dsrepl): anticipate and handle nodes leaving the cluster
Also make `claim_site/2` safer by refusing to claim a site for a node
that is already there.
2024-05-16 18:32:07 +02:00
Kjell Winblad ed16e678be fix(rule apply test): add default values to context
Fixes:
https://emqx.atlassian.net/browse/EMQX-12378
2024-05-16 17:31:44 +02:00
Kjell Winblad 520e91c8fd fix(greptime trace): remove async info as it is confusing
Fixes:
https://emqx.atlassian.net/browse/EMQX-12385
2024-05-16 16:50:09 +02:00
Ilia Averianov 22d5f17de8
Merge pull request #13025 from savonarola/0513-fix-cursor-usage
Get rid of qlc in retainer
2024-05-16 17:06:53 +03:00
Kjell Winblad b7c2f4a6d7 fix(best_effor_json): make tuple list check more efficient 2024-05-16 15:40:38 +02:00
Ilya Averyanov 1a664c941b chore(retainer): scan table in batches, improve stream usage
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-05-16 16:06:11 +03:00
Kjell Winblad ba74135079
Merge pull request #13052 from kjellwinblad/kjell/evaluate_lazy_trace_values_in_normal_log_formatters/EMQX-12380
fix: evaluate lazy values in normal log formatters when level is debug
2024-05-16 15:02:53 +02:00
Thales Macedo Garitezi 34a29e6363 feat(schema validation): implement support for `ctl conf load` 2024-05-16 09:55:42 -03:00
Ilia Averianov 322989c83f
Merge pull request #13040 from savonarola/0513-fix-http-authn-error-handling
fix(auth_http): fix query encoding
2024-05-16 15:12:57 +03:00
Zaiming (Stone) Shi 0be1249bbe
Merge pull request #13041 from zmstone/0513-http-authn-header-missing-should-result-in-meaningful-error-message
0513 http authn header missing should result in meaningful error message
2024-05-16 13:17:20 +02:00
ieQu1 71b393be89
Merge pull request #13057 from ieQu1/dev/default-n-shards
fix(ds): Change the default number of shards to 12
2024-05-16 13:05:34 +02:00
Kjell Winblad 413ad60bdb fix(cassandra): format rendered trace in a better way
Fixes:
https://emqx.atlassian.net/browse/EMQX-12393
2024-05-16 11:35:12 +02:00
Kjell Winblad 9fd8e930be fix(best_effort_json): only do tuple list to map without losing pairs 2024-05-16 11:19:48 +02:00
Ilya Averyanov e1ce6377f3 chore(streams): add stream methods, optimize streams 2024-05-16 12:04:08 +03:00
Ilia Averianov 1f7e358e1e
Merge pull request #13006 from savonarola/0509-fix-retainer-banned
fix(banned): verify delayed/taken over/retained messages against clientid_re ban rules
2024-05-16 11:37:15 +03:00
zmstone 2acde5a4e4 fix(authn/http): log meaningful error message if http header is missing 2024-05-16 11:36:52 +03:00
zmstone 4fb484d4cf chore(emqx_schema): change atom array to enum array for alarm.actions 2024-05-16 11:36:52 +03:00
Ilya Averyanov daf2e5a444 chore(auth_http): unify http request generation
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-05-16 11:36:52 +03:00
Kjell Winblad cf5b464777
Merge pull request #12937 from kjellwinblad/kjell/add_use_legacy_protocol_option_to_mongodb_auth/EMQX-12245
test(mongodb authn, authz): add test cases for use_legacy_protocol
2024-05-16 10:10:17 +02:00
Kjell Winblad 72b0c63ed2
Merge pull request #13043 from kjellwinblad/kjell/kafka_incorrect_query_type/EMQX-12363
fix(kafka trace): do not include query type in trace entry
2024-05-16 10:08:16 +02:00
ieQu1 deffb01544
fix(ds): Change the default number of shards to 12 2024-05-16 08:06:07 +02:00
Ilya Averyanov 552b62236c chore(retainer): move filter/foreach to emqx_utils_stream 2024-05-15 19:09:06 +03:00
Ilya Averyanov 0b39aaadbd chore(retainer): get rid of qlc usage 2024-05-15 19:09:06 +03:00
Thales Macedo Garitezi 2eb1bb2b13
Merge pull request #13032 from kjellwinblad/kjell/no_ctx_message_validation_failed/EMQX-12354
fix: add handling of message_validation_failed ctx for rule testing
2024-05-15 11:20:18 -03:00
Kjell Winblad 246bce85ec fix: make mysql action rendered trace include parameters
When doing non-batch inserts with the MySQL action, prepared statements
are used. We therefore need to include the parameters to the prepared
statement when tracing the rendered templates.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12335
2024-05-15 14:49:33 +02:00
Kjell Winblad c72682d81b fix: remove query mode from redis action trace as it only supports sync
Fixes:
https://emqx.atlassian.net/browse/EMQX-12336
2024-05-15 14:32:15 +02:00
Ilya Averyanov 78a87ab5a6 fix(banned): verify delayed/taken over/retained messages against clientid_re rules 2024-05-15 14:05:38 +03:00
Kjell Winblad 8cbfc4c0cd fix: evaluate lazy values in normal log formatters when level is debug
Fixes:
https://emqx.atlassian.net/browse/EMQX-12380
2024-05-15 11:47:21 +02:00
lafirest faf3492e58
Merge pull request #13039 from lafirest/fix/rocket_key
fix(rocketmq): fix a typo and a RocketMQ encoding error
2024-05-15 15:28:35 +08:00
Andrew Mayorov 3e1e4bab17
Merge pull request #13047 from keynslug/fix/EMQX-12366/api-ds-leave
feat(api-ds): provide more information on nonexistent site leave
2024-05-14 23:21:07 +02:00
Thales Macedo Garitezi bcbeadd4af test: stabilize flaky test 2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi f062517ccb test: stabilize flaky test 2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi b0aa7f25aa test: add more context to help debug flaky test
Ex: https://github.com/emqx/emqx/actions/runs/9084293110/job/24967912514?pr=13050#step:6:18330
2024-05-14 17:53:29 -03:00
Thales Macedo Garitezi fd949240c0 feat(schema validation): implement backup restore
Fixes https://emqx.atlassian.net/browse/EMQX-12346
2024-05-14 15:58:19 -03:00
Thales Macedo Garitezi bc5071590b
Merge pull request #13033 from thalesmg/fix-ds-more-client-api-fields-r57-20240513
fix(ds clients mgmt api): add more fields to disconnected durable sessions
2024-05-14 15:29:40 -03:00
Thales Macedo Garitezi 7023e6ad96
Merge pull request #13017 from thalesmg/fix-ds-subs-pages-r57-20240510
fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page
2024-05-14 15:29:29 -03:00
Thales Macedo Garitezi 2a0ac34656 Merge branch 'release-57' into sync-r57-m-20240514 2024-05-14 15:03:04 -03:00
Kjell Winblad e811f4a392 fix: add handling of message_validation_failed ctx for rule testing
Before this commit, the backend could not handle the test context for
message_validation_failed events that could be sent from the dashboard
since no handling code for this contexts existed in the backend. This
has now been fixed by adding such handling code.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12354
2024-05-14 13:07:32 -03:00
Thales Macedo Garitezi 1730a41337
Merge pull request #13046 from thalesmg/refactor-mv-rename-sv-r57-20240514
refactor: rename `message_validation` to `schema_validation`
2024-05-14 13:03:12 -03:00
Kjell Winblad adbc392265
Merge pull request #13027 from kjellwinblad/kjell/fix_trace_json_report_formatting/EMQX-12344
fix: JSON trace formatter should handle report style log entries
2024-05-14 18:02:59 +02:00
firest 437a0c0d55 fix(rocketmq): fix a typo and a RocketMQ encoding error 2024-05-14 23:01:31 +08:00
Andrew Mayorov 2f90d15ce6
fix(ds-api): adapt CLI to API changes 2024-05-14 16:55:17 +02:00
Andrew Mayorov 35e360fcbe
feat(api-ds): provide more information on nonexistent site leave 2024-05-14 15:57:41 +02:00
Thales Macedo Garitezi 257dae5211 refactor: rename `message_validation` to `schema_validation`
Request from Product team.

Fixes https://emqx.atlassian.net/browse/EMQX-12383
2024-05-14 10:32:27 -03:00
lafirest dc391d2403
Merge pull request #13009 from lafirest/fix/rate_log
feat(limiter): lift the log level of rate limiter to `warning`
2024-05-14 21:11:26 +08:00
Thales Macedo Garitezi 15acd86b31
Merge pull request #13034 from thalesmg/test-flaky-mq-r57-20240513
test(client mgmt api): attempt to fix flaky test
2024-05-14 09:01:29 -03:00
Thales Macedo Garitezi e8d2c48d3c
Merge pull request #13018 from thalesmg/fix-postgres-log-spam-r57-20240510
fix(postgres): reduce log spamming when connection goes down
2024-05-14 09:00:36 -03:00
Kjell Winblad f75c7a5cea fix(trace log entry error): better structure and naming 2024-05-14 13:38:41 +02:00
Andrew Mayorov 74ed0f578a
Merge pull request #13042 from keynslug/fix/EMQX-12204/api-ns
fix(s3-aggreg): disambiguate action schema namespace
2024-05-14 13:12:34 +02:00
Kjell Winblad 121084b2ff fix(kafka trace): do not include query type in trace entry
For Kafka that has its internal buffering the on_query_async callback is
used both for sync and async queries (for sync queries the caller waits
for a response directly after calling on_query_async). Therefore, before
this commit, the trace included incorrect query type information. This
has been fixed by removing the query type information from the trace.
This should be okay since the query type is not essential information
and can be derived from the configuration.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12363
2024-05-14 12:24:30 +02:00
ieQu1 525e4dac95
Merge pull request #13036 from ieQu1/dev/reduce-log-spam
tests: Reduce log spam in the failed test suites
2024-05-14 10:53:30 +02:00
ieQu1 f663373c57
Merge pull request #13037 from ieQu1/dev/merge-release57-240513
Sync release-57 to the master
2024-05-14 10:50:30 +02:00
Andrew Mayorov c10a48f4be
fix(s3-aggreg): disambiguate action schema namespace
Otherwise schemas for `s3` and `s3_aggregated_upload` collide when
projected into OpenAPI schema.
2024-05-14 10:49:21 +02:00
zmstone 93232d4253 fix(authn/http): log meaningful error message if http header is missing 2024-05-14 10:22:07 +02:00
Andrew Mayorov 98a93662c1
Merge pull request #13035 from keynslug/fix/EMQX-12291/pers-sys-msg
fix(sessds): persist $SYS messages as well
2024-05-14 10:05:49 +02:00
firest 7a7bd84cba fix(limiter): improved log content 2024-05-14 15:34:13 +08:00
Ilya Averyanov bca3782d73 fix(auth_http): fix query encoding
* ignore authenticator if JSON format is set up for requests, but non-utf8 data is going to be sent
* use application/json format by default
* fix encoding of query part of the requests
2024-05-14 10:32:53 +03:00
ieQu1 25c6ac3d16
Merge remote-tracking branch 'origin/release-57' 2024-05-13 22:05:29 +02:00
ieQu1 ac3f5a083d
test: Reduce log spam in the failed test suites 2024-05-13 22:00:33 +02:00
ieQu1 8506ca7919
Merge pull request #12998 from ieQu1/dev/improve-latency
Use leader's clock when calculating LTS cutoff timestamp
2024-05-13 21:54:06 +02:00
Andrew Mayorov 66b7ac4c45
fix(sessds): persist $SYS messages as well
Otherwise, persistent sessions will not be able to receive $SYS messages
whatsoever.
2024-05-13 21:27:55 +02:00
Thales Macedo Garitezi a2a5fd1f8e test(client mgmt api): attempt to fix flaky test
```
%%% emqx_mgmt_api_clients_SUITE ==> msgs_plain_encoding.t_mqueue_messages: FAILED
%%% emqx_mgmt_api_clients_SUITE ==>
Failure/Error: ?assertEqual(79, Count)
  expected: 79
       got: 100
      line: 1303
```
2024-05-13 15:53:21 -03:00
ieQu1 3da3a36863
test(ds): Add generation in the replication suite 2024-05-13 19:51:04 +02:00
ieQu1 9f7ef9f34f
fix(ds): Apply review remarks 2024-05-13 19:35:24 +02:00
Thales Macedo Garitezi e7c47f4ec0 fix(ds clients mgmt api): add more fields to disconnected durable sessions
Fixes https://emqx.atlassian.net/browse/EMQX-12369
2024-05-13 14:25:02 -03:00
Kjell Winblad 1c34a84dd2
Merge pull request #13028 from kjellwinblad/kjell/swagger_example_missing_ruleid_paramter/EMQX-12325
fix: include ruleid parameter in swagger example for apply rule endpoint
2024-05-13 17:35:31 +02:00
Thales Macedo Garitezi 81473519bd
Merge pull request #13031 from thalesmg/fix-clientsv2-sec-r57-20240513
fix(client mgmt api): require authn for endpoint
2024-05-13 12:17:31 -03:00
Thales Macedo Garitezi 39899b917d fix(client mgmt api): require authn for endpoint
Fixes https://emqx.atlassian.net/browse/EMQX-12361
2024-05-13 10:54:21 -03:00
Thales Macedo Garitezi 6b032faebc fix(client mgmt api): return expired durable sessions with `is_expired: true`
Fixes https://emqx.atlassian.net/browse/EMQX-12274
2024-05-13 10:51:24 -03:00
Thales Macedo Garitezi 0dee2d67b1 fix: always check if first query has more rows 2024-05-13 09:10:18 -03:00
Kjell Winblad c8690a8273 fix: include ruleid parameter in swagger example for apply rule endpoint
Fixes:
https://emqx.atlassian.net/browse/EMQX-12325
2024-05-13 12:28:17 +02:00
Kjell Winblad 921d82f083 fix: JSON trace formatter should handle report style log entries
Fixes:
https://emqx.atlassian.net/browse/EMQX-12344
2024-05-13 12:11:11 +02:00
zmstone 7faf73e0f4 chore(emqx_schema): change atom array to enum array for alarm.actions 2024-05-13 10:54:33 +02:00
zmstone 290ebe2fc5 fix: deny subscribing to +/# by default ACL
Prior to this change, EMQX default ACL has a deny rule to reject
subscribing to `#`.
For completeness, the default ACL should also deny `+/#` because
they are essentially equivalent.
2024-05-13 09:26:42 +02:00
JianBo He 54f6bd181b
Merge pull request #13010 from JimMoen/fix-jt808-reg-failed
fix(gw): jt808 REG_ACK failed due to faulty clientinfo
2024-05-13 14:19:18 +08:00
JimMoen 2f212f34a3
Merge pull request #12989 from JimMoen/fix-plugin-config-file-backup
fix(plugin): add a backup for the plugin config file
2024-05-11 16:38:36 +08:00
JimMoen a5ecceb755
chore: apply suggestions from code review, thanks @thalesmg 2024-05-11 11:36:40 +08:00
Thales Macedo Garitezi e25fe62cbc fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page
Fixes https://emqx.atlassian.net/browse/EMQX-12294
2024-05-10 14:48:46 -03:00
Kjell Winblad c6df069c5a
Merge pull request #13012 from kjellwinblad/kjell/fix_incorrect_config_crash/EMQX-12315
fix: listener crash if access_rules config option is incorrect
2024-05-10 17:46:40 +02:00
Thales Macedo Garitezi f690db9849 fix(postgres): reduce log spamming when connection goes down
Fixes https://emqx.atlassian.net/browse/EMQX-12334

See also: https://github.com/emqx/epgsql/pull/10
2024-05-10 12:15:01 -03:00
JimMoen 3c7e8a3bd5
test(gw): jt808 register failed 2024-05-10 23:09:22 +08:00
JimMoen f8fd4d15be
fix(gw): process jt808 client deregister packet 2024-05-10 23:00:59 +08:00
Zaiming (Stone) Shi c491b83857
Merge pull request #13014 from zmstone/0510-add-is_template-for-kinesis-partition-key
fix(kinesis): partition_key should be of 'template' type
2024-05-10 16:11:35 +02:00
Kjell Winblad b13fa37771 fix: dialyzer warning 2024-05-10 15:51:28 +02:00
Thales Macedo Garitezi 6be4e6f631 Merge branch 'release-57' into sync-r57-m-20240510 2024-05-10 10:23:28 -03:00
Thales Macedo Garitezi 0eaef18391
Merge pull request #13005 from thalesmg/fix-aggreg-sup-tree-r57-20240509
fix(aggregator): refactor supervision tree
2024-05-10 10:22:33 -03:00
Kjell Winblad 2bed5894e3 test: add test cases for listeners access_rules validation and split 2024-05-10 14:21:43 +02:00
Thales Macedo Garitezi 3a29696a48 fix(aggregator): refactor supervision tree
Instead of using a aggregator supervisor with a fixed local name, we should allow
specifying different names so each action app will spawn its own aggregator supervisor.
2024-05-10 09:06:12 -03:00
zmstone 47df968d4a fix(kinesis): partition_key should be of 'template' type 2024-05-10 13:57:38 +02:00
firest 1fefb421bc feat(limiter): lift the log level of rate limiter to `warning` 2024-05-10 18:47:55 +08:00
Kjell Winblad 0aeb2cd77f fix: listener crash if access_rules config option is incorrect
Previously when changing the access_rules configuration option of a
listener to something that was not a valid rule, the listener would
crash. This has now been fixed by the addition of a configuration
validator that checks the access_rules field.

Additionally, a configuration option converter has been added to the
access_rules field so that one can specify several rules in a single
string by using "," (comma) as separator.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12315
2024-05-10 12:27:04 +02:00
JimMoen 1a6f0c9234
chore: bump gw_jt808 app vsn 2024-05-10 17:13:23 +08:00
JimMoen b07242515f
fix(gw): jt808 REG_ACK failed due to faulty clientinfo 2024-05-10 17:12:28 +08:00
Ivan Dyachkov 900536712d
Merge pull request #12997 from thalesmg/sync-r57-m-20240508
sync `release-57` to `master`
2024-05-10 10:41:30 +02:00
Kjell Winblad 8c0b55a89e
Merge pull request #12992 from kjellwinblad/kjell/fix_rule_unescape_function/EMQX-12313
fix: make rule engine unescape convert \a to the terminal alarm char
2024-05-10 09:48:33 +02:00
lafirest da7b11020a
Merge pull request #13001 from lafirest/fix/sysk_reconn
fix(sysk): fix that the syskeeper forwarding never reconnecting
2024-05-10 11:24:28 +08:00
firest 1f3b640a3d chore: update change 2024-05-10 07:29:14 +08:00
Thales Macedo Garitezi 7b80a9aa44 fix(aggregator): namespace aggregator ids with action type
Otherwise, actions of different types but same names will clash when starting the
aggregator supervision tree.
2024-05-09 16:02:11 -03:00
Andrew Mayorov 57dda70d2f
Merge pull request #12994 from keynslug/fix/sessds/log-levels
fix(sessds): use milder log level for regular error conditions
2024-05-09 18:34:54 +02:00
lafirest 23a1b18459
Merge pull request #12983 from emqx/feat/authn_hook
feat(events): add new hook && event `client.check_authn_complete`
2024-05-09 22:36:52 +08:00
Thales Macedo Garitezi 401f0fa84b Merge branch 'release-57' into sync-r57-m-20240508 2024-05-09 09:13:30 -03:00
firest 2bc014db69 fix(events): call `client.check_authn_complete` even if authentication fails 2024-05-09 18:15:32 +08:00
firest e3a59c4037 fix(sysk): fix that the syskeeper forwarding never reconnecting 2024-05-09 16:27:58 +08:00
Ilia Averianov a1aedee446
Merge pull request #12996 from savonarola/0508-fix-cursor-usage
fix(retainer): fix qlc cursor cleanup
2024-05-09 08:04:10 +03:00
firest d2ca4e9f11 chore: update change 2024-05-09 11:04:56 +08:00
ieQu1 07aa708894
test(ds): Refactor replication suite 2024-05-09 03:56:56 +02:00
zhongwencool 107d00adee
Merge pull request #12944 from zhongwencool/mqtt-utf8-strict-mode
fix: non-utf8 clientid connect event case crash if strict_mod=false
2024-05-09 09:16:10 +08:00
ieQu1 63e51fca66
test(ds): Use streams to fill the storage 2024-05-09 02:46:57 +02:00
ieQu1 a0a3977043
feat(ds): Assign latest timestamp deterministically 2024-05-08 23:17:57 +02:00
ieQu1 2236af84ba
feat(ds): two-stage storage commit on the storage level 2024-05-08 23:17:57 +02:00
ieQu1 f250f00f3f
chore: Bump snabbkaffe version to 1.0.10 2024-05-08 23:17:57 +02:00
ieQu1 1ddbbca90e
feat(ds): Allow incremental update of the LTS trie 2024-05-08 23:17:57 +02:00
ieQu1 68ca891f41
test(ds): Use streams to create traffic 2024-05-08 23:17:57 +02:00
Zaiming (Stone) Shi 018d79b498
Merge pull request #12993 from zmstone/0508-fix-update-listener-zone-from-dashboard
0508 fix update listener zone from dashboard
2024-05-08 22:03:10 +02:00
Thales Macedo Garitezi 55f27281b3
Merge pull request #12991 from thalesmg/refactor-mv-message-type-r57-20240508
refactor(message validation): rename `message_name` to `message_type`
2024-05-08 16:58:28 -03:00
Thales Macedo Garitezi d3905b6336
Merge pull request #12990 from thalesmg/fix-mv-conf-update-r57-20240508
fix(message validation): don't call `emqx_conf:update` during boot
2024-05-08 16:58:15 -03:00
Ilya Averyanov 57287f0722 fix(retainer): fix qlc cursor cleanup 2024-05-08 21:38:37 +03:00
Thales Macedo Garitezi 8d0574abf0
Merge pull request #12979 from thalesmg/fix-ps-monitor-topic-count-r57-20240506
fix(monitor api): count persistent routes and subscriptions
2024-05-08 15:09:16 -03:00
Andrew Mayorov d84c180ccb
feat(dsrepl): avoid contacting unreachable ra servers
Assuming estabilished Erlang distribution channel is a reliable way to
tell whether a remote node is reachable.
2024-05-08 18:12:13 +02:00
Andrew Mayorov 3e956e8711
fix(sessds): use milder log level for regular error conditions
Especially when such events are emitted in (potentially) tight loops,
i.e. pulling messages from iterators pointing to remote shards.
2024-05-08 17:40:08 +02:00
zmstone 9edbad5459 fix(listener_api): do not allow update listener with unknown zone name 2024-05-08 16:37:55 +02:00
Thales Macedo Garitezi d3bc32dc35 fix(message validation): don't call `emqx_conf:update` during boot
`load/0` and `unload/0` shouldn't call cluster operations.
2024-05-08 10:18:29 -03:00
Kjell Winblad 916168c755 fix: make rule engine unescape convert \a to the terminal alarm char
The rule engine unescape function should convert the escape sequence \a
to the alarm bell symbol (ASCII 7). This bug was created as it was
assumed that Erlang convert $\a to 7 but Erlang does not handle the \a
escape sequence and instead convert it to 97 (ASCII code for a).

Fixes:
https://emqx.atlassian.net/browse/EMQX-12313
2024-05-08 15:18:13 +02:00
Thales Macedo Garitezi 5172606032 refactor(message validation): rename `message_name` to `message_type`
On Product's behest, to make naming consistent with protobuf nomenclature.

https://protobuf.dev/programming-guides/proto3/#adding-types
2024-05-08 09:13:40 -03:00
zmstone 3a2ff34433 chore: add zone in listener config example 2024-05-08 11:43:10 +02:00
JimMoen 25be9c2890
fix(plugin): use binary `NameVsn` to re-build plugin avsc serde 2024-05-08 15:43:54 +08:00
JimMoen 68c601ad72
fix(plugin): add a backup for the plugin config file 2024-05-08 15:43:53 +08:00
JianBo He 4403b4f5ce
Merge pull request #12976 from HJianBo/fix-duplicated-disconnect
Fix duplicated disconnect event
2024-05-08 14:43:19 +08:00
Kjell Winblad 72f68afbca
Merge pull request #12981 from kjellwinblad/kjell/fixup_trace_all_http_and_error
Make formatting of action result trace entries better
2024-05-08 06:10:03 +02:00
Thales Macedo Garitezi 0c7b221c42 fix(clients mgmt api): set `durable` flag for DS sessions and subscriptions 2024-05-07 14:55:50 -03:00
Kjell Winblad d5324e295f test: do cleanup in emqx_common_test_helpers:on_exit function
Thanks @thalesmg for the suggestion.
2024-05-07 18:18:18 +02:00
Thales Macedo Garitezi 02c58b67f0
Merge pull request #12986 from thalesmg/fix-mv-name-length-r57-20240507
fix(message validation): limit the length of validation name
2024-05-07 11:39:17 -03:00
Thales Macedo Garitezi 2989793f4c
Merge pull request #12980 from thalesmg/fix-ds-get-offline-client-stuff-r57-20240506
fix(mgmt clients api): hold channel info after client disconnects for display in API
2024-05-07 11:18:00 -03:00
Thales Macedo Garitezi d4d2b4c460
Merge pull request #12985 from thalesmg/fix-mv-cluster-metrics-r57-20240507
fix(message validation): apply config CRUD operations on whole cluster
2024-05-07 11:10:58 -03:00
Kjell Winblad 09ee7ec0e2 fix(rule tracing): make sure that recoverable errors are traced 2024-05-07 15:49:38 +02:00
Thales Macedo Garitezi 96bb6e7d88 fix(message validation): limit the length of validation name
Fixes https://emqx.atlassian.net/browse/EMQX-12299
2024-05-07 10:04:56 -03:00
Thales Macedo Garitezi 06f16d4ee0 fix(message validation): apply config CRUD operations on whole cluster
Fixes https://emqx.atlassian.net/browse/EMQX-12298
2024-05-07 09:45:50 -03:00
Thales Macedo Garitezi 6d6eb42fa3 fix(mgmt clients api): hold channel info after client disconnects for display in api
Fixes https://emqx.atlassian.net/browse/EMQX-12266
2024-05-07 09:22:33 -03:00
Serge Tupchii b869488d35 fix(emqx_license): don't crash if an imported backup file has no license config 2024-05-07 13:48:04 +03:00
zhongwencool a0bf06caba chore: add test case for non-utf8 topic 2024-05-07 17:02:45 +08:00
firest f641d0b2b7 feat(events): add new hook && event `client.check_authn_complete` 2024-05-07 16:24:31 +08:00
Kjell Winblad feecc36607 fix(rule tracing): clean up error tuple in the action_failed trace 2024-05-07 09:13:55 +02:00
JianBo He 1642b06bf9 test: add tests 2024-05-07 10:30:53 +08:00
Thales Macedo Garitezi 28355a4cfc fix(monitor api): count persistent routes and subscriptions
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-06 16:51:31 -03:00
Zaiming (Stone) Shi 93e31d088a
Merge pull request #12978 from JimMoen/fix-typo
chore: fix typo
2024-05-06 21:28:51 +02:00
Kjell Winblad ca88f5731b fix(rule tracing): format result traces in a more structured way 2024-05-06 17:33:59 +02:00
Thales Macedo Garitezi cd4aaf663b docs: briefly describe delivery callbacks 2024-05-06 10:00:10 -03:00
Thales Macedo Garitezi e6b1dfb095 refactor: improve error 2024-05-06 09:53:49 -03:00
Thales Macedo Garitezi 793cc4df2c fix: add fully qualified calls to enable hot beam reloads 2024-05-06 09:52:31 -03:00
Thales Macedo Garitezi edcb750d9f fix: remove unused dependency 2024-05-06 09:51:32 -03:00
Thales Macedo Garitezi 36542a4b1d docs: bump copyright years 2024-05-06 09:50:12 -03:00
Ilia Averianov e19222fc0d
Merge pull request #12971 from savonarola/0502-fix-url-parse
fix(auth,http): improve URI handling
2024-05-06 13:41:10 +03:00
JimMoen 1df2d837ee
chore: fix typo 2024-05-06 17:37:32 +08:00
ieQu1 3642bcd1b6
docs(ds): Fix comment for the builtin DS metrics 2024-05-06 11:21:32 +02:00
ieQu1 b2a633aca1
fix(ds): Use leader's clock for computing LTS safe cutoff time 2024-05-06 11:21:32 +02:00
ieQu1 1ff2e02fd9
feat(ds): Pass current time to the storage layer via argument 2024-05-06 11:21:32 +02:00
ieQu1 8ac9700aab
feat(ds): Add an API for DB-global variables 2024-05-06 11:21:32 +02:00
ieQu1 86d45522e3
fix(dsrepl): Don't reverse elements of batches 2024-05-06 11:21:32 +02:00
ieQu1 bcfa7b2209
fix(ds): Destroy LTS tries when the generation is dropped 2024-05-06 11:21:32 +02:00
ieQu1 9999ccd36c
feat(ds): Ignore safe cutoff time for streams without varying levels 2024-05-06 11:21:32 +02:00
Ilya Averyanov 31026d51f7 chore(auth,http): cache REs for parsing URIs 2024-05-06 11:46:18 +03:00
JianBo He b320b20c7b fix(mqtt): disconnected event should not be sent twice 2024-05-06 16:17:02 +08:00
JianBo He cabe2ae100 chore: fix dialyzer warning 2024-05-06 14:58:27 +08:00
JianBo He d2b6e41cd1 chore(stomp): parse \n as heartbeat frame 2024-05-06 14:39:41 +08:00
Ilya Averyanov 3b655f56cb fix(auth,http): improve URI handling 2024-05-04 09:47:13 +03:00
Zaiming (Stone) Shi e7f0c83406
Merge pull request #12970 from zmstone/0503-refactor-dashboard-listener
refactor: simplify https listener config for dashboard
2024-05-04 08:38:29 +02:00
Zaiming (Stone) Shi c5d8659f35
Merge pull request #12972 from zmstone/0503-unhide-default-dashboard-username-password
chore: unhide dashboard default_password config
2024-05-04 08:36:32 +02:00
Thales Macedo Garitezi 32493e395c refactor: avoid passing record down to template rendering function 2024-05-03 17:43:30 -03:00
Thales Macedo Garitezi 157e2c2535 refactor: make api more generic 2024-05-03 17:25:02 -03:00
Thales Macedo Garitezi a6508e2fca refactor: use better names
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2024-05-03 17:04:09 -03:00
Thales Macedo Garitezi d61f4078e2 refactor: extract blob aggregator logic to separate application
This will allow other bridges to share the same aggregation logic as S3 aggregated action.
2024-05-03 14:35:58 -03:00
zmstone 4d3156b5ed chore: unhide dashboard default_username and default_password config 2024-05-03 18:35:49 +02:00
Thales Macedo Garitezi 3818b75188
Merge pull request #12959 from thalesmg/kprodu-connector-hc-m-20240429
feat(kafka producer): add health check topic option
2024-05-03 12:48:48 -03:00
zmstone 3c5c76fcfc refactor: simplify https listener config for dashboard 2024-05-03 16:45:40 +02:00
Kjell Winblad 004dc80fb2
Merge pull request #12958 from kjellwinblad/kjell/fixup_trace
fix: rule trace formatting, republish and console stop after rendering
2024-05-03 16:24:17 +02:00
Zaiming (Stone) Shi 52357742aa
Merge pull request #12962 from zmstone/0502-fix-ssl-client-opts-to-allow-wildcard-cert-by-default
fix(ssl-clients): allow wildcard certificates by default
2024-05-03 15:19:37 +02:00
Kjell Winblad 5bfe31b691 fix: issues found during PR review (thanks @thalesmg and @zmstone)
* Simpler handling of true and false in best effort JSON formatter
* inet:ntoa/1 to format IP addresses
* Made a record for lazy formatted trace values and formatter to improve
  maintainability
* Added callback to format return value from connector
  * Extended test case to check that the format return value callback
    works
* Added handling of "lazy" trace entry data to the text formatter. Do we
  need to handle this data in the normal log formatters as well?
2024-05-03 15:04:57 +02:00
Kjell Winblad e32745bca6 test(apply rule trace): add test case for republish and console print 2024-05-03 15:02:37 +02:00
Kjell Winblad b9cd75eef0 test(emqx_mgmt_api_trace_SUITE): fix bad test cases 2024-05-03 15:02:37 +02:00
Kjell Winblad 76ccef7ce3 fix(trace api): trace should be ready after create
Fixes:
https://emqx.atlassian.net/browse/EMQX-12276
2024-05-03 15:02:37 +02:00
Kjell Winblad 9576efb716 fix(rule apply test): do not crash when the rule is not found 2024-05-03 15:02:37 +02:00
Kjell Winblad 3ba5cb7858 fix(apply rule): make sure we return something JSON compatible
Fixes:
https://emqx.atlassian.net/browse/EMQX-12275
2024-05-03 15:02:37 +02:00
Kjell Winblad ea7633c484 fix: rule trace formatting, republish and console stop after rendering
* Better rule trace formatting for many trace entries
* The republish and console actions have got working stop after
  rendering functionality
2024-05-03 15:02:36 +02:00
zmstone 37d66e90fb fix(ssl-clients): allow wildcard certificates by default 2024-05-03 12:52:42 +02:00
Ilia Averianov 3ed385201c
Merge pull request #12947 from savonarola/0426-auth-timeout
feat(authn): add connection expire based on authn data
2024-05-03 11:32:18 +03:00
Thales Macedo Garitezi 6f3da6b131 feat(kafka producer): add health check topic option
Fixes https://emqx.atlassian.net/browse/EMQX-12241

This allows more accurate health checking for Kafka Producers.  Without a topic, it's not
possible to actually probe the connection to partition leaders, so the connector might not
be reported as `disconnected` without testing a concrete topic.
2024-05-02 17:07:44 -03:00
Thales Macedo Garitezi c71f73924b
Merge pull request #12961 from thalesmg/kconsu-custom-group-id-m-20240430
feat(kafka consumer): allow custom group id
2024-05-02 14:37:42 -03:00
Ivan Dyachkov cc577e636d Merge remote-tracking branch 'upstream/release-57' into 0502-sync-release-57 2024-05-02 17:06:18 +02:00
Ivan Dyachkov 3064a1cbae
Merge pull request #12957 from id/0430-adjust-build-scripts
adjust build scripts
2024-05-02 17:05:06 +02:00
Thales Macedo Garitezi eb113fa578 fix: add non-empty validator 2024-05-02 11:19:00 -03:00
William Yang 8f780ae8bc
Merge pull request #12955 from qzhuyan/port/william/5/partial-chain
port: TLS partial chain
2024-05-02 15:34:25 +02:00
Thales Macedo Garitezi 607bf0fe44
Merge pull request #12952 from thalesmg/fix-dont-list-mv-rule-event-r57-20240429
fix(rule events): don't list validation failure event in CE API
2024-05-02 09:20:53 -03:00
Thales Macedo Garitezi 3942b371d7 feat(kafka consumer): allow custom group id
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
2024-05-02 08:59:40 -03:00
Ivan Dyachkov 035b8480b0 chore(emqx_conf): extract data dirs to sync to a macro 2024-05-02 10:25:53 +02:00
William Yang 1a4a4bb3a5 chore: fix nit 2024-05-02 10:13:57 +02:00
Thales Macedo Garitezi 14ef0b1e51 feat(kafka consumer): allow custom group id
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
2024-05-01 13:53:29 -03:00
Thales Macedo Garitezi 42cb17360e Merge branch 'release-57' into sync-r57-m-20240430 2024-04-30 14:42:22 -03:00
Thales Macedo Garitezi 7a44746492 fix(rule events): don't list validation failures in CE
Fixes https://emqx.atlassian.net/browse/EMQX-12284
2024-04-30 14:41:09 -03:00
Thales Macedo Garitezi 874f1f2428
Merge pull request #12950 from thalesmg/fix-mv-duplicated-topics-r57-20240429
fix(message validation): validate duplicated topics
2024-04-30 13:34:58 -03:00
Thales Macedo Garitezi dd6566f3c5
Merge pull request #12948 from thalesmg/fix-http-bridge-header-update-obfuscate-r57-20240429
fix(http connector): deobfuscate sensitive headers
2024-04-30 13:34:15 -03:00
Ilya Averyanov e4154dd472 feat(authn): use correct time resolution for setting channel expire in JWT authn 2024-04-30 19:01:16 +03:00
William Yang fb30207ef3 chore: fix test 2024-04-30 16:41:46 +02:00
William Yang 337c230e79 feat(partial_chain): gateway support 2024-04-30 16:41:26 +02:00
Ilya Averyanov 80d724c504 feat(authn): add connection expire based on authn data 2024-04-30 17:04:55 +03:00
Andrew Mayorov 279169105b
Merge pull request #12934 from keynslug/feat/EMQX-12204/aggreg-s3-bridge
feat(s3-bridge): implement aggregated upload action
2024-04-30 13:50:19 +02:00
zmstone 1974ec15ec fix(client_attrs): fix client_attrs extraction loop 2024-04-30 12:12:35 +02:00
Zaiming (Stone) Shi a88862ebe0
Merge pull request #12951 from zmstone/0429-fix-description-schema
fix(schema): description should be in binary() type
2024-04-30 12:03:55 +02:00
Andrew Mayorov 6d3add3646
fix(s3-aggreg): do not handle `{error, closed}` on buffer write
Because it's not really something `file:write/2` is supposed to return.
2024-04-30 10:48:16 +02:00
Andrew Mayorov 4bea938ef2
fix(s3-csv): clarify naming of CSV container concepts
Co-Authored-By: Ilya Averyanov <av@rubybox.dev>
2024-04-30 10:48:16 +02:00
Andrew Mayorov 83366cbed0
fix(s3-aggreg): ensure action works in Rule SQL contexts 2024-04-30 10:48:16 +02:00
Andrew Mayorov a1a313d992
fix(s3-aggeg): apply CSV column order setting consistently
Otherwise, columns that are part of column order could appear and
disappear from consecutive uploads, depending on if they are part
of the very first buffered event or not.
2024-04-30 10:48:16 +02:00
Andrew Mayorov f6e5eea4f7
feat(s3-aggreg): handle delivery shutdowns gracefully 2024-04-30 10:48:15 +02:00
Andrew Mayorov 339036045d
feat(s3-aggreg): support custom and default S3 object HTTP headers
I.e. configured container decides default `Content-Type` header.
2024-04-30 10:48:15 +02:00
Andrew Mayorov 5b15b2d641
docs(s3-bridge): document aggregated upload action schema 2024-04-30 10:48:15 +02:00
Andrew Mayorov ccbcc0c4e3
feat(s3-bridge): implement aggregated upload action 2024-04-30 10:48:15 +02:00
Andrew Mayorov b91ff97170
feat(s3): separate streaming upload logic into dedicated module
And use it in `emqx_s3_uploader`, while also turning it into a simple
gen_server.
2024-04-30 10:48:12 +02:00
Kjell Winblad 7b638a5829 style: fix spelling mistake (thanks @savonarola) 2024-04-30 09:28:12 +02:00
William Yang 70ffd77f99 chore(TLS-chain-test): update for OTP 26 2024-04-30 09:05:51 +02:00
zmstone c8d6976b14 feat: add conditions to variform expressions
- refactored `coalesce` function to allow lazy evaluation
- added `iif(Cond, IfExpr, EleseExpr)` to allow simple conditions
2024-04-30 07:28:27 +02:00
William Yang 650cf4b27e test(partial_chain): update tcs for OTP-25 2024-04-30 06:25:03 +02:00
William Yang 43ad665dcf fix(test): tls_verify_partial_chain 2024-04-30 06:25:03 +02:00
William Yang a29a43e5fc fix(listener): remove partial_chain in wss opts 2024-04-30 06:25:03 +02:00
William Yang 4e9c1ec0c9 chore: happy elvis 2024-04-30 06:25:03 +02:00
William Yang 8eb463c58d feat(tls): update schema for TLS keyusage 2024-04-30 06:25:03 +02:00
William Yang 90430fa66d fix(tls): undefined keyusage 2024-04-30 06:25:03 +02:00
William Yang eb1ab9adfe test(tls): verify peer keyusage 2024-04-30 06:25:03 +02:00
William Yang 8bc3a86f63 feat(config): partial_chain 2024-04-30 06:25:03 +02:00
William Yang fa4357ce89 test: port listener tls partial_chain 2024-04-30 06:25:03 +02:00
William Yang 0b95a08d32 feat(tls): port partial_chain, part 1 2024-04-30 06:25:03 +02:00
zmstone 9917293fd0 fix(schema): description should be in binary() type 2024-04-29 21:44:55 +02:00
Thales Macedo Garitezi ffedce014f fix(message validation): validate duplicated topics
Fixes https://emqx.atlassian.net/browse/EMQX-12254
2024-04-29 15:44:03 -03:00
Thales Macedo Garitezi a847389159 fix(http connector): deobfuscate sensitive headers
Fixes https://emqx.atlassian.net/browse/EMQX-12213
2024-04-29 15:20:46 -03:00
Thales Macedo Garitezi 5cf92dcb73 refactor: use `spawn_link` instead of `spawn_monitor`
This should cover the case when the resource manager is brutally killed.
2024-04-29 15:20:32 -03:00
Thales Macedo Garitezi 475077c798 fix(resource): account for ongoing channel health checks, update data and reply immediately when receiving an update 2024-04-29 15:20:32 -03:00
Thales Macedo Garitezi 79526d539a fix(resource manager): clean up any running health checks when terminating
Fixes https://github.com/emqx/emqx/pull/12812#discussion_r1555564254
2024-04-29 15:20:32 -03:00
zmstone 7a05a4754f docs: expose zone config in schema doc 2024-04-29 13:57:18 +02:00
Zaiming (Stone) Shi a41652ec31
Merge pull request #12940 from zmstone/0427-catch-hocon-syntax-error
feat(mgmt): add ignore_readonly to  configs API
2024-04-29 13:22:12 +02:00
zmstone 1db932df21 chore(mgmt): PUT /configs?ignore_readonly=true, lower log to info level 2024-04-29 10:20:47 +02:00
zhongwencool ef9a63ae17 fix: non-utf8 clientid connect event case crash if strict_mod=false 2024-04-29 14:43:29 +08:00
zmstone 07cbdc6e90 feat(mgmt): add ignore_readonly qeury-string to PUT /configs API 2024-04-28 14:45:53 +02:00
zmstone 10625eacac chore: upgrade to hocon-0.42.2
hocon pretty-print quotes more string values
if a string has '.' or '-', or if it starts with a digit 0-9,
then it's quoted.

see details here: https://github.com/emqx/hocon/pull/293
2024-04-28 14:05:30 +02:00
firest d6c203b4fd fix(dynamo): fixed the keys checking for Dynamo 2024-04-28 16:59:01 +08:00
zmstone 5f4215b333 fix(mgmt): avoid 500 error when hocon syntax error 2024-04-27 21:12:42 +02:00
JimMoen 43ac4f5dfe
fix: make bpapi check happy 2024-04-26 21:17:01 +08:00
JimMoen 5ff4e76904
refactor: rename plugins config api functions 2024-04-26 20:51:20 +08:00
Kjell Winblad db48a977bb test(mongodb authn, authz): add test cases for use_legacy_protocol
This commit adds test cases that check that the authn and authz modules
for MongoDB support the use_legacy_protocol configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12245
2024-04-26 12:41:50 +02:00
JimMoen 00cab33fde
fix: plugin's internal config api 2024-04-26 18:10:55 +08:00
Ilya Averyanov aaf57ecfbc chore(authz): improve and clarify types 2024-04-26 12:09:18 +03:00
JimMoen 11389bc086
fix: i18n file renamed 2024-04-26 11:12:24 +08:00
JimMoen 28e8131984
refactor: avoid make when do serde 2024-04-26 10:37:15 +08:00
JimMoen e5bd747b32
refactor: read avsc file when make serde 2024-04-26 10:25:19 +08:00
JimMoen 670ddae57c
chore: fix typo 2024-04-26 10:25:18 +08:00
JimMoen c884dfb451
test: make eunit happy 2024-04-26 10:25:17 +08:00
JimMoen 1869f6fd0a
fix: enusre plugin installed when do config operation 2024-04-26 10:25:16 +08:00
JimMoen b0aa3bb70f
fix(plugin): get plugin config api 2024-04-26 10:25:15 +08:00
JimMoen f343cd2021
fix: get plugin config in json 2024-04-26 10:25:14 +08:00
JimMoen d2ecccc2ff
fix: call json encoder/decoder for plugin config 2024-04-26 10:25:12 +08:00
JimMoen c180b6a417
fix(api): plugin api docs 2024-04-26 10:25:11 +08:00
JimMoen 1f00ce789f
fix(plugin): gen_server call timeout infinity 2024-04-26 10:25:10 +08:00
JimMoen c0429ca333
fix(plugin): refine schema serde log 2024-04-26 10:25:09 +08:00
JimMoen 27d1f91cac
refactor: refine function name 2024-04-26 10:25:08 +08:00
JimMoen 2686a66b14
fix: make eunit happy 2024-04-26 10:25:07 +08:00
JimMoen d2e0c09f2e
fix: make static check happy 2024-04-26 10:25:06 +08:00
JimMoen d06f410fd5
fix(plugins): read avsc file 2024-04-26 10:25:05 +08:00
JimMoen 8db5e51592
feat: plugin config with avro schema and apis 2024-04-26 10:25:04 +08:00
JimMoen 71cdcc860a
fix(plugin): plugin's mgmt api schema codes 2024-04-26 10:25:03 +08:00
Zaiming (Stone) Shi 6af651cf2f
Merge pull request #12931 from zmstone/0425-fix-acl.conf-topic-template-render-fialure-handling
0425 fix acl.conf topic template render fialure handling
2024-04-25 20:30:02 +02:00
Ilia Averianov a0e0a27f87
Merge pull request #12418 from savonarola/0129-fix-auth-claim-schema
feat(jwt_auth): improve verify_claims handling and docs
2024-04-25 19:05:13 +03:00
chengshq 75dab4dff0 fix: STOMP heartbeat 2024-04-25 23:33:15 +08:00
zmstone 01923147a2 fix(variform and authz): do not initialize empty client_attrs field
when client_attrs_init expression renders to empty string,
do not initialize the attribute.

also fixed an ACL error: a template render failure for a topic
would stop the ACL checks for the following topics if more
than one topic is configured.
2024-04-25 17:32:07 +02:00
zmstone d30b52f0f9 docs: refine acl.conf comments 2024-04-25 17:32:07 +02:00
Kjell Winblad 15594b4db6 fix(HTTP connector): retry on 503 Service Unavailable response
Previously, if an HTTP request received a 503 (Service Unavailable)
status, it was marked as a failure without retrying. This has now been
fixed so that the request is retried a configurable number of times.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12217
https://github.com/emqx/emqx/issues/12869 (partly)
2024-04-25 17:05:18 +02:00
Ilya Averyanov 407b0cd0ca feat(jwt_auth): improve verify_claims handling and docs 2024-04-25 17:49:29 +03:00
Ilia Averianov c42583550d
Merge pull request #12514 from savonarola/0214-fix-ft-responses
fix(ft): report ft assemble status from a dedicated process
2024-04-25 15:10:26 +03:00
Kjell Winblad ff09f14191 fix(http connector): remove sensitive info from headers lazily
In production code we don't need to redact the headers for a trace that
will never appear anywhere so we can improve performance by doing
removal of sensitive information lazily.
2024-04-25 13:27:14 +02:00
Kjell Winblad ef9884cf47 refactor(rule trace): templates rendered trace to increase code reuse
* The code for passing the trace context to a sub process has been
  improved to increase code reuse. This code is used when the action
  templates are rendered in a sub process.
* A macro has also been added for the error term that is thrown when the
  action shall be stopped after the templates has been rendered. This is
  also done to reduce code duplication and to reduce the risk of
  introducing bugs due to typos.
* Fix incorrect type spec

Thanks to @zmstone for suggesting these improvements in comments to a PR
(https://github.com/emqx/emqx/pull/12916).
2024-04-25 13:15:36 +02:00
Kjell Winblad 0dbaef4316 feat: add stop after render and after render trace to tdengine action 2024-04-25 11:51:46 +02:00
Kjell Winblad d6ceeb3b30 feat: add stop after render and after render trace to rabbitmq action 2024-04-25 11:51:46 +02:00
Kjell Winblad 11d9d30fc0 feat: add stop after render and after render trace to syskeeper action 2024-04-25 11:51:46 +02:00
Kjell Winblad a0b2357abb feat: add stop after render and after render trace to sqlserver action 2024-04-25 11:51:46 +02:00
Kjell Winblad 03e3ac19a9 feat: add stop after render and after render trace to s3 action 2024-04-25 11:51:46 +02:00
Kjell Winblad 22c7224267 feat: add stop after render and after render trace to rocketmq action 2024-04-25 11:51:46 +02:00
Kjell Winblad 2abc1b1141 feat: add stop after render and after render trace to redis action 2024-04-25 11:51:46 +02:00
Kjell Winblad 74fac80e7e feat: add stop after render and after render trace to pulsar action 2024-04-25 11:51:46 +02:00
Kjell Winblad 9c37c99b62 feat: add stop after render and after render trace to oracle action 2024-04-25 11:51:46 +02:00
Kjell Winblad d27f05fa60 feat: add stop after render and after render trace to influxdb action 2024-04-25 11:51:46 +02:00
Kjell Winblad 279ad186f7 feat: add stop after render and after render trace to kinesis action 2024-04-25 11:51:46 +02:00
Kjell Winblad beedc72be4 feat: add stop after render and after render trace to mongodb action 2024-04-25 11:51:46 +02:00
Kjell Winblad e2b35ea242 feat: add stop after render and after render trace to opents action 2024-04-25 11:51:46 +02:00
Kjell Winblad 9d6655bc30 feat: add stop after render and after render trace to hstreamdb action 2024-04-25 11:51:46 +02:00
Kjell Winblad 7c7590fbc8 feat: add stop after render and after render trace to greptimedb action 2024-04-25 11:51:46 +02:00
Kjell Winblad 7922d5d422 feat: add stop after render and after render trace to gcp action 2024-04-25 11:51:46 +02:00
Kjell Winblad a2dd8f5aee feat: add stop after render and after render trace to cassandra action 2024-04-25 11:51:46 +02:00
Kjell Winblad 810aa68b02 feat: add stop after render and after render trace to dynamo action 2024-04-25 11:51:46 +02:00
Kjell Winblad 120b35ac75 feat: add stop after render and after render trace to mysql action 2024-04-25 11:51:46 +02:00
Kjell Winblad b2811f96b2 refactor(rule trace): simplify function for setting trace meta data
This commit simplifies a function to set trace meta data in line with a
suggestion from  @zmstone:

https://github.com/emqx/emqx/pull/12912#discussion_r1576053856
2024-04-25 11:51:46 +02:00
Kjell Winblad 7ad354f412 feat: add stop after render and after render trace to clickhouse action 2024-04-25 11:51:46 +02:00
Kjell Winblad 32c27f1711 feat: add stop after render and after render trace to kafka action 2024-04-25 11:51:46 +02:00
Kjell Winblad b02ed4e6ec feat: add stop after render and after render trace to pgsql action 2024-04-25 11:51:46 +02:00
Kjell Winblad 1f676ce035 feat: add stop after render and after render trace to mqtt action 2024-04-25 11:51:46 +02:00
Kjell Winblad 5ca90ccced fix: improve structure of log trace entries for HTTP action
Fixes:
https://emqx.atlassian.net/browse/EMQX-12025
2024-04-25 11:51:46 +02:00
zmstone a67773e973 Merge remote-tracking branch 'origin/master' into release-57 2024-04-25 10:56:52 +02:00
zhongwencool 2cbf4dc789
Merge pull request #12923 from zhongwencool/authn-mnesia
chore: provided more specific error for wrong import method
2024-04-25 10:28:09 +08:00
zhongwencool 2a2da3766e
chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-04-25 09:16:53 +08:00
zmstone f2fa35efdf Merge remote-tracking branch 'origin/master' into 0424-merge-latest-master-to-release-57 2024-04-24 22:02:56 +02:00
Ilya Averyanov 05f544495e fix(ft): report ft assemble status from a dedicated process
Previously, the status was monitored by the channel itself.
If channel disconnected before the assemble was completed,
the status message was lost (not sent to the resonse topic)
2024-04-24 18:16:28 +03:00
Serge Tupchii 02a0faa1d6 refactor: avoid evaluating Data more than once in SLOG_THROTTE macro 2024-04-24 16:26:34 +03:00
zmstone b66d7a688e Merge remote-tracking branch 'origin/release-57' into release-57 2024-04-24 13:09:06 +02:00
zhongwencool 4f12d45436 chore: provided more specific error for wrong import method 2024-04-24 15:37:35 +08:00
zhongwencool 712008bc1d
Merge pull request #12909 from zhongwencool/esockd-bump-version
fix: bump escokd to 5.11.2 to handle udp_error/closed message
2024-04-24 09:15:39 +08:00
ieQu1 5804b70c1f
Merge pull request #12918 from ieQu1/dev/ds-readme1
docs(ds): Update README with CLI and REST API endpoints
2024-04-23 17:02:01 +02:00
ieQu1 e4c3283c9c docs(ds): Update README with CLI and REST API endpoints 2024-04-23 16:28:35 +02:00
Kjell Winblad e1eed30b5d
Merge pull request #12912 from kjellwinblad/kjell/rule_trigger_time/EMQX-12025
feat(rule tracing): add rule trigger time meta data field
2024-04-23 15:00:05 +02:00
lafirest 0d1c13661f
Merge pull request #12895 from lafirest/fix/dyndb
fix(dynamo): Added missing keys for DynamoDB
2024-04-22 23:03:00 +08:00
lafirest a81ca359fc
Merge pull request #12899 from lafirest/fix/rocmq
feat(rocketmq): add support for namespace and key dispatch strategy
2024-04-22 21:20:44 +08:00
Kjell Winblad e9d498dde2 feat(rule tracing): add rule trigger time meta data field
Fixes:
https://emqx.atlassian.net/browse/EMQX-12025
2024-04-22 13:51:48 +02:00
zhongwencool 6049b4966f fix: bump escokd to 5.11.2 to handle udp_error/closed message 2024-04-22 16:41:09 +08:00
ieQu1 f9eda1883f
Merge pull request #12903 from ieQu1/dev/ds-egress-flush-condition
fix(ds): Fix egress flush condition
2024-04-22 10:17:22 +02:00
JianBo He c5c4bb987f
Merge pull request #12902 from HJianBo/stomp
Pass the Content-type of MQTT message to the Stomp message
2024-04-22 16:02:03 +08:00
Ivan Dyachkov b723e87e54 Merge remote-tracking branch 'upstream/release-56' into 0422-sync-release-56 2024-04-22 08:12:05 +02:00
firest d5cdc07eab feat(rocketmq): add support for namespace and key dispatch strategy 2024-04-22 13:14:11 +08:00
firest 46f339dfab chore: updte change log 2024-04-22 13:12:01 +08:00
firest a1e85e3c59 fix(dynamo): Added missing keys for DynamoDB 2024-04-22 13:09:34 +08:00
Kjell Winblad 6e99f01ecd
Merge pull request #12863 from kjellwinblad/kjell/new_trace_log_formatter/EMQX-12025
feat(emqx_trace): add JSON trace log entry formatter
2024-04-22 06:25:18 +02:00
JianBo He d85df14b85
Merge pull request #12892 from HJianBo/fix-gateway-related-issues
fix(ocpp): avoid an error log in handling downstream messages
2024-04-22 11:25:26 +08:00
JianBo He 69bdcd2f24
Merge pull request #12898 from emqx/iotdb-1.3.0
feat: support iotdb 1.3.0
2024-04-22 10:56:06 +08:00
ieQu1 4c76a2574d
fix(ds): Fix egress flush condition 2024-04-21 21:51:31 +02:00
Zaiming (Stone) Shi 307cd79be2
Merge pull request #12897 from zmstone/0418-support-dynamic-license
feat(license): add business-critical customer type
2024-04-20 09:03:58 +02:00
JianBo He 5520e54147 chore: add tests 2024-04-20 08:59:36 +08:00
JianBo He aaf7953968 chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-04-20 08:36:59 +08:00
JianBo He 95f3e49edb fix(stomp): pass the Content-Type from the MQTT message 2024-04-20 08:33:55 +08:00
Andrew Mayorov 43f8346c00
fix(dssnap): ensure idempotent write of empty chunks 2024-04-19 18:52:33 +02:00
ieQu1 bac5100635
Merge pull request #12874 from ieQu1/dev/EMQX-12030-subscriptions-api
API for durable subscriptions
2024-04-19 17:11:53 +02:00
Kjell Winblad 2890bc2619 fix(tracing): remove internal extra field from the trace config
This commit removes the internal extra field from the trace config
structure exposed to the user via the HTTP API.
2024-04-19 17:05:34 +02:00
Kjell Winblad 9e46c18443 fix(emqx_trace_SUITE:t_base_create_delete): broken test case 2024-04-19 15:41:50 +02:00
zmstone ec83fbe3dc feat(license): add business critical customer type 2024-04-19 14:19:39 +02:00
ieQu1 ede7246882
fix(sessds): Avoid double-enriching transient messages 2024-04-19 13:39:04 +02:00
Ilia Averianov b7a4536f47
Merge pull request #12871 from savonarola/0412-fix-rebalance-app-dependencies
fix(rebalance): fix start order of rebalance applications
2024-04-19 13:27:41 +03:00
Zaiming (Stone) Shi a8b1224225
Merge pull request #12893 from zmstone/0416-add-is-template-flag-to-dashboard-schema
0416 add `is_template` flag to dashboard schema
2024-04-19 10:41:43 +02:00
Kjell Winblad c163956d08 fix(trace formatter): remove record field to enable rolling upgrade 2024-04-19 09:52:20 +02:00
zhongwencool b163a87386 feat: support batch_size on iotdb 2024-04-19 14:52:56 +08:00
zhongwencool c0521fd250 feat: support iotdb 1.3.0 2024-04-19 08:54:18 +08:00
ieQu1 f1e6565ddd
refactor(sessds): Move all subscription logic to the subs module 2024-04-19 01:20:29 +02:00
ieQu1 d12966db5b
test: Avoid dumping raw snabbkaffe traces to the console 2024-04-19 01:20:29 +02:00
ieQu1 180130d684
feat(sessds): List persistent subscriptions in the REST API 2024-04-19 01:20:29 +02:00
Kjell Winblad 09b414f368 test: add necessary application to test suites 2024-04-18 17:39:52 +02:00
Kjell Winblad 285bfa9367 fix: improve rendering of action_template_rendered trace 2024-04-18 17:00:47 +02:00
Kjell Winblad 7be18730e8 test(emqx_rule_engine_api_rule_apply_SUITE): remove unnecessary code 2024-04-18 15:16:18 +02:00
Kjell Winblad f5b0439724 fix(emqx_rule_engine_api_rule_apply_SUITE): flaky test case 2024-04-18 14:28:28 +02:00
Kjell Winblad 8f1486f6d3 fix: clean up trace messages to make it easier to interpret
This commit removes some redundant trace messages and renames some to
make it easier to interpret what is happening for the user.
2024-04-18 14:28:01 +02:00
zmstone 5b38d592f0 feat(http): add `is_template` as HTTP headers field property
is_template was designed to be type property.
however for HTTP headers, it's a map() type,
instead of creating a new type for it, it's easier to just
add it as a field property.
2024-04-18 13:16:29 +02:00
zmstone ede4eeae9f fix(http_bridge): path is template field 2024-04-18 13:04:56 +02:00
zmstone ca56e7e8d7 fix(kafka): headers are template fields 2024-04-18 13:04:36 +02:00
Kjell Winblad 3232ab5ea3 fix(rule tracing): unset trace meta data in try-after-end
We wrap the reset of the process trace meta data in the after clause of
a try-after-end expression to be sure we never get any lingering
incorrect meta data.
2024-04-18 11:28:50 +02:00
Kjell Winblad aa388adba9 fix(json trace format): format client_ids and rule_ids as lists 2024-04-18 11:27:39 +02:00
Kjell Winblad 10957e7d79 fix: change name of the default trace log format from plain to text 2024-04-18 10:42:51 +02:00
Kjell Winblad ef9f8a8fdf fix: unreachable clause found by dialyzer 2024-04-18 10:28:37 +02:00
Kjell Winblad 6c0ee8bb01 test(emqx_trace_SUITE): fix failure due to new field 2024-04-18 10:28:37 +02:00
Kjell Winblad 968dc2ccda fix(json trace format): use best_effort_json instead of new encoder 2024-04-18 10:28:37 +02:00
Kjell Winblad aa39835242 feat(emqx_cli): add parameter for trace format (plain or json) 2024-04-18 10:28:29 +02:00
Kjell Winblad 6fd7a06c5d fix: problems reported by dialyzer 2024-04-18 10:20:18 +02:00
Kjell Winblad a6558740e8 feat(emqx_trace): add JSON trace log entry formatter
This commit makes it possible to select the JSON trace log entry
formatter when crating a trace pattern. This will make it easier for the
dashboard and automatic tests to parse the log entries.

Fixes: EMQX-12025 (partly)
2024-04-18 10:20:11 +02:00
zmstone ab763fe665 test: fix test case flakyness 2024-04-18 09:32:05 +02:00
Kjell Winblad 2a2fadfbff
Merge pull request #12827 from kjellwinblad/kjell/emqx_rule_tracing/EMQX-11966
Add rule ID tracing support
2024-04-18 09:06:31 +02:00
JianBo He d7ebecddb4 fix(ocpp): return correct current_connections number of listenrs http api 2024-04-18 14:54:12 +08:00
Ivan Dyachkov e93bd314bc chore: 5.6.1 2024-04-18 08:03:56 +02:00
JianBo He 75632bb2cd chore(gw): update listener examples 2024-04-18 13:58:31 +08:00
Kjell Winblad b16b9d8fcc refactor: made code more readable thanks to suggestion from @zmstone 2024-04-18 06:51:30 +02:00
zmstone 55941000c0 test: make test case more stable, less flaky 2024-04-17 20:42:03 +02:00
zmstone 6ab2b004ed fix(resource_manager): update cache after channel add 2024-04-17 20:34:33 +02:00
zmstone c96ae8dd23 fix: return 503 if bridge bpapi call timeout 2024-04-17 20:34:33 +02:00
zmstone 5a4bfff9e5 refactor: add template_str type 2024-04-17 20:34:33 +02:00
zmstone 5c014f4c29 test: fix test cases 2024-04-17 20:34:33 +02:00
Kjell Winblad 95891db29a test: clean up created resources in the end of test case 2024-04-17 18:43:19 +02:00
Kjell Winblad 6cf29ba688 fix: clean up traces to make them easier to parse and understand 2024-04-17 18:20:17 +02:00
Kjell Winblad cf56050759 feat: avoid mixing request with and without the stop_after_render flag
Previously a batch of requests that was sent to a connector could
contain both requests with the stop_after_rendering flag and requests
without this flag. When this happened a warning message was generated and
the stop_after_render flags for the batch would be ignored. This commit
fixes so that a mixed batch is never created so there is no longer any
need for a warning message or ignoring flags.
2024-04-17 16:21:21 +02:00
zmstone d49e98bc4b test: fix dashboard schema validation 2024-04-17 14:09:48 +02:00
zmstone f9f14f9758 refactor(emqx_conf): raise exception at higher level for more context 2024-04-17 13:10:55 +02:00
zmstone 51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
JianBo He 2468243dfd chore: update changes 2024-04-17 18:52:48 +08:00
ieQu1 124c5047d0
feat(sessds): Add API for getting session data from the cold storage 2024-04-17 12:31:54 +02:00
ieQu1 38a2e8add9
fix(sessds): Return the number of subscriptions for offline sessions 2024-04-17 12:31:54 +02:00
ieQu1 e439a2e0f2
fix(sessds): Save protocol name and version in the session metadata 2024-04-17 12:31:54 +02:00
JianBo He 18196ec19c fix(ocpp): avoid an error log in handling dnsteeam messages 2024-04-17 17:48:19 +08:00
JianBo He c61c3157c6
Merge pull request #12888 from HJianBo/data-backup-for-license
fix(license): license file is not taking effect after importing backup files
2024-04-17 16:31:16 +08:00
JimMoen 1dfd9115cd
Merge pull request #12880 from JimMoen/EMQX-12147/influx-write-syntax-tag-type
fix(influx): literal number values in tag set
2024-04-17 10:23:57 +08:00
ieQu1 197a4c30be
fix(sessds): Strip unneccessary data from the durable session state 2024-04-17 01:21:52 +02:00
ieQu1 93bb840365
docs(ds): Update README 2024-04-17 01:21:52 +02:00
ieQu1 87ffaf89e5
refactor(sessds_state): Use macros for map keys 2024-04-17 01:21:52 +02:00
ieQu1 113a990482
feat(sessds): Support max subscriptions 2024-04-17 01:21:51 +02:00
ieQu1 6c897c26ae
fix(sessds): Commit session on unsubscribe 2024-04-17 01:21:51 +02:00
ieQu1 6c83bbe10b
feat(mgmt): Filter subscriptions by durability 2024-04-17 01:21:51 +02:00
ieQu1 b30ddc206e
fix(sessds): Immutable subscriptions
This commit fixes two issues:
- Behavior of overlapping subscriptions has been aligned with the
in-memory session.

- Fixed handling of replays when subscription changes (either by
client or EMQX configuration)
2024-04-17 01:21:51 +02:00
ieQu1 3e0c649e8e
feat(sessds): Store awaiting rel 2024-04-17 01:21:51 +02:00
zmstone df458b98d7 refactor(dashboard_schema): no need to translate labels
the trans_label implementation was ugly, it compares an anonymous
function to check if the label should be translated.

since we have stopped generating i18n message ids for dashboard
schema, this entire function is now stale, so this function
is deleted.
2024-04-16 17:54:34 +02:00
zmstone 2bd72aab44 chore: bump dashboard schema version to 0.2.0 2024-04-16 16:58:45 +02:00
Zaiming (Stone) Shi 4d38a8fd44
Merge pull request #12872 from zmstone/0412-support-variform-for-client_attrs
0412 use variform for client_attrs
2024-04-16 13:29:56 +02:00
JianBo He 6b8111c066 fix(license): license file is not taking effect after importing backup files 2024-04-16 19:07:17 +08:00
ieQu1 c645cfa5d6
fix(sessds): Graceful handling of shared subscription error 2024-04-16 12:37:01 +02:00
Ivan Dyachkov f3bb28c6af chore: 5.6.1-rc.2 2024-04-16 09:47:32 +02:00
Andrew Mayorov 5d7b2e2ce6
fix(dsrepl): attempt leadership transfer on terminate
In addition to on removal. The reasoning is basically the same: try to
avoid situations when log entries are replicated (or will be considered
replicated when the new leader is elected) but the leader terminates
before replying to the client.

To be clear: this is a stupid solution. Something much more robust is
needed.
2024-04-15 22:05:24 +02:00
Ivan Dyachkov a0ffe5e7ae chore: 5.6.1-rc.1 2024-04-15 22:05:05 +02:00
Kjell Winblad f481871792 fix(rocketmq action): we need one producer group per channel and topic
We need one producer group per channel and topic because we can have
several topics per channel due to templating.
2024-04-15 20:52:08 +02:00
Kjell Winblad 1fe92bddd0 fix(rocketmq action): make sure that topic template is respected 2024-04-15 20:37:23 +02:00
Kjell Winblad 7f9a311988 fix(rocketmq action): all actions used only one topic
This commit makes sure that a set of rocketmq workers are started for
each topic. Before this commit all actions for a rocketmq connector used
the same workers which all were bound to a single topic so all messages
got delivered to that topic regardless of the action configuration.

We should have automatic tests to check that the messages actually go to
different topics but this needs to go into another PR since we have to
deliver the fix fast and the rocketmq library does not support reading
from topics.

Fixes:
https://emqx.atlassian.net/browse/EEC-1006
2024-04-15 20:08:56 +02:00
zmstone f80d078de3 feat(variform): Add more functions 2024-04-15 16:56:52 +02:00
Andrew Mayorov 89f42f1171
fix(dsrepl): make placeholder shard process permanent under supervisor 2024-04-15 16:43:52 +02:00
Andrew Mayorov c4d1360b96
fix(dsrepl): trigger election for new ra servers unconditionallly
Otherwise we might end up in a situation when there's no member online
yet at the time of the election trigger, and the election will never
happen.
2024-04-15 16:42:29 +02:00
JimMoen 084e920c6e
test(influx): literal values or variable in tag set 2024-04-15 17:47:53 +08:00
JimMoen 8ed397d4fa
fix(influx): literal number values in tag set 2024-04-15 17:47:52 +08:00
Ivan Dyachkov 3ef160eed2 Merge remote-tracking branch 'upstream/release-56' into 0415-sync-release-56 2024-04-15 08:09:03 +02:00
zmstone b76b6fbe63 feat(variform): initialize client_attrs with variform
Moved regular expression extraction as a variform function.
2024-04-14 10:13:24 +02:00
zmstone da5b01aa46 refactor(client_attr): allow more than one initial extraction 2024-04-13 01:00:25 +02:00
zmstone 2577224bc7 fix(swagger): do not generate dummy descriptions 2024-04-13 01:00:25 +02:00
Zaiming (Stone) Shi 06b07460e8
Merge pull request #12866 from zmstone/0411-refactor-resove-TODOs
refactor: resolve some old TODOs
2024-04-13 00:10:47 +02:00
Kjell Winblad b4198185bc fix(http connector): redact sensitive information from headers 2024-04-12 15:27:03 +02:00
Kjell Winblad 6777f04780 fix: iolist_to_binrary -> unicode:characters_to_binary
It is not always safe to use iolist_to_binrary on the output of an
io_lib:format call with the ~ts placeholder.
2024-04-12 15:21:25 +02:00
Ilya Averyanov 500d4fedda fix(rebalance): fix start order of rebalance applications 2024-04-12 15:58:57 +03:00
Kjell Winblad ed5409fb6a docs(trace): add emqx_ctl documentation for the new ruleid trace 2024-04-12 11:54:14 +02:00
Kjell Winblad f444c6fc32 fix: pass stop_action_after_render=true in trace meta data
Even if there is no trace we still need to pass
stop_action_after_render=true in the trace meta data so that the action
will be stopped.
2024-04-12 11:38:01 +02:00
Kjell Winblad aa950f97fa test: fix tests with missing application 2024-04-12 11:23:19 +02:00
Kjell Winblad 9998940aa2 fix(trace): several improvements thanks to comments from @zmstone 2024-04-11 16:48:43 +02:00
Ivan Dyachkov b27fc0da26 test(emqx_machine): ensure node is down before testing open ports 2024-04-11 15:24:41 +02:00
zmstone dfd13b4ab5 test: fix dashboard schema json test cases 2024-04-11 15:12:01 +02:00
zmstone afc87ddc9e refactor: do not generate i18n msgid in dashboard schema json
dashboard has its own mind
now i18n is only used to generate docs:
1. runtime swagger spec at /api-docs
2. build-time config schema
both result in the resolved i18n text, but not msgid
2024-04-11 14:09:22 +02:00
Ivan Dyachkov 3b7cade671 chore: 5.6.1-beta.1 2024-04-11 13:49:40 +02:00
zmstone eac25194e5 refactor: resolve some old TODOs
move dashboard schema generation code to the right module
2024-04-11 13:41:34 +02:00
Zaiming (Stone) Shi e7a4210943
Merge pull request #12858 from zmstone/0410-fix-variform-number-handling
fix(variform): allow numbers to be numbers
2024-04-11 11:24:55 +02:00
zmstone 834bddadad test: delete flaky test for now 2024-04-11 09:39:17 +02:00
zmstone e6330dddec fix(variform): allow numbers to be numbers 2024-04-10 11:57:45 +02:00
Zaiming (Stone) Shi 2fea651d1d
Merge pull request #12851 from zmstone/0327-feat-add-emqx_variform
emqx_variform for string substitution and transform
2024-04-10 10:18:40 +02:00
JianBo He 91ffc95f29
Merge pull request #12855 from JimMoen/fix-share-queue-format
fix(mgmt): $queue shared topics format in mgmt topics api
2024-04-10 10:52:48 +08:00
JianBo He 5f8986209e
Merge pull request #12842 from lafirest/fix/iotdb
fix(iotdb): correctly handle undefined value of bool type
2024-04-10 09:49:16 +08:00
Serge Tupchii fae9005f87 test(emqx_mgmt_cli): test that replicants do not join a left core node 2024-04-09 19:06:13 +03:00
Serge Tupchii 55179ccfed chore: update ekka to 0.19.3
Included updates:
  - https://github.com/emqx/mria/pull/178
2024-04-09 19:04:38 +03:00
Kjell Winblad 31142df5cf fix: default value of stop_action_after_template_rendering to true
This commit changes the default value for the
stop_action_after_template_rendering option of the apply rule HTTP API
endpoint so that it is true instead of false.
2024-04-09 16:20:37 +02:00
Kjell Winblad 7bcd553786 test: move test case with rule_engine dep from emqx to emqx_rule_engine 2024-04-09 14:08:46 +02:00
Kjell Winblad 3898950017 test: fix test by adding dependency 2024-04-09 11:38:35 +02:00
Kjell Winblad 958748cf7f test: fix inter test suite problem 2024-04-09 11:10:39 +02:00
JimMoen 03a9c46ca7
fix(sys_topic): format shared topics 2024-04-09 17:06:36 +08:00
JimMoen 47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api 2024-04-09 17:06:32 +08:00