Commit Graph

12179 Commits

Author SHA1 Message Date
ieQu1 1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable 2024-05-24 19:09:34 +02:00
ieQu1 25721bceba
fix(dsstor): Delete generation metadata before dropping it 2024-05-24 19:09:33 +02:00
ieQu1 3d9837a0b8
test(ds): Add emqx_ds_replication_SUITE:t_drop_generation 2024-05-24 19:09:33 +02:00
Ilya Averyanov 327080e70e perf(emqx_placeholder): do not try to decode maps as json
fix
2024-05-24 20:08:31 +03:00
Andrew Mayorov c30b881239
test(monitor): verify historical and current metrics are consistent 2024-05-24 14:56:10 +02:00
Andrew Mayorov fd9655bc35
fix(monitor): unify metric names for consistency and simplicity 2024-05-24 14:55:48 +02:00
Andrew Mayorov 7b137c0aaf
fix(monitor): reflect durable subscriptions in historical metrics
Essentially in the same manner as they are currently reflected in the
current metrics.
2024-05-24 14:08:35 +02:00
zmstone 6ec8ae5a5f
Merge pull request #13112 from HJianBo/change-the-default-batch-size-s3-uploader
Fix(s3): change the batch_size to 1 for direct mode
2024-05-24 10:56:53 +02:00
Andrew Mayorov 29d203ae33
Merge pull request #13108 from keynslug/fix/ct-flaky/mgmt
test(mgmt): avoid inter-suite contamination by switching to cth tooling
2024-05-24 11:14:16 +03:00
JianBo He adaf238131 fix(s3): fix the convert function if the mode is missing 2024-05-24 16:02:06 +08:00
JimMoen 5b60f6a5b2
fix: make dialyzer happy 2024-05-24 14:29:02 +08:00
JimMoen 43506495ae
fix: ensure plugin configured states after join cluster 2024-05-24 13:51:56 +08:00
JimMoen 874bcd9a8b
fix: match decode plugin config map failed case 2024-05-24 10:20:50 +08:00
zhongwencool 3aa1f86d9e chore: is_content_type_json to validate_content_type_json 2024-05-24 10:13:24 +08:00
zhongwencool 5c759941d5 feat: return 415 when UNSUPPORTED_MEDIA_TYPE 2024-05-24 10:13:23 +08:00
zmstone c3c6da7ec4 chore: prepare for release cut 5.7.0-rc.1 2024-05-23 21:39:15 +02:00
Zaiming (Stone) Shi 3a4b8e6c24
Merge pull request #13106 from zmstone/0523-add-customer-type
0523 add customer type (4=BYOC) to license schema
2024-05-23 21:30:02 +02:00
Zaiming (Stone) Shi 40288b99b2
Merge pull request #13101 from JimMoen/fix-plugin-config-map
fix: allow put plugin config without schema
2024-05-23 21:18:43 +02:00
ieQu1 c952e46f08
Merge remote-tracking branch 'origin/release-57' 2024-05-23 21:04:03 +02:00
Thales Macedo Garitezi 823a6bc521
Merge pull request #13107 from thalesmg/hide-clientv2-response-type-r57-20240523
fix(client mgmt api): hide unused return type
2024-05-23 16:03:51 -03:00
Andrew Mayorov 7878d106f6
test(mgmt): avoid inter-suite contamination by switching to cth tooling 2024-05-23 20:17:34 +02:00
Andrew Mayorov b4c6968f8c
Merge pull request #13104 from keynslug/fix/EMQX-12415/last-resort-poll
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
2024-05-23 21:17:03 +03:00
ieQu1 0dc507441a
Merge pull request #13105 from ieQu1/dev/sessds-renew-streams-interval
fix(sessds): Change the defaults to renew streams every 1 second
2024-05-23 18:23:20 +02:00
Thales Macedo Garitezi b1e3c941fb fix(client mgmt api): hide unused return type
Fixes https://emqx.atlassian.net/browse/EMQX-12427
2024-05-23 13:17:43 -03:00
Andrew Mayorov d47bf8076a
test(mgmt): better isolate testsuite running environments 2024-05-23 17:53:13 +02:00
zmstone c737045dff chore: add customer type 4 (BYOC) 2024-05-23 17:30:06 +02:00
ieQu1 5584b658c9
fix(sessds): Change the defaults to renew streams every 1 second 2024-05-23 17:17:29 +02:00
Zaiming (Stone) Shi 1d99ba2a8a
Merge pull request #13102 from JimMoen/fix-ds-prom-badkey
fix: add ds metrics into prometheus zipper init acc
2024-05-23 16:57:06 +02:00
Thales Macedo Garitezi 118c1d2806
Merge pull request #13103 from thalesmg/fix-another-ds-counter-nit-r57-20240523
fix(monitor api): fix cluster metric aggregation
2024-05-23 10:57:53 -03:00
William Yang bb3f858db2
Merge pull request #13086 from qzhuyan/dev/william/quic-lb
feat: quic-lb support
2024-05-23 15:31:14 +02:00
Andrew Mayorov ba6382adae
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
This is a hack to work around the unintended issues causing shard
allocator to become idle even when there are pending transitions.
2024-05-23 14:54:01 +02:00
Andrew Mayorov 839b143fc4
fix(sessds): consider message durability disabled by default 2024-05-23 14:48:39 +02:00
Thales Macedo Garitezi c0e3a81c61 fix(monitor api): fix cluster metric aggregation
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-23 09:38:24 -03:00
Andrew Mayorov bf326acd7b
fix(dsrepl): handle stopping non-yet-running shard supervisor 2024-05-23 14:34:27 +02:00
Kjell Winblad 0e5b02c373
Merge pull request #13100 from kjellwinblad/kjell/s3_aggregate_only_has_result_trace/EMQX-12429
fix: add action rendered trace for s3 in aggregated mode
2024-05-23 14:29:18 +02:00
JimMoen a40c7d646a
fix: apply suggestions from code review, thanks @zmstone 2024-05-23 20:28:14 +08:00
JimMoen b91ecac168
refactor(prom): unify function name to avoid multiple levels call 2024-05-23 20:11:22 +08:00
JimMoen 065411c69c
fix: add ds metrics into prometheus zipper init acc 2024-05-23 20:11:21 +08:00
Kjell Winblad 89b47e8ffc fix(s3 tracing): do not format records as IO data in aggregate mode 2024-05-23 12:41:30 +02:00
Kjell Winblad fb7688ab94 fix(trace): make sure that the payload encode works with nested payloads
This commit makes sure that the trace setting for payload encode works
even when the payload is in a nested structure or when the payload key
is a binary instead of an atom.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12424
2024-05-23 12:30:56 +02:00
JimMoen 5dc96a37e2
fix: ensure plugin avsc serde before config 2024-05-23 17:46:58 +08:00
JimMoen 25a9aa1797
fix: allow put plugin config without schema 2024-05-23 17:33:34 +08:00
JimMoen 99869821fb
fix: ensure plugin config map after fresh install 2024-05-23 17:16:16 +08:00
William Yang 94e59c8fa7 feat(quic-lb): fix test 2024-05-23 10:32:23 +02:00
William Yang d8eecfe64f feat(quic-lb): add configuration 2024-05-23 10:32:20 +02:00
Kjell Winblad e10c87b825 fix: add action rendered trace for s3 in aggregated mode
Fixes:
https://emqx.atlassian.net/browse/EMQX-12429
2024-05-23 10:25:27 +02:00
Andrew Mayorov 398dc97ed6
Merge pull request #13092 from keynslug/fix/dsrepl/site-autoleave
fix(dsrepl): properly handle transaction abort during forget site
2024-05-23 10:56:23 +03:00
ieQu1 950fa82be3
Merge pull request #13097 from ieQu1/dev/ds-replay-unrecoverable-no-test
feat(sessds): Handle unrecoverable errors
2024-05-23 09:49:21 +02:00
William Yang 34a69625b5 feat: quic-lb support 2024-05-23 09:17:09 +02:00
Kjell Winblad 26c988fe11
Merge pull request #13077 from kjellwinblad/kjell/fix_action_update_race_issue/EMQX-12376
fix: action config update would sometimes not be reflected in connector
2024-05-23 09:11:48 +02:00
Andrew Mayorov 2f6729d203
Merge pull request #13089 from keynslug/fix/EMQX-12437/restore
fix(s3-bridge): restore backward config compatibility
2024-05-23 10:10:01 +03:00
zmstone b0832ecc74 test: fix a flaky one 2024-05-23 08:55:28 +02:00
JianBo He 4053356597
Merge pull request #13063 from JimMoen/improve-plugin-config-behavior
fix(plugin): serde not found on new-joined nodes
2024-05-23 11:24:44 +08:00
lafirest b1076221b0
Merge pull request #13085 from lafirest/fix/rocket_namespace
fix(rocketmq): fix namespace error for RocketMQ
2024-05-23 10:39:30 +08:00
ieQu1 07774ab060
feat(sessds): Handle unrecoverable errors 2024-05-22 22:45:37 +02:00
Thales Macedo Garitezi 63cb12d7c2
Merge pull request #13095 from thalesmg/fix-monitor-ds-counters-again-r57-20240522
fix(monitor api): add ds connections and subscriptions to old counters
2024-05-22 16:43:15 -03:00
Thales Macedo Garitezi 20a767eee8 test: fix broken test 2024-05-22 16:19:47 -03:00
ieQu1 bfd01c425d
Merge pull request #13072 from ieQu1/dev/fix-drop-generation
Idempotent drop_generation
2024-05-22 21:13:30 +02:00
ieQu1 6eb04f90a3
fix(ds): Allow to write batches to older generations 2024-05-22 20:28:16 +02:00
Thales Macedo Garitezi 5cad4497de fix(monitor api): add ds connections and subscriptions to old counters
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-22 15:19:46 -03:00
Thales Macedo Garitezi 4094032649 fix(bridge v2 api): don't attempt to start disabled connector when starting action/source
Fixes https://emqx.atlassian.net/browse/EMQX-12435
2024-05-22 13:33:55 -03:00
zmstone 1bc9600c58 Merge remote-tracking branch 'origin/release-57' 2024-05-22 18:28:23 +02:00
Kjell Winblad 88c96e26de refactor: simplify the code with maps:map/2
Thanks @thalesmg for the suggestion
2024-05-22 18:03:43 +02:00
ieQu1 59a09fb86f
fix(ds): Apply review remarks 2024-05-22 18:01:33 +02:00
ieQu1 5c78ecba40
docs(ds): Update documentation for the storage layouts 2024-05-22 18:01:33 +02:00
ieQu1 29345aaa30
fix(ds): Fix idle event generation in bitfield_lts layout 2024-05-22 18:01:33 +02:00
ieQu1 b3ded7edce
fix(ds): Fix code review remark 2024-05-22 18:01:33 +02:00
ieQu1 60edf5e9b8
fix(ds): Move responsibility of returning end_of_stream to the CBM 2024-05-22 18:01:33 +02:00
ieQu1 0ff307e789
fix(ds): Include generation ID in the storage events
Make sure storage events originating from generation X are handled in
the context of the same generation.
2024-05-22 18:01:33 +02:00
ieQu1 acdae4fba3
fix(ds): Workaround for the idempotency error when dropping gens 2024-05-22 18:01:33 +02:00
ieQu1 eb7c43ee9d
fix(ds): Always store messages in the current generation 2024-05-22 18:01:33 +02:00
ieQu1 074d98a14a
test(ds): Refactor ds_SUITE 2024-05-22 18:01:33 +02:00
ieQu1 e4a73f003a
feat(ds): Implement format_status callback
Reduce volume of logs and crash reports from DS
2024-05-22 18:01:32 +02:00
ieQu1 1526c527d0
fix(ds): Log generation operations 2024-05-22 18:01:32 +02:00
ieQu1 aca2d9586c
fix(ds): Fix return type of drop_generation 2024-05-22 18:01:32 +02:00
ieQu1 c6fc76e335
fix(ds): Perform read operations on the leader. 2024-05-22 18:01:32 +02:00
ieQu1 4580906405
fix(ds): Use erpc instead of gen_rpc for `delete_next' 2024-05-22 18:01:32 +02:00
Andrew Mayorov 8016e9adf4
fix(s3-bridge): restore backward config compatibility 2024-05-22 17:59:52 +02:00
Kjell Winblad 4b540e3bd0 fix: do not leak action configurations in alarm messages 2024-05-22 17:43:07 +02:00
Andrew Mayorov e6c5c1b598
chore(dsrepl): provide more information in rebalancing log messages 2024-05-22 17:24:08 +02:00
Andrew Mayorov c355c9ad50
fix(dsrepl): properly handle transaction abort during forget site 2024-05-22 17:22:55 +02:00
Kjell Winblad cff8b97e8a fix: handle channel updated during health check
This commit fixes an issue found by CI test case
emqx_bridge_influxdb_SUITE:t_start_stop and others. While the channel
health check process is running, the channel could be removed or updated
which could cause a crash in the resource manager or non up-to-date
alarms being triggered.
2024-05-22 12:15:04 +02:00
Kjell Winblad 39d758c4d6 fix: do not return configs for channels from emqx_resource_manager 2024-05-22 12:15:04 +02:00
Kjell Winblad 917474f694 fix: action config update would sometimes not be reflected in connector
Before this commit the following happened sometimes:

1. action status is connected
2. action config is updated to something that should change the status to
   disconnected
3. action status is still connected and the old config is being used by
   the connector even though the config has been correctly updated.

The reason for this bug is that the post_config_hook runs before the
global EMQX config is updated. The post config hook is adding the new
config to the connector. Since the new config causes the action to get
status disconnected, the adding of the action to the connector is
retried when the health check runs but this time the config will be
loaded from the global config which could cause the old config to be
loaded (this happens when the global config has not had time to get
updated).

The above problem has been fixed in this commit by only reading action
configs from the global config when the connector starts/restarts and
instead store the latest configs for the actions in the connector.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12376
2024-05-22 12:15:04 +02:00
firest f3e8037e0f fix(rocketmq): fix namespace error for RocketMQ 2024-05-22 16:29:38 +08:00
Andrew Mayorov a3cd3e31b1
Merge pull request #13084 from keynslug/fix/EMQX-12422/part-size
fix(s3-bridge): report inconsistent `min/max_part_size` option
2024-05-22 09:48:09 +02:00
JimMoen e5f7aa9817
refactor: plguin functions and types rename 2024-05-22 11:51:54 +08:00
JimMoen 33aa61daea
fix: use hocon format as plugin config 2024-05-22 06:37:58 +08:00
JimMoen 14f2a68799
fix: bpapi spec type 2024-05-22 06:01:57 +08:00
Thales Macedo Garitezi 31a35f2a15
Merge pull request #13076 from thalesmg/fix-ds-session-count-client-api-r57-20240520
fix(client mgmt api): cache disconnected durable session count for `/clients` api
2024-05-21 15:34:21 -03:00
JimMoen 5abd23af5a
test: plugin refactor 2024-05-22 02:09:37 +08:00
JimMoen 87b3b214b9
fix: make static_check happy 2024-05-22 02:09:36 +08:00
JimMoen 3ce091e9d7
fix: ensure plugin tarball and extracted 2024-05-22 02:09:35 +08:00
JimMoen 1abc8cf502
fix: ensure plugin's config on boot and join cluster 2024-05-22 02:09:34 +08:00
JimMoen 677352e498
fix: ensure installed and plugin state on boot 2024-05-22 02:09:33 +08:00
JimMoen 2076681e76
fix: put plugin config with binary namevsn key 2024-05-22 02:09:32 +08:00
JimMoen 140b7ce51e
fix(plugin): schema content only provided in enterprise edition 2024-05-22 02:09:31 +08:00
JimMoen a7f2f95318
fix: ensure avro file 2024-05-22 02:09:30 +08:00
JimMoen df7dcb2764
fix: do not let plugin start failed lead emqx start failed 2024-05-22 02:09:29 +08:00
JimMoen e0e4517d9e
fix: ensure plugin config on boot 2024-05-22 02:09:28 +08:00