Commit Graph

12481 Commits

Author SHA1 Message Date
JimMoen 756797b258
refactor: gen metric schema with desc from macros 2024-06-12 19:04:39 +08:00
JimMoen 263d2dbae2
fix: inc metric `'client.auth.anonymous'` when client anonymous 2024-06-12 19:04:39 +08:00
JimMoen 9a78a6f640
refactor: mv metrics macros in hrl file 2024-06-12 19:04:29 +08:00
JimMoen cbac4019b3
fix: rm havn't used counter idx 2024-06-12 19:01:50 +08:00
Thales Macedo Garitezi 46ab3be1f4 chore: change types of mysql and mongodb fields to `template()`
Fixes https://emqx.atlassian.net/browse/EMQX-12395
2024-06-11 17:25:53 -03:00
Thales Macedo Garitezi 0874768c1d fix(client mgmt api): allow projecting `client_attrs` from client fields
Fixes https://emqx.atlassian.net/browse/EMQX-12511
2024-06-11 17:14:33 -03:00
zmstone 8a283b1cc0
Merge pull request #13180 from zmstone/0604-optimize-tcp-ssl-connection-data-send-performance
0604 optimize tcp ssl connection data send performance
2024-06-11 22:06:57 +02:00
zmstone 09f91159c9 fix(emqx_rule_funcs): expose regex_extract function to rule engine
previoulsy only available in variform expressions,
now made available for rule-engine
2024-06-11 21:31:13 +02:00
zmstone 51810c6919
Merge pull request #13229 from zmstone/0611-auth-with-raw-cert
feat(authn): support `${cert_pem}` placeholder
2024-06-11 21:16:01 +02:00
zmstone 751f7a24e9 feat(authn): support ${cert_pem} placeholder 2024-06-11 19:54:24 +02:00
Thales Macedo Garitezi 337009c3a0 fix: declare `emqx_schema_registry` as a dependency of `emqx_schema_validation` 2024-06-11 14:03:27 -03:00
Thales Macedo Garitezi 0f9c3b4cea feat(schema validation): check references schema names and types before changing config
Fixes https://emqx.atlassian.net/browse/EMQX-12368
2024-06-11 14:03:26 -03:00
Thales Macedo Garitezi 03b226248a feat(schema registry): add check for inner types
Currently, only `protobuf` has any.
2024-06-11 13:50:11 -03:00
Thales Macedo Garitezi c70e8252fe fix: declare `emqx_schema_registry` as a dependency of `emqx_message_transformation` 2024-06-11 13:35:02 -03:00
Thales Macedo Garitezi 2c6aa80b3c
Merge pull request #13221 from thalesmg/refactor-sv-topic-index-r57-20240610
refactor(schema validation): module organization and position indexing
2024-06-11 13:29:48 -03:00
Thales Macedo Garitezi 83ff07ade0
Merge pull request #13199 from thalesmg/message-transformation-r57-20240604
feat: implement message transformation
2024-06-11 13:29:25 -03:00
Thales Macedo Garitezi ab5329be02
Merge pull request #13220 from thalesmg/refactor-channel-hc-opt-r57-20240610
fix: remove another redundant health check
2024-06-11 12:22:43 -03:00
Andrew Mayorov 2f0eb3560b
Merge pull request #13227 from keynslug/fix/EMQX-12533/bad-tpl
fix(bridge-s3): validate aggreg key template before adding a channel
2024-06-11 15:39:12 +03:00
Andrew Mayorov d4fb812352
fix(aggreg): anticipate and handle delivery startup errors 2024-06-11 13:21:41 +02:00
Andrew Mayorov 29fc30ea69
fix(bridge-s3): validate aggreg key template before adding a channel 2024-06-11 13:21:41 +02:00
Andrew Mayorov fb0da9848c
feat(tpl): add separate `placeholders/1` function
The purpose is to have a clearer view of placeholders used in a
template, without going the usual `render(Template, #{})` route that is
actually subtly misleading: it won't mention that `${}` / `${.}`
placeholder has been used.

Also unify handling of `${}` / `${.}` in a couple of places.
2024-06-11 11:42:43 +02:00
Kjell Winblad 9d22089104
Merge pull request #13173 from kjellwinblad/kjell/pgsql_timestamp_encode_etc/EEC-1036
fix(pgsql connector): handle prepared statement already exists
2024-06-11 10:53:30 +02:00
William Yang 44258204bd chore: move tls_certs test helper to apps/emqx 2024-06-11 10:06:18 +02:00
zhongwencool b657dc537c
Merge pull request #13156 from savonarola/0529-fix-dashboard-monitor
fix(dashboard_monitor): correctly merge old and new sample data
2024-06-11 10:33:15 +08:00
Thales Macedo Garitezi 05ebb17cd6 refactor: index positions rather than names
Addresses https://github.com/emqx/emqx/pull/13199#discussion_r1633096025
2024-06-10 15:54:37 -03:00
Thales Macedo Garitezi 5eff4a7544 refactor(schema validation): improve api and module organization
To conform to comment from another PR:
https://github.com/emqx/emqx/pull/13199#discussion_r1632658742

... since schema validation is the "inspiration" for message transformation.
2024-06-10 15:46:28 -03:00
Thales Macedo Garitezi d93c8540c8 feat: pass along peername down to transformation context 2024-06-10 15:35:35 -03:00
Thales Macedo Garitezi 6e0ef893f4 feat: pass along client attributes down to message transformation context 2024-06-10 15:27:04 -03:00
Thales Macedo Garitezi 3c501e4f2a fix: remove another redundant health check
Fixes https://emqx.atlassian.net/browse/EMQX-12521
2024-06-10 15:13:34 -03:00
Thales Macedo Garitezi 1a20a40218
Merge pull request #13209 from thalesmg/rm-redundant-hc-r57-20240607
chore: remove redundant health check call
2024-06-10 14:58:48 -03:00
Thales Macedo Garitezi 5629fe60c1 refactor: index positions rather than names 2024-06-10 13:33:22 -03:00
Thales Macedo Garitezi 8c7a7cf9db refactor: apply review remarks 2024-06-10 11:17:13 -03:00
Thales Macedo Garitezi e54cf2f218 refactor: use option to skip rendering as string 2024-06-10 10:49:39 -03:00
Thales Macedo Garitezi d34558954d refactor: improve api and module organization 2024-06-10 10:40:53 -03:00
Thales Macedo Garitezi 4aa1c3842f fix: declare `emqx` as a dependency in `.app.src` 2024-06-10 09:53:29 -03:00
Thales Macedo Garitezi aa416cc0ad docs: fix stale comments and dates 2024-06-10 09:53:02 -03:00
zmstone 7a09b0470d chore: cosmetic changes 2024-06-10 14:22:25 +02:00
Kjell Winblad a885f0b41a test(emqx_bridge_pgsql_SUITE): call test janitor 2024-06-10 13:45:27 +02:00
William Yang a1aa9a4375 fix(ee): emqx no longer deps on emqx_auth_ext 2024-06-10 10:32:01 +02:00
zmstone 4347f3de3e fix(bridge/mqtt): respect client ID prefix 2024-06-10 10:27:45 +02:00
William Yang 76cfc309a9 feat(tls): partial_chain not required 2024-06-10 10:25:44 +02:00
William Yang 7c37bf9965 feat(tls): ee only: TLS partial_chain and Keyusage 2024-06-10 10:24:48 +02:00
William Yang e60380d205 chore: fix nit 2024-06-10 09:55:12 +02:00
William Yang e9b813d8ef chore: fix test 2024-06-10 09:54:54 +02:00
William Yang 1739bc0c24 feat(partial_chain): gateway support 2024-06-10 09:54:44 +02:00
William Yang abbf2ef62f chore(TLS-chain-test): update for OTP 26 2024-06-08 00:16:06 +02:00
William Yang 221b748b0f test(partial_chain): update tcs for OTP-25 2024-06-08 00:16:01 +02:00
William Yang 463d1a1875 fix(test): tls_verify_partial_chain 2024-06-08 00:15:54 +02:00
William Yang 788cdbc6dd fix(listener): remove partial_chain in wss opts 2024-06-08 00:15:51 +02:00
William Yang 3fc99315e0 chore: happy elvis 2024-06-08 00:15:45 +02:00
William Yang c5dccdf526 feat(tls): update schema for TLS keyusage 2024-06-08 00:15:43 +02:00
William Yang aa25e3badd fix(tls): undefined keyusage 2024-06-08 00:15:39 +02:00
William Yang 9e5cfea8c0 test(tls): verify peer keyusage 2024-06-08 00:15:36 +02:00
William Yang c5657029ab feat(config): partial_chain 2024-06-08 00:15:34 +02:00
William Yang f7ff9496e6 test: port listener tls partial_chain 2024-06-08 00:15:31 +02:00
William Yang 1ce13242a8 feat(tls): port partial_chain, part 1 2024-06-08 00:15:28 +02:00
Kjell Winblad c13631102e fix(PgSQL connector): typo in function name 2024-06-07 19:40:18 +02:00
Andrew Mayorov b2f027c974
Merge pull request #13197 from keynslug/fix/EMQX-12522/conv-ssl
fix(bridge-s3): pass SSL options through `convert_certs/2`
2024-06-07 19:45:53 +03:00
Thales Macedo Garitezi 28a3c77e88 chore: remove redundant health check call
Since `emqx_resource_manager:channel_health_check` already does a base resource health
check, the call was repeated.
2024-06-07 11:07:12 -03:00
Kjell Winblad e63dcc84b0 fix: unused variable and better error message 2024-06-07 12:20:42 +02:00
JimMoen 78a196e215
Merge pull request #13189 from JimMoen/fix-apply-sqlserver-template
fix(utils_sql): improve insert sql regular expression
2024-06-07 13:50:19 +08:00
lafirest eb215726d5
Merge pull request #13195 from lafirest/fix/typos
fix: fix typos
2024-06-07 09:04:16 +08:00
Thales Macedo Garitezi ef36350bf6 feat: implement message transformation
Fixes https://emqx.atlassian.net/browse/EMQX-12517
2024-06-06 17:51:08 -03:00
Andrew Mayorov 8a85d7cb5a
fix(bridge-s3): pass SSL options through `convert_certs/2` 2024-06-06 19:37:38 +02:00
Andrew Mayorov c17c788438
feat(connector): allow delegate `pre_config_update` to impls 2024-06-06 19:37:34 +02:00
zmstone 2525bd0c16 perf: eliminate the loop-back fake async inet_reply message
Now that EMQX is on OTP 26 by default, the fake inet_reply
message when the send call returns ok only adds overhead.

For TCP (gen_tcp module), OTP 26 no longer supports direct port_command
so we had to swith to gen_tcp:send, then fake an async reply:
see https://github.com/emqx/esockd/pull/181

For TLS (ssl module), it has never supported async send anyway.

This commit switches to call Transport:send/2 directly,
this should result in one less loop-back message for each message.

For OTP 25 on which EMQX can still be compiled, one will have to
suffer the performance penalty for gen_tcp:send function to
do a non-optimized 'receive' for the send result.
2024-06-06 09:35:51 +02:00
zmstone 5c3f5d8085 perf: do not call inet getstat before each and every send
In a stress test environment, when alarm is enabled, there were
a lot of long_schedule warnings with the stacktrace pointing to
congestion alarm based inet:getstat function.
When alarm is disabled, the one single client is able to
hold 40,000 QoS 1 messages per second (inflight=32) throughput.
When alarm is enabled, its mqueue overflows.

This commit removes the call before each and every data send,
so to rely on emit_stats timer to trigger the congestion check.
2024-06-06 09:35:51 +02:00
firest d4118c6e8e fix: fix typos 2024-06-06 11:19:40 +08:00
JimMoen dff31b293c
perf(utils_sql): pre-build regexp 2024-06-06 11:01:38 +08:00
zmstone c41be7ee22 chore: fix dialyzer warning 2024-06-05 23:37:21 +02:00
JimMoen 144264e0d8
test: add utils_sql test cases 2024-06-06 00:30:18 +08:00
JimMoen 627b8c45d2
fix(utils_sql): improve insert sql regular expression 2024-06-06 00:30:17 +08:00
Ilya Averyanov 60556abc4f fix(dashboard_monitor): correctly merge old and new sample data 2024-06-05 18:49:52 +03:00
Kjell Winblad 2956e849eb fix(pgsql connector): better msg when failing to remove statement 2024-06-05 16:58:14 +02:00
Kjell Winblad 336089f8a7 fix: bug found by dialyzer and make test case cleaner 2024-06-05 15:53:02 +02:00
Thales Macedo Garitezi ae0379f974
Merge pull request #13181 from thalesmg/force-stop-connector-r57-20240604
fix(resource manager): force kill process if stuck when stopping/removing
2024-06-05 09:08:30 -03:00
lafirest d79193fde4
Merge pull request #13177 from lafirest/sync-authn-com
feat: sync `client.check_authn_complete` to release-57
2024-06-05 19:51:10 +08:00
Thales Macedo Garitezi 5eab221f7c
Merge pull request #13175 from thalesmg/disable-prepared-statements-postgres-r57-20240603
feat(postgres): add `disable_prepared_statements` option
2024-06-04 12:29:53 -03:00
Thales Macedo Garitezi f17aefe3d7 fix(action/source api): improve returned error message on timeout 2024-06-04 12:04:55 -03:00
zmstone cda6d5f636
Merge pull request #13172 from zmstone/0603-add-redis-arg-formatter-to-rule-engine-funcs
feat(redis): add a rule function to help formatting redis args
2024-06-04 16:56:14 +02:00
Thales Macedo Garitezi 3013189cd7 fix(resource manager): force kill process if stuck when stopping/removing
Fixes https://emqx.atlassian.net/browse/EMQX-12357
2024-06-04 11:38:24 -03:00
Thales Macedo Garitezi c07bc68e6f feat(postgres): add `disable_prepared_statements` option
Fixes https://emqx.atlassian.net/browse/EMQX-12496

Some Postgres connections, such ones made to [PGBouncer](https://www.pgbouncer.org/) or
[Supabase in Transaction Mode](https://supabase.com/), do not support some session
features like prepared statements.
2024-06-04 10:11:25 -03:00
firest 87384cae32 fix(events): call `client.check_authn_complete` even if authentication fails 2024-06-04 17:55:39 +08:00
firest 92d26ff27b chore: update change 2024-06-04 17:55:30 +08:00
firest e4e53844d5 feat(events): add new hook && event `client.check_authn_complete` 2024-06-04 17:55:21 +08:00
Kjell Winblad 572ca6433e fix(pgsql connector): improvements due to suggestions from @thalesmg
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-06-04 10:11:45 +02:00
zmstone aa7ce1f641 fix(bridge/redis): add test case for map_to_redis_hset_args 2024-06-03 23:05:43 +02:00
Kjell Winblad f2ccfff803 fix(pgsql connector): handle prepared statement already exists
In a user's log file it was found that that the pgsql driver can end up
in a situation where the prepared statement for a channel/action is not
properly removed before a channel with the same name as the prepared
statement is added to the connector. This commit handles this by
attempting to remove the old prepared statement if one already exists
when adding channel.

Related issue:
https://emqx.atlassian.net/browse/EEC-1036
2024-06-03 16:14:26 +02:00
zmstone e3ed7b59dd feat(redis): add a rule function to help formatting redis args
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.

This new function is dedicated for redis to avoid abuse for other
data integrations.
2024-06-03 14:09:15 +02:00
JimMoen 21624bc865
fix: get plugin app vsn from field `rel_apps` 2024-06-03 15:26:14 +08:00
zmstone b6ff67d712
Merge pull request #13093 from zmstone/0522-upgrade-to-brod-3.18
chore: upgrade to brod 3.18.0
2024-06-02 15:06:40 +02:00
JimMoen 8d22a5a3ae
Merge pull request #13167 from JimMoen/fix-rule-events-i18n-todo
fix: missing rule event i18n in zh_cn
2024-06-02 18:23:20 +08:00
zmstone 9637b14543 fix: upgrade to brod 3.18.0 to fix unnecessary rebalance 2024-06-01 10:57:24 +02:00
zmstone 37ce8d93a1
Merge pull request #13164 from zmstone/0530-http-acl-compatibility-with-v4
0530 http acl compatibility with v4
2024-05-31 16:28:30 +02:00
Thales Macedo Garitezi c289c7d065
Merge pull request #13148 from thalesmg/EMQX-12251-fix-conn-api-timeout-case-clause-r571-20240528
fix(connector api): handle `timeout` when waiting for connector status
2024-05-31 10:07:35 -03:00
zmstone 6fe8a09e97 fix(authz/http): rename PH_ACCESS to VAR_ACCESS 2024-05-31 14:49:54 +02:00
JimMoen 45e2c81ebc
fix: missing rule event i18n in zh_cn 2024-05-31 17:51:47 +08:00
zmstone ca32cf8d29 Merge branch 'emqx-12487-fix-authz-http-content-type' into 0530-http-acl-compatibility-with-v4 2024-05-30 22:30:20 +02:00
zmstone a00d9f17b1 chore: upgrade kafka producer lib wolff to 1.10.4
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
2024-05-30 15:40:10 +02:00
zmstone 460081a22e
Merge pull request #13131 from zmstone/0527-handle-unicode-in-template-strings
0527 handle unicode in template strings
2024-05-30 15:37:43 +02:00
Ilya Averyanov 97f9c81e19 feat(auth): add legacy ${access} placeholder 2024-05-30 15:47:48 +03:00
Ivan Dyachkov 0c4da98b52 chore: update deps 2024-05-30 11:53:00 +02:00
Ivan Dyachkov 1129c18330 fix(authz_http): fix content-type header in http request 2024-05-30 11:36:13 +02:00
zhongwencool 8ab7e75e64
Merge pull request #13139 from zhongwencool/influxdb-client-failed-msg-port
chore: improved InfluxDB connection error messaging
2024-05-30 10:57:21 +08:00
zmstone 532812f6f6 test: assert results 2024-05-29 22:10:34 +02:00
zmstone 9fe29bf5f7 chore: bump kafka bridge app vsn 2024-05-29 21:57:14 +02:00
zmstone 78a6100346 chore: fix app vsn bumps
only bug fixes so far
2024-05-29 21:56:22 +02:00
zmstone 1db9d54d17 fix: handle unicode in data integration template strings
The bug was introduced in 5.7.0 (#12893)
2024-05-29 21:53:07 +02:00
zmstone c54d25de98
Merge pull request #13070 from zmstone/0518-improve-kafka-connection-error-logs
0518 improve kafka connection error logs
2024-05-29 16:24:01 +02:00
Kjell Winblad 5017ef82a3
Merge pull request #13130 from kjellwinblad/kjell/fix_redis_trace_format/EMQX-12428
fix(redis trace): add separators in redis batch action trace
2024-05-29 15:59:32 +02:00
Kjell Winblad 6810591911
Merge pull request #13147 from kjellwinblad/kjell/fix_function_clause_when_decoding_protobuf/EMQX-12453
fix: make protobuf schema decode error reported to user less cryptic
2024-05-29 15:56:27 +02:00
Kjell Winblad 3769bbb7f4
Merge pull request #13140 from kjellwinblad/kjell/republish_trace_crash/EMQX-12474
fix: trace formatter should not crash when given a non-binary payload
2024-05-29 15:54:02 +02:00
Kjell Winblad 9b089a4e5a fix: use explain instead of msg as error explanation and add test 2024-05-29 10:20:15 +02:00
Kjell Winblad 479889f162
Merge pull request #13136 from kjellwinblad/kjell/fix_oracle_trace_format/EMQX-12433
fix(oracle action trace): parameters should not be rendered as IO Data
2024-05-29 09:12:38 +02:00
zhongwencool 7d9333a9d7 chore: bump influxdb app version 2024-05-29 11:13:17 +08:00
zhongwencool 00d74826ec chore: improved InfluxDB connection error messaging 2024-05-29 11:13:17 +08:00
Thales Macedo Garitezi 802361dbd0 fix(connector api): handle `timeout` when waiting for connector status
Fixes https://emqx.atlassian.net/browse/EMQX-12251
2024-05-28 14:19:32 -03:00
Kjell Winblad 82800faadf fix: make protobuf schema decode error reported to user less cryptic
Before this commit the user would see a cryptic warning log including
function_clause when decoding a message failed. This commit improves
this by removing function_caluse as well as some other cryptic words
from the message and adding a description describing what went wrong.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12453
2024-05-28 18:07:14 +02:00
Kjell Winblad 3151a3123e chore: bump version of emqx_redis_bridge app 2024-05-28 17:23:35 +02:00
Kjell Winblad 3f1fe413ee refactor: simplify redis trace formatting 2024-05-28 17:22:58 +02:00
Kjell Winblad cbdde7165e fix(redis trace): add separators in redis batch action trace
Logger will transform data that looks like IO data into a string which
made redis batch traces look like the spaces had been removed
from the strings. To prevent this, we render the batched commands into
binary string separated by spaces and semicolon. The components of a
single command is separated by spaces and the commands in a batch are
separated by semicolons.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12428
2024-05-28 17:22:58 +02:00
Kjell Winblad 5e3de1a4e3 fix(emqx_logger_jsonfmt:best_effort_json_test): test cases 2024-05-28 17:15:32 +02:00
zmstone 40080f5e7d
Merge pull request #13141 from zmstone/0528-test-fix-dashboard-config-test-with-stable-base-config
test: fix test case to have deterministic base config
2024-05-28 14:32:55 +02:00
Andrew Mayorov 9ae4de10ed
Merge pull request #13127 from keynslug/fix/EMQX-12439/update-leaner-error
fix(bridge-v2): report descriptive error on invalid update request
2024-05-28 13:11:41 +03:00
zmstone e9cc88cb95 test: fix test case to have deterministic base config 2024-05-28 12:03:01 +02:00
Kjell Winblad 84bc6ba28c fix: trace formatter should not crash when given a non-binary payload
Fixes:
https://emqx.atlassian.net/browse/EMQX-12474
2024-05-28 11:13:51 +02:00
Kjell Winblad e55bda6fe6 chore: bump app versions
Bump app version as required to pass the check which is done by the
following script:

./scripts/apps-version-check.sh
2024-05-28 11:01:00 +02:00
Kjell Winblad f65168982b fix(oracle action trace): parameters should not be rendered as IO Data
This forces the parameters to the database statement to be rendered as
a JSON array in JSON traces instead of being rendered as a string when
the parameters are interpreted as IO data.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12433
2024-05-28 10:59:44 +02:00
Andrew Mayorov c04aaad0a4
chore(bridge-v2): leave comment describing need for `nowarn_function` 2024-05-28 10:31:14 +02:00
ieQu1 8fbeca4321
chore: Version bumps 2024-05-28 00:14:01 +02:00
ieQu1 04305d004f
refactor(sessds): Extract metadata keys to a header 2024-05-27 23:36:20 +02:00
ieQu1 16d1425982
refactor(emqx): Move files related to durable session to a subdir 2024-05-27 23:31:15 +02:00
zmstone 2771a10d39 test: fix a flaky one 2024-05-27 20:59:50 +02:00
Andrew Mayorov 663f4fd39f
chore: bump application versions 2024-05-27 17:49:16 +02:00
Andrew Mayorov 08d88ea814
feat(bridge-api): improve error messages for Update Source API 2024-05-27 17:49:16 +02:00
Andrew Mayorov ed7c29ec26
fix(api): mention spec override in "request metadata" type 2024-05-27 16:57:45 +02:00
Andrew Mayorov 5c2a68076f
fix(bridge-v2): report descriptive error on invalid update request
Before this commit, generic validation errors were reported as union
mismatches of _all_ of the bridge schemas. After this commit, specific
schema is chosen before validation.
2024-05-27 14:37:10 +02:00
Ivan Dyachkov f393fd5304 chore: release 5.7.0 2024-05-27 14:32:23 +02:00
zmstone b2f5e50f16
Merge pull request #13120 from ieQu1/dev/ds-api-site-status
fix(dsapi): Check status of the node via Mria
2024-05-25 05:42:05 +02:00
zmstone 0763e29786 chore: prepare for 5.7.0-rc.2 release 2024-05-24 21:10:37 +02:00
ieQu1 73df01e934
fix(dsapi): Check status of the node via Mria 2024-05-24 20:59:07 +02:00
ieQu1 7f356aa3a8
Merge pull request #13115 from ieQu1/dev/fix-drop-generations2
Mitigate transient errors when dropping generations
2024-05-24 20:13:39 +02:00
ieQu1 830b62d899
fix(dsrepl): Retry sending ra commands to the leader 2024-05-24 19:09:34 +02:00
ieQu1 1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable 2024-05-24 19:09:34 +02:00
ieQu1 25721bceba
fix(dsstor): Delete generation metadata before dropping it 2024-05-24 19:09:33 +02:00
ieQu1 3d9837a0b8
test(ds): Add emqx_ds_replication_SUITE:t_drop_generation 2024-05-24 19:09:33 +02:00
Ilya Averyanov 327080e70e perf(emqx_placeholder): do not try to decode maps as json
fix
2024-05-24 20:08:31 +03:00
Andrew Mayorov c30b881239
test(monitor): verify historical and current metrics are consistent 2024-05-24 14:56:10 +02:00
Andrew Mayorov fd9655bc35
fix(monitor): unify metric names for consistency and simplicity 2024-05-24 14:55:48 +02:00
Andrew Mayorov 7b137c0aaf
fix(monitor): reflect durable subscriptions in historical metrics
Essentially in the same manner as they are currently reflected in the
current metrics.
2024-05-24 14:08:35 +02:00
zmstone 6ec8ae5a5f
Merge pull request #13112 from HJianBo/change-the-default-batch-size-s3-uploader
Fix(s3): change the batch_size to 1 for direct mode
2024-05-24 10:56:53 +02:00
Andrew Mayorov 29d203ae33
Merge pull request #13108 from keynslug/fix/ct-flaky/mgmt
test(mgmt): avoid inter-suite contamination by switching to cth tooling
2024-05-24 11:14:16 +03:00
JianBo He adaf238131 fix(s3): fix the convert function if the mode is missing 2024-05-24 16:02:06 +08:00
JimMoen 5b60f6a5b2
fix: make dialyzer happy 2024-05-24 14:29:02 +08:00
JimMoen 43506495ae
fix: ensure plugin configured states after join cluster 2024-05-24 13:51:56 +08:00
JimMoen 874bcd9a8b
fix: match decode plugin config map failed case 2024-05-24 10:20:50 +08:00
zmstone c3c6da7ec4 chore: prepare for release cut 5.7.0-rc.1 2024-05-23 21:39:15 +02:00
Zaiming (Stone) Shi 3a4b8e6c24
Merge pull request #13106 from zmstone/0523-add-customer-type
0523 add customer type (4=BYOC) to license schema
2024-05-23 21:30:02 +02:00
Zaiming (Stone) Shi 40288b99b2
Merge pull request #13101 from JimMoen/fix-plugin-config-map
fix: allow put plugin config without schema
2024-05-23 21:18:43 +02:00
Thales Macedo Garitezi 823a6bc521
Merge pull request #13107 from thalesmg/hide-clientv2-response-type-r57-20240523
fix(client mgmt api): hide unused return type
2024-05-23 16:03:51 -03:00
Andrew Mayorov 7878d106f6
test(mgmt): avoid inter-suite contamination by switching to cth tooling 2024-05-23 20:17:34 +02:00
Andrew Mayorov b4c6968f8c
Merge pull request #13104 from keynslug/fix/EMQX-12415/last-resort-poll
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
2024-05-23 21:17:03 +03:00
ieQu1 0dc507441a
Merge pull request #13105 from ieQu1/dev/sessds-renew-streams-interval
fix(sessds): Change the defaults to renew streams every 1 second
2024-05-23 18:23:20 +02:00
Thales Macedo Garitezi b1e3c941fb fix(client mgmt api): hide unused return type
Fixes https://emqx.atlassian.net/browse/EMQX-12427
2024-05-23 13:17:43 -03:00
Andrew Mayorov d47bf8076a
test(mgmt): better isolate testsuite running environments 2024-05-23 17:53:13 +02:00
zmstone c737045dff chore: add customer type 4 (BYOC) 2024-05-23 17:30:06 +02:00
ieQu1 5584b658c9
fix(sessds): Change the defaults to renew streams every 1 second 2024-05-23 17:17:29 +02:00
Zaiming (Stone) Shi 1d99ba2a8a
Merge pull request #13102 from JimMoen/fix-ds-prom-badkey
fix: add ds metrics into prometheus zipper init acc
2024-05-23 16:57:06 +02:00
Thales Macedo Garitezi 118c1d2806
Merge pull request #13103 from thalesmg/fix-another-ds-counter-nit-r57-20240523
fix(monitor api): fix cluster metric aggregation
2024-05-23 10:57:53 -03:00
Andrew Mayorov ba6382adae
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
This is a hack to work around the unintended issues causing shard
allocator to become idle even when there are pending transitions.
2024-05-23 14:54:01 +02:00
Andrew Mayorov 839b143fc4
fix(sessds): consider message durability disabled by default 2024-05-23 14:48:39 +02:00
Thales Macedo Garitezi c0e3a81c61 fix(monitor api): fix cluster metric aggregation
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-23 09:38:24 -03:00
Andrew Mayorov bf326acd7b
fix(dsrepl): handle stopping non-yet-running shard supervisor 2024-05-23 14:34:27 +02:00
Kjell Winblad 0e5b02c373
Merge pull request #13100 from kjellwinblad/kjell/s3_aggregate_only_has_result_trace/EMQX-12429
fix: add action rendered trace for s3 in aggregated mode
2024-05-23 14:29:18 +02:00
JimMoen a40c7d646a
fix: apply suggestions from code review, thanks @zmstone 2024-05-23 20:28:14 +08:00
JimMoen b91ecac168
refactor(prom): unify function name to avoid multiple levels call 2024-05-23 20:11:22 +08:00
JimMoen 065411c69c
fix: add ds metrics into prometheus zipper init acc 2024-05-23 20:11:21 +08:00
Kjell Winblad 89b47e8ffc fix(s3 tracing): do not format records as IO data in aggregate mode 2024-05-23 12:41:30 +02:00
Kjell Winblad fb7688ab94 fix(trace): make sure that the payload encode works with nested payloads
This commit makes sure that the trace setting for payload encode works
even when the payload is in a nested structure or when the payload key
is a binary instead of an atom.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12424
2024-05-23 12:30:56 +02:00
JimMoen 5dc96a37e2
fix: ensure plugin avsc serde before config 2024-05-23 17:46:58 +08:00
JimMoen 25a9aa1797
fix: allow put plugin config without schema 2024-05-23 17:33:34 +08:00
JimMoen 99869821fb
fix: ensure plugin config map after fresh install 2024-05-23 17:16:16 +08:00
Kjell Winblad e10c87b825 fix: add action rendered trace for s3 in aggregated mode
Fixes:
https://emqx.atlassian.net/browse/EMQX-12429
2024-05-23 10:25:27 +02:00
Andrew Mayorov 398dc97ed6
Merge pull request #13092 from keynslug/fix/dsrepl/site-autoleave
fix(dsrepl): properly handle transaction abort during forget site
2024-05-23 10:56:23 +03:00
ieQu1 950fa82be3
Merge pull request #13097 from ieQu1/dev/ds-replay-unrecoverable-no-test
feat(sessds): Handle unrecoverable errors
2024-05-23 09:49:21 +02:00
Kjell Winblad 26c988fe11
Merge pull request #13077 from kjellwinblad/kjell/fix_action_update_race_issue/EMQX-12376
fix: action config update would sometimes not be reflected in connector
2024-05-23 09:11:48 +02:00
Andrew Mayorov 2f6729d203
Merge pull request #13089 from keynslug/fix/EMQX-12437/restore
fix(s3-bridge): restore backward config compatibility
2024-05-23 10:10:01 +03:00
JianBo He 4053356597
Merge pull request #13063 from JimMoen/improve-plugin-config-behavior
fix(plugin): serde not found on new-joined nodes
2024-05-23 11:24:44 +08:00
lafirest b1076221b0
Merge pull request #13085 from lafirest/fix/rocket_namespace
fix(rocketmq): fix namespace error for RocketMQ
2024-05-23 10:39:30 +08:00
ieQu1 07774ab060
feat(sessds): Handle unrecoverable errors 2024-05-22 22:45:37 +02:00
Thales Macedo Garitezi 63cb12d7c2
Merge pull request #13095 from thalesmg/fix-monitor-ds-counters-again-r57-20240522
fix(monitor api): add ds connections and subscriptions to old counters
2024-05-22 16:43:15 -03:00
ieQu1 bfd01c425d
Merge pull request #13072 from ieQu1/dev/fix-drop-generation
Idempotent drop_generation
2024-05-22 21:13:30 +02:00
ieQu1 6eb04f90a3
fix(ds): Allow to write batches to older generations 2024-05-22 20:28:16 +02:00
Thales Macedo Garitezi 5cad4497de fix(monitor api): add ds connections and subscriptions to old counters
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-22 15:19:46 -03:00
Thales Macedo Garitezi 4094032649 fix(bridge v2 api): don't attempt to start disabled connector when starting action/source
Fixes https://emqx.atlassian.net/browse/EMQX-12435
2024-05-22 13:33:55 -03:00
Kjell Winblad 88c96e26de refactor: simplify the code with maps:map/2
Thanks @thalesmg for the suggestion
2024-05-22 18:03:43 +02:00
ieQu1 59a09fb86f
fix(ds): Apply review remarks 2024-05-22 18:01:33 +02:00
ieQu1 5c78ecba40
docs(ds): Update documentation for the storage layouts 2024-05-22 18:01:33 +02:00
ieQu1 29345aaa30
fix(ds): Fix idle event generation in bitfield_lts layout 2024-05-22 18:01:33 +02:00
ieQu1 b3ded7edce
fix(ds): Fix code review remark 2024-05-22 18:01:33 +02:00
ieQu1 60edf5e9b8
fix(ds): Move responsibility of returning end_of_stream to the CBM 2024-05-22 18:01:33 +02:00
ieQu1 0ff307e789
fix(ds): Include generation ID in the storage events
Make sure storage events originating from generation X are handled in
the context of the same generation.
2024-05-22 18:01:33 +02:00
ieQu1 acdae4fba3
fix(ds): Workaround for the idempotency error when dropping gens 2024-05-22 18:01:33 +02:00
ieQu1 eb7c43ee9d
fix(ds): Always store messages in the current generation 2024-05-22 18:01:33 +02:00
ieQu1 074d98a14a
test(ds): Refactor ds_SUITE 2024-05-22 18:01:33 +02:00
ieQu1 e4a73f003a
feat(ds): Implement format_status callback
Reduce volume of logs and crash reports from DS
2024-05-22 18:01:32 +02:00
ieQu1 1526c527d0
fix(ds): Log generation operations 2024-05-22 18:01:32 +02:00
ieQu1 aca2d9586c
fix(ds): Fix return type of drop_generation 2024-05-22 18:01:32 +02:00
ieQu1 c6fc76e335
fix(ds): Perform read operations on the leader. 2024-05-22 18:01:32 +02:00
ieQu1 4580906405
fix(ds): Use erpc instead of gen_rpc for `delete_next' 2024-05-22 18:01:32 +02:00
Andrew Mayorov 8016e9adf4
fix(s3-bridge): restore backward config compatibility 2024-05-22 17:59:52 +02:00
Kjell Winblad 4b540e3bd0 fix: do not leak action configurations in alarm messages 2024-05-22 17:43:07 +02:00
Andrew Mayorov e6c5c1b598
chore(dsrepl): provide more information in rebalancing log messages 2024-05-22 17:24:08 +02:00
Andrew Mayorov c355c9ad50
fix(dsrepl): properly handle transaction abort during forget site 2024-05-22 17:22:55 +02:00
Kjell Winblad cff8b97e8a fix: handle channel updated during health check
This commit fixes an issue found by CI test case
emqx_bridge_influxdb_SUITE:t_start_stop and others. While the channel
health check process is running, the channel could be removed or updated
which could cause a crash in the resource manager or non up-to-date
alarms being triggered.
2024-05-22 12:15:04 +02:00
Kjell Winblad 39d758c4d6 fix: do not return configs for channels from emqx_resource_manager 2024-05-22 12:15:04 +02:00
Kjell Winblad 917474f694 fix: action config update would sometimes not be reflected in connector
Before this commit the following happened sometimes:

1. action status is connected
2. action config is updated to something that should change the status to
   disconnected
3. action status is still connected and the old config is being used by
   the connector even though the config has been correctly updated.

The reason for this bug is that the post_config_hook runs before the
global EMQX config is updated. The post config hook is adding the new
config to the connector. Since the new config causes the action to get
status disconnected, the adding of the action to the connector is
retried when the health check runs but this time the config will be
loaded from the global config which could cause the old config to be
loaded (this happens when the global config has not had time to get
updated).

The above problem has been fixed in this commit by only reading action
configs from the global config when the connector starts/restarts and
instead store the latest configs for the actions in the connector.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12376
2024-05-22 12:15:04 +02:00
firest f3e8037e0f fix(rocketmq): fix namespace error for RocketMQ 2024-05-22 16:29:38 +08:00
Andrew Mayorov a3cd3e31b1
Merge pull request #13084 from keynslug/fix/EMQX-12422/part-size
fix(s3-bridge): report inconsistent `min/max_part_size` option
2024-05-22 09:48:09 +02:00
JimMoen e5f7aa9817
refactor: plguin functions and types rename 2024-05-22 11:51:54 +08:00
JimMoen 33aa61daea
fix: use hocon format as plugin config 2024-05-22 06:37:58 +08:00
JimMoen 14f2a68799
fix: bpapi spec type 2024-05-22 06:01:57 +08:00
Thales Macedo Garitezi 31a35f2a15
Merge pull request #13076 from thalesmg/fix-ds-session-count-client-api-r57-20240520
fix(client mgmt api): cache disconnected durable session count for `/clients` api
2024-05-21 15:34:21 -03:00
JimMoen 5abd23af5a
test: plugin refactor 2024-05-22 02:09:37 +08:00
JimMoen 87b3b214b9
fix: make static_check happy 2024-05-22 02:09:36 +08:00
JimMoen 3ce091e9d7
fix: ensure plugin tarball and extracted 2024-05-22 02:09:35 +08:00
JimMoen 1abc8cf502
fix: ensure plugin's config on boot and join cluster 2024-05-22 02:09:34 +08:00
JimMoen 677352e498
fix: ensure installed and plugin state on boot 2024-05-22 02:09:33 +08:00
JimMoen 2076681e76
fix: put plugin config with binary namevsn key 2024-05-22 02:09:32 +08:00
JimMoen 140b7ce51e
fix(plugin): schema content only provided in enterprise edition 2024-05-22 02:09:31 +08:00
JimMoen a7f2f95318
fix: ensure avro file 2024-05-22 02:09:30 +08:00
JimMoen df7dcb2764
fix: do not let plugin start failed lead emqx start failed 2024-05-22 02:09:29 +08:00
JimMoen e0e4517d9e
fix: ensure plugin config on boot 2024-05-22 02:09:28 +08:00
JimMoen e8d4e88118
fix(plugin): serde not found on new-joined nodes 2024-05-22 02:09:26 +08:00
Andrew Mayorov 28f6eb0200
fix(s3-bridge): report inconsistent `min/max_part_size` option 2024-05-21 18:49:01 +02:00
Thales Macedo Garitezi 7c5cb1acc5 fix(monitor current api): expose `disconnected_durable_sessions` count
Fixes https://emqx.atlassian.net/browse/EMQX-12423
2024-05-21 10:03:09 -03:00
Thales Macedo Garitezi 8b4a1c3d75 fix(client mgmt api): cache disconnected durable session count for `/clients` api
Fixes https://emqx.atlassian.net/browse/EMQX-12396
2024-05-20 11:48:08 -03:00
Kjell Winblad 8279d8c787
Merge pull request #13053 from kjellwinblad/kjell/fix_rule_trace_issues/EMQX-12327/EMQX-12335/EMQX-12336
Fix small action trace issues
2024-05-20 16:25:38 +02:00
Thales Macedo Garitezi a5d973b681 docs: hide `/clients_v2` from swagger api docs
Since it's not yet ready for production, we'll hide it from the API docs.
2024-05-20 10:00:43 -03:00
Kjell Winblad 3c87bcde46 fix(rule trace): restore logger metadata to its previous value 2024-05-20 10:57:15 +02:00
zmstone 6940930c3e chore: bump emqx_auth_http app vsn 2024-05-19 09:30:39 +02:00
zmstone 331d44a78a chore: improve Kafka producer health-check error logs
The Kafka producer lib wolff returns detailed error logs
about which host:port had issue, EMQX should log them to help
troubleshooting
2024-05-18 19:12:51 +02:00
Zaiming (Stone) Shi c6ee37bacd
Merge pull request #13068 from zmstone/0517-fix-dynamic-license-usage-for-dashboard
fix(license): show dynamic max_connections for bc
2024-05-18 09:49:18 +02:00
Andrew Mayorov fc3f525a67
Merge pull request #13066 from keynslug/fix/s3-bridge/desc-html
chore(s3-bridge): decrease default aggregation settings
2024-05-17 19:43:19 +02:00
zmstone b3afff9638 fix(license): show dynamic max_connections for bc 2024-05-17 17:44:21 +02:00
Thales Macedo Garitezi 388e6c8262 fix(dssubs): introduce separate gauge for subscriptions from durable sessions
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-17 10:17:16 -03:00
Kjell Winblad 5ce095f30e fix(rule trace): only include stop_after_rendering when value is true 2024-05-17 14:53:14 +02:00
Andrew Mayorov 385ac268d7
chore(s3-bridge): decrease default aggreg interval to 30 minutes 2024-05-17 14:49:18 +02:00
Andrew Mayorov c348f759af
chore(s3-bridge): decrease default aggreg limit to 100K records 2024-05-17 14:19:59 +02:00
Kjell Winblad 2e6db85578 fix(rule trace): rename rule_trigger_time(s) and cleaups
This commit renames trace fields rule_trigger_time and
rule_trigger_times to rule_trigger_ts and makes sure that the value for
rule_trigger_ts will always be a list of timestamps.
2024-05-17 14:13:36 +02:00
Andrew Mayorov 4d1db9f847
Merge pull request #13059 from keynslug/feat/EMQX-12204/single-action-schema
feat(s3-bridge): meld 2 separate actions into a single one
2024-05-17 13:27:40 +02:00
Kjell Winblad 2209b26fa5 fix(rule trace): do not leak trace meta data when tracing action result
Fixes:
https://emqx.atlassian.net/browse/EMQX-12391
2024-05-17 10:05:33 +02:00
ieQu1 5ba71ae6f7
Merge pull request #13062 from ieQu1/dev/ds-better-naming
fix(sessds): Rename last_alive_update_interval -> heartbeat_interval
2024-05-17 09:55:40 +02:00
Andrew Mayorov 39c2433afb
fix(s3-bridge): update config typespecs 2024-05-17 09:47:16 +02:00
Andrew Mayorov 86f99959b0
Merge pull request #13054 from keynslug/fix/EMQX-12365/node-leave
fix(dsrepl): anticipate and handle nodes leaving the cluster
2024-05-17 09:43:15 +02:00
Andrew Mayorov 5b2c2b336c
feat(s3-bridge): meld 2 separate actions into a single one
Discriminated through `mode` parameter. Also rename "simple" action to
a "direct" upload mode.
2024-05-17 09:34:21 +02:00
zhongwencool 1ac7aa151e
Merge pull request #13048 from thalesmg/sv-backup-r57-20240514
feat(schema validation): implement backup restore
2024-05-17 09:47:04 +08:00
ieQu1 6f30c1c541
fix(sessds): Rename last_alive_update_interval -> heartbeat_interval 2024-05-17 01:20:10 +02:00
ieQu1 73f17249e9
fix(sessds): Rename session_persistence to durable_sessions 2024-05-16 21:31:32 +02:00
ieQu1 ee6e7174cf
fix(sessds): Rename the durable messages DB to `messages` 2024-05-16 21:31:32 +02:00
Andrew Mayorov 5157e61418
fix(dsrepl): verify if shards already allocated first 2024-05-16 18:56:54 +02:00
Andrew Mayorov 0119728d45
feat(dsrepl): also reflect pending transitions in ds status 2024-05-16 18:56:21 +02:00
Andrew Mayorov 26c4a4f597
feat(dsrepl): reflect conflicts and inconsistencies in ds status 2024-05-16 18:32:08 +02:00
Andrew Mayorov 7e86e3e61c
fix(dsrepl): anticipate and handle nodes leaving the cluster
Also make `claim_site/2` safer by refusing to claim a site for a node
that is already there.
2024-05-16 18:32:07 +02:00
Kjell Winblad ed16e678be fix(rule apply test): add default values to context
Fixes:
https://emqx.atlassian.net/browse/EMQX-12378
2024-05-16 17:31:44 +02:00
Kjell Winblad 520e91c8fd fix(greptime trace): remove async info as it is confusing
Fixes:
https://emqx.atlassian.net/browse/EMQX-12385
2024-05-16 16:50:09 +02:00
Kjell Winblad b7c2f4a6d7 fix(best_effor_json): make tuple list check more efficient 2024-05-16 15:40:38 +02:00
Kjell Winblad ba74135079
Merge pull request #13052 from kjellwinblad/kjell/evaluate_lazy_trace_values_in_normal_log_formatters/EMQX-12380
fix: evaluate lazy values in normal log formatters when level is debug
2024-05-16 15:02:53 +02:00
Thales Macedo Garitezi 34a29e6363 feat(schema validation): implement support for `ctl conf load` 2024-05-16 09:55:42 -03:00
Zaiming (Stone) Shi 0be1249bbe
Merge pull request #13041 from zmstone/0513-http-authn-header-missing-should-result-in-meaningful-error-message
0513 http authn header missing should result in meaningful error message
2024-05-16 13:17:20 +02:00
ieQu1 71b393be89
Merge pull request #13057 from ieQu1/dev/default-n-shards
fix(ds): Change the default number of shards to 12
2024-05-16 13:05:34 +02:00
Kjell Winblad 413ad60bdb fix(cassandra): format rendered trace in a better way
Fixes:
https://emqx.atlassian.net/browse/EMQX-12393
2024-05-16 11:35:12 +02:00
Kjell Winblad 9fd8e930be fix(best_effort_json): only do tuple list to map without losing pairs 2024-05-16 11:19:48 +02:00
Kjell Winblad 72b0c63ed2
Merge pull request #13043 from kjellwinblad/kjell/kafka_incorrect_query_type/EMQX-12363
fix(kafka trace): do not include query type in trace entry
2024-05-16 10:08:16 +02:00
ieQu1 deffb01544
fix(ds): Change the default number of shards to 12 2024-05-16 08:06:07 +02:00
Thales Macedo Garitezi 2eb1bb2b13
Merge pull request #13032 from kjellwinblad/kjell/no_ctx_message_validation_failed/EMQX-12354
fix: add handling of message_validation_failed ctx for rule testing
2024-05-15 11:20:18 -03:00
Kjell Winblad 246bce85ec fix: make mysql action rendered trace include parameters
When doing non-batch inserts with the MySQL action, prepared statements
are used. We therefore need to include the parameters to the prepared
statement when tracing the rendered templates.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12335
2024-05-15 14:49:33 +02:00
Kjell Winblad c72682d81b fix: remove query mode from redis action trace as it only supports sync
Fixes:
https://emqx.atlassian.net/browse/EMQX-12336
2024-05-15 14:32:15 +02:00
Kjell Winblad 8cbfc4c0cd fix: evaluate lazy values in normal log formatters when level is debug
Fixes:
https://emqx.atlassian.net/browse/EMQX-12380
2024-05-15 11:47:21 +02:00
lafirest faf3492e58
Merge pull request #13039 from lafirest/fix/rocket_key
fix(rocketmq): fix a typo and a RocketMQ encoding error
2024-05-15 15:28:35 +08:00
Andrew Mayorov 3e1e4bab17
Merge pull request #13047 from keynslug/fix/EMQX-12366/api-ds-leave
feat(api-ds): provide more information on nonexistent site leave
2024-05-14 23:21:07 +02:00
Thales Macedo Garitezi fd949240c0 feat(schema validation): implement backup restore
Fixes https://emqx.atlassian.net/browse/EMQX-12346
2024-05-14 15:58:19 -03:00
Thales Macedo Garitezi bc5071590b
Merge pull request #13033 from thalesmg/fix-ds-more-client-api-fields-r57-20240513
fix(ds clients mgmt api): add more fields to disconnected durable sessions
2024-05-14 15:29:40 -03:00
Thales Macedo Garitezi 7023e6ad96
Merge pull request #13017 from thalesmg/fix-ds-subs-pages-r57-20240510
fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page
2024-05-14 15:29:29 -03:00
Kjell Winblad e811f4a392 fix: add handling of message_validation_failed ctx for rule testing
Before this commit, the backend could not handle the test context for
message_validation_failed events that could be sent from the dashboard
since no handling code for this contexts existed in the backend. This
has now been fixed by adding such handling code.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12354
2024-05-14 13:07:32 -03:00
Thales Macedo Garitezi 1730a41337
Merge pull request #13046 from thalesmg/refactor-mv-rename-sv-r57-20240514
refactor: rename `message_validation` to `schema_validation`
2024-05-14 13:03:12 -03:00
Kjell Winblad adbc392265
Merge pull request #13027 from kjellwinblad/kjell/fix_trace_json_report_formatting/EMQX-12344
fix: JSON trace formatter should handle report style log entries
2024-05-14 18:02:59 +02:00
firest 437a0c0d55 fix(rocketmq): fix a typo and a RocketMQ encoding error 2024-05-14 23:01:31 +08:00
Andrew Mayorov 2f90d15ce6
fix(ds-api): adapt CLI to API changes 2024-05-14 16:55:17 +02:00
Andrew Mayorov 35e360fcbe
feat(api-ds): provide more information on nonexistent site leave 2024-05-14 15:57:41 +02:00
Thales Macedo Garitezi 257dae5211 refactor: rename `message_validation` to `schema_validation`
Request from Product team.

Fixes https://emqx.atlassian.net/browse/EMQX-12383
2024-05-14 10:32:27 -03:00
Thales Macedo Garitezi 15acd86b31
Merge pull request #13034 from thalesmg/test-flaky-mq-r57-20240513
test(client mgmt api): attempt to fix flaky test
2024-05-14 09:01:29 -03:00
Thales Macedo Garitezi e8d2c48d3c
Merge pull request #13018 from thalesmg/fix-postgres-log-spam-r57-20240510
fix(postgres): reduce log spamming when connection goes down
2024-05-14 09:00:36 -03:00
Kjell Winblad f75c7a5cea fix(trace log entry error): better structure and naming 2024-05-14 13:38:41 +02:00
Andrew Mayorov 74ed0f578a
Merge pull request #13042 from keynslug/fix/EMQX-12204/api-ns
fix(s3-aggreg): disambiguate action schema namespace
2024-05-14 13:12:34 +02:00
Kjell Winblad 121084b2ff fix(kafka trace): do not include query type in trace entry
For Kafka that has its internal buffering the on_query_async callback is
used both for sync and async queries (for sync queries the caller waits
for a response directly after calling on_query_async). Therefore, before
this commit, the trace included incorrect query type information. This
has been fixed by removing the query type information from the trace.
This should be okay since the query type is not essential information
and can be derived from the configuration.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12363
2024-05-14 12:24:30 +02:00
ieQu1 525e4dac95
Merge pull request #13036 from ieQu1/dev/reduce-log-spam
tests: Reduce log spam in the failed test suites
2024-05-14 10:53:30 +02:00
Andrew Mayorov c10a48f4be
fix(s3-aggreg): disambiguate action schema namespace
Otherwise schemas for `s3` and `s3_aggregated_upload` collide when
projected into OpenAPI schema.
2024-05-14 10:49:21 +02:00
zmstone 93232d4253 fix(authn/http): log meaningful error message if http header is missing 2024-05-14 10:22:07 +02:00
Andrew Mayorov 98a93662c1
Merge pull request #13035 from keynslug/fix/EMQX-12291/pers-sys-msg
fix(sessds): persist $SYS messages as well
2024-05-14 10:05:49 +02:00
ieQu1 ac3f5a083d
test: Reduce log spam in the failed test suites 2024-05-13 22:00:33 +02:00
ieQu1 8506ca7919
Merge pull request #12998 from ieQu1/dev/improve-latency
Use leader's clock when calculating LTS cutoff timestamp
2024-05-13 21:54:06 +02:00
Andrew Mayorov 66b7ac4c45
fix(sessds): persist $SYS messages as well
Otherwise, persistent sessions will not be able to receive $SYS messages
whatsoever.
2024-05-13 21:27:55 +02:00
Thales Macedo Garitezi a2a5fd1f8e test(client mgmt api): attempt to fix flaky test
```
%%% emqx_mgmt_api_clients_SUITE ==> msgs_plain_encoding.t_mqueue_messages: FAILED
%%% emqx_mgmt_api_clients_SUITE ==>
Failure/Error: ?assertEqual(79, Count)
  expected: 79
       got: 100
      line: 1303
```
2024-05-13 15:53:21 -03:00
ieQu1 3da3a36863
test(ds): Add generation in the replication suite 2024-05-13 19:51:04 +02:00
ieQu1 9f7ef9f34f
fix(ds): Apply review remarks 2024-05-13 19:35:24 +02:00
Thales Macedo Garitezi e7c47f4ec0 fix(ds clients mgmt api): add more fields to disconnected durable sessions
Fixes https://emqx.atlassian.net/browse/EMQX-12369
2024-05-13 14:25:02 -03:00
Kjell Winblad 1c34a84dd2
Merge pull request #13028 from kjellwinblad/kjell/swagger_example_missing_ruleid_paramter/EMQX-12325
fix: include ruleid parameter in swagger example for apply rule endpoint
2024-05-13 17:35:31 +02:00
Thales Macedo Garitezi 81473519bd
Merge pull request #13031 from thalesmg/fix-clientsv2-sec-r57-20240513
fix(client mgmt api): require authn for endpoint
2024-05-13 12:17:31 -03:00
Thales Macedo Garitezi 39899b917d fix(client mgmt api): require authn for endpoint
Fixes https://emqx.atlassian.net/browse/EMQX-12361
2024-05-13 10:54:21 -03:00
Thales Macedo Garitezi 6b032faebc fix(client mgmt api): return expired durable sessions with `is_expired: true`
Fixes https://emqx.atlassian.net/browse/EMQX-12274
2024-05-13 10:51:24 -03:00
Thales Macedo Garitezi 0dee2d67b1 fix: always check if first query has more rows 2024-05-13 09:10:18 -03:00
Kjell Winblad c8690a8273 fix: include ruleid parameter in swagger example for apply rule endpoint
Fixes:
https://emqx.atlassian.net/browse/EMQX-12325
2024-05-13 12:28:17 +02:00
Kjell Winblad 921d82f083 fix: JSON trace formatter should handle report style log entries
Fixes:
https://emqx.atlassian.net/browse/EMQX-12344
2024-05-13 12:11:11 +02:00
zmstone 7faf73e0f4 chore(emqx_schema): change atom array to enum array for alarm.actions 2024-05-13 10:54:33 +02:00
JianBo He 54f6bd181b
Merge pull request #13010 from JimMoen/fix-jt808-reg-failed
fix(gw): jt808 REG_ACK failed due to faulty clientinfo
2024-05-13 14:19:18 +08:00
JimMoen 2f212f34a3
Merge pull request #12989 from JimMoen/fix-plugin-config-file-backup
fix(plugin): add a backup for the plugin config file
2024-05-11 16:38:36 +08:00
JimMoen a5ecceb755
chore: apply suggestions from code review, thanks @thalesmg 2024-05-11 11:36:40 +08:00
Thales Macedo Garitezi e25fe62cbc fix(subs mgmt api): attempt to return mixed clients ds/non-ds in the same page
Fixes https://emqx.atlassian.net/browse/EMQX-12294
2024-05-10 14:48:46 -03:00
Kjell Winblad c6df069c5a
Merge pull request #13012 from kjellwinblad/kjell/fix_incorrect_config_crash/EMQX-12315
fix: listener crash if access_rules config option is incorrect
2024-05-10 17:46:40 +02:00
Thales Macedo Garitezi f690db9849 fix(postgres): reduce log spamming when connection goes down
Fixes https://emqx.atlassian.net/browse/EMQX-12334

See also: https://github.com/emqx/epgsql/pull/10
2024-05-10 12:15:01 -03:00
JimMoen 3c7e8a3bd5
test(gw): jt808 register failed 2024-05-10 23:09:22 +08:00
JimMoen f8fd4d15be
fix(gw): process jt808 client deregister packet 2024-05-10 23:00:59 +08:00
Zaiming (Stone) Shi c491b83857
Merge pull request #13014 from zmstone/0510-add-is_template-for-kinesis-partition-key
fix(kinesis): partition_key should be of 'template' type
2024-05-10 16:11:35 +02:00
Kjell Winblad b13fa37771 fix: dialyzer warning 2024-05-10 15:51:28 +02:00
Thales Macedo Garitezi 0eaef18391
Merge pull request #13005 from thalesmg/fix-aggreg-sup-tree-r57-20240509
fix(aggregator): refactor supervision tree
2024-05-10 10:22:33 -03:00
Kjell Winblad 2bed5894e3 test: add test cases for listeners access_rules validation and split 2024-05-10 14:21:43 +02:00
Thales Macedo Garitezi 3a29696a48 fix(aggregator): refactor supervision tree
Instead of using a aggregator supervisor with a fixed local name, we should allow
specifying different names so each action app will spawn its own aggregator supervisor.
2024-05-10 09:06:12 -03:00
zmstone 47df968d4a fix(kinesis): partition_key should be of 'template' type 2024-05-10 13:57:38 +02:00
Kjell Winblad 0aeb2cd77f fix: listener crash if access_rules config option is incorrect
Previously when changing the access_rules configuration option of a
listener to something that was not a valid rule, the listener would
crash. This has now been fixed by the addition of a configuration
validator that checks the access_rules field.

Additionally, a configuration option converter has been added to the
access_rules field so that one can specify several rules in a single
string by using "," (comma) as separator.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12315
2024-05-10 12:27:04 +02:00
JimMoen 1a6f0c9234
chore: bump gw_jt808 app vsn 2024-05-10 17:13:23 +08:00
JimMoen b07242515f
fix(gw): jt808 REG_ACK failed due to faulty clientinfo 2024-05-10 17:12:28 +08:00
Kjell Winblad 8c0b55a89e
Merge pull request #12992 from kjellwinblad/kjell/fix_rule_unescape_function/EMQX-12313
fix: make rule engine unescape convert \a to the terminal alarm char
2024-05-10 09:48:33 +02:00
lafirest da7b11020a
Merge pull request #13001 from lafirest/fix/sysk_reconn
fix(sysk): fix that the syskeeper forwarding never reconnecting
2024-05-10 11:24:28 +08:00
firest 1f3b640a3d chore: update change 2024-05-10 07:29:14 +08:00
Thales Macedo Garitezi 7b80a9aa44 fix(aggregator): namespace aggregator ids with action type
Otherwise, actions of different types but same names will clash when starting the
aggregator supervision tree.
2024-05-09 16:02:11 -03:00
Andrew Mayorov 57dda70d2f
Merge pull request #12994 from keynslug/fix/sessds/log-levels
fix(sessds): use milder log level for regular error conditions
2024-05-09 18:34:54 +02:00
firest e3a59c4037 fix(sysk): fix that the syskeeper forwarding never reconnecting 2024-05-09 16:27:58 +08:00
Ilia Averianov a1aedee446
Merge pull request #12996 from savonarola/0508-fix-cursor-usage
fix(retainer): fix qlc cursor cleanup
2024-05-09 08:04:10 +03:00
ieQu1 07aa708894
test(ds): Refactor replication suite 2024-05-09 03:56:56 +02:00
ieQu1 63e51fca66
test(ds): Use streams to fill the storage 2024-05-09 02:46:57 +02:00
ieQu1 a0a3977043
feat(ds): Assign latest timestamp deterministically 2024-05-08 23:17:57 +02:00
ieQu1 2236af84ba
feat(ds): two-stage storage commit on the storage level 2024-05-08 23:17:57 +02:00
ieQu1 f250f00f3f
chore: Bump snabbkaffe version to 1.0.10 2024-05-08 23:17:57 +02:00
ieQu1 1ddbbca90e
feat(ds): Allow incremental update of the LTS trie 2024-05-08 23:17:57 +02:00
ieQu1 68ca891f41
test(ds): Use streams to create traffic 2024-05-08 23:17:57 +02:00
Zaiming (Stone) Shi 018d79b498
Merge pull request #12993 from zmstone/0508-fix-update-listener-zone-from-dashboard
0508 fix update listener zone from dashboard
2024-05-08 22:03:10 +02:00
Thales Macedo Garitezi 55f27281b3
Merge pull request #12991 from thalesmg/refactor-mv-message-type-r57-20240508
refactor(message validation): rename `message_name` to `message_type`
2024-05-08 16:58:28 -03:00
Thales Macedo Garitezi d3905b6336
Merge pull request #12990 from thalesmg/fix-mv-conf-update-r57-20240508
fix(message validation): don't call `emqx_conf:update` during boot
2024-05-08 16:58:15 -03:00
Ilya Averyanov 57287f0722 fix(retainer): fix qlc cursor cleanup 2024-05-08 21:38:37 +03:00