yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,014 Commits 393 Branches 684 Tags 74 MiB
e99fee68c0
Commit Graph

12393 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
9b3c806ba7 feat(message transformation): implement dryrun endpoint 
Follow up to https://github.com/emqx/emqx/pull/13199
 2024-06-14 10:19:36 -03:00
zhongwencool
a6e3a09118 feat: make the dashboard restart quicker 2024-06-14 18:22:34 +08:00
JimMoen
0f0e7d18db
Merge pull request #13261 from JimMoen/fix-typo 
chore: fix typo
 2024-06-14 16:55:35 +08:00
JimMoen
6823c79ae0
chore: fix typo 2024-06-14 16:04:11 +08:00
Thales Macedo Garitezi
6ccf1dcbf9
Merge pull request #13238 from thalesmg/fix-authz-http-content-type-handling-r57-20240612 
fix(http authz): handle unknown content types in responses
 2024-06-13 09:10:43 -03:00
Thales Macedo Garitezi
62cf277cd7
Merge pull request #13235 from thalesmg/fix-template-types-r57-20240611 
chore: change types of mysql and mongodb fields to `template()`
 2024-06-13 09:08:40 -03:00
Thales Macedo Garitezi
c81a3ebc0a
Merge pull request #13234 from thalesmg/fix-clients-api-client-attrs-union-r57-20240611 
fix(client mgmt api): allow projecting `client_attrs` from client fields
 2024-06-13 09:08:29 -03:00
William Yang
723144199e
Merge pull request #13211 from qzhuyan/fix/william/rel57/ee-tls 
Fix/william/rel57/ee tls
 2024-06-13 13:19:44 +02:00
Andrew Mayorov
0f53b33417
chore(s3): make schema fields with defaults optional 2024-06-13 11:57:36 +02:00
Thales Macedo Garitezi
2c264d9a4b fix(http authz): handle unknown content types in responses 
Fixes https://emqx.atlassian.net/browse/EMQX-12530
 2024-06-12 14:31:12 -03:00
JimMoen
daeb10151b
chore: fix typos 2024-06-12 19:06:34 +08:00
JimMoen
cde4cb1358
fix: metrics macro with description 2024-06-12 19:06:33 +08:00
JimMoen
756797b258
refactor: gen metric schema with desc from macros 2024-06-12 19:04:39 +08:00
JimMoen
263d2dbae2
fix: inc metric 'client.auth.anonymous' when client anonymous 2024-06-12 19:04:39 +08:00
JimMoen
9a78a6f640
refactor: mv metrics macros in hrl file 2024-06-12 19:04:29 +08:00
JimMoen
cbac4019b3
fix: rm havn't used counter idx 2024-06-12 19:01:50 +08:00
Thales Macedo Garitezi
46ab3be1f4 chore: change types of mysql and mongodb fields to template() 
Fixes https://emqx.atlassian.net/browse/EMQX-12395
 2024-06-11 17:25:53 -03:00
Thales Macedo Garitezi
0874768c1d fix(client mgmt api): allow projecting client_attrs from client fields 
Fixes https://emqx.atlassian.net/browse/EMQX-12511
 2024-06-11 17:14:33 -03:00
zmstone
8a283b1cc0
Merge pull request #13180 from zmstone/0604-optimize-tcp-ssl-connection-data-send-performance 
0604 optimize tcp ssl connection data send performance
 2024-06-11 22:06:57 +02:00
zmstone
09f91159c9 fix(emqx_rule_funcs): expose regex_extract function to rule engine 
previoulsy only available in variform expressions,
now made available for rule-engine
 2024-06-11 21:31:13 +02:00
zmstone
51810c6919
Merge pull request #13229 from zmstone/0611-auth-with-raw-cert 
feat(authn): support `${cert_pem}` placeholder
 2024-06-11 21:16:01 +02:00
zmstone
751f7a24e9 feat(authn): support ${cert_pem} placeholder 2024-06-11 19:54:24 +02:00
Thales Macedo Garitezi
337009c3a0 fix: declare emqx_schema_registry as a dependency of emqx_schema_validation 2024-06-11 14:03:27 -03:00
Thales Macedo Garitezi
0f9c3b4cea feat(schema validation): check references schema names and types before changing config 
Fixes https://emqx.atlassian.net/browse/EMQX-12368
 2024-06-11 14:03:26 -03:00
Thales Macedo Garitezi
03b226248a feat(schema registry): add check for inner types 
Currently, only `protobuf` has any.
 2024-06-11 13:50:11 -03:00
Thales Macedo Garitezi
c70e8252fe fix: declare emqx_schema_registry as a dependency of emqx_message_transformation 2024-06-11 13:35:02 -03:00
Thales Macedo Garitezi
2c6aa80b3c
Merge pull request #13221 from thalesmg/refactor-sv-topic-index-r57-20240610 
refactor(schema validation): module organization and position indexing
 2024-06-11 13:29:48 -03:00
Thales Macedo Garitezi
83ff07ade0
Merge pull request #13199 from thalesmg/message-transformation-r57-20240604 
feat: implement message transformation
 2024-06-11 13:29:25 -03:00
Thales Macedo Garitezi
ab5329be02
Merge pull request #13220 from thalesmg/refactor-channel-hc-opt-r57-20240610 
fix: remove another redundant health check
 2024-06-11 12:22:43 -03:00
Andrew Mayorov
2f0eb3560b
Merge pull request #13227 from keynslug/fix/EMQX-12533/bad-tpl 
fix(bridge-s3): validate aggreg key template before adding a channel
 2024-06-11 15:39:12 +03:00
Andrew Mayorov
d4fb812352
fix(aggreg): anticipate and handle delivery startup errors 2024-06-11 13:21:41 +02:00
Andrew Mayorov
29fc30ea69
fix(bridge-s3): validate aggreg key template before adding a channel 2024-06-11 13:21:41 +02:00
Andrew Mayorov
fb0da9848c
feat(tpl): add separate placeholders/1 function 
The purpose is to have a clearer view of placeholders used in a
template, without going the usual `render(Template, #{})` route that is
actually subtly misleading: it won't mention that `${}` / `${.}`
placeholder has been used.

Also unify handling of `${}` / `${.}` in a couple of places.
 2024-06-11 11:42:43 +02:00
Kjell Winblad
9d22089104
Merge pull request #13173 from kjellwinblad/kjell/pgsql_timestamp_encode_etc/EEC-1036 
fix(pgsql connector): handle prepared statement already exists
 2024-06-11 10:53:30 +02:00
William Yang
44258204bd chore: move tls_certs test helper to apps/emqx 2024-06-11 10:06:18 +02:00
zhongwencool
b657dc537c
Merge pull request #13156 from savonarola/0529-fix-dashboard-monitor 
fix(dashboard_monitor): correctly merge old and new sample data
 2024-06-11 10:33:15 +08:00
Thales Macedo Garitezi
05ebb17cd6 refactor: index positions rather than names 
Addresses https://github.com/emqx/emqx/pull/13199#discussion_r1633096025
 2024-06-10 15:54:37 -03:00
Thales Macedo Garitezi
5eff4a7544 refactor(schema validation): improve api and module organization 
To conform to comment from another PR:
https://github.com/emqx/emqx/pull/13199#discussion_r1632658742

... since schema validation is the "inspiration" for message transformation.
 2024-06-10 15:46:28 -03:00
Thales Macedo Garitezi
d93c8540c8 feat: pass along peername down to transformation context 2024-06-10 15:35:35 -03:00
Thales Macedo Garitezi
6e0ef893f4 feat: pass along client attributes down to message transformation context 2024-06-10 15:27:04 -03:00
Thales Macedo Garitezi
3c501e4f2a fix: remove another redundant health check 
Fixes https://emqx.atlassian.net/browse/EMQX-12521
 2024-06-10 15:13:34 -03:00
Thales Macedo Garitezi
1a20a40218
Merge pull request #13209 from thalesmg/rm-redundant-hc-r57-20240607 
chore: remove redundant health check call
 2024-06-10 14:58:48 -03:00
Thales Macedo Garitezi
5629fe60c1 refactor: index positions rather than names 2024-06-10 13:33:22 -03:00
Thales Macedo Garitezi
8c7a7cf9db refactor: apply review remarks 2024-06-10 11:17:13 -03:00
Thales Macedo Garitezi
e54cf2f218 refactor: use option to skip rendering as string 2024-06-10 10:49:39 -03:00
Thales Macedo Garitezi
d34558954d refactor: improve api and module organization 2024-06-10 10:40:53 -03:00
Thales Macedo Garitezi
4aa1c3842f fix: declare emqx as a dependency in .app.src 2024-06-10 09:53:29 -03:00
Thales Macedo Garitezi
aa416cc0ad docs: fix stale comments and dates 2024-06-10 09:53:02 -03:00
zmstone
7a09b0470d chore: cosmetic changes 2024-06-10 14:22:25 +02:00
Kjell Winblad
a885f0b41a test(emqx_bridge_pgsql_SUITE): call test janitor 2024-06-10 13:45:27 +02:00
William Yang
a1aa9a4375 fix(ee): emqx no longer deps on emqx_auth_ext 2024-06-10 10:32:01 +02:00
zmstone
4347f3de3e fix(bridge/mqtt): respect client ID prefix 2024-06-10 10:27:45 +02:00
William Yang
76cfc309a9 feat(tls): partial_chain not required 2024-06-10 10:25:44 +02:00
William Yang
7c37bf9965 feat(tls): ee only: TLS partial_chain and Keyusage 2024-06-10 10:24:48 +02:00
William Yang
e60380d205 chore: fix nit 2024-06-10 09:55:12 +02:00
William Yang
e9b813d8ef chore: fix test 2024-06-10 09:54:54 +02:00
William Yang
1739bc0c24 feat(partial_chain): gateway support 2024-06-10 09:54:44 +02:00
William Yang
abbf2ef62f chore(TLS-chain-test): update for OTP 26 2024-06-08 00:16:06 +02:00
William Yang
221b748b0f test(partial_chain): update tcs for OTP-25 2024-06-08 00:16:01 +02:00
William Yang
463d1a1875 fix(test): tls_verify_partial_chain 2024-06-08 00:15:54 +02:00
William Yang
788cdbc6dd fix(listener): remove partial_chain in wss opts 2024-06-08 00:15:51 +02:00
William Yang
3fc99315e0 chore: happy elvis 2024-06-08 00:15:45 +02:00
William Yang
c5dccdf526 feat(tls): update schema for TLS keyusage 2024-06-08 00:15:43 +02:00
William Yang
aa25e3badd fix(tls): undefined keyusage 2024-06-08 00:15:39 +02:00
William Yang
9e5cfea8c0 test(tls): verify peer keyusage 2024-06-08 00:15:36 +02:00
William Yang
c5657029ab feat(config): partial_chain 2024-06-08 00:15:34 +02:00
William Yang
f7ff9496e6 test: port listener tls partial_chain 2024-06-08 00:15:31 +02:00
William Yang
1ce13242a8 feat(tls): port partial_chain, part 1 2024-06-08 00:15:28 +02:00
Kjell Winblad
c13631102e fix(PgSQL connector): typo in function name 2024-06-07 19:40:18 +02:00
Andrew Mayorov
b2f027c974
Merge pull request #13197 from keynslug/fix/EMQX-12522/conv-ssl 
fix(bridge-s3): pass SSL options through `convert_certs/2`
 2024-06-07 19:45:53 +03:00
Thales Macedo Garitezi
28a3c77e88 chore: remove redundant health check call 
Since `emqx_resource_manager:channel_health_check` already does a base resource health
check, the call was repeated.
 2024-06-07 11:07:12 -03:00
Kjell Winblad
e63dcc84b0 fix: unused variable and better error message 2024-06-07 12:20:42 +02:00
JimMoen
78a196e215
Merge pull request #13189 from JimMoen/fix-apply-sqlserver-template 
fix(utils_sql): improve insert sql regular expression
 2024-06-07 13:50:19 +08:00
lafirest
eb215726d5
Merge pull request #13195 from lafirest/fix/typos 
fix: fix typos
 2024-06-07 09:04:16 +08:00
Thales Macedo Garitezi
ef36350bf6 feat: implement message transformation 
Fixes https://emqx.atlassian.net/browse/EMQX-12517
 2024-06-06 17:51:08 -03:00
Andrew Mayorov
8a85d7cb5a
fix(bridge-s3): pass SSL options through convert_certs/2 2024-06-06 19:37:38 +02:00
Andrew Mayorov
c17c788438
feat(connector): allow delegate pre_config_update to impls 2024-06-06 19:37:34 +02:00
zmstone
2525bd0c16 perf: eliminate the loop-back fake async inet_reply message 
Now that EMQX is on OTP 26 by default, the fake inet_reply
message when the send call returns ok only adds overhead.

For TCP (gen_tcp module), OTP 26 no longer supports direct port_command
so we had to swith to gen_tcp:send, then fake an async reply:
see https://github.com/emqx/esockd/pull/181

For TLS (ssl module), it has never supported async send anyway.

This commit switches to call Transport:send/2 directly,
this should result in one less loop-back message for each message.

For OTP 25 on which EMQX can still be compiled, one will have to
suffer the performance penalty for gen_tcp:send function to
do a non-optimized 'receive' for the send result.
 2024-06-06 09:35:51 +02:00
zmstone
5c3f5d8085 perf: do not call inet getstat before each and every send 
In a stress test environment, when alarm is enabled, there were
a lot of long_schedule warnings with the stacktrace pointing to
congestion alarm based inet:getstat function.
When alarm is disabled, the one single client is able to
hold 40,000 QoS 1 messages per second (inflight=32) throughput.
When alarm is enabled, its mqueue overflows.

This commit removes the call before each and every data send,
so to rely on emit_stats timer to trigger the congestion check.
 2024-06-06 09:35:51 +02:00
firest
d4118c6e8e fix: fix typos 2024-06-06 11:19:40 +08:00
JimMoen
dff31b293c
perf(utils_sql): pre-build regexp 2024-06-06 11:01:38 +08:00
zmstone
c41be7ee22 chore: fix dialyzer warning 2024-06-05 23:37:21 +02:00
JimMoen
144264e0d8
test: add utils_sql test cases 2024-06-06 00:30:18 +08:00
JimMoen
627b8c45d2
fix(utils_sql): improve insert sql regular expression 2024-06-06 00:30:17 +08:00
Ilya Averyanov
60556abc4f fix(dashboard_monitor): correctly merge old and new sample data 2024-06-05 18:49:52 +03:00
Kjell Winblad
2956e849eb fix(pgsql connector): better msg when failing to remove statement 2024-06-05 16:58:14 +02:00
Kjell Winblad
336089f8a7 fix: bug found by dialyzer and make test case cleaner 2024-06-05 15:53:02 +02:00
Thales Macedo Garitezi
ae0379f974
Merge pull request #13181 from thalesmg/force-stop-connector-r57-20240604 
fix(resource manager): force kill process if stuck when stopping/removing
 2024-06-05 09:08:30 -03:00
lafirest
d79193fde4
Merge pull request #13177 from lafirest/sync-authn-com 
feat: sync `client.check_authn_complete` to release-57
 2024-06-05 19:51:10 +08:00
Thales Macedo Garitezi
5eab221f7c
Merge pull request #13175 from thalesmg/disable-prepared-statements-postgres-r57-20240603 
feat(postgres): add `disable_prepared_statements` option
 2024-06-04 12:29:53 -03:00
Thales Macedo Garitezi
f17aefe3d7 fix(action/source api): improve returned error message on timeout 2024-06-04 12:04:55 -03:00
zmstone
cda6d5f636
Merge pull request #13172 from zmstone/0603-add-redis-arg-formatter-to-rule-engine-funcs 
feat(redis): add a rule function to help formatting redis args
 2024-06-04 16:56:14 +02:00
Thales Macedo Garitezi
3013189cd7 fix(resource manager): force kill process if stuck when stopping/removing 
Fixes https://emqx.atlassian.net/browse/EMQX-12357
 2024-06-04 11:38:24 -03:00
Thales Macedo Garitezi
c07bc68e6f feat(postgres): add disable_prepared_statements option 
Fixes https://emqx.atlassian.net/browse/EMQX-12496

Some Postgres connections, such ones made to [PGBouncer](https://www.pgbouncer.org/) or
[Supabase in Transaction Mode](https://supabase.com/), do not support some session
features like prepared statements.
 2024-06-04 10:11:25 -03:00
firest
87384cae32 fix(events): call client.check_authn_complete even if authentication fails 2024-06-04 17:55:39 +08:00
firest
92d26ff27b chore: update change 2024-06-04 17:55:30 +08:00
firest
e4e53844d5 feat(events): add new hook && event client.check_authn_complete 2024-06-04 17:55:21 +08:00
Kjell Winblad
572ca6433e fix(pgsql connector): improvements due to suggestions from @thalesmg 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-06-04 10:11:45 +02:00
zmstone
aa7ce1f641 fix(bridge/redis): add test case for map_to_redis_hset_args 2024-06-03 23:05:43 +02:00
Kjell Winblad
f2ccfff803 fix(pgsql connector): handle prepared statement already exists 
In a user's log file it was found that that the pgsql driver can end up
in a situation where the prepared statement for a channel/action is not
properly removed before a channel with the same name as the prepared
statement is added to the connector. This commit handles this by
attempting to remove the old prepared statement if one already exists
when adding channel.

Related issue:
https://emqx.atlassian.net/browse/EEC-1036
 2024-06-03 16:14:26 +02:00
zmstone
e3ed7b59dd feat(redis): add a rule function to help formatting redis args 
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.

This new function is dedicated for redis to avoid abuse for other
data integrations.
 2024-06-03 14:09:15 +02:00
JimMoen
21624bc865
fix: get plugin app vsn from field rel_apps 2024-06-03 15:26:14 +08:00
zmstone
b6ff67d712
Merge pull request #13093 from zmstone/0522-upgrade-to-brod-3.18 
chore: upgrade to brod 3.18.0
 2024-06-02 15:06:40 +02:00
JimMoen
8d22a5a3ae
Merge pull request #13167 from JimMoen/fix-rule-events-i18n-todo 
fix: missing rule event i18n in zh_cn
 2024-06-02 18:23:20 +08:00
zmstone
9637b14543 fix: upgrade to brod 3.18.0 to fix unnecessary rebalance 2024-06-01 10:57:24 +02:00
zmstone
37ce8d93a1
Merge pull request #13164 from zmstone/0530-http-acl-compatibility-with-v4 
0530 http acl compatibility with v4
 2024-05-31 16:28:30 +02:00
Thales Macedo Garitezi
c289c7d065
Merge pull request #13148 from thalesmg/EMQX-12251-fix-conn-api-timeout-case-clause-r571-20240528 
fix(connector api): handle `timeout` when waiting for connector status
 2024-05-31 10:07:35 -03:00
zmstone
6fe8a09e97 fix(authz/http): rename PH_ACCESS to VAR_ACCESS 2024-05-31 14:49:54 +02:00
JimMoen
45e2c81ebc
fix: missing rule event i18n in zh_cn 2024-05-31 17:51:47 +08:00
zmstone
ca32cf8d29 Merge branch 'emqx-12487-fix-authz-http-content-type' into 0530-http-acl-compatibility-with-v4 2024-05-30 22:30:20 +02:00
zmstone
a00d9f17b1 chore: upgrade kafka producer lib wolff to 1.10.4 
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
 2024-05-30 15:40:10 +02:00
zmstone
460081a22e
Merge pull request #13131 from zmstone/0527-handle-unicode-in-template-strings 
0527 handle unicode in template strings
 2024-05-30 15:37:43 +02:00
Ilya Averyanov
97f9c81e19 feat(auth): add legacy ${access} placeholder 2024-05-30 15:47:48 +03:00
Ivan Dyachkov
0c4da98b52 chore: update deps 2024-05-30 11:53:00 +02:00
Ivan Dyachkov
1129c18330 fix(authz_http): fix content-type header in http request 2024-05-30 11:36:13 +02:00
zhongwencool
8ab7e75e64
Merge pull request #13139 from zhongwencool/influxdb-client-failed-msg-port 
chore: improved InfluxDB connection error messaging
 2024-05-30 10:57:21 +08:00
zmstone
532812f6f6 test: assert results 2024-05-29 22:10:34 +02:00
zmstone
9fe29bf5f7 chore: bump kafka bridge app vsn 2024-05-29 21:57:14 +02:00
zmstone
78a6100346 chore: fix app vsn bumps 
only bug fixes so far
 2024-05-29 21:56:22 +02:00
zmstone
1db9d54d17 fix: handle unicode in data integration template strings 
The bug was introduced in 5.7.0 (#12893)
 2024-05-29 21:53:07 +02:00
zmstone
c54d25de98
Merge pull request #13070 from zmstone/0518-improve-kafka-connection-error-logs 
0518 improve kafka connection error logs
 2024-05-29 16:24:01 +02:00
Kjell Winblad
5017ef82a3
Merge pull request #13130 from kjellwinblad/kjell/fix_redis_trace_format/EMQX-12428 
fix(redis trace): add separators in redis batch action trace
 2024-05-29 15:59:32 +02:00
Kjell Winblad
6810591911
Merge pull request #13147 from kjellwinblad/kjell/fix_function_clause_when_decoding_protobuf/EMQX-12453 
fix: make protobuf schema decode error reported to user less cryptic
 2024-05-29 15:56:27 +02:00
Kjell Winblad
3769bbb7f4
Merge pull request #13140 from kjellwinblad/kjell/republish_trace_crash/EMQX-12474 
fix: trace formatter should not crash when given a non-binary payload
 2024-05-29 15:54:02 +02:00
Kjell Winblad
9b089a4e5a fix: use explain instead of msg as error explanation and add test 2024-05-29 10:20:15 +02:00
Kjell Winblad
479889f162
Merge pull request #13136 from kjellwinblad/kjell/fix_oracle_trace_format/EMQX-12433 
fix(oracle action trace): parameters should not be rendered as IO Data
 2024-05-29 09:12:38 +02:00
zhongwencool
7d9333a9d7 chore: bump influxdb app version 2024-05-29 11:13:17 +08:00
zhongwencool
00d74826ec chore: improved InfluxDB connection error messaging 2024-05-29 11:13:17 +08:00
Thales Macedo Garitezi
802361dbd0 fix(connector api): handle timeout when waiting for connector status 
Fixes https://emqx.atlassian.net/browse/EMQX-12251
 2024-05-28 14:19:32 -03:00
Kjell Winblad
82800faadf fix: make protobuf schema decode error reported to user less cryptic 
Before this commit the user would see a cryptic warning log including
function_clause when decoding a message failed. This commit improves
this by removing function_caluse as well as some other cryptic words
from the message and adding a description describing what went wrong.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12453
 2024-05-28 18:07:14 +02:00
Kjell Winblad
3151a3123e chore: bump version of emqx_redis_bridge app 2024-05-28 17:23:35 +02:00
Kjell Winblad
3f1fe413ee refactor: simplify redis trace formatting 2024-05-28 17:22:58 +02:00
Kjell Winblad
cbdde7165e fix(redis trace): add separators in redis batch action trace 
Logger will transform data that looks like IO data into a string which
made redis batch traces look like the spaces had been removed
from the strings. To prevent this, we render the batched commands into
binary string separated by spaces and semicolon. The components of a
single command is separated by spaces and the commands in a batch are
separated by semicolons.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12428
 2024-05-28 17:22:58 +02:00
Kjell Winblad
5e3de1a4e3 fix(emqx_logger_jsonfmt:best_effort_json_test): test cases 2024-05-28 17:15:32 +02:00
zmstone
40080f5e7d
Merge pull request #13141 from zmstone/0528-test-fix-dashboard-config-test-with-stable-base-config 
test: fix test case to have deterministic base config
 2024-05-28 14:32:55 +02:00
Andrew Mayorov
9ae4de10ed
Merge pull request #13127 from keynslug/fix/EMQX-12439/update-leaner-error 
fix(bridge-v2): report descriptive error on invalid update request
 2024-05-28 13:11:41 +03:00
zmstone
e9cc88cb95 test: fix test case to have deterministic base config 2024-05-28 12:03:01 +02:00
Kjell Winblad
84bc6ba28c fix: trace formatter should not crash when given a non-binary payload 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12474
 2024-05-28 11:13:51 +02:00
Kjell Winblad
e55bda6fe6 chore: bump app versions 
Bump app version as required to pass the check which is done by the
following script:

./scripts/apps-version-check.sh
 2024-05-28 11:01:00 +02:00
Kjell Winblad
f65168982b fix(oracle action trace): parameters should not be rendered as IO Data 
This forces the parameters to the database statement to be rendered as
a JSON array in JSON traces instead of being rendered as a string when
the parameters are interpreted as IO data.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12433
 2024-05-28 10:59:44 +02:00
Andrew Mayorov
c04aaad0a4
chore(bridge-v2): leave comment describing need for nowarn_function 2024-05-28 10:31:14 +02:00
ieQu1
8fbeca4321
chore: Version bumps 2024-05-28 00:14:01 +02:00
ieQu1
04305d004f
refactor(sessds): Extract metadata keys to a header 2024-05-27 23:36:20 +02:00
ieQu1
16d1425982
refactor(emqx): Move files related to durable session to a subdir 2024-05-27 23:31:15 +02:00
zmstone
2771a10d39 test: fix a flaky one 2024-05-27 20:59:50 +02:00
Andrew Mayorov
663f4fd39f
chore: bump application versions 2024-05-27 17:49:16 +02:00
Andrew Mayorov
08d88ea814
feat(bridge-api): improve error messages for Update Source API 2024-05-27 17:49:16 +02:00
Andrew Mayorov
ed7c29ec26
fix(api): mention spec override in "request metadata" type 2024-05-27 16:57:45 +02:00
Andrew Mayorov
5c2a68076f
fix(bridge-v2): report descriptive error on invalid update request 
Before this commit, generic validation errors were reported as union
mismatches of _all_ of the bridge schemas. After this commit, specific
schema is chosen before validation.
 2024-05-27 14:37:10 +02:00
Ivan Dyachkov
f393fd5304 chore: release 5.7.0 2024-05-27 14:32:23 +02:00
zmstone
b2f5e50f16
Merge pull request #13120 from ieQu1/dev/ds-api-site-status 
fix(dsapi): Check status of the node via Mria
 2024-05-25 05:42:05 +02:00
zmstone
0763e29786 chore: prepare for 5.7.0-rc.2 release 2024-05-24 21:10:37 +02:00
ieQu1
73df01e934
fix(dsapi): Check status of the node via Mria 2024-05-24 20:59:07 +02:00
ieQu1
7f356aa3a8
Merge pull request #13115 from ieQu1/dev/fix-drop-generations2 
Mitigate transient errors when dropping generations
 2024-05-24 20:13:39 +02:00
ieQu1
830b62d899
fix(dsrepl): Retry sending ra commands to the leader 2024-05-24 19:09:34 +02:00
ieQu1
1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable 2024-05-24 19:09:34 +02:00
ieQu1
25721bceba
fix(dsstor): Delete generation metadata before dropping it 2024-05-24 19:09:33 +02:00
ieQu1
3d9837a0b8
test(ds): Add emqx_ds_replication_SUITE:t_drop_generation 2024-05-24 19:09:33 +02:00
Ilya Averyanov
327080e70e perf(emqx_placeholder): do not try to decode maps as json 
fix
 2024-05-24 20:08:31 +03:00
Andrew Mayorov
c30b881239
test(monitor): verify historical and current metrics are consistent 2024-05-24 14:56:10 +02:00
Andrew Mayorov
fd9655bc35
fix(monitor): unify metric names for consistency and simplicity 2024-05-24 14:55:48 +02:00
Andrew Mayorov
7b137c0aaf
fix(monitor): reflect durable subscriptions in historical metrics 
Essentially in the same manner as they are currently reflected in the
current metrics.
 2024-05-24 14:08:35 +02:00
zmstone
6ec8ae5a5f
Merge pull request #13112 from HJianBo/change-the-default-batch-size-s3-uploader 
Fix(s3): change the batch_size to 1 for direct mode
 2024-05-24 10:56:53 +02:00
Andrew Mayorov
29d203ae33
Merge pull request #13108 from keynslug/fix/ct-flaky/mgmt 
test(mgmt): avoid inter-suite contamination by switching to cth tooling
 2024-05-24 11:14:16 +03:00
JianBo He
adaf238131 fix(s3): fix the convert function if the mode is missing 2024-05-24 16:02:06 +08:00
JimMoen
5b60f6a5b2
fix: make dialyzer happy 2024-05-24 14:29:02 +08:00
JimMoen
43506495ae
fix: ensure plugin configured states after join cluster 2024-05-24 13:51:56 +08:00
JimMoen
874bcd9a8b
fix: match decode plugin config map failed case 2024-05-24 10:20:50 +08:00
zmstone
c3c6da7ec4 chore: prepare for release cut 5.7.0-rc.1 2024-05-23 21:39:15 +02:00
Zaiming (Stone) Shi
3a4b8e6c24
Merge pull request #13106 from zmstone/0523-add-customer-type 
0523 add customer type (4=BYOC) to license schema
 2024-05-23 21:30:02 +02:00
Zaiming (Stone) Shi
40288b99b2
Merge pull request #13101 from JimMoen/fix-plugin-config-map 
fix: allow put plugin config without schema
 2024-05-23 21:18:43 +02:00
Thales Macedo Garitezi
823a6bc521
Merge pull request #13107 from thalesmg/hide-clientv2-response-type-r57-20240523 
fix(client mgmt api): hide unused return type
 2024-05-23 16:03:51 -03:00
Andrew Mayorov
7878d106f6
test(mgmt): avoid inter-suite contamination by switching to cth tooling 2024-05-23 20:17:34 +02:00
Andrew Mayorov
b4c6968f8c
Merge pull request #13104 from keynslug/fix/EMQX-12415/last-resort-poll 
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
 2024-05-23 21:17:03 +03:00
ieQu1
0dc507441a
Merge pull request #13105 from ieQu1/dev/sessds-renew-streams-interval 
fix(sessds): Change the defaults to renew streams every 1 second
 2024-05-23 18:23:20 +02:00
Thales Macedo Garitezi
b1e3c941fb fix(client mgmt api): hide unused return type 
Fixes https://emqx.atlassian.net/browse/EMQX-12427
 2024-05-23 13:17:43 -03:00
Andrew Mayorov
d47bf8076a
test(mgmt): better isolate testsuite running environments 2024-05-23 17:53:13 +02:00
zmstone
c737045dff chore: add customer type 4 (BYOC) 2024-05-23 17:30:06 +02:00
ieQu1
5584b658c9
fix(sessds): Change the defaults to renew streams every 1 second 2024-05-23 17:17:29 +02:00
Zaiming (Stone) Shi
1d99ba2a8a
Merge pull request #13102 from JimMoen/fix-ds-prom-badkey 
fix: add ds metrics into prometheus zipper init acc
 2024-05-23 16:57:06 +02:00
Thales Macedo Garitezi
118c1d2806
Merge pull request #13103 from thalesmg/fix-another-ds-counter-nit-r57-20240523 
fix(monitor api): fix cluster metric aggregation
 2024-05-23 10:57:53 -03:00
Andrew Mayorov
ba6382adae
fix(dsrepl): trigger "last-resort" pending transitions handler when idle 
This is a hack to work around the unintended issues causing shard
allocator to become idle even when there are pending transitions.
 2024-05-23 14:54:01 +02:00
Andrew Mayorov
839b143fc4
fix(sessds): consider message durability disabled by default 2024-05-23 14:48:39 +02:00
Thales Macedo Garitezi
c0e3a81c61 fix(monitor api): fix cluster metric aggregation 
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
 2024-05-23 09:38:24 -03:00
Andrew Mayorov
bf326acd7b
fix(dsrepl): handle stopping non-yet-running shard supervisor 2024-05-23 14:34:27 +02:00
Kjell Winblad
0e5b02c373
Merge pull request #13100 from kjellwinblad/kjell/s3_aggregate_only_has_result_trace/EMQX-12429 
fix: add action rendered trace for s3 in aggregated mode
 2024-05-23 14:29:18 +02:00
JimMoen
a40c7d646a
fix: apply suggestions from code review, thanks @zmstone 2024-05-23 20:28:14 +08:00
JimMoen
b91ecac168
refactor(prom): unify function name to avoid multiple levels call 2024-05-23 20:11:22 +08:00
JimMoen
065411c69c
fix: add ds metrics into prometheus zipper init acc 2024-05-23 20:11:21 +08:00
Kjell Winblad
89b47e8ffc fix(s3 tracing): do not format records as IO data in aggregate mode 2024-05-23 12:41:30 +02:00
Kjell Winblad
fb7688ab94 fix(trace): make sure that the payload encode works with nested payloads 
This commit makes sure that the trace setting for payload encode works
even when the payload is in a nested structure or when the payload key
is a binary instead of an atom.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12424
 2024-05-23 12:30:56 +02:00
JimMoen
5dc96a37e2
fix: ensure plugin avsc serde before config 2024-05-23 17:46:58 +08:00
JimMoen
25a9aa1797
fix: allow put plugin config without schema 2024-05-23 17:33:34 +08:00
JimMoen
99869821fb
fix: ensure plugin config map after fresh install 2024-05-23 17:16:16 +08:00
Kjell Winblad
e10c87b825 fix: add action rendered trace for s3 in aggregated mode 
Fixes:
https://emqx.atlassian.net/browse/EMQX-12429
 2024-05-23 10:25:27 +02:00
Andrew Mayorov
398dc97ed6
Merge pull request #13092 from keynslug/fix/dsrepl/site-autoleave 
fix(dsrepl): properly handle transaction abort during forget site
 2024-05-23 10:56:23 +03:00
ieQu1
950fa82be3
Merge pull request #13097 from ieQu1/dev/ds-replay-unrecoverable-no-test 
feat(sessds): Handle unrecoverable errors
 2024-05-23 09:49:21 +02:00
Kjell Winblad
26c988fe11
Merge pull request #13077 from kjellwinblad/kjell/fix_action_update_race_issue/EMQX-12376 
fix: action config update would sometimes not be reflected in connector
 2024-05-23 09:11:48 +02:00
Andrew Mayorov
2f6729d203
Merge pull request #13089 from keynslug/fix/EMQX-12437/restore 
fix(s3-bridge): restore backward config compatibility
 2024-05-23 10:10:01 +03:00
JianBo He
4053356597
Merge pull request #13063 from JimMoen/improve-plugin-config-behavior 
fix(plugin): serde not found on new-joined nodes
 2024-05-23 11:24:44 +08:00
lafirest
b1076221b0
Merge pull request #13085 from lafirest/fix/rocket_namespace 
fix(rocketmq): fix namespace error for RocketMQ
 2024-05-23 10:39:30 +08:00
ieQu1
07774ab060
feat(sessds): Handle unrecoverable errors 2024-05-22 22:45:37 +02:00
Thales Macedo Garitezi
63cb12d7c2
Merge pull request #13095 from thalesmg/fix-monitor-ds-counters-again-r57-20240522 
fix(monitor api): add ds connections and subscriptions to old counters
 2024-05-22 16:43:15 -03:00
ieQu1
bfd01c425d
Merge pull request #13072 from ieQu1/dev/fix-drop-generation 
Idempotent drop_generation
 2024-05-22 21:13:30 +02:00
ieQu1
6eb04f90a3
fix(ds): Allow to write batches to older generations 2024-05-22 20:28:16 +02:00
Thales Macedo Garitezi
5cad4497de fix(monitor api): add ds connections and subscriptions to old counters 
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
 2024-05-22 15:19:46 -03:00
Thales Macedo Garitezi
4094032649 fix(bridge v2 api): don't attempt to start disabled connector when starting action/source 
Fixes https://emqx.atlassian.net/browse/EMQX-12435
 2024-05-22 13:33:55 -03:00
Kjell Winblad
88c96e26de refactor: simplify the code with maps:map/2 
Thanks @thalesmg for the suggestion
 2024-05-22 18:03:43 +02:00
ieQu1
59a09fb86f
fix(ds): Apply review remarks 2024-05-22 18:01:33 +02:00
ieQu1
5c78ecba40
docs(ds): Update documentation for the storage layouts 2024-05-22 18:01:33 +02:00
ieQu1
29345aaa30
fix(ds): Fix idle event generation in bitfield_lts layout 2024-05-22 18:01:33 +02:00
ieQu1
b3ded7edce
fix(ds): Fix code review remark 2024-05-22 18:01:33 +02:00
ieQu1
60edf5e9b8
fix(ds): Move responsibility of returning end_of_stream to the CBM 2024-05-22 18:01:33 +02:00
ieQu1
0ff307e789
fix(ds): Include generation ID in the storage events 
Make sure storage events originating from generation X are handled in
the context of the same generation.
 2024-05-22 18:01:33 +02:00
ieQu1
acdae4fba3
fix(ds): Workaround for the idempotency error when dropping gens 2024-05-22 18:01:33 +02:00
ieQu1
eb7c43ee9d
fix(ds): Always store messages in the current generation 2024-05-22 18:01:33 +02:00
ieQu1
074d98a14a
test(ds): Refactor ds_SUITE 2024-05-22 18:01:33 +02:00
ieQu1
e4a73f003a
feat(ds): Implement format_status callback 
Reduce volume of logs and crash reports from DS
 2024-05-22 18:01:32 +02:00
ieQu1
1526c527d0
fix(ds): Log generation operations 2024-05-22 18:01:32 +02:00
ieQu1
aca2d9586c
fix(ds): Fix return type of drop_generation 2024-05-22 18:01:32 +02:00
ieQu1
c6fc76e335
fix(ds): Perform read operations on the leader. 2024-05-22 18:01:32 +02:00
ieQu1
4580906405
fix(ds): Use erpc instead of gen_rpc for `delete_next' 2024-05-22 18:01:32 +02:00
Andrew Mayorov
8016e9adf4
fix(s3-bridge): restore backward config compatibility 2024-05-22 17:59:52 +02:00
Kjell Winblad
4b540e3bd0 fix: do not leak action configurations in alarm messages 2024-05-22 17:43:07 +02:00
Andrew Mayorov
e6c5c1b598
chore(dsrepl): provide more information in rebalancing log messages 2024-05-22 17:24:08 +02:00
Andrew Mayorov
c355c9ad50
fix(dsrepl): properly handle transaction abort during forget site 2024-05-22 17:22:55 +02:00
Kjell Winblad
cff8b97e8a fix: handle channel updated during health check 
This commit fixes an issue found by CI test case
emqx_bridge_influxdb_SUITE:t_start_stop and others. While the channel
health check process is running, the channel could be removed or updated
which could cause a crash in the resource manager or non up-to-date
alarms being triggered.
 2024-05-22 12:15:04 +02:00
Kjell Winblad
39d758c4d6 fix: do not return configs for channels from emqx_resource_manager 2024-05-22 12:15:04 +02:00
Kjell Winblad
917474f694 fix: action config update would sometimes not be reflected in connector 
Before this commit the following happened sometimes:

1. action status is connected
2. action config is updated to something that should change the status to
   disconnected
3. action status is still connected and the old config is being used by
   the connector even though the config has been correctly updated.

The reason for this bug is that the post_config_hook runs before the
global EMQX config is updated. The post config hook is adding the new
config to the connector. Since the new config causes the action to get
status disconnected, the adding of the action to the connector is
retried when the health check runs but this time the config will be
loaded from the global config which could cause the old config to be
loaded (this happens when the global config has not had time to get
updated).

The above problem has been fixed in this commit by only reading action
configs from the global config when the connector starts/restarts and
instead store the latest configs for the actions in the connector.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12376
 2024-05-22 12:15:04 +02:00
firest
f3e8037e0f fix(rocketmq): fix namespace error for RocketMQ 2024-05-22 16:29:38 +08:00
Andrew Mayorov
a3cd3e31b1
Merge pull request #13084 from keynslug/fix/EMQX-12422/part-size 
fix(s3-bridge): report inconsistent `min/max_part_size` option
 2024-05-22 09:48:09 +02:00
JimMoen
e5f7aa9817
refactor: plguin functions and types rename 2024-05-22 11:51:54 +08:00
JimMoen
33aa61daea
fix: use hocon format as plugin config 2024-05-22 06:37:58 +08:00
JimMoen
14f2a68799
fix: bpapi spec type 2024-05-22 06:01:57 +08:00
Thales Macedo Garitezi
31a35f2a15
Merge pull request #13076 from thalesmg/fix-ds-session-count-client-api-r57-20240520 
fix(client mgmt api): cache disconnected durable session count for `/clients` api
 2024-05-21 15:34:21 -03:00
JimMoen
5abd23af5a
test: plugin refactor 2024-05-22 02:09:37 +08:00
JimMoen
87b3b214b9
fix: make static_check happy 2024-05-22 02:09:36 +08:00
JimMoen
3ce091e9d7
fix: ensure plugin tarball and extracted 2024-05-22 02:09:35 +08:00
JimMoen
1abc8cf502
fix: ensure plugin's config on boot and join cluster 2024-05-22 02:09:34 +08:00
JimMoen
677352e498
fix: ensure installed and plugin state on boot 2024-05-22 02:09:33 +08:00
JimMoen
2076681e76
fix: put plugin config with binary namevsn key 2024-05-22 02:09:32 +08:00
JimMoen
140b7ce51e
fix(plugin): schema content only provided in enterprise edition 2024-05-22 02:09:31 +08:00
JimMoen
a7f2f95318
fix: ensure avro file 2024-05-22 02:09:30 +08:00
JimMoen
df7dcb2764
fix: do not let plugin start failed lead emqx start failed 2024-05-22 02:09:29 +08:00
JimMoen
e0e4517d9e
fix: ensure plugin config on boot 2024-05-22 02:09:28 +08:00
JimMoen
e8d4e88118
fix(plugin): serde not found on new-joined nodes 2024-05-22 02:09:26 +08:00
Andrew Mayorov
28f6eb0200
fix(s3-bridge): report inconsistent min/max_part_size option 2024-05-21 18:49:01 +02:00
Thales Macedo Garitezi
7c5cb1acc5 fix(monitor current api): expose disconnected_durable_sessions count 
Fixes https://emqx.atlassian.net/browse/EMQX-12423
 2024-05-21 10:03:09 -03:00
Thales Macedo Garitezi
8b4a1c3d75 fix(client mgmt api): cache disconnected durable session count for /clients api 
Fixes https://emqx.atlassian.net/browse/EMQX-12396
 2024-05-20 11:48:08 -03:00
Kjell Winblad
8279d8c787
Merge pull request #13053 from kjellwinblad/kjell/fix_rule_trace_issues/EMQX-12327/EMQX-12335/EMQX-12336 
Fix small action trace issues
 2024-05-20 16:25:38 +02:00