Commit Graph

12282 Commits

Author SHA1 Message Date
JimMoen
dff31b293c
perf(utils_sql): pre-build regexp 2024-06-06 11:01:38 +08:00
zmstone
c41be7ee22 chore: fix dialyzer warning 2024-06-05 23:37:21 +02:00
Ilia Averianov
dc2e6d1695
Merge pull request #13163 from savonarola/0523-ds-shared-subs-dispatch
Inject shared subcription handling into durable session
2024-06-05 23:59:50 +03:00
Serge Tupchii
159da912c2 feat(emqx_dashboard_swagger): apply field validator in request body check 2024-06-05 19:35:04 +03:00
JimMoen
144264e0d8
test: add utils_sql test cases 2024-06-06 00:30:18 +08:00
JimMoen
627b8c45d2
fix(utils_sql): improve insert sql regular expression 2024-06-06 00:30:17 +08:00
Ilya Averyanov
b075b7120c feat(sessds): implement dispatchig between CE/EE shared sub agents 2024-06-05 19:17:58 +03:00
Ilya Averyanov
60556abc4f fix(dashboard_monitor): correctly merge old and new sample data 2024-06-05 18:49:52 +03:00
Kjell Winblad
2956e849eb fix(pgsql connector): better msg when failing to remove statement 2024-06-05 16:58:14 +02:00
Kjell Winblad
336089f8a7 fix: bug found by dialyzer and make test case cleaner 2024-06-05 15:53:02 +02:00
Thales Macedo Garitezi
ae0379f974
Merge pull request #13181 from thalesmg/force-stop-connector-r57-20240604
fix(resource manager): force kill process if stuck when stopping/removing
2024-06-05 09:08:30 -03:00
lafirest
d79193fde4
Merge pull request #13177 from lafirest/sync-authn-com
feat: sync `client.check_authn_complete` to release-57
2024-06-05 19:51:10 +08:00
Thales Macedo Garitezi
5eab221f7c
Merge pull request #13175 from thalesmg/disable-prepared-statements-postgres-r57-20240603
feat(postgres): add `disable_prepared_statements` option
2024-06-04 12:29:53 -03:00
Thales Macedo Garitezi
f17aefe3d7 fix(action/source api): improve returned error message on timeout 2024-06-04 12:04:55 -03:00
zmstone
cda6d5f636
Merge pull request #13172 from zmstone/0603-add-redis-arg-formatter-to-rule-engine-funcs
feat(redis): add a rule function to help formatting redis args
2024-06-04 16:56:14 +02:00
Thales Macedo Garitezi
3013189cd7 fix(resource manager): force kill process if stuck when stopping/removing
Fixes https://emqx.atlassian.net/browse/EMQX-12357
2024-06-04 11:38:24 -03:00
Ilya Averyanov
9cdfbb0845 feat(sessds): make ds session be aware of shared subscriptions 2024-06-04 17:15:30 +03:00
Thales Macedo Garitezi
c07bc68e6f feat(postgres): add disable_prepared_statements option
Fixes https://emqx.atlassian.net/browse/EMQX-12496

Some Postgres connections, such ones made to [PGBouncer](https://www.pgbouncer.org/) or
[Supabase in Transaction Mode](https://supabase.com/), do not support some session
features like prepared statements.
2024-06-04 10:11:25 -03:00
firest
87384cae32 fix(events): call client.check_authn_complete even if authentication fails 2024-06-04 17:55:39 +08:00
firest
92d26ff27b chore: update change 2024-06-04 17:55:30 +08:00
firest
e4e53844d5 feat(events): add new hook && event client.check_authn_complete 2024-06-04 17:55:21 +08:00
Ilia Averianov
dc95218c68
Merge pull request #13144 from savonarola/0528-increase-bridge-buffer-drop-log-level
chore(bridges): increase log level for bridge buffer overflow reports
2024-06-04 12:37:50 +03:00
Kjell Winblad
572ca6433e fix(pgsql connector): improvements due to suggestions from @thalesmg
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-06-04 10:11:45 +02:00
zhongwencool
a8a67a2ac9
Merge pull request #13151 from zhongwencool/authz-trace-log
chore: make authz's logs easier to understand
2024-06-04 11:12:09 +08:00
zmstone
aa7ce1f641 fix(bridge/redis): add test case for map_to_redis_hset_args 2024-06-03 23:05:43 +02:00
Kjell Winblad
f2ccfff803 fix(pgsql connector): handle prepared statement already exists
In a user's log file it was found that that the pgsql driver can end up
in a situation where the prepared statement for a channel/action is not
properly removed before a channel with the same name as the prepared
statement is added to the connector. This commit handles this by
attempting to remove the old prepared statement if one already exists
when adding channel.

Related issue:
https://emqx.atlassian.net/browse/EEC-1036
2024-06-03 16:14:26 +02:00
zmstone
e3ed7b59dd feat(redis): add a rule function to help formatting redis args
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.

This new function is dedicated for redis to avoid abuse for other
data integrations.
2024-06-03 14:09:15 +02:00
Thales Macedo Garitezi
c5547543e0
Merge pull request #13069 from thalesmg/azure-blob-storage-action-m-20240508
feat: implement azure blob storage action
2024-06-03 08:56:37 -03:00
Ilia Averianov
193978af90
Merge pull request #13168 from savonarola/0530-fix-mgmt-test
chore(tests): improve test stability
2024-06-03 14:44:12 +03:00
zhongwencool
fda365a87b chore: make authz's logs easier to understand 2024-06-03 18:01:57 +08:00
JimMoen
21624bc865
fix: get plugin app vsn from field rel_apps 2024-06-03 15:26:14 +08:00
zmstone
b6ff67d712
Merge pull request #13093 from zmstone/0522-upgrade-to-brod-3.18
chore: upgrade to brod 3.18.0
2024-06-02 15:06:40 +02:00
JimMoen
8d22a5a3ae
Merge pull request #13167 from JimMoen/fix-rule-events-i18n-todo
fix: missing rule event i18n in zh_cn
2024-06-02 18:23:20 +08:00
zmstone
9637b14543 fix: upgrade to brod 3.18.0 to fix unnecessary rebalance 2024-06-01 10:57:24 +02:00
Ilya Averyanov
57741a41e9 chore(tests): improve test stability 2024-05-31 22:27:31 +03:00
zmstone
fb07f40bea test: fix default value of retry_interval in test case 2024-05-31 17:53:12 +02:00
zmstone
20c92407c7 feat: change message retry_interval default value to infinity
Previous Setting: The default value for `retry_interval` was 30 seconds.

New Default: The default `retry_interval` has been changed to 'infinity'.
With this update, EMQX will not automatically retry message deliveries
by default.

Compliance and Compatibility: Aligning with MQTT specification
standards, in-session message delivery retries are not typically
compliant. We recognize that some users depend on this feature, so the
option to configure retries remains available for backward
compatibility.
2024-05-31 17:53:12 +02:00
zmstone
37ce8d93a1
Merge pull request #13164 from zmstone/0530-http-acl-compatibility-with-v4
0530 http acl compatibility with v4
2024-05-31 16:28:30 +02:00
Thales Macedo Garitezi
347971e24a fix: check max_block_size for direct transfers 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
4859cebd9f refactor: move min_block_size one level up 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
9a1d3ea04f chore: bump app vsns 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
616f14ae53 fix: avoid uploading blocks too frequently and splitting large buffers
Fixes https://github.com/emqx/emqx/pull/13069#discussion_r1613706747
2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
af99829a21 refactor: move common templating logic to emqx_connector_aggreg_buffer_ctx 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
c916c83c7c feat: implement azure blob storage action
Fixes https://emqx.atlassian.net/browse/EMQX-12280
2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
60d24c6ad5 refactor: simplify resource_opts definition 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi
729441d1ce refactor(connector aggregator): move shared schemas to app 2024-05-31 11:07:44 -03:00
Ilya Averyanov
db713cb656 chore(bridges): increase log level for bridge buffer overflow reports
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2024-05-31 16:35:18 +03:00
Thales Macedo Garitezi
c289c7d065
Merge pull request #13148 from thalesmg/EMQX-12251-fix-conn-api-timeout-case-clause-r571-20240528
fix(connector api): handle `timeout` when waiting for connector status
2024-05-31 10:07:35 -03:00
zmstone
6fe8a09e97 fix(authz/http): rename PH_ACCESS to VAR_ACCESS 2024-05-31 14:49:54 +02:00
JimMoen
45e2c81ebc
fix: missing rule event i18n in zh_cn 2024-05-31 17:51:47 +08:00
Ivan Dyachkov
29ad07ce29 Merge remote-tracking branch 'upstream/release-57' into 0531-sync-release-57 2024-05-31 07:32:26 +02:00
zmstone
ca32cf8d29 Merge branch 'emqx-12487-fix-authz-http-content-type' into 0530-http-acl-compatibility-with-v4 2024-05-30 22:30:20 +02:00
zmstone
a00d9f17b1 chore: upgrade kafka producer lib wolff to 1.10.4
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
2024-05-30 15:40:10 +02:00
zmstone
460081a22e
Merge pull request #13131 from zmstone/0527-handle-unicode-in-template-strings
0527 handle unicode in template strings
2024-05-30 15:37:43 +02:00
Ilya Averyanov
97f9c81e19 feat(auth): add legacy ${access} placeholder 2024-05-30 15:47:48 +03:00
Ivan Dyachkov
0c4da98b52 chore: update deps 2024-05-30 11:53:00 +02:00
Ivan Dyachkov
1129c18330 fix(authz_http): fix content-type header in http request 2024-05-30 11:36:13 +02:00
zhongwencool
8ab7e75e64
Merge pull request #13139 from zhongwencool/influxdb-client-failed-msg-port
chore: improved InfluxDB connection error messaging
2024-05-30 10:57:21 +08:00
zmstone
532812f6f6 test: assert results 2024-05-29 22:10:34 +02:00
zmstone
9fe29bf5f7 chore: bump kafka bridge app vsn 2024-05-29 21:57:14 +02:00
zmstone
78a6100346 chore: fix app vsn bumps
only bug fixes so far
2024-05-29 21:56:22 +02:00
zmstone
1db9d54d17 fix: handle unicode in data integration template strings
The bug was introduced in 5.7.0 (#12893)
2024-05-29 21:53:07 +02:00
zmstone
c54d25de98
Merge pull request #13070 from zmstone/0518-improve-kafka-connection-error-logs
0518 improve kafka connection error logs
2024-05-29 16:24:01 +02:00
Kjell Winblad
5017ef82a3
Merge pull request #13130 from kjellwinblad/kjell/fix_redis_trace_format/EMQX-12428
fix(redis trace): add separators in redis batch action trace
2024-05-29 15:59:32 +02:00
Kjell Winblad
6810591911
Merge pull request #13147 from kjellwinblad/kjell/fix_function_clause_when_decoding_protobuf/EMQX-12453
fix: make protobuf schema decode error reported to user less cryptic
2024-05-29 15:56:27 +02:00
Kjell Winblad
3769bbb7f4
Merge pull request #13140 from kjellwinblad/kjell/republish_trace_crash/EMQX-12474
fix: trace formatter should not crash when given a non-binary payload
2024-05-29 15:54:02 +02:00
Kjell Winblad
9b089a4e5a fix: use explain instead of msg as error explanation and add test 2024-05-29 10:20:15 +02:00
Kjell Winblad
479889f162
Merge pull request #13136 from kjellwinblad/kjell/fix_oracle_trace_format/EMQX-12433
fix(oracle action trace): parameters should not be rendered as IO Data
2024-05-29 09:12:38 +02:00
zhongwencool
7d9333a9d7 chore: bump influxdb app version 2024-05-29 11:13:17 +08:00
zhongwencool
00d74826ec chore: improved InfluxDB connection error messaging 2024-05-29 11:13:17 +08:00
Thales Macedo Garitezi
802361dbd0 fix(connector api): handle timeout when waiting for connector status
Fixes https://emqx.atlassian.net/browse/EMQX-12251
2024-05-28 14:19:32 -03:00
Kjell Winblad
82800faadf fix: make protobuf schema decode error reported to user less cryptic
Before this commit the user would see a cryptic warning log including
function_clause when decoding a message failed. This commit improves
this by removing function_caluse as well as some other cryptic words
from the message and adding a description describing what went wrong.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12453
2024-05-28 18:07:14 +02:00
Kjell Winblad
3151a3123e chore: bump version of emqx_redis_bridge app 2024-05-28 17:23:35 +02:00
Kjell Winblad
3f1fe413ee refactor: simplify redis trace formatting 2024-05-28 17:22:58 +02:00
Kjell Winblad
cbdde7165e fix(redis trace): add separators in redis batch action trace
Logger will transform data that looks like IO data into a string which
made redis batch traces look like the spaces had been removed
from the strings. To prevent this, we render the batched commands into
binary string separated by spaces and semicolon. The components of a
single command is separated by spaces and the commands in a batch are
separated by semicolons.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12428
2024-05-28 17:22:58 +02:00
Kjell Winblad
5e3de1a4e3 fix(emqx_logger_jsonfmt:best_effort_json_test): test cases 2024-05-28 17:15:32 +02:00
zmstone
062ab31ecf Merge remote-tracking branch 'origin/release-57' into 0527-port-back-diverged-modules 2024-05-28 14:35:25 +02:00
zmstone
40080f5e7d
Merge pull request #13141 from zmstone/0528-test-fix-dashboard-config-test-with-stable-base-config
test: fix test case to have deterministic base config
2024-05-28 14:32:55 +02:00
Andrew Mayorov
9ae4de10ed
Merge pull request #13127 from keynslug/fix/EMQX-12439/update-leaner-error
fix(bridge-v2): report descriptive error on invalid update request
2024-05-28 13:11:41 +03:00
zmstone
e9cc88cb95 test: fix test case to have deterministic base config 2024-05-28 12:03:01 +02:00
Kjell Winblad
84bc6ba28c fix: trace formatter should not crash when given a non-binary payload
Fixes:
https://emqx.atlassian.net/browse/EMQX-12474
2024-05-28 11:13:51 +02:00
Kjell Winblad
e55bda6fe6 chore: bump app versions
Bump app version as required to pass the check which is done by the
following script:

./scripts/apps-version-check.sh
2024-05-28 11:01:00 +02:00
Kjell Winblad
f65168982b fix(oracle action trace): parameters should not be rendered as IO Data
This forces the parameters to the database statement to be rendered as
a JSON array in JSON traces instead of being rendered as a string when
the parameters are interpreted as IO data.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12433
2024-05-28 10:59:44 +02:00
zmstone
8f2ca71d8b test: fix test case to have deterministic base config 2024-05-28 10:57:20 +02:00
Andrew Mayorov
c04aaad0a4
chore(bridge-v2): leave comment describing need for nowarn_function 2024-05-28 10:31:14 +02:00
ieQu1
8fbeca4321
chore: Version bumps 2024-05-28 00:14:01 +02:00
ieQu1
04305d004f
refactor(sessds): Extract metadata keys to a header 2024-05-27 23:36:20 +02:00
ieQu1
16d1425982
refactor(emqx): Move files related to durable session to a subdir 2024-05-27 23:31:15 +02:00
zmstone
1148091f0b chore: restore test code 2024-05-27 23:05:21 +02:00
zmstone
96ef37d0ea chore: port diverged modules back to oss 2024-05-27 22:40:08 +02:00
zmstone
2771a10d39 test: fix a flaky one 2024-05-27 20:59:50 +02:00
zmstone
238c207b09 chore: bump app versions 2024-05-27 20:26:52 +02:00
zmstone
41cbfcfaa1 Merge remote-tracking branch 'origin/release-57' into 0527-sync-5.7.0-to-master 2024-05-27 20:10:44 +02:00
Andrew Mayorov
663f4fd39f
chore: bump application versions 2024-05-27 17:49:16 +02:00
Andrew Mayorov
08d88ea814
feat(bridge-api): improve error messages for Update Source API 2024-05-27 17:49:16 +02:00
Andrew Mayorov
ed7c29ec26
fix(api): mention spec override in "request metadata" type 2024-05-27 16:57:45 +02:00
Andrew Mayorov
5c2a68076f
fix(bridge-v2): report descriptive error on invalid update request
Before this commit, generic validation errors were reported as union
mismatches of _all_ of the bridge schemas. After this commit, specific
schema is chosen before validation.
2024-05-27 14:37:10 +02:00
Ivan Dyachkov
f393fd5304 chore: release 5.7.0 2024-05-27 14:32:23 +02:00
zmstone
b2f5e50f16
Merge pull request #13120 from ieQu1/dev/ds-api-site-status
fix(dsapi): Check status of the node via Mria
2024-05-25 05:42:05 +02:00
zmstone
0763e29786 chore: prepare for 5.7.0-rc.2 release 2024-05-24 21:10:37 +02:00