03b226248a
feat(schema registry): add check for inner types
...
Currently, only `protobuf` has any.
2024-06-11 13:50:11 -03:00
c70e8252fe
fix: declare `emqx_schema_registry` as a dependency of `emqx_message_transformation`
2024-06-11 13:35:02 -03:00
2c6aa80b3c
Merge pull request #13221 from thalesmg/refactor-sv-topic-index-r57-20240610
...
refactor(schema validation): module organization and position indexing
2024-06-11 13:29:48 -03:00
83ff07ade0
Merge pull request #13199 from thalesmg/message-transformation-r57-20240604
...
feat: implement message transformation
2024-06-11 13:29:25 -03:00
ab5329be02
Merge pull request #13220 from thalesmg/refactor-channel-hc-opt-r57-20240610
...
fix: remove another redundant health check
2024-06-11 12:22:43 -03:00
2f0eb3560b
Merge pull request #13227 from keynslug/fix/EMQX-12533/bad-tpl
...
fix(bridge-s3): validate aggreg key template before adding a channel
2024-06-11 15:39:12 +03:00
d4fb812352
fix(aggreg): anticipate and handle delivery startup errors
2024-06-11 13:21:41 +02:00
29fc30ea69
fix(bridge-s3): validate aggreg key template before adding a channel
2024-06-11 13:21:41 +02:00
fb0da9848c
feat(tpl): add separate `placeholders/1` function
...
The purpose is to have a clearer view of placeholders used in a
template, without going the usual `render(Template, #{})` route that is
actually subtly misleading: it won't mention that `${}` / `${.}`
placeholder has been used.
Also unify handling of `${}` / `${.}` in a couple of places.
2024-06-11 11:42:43 +02:00
9d22089104
Merge pull request #13173 from kjellwinblad/kjell/pgsql_timestamp_encode_etc/EEC-1036
...
fix(pgsql connector): handle prepared statement already exists
2024-06-11 10:53:30 +02:00
44258204bd
chore: move tls_certs test helper to apps/emqx
2024-06-11 10:06:18 +02:00
b657dc537c
Merge pull request #13156 from savonarola/0529-fix-dashboard-monitor
...
fix(dashboard_monitor): correctly merge old and new sample data
2024-06-11 10:33:15 +08:00
05ebb17cd6
refactor: index positions rather than names
...
Addresses https://github.com/emqx/emqx/pull/13199#discussion_r1633096025
2024-06-10 15:54:37 -03:00
5eff4a7544
refactor(schema validation): improve api and module organization
...
To conform to comment from another PR:
https://github.com/emqx/emqx/pull/13199#discussion_r1632658742
... since schema validation is the "inspiration" for message transformation.
2024-06-10 15:46:28 -03:00
d93c8540c8
feat: pass along peername down to transformation context
2024-06-10 15:35:35 -03:00
6e0ef893f4
feat: pass along client attributes down to message transformation context
2024-06-10 15:27:04 -03:00
3c501e4f2a
fix: remove another redundant health check
...
Fixes https://emqx.atlassian.net/browse/EMQX-12521
2024-06-10 15:13:34 -03:00
1a20a40218
Merge pull request #13209 from thalesmg/rm-redundant-hc-r57-20240607
...
chore: remove redundant health check call
2024-06-10 14:58:48 -03:00
5629fe60c1
refactor: index positions rather than names
2024-06-10 13:33:22 -03:00
8c7a7cf9db
refactor: apply review remarks
2024-06-10 11:17:13 -03:00
e54cf2f218
refactor: use option to skip rendering as string
2024-06-10 10:49:39 -03:00
d34558954d
refactor: improve api and module organization
2024-06-10 10:40:53 -03:00
4aa1c3842f
fix: declare `emqx` as a dependency in `.app.src`
2024-06-10 09:53:29 -03:00
aa416cc0ad
docs: fix stale comments and dates
2024-06-10 09:53:02 -03:00
7a09b0470d
chore: cosmetic changes
2024-06-10 14:22:25 +02:00
a885f0b41a
test(emqx_bridge_pgsql_SUITE): call test janitor
2024-06-10 13:45:27 +02:00
a1aa9a4375
fix(ee): emqx no longer deps on emqx_auth_ext
2024-06-10 10:32:01 +02:00
4347f3de3e
fix(bridge/mqtt): respect client ID prefix
2024-06-10 10:27:45 +02:00
76cfc309a9
feat(tls): partial_chain not required
2024-06-10 10:25:44 +02:00
7c37bf9965
feat(tls): ee only: TLS partial_chain and Keyusage
2024-06-10 10:24:48 +02:00
e60380d205
chore: fix nit
2024-06-10 09:55:12 +02:00
e9b813d8ef
chore: fix test
2024-06-10 09:54:54 +02:00
1739bc0c24
feat(partial_chain): gateway support
2024-06-10 09:54:44 +02:00
abbf2ef62f
chore(TLS-chain-test): update for OTP 26
2024-06-08 00:16:06 +02:00
221b748b0f
test(partial_chain): update tcs for OTP-25
2024-06-08 00:16:01 +02:00
463d1a1875
fix(test): tls_verify_partial_chain
2024-06-08 00:15:54 +02:00
788cdbc6dd
fix(listener): remove partial_chain in wss opts
2024-06-08 00:15:51 +02:00
3fc99315e0
chore: happy elvis
2024-06-08 00:15:45 +02:00
c5dccdf526
feat(tls): update schema for TLS keyusage
2024-06-08 00:15:43 +02:00
aa25e3badd
fix(tls): undefined keyusage
2024-06-08 00:15:39 +02:00
9e5cfea8c0
test(tls): verify peer keyusage
2024-06-08 00:15:36 +02:00
c5657029ab
feat(config): partial_chain
2024-06-08 00:15:34 +02:00
f7ff9496e6
test: port listener tls partial_chain
2024-06-08 00:15:31 +02:00
1ce13242a8
feat(tls): port partial_chain, part 1
2024-06-08 00:15:28 +02:00
c13631102e
fix(PgSQL connector): typo in function name
2024-06-07 19:40:18 +02:00
b2f027c974
Merge pull request #13197 from keynslug/fix/EMQX-12522/conv-ssl
...
fix(bridge-s3): pass SSL options through `convert_certs/2`
2024-06-07 19:45:53 +03:00
28a3c77e88
chore: remove redundant health check call
...
Since `emqx_resource_manager:channel_health_check` already does a base resource health
check, the call was repeated.
2024-06-07 11:07:12 -03:00
e63dcc84b0
fix: unused variable and better error message
2024-06-07 12:20:42 +02:00
78a196e215
Merge pull request #13189 from JimMoen/fix-apply-sqlserver-template
...
fix(utils_sql): improve insert sql regular expression
2024-06-07 13:50:19 +08:00
eb215726d5
Merge pull request #13195 from lafirest/fix/typos
...
fix: fix typos
2024-06-07 09:04:16 +08:00
ef36350bf6
feat: implement message transformation
...
Fixes https://emqx.atlassian.net/browse/EMQX-12517
2024-06-06 17:51:08 -03:00
8a85d7cb5a
fix(bridge-s3): pass SSL options through `convert_certs/2`
2024-06-06 19:37:38 +02:00
c17c788438
feat(connector): allow delegate `pre_config_update` to impls
2024-06-06 19:37:34 +02:00
2525bd0c16
perf: eliminate the loop-back fake async inet_reply message
...
Now that EMQX is on OTP 26 by default, the fake inet_reply
message when the send call returns ok only adds overhead.
For TCP (gen_tcp module), OTP 26 no longer supports direct port_command
so we had to swith to gen_tcp:send, then fake an async reply:
see https://github.com/emqx/esockd/pull/181
For TLS (ssl module), it has never supported async send anyway.
This commit switches to call Transport:send/2 directly,
this should result in one less loop-back message for each message.
For OTP 25 on which EMQX can still be compiled, one will have to
suffer the performance penalty for gen_tcp:send function to
do a non-optimized 'receive' for the send result.
2024-06-06 09:35:51 +02:00
5c3f5d8085
perf: do not call inet getstat before each and every send
...
In a stress test environment, when alarm is enabled, there were
a lot of long_schedule warnings with the stacktrace pointing to
congestion alarm based inet:getstat function.
When alarm is disabled, the one single client is able to
hold 40,000 QoS 1 messages per second (inflight=32) throughput.
When alarm is enabled, its mqueue overflows.
This commit removes the call before each and every data send,
so to rely on emit_stats timer to trigger the congestion check.
2024-06-06 09:35:51 +02:00
d4118c6e8e
fix: fix typos
2024-06-06 11:19:40 +08:00
dff31b293c
perf(utils_sql): pre-build regexp
2024-06-06 11:01:38 +08:00
c41be7ee22
chore: fix dialyzer warning
2024-06-05 23:37:21 +02:00
144264e0d8
test: add utils_sql test cases
2024-06-06 00:30:18 +08:00
627b8c45d2
fix(utils_sql): improve insert sql regular expression
2024-06-06 00:30:17 +08:00
60556abc4f
fix(dashboard_monitor): correctly merge old and new sample data
2024-06-05 18:49:52 +03:00
2956e849eb
fix(pgsql connector): better msg when failing to remove statement
2024-06-05 16:58:14 +02:00
336089f8a7
fix: bug found by dialyzer and make test case cleaner
2024-06-05 15:53:02 +02:00
ae0379f974
Merge pull request #13181 from thalesmg/force-stop-connector-r57-20240604
...
fix(resource manager): force kill process if stuck when stopping/removing
2024-06-05 09:08:30 -03:00
d79193fde4
Merge pull request #13177 from lafirest/sync-authn-com
...
feat: sync `client.check_authn_complete` to release-57
2024-06-05 19:51:10 +08:00
5eab221f7c
Merge pull request #13175 from thalesmg/disable-prepared-statements-postgres-r57-20240603
...
feat(postgres): add `disable_prepared_statements` option
2024-06-04 12:29:53 -03:00
f17aefe3d7
fix(action/source api): improve returned error message on timeout
2024-06-04 12:04:55 -03:00
cda6d5f636
Merge pull request #13172 from zmstone/0603-add-redis-arg-formatter-to-rule-engine-funcs
...
feat(redis): add a rule function to help formatting redis args
2024-06-04 16:56:14 +02:00
3013189cd7
fix(resource manager): force kill process if stuck when stopping/removing
...
Fixes https://emqx.atlassian.net/browse/EMQX-12357
2024-06-04 11:38:24 -03:00
c07bc68e6f
feat(postgres): add `disable_prepared_statements` option
...
Fixes https://emqx.atlassian.net/browse/EMQX-12496
Some Postgres connections, such ones made to [PGBouncer](https://www.pgbouncer.org/ ) or
[Supabase in Transaction Mode](https://supabase.com/ ), do not support some session
features like prepared statements.
2024-06-04 10:11:25 -03:00
87384cae32
fix(events): call `client.check_authn_complete` even if authentication fails
2024-06-04 17:55:39 +08:00
92d26ff27b
chore: update change
2024-06-04 17:55:30 +08:00
e4e53844d5
feat(events): add new hook && event `client.check_authn_complete`
2024-06-04 17:55:21 +08:00
572ca6433e
fix(pgsql connector): improvements due to suggestions from @thalesmg
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-06-04 10:11:45 +02:00
aa7ce1f641
fix(bridge/redis): add test case for map_to_redis_hset_args
2024-06-03 23:05:43 +02:00
f2ccfff803
fix(pgsql connector): handle prepared statement already exists
...
In a user's log file it was found that that the pgsql driver can end up
in a situation where the prepared statement for a channel/action is not
properly removed before a channel with the same name as the prepared
statement is added to the connector. This commit handles this by
attempting to remove the old prepared statement if one already exists
when adding channel.
Related issue:
https://emqx.atlassian.net/browse/EEC-1036
2024-06-03 16:14:26 +02:00
e3ed7b59dd
feat(redis): add a rule function to help formatting redis args
...
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.
This new function is dedicated for redis to avoid abuse for other
data integrations.
2024-06-03 14:09:15 +02:00
21624bc865
fix: get plugin app vsn from field `rel_apps`
2024-06-03 15:26:14 +08:00
b6ff67d712
Merge pull request #13093 from zmstone/0522-upgrade-to-brod-3.18
...
chore: upgrade to brod 3.18.0
2024-06-02 15:06:40 +02:00
8d22a5a3ae
Merge pull request #13167 from JimMoen/fix-rule-events-i18n-todo
...
fix: missing rule event i18n in zh_cn
2024-06-02 18:23:20 +08:00
9637b14543
fix: upgrade to brod 3.18.0 to fix unnecessary rebalance
2024-06-01 10:57:24 +02:00
37ce8d93a1
Merge pull request #13164 from zmstone/0530-http-acl-compatibility-with-v4
...
0530 http acl compatibility with v4
2024-05-31 16:28:30 +02:00
c289c7d065
Merge pull request #13148 from thalesmg/EMQX-12251-fix-conn-api-timeout-case-clause-r571-20240528
...
fix(connector api): handle `timeout` when waiting for connector status
2024-05-31 10:07:35 -03:00
6fe8a09e97
fix(authz/http): rename PH_ACCESS to VAR_ACCESS
2024-05-31 14:49:54 +02:00
45e2c81ebc
fix: missing rule event i18n in zh_cn
2024-05-31 17:51:47 +08:00
ca32cf8d29
Merge branch 'emqx-12487-fix-authz-http-content-type' into 0530-http-acl-compatibility-with-v4
2024-05-30 22:30:20 +02:00
a00d9f17b1
chore: upgrade kafka producer lib wolff to 1.10.4
...
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
2024-05-30 15:40:10 +02:00
460081a22e
Merge pull request #13131 from zmstone/0527-handle-unicode-in-template-strings
...
0527 handle unicode in template strings
2024-05-30 15:37:43 +02:00
97f9c81e19
feat(auth): add legacy ${access} placeholder
2024-05-30 15:47:48 +03:00
0c4da98b52
chore: update deps
2024-05-30 11:53:00 +02:00
1129c18330
fix(authz_http): fix content-type header in http request
2024-05-30 11:36:13 +02:00
8ab7e75e64
Merge pull request #13139 from zhongwencool/influxdb-client-failed-msg-port
...
chore: improved InfluxDB connection error messaging
2024-05-30 10:57:21 +08:00
532812f6f6
test: assert results
2024-05-29 22:10:34 +02:00
9fe29bf5f7
chore: bump kafka bridge app vsn
2024-05-29 21:57:14 +02:00
78a6100346
chore: fix app vsn bumps
...
only bug fixes so far
2024-05-29 21:56:22 +02:00
1db9d54d17
fix: handle unicode in data integration template strings
...
The bug was introduced in 5.7.0 (#12893 )
2024-05-29 21:53:07 +02:00
c54d25de98
Merge pull request #13070 from zmstone/0518-improve-kafka-connection-error-logs
...
0518 improve kafka connection error logs
2024-05-29 16:24:01 +02:00
5017ef82a3
Merge pull request #13130 from kjellwinblad/kjell/fix_redis_trace_format/EMQX-12428
...
fix(redis trace): add separators in redis batch action trace
2024-05-29 15:59:32 +02:00
6810591911
Merge pull request #13147 from kjellwinblad/kjell/fix_function_clause_when_decoding_protobuf/EMQX-12453
...
fix: make protobuf schema decode error reported to user less cryptic
2024-05-29 15:56:27 +02:00
3769bbb7f4
Merge pull request #13140 from kjellwinblad/kjell/republish_trace_crash/EMQX-12474
...
fix: trace formatter should not crash when given a non-binary payload
2024-05-29 15:54:02 +02:00
9b089a4e5a
fix: use explain instead of msg as error explanation and add test
2024-05-29 10:20:15 +02:00
479889f162
Merge pull request #13136 from kjellwinblad/kjell/fix_oracle_trace_format/EMQX-12433
...
fix(oracle action trace): parameters should not be rendered as IO Data
2024-05-29 09:12:38 +02:00
7d9333a9d7
chore: bump influxdb app version
2024-05-29 11:13:17 +08:00
00d74826ec
chore: improved InfluxDB connection error messaging
2024-05-29 11:13:17 +08:00
802361dbd0
fix(connector api): handle `timeout` when waiting for connector status
...
Fixes https://emqx.atlassian.net/browse/EMQX-12251
2024-05-28 14:19:32 -03:00
82800faadf
fix: make protobuf schema decode error reported to user less cryptic
...
Before this commit the user would see a cryptic warning log including
function_clause when decoding a message failed. This commit improves
this by removing function_caluse as well as some other cryptic words
from the message and adding a description describing what went wrong.
Fixes:
https://emqx.atlassian.net/browse/EMQX-12453
2024-05-28 18:07:14 +02:00
3151a3123e
chore: bump version of emqx_redis_bridge app
2024-05-28 17:23:35 +02:00
3f1fe413ee
refactor: simplify redis trace formatting
2024-05-28 17:22:58 +02:00
cbdde7165e
fix(redis trace): add separators in redis batch action trace
...
Logger will transform data that looks like IO data into a string which
made redis batch traces look like the spaces had been removed
from the strings. To prevent this, we render the batched commands into
binary string separated by spaces and semicolon. The components of a
single command is separated by spaces and the commands in a batch are
separated by semicolons.
Fixes:
https://emqx.atlassian.net/browse/EMQX-12428
2024-05-28 17:22:58 +02:00
5e3de1a4e3
fix(emqx_logger_jsonfmt:best_effort_json_test): test cases
2024-05-28 17:15:32 +02:00
40080f5e7d
Merge pull request #13141 from zmstone/0528-test-fix-dashboard-config-test-with-stable-base-config
...
test: fix test case to have deterministic base config
2024-05-28 14:32:55 +02:00
9ae4de10ed
Merge pull request #13127 from keynslug/fix/EMQX-12439/update-leaner-error
...
fix(bridge-v2): report descriptive error on invalid update request
2024-05-28 13:11:41 +03:00
e9cc88cb95
test: fix test case to have deterministic base config
2024-05-28 12:03:01 +02:00
84bc6ba28c
fix: trace formatter should not crash when given a non-binary payload
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-12474
2024-05-28 11:13:51 +02:00
e55bda6fe6
chore: bump app versions
...
Bump app version as required to pass the check which is done by the
following script:
./scripts/apps-version-check.sh
2024-05-28 11:01:00 +02:00
f65168982b
fix(oracle action trace): parameters should not be rendered as IO Data
...
This forces the parameters to the database statement to be rendered as
a JSON array in JSON traces instead of being rendered as a string when
the parameters are interpreted as IO data.
Fixes:
https://emqx.atlassian.net/browse/EMQX-12433
2024-05-28 10:59:44 +02:00
c04aaad0a4
chore(bridge-v2): leave comment describing need for `nowarn_function`
2024-05-28 10:31:14 +02:00
8fbeca4321
chore: Version bumps
2024-05-28 00:14:01 +02:00
04305d004f
refactor(sessds): Extract metadata keys to a header
2024-05-27 23:36:20 +02:00
16d1425982
refactor(emqx): Move files related to durable session to a subdir
2024-05-27 23:31:15 +02:00
2771a10d39
test: fix a flaky one
2024-05-27 20:59:50 +02:00
663f4fd39f
chore: bump application versions
2024-05-27 17:49:16 +02:00
08d88ea814
feat(bridge-api): improve error messages for Update Source API
2024-05-27 17:49:16 +02:00
ed7c29ec26
fix(api): mention spec override in "request metadata" type
2024-05-27 16:57:45 +02:00
5c2a68076f
fix(bridge-v2): report descriptive error on invalid update request
...
Before this commit, generic validation errors were reported as union
mismatches of _all_ of the bridge schemas. After this commit, specific
schema is chosen before validation.
2024-05-27 14:37:10 +02:00
f393fd5304
chore: release 5.7.0
2024-05-27 14:32:23 +02:00
b2f5e50f16
Merge pull request #13120 from ieQu1/dev/ds-api-site-status
...
fix(dsapi): Check status of the node via Mria
2024-05-25 05:42:05 +02:00
0763e29786
chore: prepare for 5.7.0-rc.2 release
2024-05-24 21:10:37 +02:00
73df01e934
fix(dsapi): Check status of the node via Mria
2024-05-24 20:59:07 +02:00
7f356aa3a8
Merge pull request #13115 from ieQu1/dev/fix-drop-generations2
...
Mitigate transient errors when dropping generations
2024-05-24 20:13:39 +02:00
830b62d899
fix(dsrepl): Retry sending ra commands to the leader
2024-05-24 19:09:34 +02:00
1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable
2024-05-24 19:09:34 +02:00
25721bceba
fix(dsstor): Delete generation metadata before dropping it
2024-05-24 19:09:33 +02:00
3d9837a0b8
test(ds): Add emqx_ds_replication_SUITE:t_drop_generation
2024-05-24 19:09:33 +02:00
327080e70e
perf(emqx_placeholder): do not try to decode maps as json
...
fix
2024-05-24 20:08:31 +03:00
c30b881239
test(monitor): verify historical and current metrics are consistent
2024-05-24 14:56:10 +02:00
fd9655bc35
fix(monitor): unify metric names for consistency and simplicity
2024-05-24 14:55:48 +02:00
7b137c0aaf
fix(monitor): reflect durable subscriptions in historical metrics
...
Essentially in the same manner as they are currently reflected in the
current metrics.
2024-05-24 14:08:35 +02:00
6ec8ae5a5f
Merge pull request #13112 from HJianBo/change-the-default-batch-size-s3-uploader
...
Fix(s3): change the batch_size to 1 for direct mode
2024-05-24 10:56:53 +02:00
29d203ae33
Merge pull request #13108 from keynslug/fix/ct-flaky/mgmt
...
test(mgmt): avoid inter-suite contamination by switching to cth tooling
2024-05-24 11:14:16 +03:00
adaf238131
fix(s3): fix the convert function if the mode is missing
2024-05-24 16:02:06 +08:00
5b60f6a5b2
fix: make dialyzer happy
2024-05-24 14:29:02 +08:00
43506495ae
fix: ensure plugin configured states after join cluster
2024-05-24 13:51:56 +08:00
874bcd9a8b
fix: match decode plugin config map failed case
2024-05-24 10:20:50 +08:00
c3c6da7ec4
chore: prepare for release cut 5.7.0-rc.1
2024-05-23 21:39:15 +02:00
3a4b8e6c24
Merge pull request #13106 from zmstone/0523-add-customer-type
...
0523 add customer type (4=BYOC) to license schema
2024-05-23 21:30:02 +02:00
40288b99b2
Merge pull request #13101 from JimMoen/fix-plugin-config-map
...
fix: allow put plugin config without schema
2024-05-23 21:18:43 +02:00
823a6bc521
Merge pull request #13107 from thalesmg/hide-clientv2-response-type-r57-20240523
...
fix(client mgmt api): hide unused return type
2024-05-23 16:03:51 -03:00
7878d106f6
test(mgmt): avoid inter-suite contamination by switching to cth tooling
2024-05-23 20:17:34 +02:00
b4c6968f8c
Merge pull request #13104 from keynslug/fix/EMQX-12415/last-resort-poll
...
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
2024-05-23 21:17:03 +03:00
Thales Macedo Garitezi