Commit Graph

13044 Commits

Author SHA1 Message Date
JianBo He b39557f6fd
Merge pull request #13336 from zhongwencool/authn-boostrap-file
feat: support bootstrap_file on authentication for build-in-database
2024-06-28 09:07:53 +08:00
zhongwencool 2c48d7e0f0
Merge pull request #13347 from zhongwencool/improve-check-oom-log
chore: log shutdown reason for check_oom trace log
2024-06-28 08:06:11 +08:00
Thales Macedo Garitezi 79f15b1daa test(schema registry): add test asserting the behavior of empty message roundtrip
Relates to https://emqx.atlassian.net/browse/EMQX-10866
2024-06-27 17:49:57 -03:00
Thales Macedo Garitezi e76e94b497 perf(mgmt): optimize bulk unsubscribe when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi 0b329dbf06 perf(mgmt): optimize bulk subscribe when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi c49900af50 perf(mgmt): optimize clientid lookup when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi 7b7f44b9ac fix(client mgmt api): make bulk subscribe work again in clusters
Fixes https://emqx.atlassian.net/browse/EMQX-12337
2024-06-27 09:37:55 -03:00
Thales Macedo Garitezi 6f00df6452 fix(schema registry api): validate schema name when creating
Fixes https://emqx.atlassian.net/browse/EMQX-10958
2024-06-27 09:26:41 -03:00
Thales Macedo Garitezi ff8c2bc1d8 feat(authz): add `ignore` metric for each source type
Fixes https://emqx.atlassian.net/browse/EMQX-12411
2024-06-27 09:22:45 -03:00
Thales Macedo Garitezi 4bd0abc93f chore: bump app vsns 2024-06-27 09:22:06 -03:00
Thales Macedo Garitezi 4c3c86e919 chore: bump wolff -> 2.0.0 2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi 164a507899 test(pulsar): add testcase for different producers using the same topic 2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi ed5e6599d9 fix(buffer worker, kafka): send reply when async call fails immediately
Fixes https://emqx.atlassian.net/browse/EMQX-12585
2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi 6023012f8b fix(kafka and derivatives): add `alias` config to avoid clashes with same topic
Fixes https://emqx.atlassian.net/browse/EMQX-12592
2024-06-27 09:20:00 -03:00
JianBo He 3b21c41690 feat: support to extract the client peersni field to clientinfo 2024-06-27 17:13:10 +08:00
zmstone b6e7d7566d Merge remote-tracking branch 'origin/release-572' into release-57 2024-06-27 11:09:02 +02:00
JimMoen ef28579c4a
Merge pull request #13334 from JimMoen/fix-mqtt-username-password-flag
fix(mqtt): check password flag to respect protocol spec
2024-06-27 16:16:53 +08:00
zhongwencool 6a78951715 chore: log shutdown reason for check_oom trace log 2024-06-27 14:52:22 +08:00
lafirest 68d747b7b9
Merge pull request #13324 from lafirest/feat/oidc-572
feat(sso): add OIDC support
2024-06-27 11:24:05 +08:00
firest 3d398873f1 fix(oidc): return to dashboard when provider calls back
fixed a bug when updating config
2024-06-27 10:22:07 +08:00
zhongwencool 9594b6df32 chore: warning overrided when restart authn 2024-06-27 10:16:04 +08:00
zhongwencool 24d2534641 chore: apply review suggestion 2024-06-27 10:16:04 +08:00
zhongwencool 5265c3cc1f feat: support bootstrap_file on build-in-db authn 2024-06-27 10:16:04 +08:00
zmstone 557a843c69 chore: minimize oss/platform diff 2024-06-26 22:48:50 +02:00
zmstone 82e7b75a02 chore: bump app versions 2024-06-26 22:47:18 +02:00
zmstone 98f70ea8d8 Merge remote-tracking branch 'origin/release-572' into release-57 2024-06-26 22:28:08 +02:00
Andrew Mayorov d8963c836e
Merge pull request #13332 from keynslug/ft/EMQX-12571/error-mapping
feat(bridge-s3): provide more meaningful error details in status
2024-06-26 21:13:51 +02:00
Andrew Mayorov e1de18ef10
test(dsrepl): await stable state before running testcase 2024-06-26 20:44:35 +02:00
Andrew Mayorov 30efa1f57e
test(dsrepl): relax crash-recover testcase to tolerate message loss
Which is quite an expected occasion for this kind of stress test.
2024-06-26 20:44:34 +02:00
Andrew Mayorov 3d296abde9
fix(dsrepl): classify ra error conditions more carefully
Most importantly: avoid automatic retries of `shutdown` and `nodedown`
errors as this could easily lead to Raft log entries duplication.
2024-06-26 20:44:34 +02:00
Andrew Mayorov 733751fadd
refactor(dsstore): keep passing `Options` to both prepare + commit 2024-06-26 20:44:34 +02:00
Andrew Mayorov 5b5f33c421
chore(dsstore): resurrect `prepare_batch` entry tracepoint 2024-06-26 20:44:34 +02:00
Andrew Mayorov 8ff48ac5ea
feat(dsrepl): rely on accumulated log size to decide when to flush 2024-06-26 20:44:34 +02:00
Andrew Mayorov b6a249baa9
feat(cth-peer): add brutal `kill/1` facility 2024-06-26 20:44:34 +02:00
Andrew Mayorov 8db70b5bbc
test(dsrepl): add crash-restart-recover testcase
That verifies nothing is lost in the event of abrupt node failures.
2024-06-26 20:44:33 +02:00
Andrew Mayorov ae89b61af0
feat(cth-cluster): make `restart/1` more generic 2024-06-26 20:44:33 +02:00
Andrew Mayorov 5fd5fc76e5
fix(dsstore): ensure backward compatibility 2024-06-26 20:44:33 +02:00
Andrew Mayorov 8538a5a5b6
test(dsrepl): anticipate transitionless membership changes
E.g. when a membership change is applied twice in a row.
2024-06-26 20:44:33 +02:00
Andrew Mayorov cd0663074e
test(dsrepl): add `add_generation` events into the mix
They usually cause storage layer to perform flushes, and thus enable
testing `handle_flush/1` codepath in different circumstances.
2024-06-26 20:44:33 +02:00
Andrew Mayorov 2180cc7c26
fix(dsstore): avoid storing `cf_refs()` in the RocksDB itself
This is both pointless and confusing.
2024-06-26 20:44:33 +02:00
Andrew Mayorov 80ea2e62f7
fix(stream): ensure that `chain/1` preserves the order 2024-06-26 20:44:32 +02:00
Andrew Mayorov 7895e9cc45
feat(dsstore): make WAL-less mode optional
And make the upper layer choose when to use it.
2024-06-26 20:44:32 +02:00
Andrew Mayorov 0c0757b8c2
feat(dsrepl): enable WAL-less batch writes 2024-06-26 20:44:32 +02:00
Andrew Mayorov 2705226eb5
feat(dsrepl): release log entries occasionally
Also make tracepoints in `apply/3` callback implementation more
uniform.
2024-06-26 20:44:32 +02:00
Andrew Mayorov b6894c18fa
chore(dsrepl): improve tracepoints usability a bit 2024-06-26 20:44:32 +02:00
Thales Macedo Garitezi 263e654208
Merge pull request #13340 from thalesmg/20240626-r572-fix-docs-conf-load
fix: don't crash if application is already loaded
2024-06-26 15:15:49 -03:00
Thales Macedo Garitezi e81494a132 fix: don't crash if application is already loaded
```
Runtime terminating during boot ({{badmatch,{error,{already_loaded,wolff}}},[{emqx_conf,load,2,[{file,"emqx_conf.erl"},{line,167}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1686}]},{emqx_conf,dump_schema,2,[{file,"emqx_conf.erl"},{line,150}]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,746}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,494}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,136}]},{init,start_it,1,[]},{init,start_em,1,[]}]})
```
2024-06-26 13:41:32 -03:00
Thales Macedo Garitezi 954adc71c4 test: attempt to fix flaky test
https://github.com/emqx/emqx/actions/runs/9662725303/job/26653594859?pr=13328#step:6:186

```
%%% emqx_resource_SUITE ==> t_expiration_retry: FAILED
%%% emqx_resource_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,error,
               {badmatch,{ok,timeout}},
               [{emqx_resource_SUITE,'-do_t_expiration_retry/0-fun-12-',0,
                    [{file,
                         "/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
                     {line,2569}]},
                {emqx_resource_SUITE,do_t_expiration_retry,0,
                    [{file,
                         "/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
                     {line,2518}]}]}}},
 [{emqx_resource_SUITE,do_t_expiration_retry,0,
      [{file,"/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
       {line,2594}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]}
```
2024-06-26 13:40:49 -03:00
zmstone ef1b6d2a55 chore: keep enterprise edition version updated as well 2024-06-26 11:40:49 +02:00
zmstone 465ae507d8 chore: update release version to v5.7.1 2024-06-26 10:49:59 +02:00
zmstone af9bfc21cb
Merge pull request #13312 from zmstone/0621-fix-regex_extract-docs
doc: fix regex_extract function doc
2024-06-26 10:48:40 +02:00
zmstone 7f73f27d56
docs: fix typo 2024-06-26 10:18:08 +02:00
zhongwencool 4942f6f75a feat: improve keepalive_multiplier and keepalive_check_interval 2024-06-26 15:43:48 +08:00
Thales Macedo Garitezi 21c01f32ff
Merge pull request #13305 from kjellwinblad/kjell/redis_conn_always_time_out_no_username_password/EMQX-12557
fix: redis connector should not timeout because no username and password
2024-06-25 16:06:21 -03:00
Andrew Mayorov da214be5a1
test(bridge-s3): adapt testcase to different CI environment 2024-06-25 18:35:53 +02:00
Andrew Mayorov 10e9fed22b
Merge pull request #13326 from keynslug/fix/ct/api-clients-flaky
test(api-clients): refactor suite to isolate flakiness
2024-06-25 18:26:42 +02:00
JimMoen ed130fdc57
test: MQTT CONNECT flags check 2024-06-25 23:25:42 +08:00
JimMoen 02a9885aa5
fix(mqtt): check password flag to respect protocol spec 2024-06-25 23:25:36 +08:00
JimMoen 983f02ea1b
refactor: separate CONNECT flags validation funcs 2024-06-25 23:15:39 +08:00
Kjell Winblad 130571b56e fix: code improvements thanks to comments from @thalesmg 2024-06-25 12:00:36 -03:00
Kjell Winblad 6190192cbc fix: redis connector should not timeout because no username and password
A redis connector of type single or sentinel always got a timeout error
when doing the connector test in the dashboard if no username or
password was provided. This commit makes sure that the user instead get
an informative error message. Additionally, this commit adds more
more error information for all redis connector types.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12557
2024-06-25 12:00:36 -03:00
Thales Macedo Garitezi cc89a20ed9
Merge pull request #13318 from thalesmg/test-flaky-gcp-consu-r57-20240621
test(gcp consumer): stabilize flaky test
2024-06-25 11:52:20 -03:00
Andrew Mayorov 486a041adf
feat(bridge-s3): also map credentials / aggreg upload errors 2024-06-25 16:51:07 +02:00
Andrew Mayorov fb9afd8313
feat(bridge-s3): beautify posix write errors 2024-06-25 16:51:07 +02:00
Andrew Mayorov f3ffbd4710
feat(bridge-s3): provide more meaningful error details in status 2024-06-25 16:51:06 +02:00
Andrew Mayorov 98e4ea6fde
feat(bridge-s3): make validation errors more readable
And also turn them into schema-level validations, instead of
bridge-level error conditions.
2024-06-25 16:51:06 +02:00
Thales Macedo Garitezi 1a497bcaf2 fix(greptime): correctly define grpc options for `grpcbox_channel`
Will probably fix CI flakiness.
2024-06-25 09:47:31 -03:00
Thales Macedo Garitezi c9ec5ac87b test(gcp consumer): stabilize flaky test
https://github.com/emqx/emqx/actions/runs/9614788348/job/26526973635?pr=13317#step:5:1463

```
%%% emqx_bridge_gcp_pubsub_consumer_SUITE ==> t_connection_down_before_starting: FAILED
%%% emqx_bridge_gcp_pubsub_consumer_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,error,
               {assertMatch,
                   [{module,emqx_bridge_gcp_pubsub_consumer_SUITE},
                    {line,1451},
                    {expression,"health_check ( Config )"},
                    {pattern,"{ ok , connecting }"},
                    {value,{ok,disconnected}}]},
               [{emqx_bridge_gcp_pubsub_consumer_SUITE,
                    '-t_connection_down_before_starting/1-fun-11-',4,
                    [{file,
                         "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
                     {line,1451}]},
                {emqx_bridge_gcp_pubsub_consumer_SUITE,
                    t_connection_down_before_starting,1,
                    [{file,
                         "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
                     {line,1427}]}]}}},
 [{emqx_bridge_gcp_pubsub_consumer_SUITE,t_connection_down_before_starting,1,
      [{file,
           "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
       {line,1462}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]}
```
2024-06-25 09:36:55 -03:00
ieQu1 9f30da334f
Merge pull request #13248 from ieQu1/dev/EMQX-12491-local-backend
replace builtin DS backend with builtin_local and builtin_raft
2024-06-24 23:07:57 +02:00
ieQu1 d349f84f04
fix(ds): Apply remarks 2024-06-24 20:20:40 +02:00
Andrew Mayorov d12b985507
test(api-clients): refactor suite to isolate flakiness 2024-06-24 16:54:20 +02:00
zmstone 5d1608f34b chore: minimize oss/platform diff 2024-06-24 15:30:03 +02:00
Thales Macedo Garitezi debf1e6cd5
Merge pull request #13316 from thalesmg/fix-mt-read-user-prop-m-20240621
fix(message transformation): correctly read from user properties in context
2024-06-24 09:31:48 -03:00
firest abc255bb02 fix(oidc): make CI happy 2024-06-24 18:11:02 +08:00
firest 892420e2c6 feat(oidc): be more compatible with okta 2024-06-24 16:51:54 +08:00
firest ddb197951e feat(oidc): implement JWKS, private_key_jwt, DPoP 2024-06-24 16:51:43 +08:00
firest 9c0df3c0a8 feat(oidc): support the PKCE extension 2024-06-24 16:51:32 +08:00
firest 5e2693c9b4 feat(oidc): implement session management 2024-06-24 16:51:20 +08:00
firest 512b4b9cbb feat(sso): add OIDC support 2024-06-24 16:50:58 +08:00
JianBo He 23acbb664e
Merge pull request #13258 from HJianBo/fix-boot-order-for-gws
fix: gateway app must depends on all auth apps
2024-06-24 16:31:27 +08:00
zmstone f9e17d6c25
Merge pull request #13269 from JimMoen/improve-plugin-behavior
fix: mark fresh install to cp the default configuration file directly
2024-06-24 09:17:25 +02:00
Ivan Dyachkov 4a81c3ac18
Merge pull request #13298 from qzhuyan/fix/william/hot-conf-disable-partial-chain
fix(tls): disable partial_chain in hot config
2024-06-24 09:02:07 +02:00
ieQu1 c0472a06f1
test(ds): Set initial conditions in repl. suite explicitly 2024-06-22 02:55:05 +02:00
Thales Macedo Garitezi ec83d999bf Merge remote-tracking branch 'origin/release-57' into 20240621-sync-r57-m 2024-06-21 14:11:08 -03:00
ieQu1 9a58d71378
fix(ds): Move DS backend supervision trees to their own apps 2024-06-21 18:22:04 +02:00
ieQu1 3d69ec496a
fix(schema): Transform config of `singleton' discovery_strategy 2024-06-21 17:16:25 +02:00
ieQu1 d7d878fd43
test(ds): Fix emqx standalone test profile selection 2024-06-21 17:16:25 +02:00
ieQu1 8aa27488b6
test: Disable certain DS-related suites in CE 2024-06-21 17:16:24 +02:00
ieQu1 1d3b1868fb
test(ds): Use close_db API 2024-06-21 17:16:24 +02:00
ieQu1 b2f7815a7f
test(ds): Don't start raft explicitly 2024-06-21 17:16:24 +02:00
ieQu1 3851fc189f
fix(ds): Avoid reverse dependencies from storage to repl. layer 2024-06-21 17:16:24 +02:00
ieQu1 bc915216a0
feat(ds): Support metrics for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 be6c5e172f
fix(ds): Disable DS management APIs for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 99c9b56cf3
feat(ds_buffer): Add `ls' function to list all local buffers 2024-06-21 17:16:24 +02:00
ieQu1 5a8818edf3
feat(ds): Add schema for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 4484f30021
feat(ds_schema): Add schema for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 8990b1312b
refactor(ds_schema): Rename backend builtin -> builtin_raft 2024-06-21 17:16:24 +02:00
ieQu1 ea48b1265d
refactor(ds_schema): Extract common builtin fields 2024-06-21 17:16:24 +02:00
ieQu1 abe41de19b
refactor(ds_schema): builtin_local_buffer -> builtin_buffer 2024-06-21 17:16:24 +02:00
ieQu1 ecb172b07e
refactor(ds): Rename egress metrics to 'buffer' 2024-06-21 17:16:24 +02:00
ieQu1 09c3ae795d
refactor(ds_raft): Replace egress server with common emqx_ds_buffer 2024-06-21 17:16:24 +02:00
ieQu1 a0fbd37e58
refactor(emqx): Use emqx_ds_backends application 2024-06-21 17:16:24 +02:00
ieQu1 ef09cfcd71
feat(ds): Add `emqx_ds_backends` application 2024-06-21 17:16:24 +02:00
ieQu1 279619fc80
feat(ds): Add `builtin_local' backend 2024-06-21 17:16:24 +02:00
ieQu1 a8ea0ae4e5
refactor(ds): Extract DS replication layer to a separate application 2024-06-21 17:16:24 +02:00
ieQu1 63f1856a2c
feat(ds): Dynamic backend registration 2024-06-21 17:16:24 +02:00
ieQu1 83dc8f4d77
fix(ds): Fix return values of emqx_ds_storage_layer functions 2024-06-21 17:16:23 +02:00
ieQu1 a18d1987a2
test(ds): Add a helper function for diffing messages 2024-06-21 17:16:23 +02:00
Ilia Averianov f1b8c356a6
Merge pull request #13299 from savonarola/0617-shared-sub-leader-poc
feat(queue): implement PoC version of session ↔️ shared group leader interaction
2024-06-21 16:45:17 +03:00
Thales Macedo Garitezi 28c7d94bd2 fix(message transformation): correctly read from user properties in context
Fixes https://emqx.atlassian.net/browse/EMQX-12582
2024-06-21 09:39:22 -03:00
Ilya Averyanov db28a042d5 feat(queue): handle renew_lease_timeout 2024-06-21 15:18:37 +03:00
zhongwencool 75a524c916 test: add more debug msg to flaky cluster_rpc SUITE 2024-06-21 18:04:24 +08:00
Ilya Averyanov 2096755ad6 feat(queue): reorganize group sm callbacks and methods 2024-06-21 13:01:38 +03:00
Ilya Averyanov b9c5911883 feat(queue): implement `find_leader_timeout` event and so the `connecting` group_sm state 2024-06-21 12:22:17 +03:00
SergeTupchiy fb266fbf8c
Merge pull request #13300 from SergeTupchiy/EMQX-12195-cluster-link-conf-backup
cluster link conf backup
2024-06-21 11:42:57 +03:00
zmstone 8a99995810 docs: fix regex_extract function doc 2024-06-21 09:25:09 +02:00
JianBo He d8e6c07ca0 chore: fix failed tests 2024-06-21 11:04:23 +08:00
JianBo He a8af90f912 chore: add changes 2024-06-21 09:39:14 +08:00
JianBo He 08f085d823 fix: gateway app must depends on all auth apps 2024-06-21 09:39:14 +08:00
Ilya Averyanov 979fb58e50 feat(queue): use tp to trace new lease events 2024-06-20 21:12:15 +03:00
Ilya Averyanov 1205e34650 feat(queue): remove too useless header file 2024-06-20 20:48:56 +03:00
JimMoen acc8bf3405
Merge pull request #13222 from JimMoen/fix-will-qos-retain-flag-check
fix: check willretain and willqos when WillFlag set to `true`
2024-06-21 01:28:23 +08:00
Ivan Dyachkov ba4fb0d3f9
Merge pull request #13290 from kjellwinblad/kjell/format_rule_cli/EMQX-12548
fix: do not crash when showing rule unsing the cmd interface
2024-06-20 14:55:33 +02:00
zmstone 2a0071aa01
Merge pull request #13295 from thalesmg/test-flaky-greptime-mkII-r57-20240619
test(greptime): attempt to fix flaky tests (attempt without driver patch)
2024-06-20 14:49:21 +02:00
Ilya Averyanov be175d205c feat(queue): add JIRA task links for TODOs 2024-06-20 15:38:16 +03:00
Ilya Averyanov a9c976b6c1 feat(queue): fix notes about the usage of agent<->leader protocol-related data structures 2024-06-20 15:24:25 +03:00
Ilya Averyanov d0cff63ed6 feat(queue): remove unused hrl-guard 2024-06-20 15:18:47 +03:00
Ilya Averyanov a92460d38f feat(queue): improve hints for macro clauses 2024-06-20 15:17:08 +03:00
Serge Tupchii ac8762be10 chore: upgrade ekka to 0.19.5 (mria 0.8.8)
mria 0.8.8 heals a network partition once majority of core nodes are alive.
Previously, the autoheal worked only when all core nodes were reachable.

Fixes: EMQX-10974
2024-06-20 12:53:58 +03:00
William Yang 464a0a82f0 fix(tls): move default_root_fun to ?CONST_MOD_V1 2024-06-20 11:38:52 +02:00
zmstone 7d32275ebe chore: minimize oss/platform diff 2024-06-20 10:36:01 +02:00
Kjell Winblad 12859b8959 fix: handle yet another action format 2024-06-20 10:03:49 +02:00
JimMoen 675abd7512
test: will retain and willqos in connect flags 2024-06-20 10:32:15 +08:00
JimMoen cf9d6943d5
fix: check willretain and willqos when WillFlag set to `true` 2024-06-20 10:32:15 +08:00
JianBo He c307c1dfc2 chore: cut v5.7.1-alpha.1 2024-06-20 09:40:42 +08:00
Ilya Averyanov 6da10036dc feat(queue): fix dialyzer issues 2024-06-19 22:25:53 +03:00
Ilya Averyanov a4bbab4aa2 feat(queue): document general entity layout 2024-06-19 22:24:20 +03:00
JimMoen ff742d926a
fix: `?SLOG` with msg field 2024-06-20 02:18:26 +08:00
Serge Tupchii 71f5eaf11e fix(data_backup): import `cluster.links` conf 2024-06-19 20:47:06 +03:00
Serge Tupchii ef5cf4fac3 chore(clusterlink): fix comment 2024-06-19 20:46:00 +03:00
Ilya Averyanov 2b0146663a feat(queue): add smoke test for ds shared subs 2024-06-19 19:32:23 +03:00
Ilya Averyanov e3c4816035 feat(queue): move group subscription state machine to its own module 2024-06-19 19:22:10 +03:00
Thales Macedo Garitezi 9559ba2f06 test: remove unnecessary lines 2024-06-19 13:22:02 -03:00
William Yang 8d04545f03 test(tls): root_fun is set even when partial_chain is false 2024-06-19 17:34:04 +02:00
William Yang 41239ae766 fix(tls): disable partial_chain in hot config 2024-06-19 17:03:05 +02:00
zmstone 021b6b3902 Merge remote-tracking branch 'origin/release-57' into 0619-sync-release-57-to-master 2024-06-19 15:06:41 +02:00
zmstone 02a6ee1ef4
Merge pull request #13279 from zmstone/0617-json-decode-payload-for-map_to_redis_hset_args
fix(redis): json decode if arg for map_to_redis_hset_args is string
2024-06-19 14:55:21 +02:00
zmstone 213e4785e7 Merge remote-tracking branch 'origin/release-57' into 0619-sync-release-57-to-master 2024-06-19 14:54:48 +02:00
JimMoen a6fa3e82d9
fix: prevent use pd 2024-06-19 20:15:07 +08:00
JimMoen b7b2a08399
fix: match plugin_not_found reason 2024-06-19 20:15:07 +08:00
JimMoen 28293284a7
fix: let crash when plugin not found in cluster 2024-06-19 20:15:07 +08:00
JimMoen 19e039e0d2
fix: mark fresh install to cp the default configuration file directly 2024-06-19 20:15:07 +08:00
Thales Macedo Garitezi ff2e6e1434 test(greptime): revert driver patch to test ci again 2024-06-19 09:12:30 -03:00
zmstone 95d387a790
Merge pull request #13294 from zmstone/0618-per-release-edition-code-injection
0618 per release edition code injection
2024-06-19 11:53:57 +02:00
zmstone f0e5721959 refactor: make dialyzer happy 2024-06-19 10:42:08 +02:00
SergeTupchiy 0d098a01ef
Merge pull request #13293 from SergeTupchiy/EMQX-12345-reindex-retained-msgs-on-import
reindex retained msgs on import
2024-06-19 11:23:08 +03:00
Thales Macedo Garitezi ffec1c7fe0 fix(greptime): attempt patch for cleaning up channels when worker terminates 2024-06-18 22:16:03 -03:00
Thales Macedo Garitezi 9dd419d822 test(greptime): attempt to fix flaky tests
```
=WARNING REPORT==== 18-Jun-2024::12:42:04.739109 ===
    reason: no_endpoints
    msg: failed_to_start_greptimedb_connector
    client: #{protocol => http,
              pool => <<"connector:greptimedb:emqx_bridge_greptimedb_SUITE">>,
              cli_opts =>
                  [{endpoints,[{http,"toxiproxy",4001}]},
                   {pool_size,2},
                   {pool_type,random},
                   {auto_reconnect,1},
                   {gprc_options,#{sync_start => true,
                                   connect_timeout => 5000}},
                   {dbname,"public"},
                   {auth,{basic,#{password => <<"******">>,
                                  username => "greptime_user"}}},
                   {https_enabled,false}]}
    connector: <<"connector:greptimedb:emqx_bridge_greptimedb_SUITE">>
```
2024-06-18 18:23:22 -03:00
zmstone 54fc605cc5 refactor: move schema injection from hrl to erl 2024-06-18 22:22:53 +02:00
zmstone a53fa977df refactor: delete an unused macro 2024-06-18 22:22:48 +02:00
Thales Macedo Garitezi e69ba33420
Merge pull request #13291 from thalesmg/fix-ds-mgmt-node-down-r57-20240618
fix(dsreplmeta): check site status when fetching shard info
2024-06-18 15:14:08 -03:00
Ilya Averyanov bca743054b feat(queue): implement backbones of queue agent, leader and leader registry 2024-06-18 21:03:51 +03:00
Thales Macedo Garitezi 4c54ab6379 test: remove tests per review request 2024-06-18 13:08:19 -03:00
Kjell Winblad 4a318f8f51 style: fix type problem found by dialyzer 2024-06-18 17:02:26 +02:00
Serge Tupchii baa7996289 fix(data_backup): allow exporting `ram_copies` Mnesia tables
Currently, ram tables can be used for message retainer.
2024-06-18 17:31:33 +03:00
Serge Tupchii 60882a616e fix(data_backup): re-index retained messages after importing 2024-06-18 17:31:19 +03:00
Thales Macedo Garitezi 6bde6aa711
Merge pull request #13280 from thalesmg/fix-plugin-elixir-r57-20240617
fix(plugins): do not stop Elixir protected apps
2024-06-18 10:41:12 -03:00
Thales Macedo Garitezi 8ce16fd7d9 fix(dsreplmeta): check site status when fetching shard info
Fixes https://emqx.atlassian.net/browse/EMQX-12356
2024-06-18 10:40:09 -03:00
Kjell Winblad a4f855108c fix: do not crash when showing rule unsing the cmd interface
Before the change the command line command

    $ bin/emqx ctl rules show rule_0hyd

would crash if the rule had bridge actions. This has now been fixed by
adding a handler for printing  bridge actions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12548
2024-06-18 15:17:25 +02:00
Kjell Winblad 3f7723b2dc
Merge pull request #13274 from kjellwinblad/kjell/add_ssl_to_rocksdb/EMQX-12289
feat: add SSL support to RocketMQ connector
2024-06-18 09:38:28 +02:00
Thales Macedo Garitezi 20cffb54d4 Merge branch 'release-57' into sync-r57-m-20240617 2024-06-17 17:10:12 -03:00
Thales Macedo Garitezi 6897f0141b fix(plugins): do not stop Elixir protected apps
On one hand, Elixir plugins might include Elixir itself, when targetting a non-Elixir EMQX
release.  If, on the other hand, the EMQX release already includes Elixir, we shouldn't
stop Elixir nor IEx, or the running IEx shell will break.
2024-06-17 16:42:23 -03:00
Ilia Averianov ad993437aa
Merge pull request #13278 from savonarola/0617-release-57-sync
Sync release-57
2024-06-17 21:25:00 +03:00
zmstone 9479c8d33b fix(redis): json decode if arg for map_to_redis_hset_args is string 2024-06-17 20:19:53 +02:00
Andrew Mayorov eede9f349e
Merge pull request #13276 from keynslug/fix/ds/inherited-lts-lost
fix(dsstore): persist inherited LTS trie
2024-06-17 20:25:31 +03:00
zmstone ae22a64157
Merge pull request #13277 from zmstone/0617-handle-kafka-message_too_large-return
0617 handle kafka message too large return
2024-06-17 19:15:33 +02:00
Kjell Winblad 0cbbae9655 fix: make fixes due to comments from @thalesmg 2024-06-17 19:06:12 +02:00
Kjell Winblad c39c544c96 feat: add SSL support to RocketMQ connector 2024-06-17 19:06:12 +02:00
Thales Macedo Garitezi b64f0c0ca7
Merge pull request #13231 from thalesmg/mt-test-endpoint-r57-20240611
feat(message transformation): implement dryrun endpoint
2024-06-17 13:11:39 -03:00
Thales Macedo Garitezi 240bb09070
Merge pull request #13275 from thalesmg/test-flaky-sk-r57-20240617
test: attempt to fix flaky test
2024-06-17 13:01:08 -03:00
Ilya Averyanov f8e6aab86f Merge branch 'release-57' into 0617-release-57-sync
* release-57:
  chore(auth,http): cache REs for parsing URIs
  fix(auth,http): improve URI handling
  chore: revert ULOG/ELOG
  test: generate dispatch.eterm in dashboard test
  docs: refine change log
  feat: make the dashboard restart quicker
  chore: fix typo
  fix(http authz): handle unknown content types in responses
  chore: change types of mysql and mongodb fields to `template()`
  fix(client mgmt api): allow projecting `client_attrs` from client fields
  fix(emqx_rule_funcs): expose regex_extract function to rule engine
2024-06-17 18:53:39 +03:00
Andrew Mayorov 82588fbc35
chore(dsstore): refine module and callback typespecs 2024-06-17 17:53:39 +02:00
Andrew Mayorov 05d97397d3
fix(dsstore): use correct comparison when walking over generations 2024-06-17 17:53:39 +02:00
Andrew Mayorov 22009bcc58
test(dsstore): verify that inherited TLS trie is persisted 2024-06-17 17:53:38 +02:00
Andrew Mayorov 68f6556856
fix(dsstore): persist inherited LTS trie
Before this commit, inherited trie was actually only kept in memory
cache.

Also simplify storage backend behaviour around inheriting previous
generation's legacy.
2024-06-17 17:53:38 +02:00
SergeTupchiy 830266b4d5
Merge pull request #13126 from SergeTupchiy/EMQX-11967-cross-cluster-route-replication-syncer-actor
cluster linking
2024-06-17 18:52:46 +03:00
zmstone 3ac4ddcbe3 fix(kafka): handle message_too_large
bump 'failed' counter
2024-06-17 17:27:25 +02:00
zmstone 00dfdc22cf chore: pin Kafka client wolff 0.10.5 2024-06-17 17:27:25 +02:00
zhongwencool 5ab4b321a0
Merge pull request #13267 from zhongwencool/use-redbug-for-trace
chore(debug): use redbug for default trace tools
2024-06-17 22:10:18 +08:00
Thales Macedo Garitezi fb492e3dc5
Merge pull request #13264 from thalesmg/fix-postgres-disabled-prepared-r57-20240614
fix(postgres): authn/authz/batch requests when prepared statements are disabled
2024-06-17 11:05:05 -03:00
Thales Macedo Garitezi 5665a4917c
Merge pull request #13266 from thalesmg/fix-s3-release-app-r57-20240614
fix: add `emqx_s3` to `reboot_lists.eterm`
2024-06-17 10:48:12 -03:00
Thales Macedo Garitezi ec4f462684 refactor: apply review requests 2024-06-17 10:39:32 -03:00
Thales Macedo Garitezi 1464094967 test: attempt to fix flaky test
```
%%% emqx_persistent_session_ds_SUITE ==> t_session_discard_persistent_to_persistent: FAILED
%%% emqx_persistent_session_ds_SUITE ==> {noproc,{gen_server,call,[snabbkaffe_collector,flush_trace,infinity]}}
```
2024-06-17 10:34:57 -03:00
Thales Macedo Garitezi b71212b022 refactor: apply review comments 2024-06-17 09:40:33 -03:00
Thales Macedo Garitezi f5eb3e7471
Merge pull request #13250 from thalesmg/ds-singleton-m-20240613
feat: enforce singleton discovery strategy when using DS `builtin_local` backend
2024-06-17 09:34:05 -03:00
Ilya Averyanov 19c9f0d76f chore(auth,http): cache REs for parsing URIs 2024-06-17 14:42:29 +03:00
Ilya Averyanov f7ac829f28 fix(auth,http): improve URI handling 2024-06-17 14:42:29 +03:00
Serge Tupchii a905a6048c chore(clusterlink): rename link `upstream` field to `name` 2024-06-17 12:24:51 +03:00
Serge Tupchii a95a08efd3 test(clusterlink): add more test cases 2024-06-17 12:24:51 +03:00
Serge Tupchii 44c37571cc fix(clusterlink): ignore not_registered error 2024-06-17 12:24:51 +03:00
Serge Tupchii 00f912928f fix: fix emqx_external_broker:forward/1 spec 2024-06-17 12:24:51 +03:00
Andrew Mayorov ed16ff07df refactor(broker): simplify external broker behaviour 2024-06-17 12:24:51 +03:00
Serge Tupchii d282c61120 feat(clusterlink): update only necessary resources when a link config is changed 2024-06-17 12:24:50 +03:00
Serge Tupchii ff16521d4f fix(clusterlink): add schema descriptions, fix dialyzer warnings, add resource_opts 2024-06-17 12:23:52 +03:00
Serge Tupchii 94e81ba812 feat(clusterlink): implement actor config handler 2024-06-17 12:23:52 +03:00
Serge Tupchii c871b37453 fix(clusterlink): add link topics schema validator 2024-06-17 12:23:52 +03:00
Andrew Mayorov 780a0bf807 fix(cluster-link): clear exit signal of failed-to-connect client 2024-06-17 12:23:52 +03:00
Andrew Mayorov d0df4de2a3 test(cluster-link): add e2e replication actor GC testcase 2024-06-17 12:23:52 +03:00
Andrew Mayorov c4840b30d2 fix(cluster-link): deduplicate routes down to dest cluster 2024-06-17 12:23:52 +03:00
Andrew Mayorov ede35df24a fix(cluster-link): cancel heartbeats on client down 2024-06-17 12:23:52 +03:00
Andrew Mayorov e0604e3af6 fix(cluster-link): anticipate clients may occasionally retry 2024-06-17 12:23:52 +03:00
Andrew Mayorov 0219b8bd4d feat(cluster-link): add simple replication actor GC process 2024-06-17 12:23:52 +03:00
Andrew Mayorov 7fccb5dbc9 test(topic): add more `intersection/2` testcases 2024-06-17 12:23:52 +03:00
Andrew Mayorov 7b8f466adf feat(topic): avoid `lists:reverse` when intersecting 2024-06-17 12:23:52 +03:00
Andrew Mayorov 24be189728 fix(topic): respect special topic rules when intersecting 2024-06-17 12:23:51 +03:00
Andrew Mayorov de1ac131f7 test(cluster-link): fix test suite setup 2024-06-17 12:23:51 +03:00
Serge Tupchii 58eaf07627 fix(clusterlink): valide config to disallow duplicated cluster links 2024-06-17 12:23:51 +03:00
Andrew Mayorov 54d51d0982 test(cluster-link): draft basic integration test suite 2024-06-17 12:23:51 +03:00
Andrew Mayorov e9c24090d4 fix(cluster-link): avoid starting ps syncer if persistence disabled 2024-06-17 12:23:51 +03:00
Andrew Mayorov 036c7e8492 test(cluster-link): adapt extrouter testsuite to new APIs 2024-06-17 12:23:51 +03:00
Andrew Mayorov 45eda4f3b9 fix(cluster-link): adapt heartbeat / reincarnation handling to new API 2024-06-17 12:23:51 +03:00
Andrew Mayorov 43d114546c feat(cluster-link): preserve replication actor state in pdict 2024-06-17 12:23:51 +03:00
Andrew Mayorov 5771a41a32 fix(cluster-link): ensure replication actor bootstraps do heartbeats 2024-06-17 12:23:51 +03:00
Andrew Mayorov d4b449c6e1 feat(cluster-link): implement replication actor heartbeats 2024-06-17 12:23:51 +03:00
Serge Tupchii faa4420e1f fix(clusterlink): improve actor error handling
Add status and error reason to the actor state, report alarms.
2024-06-17 12:23:51 +03:00
Serge Tupchii 21711c6e0d fix(clusterlink): communicate bootstrap requirements via actor handshake
`session_present` flag is not reliable to decide whether bootstrap is needed if durable sessions is enabled.
In this case, the client session may survive cluster restart while all the external routes are lost, as they are not persistent.
2024-06-17 12:23:51 +03:00
Serge Tupchii d5e82cdfac refactor(clusterlink): avoid unnecessary `maybe_` external_broker CB names 2024-06-17 12:23:51 +03:00
Serge Tupchii d578ac3f9e fix(clusterlink): match correct timer ref in router actor 2024-06-17 12:23:51 +03:00
Serge Tupchii b1aeb35370 feat(clusterlink): implement actor init handshake 2024-06-17 12:23:51 +03:00
Serge Tupchii ac19cf89df chore(clusterlink): remove code related to the rejected coordinator-based implementation 2024-06-17 12:23:51 +03:00
Serge Tupchii e26e7acaa1 refactor(clusterlink): use `emqx_bridge_mqtt_lib:clientid_base/1` to construct routesync client id 2024-06-17 12:23:51 +03:00
Serge Tupchii e7305c62ee feat(clusterlink): replicate shared subscription and persistent session routes 2024-06-17 12:23:51 +03:00
Serge Tupchii f036b641eb feat(clusterlink): integrate node local routes replication and message forwarding 2024-06-17 12:23:51 +03:00
Serge Tupchii 7df91d852c feat(clusterlink): integrate node local syncer/actor implementation
- support multiple cluster links in extrouter
 - apply extrouter ops on 'message.publish' hook
 - fix several minor bugs
2024-06-17 12:23:51 +03:00
Andrew Mayorov cbd01ae818 feat(clusterlink): add node-local route sync actor implementation 2024-06-17 12:23:51 +03:00
Andrew Mayorov 7b95273218 feat(routesync): make syncer a bit more generic and reusable 2024-06-17 12:23:51 +03:00
Andrew Mayorov 5bd9ee5c7f feat(utils): add `emqx_utils_ets:keyfoldl/3` function
Designed to be used with `bag` / `duplicate_bag` tables.
2024-06-17 12:23:51 +03:00
Andrew Mayorov 4097585f5d fix(clusterlink): ensure extrouter works on replicants
This is sort of a quick fix to make things safe, but it will likely be
a subject to the same drawbacks as the regular router in high-latency
deployments: reduced throughput.
2024-06-17 12:23:51 +03:00
Andrew Mayorov a53524c826 fix(cth-cluster): fix occasional case clauses during cluster bootup 2024-06-17 12:23:51 +03:00
Andrew Mayorov 2dd99c5a08 feat(clusterlink): add facility to reconstruct remote routing table 2024-06-17 12:23:51 +03:00
Serge Tupchii f08342c704 feat: cluster link prototype WIP 2024-06-17 12:23:48 +03:00
lafirest 98a54994c0
Merge pull request #13196 from lafirest/fix/rules_len
fix(authz_mnesia): add a soft limit in the API for the length of ACL rules
2024-06-17 15:47:59 +08:00
zhongwencool 17809a5803 chore: fix dialyzer warning 2024-06-17 14:07:30 +08:00
firest 1716852057 fix(authz_mnesia): improve field names and changes 2024-06-17 10:09:25 +08:00
zhongwencool a49e5877ec chore(debug): use redbug for default trace tools 2024-06-17 09:28:23 +08:00
zmstone 8daaf7b727
Merge pull request #13233 from zmstone/0611-expose-regex_extract-func
fix(emqx_rule_funcs): expose regex_extract function to rule engine
2024-06-16 16:16:48 +02:00
zmstone 2b7f3a597f
Merge pull request #13245 from qzhuyan/merge/william/sync-rel57-to-master
sync release57 to master
2024-06-15 09:13:55 +02:00
zhongwencool d433fc7c30
Merge pull request #13236 from zhongwencool/bug-schema-validation
fix: some minor bug fixes
2024-06-15 06:40:28 +08:00
Thales Macedo Garitezi 198ccd320b fix: add `emqx_s3` to `reboot_lists.eterm`
Otherwise, it's unconditionally started as a permanent application instead of being
controlled by `emqx_machine`.

```sh
ͳ rg start_boot _build/emqx-enterprise/rel/emqx/releases/5.7.0-g4e6be8a6/start.script | rg emqx_s3
     {apply,{application,start_boot,[emqx_s3,permanent]}},
```
2024-06-14 15:06:18 -03:00
Thales Macedo Garitezi 6e702cd16c fix(postgres): fix usage by authn/authz modules when prepared statements are disabled
Fixes https://emqx.atlassian.net/browse/EMQX-12550
2024-06-14 12:19:46 -03:00
zmstone 626aae6edf chore: fix bad conflict resolution 2024-06-14 16:57:53 +02:00
zhongwencool d95f17fe77 chore: revert ULOG/ELOG 2024-06-14 22:11:27 +08:00
Thales Macedo Garitezi fd5e844cc1 fix(postgres): correctly handle batch queries with disabled prepared statements
Fixes https://emqx.atlassian.net/browse/EMQX-12549
2024-06-14 11:10:21 -03:00
zhongwencool f713f13b2c test: generate dispatch.eterm in dashboard test 2024-06-14 21:52:58 +08:00
Thales Macedo Garitezi 9b3c806ba7 feat(message transformation): implement dryrun endpoint
Follow up to https://github.com/emqx/emqx/pull/13199
2024-06-14 10:19:36 -03:00
zhongwencool a6e3a09118 feat: make the dashboard restart quicker 2024-06-14 18:22:34 +08:00
JimMoen 0f0e7d18db
Merge pull request #13261 from JimMoen/fix-typo
chore: fix typo
2024-06-14 16:55:35 +08:00
JimMoen 6823c79ae0
chore: fix typo 2024-06-14 16:04:11 +08:00
William Yang b86d631744 test: fix tc t_handle_outing_non_utf8_topic 2024-06-14 08:17:19 +02:00
William Yang d39b8caff9 Merge branch 'release-57' into merge/william/sync-rel57-to-master 2024-06-14 08:17:09 +02:00
lafirest afd4b46f72
Merge pull request #13225 from lafirest/fix/mysql_redact
fix(auth): redact sensitive data for the authn/authz APIs
2024-06-14 09:19:21 +08:00
Thales Macedo Garitezi 01635722e9
docs: improve descriptions
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2024-06-13 18:34:44 -03:00
Thales Macedo Garitezi 52e6c88941 feat: enforce singleton discovery strategy when using DS `builtin_local` backend
Fixes https://emqx.atlassian.net/browse/EMQX-12493
2024-06-13 17:04:23 -03:00
Thales Macedo Garitezi cba5c7bb45 refactor: rm dead code 2024-06-13 16:57:34 -03:00
Thales Macedo Garitezi 4d876f2af2 chore: bump ekka -> 0.19.4
https://github.com/emqx/ekka/pull/236
2024-06-13 16:35:58 -03:00
Thales Macedo Garitezi 3adf64e637
Merge pull request #13240 from thalesmg/ds-cluster-purge-m-20240612
feat: support purging durable sessions during cluster purge
2024-06-13 09:30:18 -03:00
Thales Macedo Garitezi 6ccf1dcbf9
Merge pull request #13238 from thalesmg/fix-authz-http-content-type-handling-r57-20240612
fix(http authz): handle unknown content types in responses
2024-06-13 09:10:43 -03:00
Thales Macedo Garitezi 62cf277cd7
Merge pull request #13235 from thalesmg/fix-template-types-r57-20240611
chore: change types of mysql and mongodb fields to `template()`
2024-06-13 09:08:40 -03:00
Thales Macedo Garitezi c81a3ebc0a
Merge pull request #13234 from thalesmg/fix-clients-api-client-attrs-union-r57-20240611
fix(client mgmt api): allow projecting `client_attrs` from client fields
2024-06-13 09:08:29 -03:00
William Yang 1664ea4ad4 Revert: TLS partial-chain and keyUsage #12955 #12977
This reverts commit 28b17a2562.
This reverts commit 01467246fc.
This reverts commit c3f8ba5762.
This reverts commit 1a4a4bb3a5.
This reverts commit fb30207ef3.
This reverts commit 337c230e79.
This reverts commit 3a674f44f1.
This reverts commit 70ffd77f99.
This reverts commit 03b0935564.
This reverts commit 650cf4b27e.
This reverts commit 43ad665dcf.
This reverts commit a29a43e5fc.
This reverts commit 4e9c1ec0c9.
This reverts commit 8eb463c58d.
This reverts commit 90430fa66d.
This reverts commit eb1ab9adfe.
This reverts commit 8bc3a86f63.
This reverts commit fa4357ce89.
This reverts commit 0b95a08d32.
2024-06-13 13:46:45 +02:00
William Yang 723144199e
Merge pull request #13211 from qzhuyan/fix/william/rel57/ee-tls
Fix/william/rel57/ee tls
2024-06-13 13:19:44 +02:00
Andrew Mayorov 0f53b33417
chore(s3): make schema fields with defaults optional 2024-06-13 11:57:36 +02:00
zhongwencool 0eee2edd28 test: fix api_config SUITE failed 2024-06-13 14:55:24 +08:00
zhongwencool 2b1fa1711b fix: observer command crash when can't find object code 2024-06-13 14:55:23 +08:00
zhongwencool 623845d07d fix: don't allow set active_n to negative int 2024-06-13 14:55:23 +08:00
zhongwencool 825bfe4717 fix: load bad configs return unknown msg 2024-06-13 14:55:23 +08:00
zhongwencool 5d3f464ec3 fix: ws/wss's max_frame_size should > 0 2024-06-13 14:55:23 +08:00
zhongwencool 800a79adde test: test failed with_mfa 2024-06-13 09:50:52 +08:00
Thales Macedo Garitezi 2c264d9a4b fix(http authz): handle unknown content types in responses
Fixes https://emqx.atlassian.net/browse/EMQX-12530
2024-06-12 14:31:12 -03:00
zhongwencool c72b455a83 feat: add hidden option to record MFA and line in the log is very useful for debug. 2024-06-12 22:51:45 +08:00
Thales Macedo Garitezi d0e6f22a79 feat: support purging durable sessions during cluster purge
Fixes https://emqx.atlassian.net/browse/EMQX-12405
2024-06-12 11:19:53 -03:00
JimMoen daeb10151b
chore: fix typos 2024-06-12 19:06:34 +08:00
JimMoen cde4cb1358
fix: metrics macro with description 2024-06-12 19:06:33 +08:00
JimMoen 756797b258
refactor: gen metric schema with desc from macros 2024-06-12 19:04:39 +08:00
JimMoen 263d2dbae2
fix: inc metric `'client.auth.anonymous'` when client anonymous 2024-06-12 19:04:39 +08:00
JimMoen 9a78a6f640
refactor: mv metrics macros in hrl file 2024-06-12 19:04:29 +08:00
JimMoen cbac4019b3
fix: rm havn't used counter idx 2024-06-12 19:01:50 +08:00
Thales Macedo Garitezi 46ab3be1f4 chore: change types of mysql and mongodb fields to `template()`
Fixes https://emqx.atlassian.net/browse/EMQX-12395
2024-06-11 17:25:53 -03:00
Thales Macedo Garitezi 0874768c1d fix(client mgmt api): allow projecting `client_attrs` from client fields
Fixes https://emqx.atlassian.net/browse/EMQX-12511
2024-06-11 17:14:33 -03:00
zmstone 8a283b1cc0
Merge pull request #13180 from zmstone/0604-optimize-tcp-ssl-connection-data-send-performance
0604 optimize tcp ssl connection data send performance
2024-06-11 22:06:57 +02:00
zmstone 09f91159c9 fix(emqx_rule_funcs): expose regex_extract function to rule engine
previoulsy only available in variform expressions,
now made available for rule-engine
2024-06-11 21:31:13 +02:00
zmstone 51810c6919
Merge pull request #13229 from zmstone/0611-auth-with-raw-cert
feat(authn): support `${cert_pem}` placeholder
2024-06-11 21:16:01 +02:00
Thales Macedo Garitezi db572d35a7 Merge remote-tracking branch 'origin/release-57' into sync-r57-m-20240611 2024-06-11 15:34:54 -03:00
zmstone 751f7a24e9 feat(authn): support ${cert_pem} placeholder 2024-06-11 19:54:24 +02:00
Thales Macedo Garitezi 337009c3a0 fix: declare `emqx_schema_registry` as a dependency of `emqx_schema_validation` 2024-06-11 14:03:27 -03:00
Thales Macedo Garitezi 0f9c3b4cea feat(schema validation): check references schema names and types before changing config
Fixes https://emqx.atlassian.net/browse/EMQX-12368
2024-06-11 14:03:26 -03:00
Thales Macedo Garitezi 03b226248a feat(schema registry): add check for inner types
Currently, only `protobuf` has any.
2024-06-11 13:50:11 -03:00
Thales Macedo Garitezi c70e8252fe fix: declare `emqx_schema_registry` as a dependency of `emqx_message_transformation` 2024-06-11 13:35:02 -03:00
Thales Macedo Garitezi b2d716909f
Merge pull request #13200 from thalesmg/update-erlazure-no-gen-server-m-20240606
chore: refactor azure blob connector to use new `erlazure` without `gen_server`
2024-06-11 13:30:14 -03:00
Thales Macedo Garitezi 686bcc8a48
Merge pull request #13207 from thalesmg/republish-metrics-m-20240607
fix(rule actions): check `republish` action result for metrics
2024-06-11 13:29:58 -03:00
Thales Macedo Garitezi 2c6aa80b3c
Merge pull request #13221 from thalesmg/refactor-sv-topic-index-r57-20240610
refactor(schema validation): module organization and position indexing
2024-06-11 13:29:48 -03:00
Thales Macedo Garitezi 83ff07ade0
Merge pull request #13199 from thalesmg/message-transformation-r57-20240604
feat: implement message transformation
2024-06-11 13:29:25 -03:00
Thales Macedo Garitezi ab5329be02
Merge pull request #13220 from thalesmg/refactor-channel-hc-opt-r57-20240610
fix: remove another redundant health check
2024-06-11 12:22:43 -03:00
Andrew Mayorov 2f0eb3560b
Merge pull request #13227 from keynslug/fix/EMQX-12533/bad-tpl
fix(bridge-s3): validate aggreg key template before adding a channel
2024-06-11 15:39:12 +03:00
Andrew Mayorov d4fb812352
fix(aggreg): anticipate and handle delivery startup errors 2024-06-11 13:21:41 +02:00
Andrew Mayorov 29fc30ea69
fix(bridge-s3): validate aggreg key template before adding a channel 2024-06-11 13:21:41 +02:00
Andrew Mayorov fb0da9848c
feat(tpl): add separate `placeholders/1` function
The purpose is to have a clearer view of placeholders used in a
template, without going the usual `render(Template, #{})` route that is
actually subtly misleading: it won't mention that `${}` / `${.}`
placeholder has been used.

Also unify handling of `${}` / `${.}` in a couple of places.
2024-06-11 11:42:43 +02:00
Kjell Winblad 9d22089104
Merge pull request #13173 from kjellwinblad/kjell/pgsql_timestamp_encode_etc/EEC-1036
fix(pgsql connector): handle prepared statement already exists
2024-06-11 10:53:30 +02:00
William Yang 44258204bd chore: move tls_certs test helper to apps/emqx 2024-06-11 10:06:18 +02:00
firest e64f60b73f fix(auth): redact sensitive data for the authn/authz APIs 2024-06-11 16:05:11 +08:00
zhongwencool b657dc537c
Merge pull request #13156 from savonarola/0529-fix-dashboard-monitor
fix(dashboard_monitor): correctly merge old and new sample data
2024-06-11 10:33:15 +08:00
Thales Macedo Garitezi 05ebb17cd6 refactor: index positions rather than names
Addresses https://github.com/emqx/emqx/pull/13199#discussion_r1633096025
2024-06-10 15:54:37 -03:00
Thales Macedo Garitezi 5eff4a7544 refactor(schema validation): improve api and module organization
To conform to comment from another PR:
https://github.com/emqx/emqx/pull/13199#discussion_r1632658742

... since schema validation is the "inspiration" for message transformation.
2024-06-10 15:46:28 -03:00
Thales Macedo Garitezi d93c8540c8 feat: pass along peername down to transformation context 2024-06-10 15:35:35 -03:00
Thales Macedo Garitezi 6e0ef893f4 feat: pass along client attributes down to message transformation context 2024-06-10 15:27:04 -03:00
Thales Macedo Garitezi 3c501e4f2a fix: remove another redundant health check
Fixes https://emqx.atlassian.net/browse/EMQX-12521
2024-06-10 15:13:34 -03:00
Thales Macedo Garitezi 1a20a40218
Merge pull request #13209 from thalesmg/rm-redundant-hc-r57-20240607
chore: remove redundant health check call
2024-06-10 14:58:48 -03:00
Thales Macedo Garitezi e586178479 chore: refactor azure blob connector to use new `erlazure` without `gen_server`
https://github.com/dkataskin/erlazure/pull/43 removes unnecessary usage of `gen_server`
from the driver.
2024-06-10 13:42:28 -03:00
Thales Macedo Garitezi 5629fe60c1 refactor: index positions rather than names 2024-06-10 13:33:22 -03:00
Thales Macedo Garitezi 8c7a7cf9db refactor: apply review remarks 2024-06-10 11:17:13 -03:00
Thales Macedo Garitezi e54cf2f218 refactor: use option to skip rendering as string 2024-06-10 10:49:39 -03:00
Thales Macedo Garitezi d34558954d refactor: improve api and module organization 2024-06-10 10:40:53 -03:00
Thales Macedo Garitezi 4aa1c3842f fix: declare `emqx` as a dependency in `.app.src` 2024-06-10 09:53:29 -03:00
Thales Macedo Garitezi aa416cc0ad docs: fix stale comments and dates 2024-06-10 09:53:02 -03:00
zmstone 7a09b0470d chore: cosmetic changes 2024-06-10 14:22:25 +02:00
Kjell Winblad a885f0b41a test(emqx_bridge_pgsql_SUITE): call test janitor 2024-06-10 13:45:27 +02:00
William Yang a1aa9a4375 fix(ee): emqx no longer deps on emqx_auth_ext 2024-06-10 10:32:01 +02:00
zmstone 4347f3de3e fix(bridge/mqtt): respect client ID prefix 2024-06-10 10:27:45 +02:00
William Yang 76cfc309a9 feat(tls): partial_chain not required 2024-06-10 10:25:44 +02:00
William Yang 7c37bf9965 feat(tls): ee only: TLS partial_chain and Keyusage 2024-06-10 10:24:48 +02:00
William Yang e60380d205 chore: fix nit 2024-06-10 09:55:12 +02:00
William Yang e9b813d8ef chore: fix test 2024-06-10 09:54:54 +02:00
William Yang 1739bc0c24 feat(partial_chain): gateway support 2024-06-10 09:54:44 +02:00
zmstone 45dd7816d7 Merge remote-tracking branch 'origin/release-57' 2024-06-09 10:11:15 +02:00
William Yang abbf2ef62f chore(TLS-chain-test): update for OTP 26 2024-06-08 00:16:06 +02:00
William Yang 221b748b0f test(partial_chain): update tcs for OTP-25 2024-06-08 00:16:01 +02:00
William Yang 463d1a1875 fix(test): tls_verify_partial_chain 2024-06-08 00:15:54 +02:00
William Yang 788cdbc6dd fix(listener): remove partial_chain in wss opts 2024-06-08 00:15:51 +02:00
William Yang 3fc99315e0 chore: happy elvis 2024-06-08 00:15:45 +02:00
William Yang c5dccdf526 feat(tls): update schema for TLS keyusage 2024-06-08 00:15:43 +02:00
William Yang aa25e3badd fix(tls): undefined keyusage 2024-06-08 00:15:39 +02:00
William Yang 9e5cfea8c0 test(tls): verify peer keyusage 2024-06-08 00:15:36 +02:00
William Yang c5657029ab feat(config): partial_chain 2024-06-08 00:15:34 +02:00
William Yang f7ff9496e6 test: port listener tls partial_chain 2024-06-08 00:15:31 +02:00
William Yang 1ce13242a8 feat(tls): port partial_chain, part 1 2024-06-08 00:15:28 +02:00
Kjell Winblad c13631102e fix(PgSQL connector): typo in function name 2024-06-07 19:40:18 +02:00
Andrew Mayorov b2f027c974
Merge pull request #13197 from keynslug/fix/EMQX-12522/conv-ssl
fix(bridge-s3): pass SSL options through `convert_certs/2`
2024-06-07 19:45:53 +03:00
Thales Macedo Garitezi 39615e1cb6 fix(rule actions): check `republish` action result for metrics
Fixes https://emqx.atlassian.net/browse/EMQX-12328
2024-06-07 11:17:43 -03:00
Thales Macedo Garitezi 28a3c77e88 chore: remove redundant health check call
Since `emqx_resource_manager:channel_health_check` already does a base resource health
check, the call was repeated.
2024-06-07 11:07:12 -03:00
Kjell Winblad e63dcc84b0 fix: unused variable and better error message 2024-06-07 12:20:42 +02:00
Kjell Winblad a3f3f96781 test(emqx_bridge_v2_pgsql_SUITE): fix broken test case
We have to delete the connector and action created by the previous step
of the test case so we don't get a name conflict.
2024-06-07 12:03:34 +02:00
JimMoen 78a196e215
Merge pull request #13189 from JimMoen/fix-apply-sqlserver-template
fix(utils_sql): improve insert sql regular expression
2024-06-07 13:50:19 +08:00
lafirest eb215726d5
Merge pull request #13195 from lafirest/fix/typos
fix: fix typos
2024-06-07 09:04:16 +08:00
Thales Macedo Garitezi ef36350bf6 feat: implement message transformation
Fixes https://emqx.atlassian.net/browse/EMQX-12517
2024-06-06 17:51:08 -03:00
Andrew Mayorov 8a85d7cb5a
fix(bridge-s3): pass SSL options through `convert_certs/2` 2024-06-06 19:37:38 +02:00
Andrew Mayorov c17c788438
feat(connector): allow delegate `pre_config_update` to impls 2024-06-06 19:37:34 +02:00
zmstone ebf17c8143 Merge remote-tracking branch 'origin/release-57' into 0606-merge-release-57-to-master 2024-06-06 17:43:36 +02:00
firest 3ae26c8a54 chore: update changes 2024-06-06 18:13:24 +08:00
firest 5532c7b0a6 fix(authz_mnesia): add a soft limit in the API for the length of ACL rules 2024-06-06 16:22:53 +08:00
zmstone 2525bd0c16 perf: eliminate the loop-back fake async inet_reply message
Now that EMQX is on OTP 26 by default, the fake inet_reply
message when the send call returns ok only adds overhead.

For TCP (gen_tcp module), OTP 26 no longer supports direct port_command
so we had to swith to gen_tcp:send, then fake an async reply:
see https://github.com/emqx/esockd/pull/181

For TLS (ssl module), it has never supported async send anyway.

This commit switches to call Transport:send/2 directly,
this should result in one less loop-back message for each message.

For OTP 25 on which EMQX can still be compiled, one will have to
suffer the performance penalty for gen_tcp:send function to
do a non-optimized 'receive' for the send result.
2024-06-06 09:35:51 +02:00
zmstone 5c3f5d8085 perf: do not call inet getstat before each and every send
In a stress test environment, when alarm is enabled, there were
a lot of long_schedule warnings with the stacktrace pointing to
congestion alarm based inet:getstat function.
When alarm is disabled, the one single client is able to
hold 40,000 QoS 1 messages per second (inflight=32) throughput.
When alarm is enabled, its mqueue overflows.

This commit removes the call before each and every data send,
so to rely on emit_stats timer to trigger the congestion check.
2024-06-06 09:35:51 +02:00
firest d4118c6e8e fix: fix typos 2024-06-06 11:19:40 +08:00
JimMoen dff31b293c
perf(utils_sql): pre-build regexp 2024-06-06 11:01:38 +08:00
zmstone c41be7ee22 chore: fix dialyzer warning 2024-06-05 23:37:21 +02:00
Ilia Averianov dc2e6d1695
Merge pull request #13163 from savonarola/0523-ds-shared-subs-dispatch
Inject shared subcription handling into durable session
2024-06-05 23:59:50 +03:00
Serge Tupchii 159da912c2 feat(emqx_dashboard_swagger): apply field validator in request body check 2024-06-05 19:35:04 +03:00
JimMoen 144264e0d8
test: add utils_sql test cases 2024-06-06 00:30:18 +08:00
JimMoen 627b8c45d2
fix(utils_sql): improve insert sql regular expression 2024-06-06 00:30:17 +08:00
Ilya Averyanov b075b7120c feat(sessds): implement dispatchig between CE/EE shared sub agents 2024-06-05 19:17:58 +03:00
Ilya Averyanov 60556abc4f fix(dashboard_monitor): correctly merge old and new sample data 2024-06-05 18:49:52 +03:00
Kjell Winblad 2956e849eb fix(pgsql connector): better msg when failing to remove statement 2024-06-05 16:58:14 +02:00
Kjell Winblad 336089f8a7 fix: bug found by dialyzer and make test case cleaner 2024-06-05 15:53:02 +02:00
Thales Macedo Garitezi ae0379f974
Merge pull request #13181 from thalesmg/force-stop-connector-r57-20240604
fix(resource manager): force kill process if stuck when stopping/removing
2024-06-05 09:08:30 -03:00
lafirest d79193fde4
Merge pull request #13177 from lafirest/sync-authn-com
feat: sync `client.check_authn_complete` to release-57
2024-06-05 19:51:10 +08:00
Thales Macedo Garitezi 5eab221f7c
Merge pull request #13175 from thalesmg/disable-prepared-statements-postgres-r57-20240603
feat(postgres): add `disable_prepared_statements` option
2024-06-04 12:29:53 -03:00
Thales Macedo Garitezi f17aefe3d7 fix(action/source api): improve returned error message on timeout 2024-06-04 12:04:55 -03:00
zmstone cda6d5f636
Merge pull request #13172 from zmstone/0603-add-redis-arg-formatter-to-rule-engine-funcs
feat(redis): add a rule function to help formatting redis args
2024-06-04 16:56:14 +02:00
Thales Macedo Garitezi 3013189cd7 fix(resource manager): force kill process if stuck when stopping/removing
Fixes https://emqx.atlassian.net/browse/EMQX-12357
2024-06-04 11:38:24 -03:00
Ilya Averyanov 9cdfbb0845 feat(sessds): make ds session be aware of shared subscriptions 2024-06-04 17:15:30 +03:00
Thales Macedo Garitezi c07bc68e6f feat(postgres): add `disable_prepared_statements` option
Fixes https://emqx.atlassian.net/browse/EMQX-12496

Some Postgres connections, such ones made to [PGBouncer](https://www.pgbouncer.org/) or
[Supabase in Transaction Mode](https://supabase.com/), do not support some session
features like prepared statements.
2024-06-04 10:11:25 -03:00
firest 87384cae32 fix(events): call `client.check_authn_complete` even if authentication fails 2024-06-04 17:55:39 +08:00
firest 92d26ff27b chore: update change 2024-06-04 17:55:30 +08:00
firest e4e53844d5 feat(events): add new hook && event `client.check_authn_complete` 2024-06-04 17:55:21 +08:00
Ilia Averianov dc95218c68
Merge pull request #13144 from savonarola/0528-increase-bridge-buffer-drop-log-level
chore(bridges): increase log level for bridge buffer overflow reports
2024-06-04 12:37:50 +03:00
Kjell Winblad 572ca6433e fix(pgsql connector): improvements due to suggestions from @thalesmg
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-06-04 10:11:45 +02:00
zhongwencool a8a67a2ac9
Merge pull request #13151 from zhongwencool/authz-trace-log
chore: make authz's logs easier to understand
2024-06-04 11:12:09 +08:00
zmstone aa7ce1f641 fix(bridge/redis): add test case for map_to_redis_hset_args 2024-06-03 23:05:43 +02:00
Kjell Winblad f2ccfff803 fix(pgsql connector): handle prepared statement already exists
In a user's log file it was found that that the pgsql driver can end up
in a situation where the prepared statement for a channel/action is not
properly removed before a channel with the same name as the prepared
statement is added to the connector. This commit handles this by
attempting to remove the old prepared statement if one already exists
when adding channel.

Related issue:
https://emqx.atlassian.net/browse/EEC-1036
2024-06-03 16:14:26 +02:00
zmstone e3ed7b59dd feat(redis): add a rule function to help formatting redis args
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.

This new function is dedicated for redis to avoid abuse for other
data integrations.
2024-06-03 14:09:15 +02:00
Thales Macedo Garitezi c5547543e0
Merge pull request #13069 from thalesmg/azure-blob-storage-action-m-20240508
feat: implement azure blob storage action
2024-06-03 08:56:37 -03:00
Ilia Averianov 193978af90
Merge pull request #13168 from savonarola/0530-fix-mgmt-test
chore(tests): improve test stability
2024-06-03 14:44:12 +03:00
zhongwencool fda365a87b chore: make authz's logs easier to understand 2024-06-03 18:01:57 +08:00
JimMoen 21624bc865
fix: get plugin app vsn from field `rel_apps` 2024-06-03 15:26:14 +08:00
zmstone b6ff67d712
Merge pull request #13093 from zmstone/0522-upgrade-to-brod-3.18
chore: upgrade to brod 3.18.0
2024-06-02 15:06:40 +02:00
JimMoen 8d22a5a3ae
Merge pull request #13167 from JimMoen/fix-rule-events-i18n-todo
fix: missing rule event i18n in zh_cn
2024-06-02 18:23:20 +08:00
zmstone 9637b14543 fix: upgrade to brod 3.18.0 to fix unnecessary rebalance 2024-06-01 10:57:24 +02:00
Ilya Averyanov 57741a41e9 chore(tests): improve test stability 2024-05-31 22:27:31 +03:00
zmstone fb07f40bea test: fix default value of retry_interval in test case 2024-05-31 17:53:12 +02:00
zmstone 20c92407c7 feat: change message retry_interval default value to infinity
Previous Setting: The default value for `retry_interval` was 30 seconds.

New Default: The default `retry_interval` has been changed to 'infinity'.
With this update, EMQX will not automatically retry message deliveries
by default.

Compliance and Compatibility: Aligning with MQTT specification
standards, in-session message delivery retries are not typically
compliant. We recognize that some users depend on this feature, so the
option to configure retries remains available for backward
compatibility.
2024-05-31 17:53:12 +02:00
zmstone 37ce8d93a1
Merge pull request #13164 from zmstone/0530-http-acl-compatibility-with-v4
0530 http acl compatibility with v4
2024-05-31 16:28:30 +02:00
Thales Macedo Garitezi 347971e24a fix: check `max_block_size` for direct transfers 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi 4859cebd9f refactor: move `min_block_size` one level up 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi 9a1d3ea04f chore: bump app vsns 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi 616f14ae53 fix: avoid uploading blocks too frequently and splitting large buffers
Fixes https://github.com/emqx/emqx/pull/13069#discussion_r1613706747
2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi af99829a21 refactor: move common templating logic to `emqx_connector_aggreg_buffer_ctx` 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi c916c83c7c feat: implement azure blob storage action
Fixes https://emqx.atlassian.net/browse/EMQX-12280
2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi 60d24c6ad5 refactor: simplify resource_opts definition 2024-05-31 11:07:44 -03:00
Thales Macedo Garitezi 729441d1ce refactor(connector aggregator): move shared schemas to app 2024-05-31 11:07:44 -03:00
Ilya Averyanov db713cb656 chore(bridges): increase log level for bridge buffer overflow reports
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2024-05-31 16:35:18 +03:00
Thales Macedo Garitezi c289c7d065
Merge pull request #13148 from thalesmg/EMQX-12251-fix-conn-api-timeout-case-clause-r571-20240528
fix(connector api): handle `timeout` when waiting for connector status
2024-05-31 10:07:35 -03:00
zmstone 6fe8a09e97 fix(authz/http): rename PH_ACCESS to VAR_ACCESS 2024-05-31 14:49:54 +02:00
JimMoen 45e2c81ebc
fix: missing rule event i18n in zh_cn 2024-05-31 17:51:47 +08:00
Ivan Dyachkov 29ad07ce29 Merge remote-tracking branch 'upstream/release-57' into 0531-sync-release-57 2024-05-31 07:32:26 +02:00
zmstone ca32cf8d29 Merge branch 'emqx-12487-fix-authz-http-content-type' into 0530-http-acl-compatibility-with-v4 2024-05-30 22:30:20 +02:00
zmstone a00d9f17b1 chore: upgrade kafka producer lib wolff to 1.10.4
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
2024-05-30 15:40:10 +02:00
zmstone 460081a22e
Merge pull request #13131 from zmstone/0527-handle-unicode-in-template-strings
0527 handle unicode in template strings
2024-05-30 15:37:43 +02:00
Ilya Averyanov 97f9c81e19 feat(auth): add legacy ${access} placeholder 2024-05-30 15:47:48 +03:00
Ivan Dyachkov 0c4da98b52 chore: update deps 2024-05-30 11:53:00 +02:00
Ivan Dyachkov 1129c18330 fix(authz_http): fix content-type header in http request 2024-05-30 11:36:13 +02:00
zhongwencool 8ab7e75e64
Merge pull request #13139 from zhongwencool/influxdb-client-failed-msg-port
chore: improved InfluxDB connection error messaging
2024-05-30 10:57:21 +08:00
zmstone 532812f6f6 test: assert results 2024-05-29 22:10:34 +02:00
zmstone 9fe29bf5f7 chore: bump kafka bridge app vsn 2024-05-29 21:57:14 +02:00
zmstone 78a6100346 chore: fix app vsn bumps
only bug fixes so far
2024-05-29 21:56:22 +02:00
zmstone 1db9d54d17 fix: handle unicode in data integration template strings
The bug was introduced in 5.7.0 (#12893)
2024-05-29 21:53:07 +02:00
zmstone c54d25de98
Merge pull request #13070 from zmstone/0518-improve-kafka-connection-error-logs
0518 improve kafka connection error logs
2024-05-29 16:24:01 +02:00
Kjell Winblad 5017ef82a3
Merge pull request #13130 from kjellwinblad/kjell/fix_redis_trace_format/EMQX-12428
fix(redis trace): add separators in redis batch action trace
2024-05-29 15:59:32 +02:00
Kjell Winblad 6810591911
Merge pull request #13147 from kjellwinblad/kjell/fix_function_clause_when_decoding_protobuf/EMQX-12453
fix: make protobuf schema decode error reported to user less cryptic
2024-05-29 15:56:27 +02:00
Kjell Winblad 3769bbb7f4
Merge pull request #13140 from kjellwinblad/kjell/republish_trace_crash/EMQX-12474
fix: trace formatter should not crash when given a non-binary payload
2024-05-29 15:54:02 +02:00
Kjell Winblad 9b089a4e5a fix: use explain instead of msg as error explanation and add test 2024-05-29 10:20:15 +02:00
Kjell Winblad 479889f162
Merge pull request #13136 from kjellwinblad/kjell/fix_oracle_trace_format/EMQX-12433
fix(oracle action trace): parameters should not be rendered as IO Data
2024-05-29 09:12:38 +02:00
zhongwencool 7d9333a9d7 chore: bump influxdb app version 2024-05-29 11:13:17 +08:00
zhongwencool 00d74826ec chore: improved InfluxDB connection error messaging 2024-05-29 11:13:17 +08:00
Thales Macedo Garitezi 802361dbd0 fix(connector api): handle `timeout` when waiting for connector status
Fixes https://emqx.atlassian.net/browse/EMQX-12251
2024-05-28 14:19:32 -03:00
Kjell Winblad 82800faadf fix: make protobuf schema decode error reported to user less cryptic
Before this commit the user would see a cryptic warning log including
function_clause when decoding a message failed. This commit improves
this by removing function_caluse as well as some other cryptic words
from the message and adding a description describing what went wrong.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12453
2024-05-28 18:07:14 +02:00
Kjell Winblad 3151a3123e chore: bump version of emqx_redis_bridge app 2024-05-28 17:23:35 +02:00
Kjell Winblad 3f1fe413ee refactor: simplify redis trace formatting 2024-05-28 17:22:58 +02:00
Kjell Winblad cbdde7165e fix(redis trace): add separators in redis batch action trace
Logger will transform data that looks like IO data into a string which
made redis batch traces look like the spaces had been removed
from the strings. To prevent this, we render the batched commands into
binary string separated by spaces and semicolon. The components of a
single command is separated by spaces and the commands in a batch are
separated by semicolons.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12428
2024-05-28 17:22:58 +02:00
Kjell Winblad 5e3de1a4e3 fix(emqx_logger_jsonfmt:best_effort_json_test): test cases 2024-05-28 17:15:32 +02:00
zmstone 062ab31ecf Merge remote-tracking branch 'origin/release-57' into 0527-port-back-diverged-modules 2024-05-28 14:35:25 +02:00
zmstone 40080f5e7d
Merge pull request #13141 from zmstone/0528-test-fix-dashboard-config-test-with-stable-base-config
test: fix test case to have deterministic base config
2024-05-28 14:32:55 +02:00
Andrew Mayorov 9ae4de10ed
Merge pull request #13127 from keynslug/fix/EMQX-12439/update-leaner-error
fix(bridge-v2): report descriptive error on invalid update request
2024-05-28 13:11:41 +03:00
zmstone e9cc88cb95 test: fix test case to have deterministic base config 2024-05-28 12:03:01 +02:00
Kjell Winblad 84bc6ba28c fix: trace formatter should not crash when given a non-binary payload
Fixes:
https://emqx.atlassian.net/browse/EMQX-12474
2024-05-28 11:13:51 +02:00
Kjell Winblad e55bda6fe6 chore: bump app versions
Bump app version as required to pass the check which is done by the
following script:

./scripts/apps-version-check.sh
2024-05-28 11:01:00 +02:00
Kjell Winblad f65168982b fix(oracle action trace): parameters should not be rendered as IO Data
This forces the parameters to the database statement to be rendered as
a JSON array in JSON traces instead of being rendered as a string when
the parameters are interpreted as IO data.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12433
2024-05-28 10:59:44 +02:00
zmstone 8f2ca71d8b test: fix test case to have deterministic base config 2024-05-28 10:57:20 +02:00
Andrew Mayorov c04aaad0a4
chore(bridge-v2): leave comment describing need for `nowarn_function` 2024-05-28 10:31:14 +02:00
ieQu1 8fbeca4321
chore: Version bumps 2024-05-28 00:14:01 +02:00
ieQu1 04305d004f
refactor(sessds): Extract metadata keys to a header 2024-05-27 23:36:20 +02:00
ieQu1 16d1425982
refactor(emqx): Move files related to durable session to a subdir 2024-05-27 23:31:15 +02:00
zmstone 1148091f0b chore: restore test code 2024-05-27 23:05:21 +02:00
zmstone 96ef37d0ea chore: port diverged modules back to oss 2024-05-27 22:40:08 +02:00
zmstone 2771a10d39 test: fix a flaky one 2024-05-27 20:59:50 +02:00
zmstone 238c207b09 chore: bump app versions 2024-05-27 20:26:52 +02:00
zmstone 41cbfcfaa1 Merge remote-tracking branch 'origin/release-57' into 0527-sync-5.7.0-to-master 2024-05-27 20:10:44 +02:00
Andrew Mayorov 663f4fd39f
chore: bump application versions 2024-05-27 17:49:16 +02:00
Andrew Mayorov 08d88ea814
feat(bridge-api): improve error messages for Update Source API 2024-05-27 17:49:16 +02:00
Andrew Mayorov ed7c29ec26
fix(api): mention spec override in "request metadata" type 2024-05-27 16:57:45 +02:00
Andrew Mayorov 5c2a68076f
fix(bridge-v2): report descriptive error on invalid update request
Before this commit, generic validation errors were reported as union
mismatches of _all_ of the bridge schemas. After this commit, specific
schema is chosen before validation.
2024-05-27 14:37:10 +02:00
Ivan Dyachkov f393fd5304 chore: release 5.7.0 2024-05-27 14:32:23 +02:00
zmstone b2f5e50f16
Merge pull request #13120 from ieQu1/dev/ds-api-site-status
fix(dsapi): Check status of the node via Mria
2024-05-25 05:42:05 +02:00
zmstone 0763e29786 chore: prepare for 5.7.0-rc.2 release 2024-05-24 21:10:37 +02:00
ieQu1 73df01e934
fix(dsapi): Check status of the node via Mria 2024-05-24 20:59:07 +02:00
ieQu1 7f356aa3a8
Merge pull request #13115 from ieQu1/dev/fix-drop-generations2
Mitigate transient errors when dropping generations
2024-05-24 20:13:39 +02:00
ieQu1 830b62d899
fix(dsrepl): Retry sending ra commands to the leader 2024-05-24 19:09:34 +02:00
ieQu1 1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable 2024-05-24 19:09:34 +02:00
ieQu1 25721bceba
fix(dsstor): Delete generation metadata before dropping it 2024-05-24 19:09:33 +02:00
ieQu1 3d9837a0b8
test(ds): Add emqx_ds_replication_SUITE:t_drop_generation 2024-05-24 19:09:33 +02:00
Ilya Averyanov 327080e70e perf(emqx_placeholder): do not try to decode maps as json
fix
2024-05-24 20:08:31 +03:00
Andrew Mayorov c30b881239
test(monitor): verify historical and current metrics are consistent 2024-05-24 14:56:10 +02:00
Andrew Mayorov fd9655bc35
fix(monitor): unify metric names for consistency and simplicity 2024-05-24 14:55:48 +02:00
Andrew Mayorov 7b137c0aaf
fix(monitor): reflect durable subscriptions in historical metrics
Essentially in the same manner as they are currently reflected in the
current metrics.
2024-05-24 14:08:35 +02:00
zmstone 6ec8ae5a5f
Merge pull request #13112 from HJianBo/change-the-default-batch-size-s3-uploader
Fix(s3): change the batch_size to 1 for direct mode
2024-05-24 10:56:53 +02:00
Andrew Mayorov 29d203ae33
Merge pull request #13108 from keynslug/fix/ct-flaky/mgmt
test(mgmt): avoid inter-suite contamination by switching to cth tooling
2024-05-24 11:14:16 +03:00
JianBo He adaf238131 fix(s3): fix the convert function if the mode is missing 2024-05-24 16:02:06 +08:00
JimMoen 5b60f6a5b2
fix: make dialyzer happy 2024-05-24 14:29:02 +08:00
JimMoen 43506495ae
fix: ensure plugin configured states after join cluster 2024-05-24 13:51:56 +08:00
JimMoen 874bcd9a8b
fix: match decode plugin config map failed case 2024-05-24 10:20:50 +08:00
zhongwencool 3aa1f86d9e chore: is_content_type_json to validate_content_type_json 2024-05-24 10:13:24 +08:00
zhongwencool 5c759941d5 feat: return 415 when UNSUPPORTED_MEDIA_TYPE 2024-05-24 10:13:23 +08:00
zmstone c3c6da7ec4 chore: prepare for release cut 5.7.0-rc.1 2024-05-23 21:39:15 +02:00
Zaiming (Stone) Shi 3a4b8e6c24
Merge pull request #13106 from zmstone/0523-add-customer-type
0523 add customer type (4=BYOC) to license schema
2024-05-23 21:30:02 +02:00
Zaiming (Stone) Shi 40288b99b2
Merge pull request #13101 from JimMoen/fix-plugin-config-map
fix: allow put plugin config without schema
2024-05-23 21:18:43 +02:00
ieQu1 c952e46f08
Merge remote-tracking branch 'origin/release-57' 2024-05-23 21:04:03 +02:00
Thales Macedo Garitezi 823a6bc521
Merge pull request #13107 from thalesmg/hide-clientv2-response-type-r57-20240523
fix(client mgmt api): hide unused return type
2024-05-23 16:03:51 -03:00
Andrew Mayorov 7878d106f6
test(mgmt): avoid inter-suite contamination by switching to cth tooling 2024-05-23 20:17:34 +02:00
Andrew Mayorov b4c6968f8c
Merge pull request #13104 from keynslug/fix/EMQX-12415/last-resort-poll
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
2024-05-23 21:17:03 +03:00
ieQu1 0dc507441a
Merge pull request #13105 from ieQu1/dev/sessds-renew-streams-interval
fix(sessds): Change the defaults to renew streams every 1 second
2024-05-23 18:23:20 +02:00
Thales Macedo Garitezi b1e3c941fb fix(client mgmt api): hide unused return type
Fixes https://emqx.atlassian.net/browse/EMQX-12427
2024-05-23 13:17:43 -03:00
Andrew Mayorov d47bf8076a
test(mgmt): better isolate testsuite running environments 2024-05-23 17:53:13 +02:00
zmstone c737045dff chore: add customer type 4 (BYOC) 2024-05-23 17:30:06 +02:00
ieQu1 5584b658c9
fix(sessds): Change the defaults to renew streams every 1 second 2024-05-23 17:17:29 +02:00
Zaiming (Stone) Shi 1d99ba2a8a
Merge pull request #13102 from JimMoen/fix-ds-prom-badkey
fix: add ds metrics into prometheus zipper init acc
2024-05-23 16:57:06 +02:00
Thales Macedo Garitezi 118c1d2806
Merge pull request #13103 from thalesmg/fix-another-ds-counter-nit-r57-20240523
fix(monitor api): fix cluster metric aggregation
2024-05-23 10:57:53 -03:00
William Yang bb3f858db2
Merge pull request #13086 from qzhuyan/dev/william/quic-lb
feat: quic-lb support
2024-05-23 15:31:14 +02:00
Andrew Mayorov ba6382adae
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
This is a hack to work around the unintended issues causing shard
allocator to become idle even when there are pending transitions.
2024-05-23 14:54:01 +02:00
Andrew Mayorov 839b143fc4
fix(sessds): consider message durability disabled by default 2024-05-23 14:48:39 +02:00
Thales Macedo Garitezi c0e3a81c61 fix(monitor api): fix cluster metric aggregation
Fixes https://emqx.atlassian.net/browse/EMQX-12423
Fixes https://emqx.atlassian.net/browse/EMQX-12267
2024-05-23 09:38:24 -03:00
Andrew Mayorov bf326acd7b
fix(dsrepl): handle stopping non-yet-running shard supervisor 2024-05-23 14:34:27 +02:00
Kjell Winblad 0e5b02c373
Merge pull request #13100 from kjellwinblad/kjell/s3_aggregate_only_has_result_trace/EMQX-12429
fix: add action rendered trace for s3 in aggregated mode
2024-05-23 14:29:18 +02:00