Commit Graph

13044 Commits

Author SHA1 Message Date
Thales Macedo Garitezi b91515b131 fix(schema registry mix): gpb is a runtime dep 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 6d94809950
Merge pull request #13415 from thalesmg/20240703-m-couchbase-action
feat: implement couchbase connector and action
2024-07-09 15:53:11 -03:00
Thales Macedo Garitezi 3c370a90aa
Merge pull request #13436 from thalesmg/20240708-r57-custom-headers-jwks
feat(jwks): allow specifying custom request headers
2024-07-09 15:52:44 -03:00
Thales Macedo Garitezi 9f8a1885a7 fix(resource manager): disentangle connector and channel health check frequencies
Fixes https://emqx.atlassian.net/browse/EMQX-12674
2024-07-09 14:53:39 -03:00
Thales Macedo Garitezi d25c4ba06f
Merge pull request #13421 from thalesmg/20240705-r57-docs-mt-api-examples
docs(message transformation): add api examples
2024-07-09 10:16:53 -03:00
ieQu1 3721be65ee
fix(ds): Improve comments 2024-07-09 13:15:15 +02:00
ieQu1 d7732a6aac
test(ds): Attempt to stabilize a flaky test 2024-07-09 13:15:15 +02:00
ieQu1 e70c1cfea3
test(ds): Improve stability of replication test suite 2024-07-09 13:15:15 +02:00
ieQu1 dc4ae82798
test(ds): Add message ID 2024-07-09 13:15:14 +02:00
ieQu1 d1b574a67e
perf(dslocal): Run heavy-duty operations in a temporary process 2024-07-09 13:15:14 +02:00
ieQu1 661f79544b
fix(ds): Optimize hot loop of skipstream storage layout 2024-07-09 13:15:14 +02:00
ieQu1 23dafbb03b
feat(ds): Add a benchmarking tool for storage efficiency analysis 2024-07-09 13:15:14 +02:00
ieQu1 afeb2ab8aa
feat(ds): Add metrics for skipstream layout 2024-07-09 13:15:14 +02:00
ieQu1 b68ebb9a73
test(dsrepl): Generalize tests to use different storage layouts 2024-07-09 13:15:14 +02:00
ieQu1 8c5e4a2376
test(ds): Generalize storage layout test suite for different layouts 2024-07-09 13:15:14 +02:00
ieQu1 086e7256f5
feat(ds): Add configuration schema for skipstream LTS layout 2024-07-09 13:15:14 +02:00
ieQu1 a4642d4d06
feat(ds): Add a new storage layout engine: "skipstream"
This layout is based on LTS as well, but it uses separate index
streams for constrained replay of streams with learned wildcards
2024-07-09 13:15:14 +02:00
ieQu1 de48077ac4
test(ds): Add new helper functions
- Improve message comparison
- Add set operations
2024-07-09 13:15:14 +02:00
ieQu1 210556e545
feat(ds): Generalize value serialization
- Add a new utility module for message serialization
- Add experimental serializer based on ASN.1
2024-07-09 13:15:14 +02:00
ieQu1 843973ef32
fix(ds): bitfield_lts: static_key_size -> static_key_bits 2024-07-09 13:15:14 +02:00
ieQu1 f84fb34692
feat(ds_lts): New APIs: info, reverse lookups and topic compression 2024-07-09 13:15:14 +02:00
ieQu1 eb80402ccb
fix(ds): Improve typespecs and descriptions in storage_layer 2024-07-09 13:15:14 +02:00
ieQu1 71dad0242e
docs(ds): Move Raft-related parts to emqx_ds_builtin_raft README 2024-07-09 13:15:14 +02:00
ieQu1 afe1c5617d
refactor(ds): Rename macros for bitfield_lts metrics 2024-07-09 13:15:14 +02:00
ieQu1 0f2c19b656
refactor(ds): Move end_of_stream detection logic for delete_next 2024-07-09 13:15:14 +02:00
ieQu1 b565976794
fix(ds): Fix hashing of empty wildcard topic levels in bitfield_lts 2024-07-09 13:15:13 +02:00
Thales Macedo Garitezi f00bb383d4 fix(ds builtin local): use `-include_lib` instead of relative path 2024-07-08 16:57:55 -03:00
Thales Macedo Garitezi 811184ddad feat(jwks): allow specifying custom request headers
Fixes https://emqx.atlassian.net/browse/EMQX-12655
2024-07-08 15:40:52 -03:00
Thales Macedo Garitezi d34fc7a03a
Merge pull request #13420 from thalesmg/20240705-r57-fix-mt-empty-topics
fix(schema validation & message transformation): forbid empty topic filter list
2024-07-08 10:15:08 -03:00
Thales Macedo Garitezi 166f5e5f12
Merge pull request #13426 from thalesmg/20240705-r57-test-flaky-plugin-start-enabled
test(plugins): fix flaky test
2024-07-08 09:19:53 -03:00
zhongwencool 820789a09f fix: redact status when mark_fix_log begin 2024-07-08 17:32:45 +08:00
zhongwencool 457ea93570 test: add cluster_sync cli test 2024-07-08 17:04:41 +08:00
zhongwencool f490a0cba2 feat: don't reset tnx_id when cluster_fix 2024-07-08 17:04:41 +08:00
zhongwencool 298211d101 chore: apply suggestions from code review
Co-authored-by: zmstone <zmstone@gmail.com>
2024-07-08 17:04:41 +08:00
zhongwencool bdf3fc63a6 chore: add config leader to suggestion 2024-07-08 17:04:41 +08:00
zhongwencool 22fc3c49cc chore: combine some common code into one function 2024-07-08 17:04:41 +08:00
zhongwencool 5b105fcdbb chore: move emqx_conf_proto_v3 to emqx_conf_proto_v4 2024-07-08 17:04:41 +08:00
zhongwencool 3ed4340145 test: fix cluster_rpc test failed 2024-07-08 17:04:41 +08:00
zhongwencool 2069910ad1 feat: add cluster fix command 2024-07-08 17:04:41 +08:00
JimMoen ae3b8fe146
test: create jwt authenticator with bad public key 2024-07-08 16:52:18 +08:00
JimMoen f76444fbf8
fix: create authn jwt with bad public key 2024-07-08 16:52:18 +08:00
Shawn 5fca0a16f9 feat: rename emqx_relup to emqx_post_upgrade 2024-07-08 10:33:09 +08:00
zhongwencool 29d7a511f1
Merge pull request #13419 from zhongwencool/port-pr
Port: some minor bug fixes from master
2024-07-06 14:42:51 +08:00
Thales Macedo Garitezi f1b4467fe1 test(plugins): fix flaky test
Hypothesis is that both peer nodes were using the same directory and stepping on each
other's toes.
2024-07-05 14:17:42 -03:00
Thales Macedo Garitezi 70fab51354 fix: handle `max_heap_size` = 0
Fixes https://github.com/emqx/emqx/issues/13417

Fixes https://emqx.atlassian.net/browse/EMQX-12659
2024-07-05 13:10:37 -03:00
Kjell Winblad c8258cebe8
Merge pull request #13414 from kjellwinblad/kjell/review_connector_error_logs_rabbitmq_etc/EMQX-12462
fix: make RabbitMQ error log messages easier to understand
2024-07-05 15:01:15 +02:00
Thales Macedo Garitezi 36ee7bed77 docs(message transformation): add api examples
Fixes https://emqx.atlassian.net/browse/EMQX-12645
2024-07-05 09:59:27 -03:00
Thales Macedo Garitezi e7351d949d fix(schema validation): forbid empty topic filter list 2024-07-05 09:51:43 -03:00
Thales Macedo Garitezi e99fee68c0 fix(message transformation): forbid empty topic filter list
Fixes https://emqx.atlassian.net/browse/EMQX-12646
2024-07-05 09:49:03 -03:00
zhongwencool 7d851872ec chore: update emqx_module's app version 2024-07-05 19:21:28 +08:00
zhongwencool d94fcb9cfd test: fix api_config SUITE failed 2024-07-05 17:34:39 +08:00
zhongwencool ba3097dc56 fix: observer command crash when can't find object code 2024-07-05 17:34:33 +08:00
zhongwencool f0a1d785ca fix: don't allow set active_n to negative int 2024-07-05 17:34:25 +08:00
zhongwencool 8aab919f74 fix: load bad configs return unknown msg 2024-07-05 17:34:17 +08:00
zhongwencool b4cffc581b fix: ws/wss's max_frame_size should > 0 2024-07-05 17:34:09 +08:00
zhongwencool 094259f444
Merge pull request #13408 from zhongwencool/password-crash
chore: improve auth error for invalid salt/password type
2024-07-05 11:44:21 +08:00
JimMoen 9d0b5a9bc6
Merge pull request #13412 from JimMoen/fix-cert-notafter-badmatch
fix: obtain cert expiry epoch failed due to formated `generalTime`
2024-07-05 10:13:22 +08:00
zhongwencool d3d3303dcb chore: improve auth error for invalid salt/password type 2024-07-05 10:12:24 +08:00
lafirest aa84ca5a88
Merge pull request #13386 from lafirest/feat/banned_boot_57
feat(banned): add a bootstrap file for banned
2024-07-05 08:47:40 +08:00
Thales Macedo Garitezi c4dd167cb9 feat: implement couchbase connector and action
Fixes https://emqx.atlassian.net/browse/EMQX-12545
2024-07-04 17:51:59 -03:00
Thales Macedo Garitezi b333babb4c
Merge pull request #13401 from thalesmg/20240703-r57-authz-ignore-api-metrics
fix(authz api): add new `ignore` metric to status response
2024-07-04 17:07:08 -03:00
JimMoen d84d31cbc5
test: cert expiry epoch with `generalTime` formatted 2024-07-05 01:47:13 +08:00
Kjell Winblad b994e0f1c0 fix: make RabbitMQ error log messages easier to understand
Fixes:
https://emqx.atlassian.net/browse/EMQX-12462
2024-07-04 16:43:53 +02:00
Thales Macedo Garitezi 420493deb4 chore: add missing mix files to new apps 2024-07-04 10:27:59 -03:00
Kjell Winblad d206d24975 fix: only set default for max_conn_rate and update test case
This revert the change in commit e291dcd for all listener "short
path fields" except the field max_conn_rate so they no longer have a
default value. It also updates a test case that assume that no listener
config is created by default but this is no longer the case when there
is a default value for the max_conn_rate field.
2024-07-04 14:32:10 +02:00
Thales Macedo Garitezi f758fd9279
Merge pull request #13405 from thalesmg/20240703-r57-test-flaky-table-removed-postgres
test(postgres bridge): attempt to stabilize flaky test
2024-07-04 09:25:15 -03:00
JimMoen 8c6cd69caa
fix: obtain cert expiry epoch failed due to formated `generalTime` 2024-07-04 18:23:56 +08:00
firest 913e0ce18b feat(banned): add a bootstrap file for banned 2024-07-04 09:12:12 +08:00
zmstone 947cddb2eb test: invalid map key is caught by hocon
now that hocon has a built-in map key validation,
some of the resource name validations are cought by hocon
2024-07-03 23:00:18 +02:00
zmstone eaaee725c2 fix: upgrade to hocon 0.43.1
included 3 changes since 0.42.2

- allow validation of map keys
- improve crash stacktrace report
- avoid dumping array environment variable values
2024-07-03 23:00:14 +02:00
zmstone 7ee5b90084
Merge pull request #13400 from zmstone/0605-ACL-rules-in-http-authentication-response
feat(auth): support HTTP authn return ACL rules
2024-07-03 21:51:07 +02:00
Thales Macedo Garitezi 32ace85e1c Merge branch 'release-57' into 20240703-m-sync-r57 2024-07-03 16:03:15 -03:00
Thales Macedo Garitezi ae24b7a37b
Merge pull request #13404 from thalesmg/20240703-r57-refactor-mix-common-deps
ci(mix): refactor to avoid conflicts
2024-07-03 15:57:05 -03:00
Thales Macedo Garitezi 72579f9014 test(postgres bridge): attempt to stabilize flaky test
```
%%% emqx_bridge_pgsql_SUITE ==> tcp.sync.with_batch.t_table_removed: FAILED
%%% emqx_bridge_pgsql_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,exit,
               {test_case_failed,
                   "unexpected result: {error,{recoverable_error,sync_required}}"},
               [{emqx_bridge_pgsql_SUITE,'-t_table_removed/1-fun-3-',3,
                    [{file,
                         "/emqx/apps/emqx_bridge_pgsql/test/emqx_bridge_pgsql_SUITE.erl"},
                     {line,822}]},
```

```
Error: -03T17:52:54.046809+00:00 [error] Generic server <0.352770.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,{state,ssl,{sslsocket,{gen_tcp,#Port<0.1671>,tls_connection,undefined},[<0.352774.0>,<0.352773.0>]},<<>>,{336,-2093820527},on_message,{codec,#{},[null,undefined],{oid_db,#{16 =>
...
2024-07-03T17:52:54.075446+00:00 [critical] Run stage failed: exit:{test_case_failed,"unexpected result: {error,\n                    {resource_error,\n                     #{reason => exception,\n                       msg =>\n                        #{error =>\n                           {exit,\n                            {{undef,\n                              [{undefined,handle_message,\n                                [90,<<\"I\">>,\n                                 {state,ssl,\n                                  {sslsocket,\n                                   {gen_tcp,#Port<0.1671>,tls_connection,\n
```
2024-07-03 15:05:00 -03:00
Thales Macedo Garitezi afb65817c4 ci(mix): refactor to avoid conflicts
This introduces a refactoring that will hopefully help avoid conflicts with ongoing work
to migrate our build / CI tooling to Mix.
2024-07-03 14:37:25 -03:00
Andrew Mayorov 187f5e5936
Merge pull request #13391 from keynslug/perf/EMQX-12611/avoid-stream-shuffle
perf(sessds): rotate through streams with iterators when fetching
2024-07-03 17:46:29 +02:00
Andrew Mayorov d7d5eb2c52
test(bridge-s3): correct assertions to get rid of flakiness 2024-07-03 16:40:34 +02:00
Thales Macedo Garitezi 532c7831b2
Merge pull request #13384 from thalesmg/20240701-r57-mt-new-fields
feat(message transformation): add more read-only fields to context
2024-07-03 11:27:27 -03:00
Thales Macedo Garitezi 35f1ddc0eb
Merge pull request #13393 from thalesmg/20240702-r57-test-plugin-start-enabled
fix(plugins): ensure plugin apps are restarted when restarting `emqx_plugins`
2024-07-03 11:27:14 -03:00
zmstone 9194756963 feat(auth): support HTTP authn return ACL rules 2024-07-03 15:37:11 +02:00
Andrew Mayorov 950f4d9483
fix(sessds): defend restartable stream iterator from infinite loop 2024-07-03 15:21:16 +02:00
Andrew Mayorov 947af1faaf
chore(sessds): rename `iterate` -> `iter_next` for consistency 2024-07-03 15:21:16 +02:00
Thales Macedo Garitezi c3579f338b fix(authz api): add new `ignore` metric to status response
Fixes https://emqx.atlassian.net/browse/EMQX-12411
2024-07-03 10:16:18 -03:00
Thales Macedo Garitezi e1420a27bb fix(plugins): ensure plugin apps are restarted when restarting `emqx_plugins`
Fixes https://emqx.atlassian.net/browse/EMQX-12628
Fixes https://github.com/emqx/emqx/issues/13378
2024-07-03 09:51:42 -03:00
Thales Macedo Garitezi 2c3209e258
Merge pull request #13395 from thalesmg/20240702-r57-test-flaky-cluster-invite-timeout
test: attempt to fix flaky tests
2024-07-03 09:32:58 -03:00
zhongwencool fe256363ad
Merge pull request #13331 from lafirest/fix/error-msg
fix(api_keys): improved the error message when bootstrapping api_key
2024-07-03 15:38:44 +08:00
zhongwencool cfa29eaa6f Merge remote-tracking branch 'upstream/release-57' into 20240702-m-sync-r57-mix-umbrella 2024-07-03 15:30:11 +08:00
zhongwencool 5f321702e7
Merge pull request #13398 from zhongwencool/authz-db-clear-table
fix: don't destory when authz'source unchanged
2024-07-03 15:29:39 +08:00
JianBo He 9f44c50025
Merge pull request #13350 from HJianBo/add-peersni-to-client-attr
feat: support to extract the client peersni field to clientinfo
2024-07-03 15:29:21 +08:00
zhongwencool cfa7c3bf04 Merge remote-tracking branch 'upstream/release-57' into 20240702-m-sync-r57-mix-umbrella 2024-07-03 15:15:58 +08:00
zhongwencool 112433da87 fix: don't destory when authz'source unchanged 2024-07-03 13:13:23 +08:00
firest ac6bbd2977 fix(api_keys): improved the error message when bootstrapping api_key 2024-07-03 11:08:47 +08:00
zhongwencool 0dfa3e8c86 chore: ensure the module is loaded 2024-07-03 10:11:33 +08:00
JianBo He 920e039487 chore: fix failed tests 2024-07-03 09:55:45 +08:00
zhongwencool 7c02e1979e chore: add cluster_rpc_opts() type in emqx_config 2024-07-03 09:49:14 +08:00
zhongwencool 059baf9ea5 test: add authz test case 2024-07-03 09:49:13 +08:00
zhongwencool e1c3b7587d feat: do not fail on other nodes when the RPC succeeds on the first node 2024-07-03 09:49:13 +08:00
Thales Macedo Garitezi 9ef3eff4c6
Merge pull request #13394 from thalesmg/20240702-r57-atom-leak-schema-registry
fix(schema registry): avoid atom leak
2024-07-02 17:40:26 -03:00
Thales Macedo Garitezi fdf43455d9
Merge pull request #13383 from thalesmg/20240701-r57-test-flaky-gconsu-multiple-pull-workers
test: attempts to stabilize flaky tests
2024-07-02 17:40:10 -03:00
Thales Macedo Garitezi b0c0c02df9 test: attempt to fix flaky tests
```
%%% emqx_mgmt_api_cluster_SUITE ==> t_cluster_invite_api_timeout: FAILED
%%% emqx_mgmt_api_cluster_SUITE ==>
Failure/Error: ?assertMatch([ # { core_node := Core1 , replicant_nodes := [ # { node := Replicant , streams := _ } ] } , # { core_node := Core2 , replicant_nodes := [ # { node := Replicant , streams := _ } ] } ], lists : sort ( Core1Resp ))
  expected: = [ # { core_node := Core1 , replicant_nodes := [ # { node := Replicant , streams := _ } ] } , # { core_node := Core2 , replicant_nodes := [ # { node := Replicant , streams := _ } ] } ]
       got: [#{core_node => 'data_backup_core1@127.0.0.1',
               replicant_nodes => []},
             #{core_node => 'data_backup_core2@127.0.0.1',
               replicant_nodes =>
                   [#{node => 'data_backup_replicant@127.0.0.1',
                      streams => 7}]}]
      line: 111
```
2024-07-02 15:15:03 -03:00
Thales Macedo Garitezi ea30d50125 fix(schema registry): avoid atom leak
Fixes https://emqx.atlassian.net/browse/EMQX-12603
2024-07-02 14:01:21 -03:00
Andrew Mayorov 1d5669d008
fix(sessds): tolerate removal of nonexistent gbt entries 2024-07-02 17:15:41 +02:00
Thales Macedo Garitezi 5532f40d83 Merge branch 'release-57' into 20240702-m-sync-r57-mix-umbrella 2024-07-02 11:52:36 -03:00
Thales Macedo Garitezi f64bd313aa
Merge pull request #13263 from thalesmg/mix-umbrella-m-20240612
perf: "mixify" project to improve release speed (part 1)
2024-07-02 11:36:09 -03:00
zhongwencool 55298ab6f3
Merge pull request #13387 from zhongwencool/dont-override-authn-users
fix: don't override authn users when import_user from authn.boostrap_file
2024-07-02 22:33:16 +08:00
lafirest 3f0d59300b
Merge pull request #13348 from lafirest/fix/drop_payload
feat(log): allows custom the encoding of payload in the log
2024-07-02 21:47:48 +08:00
Andrew Mayorov a57917b66b
perf(sessds): rotate through streams with iterators when fetching
This avoids expensive shuffling of the whole list of fetchable streams,
which can be quite long.
2024-07-02 15:42:33 +02:00
Andrew Mayorov 9a4f3f88e3
feat(sessds): allow stream iteration starting from a specific key 2024-07-02 15:40:40 +02:00
Andrew Mayorov dc73b957b3
feat(sessds): use trees to hold streams in session state 2024-07-02 15:39:02 +02:00
Thales Macedo Garitezi 4df2e0be85 fix: bad rebase conflict resolution 2024-07-02 10:21:52 -03:00
zhongwencool 08596f886a feat: add default csv file in authn-built-in-db 2024-07-02 21:18:38 +08:00
zhongwencool 8e904099c7 fix: add boostrap_file/type to post authn_api 2024-07-02 21:18:37 +08:00
zhongwencool 4d912516c8 fix: don't override authn users when import_user from authn.boostrap_file 2024-07-02 21:18:37 +08:00
Thales Macedo Garitezi bd0c8f0204 fix: bad conflict resolution 2024-07-02 09:47:49 -03:00
Thales Macedo Garitezi 89bd69eb50 fix(bridge v1 api): return correct http response code when incompatible 2024-07-02 09:46:06 -03:00
Thales Macedo Garitezi 8843fcbbf4 perf: use manifest to track proto file compilation 2024-07-02 09:45:47 -03:00
Thales Macedo Garitezi 19f3b030f9 chore: preparing to run common tests / eunit with mix 2024-07-02 09:45:45 -03:00
Thales Macedo Garitezi b74828d7ea perf: "mixify" project to improve release speed
Fixes https://emqx.atlassian.net/browse/EMQX-12527
2024-07-02 09:39:48 -03:00
lafirest dea2bf19b1
Merge pull request #13389 from lafirest/fix/builtin_acl
fix: fix two minor issues of bulti-in authn/authz
2024-07-02 20:19:53 +08:00
Thales Macedo Garitezi 5a0bae2318
Merge pull request #13367 from thalesmg/20240628-r57-test-flaky-authz-cache
test(authz cache): attempt to fix flaky test
2024-07-02 09:14:06 -03:00
Thales Macedo Garitezi 24ac241727
Merge pull request #13380 from thalesmg/20240701-r57-fix-http-bridge-hc-return-value
fix(http action): use correct return value for channel health check
2024-07-02 09:13:53 -03:00
Thales Macedo Garitezi db07a1ebea feat(message transformation): add more read-only fields to context
Fixes https://emqx.atlassian.net/browse/EMQX-12583
2024-07-02 09:13:19 -03:00
firest 686f79c036 fix: fix two minor issues of bulti-in authn/authz
1. the `Derived Key Length` for `pbkdf2` should be a positive integer.
2. fix topics in the authorization rules might be parsed incorrectly
2024-07-02 19:35:48 +08:00
firest a46440d00a feat(log): allows custom the encoding of payload in the log 2024-07-02 19:33:31 +08:00
JianBo He 7e0bcd4eda chore: tests in otp26 2024-07-02 15:26:07 +08:00
Thales Macedo Garitezi 317b29451f test(gcp consumer): attempt to stabilize flaky test
```
%%% emqx_bridge_kafka_impl_consumer_SUITE ==> ssl.t_start_and_consume_ok: FAILED
%%% emqx_bridge_kafka_impl_consumer_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,error,
               {badmatch,{{1,0},timeout}},
               [{emqx_bridge_kafka_impl_consumer_SUITE,
                    '-t_start_and_consume_ok/1-fun-12-',4,
                    [{file,
                         "/emqx/apps/emqx_bridge_kafka/test/emqx_bridge_kafka_impl_consumer_SUITE.erl"},
                     {line,1184}]},
                {emqx_bridge_kafka_impl_consumer_SUITE,
                    t_start_and_consume_ok,1,
                    [{file,
                         "/emqx/apps/emqx_bridge_kafka/test/emqx_bridge_kafka_impl_consumer_SUITE.erl"},
                     {line,1171}]}]}}},
 [{emqx_bridge_kafka_impl_consumer_SUITE,t_start_and_consume_ok,1,
      [{file,
           "/emqx/apps/emqx_bridge_kafka/test/emqx_bridge_kafka_impl_consumer_SUITE.erl"},
       {line,1240}]},
```
2024-07-01 17:57:17 -03:00
Thales Macedo Garitezi c04e93838f test(eviction agent): attempt to stabilize flaky test
```
%%% emqx_eviction_agent_SUITE ==> t_session_serialization: FAILED
%%% emqx_eviction_agent_SUITE ==>
Failure/Error: ?assertEqual(1, emqx_eviction_agent : session_count ( ))
  expected: 1
       got: 2
      line: 348
```
2024-07-01 16:10:32 -03:00
Thales Macedo Garitezi 59084dbfbe fix(message transformation): correctly read from user properties in context
Port of https://github.com/emqx/emqx/pull/13316 to release-57

Fixes https://emqx.atlassian.net/browse/EMQX-12582
2024-07-01 14:53:42 -03:00
Thales Macedo Garitezi 6dbb561944
Merge pull request #13355 from thalesmg/20240627-r57-fix-connector-api-bad-ssl-config
fix(connector api): handle bad tls config file conversion errors
2024-07-01 14:26:09 -03:00
Thales Macedo Garitezi b14856cf1a refactor: improve error messages 2024-07-01 13:21:17 -03:00
Thales Macedo Garitezi 72664780df fix(http action): use correct return value for channel health check
Fixes https://emqx.atlassian.net/browse/EMQX-12622
2024-07-01 11:17:57 -03:00
Ivan Dyachkov 532f04da9d
Merge pull request #13373 from id/0701-sync-release-57
sync release-57
2024-07-01 16:02:29 +02:00
Thales Macedo Garitezi d1e9b097d1
Merge pull request #13365 from thalesmg/20240628-r57-fix-dashboard-add-default-user-concurrently
fix(dashboard): handle add default user race condition
2024-07-01 10:22:32 -03:00
Ivan Dyachkov 505f568c32 chore: bump apps versions 2024-07-01 13:52:27 +02:00
Ivan Dyachkov e28750b522 Merge remote-tracking branch 'upstream/release-57' into 0701-sync-release-57 2024-07-01 13:52:14 +02:00
Kjell Winblad e291dcdd18 fix: default value for max_conn_rate etc should be set to infinity
Before this commit the default value for the fields max_conn_rate,
messages_rate and bytes_rate were not set. This is fixed by setting the
default value to infinity. This breaks the corresponding dashboard
fields (they can not be edited) so the dashboard also needs to be
updated.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12514
2024-07-01 13:16:26 +02:00
lafirest bd075caf56
Merge pull request #13358 from lafirest/fix/authn_reason
fix(events): fixed the reason format of the `authn_complete_event`
2024-07-01 18:46:18 +08:00
lafirest 6c665037de
Merge pull request #13372 from lafirest/fix/gateway_license
fix: limit gateway connections with license
2024-07-01 18:46:05 +08:00
firest a0644d4612 chore: update app version 2024-07-01 16:32:08 +08:00
firest b5d507bad8 fix: limit gateway connections with license 2024-06-30 20:06:04 +08:00
Thales Macedo Garitezi ec6e862539 test(authz cache): attempt to fix flaky test
Hypothesis: some race condition involving the previous test case, which uses the same
clientid.

```
Testing apps.emqx.emqx_authz_cache_SUITE: *** FAILED test case 3 of 3 ***
%%% emqx_authz_cache_SUITE ==> t_drain_authz_cache: FAILED
%%% emqx_authz_cache_SUITE ==>
Failure/Error: ?assertEqual([], list_cache ( ClientPid ))
  expected: []
       got: [{{#{qos => 0,action_type => publish,retain => false},<<"t1">>},
              {allow,1719599365019}}]
      line: 72
```
2024-06-28 15:40:55 -03:00
Thales Macedo Garitezi 9215b3710f
Merge pull request #13327 from thalesmg/fix-kprodu-delete-wolff-r57-20240624
fix(kafka and derivatives): add `alias` config to avoid clashes with same topic
2024-06-28 15:22:11 -03:00
Thales Macedo Garitezi b69f298058 fix(dashboard): handle add default user race condition
This can happen at least in tests, when nodes boot concurrently.
2024-06-28 15:18:23 -03:00
Thales Macedo Garitezi 795d280861 fix(connector api): handle bad tls config file conversion errors
Fixes https://emqx.atlassian.net/browse/EMQX-12581
2024-06-28 14:31:31 -03:00
Thales Macedo Garitezi 9ede62c9b1
Merge pull request #13356 from thalesmg/20240626-m-test-sparkplug-empty-roundtrip
test(schema registry): add test asserting the behavior of empty message roundtrip
2024-06-28 12:08:38 -03:00
Thales Macedo Garitezi 2a9c27d206
Merge pull request #13345 from thalesmg/20240626-r572-fix-validate-schema-reg-name
fix(schema registry api): validate schema name when creating
2024-06-28 12:08:27 -03:00
Thales Macedo Garitezi 063e7657b5
Merge pull request #13344 from thalesmg/20240626-r572-multi-node-bulk-subscribe
fix(client mgmt api): make bulk subscribe work again in clusters
2024-06-28 12:08:16 -03:00
Thales Macedo Garitezi 067beece75 chore: add count to thrown error 2024-06-28 12:06:47 -03:00
JianBo He 7cf0e69fdf chore: fix failed tests 2024-06-28 14:07:59 +08:00
firest e8176b80a6 fix(events): fixed the reason format of the `authn_complete_event` 2024-06-28 11:09:28 +08:00
JianBo He b39557f6fd
Merge pull request #13336 from zhongwencool/authn-boostrap-file
feat: support bootstrap_file on authentication for build-in-database
2024-06-28 09:07:53 +08:00
zhongwencool 2c48d7e0f0
Merge pull request #13347 from zhongwencool/improve-check-oom-log
chore: log shutdown reason for check_oom trace log
2024-06-28 08:06:11 +08:00
Thales Macedo Garitezi 79f15b1daa test(schema registry): add test asserting the behavior of empty message roundtrip
Relates to https://emqx.atlassian.net/browse/EMQX-10866
2024-06-27 17:49:57 -03:00
Thales Macedo Garitezi e76e94b497 perf(mgmt): optimize bulk unsubscribe when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi 0b329dbf06 perf(mgmt): optimize bulk subscribe when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi c49900af50 perf(mgmt): optimize clientid lookup when registry is enabled 2024-06-27 10:32:55 -03:00
Thales Macedo Garitezi 7b7f44b9ac fix(client mgmt api): make bulk subscribe work again in clusters
Fixes https://emqx.atlassian.net/browse/EMQX-12337
2024-06-27 09:37:55 -03:00
Thales Macedo Garitezi 6f00df6452 fix(schema registry api): validate schema name when creating
Fixes https://emqx.atlassian.net/browse/EMQX-10958
2024-06-27 09:26:41 -03:00
Thales Macedo Garitezi ff8c2bc1d8 feat(authz): add `ignore` metric for each source type
Fixes https://emqx.atlassian.net/browse/EMQX-12411
2024-06-27 09:22:45 -03:00
Thales Macedo Garitezi 4bd0abc93f chore: bump app vsns 2024-06-27 09:22:06 -03:00
Thales Macedo Garitezi 4c3c86e919 chore: bump wolff -> 2.0.0 2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi 164a507899 test(pulsar): add testcase for different producers using the same topic 2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi ed5e6599d9 fix(buffer worker, kafka): send reply when async call fails immediately
Fixes https://emqx.atlassian.net/browse/EMQX-12585
2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi 6023012f8b fix(kafka and derivatives): add `alias` config to avoid clashes with same topic
Fixes https://emqx.atlassian.net/browse/EMQX-12592
2024-06-27 09:20:00 -03:00
JianBo He 3b21c41690 feat: support to extract the client peersni field to clientinfo 2024-06-27 17:13:10 +08:00
zmstone b6e7d7566d Merge remote-tracking branch 'origin/release-572' into release-57 2024-06-27 11:09:02 +02:00
JimMoen ef28579c4a
Merge pull request #13334 from JimMoen/fix-mqtt-username-password-flag
fix(mqtt): check password flag to respect protocol spec
2024-06-27 16:16:53 +08:00
zhongwencool 6a78951715 chore: log shutdown reason for check_oom trace log 2024-06-27 14:52:22 +08:00
lafirest 68d747b7b9
Merge pull request #13324 from lafirest/feat/oidc-572
feat(sso): add OIDC support
2024-06-27 11:24:05 +08:00
firest 3d398873f1 fix(oidc): return to dashboard when provider calls back
fixed a bug when updating config
2024-06-27 10:22:07 +08:00
zhongwencool 9594b6df32 chore: warning overrided when restart authn 2024-06-27 10:16:04 +08:00
zhongwencool 24d2534641 chore: apply review suggestion 2024-06-27 10:16:04 +08:00
zhongwencool 5265c3cc1f feat: support bootstrap_file on build-in-db authn 2024-06-27 10:16:04 +08:00
zmstone 557a843c69 chore: minimize oss/platform diff 2024-06-26 22:48:50 +02:00
zmstone 82e7b75a02 chore: bump app versions 2024-06-26 22:47:18 +02:00
zmstone 98f70ea8d8 Merge remote-tracking branch 'origin/release-572' into release-57 2024-06-26 22:28:08 +02:00
Andrew Mayorov d8963c836e
Merge pull request #13332 from keynslug/ft/EMQX-12571/error-mapping
feat(bridge-s3): provide more meaningful error details in status
2024-06-26 21:13:51 +02:00
Andrew Mayorov e1de18ef10
test(dsrepl): await stable state before running testcase 2024-06-26 20:44:35 +02:00
Andrew Mayorov 30efa1f57e
test(dsrepl): relax crash-recover testcase to tolerate message loss
Which is quite an expected occasion for this kind of stress test.
2024-06-26 20:44:34 +02:00
Andrew Mayorov 3d296abde9
fix(dsrepl): classify ra error conditions more carefully
Most importantly: avoid automatic retries of `shutdown` and `nodedown`
errors as this could easily lead to Raft log entries duplication.
2024-06-26 20:44:34 +02:00
Andrew Mayorov 733751fadd
refactor(dsstore): keep passing `Options` to both prepare + commit 2024-06-26 20:44:34 +02:00
Andrew Mayorov 5b5f33c421
chore(dsstore): resurrect `prepare_batch` entry tracepoint 2024-06-26 20:44:34 +02:00
Andrew Mayorov 8ff48ac5ea
feat(dsrepl): rely on accumulated log size to decide when to flush 2024-06-26 20:44:34 +02:00
Andrew Mayorov b6a249baa9
feat(cth-peer): add brutal `kill/1` facility 2024-06-26 20:44:34 +02:00
Andrew Mayorov 8db70b5bbc
test(dsrepl): add crash-restart-recover testcase
That verifies nothing is lost in the event of abrupt node failures.
2024-06-26 20:44:33 +02:00
Andrew Mayorov ae89b61af0
feat(cth-cluster): make `restart/1` more generic 2024-06-26 20:44:33 +02:00
Andrew Mayorov 5fd5fc76e5
fix(dsstore): ensure backward compatibility 2024-06-26 20:44:33 +02:00
Andrew Mayorov 8538a5a5b6
test(dsrepl): anticipate transitionless membership changes
E.g. when a membership change is applied twice in a row.
2024-06-26 20:44:33 +02:00
Andrew Mayorov cd0663074e
test(dsrepl): add `add_generation` events into the mix
They usually cause storage layer to perform flushes, and thus enable
testing `handle_flush/1` codepath in different circumstances.
2024-06-26 20:44:33 +02:00
Andrew Mayorov 2180cc7c26
fix(dsstore): avoid storing `cf_refs()` in the RocksDB itself
This is both pointless and confusing.
2024-06-26 20:44:33 +02:00
Andrew Mayorov 80ea2e62f7
fix(stream): ensure that `chain/1` preserves the order 2024-06-26 20:44:32 +02:00
Andrew Mayorov 7895e9cc45
feat(dsstore): make WAL-less mode optional
And make the upper layer choose when to use it.
2024-06-26 20:44:32 +02:00
Andrew Mayorov 0c0757b8c2
feat(dsrepl): enable WAL-less batch writes 2024-06-26 20:44:32 +02:00
Andrew Mayorov 2705226eb5
feat(dsrepl): release log entries occasionally
Also make tracepoints in `apply/3` callback implementation more
uniform.
2024-06-26 20:44:32 +02:00
Andrew Mayorov b6894c18fa
chore(dsrepl): improve tracepoints usability a bit 2024-06-26 20:44:32 +02:00
Thales Macedo Garitezi 263e654208
Merge pull request #13340 from thalesmg/20240626-r572-fix-docs-conf-load
fix: don't crash if application is already loaded
2024-06-26 15:15:49 -03:00
Thales Macedo Garitezi e81494a132 fix: don't crash if application is already loaded
```
Runtime terminating during boot ({{badmatch,{error,{already_loaded,wolff}}},[{emqx_conf,load,2,[{file,"emqx_conf.erl"},{line,167}]},{lists,foreach_1,2,[{file,"lists.erl"},{line,1686}]},{emqx_conf,dump_schema,2,[{file,"emqx_conf.erl"},{line,150}]},{erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,746}]},{erl_eval,expr,6,[{file,"erl_eval.erl"},{line,494}]},{erl_eval,exprs,6,[{file,"erl_eval.erl"},{line,136}]},{init,start_it,1,[]},{init,start_em,1,[]}]})
```
2024-06-26 13:41:32 -03:00
Thales Macedo Garitezi 954adc71c4 test: attempt to fix flaky test
https://github.com/emqx/emqx/actions/runs/9662725303/job/26653594859?pr=13328#step:6:186

```
%%% emqx_resource_SUITE ==> t_expiration_retry: FAILED
%%% emqx_resource_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,error,
               {badmatch,{ok,timeout}},
               [{emqx_resource_SUITE,'-do_t_expiration_retry/0-fun-12-',0,
                    [{file,
                         "/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
                     {line,2569}]},
                {emqx_resource_SUITE,do_t_expiration_retry,0,
                    [{file,
                         "/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
                     {line,2518}]}]}}},
 [{emqx_resource_SUITE,do_t_expiration_retry,0,
      [{file,"/__w/emqx/emqx/apps/emqx_resource/test/emqx_resource_SUITE.erl"},
       {line,2594}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]}
```
2024-06-26 13:40:49 -03:00
zmstone ef1b6d2a55 chore: keep enterprise edition version updated as well 2024-06-26 11:40:49 +02:00
zmstone 465ae507d8 chore: update release version to v5.7.1 2024-06-26 10:49:59 +02:00
zmstone af9bfc21cb
Merge pull request #13312 from zmstone/0621-fix-regex_extract-docs
doc: fix regex_extract function doc
2024-06-26 10:48:40 +02:00
zmstone 7f73f27d56
docs: fix typo 2024-06-26 10:18:08 +02:00
zhongwencool 4942f6f75a feat: improve keepalive_multiplier and keepalive_check_interval 2024-06-26 15:43:48 +08:00
Thales Macedo Garitezi 21c01f32ff
Merge pull request #13305 from kjellwinblad/kjell/redis_conn_always_time_out_no_username_password/EMQX-12557
fix: redis connector should not timeout because no username and password
2024-06-25 16:06:21 -03:00
Andrew Mayorov da214be5a1
test(bridge-s3): adapt testcase to different CI environment 2024-06-25 18:35:53 +02:00
Andrew Mayorov 10e9fed22b
Merge pull request #13326 from keynslug/fix/ct/api-clients-flaky
test(api-clients): refactor suite to isolate flakiness
2024-06-25 18:26:42 +02:00
JimMoen ed130fdc57
test: MQTT CONNECT flags check 2024-06-25 23:25:42 +08:00
JimMoen 02a9885aa5
fix(mqtt): check password flag to respect protocol spec 2024-06-25 23:25:36 +08:00
JimMoen 983f02ea1b
refactor: separate CONNECT flags validation funcs 2024-06-25 23:15:39 +08:00
Kjell Winblad 130571b56e fix: code improvements thanks to comments from @thalesmg 2024-06-25 12:00:36 -03:00
Kjell Winblad 6190192cbc fix: redis connector should not timeout because no username and password
A redis connector of type single or sentinel always got a timeout error
when doing the connector test in the dashboard if no username or
password was provided. This commit makes sure that the user instead get
an informative error message. Additionally, this commit adds more
more error information for all redis connector types.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12557
2024-06-25 12:00:36 -03:00
Thales Macedo Garitezi cc89a20ed9
Merge pull request #13318 from thalesmg/test-flaky-gcp-consu-r57-20240621
test(gcp consumer): stabilize flaky test
2024-06-25 11:52:20 -03:00
Andrew Mayorov 486a041adf
feat(bridge-s3): also map credentials / aggreg upload errors 2024-06-25 16:51:07 +02:00
Andrew Mayorov fb9afd8313
feat(bridge-s3): beautify posix write errors 2024-06-25 16:51:07 +02:00
Andrew Mayorov f3ffbd4710
feat(bridge-s3): provide more meaningful error details in status 2024-06-25 16:51:06 +02:00
Andrew Mayorov 98e4ea6fde
feat(bridge-s3): make validation errors more readable
And also turn them into schema-level validations, instead of
bridge-level error conditions.
2024-06-25 16:51:06 +02:00
Thales Macedo Garitezi 1a497bcaf2 fix(greptime): correctly define grpc options for `grpcbox_channel`
Will probably fix CI flakiness.
2024-06-25 09:47:31 -03:00
Thales Macedo Garitezi c9ec5ac87b test(gcp consumer): stabilize flaky test
https://github.com/emqx/emqx/actions/runs/9614788348/job/26526973635?pr=13317#step:5:1463

```
%%% emqx_bridge_gcp_pubsub_consumer_SUITE ==> t_connection_down_before_starting: FAILED
%%% emqx_bridge_gcp_pubsub_consumer_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,error,
               {assertMatch,
                   [{module,emqx_bridge_gcp_pubsub_consumer_SUITE},
                    {line,1451},
                    {expression,"health_check ( Config )"},
                    {pattern,"{ ok , connecting }"},
                    {value,{ok,disconnected}}]},
               [{emqx_bridge_gcp_pubsub_consumer_SUITE,
                    '-t_connection_down_before_starting/1-fun-11-',4,
                    [{file,
                         "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
                     {line,1451}]},
                {emqx_bridge_gcp_pubsub_consumer_SUITE,
                    t_connection_down_before_starting,1,
                    [{file,
                         "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
                     {line,1427}]}]}}},
 [{emqx_bridge_gcp_pubsub_consumer_SUITE,t_connection_down_before_starting,1,
      [{file,
           "/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},
       {line,1462}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1302}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1234}]}]}
```
2024-06-25 09:36:55 -03:00
ieQu1 9f30da334f
Merge pull request #13248 from ieQu1/dev/EMQX-12491-local-backend
replace builtin DS backend with builtin_local and builtin_raft
2024-06-24 23:07:57 +02:00
ieQu1 d349f84f04
fix(ds): Apply remarks 2024-06-24 20:20:40 +02:00
Andrew Mayorov d12b985507
test(api-clients): refactor suite to isolate flakiness 2024-06-24 16:54:20 +02:00
zmstone 5d1608f34b chore: minimize oss/platform diff 2024-06-24 15:30:03 +02:00
Thales Macedo Garitezi debf1e6cd5
Merge pull request #13316 from thalesmg/fix-mt-read-user-prop-m-20240621
fix(message transformation): correctly read from user properties in context
2024-06-24 09:31:48 -03:00
firest abc255bb02 fix(oidc): make CI happy 2024-06-24 18:11:02 +08:00
firest 892420e2c6 feat(oidc): be more compatible with okta 2024-06-24 16:51:54 +08:00
firest ddb197951e feat(oidc): implement JWKS, private_key_jwt, DPoP 2024-06-24 16:51:43 +08:00
firest 9c0df3c0a8 feat(oidc): support the PKCE extension 2024-06-24 16:51:32 +08:00
firest 5e2693c9b4 feat(oidc): implement session management 2024-06-24 16:51:20 +08:00
firest 512b4b9cbb feat(sso): add OIDC support 2024-06-24 16:50:58 +08:00
JianBo He 23acbb664e
Merge pull request #13258 from HJianBo/fix-boot-order-for-gws
fix: gateway app must depends on all auth apps
2024-06-24 16:31:27 +08:00
zmstone f9e17d6c25
Merge pull request #13269 from JimMoen/improve-plugin-behavior
fix: mark fresh install to cp the default configuration file directly
2024-06-24 09:17:25 +02:00
Ivan Dyachkov 4a81c3ac18
Merge pull request #13298 from qzhuyan/fix/william/hot-conf-disable-partial-chain
fix(tls): disable partial_chain in hot config
2024-06-24 09:02:07 +02:00
ieQu1 c0472a06f1
test(ds): Set initial conditions in repl. suite explicitly 2024-06-22 02:55:05 +02:00
Thales Macedo Garitezi ec83d999bf Merge remote-tracking branch 'origin/release-57' into 20240621-sync-r57-m 2024-06-21 14:11:08 -03:00
ieQu1 9a58d71378
fix(ds): Move DS backend supervision trees to their own apps 2024-06-21 18:22:04 +02:00
ieQu1 3d69ec496a
fix(schema): Transform config of `singleton' discovery_strategy 2024-06-21 17:16:25 +02:00
ieQu1 d7d878fd43
test(ds): Fix emqx standalone test profile selection 2024-06-21 17:16:25 +02:00
ieQu1 8aa27488b6
test: Disable certain DS-related suites in CE 2024-06-21 17:16:24 +02:00
ieQu1 1d3b1868fb
test(ds): Use close_db API 2024-06-21 17:16:24 +02:00
ieQu1 b2f7815a7f
test(ds): Don't start raft explicitly 2024-06-21 17:16:24 +02:00
ieQu1 3851fc189f
fix(ds): Avoid reverse dependencies from storage to repl. layer 2024-06-21 17:16:24 +02:00
ieQu1 bc915216a0
feat(ds): Support metrics for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 be6c5e172f
fix(ds): Disable DS management APIs for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 99c9b56cf3
feat(ds_buffer): Add `ls' function to list all local buffers 2024-06-21 17:16:24 +02:00
ieQu1 5a8818edf3
feat(ds): Add schema for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 4484f30021
feat(ds_schema): Add schema for builtin_local backend 2024-06-21 17:16:24 +02:00
ieQu1 8990b1312b
refactor(ds_schema): Rename backend builtin -> builtin_raft 2024-06-21 17:16:24 +02:00
ieQu1 ea48b1265d
refactor(ds_schema): Extract common builtin fields 2024-06-21 17:16:24 +02:00
ieQu1 abe41de19b
refactor(ds_schema): builtin_local_buffer -> builtin_buffer 2024-06-21 17:16:24 +02:00
ieQu1 ecb172b07e
refactor(ds): Rename egress metrics to 'buffer' 2024-06-21 17:16:24 +02:00
ieQu1 09c3ae795d
refactor(ds_raft): Replace egress server with common emqx_ds_buffer 2024-06-21 17:16:24 +02:00
ieQu1 a0fbd37e58
refactor(emqx): Use emqx_ds_backends application 2024-06-21 17:16:24 +02:00
ieQu1 ef09cfcd71
feat(ds): Add `emqx_ds_backends` application 2024-06-21 17:16:24 +02:00
ieQu1 279619fc80
feat(ds): Add `builtin_local' backend 2024-06-21 17:16:24 +02:00
ieQu1 a8ea0ae4e5
refactor(ds): Extract DS replication layer to a separate application 2024-06-21 17:16:24 +02:00
ieQu1 63f1856a2c
feat(ds): Dynamic backend registration 2024-06-21 17:16:24 +02:00
ieQu1 83dc8f4d77
fix(ds): Fix return values of emqx_ds_storage_layer functions 2024-06-21 17:16:23 +02:00
ieQu1 a18d1987a2
test(ds): Add a helper function for diffing messages 2024-06-21 17:16:23 +02:00
Ilia Averianov f1b8c356a6
Merge pull request #13299 from savonarola/0617-shared-sub-leader-poc
feat(queue): implement PoC version of session ↔️ shared group leader interaction
2024-06-21 16:45:17 +03:00
Thales Macedo Garitezi 28c7d94bd2 fix(message transformation): correctly read from user properties in context
Fixes https://emqx.atlassian.net/browse/EMQX-12582
2024-06-21 09:39:22 -03:00
Ilya Averyanov db28a042d5 feat(queue): handle renew_lease_timeout 2024-06-21 15:18:37 +03:00
zhongwencool 75a524c916 test: add more debug msg to flaky cluster_rpc SUITE 2024-06-21 18:04:24 +08:00
Ilya Averyanov 2096755ad6 feat(queue): reorganize group sm callbacks and methods 2024-06-21 13:01:38 +03:00
Ilya Averyanov b9c5911883 feat(queue): implement `find_leader_timeout` event and so the `connecting` group_sm state 2024-06-21 12:22:17 +03:00
SergeTupchiy fb266fbf8c
Merge pull request #13300 from SergeTupchiy/EMQX-12195-cluster-link-conf-backup
cluster link conf backup
2024-06-21 11:42:57 +03:00
zmstone 8a99995810 docs: fix regex_extract function doc 2024-06-21 09:25:09 +02:00
JianBo He d8e6c07ca0 chore: fix failed tests 2024-06-21 11:04:23 +08:00
JianBo He a8af90f912 chore: add changes 2024-06-21 09:39:14 +08:00
JianBo He 08f085d823 fix: gateway app must depends on all auth apps 2024-06-21 09:39:14 +08:00
Ilya Averyanov 979fb58e50 feat(queue): use tp to trace new lease events 2024-06-20 21:12:15 +03:00
Ilya Averyanov 1205e34650 feat(queue): remove too useless header file 2024-06-20 20:48:56 +03:00
JimMoen acc8bf3405
Merge pull request #13222 from JimMoen/fix-will-qos-retain-flag-check
fix: check willretain and willqos when WillFlag set to `true`
2024-06-21 01:28:23 +08:00
Ivan Dyachkov ba4fb0d3f9
Merge pull request #13290 from kjellwinblad/kjell/format_rule_cli/EMQX-12548
fix: do not crash when showing rule unsing the cmd interface
2024-06-20 14:55:33 +02:00
zmstone 2a0071aa01
Merge pull request #13295 from thalesmg/test-flaky-greptime-mkII-r57-20240619
test(greptime): attempt to fix flaky tests (attempt without driver patch)
2024-06-20 14:49:21 +02:00
Ilya Averyanov be175d205c feat(queue): add JIRA task links for TODOs 2024-06-20 15:38:16 +03:00
Ilya Averyanov a9c976b6c1 feat(queue): fix notes about the usage of agent<->leader protocol-related data structures 2024-06-20 15:24:25 +03:00
Ilya Averyanov d0cff63ed6 feat(queue): remove unused hrl-guard 2024-06-20 15:18:47 +03:00
Ilya Averyanov a92460d38f feat(queue): improve hints for macro clauses 2024-06-20 15:17:08 +03:00
Serge Tupchii ac8762be10 chore: upgrade ekka to 0.19.5 (mria 0.8.8)
mria 0.8.8 heals a network partition once majority of core nodes are alive.
Previously, the autoheal worked only when all core nodes were reachable.

Fixes: EMQX-10974
2024-06-20 12:53:58 +03:00
William Yang 464a0a82f0 fix(tls): move default_root_fun to ?CONST_MOD_V1 2024-06-20 11:38:52 +02:00
zmstone 7d32275ebe chore: minimize oss/platform diff 2024-06-20 10:36:01 +02:00
Kjell Winblad 12859b8959 fix: handle yet another action format 2024-06-20 10:03:49 +02:00
JimMoen 675abd7512
test: will retain and willqos in connect flags 2024-06-20 10:32:15 +08:00
JimMoen cf9d6943d5
fix: check willretain and willqos when WillFlag set to `true` 2024-06-20 10:32:15 +08:00
JianBo He c307c1dfc2 chore: cut v5.7.1-alpha.1 2024-06-20 09:40:42 +08:00
Ilya Averyanov 6da10036dc feat(queue): fix dialyzer issues 2024-06-19 22:25:53 +03:00
Ilya Averyanov a4bbab4aa2 feat(queue): document general entity layout 2024-06-19 22:24:20 +03:00
JimMoen ff742d926a
fix: `?SLOG` with msg field 2024-06-20 02:18:26 +08:00
Serge Tupchii 71f5eaf11e fix(data_backup): import `cluster.links` conf 2024-06-19 20:47:06 +03:00
Serge Tupchii ef5cf4fac3 chore(clusterlink): fix comment 2024-06-19 20:46:00 +03:00
Ilya Averyanov 2b0146663a feat(queue): add smoke test for ds shared subs 2024-06-19 19:32:23 +03:00
Ilya Averyanov e3c4816035 feat(queue): move group subscription state machine to its own module 2024-06-19 19:22:10 +03:00
Thales Macedo Garitezi 9559ba2f06 test: remove unnecessary lines 2024-06-19 13:22:02 -03:00
William Yang 8d04545f03 test(tls): root_fun is set even when partial_chain is false 2024-06-19 17:34:04 +02:00
William Yang 41239ae766 fix(tls): disable partial_chain in hot config 2024-06-19 17:03:05 +02:00
zmstone 021b6b3902 Merge remote-tracking branch 'origin/release-57' into 0619-sync-release-57-to-master 2024-06-19 15:06:41 +02:00
zmstone 02a6ee1ef4
Merge pull request #13279 from zmstone/0617-json-decode-payload-for-map_to_redis_hset_args
fix(redis): json decode if arg for map_to_redis_hset_args is string
2024-06-19 14:55:21 +02:00
zmstone 213e4785e7 Merge remote-tracking branch 'origin/release-57' into 0619-sync-release-57-to-master 2024-06-19 14:54:48 +02:00
JimMoen a6fa3e82d9
fix: prevent use pd 2024-06-19 20:15:07 +08:00
JimMoen b7b2a08399
fix: match plugin_not_found reason 2024-06-19 20:15:07 +08:00
JimMoen 28293284a7
fix: let crash when plugin not found in cluster 2024-06-19 20:15:07 +08:00
JimMoen 19e039e0d2
fix: mark fresh install to cp the default configuration file directly 2024-06-19 20:15:07 +08:00
Thales Macedo Garitezi ff2e6e1434 test(greptime): revert driver patch to test ci again 2024-06-19 09:12:30 -03:00
zmstone 95d387a790
Merge pull request #13294 from zmstone/0618-per-release-edition-code-injection
0618 per release edition code injection
2024-06-19 11:53:57 +02:00
zmstone f0e5721959 refactor: make dialyzer happy 2024-06-19 10:42:08 +02:00
SergeTupchiy 0d098a01ef
Merge pull request #13293 from SergeTupchiy/EMQX-12345-reindex-retained-msgs-on-import
reindex retained msgs on import
2024-06-19 11:23:08 +03:00
Thales Macedo Garitezi ffec1c7fe0 fix(greptime): attempt patch for cleaning up channels when worker terminates 2024-06-18 22:16:03 -03:00
Thales Macedo Garitezi 9dd419d822 test(greptime): attempt to fix flaky tests
```
=WARNING REPORT==== 18-Jun-2024::12:42:04.739109 ===
    reason: no_endpoints
    msg: failed_to_start_greptimedb_connector
    client: #{protocol => http,
              pool => <<"connector:greptimedb:emqx_bridge_greptimedb_SUITE">>,
              cli_opts =>
                  [{endpoints,[{http,"toxiproxy",4001}]},
                   {pool_size,2},
                   {pool_type,random},
                   {auto_reconnect,1},
                   {gprc_options,#{sync_start => true,
                                   connect_timeout => 5000}},
                   {dbname,"public"},
                   {auth,{basic,#{password => <<"******">>,
                                  username => "greptime_user"}}},
                   {https_enabled,false}]}
    connector: <<"connector:greptimedb:emqx_bridge_greptimedb_SUITE">>
```
2024-06-18 18:23:22 -03:00
zmstone 54fc605cc5 refactor: move schema injection from hrl to erl 2024-06-18 22:22:53 +02:00
zmstone a53fa977df refactor: delete an unused macro 2024-06-18 22:22:48 +02:00
Thales Macedo Garitezi e69ba33420
Merge pull request #13291 from thalesmg/fix-ds-mgmt-node-down-r57-20240618
fix(dsreplmeta): check site status when fetching shard info
2024-06-18 15:14:08 -03:00
Ilya Averyanov bca743054b feat(queue): implement backbones of queue agent, leader and leader registry 2024-06-18 21:03:51 +03:00
Thales Macedo Garitezi 4c54ab6379 test: remove tests per review request 2024-06-18 13:08:19 -03:00
Kjell Winblad 4a318f8f51 style: fix type problem found by dialyzer 2024-06-18 17:02:26 +02:00
Serge Tupchii baa7996289 fix(data_backup): allow exporting `ram_copies` Mnesia tables
Currently, ram tables can be used for message retainer.
2024-06-18 17:31:33 +03:00
Serge Tupchii 60882a616e fix(data_backup): re-index retained messages after importing 2024-06-18 17:31:19 +03:00
Thales Macedo Garitezi 6bde6aa711
Merge pull request #13280 from thalesmg/fix-plugin-elixir-r57-20240617
fix(plugins): do not stop Elixir protected apps
2024-06-18 10:41:12 -03:00
Thales Macedo Garitezi 8ce16fd7d9 fix(dsreplmeta): check site status when fetching shard info
Fixes https://emqx.atlassian.net/browse/EMQX-12356
2024-06-18 10:40:09 -03:00
Kjell Winblad a4f855108c fix: do not crash when showing rule unsing the cmd interface
Before the change the command line command

    $ bin/emqx ctl rules show rule_0hyd

would crash if the rule had bridge actions. This has now been fixed by
adding a handler for printing  bridge actions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12548
2024-06-18 15:17:25 +02:00
Kjell Winblad 3f7723b2dc
Merge pull request #13274 from kjellwinblad/kjell/add_ssl_to_rocksdb/EMQX-12289
feat: add SSL support to RocketMQ connector
2024-06-18 09:38:28 +02:00
Thales Macedo Garitezi 20cffb54d4 Merge branch 'release-57' into sync-r57-m-20240617 2024-06-17 17:10:12 -03:00
Thales Macedo Garitezi 6897f0141b fix(plugins): do not stop Elixir protected apps
On one hand, Elixir plugins might include Elixir itself, when targetting a non-Elixir EMQX
release.  If, on the other hand, the EMQX release already includes Elixir, we shouldn't
stop Elixir nor IEx, or the running IEx shell will break.
2024-06-17 16:42:23 -03:00
Ilia Averianov ad993437aa
Merge pull request #13278 from savonarola/0617-release-57-sync
Sync release-57
2024-06-17 21:25:00 +03:00
zmstone 9479c8d33b fix(redis): json decode if arg for map_to_redis_hset_args is string 2024-06-17 20:19:53 +02:00
Andrew Mayorov eede9f349e
Merge pull request #13276 from keynslug/fix/ds/inherited-lts-lost
fix(dsstore): persist inherited LTS trie
2024-06-17 20:25:31 +03:00
zmstone ae22a64157
Merge pull request #13277 from zmstone/0617-handle-kafka-message_too_large-return
0617 handle kafka message too large return
2024-06-17 19:15:33 +02:00
Kjell Winblad 0cbbae9655 fix: make fixes due to comments from @thalesmg 2024-06-17 19:06:12 +02:00
Kjell Winblad c39c544c96 feat: add SSL support to RocketMQ connector 2024-06-17 19:06:12 +02:00
Thales Macedo Garitezi b64f0c0ca7
Merge pull request #13231 from thalesmg/mt-test-endpoint-r57-20240611
feat(message transformation): implement dryrun endpoint
2024-06-17 13:11:39 -03:00
Thales Macedo Garitezi 240bb09070
Merge pull request #13275 from thalesmg/test-flaky-sk-r57-20240617
test: attempt to fix flaky test
2024-06-17 13:01:08 -03:00
Ilya Averyanov f8e6aab86f Merge branch 'release-57' into 0617-release-57-sync
* release-57:
  chore(auth,http): cache REs for parsing URIs
  fix(auth,http): improve URI handling
  chore: revert ULOG/ELOG
  test: generate dispatch.eterm in dashboard test
  docs: refine change log
  feat: make the dashboard restart quicker
  chore: fix typo
  fix(http authz): handle unknown content types in responses
  chore: change types of mysql and mongodb fields to `template()`
  fix(client mgmt api): allow projecting `client_attrs` from client fields
  fix(emqx_rule_funcs): expose regex_extract function to rule engine
2024-06-17 18:53:39 +03:00
Andrew Mayorov 82588fbc35
chore(dsstore): refine module and callback typespecs 2024-06-17 17:53:39 +02:00
Andrew Mayorov 05d97397d3
fix(dsstore): use correct comparison when walking over generations 2024-06-17 17:53:39 +02:00
Andrew Mayorov 22009bcc58
test(dsstore): verify that inherited TLS trie is persisted 2024-06-17 17:53:38 +02:00
Andrew Mayorov 68f6556856
fix(dsstore): persist inherited LTS trie
Before this commit, inherited trie was actually only kept in memory
cache.

Also simplify storage backend behaviour around inheriting previous
generation's legacy.
2024-06-17 17:53:38 +02:00
SergeTupchiy 830266b4d5
Merge pull request #13126 from SergeTupchiy/EMQX-11967-cross-cluster-route-replication-syncer-actor
cluster linking
2024-06-17 18:52:46 +03:00
zmstone 3ac4ddcbe3 fix(kafka): handle message_too_large
bump 'failed' counter
2024-06-17 17:27:25 +02:00
zmstone 00dfdc22cf chore: pin Kafka client wolff 0.10.5 2024-06-17 17:27:25 +02:00
zhongwencool 5ab4b321a0
Merge pull request #13267 from zhongwencool/use-redbug-for-trace
chore(debug): use redbug for default trace tools
2024-06-17 22:10:18 +08:00
Thales Macedo Garitezi fb492e3dc5
Merge pull request #13264 from thalesmg/fix-postgres-disabled-prepared-r57-20240614
fix(postgres): authn/authz/batch requests when prepared statements are disabled
2024-06-17 11:05:05 -03:00
Thales Macedo Garitezi 5665a4917c
Merge pull request #13266 from thalesmg/fix-s3-release-app-r57-20240614
fix: add `emqx_s3` to `reboot_lists.eterm`
2024-06-17 10:48:12 -03:00
Thales Macedo Garitezi ec4f462684 refactor: apply review requests 2024-06-17 10:39:32 -03:00
Thales Macedo Garitezi 1464094967 test: attempt to fix flaky test
```
%%% emqx_persistent_session_ds_SUITE ==> t_session_discard_persistent_to_persistent: FAILED
%%% emqx_persistent_session_ds_SUITE ==> {noproc,{gen_server,call,[snabbkaffe_collector,flush_trace,infinity]}}
```
2024-06-17 10:34:57 -03:00
Thales Macedo Garitezi b71212b022 refactor: apply review comments 2024-06-17 09:40:33 -03:00
Thales Macedo Garitezi f5eb3e7471
Merge pull request #13250 from thalesmg/ds-singleton-m-20240613
feat: enforce singleton discovery strategy when using DS `builtin_local` backend
2024-06-17 09:34:05 -03:00
Ilya Averyanov 19c9f0d76f chore(auth,http): cache REs for parsing URIs 2024-06-17 14:42:29 +03:00
Ilya Averyanov f7ac829f28 fix(auth,http): improve URI handling 2024-06-17 14:42:29 +03:00
Serge Tupchii a905a6048c chore(clusterlink): rename link `upstream` field to `name` 2024-06-17 12:24:51 +03:00
Serge Tupchii a95a08efd3 test(clusterlink): add more test cases 2024-06-17 12:24:51 +03:00
Serge Tupchii 44c37571cc fix(clusterlink): ignore not_registered error 2024-06-17 12:24:51 +03:00