Commit Graph

12813 Commits

Author SHA1 Message Date
firest 83cc3ffeb0 fix(banned): let the bootfile of banned be optional 2024-07-12 13:58:14 +08:00
lafirest 1b7d23cef4
Merge pull request #13451 from lafirest/fix/def_banned_file
fix: do not convert a empty file name to a empty list
2024-07-12 13:50:32 +08:00
Thales Macedo Garitezi 2816170e9d chore: add `$events.message_transformation_failed` to rule engine tester
Fixes https://emqx.atlassian.net/browse/EMQX-12679
2024-07-11 17:52:08 -03:00
Thales Macedo Garitezi 5f595966d8 chore(message transformation): allow empty operation list
Fixes https://emqx.atlassian.net/browse/EMQX-12682
2024-07-11 17:52:08 -03:00
Thales Macedo Garitezi 04b547d6f5 fix(schema registry): handle large names during lookup
Fixes https://emqx.atlassian.net/browse/EMQX-12692
2024-07-11 14:35:31 -03:00
Thales Macedo Garitezi f3c6d10f76 fix(mix): fix compile paths and deps 2024-07-11 14:19:23 -03:00
Thales Macedo Garitezi 01d89be743 feat(message transformation): add timestamp and pub_props fields to read context
Fixes https://emqx.atlassian.net/browse/EMQX-12684

Fixes https://emqx.atlassian.net/browse/EMQX-12678
2024-07-11 12:15:31 -03:00
Thales Macedo Garitezi 44e4b3616d feat(variform): allow hyphens in identifiers
Fixes https://emqx.atlassian.net/browse/EMQX-12683
2024-07-11 12:15:23 -03:00
Kjell Winblad a4cc3ba9e8
Merge pull request #13375 from kjellwinblad/kjell/fix_connector_lister_speed_limit_clearing/EMQX-12514
fix: default value for max_conn_rate etc should be set to infinity
2024-07-11 16:36:01 +02:00
firest d9b5c5863b fix: do not convert a empty file name to a empty list 2024-07-11 18:12:38 +08:00
lafirest c9e12f30cd
Merge pull request #13444 from lafirest/fix/oidc
fix(oidc): Avoid crashes and avoid deleting jwks on update
2024-07-11 17:55:50 +08:00
zmstone 7664b06e98
Merge pull request #13434 from zmstone/0704-refine-rpc-config
0704 refine rpc config
2024-07-11 10:25:45 +02:00
ieQu1 4825079964
chore(ds): Support platform profile 2024-07-10 12:03:23 +02:00
firest b0e3e405cf fix(oidc): Avoid crashes and avoid deleting jwks on update 2024-07-10 15:22:43 +08:00
JimMoen 44d533fe6d
Merge pull request #13432 from JimMoen/0705-fix-jwt-pem-check
fix: create authn jwt with bad public key
2024-07-10 10:33:54 +08:00
zmstone 7a23ae7b4d refactor: expose only server_port for rpc
previously, there were 4 port configs:
- tcp_server_port
- ssl_server_port
- tcp_client_port
- ssl_client_port
2024-07-09 22:26:29 +02:00
zmstone ee13773496 refactor: rename rpc.tcp_client_num to rpc.client_num
tcp_client_num is kept as an alias
2024-07-09 22:26:29 +02:00
Thales Macedo Garitezi 48e604bda8 fix(mix grpc): include default erlc options 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 818070ad44 test(mix): add integration test path 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 5279ad76be fix(grpc compiler): unload apps to avoid side effects 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi b91515b131 fix(schema registry mix): gpb is a runtime dep 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 6d94809950
Merge pull request #13415 from thalesmg/20240703-m-couchbase-action
feat: implement couchbase connector and action
2024-07-09 15:53:11 -03:00
Thales Macedo Garitezi 3c370a90aa
Merge pull request #13436 from thalesmg/20240708-r57-custom-headers-jwks
feat(jwks): allow specifying custom request headers
2024-07-09 15:52:44 -03:00
Thales Macedo Garitezi 9f8a1885a7 fix(resource manager): disentangle connector and channel health check frequencies
Fixes https://emqx.atlassian.net/browse/EMQX-12674
2024-07-09 14:53:39 -03:00
Thales Macedo Garitezi d25c4ba06f
Merge pull request #13421 from thalesmg/20240705-r57-docs-mt-api-examples
docs(message transformation): add api examples
2024-07-09 10:16:53 -03:00
ieQu1 3721be65ee
fix(ds): Improve comments 2024-07-09 13:15:15 +02:00
ieQu1 d7732a6aac
test(ds): Attempt to stabilize a flaky test 2024-07-09 13:15:15 +02:00
ieQu1 e70c1cfea3
test(ds): Improve stability of replication test suite 2024-07-09 13:15:15 +02:00
ieQu1 dc4ae82798
test(ds): Add message ID 2024-07-09 13:15:14 +02:00
ieQu1 d1b574a67e
perf(dslocal): Run heavy-duty operations in a temporary process 2024-07-09 13:15:14 +02:00
ieQu1 661f79544b
fix(ds): Optimize hot loop of skipstream storage layout 2024-07-09 13:15:14 +02:00
ieQu1 23dafbb03b
feat(ds): Add a benchmarking tool for storage efficiency analysis 2024-07-09 13:15:14 +02:00
ieQu1 afeb2ab8aa
feat(ds): Add metrics for skipstream layout 2024-07-09 13:15:14 +02:00
ieQu1 b68ebb9a73
test(dsrepl): Generalize tests to use different storage layouts 2024-07-09 13:15:14 +02:00
ieQu1 8c5e4a2376
test(ds): Generalize storage layout test suite for different layouts 2024-07-09 13:15:14 +02:00
ieQu1 086e7256f5
feat(ds): Add configuration schema for skipstream LTS layout 2024-07-09 13:15:14 +02:00
ieQu1 a4642d4d06
feat(ds): Add a new storage layout engine: "skipstream"
This layout is based on LTS as well, but it uses separate index
streams for constrained replay of streams with learned wildcards
2024-07-09 13:15:14 +02:00
ieQu1 de48077ac4
test(ds): Add new helper functions
- Improve message comparison
- Add set operations
2024-07-09 13:15:14 +02:00
ieQu1 210556e545
feat(ds): Generalize value serialization
- Add a new utility module for message serialization
- Add experimental serializer based on ASN.1
2024-07-09 13:15:14 +02:00
ieQu1 843973ef32
fix(ds): bitfield_lts: static_key_size -> static_key_bits 2024-07-09 13:15:14 +02:00
ieQu1 f84fb34692
feat(ds_lts): New APIs: info, reverse lookups and topic compression 2024-07-09 13:15:14 +02:00
ieQu1 eb80402ccb
fix(ds): Improve typespecs and descriptions in storage_layer 2024-07-09 13:15:14 +02:00
ieQu1 71dad0242e
docs(ds): Move Raft-related parts to emqx_ds_builtin_raft README 2024-07-09 13:15:14 +02:00
ieQu1 afe1c5617d
refactor(ds): Rename macros for bitfield_lts metrics 2024-07-09 13:15:14 +02:00
ieQu1 0f2c19b656
refactor(ds): Move end_of_stream detection logic for delete_next 2024-07-09 13:15:14 +02:00
ieQu1 b565976794
fix(ds): Fix hashing of empty wildcard topic levels in bitfield_lts 2024-07-09 13:15:13 +02:00
Thales Macedo Garitezi f00bb383d4 fix(ds builtin local): use `-include_lib` instead of relative path 2024-07-08 16:57:55 -03:00
Thales Macedo Garitezi 811184ddad feat(jwks): allow specifying custom request headers
Fixes https://emqx.atlassian.net/browse/EMQX-12655
2024-07-08 15:40:52 -03:00
Thales Macedo Garitezi d34fc7a03a
Merge pull request #13420 from thalesmg/20240705-r57-fix-mt-empty-topics
fix(schema validation & message transformation): forbid empty topic filter list
2024-07-08 10:15:08 -03:00
Thales Macedo Garitezi 166f5e5f12
Merge pull request #13426 from thalesmg/20240705-r57-test-flaky-plugin-start-enabled
test(plugins): fix flaky test
2024-07-08 09:19:53 -03:00
zhongwencool 820789a09f fix: redact status when mark_fix_log begin 2024-07-08 17:32:45 +08:00
zhongwencool 457ea93570 test: add cluster_sync cli test 2024-07-08 17:04:41 +08:00
zhongwencool f490a0cba2 feat: don't reset tnx_id when cluster_fix 2024-07-08 17:04:41 +08:00
zhongwencool 298211d101 chore: apply suggestions from code review
Co-authored-by: zmstone <zmstone@gmail.com>
2024-07-08 17:04:41 +08:00
zhongwencool bdf3fc63a6 chore: add config leader to suggestion 2024-07-08 17:04:41 +08:00
zhongwencool 22fc3c49cc chore: combine some common code into one function 2024-07-08 17:04:41 +08:00
zhongwencool 5b105fcdbb chore: move emqx_conf_proto_v3 to emqx_conf_proto_v4 2024-07-08 17:04:41 +08:00
zhongwencool 3ed4340145 test: fix cluster_rpc test failed 2024-07-08 17:04:41 +08:00
zhongwencool 2069910ad1 feat: add cluster fix command 2024-07-08 17:04:41 +08:00
JimMoen ae3b8fe146
test: create jwt authenticator with bad public key 2024-07-08 16:52:18 +08:00
JimMoen f76444fbf8
fix: create authn jwt with bad public key 2024-07-08 16:52:18 +08:00
zhongwencool 29d7a511f1
Merge pull request #13419 from zhongwencool/port-pr
Port: some minor bug fixes from master
2024-07-06 14:42:51 +08:00
Thales Macedo Garitezi f1b4467fe1 test(plugins): fix flaky test
Hypothesis is that both peer nodes were using the same directory and stepping on each
other's toes.
2024-07-05 14:17:42 -03:00
Thales Macedo Garitezi 70fab51354 fix: handle `max_heap_size` = 0
Fixes https://github.com/emqx/emqx/issues/13417

Fixes https://emqx.atlassian.net/browse/EMQX-12659
2024-07-05 13:10:37 -03:00
Kjell Winblad c8258cebe8
Merge pull request #13414 from kjellwinblad/kjell/review_connector_error_logs_rabbitmq_etc/EMQX-12462
fix: make RabbitMQ error log messages easier to understand
2024-07-05 15:01:15 +02:00
Thales Macedo Garitezi 36ee7bed77 docs(message transformation): add api examples
Fixes https://emqx.atlassian.net/browse/EMQX-12645
2024-07-05 09:59:27 -03:00
Thales Macedo Garitezi e7351d949d fix(schema validation): forbid empty topic filter list 2024-07-05 09:51:43 -03:00
Thales Macedo Garitezi e99fee68c0 fix(message transformation): forbid empty topic filter list
Fixes https://emqx.atlassian.net/browse/EMQX-12646
2024-07-05 09:49:03 -03:00
zhongwencool 7d851872ec chore: update emqx_module's app version 2024-07-05 19:21:28 +08:00
zhongwencool d94fcb9cfd test: fix api_config SUITE failed 2024-07-05 17:34:39 +08:00
zhongwencool ba3097dc56 fix: observer command crash when can't find object code 2024-07-05 17:34:33 +08:00
zhongwencool f0a1d785ca fix: don't allow set active_n to negative int 2024-07-05 17:34:25 +08:00
zhongwencool 8aab919f74 fix: load bad configs return unknown msg 2024-07-05 17:34:17 +08:00
zhongwencool b4cffc581b fix: ws/wss's max_frame_size should > 0 2024-07-05 17:34:09 +08:00
zhongwencool 094259f444
Merge pull request #13408 from zhongwencool/password-crash
chore: improve auth error for invalid salt/password type
2024-07-05 11:44:21 +08:00
JimMoen 9d0b5a9bc6
Merge pull request #13412 from JimMoen/fix-cert-notafter-badmatch
fix: obtain cert expiry epoch failed due to formated `generalTime`
2024-07-05 10:13:22 +08:00
zhongwencool d3d3303dcb chore: improve auth error for invalid salt/password type 2024-07-05 10:12:24 +08:00
lafirest aa84ca5a88
Merge pull request #13386 from lafirest/feat/banned_boot_57
feat(banned): add a bootstrap file for banned
2024-07-05 08:47:40 +08:00
Thales Macedo Garitezi c4dd167cb9 feat: implement couchbase connector and action
Fixes https://emqx.atlassian.net/browse/EMQX-12545
2024-07-04 17:51:59 -03:00
Thales Macedo Garitezi b333babb4c
Merge pull request #13401 from thalesmg/20240703-r57-authz-ignore-api-metrics
fix(authz api): add new `ignore` metric to status response
2024-07-04 17:07:08 -03:00
JimMoen d84d31cbc5
test: cert expiry epoch with `generalTime` formatted 2024-07-05 01:47:13 +08:00
Kjell Winblad b994e0f1c0 fix: make RabbitMQ error log messages easier to understand
Fixes:
https://emqx.atlassian.net/browse/EMQX-12462
2024-07-04 16:43:53 +02:00
Thales Macedo Garitezi 420493deb4 chore: add missing mix files to new apps 2024-07-04 10:27:59 -03:00
Kjell Winblad d206d24975 fix: only set default for max_conn_rate and update test case
This revert the change in commit e291dcd for all listener "short
path fields" except the field max_conn_rate so they no longer have a
default value. It also updates a test case that assume that no listener
config is created by default but this is no longer the case when there
is a default value for the max_conn_rate field.
2024-07-04 14:32:10 +02:00
Thales Macedo Garitezi f758fd9279
Merge pull request #13405 from thalesmg/20240703-r57-test-flaky-table-removed-postgres
test(postgres bridge): attempt to stabilize flaky test
2024-07-04 09:25:15 -03:00
JimMoen 8c6cd69caa
fix: obtain cert expiry epoch failed due to formated `generalTime` 2024-07-04 18:23:56 +08:00
firest 913e0ce18b feat(banned): add a bootstrap file for banned 2024-07-04 09:12:12 +08:00
zmstone 947cddb2eb test: invalid map key is caught by hocon
now that hocon has a built-in map key validation,
some of the resource name validations are cought by hocon
2024-07-03 23:00:18 +02:00
zmstone eaaee725c2 fix: upgrade to hocon 0.43.1
included 3 changes since 0.42.2

- allow validation of map keys
- improve crash stacktrace report
- avoid dumping array environment variable values
2024-07-03 23:00:14 +02:00
zmstone 7ee5b90084
Merge pull request #13400 from zmstone/0605-ACL-rules-in-http-authentication-response
feat(auth): support HTTP authn return ACL rules
2024-07-03 21:51:07 +02:00
Thales Macedo Garitezi 32ace85e1c Merge branch 'release-57' into 20240703-m-sync-r57 2024-07-03 16:03:15 -03:00
Thales Macedo Garitezi ae24b7a37b
Merge pull request #13404 from thalesmg/20240703-r57-refactor-mix-common-deps
ci(mix): refactor to avoid conflicts
2024-07-03 15:57:05 -03:00
Thales Macedo Garitezi 72579f9014 test(postgres bridge): attempt to stabilize flaky test
```
%%% emqx_bridge_pgsql_SUITE ==> tcp.sync.with_batch.t_table_removed: FAILED
%%% emqx_bridge_pgsql_SUITE ==> {{panic,
     #{msg => "Unexpected result",
       result =>
           {run_stage_failed,exit,
               {test_case_failed,
                   "unexpected result: {error,{recoverable_error,sync_required}}"},
               [{emqx_bridge_pgsql_SUITE,'-t_table_removed/1-fun-3-',3,
                    [{file,
                         "/emqx/apps/emqx_bridge_pgsql/test/emqx_bridge_pgsql_SUITE.erl"},
                     {line,822}]},
```

```
Error: -03T17:52:54.046809+00:00 [error] Generic server <0.352770.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,{state,ssl,{sslsocket,{gen_tcp,#Port<0.1671>,tls_connection,undefined},[<0.352774.0>,<0.352773.0>]},<<>>,{336,-2093820527},on_message,{codec,#{},[null,undefined],{oid_db,#{16 =>
...
2024-07-03T17:52:54.075446+00:00 [critical] Run stage failed: exit:{test_case_failed,"unexpected result: {error,\n                    {resource_error,\n                     #{reason => exception,\n                       msg =>\n                        #{error =>\n                           {exit,\n                            {{undef,\n                              [{undefined,handle_message,\n                                [90,<<\"I\">>,\n                                 {state,ssl,\n                                  {sslsocket,\n                                   {gen_tcp,#Port<0.1671>,tls_connection,\n
```
2024-07-03 15:05:00 -03:00
Thales Macedo Garitezi afb65817c4 ci(mix): refactor to avoid conflicts
This introduces a refactoring that will hopefully help avoid conflicts with ongoing work
to migrate our build / CI tooling to Mix.
2024-07-03 14:37:25 -03:00
Andrew Mayorov 187f5e5936
Merge pull request #13391 from keynslug/perf/EMQX-12611/avoid-stream-shuffle
perf(sessds): rotate through streams with iterators when fetching
2024-07-03 17:46:29 +02:00
Andrew Mayorov d7d5eb2c52
test(bridge-s3): correct assertions to get rid of flakiness 2024-07-03 16:40:34 +02:00
Thales Macedo Garitezi 532c7831b2
Merge pull request #13384 from thalesmg/20240701-r57-mt-new-fields
feat(message transformation): add more read-only fields to context
2024-07-03 11:27:27 -03:00
Thales Macedo Garitezi 35f1ddc0eb
Merge pull request #13393 from thalesmg/20240702-r57-test-plugin-start-enabled
fix(plugins): ensure plugin apps are restarted when restarting `emqx_plugins`
2024-07-03 11:27:14 -03:00
zmstone 9194756963 feat(auth): support HTTP authn return ACL rules 2024-07-03 15:37:11 +02:00
Andrew Mayorov 950f4d9483
fix(sessds): defend restartable stream iterator from infinite loop 2024-07-03 15:21:16 +02:00