Commit Graph

12874 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 8d535bbd24
Merge pull request #13464 from thalesmg/20240712-m-res-manager-shutdown-logs
chore: attempt to reduce race condition supervisor noproc shutdown error logs
2024-07-19 14:57:56 -03:00
Thales Macedo Garitezi 4d174b8678 feat(sources & actions api): add dependent rules to response
Fixes https://emqx.atlassian.net/browse/EMQX-12654
2024-07-19 14:43:55 -03:00
Thales Macedo Garitezi eb2d3a3b7e chore: attempt to reduce race condition supervisor shutdown errors
Fixes https://emqx.atlassian.net/browse/EMQX-12442

e.g.:
```
2024-05-23T08:52:39.811845+00:00 [error] Supervisor: {local,emqx_resource_manager_sup}. Context: shutdown_error. Reason: noproc. Offender: id=<<99, 101, 110, 115, 111, 114, 101, 100>>,pid=<0.7752.1030>.
```

It could be just a race condition, as it seems to be the case for resource manager: i) a call is made to the process to stop it; ii) the call times out; iii) the after clause ends up calling supervisor:terminate_child; iv) while the supervisor is finding the child to terminate, the process actually finishes terminating, and the supervisor receives a noproc reason back.
2024-07-19 10:57:00 -03:00
Thales Macedo Garitezi ae828e8cfb feat(connectors api): add dependent actions and sources to response
Fixes https://emqx.atlassian.net/browse/EMQX-12654
2024-07-19 10:33:48 -03:00
Thales Macedo Garitezi 464e202742
Merge pull request #13488 from thalesmg/20240718-m-mix-fix-machine-dep
chore(new mix build): fix app dependency for release
2024-07-19 09:21:02 -03:00
Andrew Mayorov b7200656a5
Merge pull request #13486 from keynslug/fix/ci/ds-raft-flaky-next
test(dsraft): attempt to stabilize flaky testcases
2024-07-19 12:15:36 +02:00
Shawn fc3405fe4c fix: bp_api for relup 2024-07-19 17:35:36 +08:00
Shawn f11dfce292 ci: suppress dialyzer checks for quicer and odbc types 2024-07-19 17:11:49 +08:00
Shawn c61828460a chore: emqx_utils_api:with_node/2 support simple http-code 2024-07-19 15:02:14 +08:00
Shawn 4d25f28bb2 fix: dialyzer checks 2024-07-19 14:19:50 +08:00
Shawn 3ad7dc262b fix: some sanity-checks 2024-07-19 11:38:44 +08:00
Shawn 3c8ef35b18 fix: show relup status even if no packages installed 2024-07-19 11:20:37 +08:00
Thales Macedo Garitezi 01883e9759 chore(new mix build): fix app dependency for release 2024-07-18 11:33:02 -03:00
Thales Macedo Garitezi ca47e4768d
Merge pull request #13485 from thalesmg/20240717-r57-rm-dead-code-purge-proto
chore: remove dead code
2024-07-18 09:22:27 -03:00
Andrew Mayorov 3a893626b8
Merge pull request #13474 from keynslug/ft/EMQX-12309/ds-cas-api
feat(ds): allow isolated batches with preconditions
2024-07-18 12:52:13 +02:00
Shawn 2008130071 feat: add HTTP APIs for relup 2024-07-18 18:48:38 +08:00
Ivan Dyachkov 4a04ffdca1
Merge pull request #13483 from id/20240717-sync-release-57
sync release-57
2024-07-18 10:41:49 +02:00
Ivan Dyachkov c2d49ff34f chore(rmq_tests): fix rabbitmq tests
Co-authored-by: Ilya Averyanov <av@rubybox.dev>
2024-07-18 08:34:00 +02:00
zhongwencool ac52bf39ce
Merge pull request #13443 from zhongwencool/cluster-link-cli-load
fix: update cluster.links via cli
2024-07-18 09:13:59 +08:00
Andrew Mayorov 0e545ffcec
feat(ds): add dedicated `#message_matcher{}` for preconditions 2024-07-17 21:27:17 +02:00
Andrew Mayorov 2e89656a90
test(dsraft): start `t_replication_transfers_snapshots` from stable state 2024-07-17 20:01:55 +02:00
Andrew Mayorov 466fa41ec3
fix(dsraft): rely on last resort timeout with unresponsive replicas
This simplifies the shard transition scheduling logic and makes it less
prone to races.
2024-07-17 19:24:38 +02:00
Thales Macedo Garitezi 93c725732c chore: remove dead code 2024-07-17 12:56:45 -03:00
Thales Macedo Garitezi 4edbcc55e7
Merge pull request #13463 from thalesmg/20240712-m-gprodu-backoff-retry
feat(gcp pubsub producer): retry on 502 and 503 http status code responses
2024-07-17 12:49:27 -03:00
Thales Macedo Garitezi cd8bf2725a
Merge pull request #13453 from thalesmg/20240711-r57-mt-fixes
batch of message transformation fixes
2024-07-17 12:45:19 -03:00
Andrew Mayorov 0c05b3f019
fix(ds): make conditionals less confusing 2024-07-17 16:23:41 +02:00
Andrew Mayorov 78fe9304be
Merge pull request #13462 from keynslug/fix/ci/flaky-ds-raft
fix(dsraft): preserve pending replica set transitions
2024-07-17 16:11:59 +02:00
Andrew Mayorov 14022aded1
feat(ds): allow isolated batches with preconditions
Namely, single message deletions and preconditions that can be used to
build complex "compare-and-swap"-style operations. Also allow user to
declare that atomic batches support is needed for a DB.
2024-07-17 15:57:17 +02:00
Andrew Mayorov 02e1007a16
feat(dslocal): implement `force_monotonic_timestamps => false` 2024-07-17 15:49:50 +02:00
zhongwencool 3381eecd6f chore: apply code review 2024-07-17 21:27:07 +08:00
zhongwencool 3c832db13d test: test cluster.links reloaded 2024-07-17 21:06:48 +08:00
zhongwencool 937fb153c2 fix: fill_default/3 should populate default values for all parameters except the 'ds' 2024-07-17 21:06:48 +08:00
zhongwencool 7b6b9580c8 test: add test for updating cluster.links 2024-07-17 21:06:48 +08:00
zhongwencool 2783192f77 fix: update cluster.links via cli 2024-07-17 21:06:48 +08:00
zhongwencool 083537daa3 fix: retry not_found if conf file not exist 2024-07-17 21:06:48 +08:00
Andrew Mayorov ae3812da85
feat(ds): allow to turn monotonic timestamps off for DB
That tells implementation how to assign timestamps to messages. Current
implicit default is now `force_monotonic_timestamps => true`.
2024-07-17 14:40:23 +02:00
Andrew Mayorov 6b130c6422
fix(dsraft): preserve pending replica set transitions
Otherwise, information about pending replica set transitions taking a
long time to complete could be lost on subsequent target set changes and
node crashes.
2024-07-17 12:17:07 +02:00
Ivan Dyachkov 292b331064 Merge remote-tracking branch 'upstream/release-57' into 20240717-sync-release-57 2024-07-17 11:29:25 +02:00
zhongwencool 52031441cf chore: add authz tag to match_rule_error log 2024-07-17 12:21:32 +08:00
zhongwencool 604cff4887 feat: add rule tag to rule_engine log 2024-07-17 12:15:57 +08:00
lafirest 6697035812
Merge pull request #13441 from lafirest/feat/coap
feat(coap): use content-sensitive udp proxy for coap
2024-07-17 10:01:51 +08:00
Thales Macedo Garitezi 1ad02a11e2
Merge pull request #13455 from thalesmg/20240711-m-mix-umbrella-part-III-no-ci
sync new mix build work to master
2024-07-16 14:41:10 -03:00
Andrew Mayorov d04915d6a6
test(dsraft): increase `ra_server` logging level for debugging 2024-07-16 15:54:49 +02:00
Andrew Mayorov 78bb102311
test(dsraft): attempt to start select testcases from stable state 2024-07-16 15:54:49 +02:00
Thales Macedo Garitezi 4a08bfc93f feat(mix ct): improve failure logging 2024-07-15 09:26:04 -03:00
Thales Macedo Garitezi 02a0ccfdd1 ci: preparations for new mix build 2024-07-15 09:26:04 -03:00
Thales Macedo Garitezi 70786d6aca test: fix suite apps 2024-07-15 09:26:04 -03:00
Thales Macedo Garitezi 066fd0481b feat(mix): compile asn1 files 2024-07-15 09:26:04 -03:00
Thales Macedo Garitezi 9e4a84cf76
Merge pull request #13442 from thalesmg/20240709-r57-decouple-connector-action-hc
fix(resource manager): disentangle connector and channel health check frequencies
2024-07-15 09:12:07 -03:00
firest ec183f1d4c test(coap): fix ci errors 2024-07-15 10:52:37 +08:00
ieQu1 46c2c75b7b
fix(connection): Make process_msg function tail-recursive 2024-07-14 06:00:00 +02:00
Thales Macedo Garitezi 0e57b39cf2 feat(gcp pubsub producer): retry on 502 and 503 http status code responses
Fixes https://emqx.atlassian.net/browse/EMQX-12625
2024-07-12 15:29:59 -03:00
Andrew Mayorov 2401a2fb80
test(dsraft): run `t_join_leave_errors` case in tracing context 2024-07-12 18:28:24 +02:00
Thales Macedo Garitezi 96c9020727 chore: improve protobuf decoding error messages
Fixes https://emqx.atlassian.net/browse/EMQX-12677
2024-07-12 13:27:30 -03:00
Andrew Mayorov af81800aec
chore(dsraft): log a bit more informative messages in shard allocator 2024-07-12 18:24:58 +02:00
Andrew Mayorov 8e8b382ec0
chore(dsraft): provide more details when replica is unready 2024-07-12 18:23:23 +02:00
Andrew Mayorov 70a760850f
chore(dsraft): correct comment spelling errors 2024-07-12 15:27:29 +02:00
Andrew Mayorov 205ad507ea
test(dsraft): attempt to ensure testcases start from stable state
Where "stable state" is currently defined as "everyone knows and agrees
on the current leader".
2024-07-12 15:26:00 +02:00
Ilya Averyanov 9ca8aeb155 chore(mgmt): reduce test flakyness 2024-07-12 12:10:49 +03:00
firest 854754eb60 feat(coap): use content-sensitive udp proxy for coap 2024-07-12 16:23:46 +08:00
firest 83cc3ffeb0 fix(banned): let the bootfile of banned be optional 2024-07-12 13:58:14 +08:00
lafirest 1b7d23cef4
Merge pull request #13451 from lafirest/fix/def_banned_file
fix: do not convert a empty file name to a empty list
2024-07-12 13:50:32 +08:00
Thales Macedo Garitezi 2816170e9d chore: add `$events.message_transformation_failed` to rule engine tester
Fixes https://emqx.atlassian.net/browse/EMQX-12679
2024-07-11 17:52:08 -03:00
Thales Macedo Garitezi 5f595966d8 chore(message transformation): allow empty operation list
Fixes https://emqx.atlassian.net/browse/EMQX-12682
2024-07-11 17:52:08 -03:00
Thales Macedo Garitezi 04b547d6f5 fix(schema registry): handle large names during lookup
Fixes https://emqx.atlassian.net/browse/EMQX-12692
2024-07-11 14:35:31 -03:00
Thales Macedo Garitezi f3c6d10f76 fix(mix): fix compile paths and deps 2024-07-11 14:19:23 -03:00
Thales Macedo Garitezi 01d89be743 feat(message transformation): add timestamp and pub_props fields to read context
Fixes https://emqx.atlassian.net/browse/EMQX-12684

Fixes https://emqx.atlassian.net/browse/EMQX-12678
2024-07-11 12:15:31 -03:00
Thales Macedo Garitezi 44e4b3616d feat(variform): allow hyphens in identifiers
Fixes https://emqx.atlassian.net/browse/EMQX-12683
2024-07-11 12:15:23 -03:00
Kjell Winblad a4cc3ba9e8
Merge pull request #13375 from kjellwinblad/kjell/fix_connector_lister_speed_limit_clearing/EMQX-12514
fix: default value for max_conn_rate etc should be set to infinity
2024-07-11 16:36:01 +02:00
firest d9b5c5863b fix: do not convert a empty file name to a empty list 2024-07-11 18:12:38 +08:00
lafirest c9e12f30cd
Merge pull request #13444 from lafirest/fix/oidc
fix(oidc): Avoid crashes and avoid deleting jwks on update
2024-07-11 17:55:50 +08:00
zmstone 7664b06e98
Merge pull request #13434 from zmstone/0704-refine-rpc-config
0704 refine rpc config
2024-07-11 10:25:45 +02:00
ieQu1 4825079964
chore(ds): Support platform profile 2024-07-10 12:03:23 +02:00
firest b0e3e405cf fix(oidc): Avoid crashes and avoid deleting jwks on update 2024-07-10 15:22:43 +08:00
JimMoen 44d533fe6d
Merge pull request #13432 from JimMoen/0705-fix-jwt-pem-check
fix: create authn jwt with bad public key
2024-07-10 10:33:54 +08:00
zmstone 7a23ae7b4d refactor: expose only server_port for rpc
previously, there were 4 port configs:
- tcp_server_port
- ssl_server_port
- tcp_client_port
- ssl_client_port
2024-07-09 22:26:29 +02:00
zmstone ee13773496 refactor: rename rpc.tcp_client_num to rpc.client_num
tcp_client_num is kept as an alias
2024-07-09 22:26:29 +02:00
Thales Macedo Garitezi 48e604bda8 fix(mix grpc): include default erlc options 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 818070ad44 test(mix): add integration test path 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 5279ad76be fix(grpc compiler): unload apps to avoid side effects 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi b91515b131 fix(schema registry mix): gpb is a runtime dep 2024-07-09 15:56:30 -03:00
Thales Macedo Garitezi 6d94809950
Merge pull request #13415 from thalesmg/20240703-m-couchbase-action
feat: implement couchbase connector and action
2024-07-09 15:53:11 -03:00
Thales Macedo Garitezi 3c370a90aa
Merge pull request #13436 from thalesmg/20240708-r57-custom-headers-jwks
feat(jwks): allow specifying custom request headers
2024-07-09 15:52:44 -03:00
Thales Macedo Garitezi 9f8a1885a7 fix(resource manager): disentangle connector and channel health check frequencies
Fixes https://emqx.atlassian.net/browse/EMQX-12674
2024-07-09 14:53:39 -03:00
Thales Macedo Garitezi d25c4ba06f
Merge pull request #13421 from thalesmg/20240705-r57-docs-mt-api-examples
docs(message transformation): add api examples
2024-07-09 10:16:53 -03:00
ieQu1 3721be65ee
fix(ds): Improve comments 2024-07-09 13:15:15 +02:00
ieQu1 d7732a6aac
test(ds): Attempt to stabilize a flaky test 2024-07-09 13:15:15 +02:00
ieQu1 e70c1cfea3
test(ds): Improve stability of replication test suite 2024-07-09 13:15:15 +02:00
ieQu1 dc4ae82798
test(ds): Add message ID 2024-07-09 13:15:14 +02:00
ieQu1 d1b574a67e
perf(dslocal): Run heavy-duty operations in a temporary process 2024-07-09 13:15:14 +02:00
ieQu1 661f79544b
fix(ds): Optimize hot loop of skipstream storage layout 2024-07-09 13:15:14 +02:00
ieQu1 23dafbb03b
feat(ds): Add a benchmarking tool for storage efficiency analysis 2024-07-09 13:15:14 +02:00
ieQu1 afeb2ab8aa
feat(ds): Add metrics for skipstream layout 2024-07-09 13:15:14 +02:00
ieQu1 b68ebb9a73
test(dsrepl): Generalize tests to use different storage layouts 2024-07-09 13:15:14 +02:00
ieQu1 8c5e4a2376
test(ds): Generalize storage layout test suite for different layouts 2024-07-09 13:15:14 +02:00
ieQu1 086e7256f5
feat(ds): Add configuration schema for skipstream LTS layout 2024-07-09 13:15:14 +02:00
ieQu1 a4642d4d06
feat(ds): Add a new storage layout engine: "skipstream"
This layout is based on LTS as well, but it uses separate index
streams for constrained replay of streams with learned wildcards
2024-07-09 13:15:14 +02:00
ieQu1 de48077ac4
test(ds): Add new helper functions
- Improve message comparison
- Add set operations
2024-07-09 13:15:14 +02:00
ieQu1 210556e545
feat(ds): Generalize value serialization
- Add a new utility module for message serialization
- Add experimental serializer based on ASN.1
2024-07-09 13:15:14 +02:00
ieQu1 843973ef32
fix(ds): bitfield_lts: static_key_size -> static_key_bits 2024-07-09 13:15:14 +02:00