91ffc95f29
Merge pull request #12855 from JimMoen/fix-share-queue-format
...
fix(mgmt): $queue shared topics format in mgmt topics api
2024-04-10 10:52:48 +08:00
5f8986209e
Merge pull request #12842 from lafirest/fix/iotdb
...
fix(iotdb): correctly handle undefined value of bool type
2024-04-10 09:49:16 +08:00
fae9005f87
test(emqx_mgmt_cli): test that replicants do not join a left core node
2024-04-09 19:06:13 +03:00
55179ccfed
chore: update ekka to 0.19.3
...
Included updates:
- https://github.com/emqx/mria/pull/178
2024-04-09 19:04:38 +03:00
31142df5cf
fix: default value of stop_action_after_template_rendering to true
...
This commit changes the default value for the
stop_action_after_template_rendering option of the apply rule HTTP API
endpoint so that it is true instead of false.
2024-04-09 16:20:37 +02:00
7bcd553786
test: move test case with rule_engine dep from emqx to emqx_rule_engine
2024-04-09 14:08:46 +02:00
3898950017
test: fix test by adding dependency
2024-04-09 11:38:35 +02:00
958748cf7f
test: fix inter test suite problem
2024-04-09 11:10:39 +02:00
03a9c46ca7
fix(sys_topic): format shared topics
2024-04-09 17:06:36 +08:00
47e0f3bb1f
fix(mgmt): $queue shared topics format in mgmt topics api
2024-04-09 17:06:32 +08:00
8bbfa9ca8b
Merge pull request #12839 from keynslug/feat/EMQX-12110/rebalancing-impl
...
feat(dsrepl): implement membership changes and rebalancing
2024-04-09 10:46:31 +02:00
53b78086ed
chore: fix xref checks
2024-04-09 09:34:05 +02:00
8381132919
chore: bump emqx_conf to 0.1.36
2024-04-09 14:44:11 +08:00
a1495689c0
fix: clean self node's cluster commit when leave cluster
2024-04-09 14:13:25 +08:00
6cb00cc8c7
Merge pull request #12844 from JimMoen/EMQX-12012/cpu-use-idle-two-decimal-places
...
fix: cpu usage and idle use two decimal places
2024-04-09 10:11:15 +08:00
d12e907209
fix(dsrepl): correctly handle ra membership change command results
...
Before this change, results similar to `{error, {no_more_servers_to_try,
[{error, nodedown}, {error, not_member}]}}` were considered retryable
failures, which is incorrect.
2024-04-08 22:44:34 +02:00
3223797ae5
fix(dsrepl): attempt leadership transfer before server removal
...
This should make it much less likely to hit weird edge cases that lead
to duplicate Raft log entries because of client retries upon receiving
`shutdown` from the leader being removed.
2024-04-08 22:43:58 +02:00
1e95bd4da6
test(dsrepl): test unresponsive nodes removal / node restarts
2024-04-08 21:27:56 +02:00
41677eb785
refactor: make elvis happy
2024-04-08 21:25:58 +02:00
bf12efac6d
fix(variform): add basic tests
2024-04-08 21:08:43 +02:00
7a836317ac
fix(dsrepl): trigger unfinished shard transition upon startup
...
Also provide a trivial API to trigger them by hand.
2024-04-08 16:12:42 +02:00
75bb7f5cdc
fix(dsrepl): retry only `{add, Site}` crashed membership transitions
...
To minimize the potential negative impact of removal transitions that
crash for some unknown and unusual reasons.
2024-04-08 16:04:33 +02:00
9628a00a82
docs(emqx_rule_api apply rule): fix doc strings
2024-04-08 15:34:29 +02:00
ba96edb061
fix(clients api): use alternative base64 function for OTP 25
...
Fixes https://github.com/emqx/emqx/pull/12798#discussion_r1555524603
2024-04-08 10:23:10 -03:00
600526a0e4
test(emqx_bridge_http_SUITE): test case after name change
2024-04-08 14:54:33 +02:00
02ee873094
docs(emqx_rule_api_schema): fix type spec
2024-04-08 13:42:15 +02:00
4c0cc079c2
fix(dsrepl): apply unnecessary rebalancing transitions cleanly
2024-04-08 13:25:45 +02:00
dcde30c38a
test(dsrepl): add two more testcases for rebalancing
2024-04-08 13:22:31 +02:00
b57725f996
test(resource_SUITE): do test case fixes needed due to rule tracing work
2024-04-08 12:41:35 +02:00
79440064fe
style: fix problems reported by elvis
2024-04-08 11:03:55 +02:00
e11c4a9c83
Merge pull request #12826 from emqx/import-source-bridges
...
fix: source bridges missing after restore the backup files
2024-04-08 16:02:15 +08:00
3e87d4bf9f
Merge pull request #12772 from killme2008/feature/upgrade-greptimedb-ingester-ci
...
feat: update greptimedb client lib and ci version
2024-04-08 09:02:03 +02:00
282cbb18be
fix: cpu usage and idle use two decimal places
...
- prometheus
- opentelemetry
2024-04-08 14:14:09 +08:00
2ace9bb893
chore(dsrepl): sprinkle few comments and typespecs for exports
2024-04-07 22:51:56 +02:00
ecaad348a7
chore(dsrepl): update few outdated comments / TODOs
2024-04-07 22:51:56 +02:00
6293efb995
fix(dsrepl): retry crashed membership transitions
2024-04-07 22:51:56 +02:00
826ce5806d
fix(dsrepl): ensure that new member UID matches server's UID
...
Before that change, UIDs supplied in the `ra:add_member/3` were not
the same as those servers were using. This haven't caused any issues
for some reason, but it's better to ensure that UIDs are the same.
2024-04-07 22:31:24 +02:00
e89dc32c90
ci: run emqx_management both with ee and ce profile
2024-04-07 18:33:52 +08:00
f40e47365c
fix(iotdb): correctly handle undefined value of bool type
2024-04-07 17:26:15 +08:00
1c81c79a2c
chore: add testcase for importing retained msgs and sources
2024-04-07 17:24:26 +08:00
9d1a69aaa9
fix: cannot import retained messages
2024-04-07 17:03:14 +08:00
5479932190
feat(apply rule test): make option to stop action after render work
...
This commit makes the apply rule HTTP API option to stop an action work
for the HTTP action, and adds infrastructure that makes it easy to add
this functionality to other actions.
2024-04-06 17:21:12 +02:00
ef705c2285
feat: add apply rule API, clientid/ruleid tracing for rule and connector
...
This commit adds:
* Support for forwarding the rule id and client id to the connector so
that events such as template rendered successfully can be traced.
* HTTP API for for applying/activating a rule with the given context
2024-04-06 17:20:47 +02:00
04ba2aaf8a
Merge pull request #12830 from thalesmg/async-channel-hc-m-20240404
...
feat(resource): non-blocking channel health checks
2024-04-05 14:24:09 -03:00
556ffc78c9
feat(dsrepl): implement membership changes and rebalancing
2024-04-05 18:57:28 +02:00
d6058b7f51
feat(dsrepl): allow to subscribe to DB metadata changes
...
Currently, only shard metadata changes are announced to the
subscribers.
2024-04-05 17:40:55 +02:00
a07295d3bc
fix(ds): address shards in the supervisor properly
2024-04-05 17:40:38 +02:00
3f7b14c861
Merge pull request #12833 from ieQu1/dev/ds-cluster-api
...
feat(ds): Add REST API for durable storage
2024-04-05 17:33:20 +02:00
2504b8126b
feat(ds): Pass mgmt_ds REST API calls to the application
2024-04-05 15:22:06 +02:00
46261440cb
feat(ds): Add a CLI for managing DB replicas
2024-04-05 15:22:06 +02:00
a62db08676
feat(ds): Add REST API for durable storage
2024-04-05 15:22:06 +02:00
d09787d1a6
fix(ds): Fix return types in replication_layer_meta
2024-04-05 15:22:06 +02:00
be47fe49ad
chore: bump ecql version to 0.7.0
...
PR: https://github.com/emqx/ecql/pull/13
No functional changes, just switch gen_fsm to gen_statem.
2024-04-05 13:31:33 +02:00
70396e9766
Merge pull request #12825 from keynslug/feat/EMQX-12110/repl-meta-api
...
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 22:32:03 +02:00
df6c5b35fe
feat(dsrepl): add more primitive operations to modify DB sites
2024-04-04 21:22:49 +02:00
bb8ffee18c
feat(dsrepl): add API to get current DB replication sites
2024-04-04 21:22:02 +02:00
ad52f7838e
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 21:22:01 +02:00
60cad74286
feat(resource): non-blocking channel health checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-12015
Continuation of https://github.com/emqx/emqx/pull/12812
2024-04-04 16:13:30 -03:00
8d58b40f33
Merge pull request #12831 from thalesmg/ds-checkpoint-clean-m-20240404
...
feat(ds): clear all checkpoints when (re)starting storage layer
2024-04-04 15:41:50 -03:00
217b35bce5
Merge pull request #12798 from thalesmg/ds-client-api-v2-m-20240327
...
feat(client mgmt api): add cursor-based list API
2024-04-04 15:10:49 -03:00
c57c36adb2
feat(ds): clear all checkpoints when (re)starting storage layer
...
Fixes https://emqx.atlassian.net/browse/EMQX-12143
2024-04-04 14:05:52 -03:00
59a442cdb5
feat(rule trace): add support for ruleid as a trace type
2024-04-04 14:55:32 +02:00
069cd4fbb4
Merge pull request #12812 from thalesmg/async-res-manager-health-check-m-20240328
...
feat(resource manager): perform non-blocking health checks
2024-04-04 09:07:02 -03:00
0e79b543cf
refactor: move variform to emqx_utils
2024-04-04 11:10:56 +02:00
f37ed3a40a
fix(ds): Limit the number of retries in egress to 0
2024-04-03 16:38:49 +02:00
319ec50c0d
fix: source bridges missing after restore the backup files
2024-04-03 18:26:51 +08:00
2bbfada7af
fix(ds): Make async batches truly async
2024-04-03 11:57:47 +02:00
92ca90c0ca
fix(ds): Improve egress logging
2024-04-03 11:57:47 +02:00
ae5935e7f7
test(ds): Attempt to stabilize metrics_worker tests in CI
2024-04-02 19:14:10 +02:00
4382971443
fix(ds): Preserve errors in the egress
2024-04-02 16:47:43 +02:00
94ca7ad0f8
feat(ds): Report counters for LTS storage layout
2024-04-02 16:47:43 +02:00
f14c253dea
fix(prometheus): Don't add DS metrics when feature is disabled
2024-04-02 16:47:43 +02:00
b379f331de
fix(sessds): Handle errors when storing messages
2024-04-02 16:47:41 +02:00
f41e538526
feat(sessds): Observe next time
2024-04-02 16:45:52 +02:00
b9ad241658
feat(sessds): Add metrics for the number of persisted messages
2024-04-02 16:45:52 +02:00
75b092bf0e
fix(ds): Actually retry sending batch
2024-04-02 16:45:49 +02:00
0de255cac8
feat(ds): Report egress flush time
2024-04-02 16:25:04 +02:00
044f3d4ef5
fix(ds): Don't reverse entries in the atomic batch
2024-04-02 16:25:04 +02:00
606f2a88cd
feat(ds): Add egress metrics
2024-04-02 16:25:04 +02:00
c9de336234
feat(ds): Add metrics worker to the builtin db supervision tree
2024-04-02 16:25:04 +02:00
d8204021dc
refactor(metrics): Move metrics worker to emqx_utils application
2024-04-02 16:25:04 +02:00
2097e854fc
feat(client mgmt api): add cursor-based list API
...
Fixes https://emqx.atlassian.net/browse/EMQX-12028
2024-04-02 10:55:28 -03:00
778e897f1f
chore(dsrepl): describe snapshot ownership and few shortcomings
2024-04-02 13:48:51 +02:00
c666c65c6a
test(ds): factor out storage iteration into helper module
2024-04-02 13:48:51 +02:00
7cebf598a8
chore(dsrepl): simplify snapshot transfer code a bit
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-04-02 13:48:51 +02:00
e029b8f996
test(dsrepl): wait for whole cluster readiness
...
To minimize the chance of flaky tests due to the shards not being
completely online.
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-04-02 13:48:50 +02:00
e8b06a6a9f
chore(dsrepl): mark few more BPAPI targets as obsolete
2024-04-02 13:48:50 +02:00
d31cd0c728
feat(ds): ensure LTS state ids are deterministic
2024-04-02 13:48:50 +02:00
2cd357a5bd
fix(ds): ensure store batch is idempotent wrt generations
2024-04-02 13:48:50 +02:00
77a022bd93
feat(dsrepl): transfer storage snapshot during ra snapshot recovery
2024-04-02 13:48:49 +02:00
b8b9b7739b
chore(ds): slightly simplify working with storage generations
2024-04-02 13:48:08 +02:00
2d074df209
Merge pull request #12797 from keynslug/fix/dsrepl-error-handling
...
fix(dsrepl): handle RPC errors gracefully when storage is down
2024-04-02 13:40:31 +02:00
5759ba5162
chore: bump app version
2024-04-02 17:09:22 +08:00
50bceee9ab
fix(stats): `'subscribers.count'` contains shared-subscriber
2024-04-02 16:56:40 +08:00
0f4b148294
refactor: uniform shared_sub table macros
2024-04-02 16:56:39 +08:00
1a4cfc2a2d
fix(api_schema): removed metrics schema in api spec
...
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622 ).
2024-04-02 16:56:36 +08:00
bade09b56e
feat(resource manager): perform non-blocking resource health checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-12015
This introduces only _resource_ non-blocking health checks. _Channel_ non-blocking health
checks may be introduced later.
2024-04-01 14:46:15 -03:00
c62410ff75
refactor: remove already bound variable
2024-04-01 17:03:50 +03:00
ceb04ba06d
fix(emqx_mgmt): do not attempt to get a stacktrace of a remote client connection process
2024-04-01 16:42:12 +03:00
42af1f9d63
fix: handle internal timeout errors in client Mqueue/Inflight APIs
2024-03-29 23:03:35 +02:00
f5a820cb10
fix(emqx_mgmt): catch OOM shutdown exits properly when calling a client conn process
...
The exit reason is expected to include gen_server `Location`:
`{{shutdown, OOMInfo}, Location}`.
2024-03-29 13:09:08 +02:00
bfca3ebc71
feat(variform): support array syntax '[' and ']'
2024-03-28 19:34:57 +01:00
5f26e4ed5e
feat(variform): implement variform engine
2024-03-28 19:34:57 +01:00
2e528d1dd8
Merge pull request #12802 from SergeTupchiy/EMQX-11826-prevent-left-node-from-rejoining-5.6.1
...
prevent a left node from rejoining the same cluster
2024-03-28 19:49:18 +02:00
ad95473aae
refactor: move string functions to emqx_variform
2024-03-28 18:03:37 +01:00
3eda182e9a
fix: prevent a node from discovering and re-joining the same cluster after it has (manually) left it.
2024-03-28 18:09:27 +02:00
9bf65a415b
feat(variform): add a variable transformer
2024-03-28 16:11:26 +01:00
35c43eb8a0
feat(sessds): handle recoverable errors in stream scheduler
2024-03-28 15:17:01 +01:00
fa66a640c3
fix(dsrepl): handle RPC errors gracefully when storage is down
2024-03-28 15:17:01 +01:00
63c017a72f
Merge pull request #12804 from SergeTupchiy/EMQX-12058-improve-force-shutdown-error-reason-5.6.1
...
chore: rename `message_queue_too_long` error reason to `mailbox_overflow` (5.6.1 port)
2024-03-28 16:14:51 +02:00
8fb4ef9fe3
test: fix flaky test
2024-03-28 10:53:44 -03:00
93c87fcb25
Merge pull request #12803 from SergeTupchiy/EMQX-11808-remove-uploaded-invalid-backups-5.6.1
...
fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid (5.6.1 port)
2024-03-28 15:37:10 +02:00
9b4d960888
chore: prepare for 5.7.x releases
2024-03-28 13:52:48 +01:00
04bf763890
fix(kafka-based bridges): avoid trying to get raw config for replayq dir
...
Fixes https://emqx.atlassian.net/browse/EMQX-12049
2024-03-28 09:13:34 -03:00
b1855f95c1
fix(bpapi): Add exceptions for experimental features
2024-03-28 12:07:45 +01:00
8c6d8bdd12
fix(bpapi): Add exceptions for experimental features
2024-03-28 12:03:41 +01:00
906a77d167
chore: rename `message_queue_too_long` error reason to `mailbox_overflow`
...
`mailbox_overflow` is consistent with the corresponding config parameter:
'force_shutdown.max_mailbox_size'
2024-03-28 12:56:00 +02:00
1d1f595e6f
fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid
2024-03-28 12:54:51 +02:00
02b72d79fc
fix(bpapi): Fix comparison of releases during compatibility check
2024-03-28 11:21:30 +01:00
db9efb9317
chore: bump apps versions
2024-03-28 10:19:09 +01:00
f4446ec680
Merge remote-tracking branch 'upstream/release-56' into 0328-sync-release-56
2024-03-28 09:59:54 +01:00
ea917d6b2b
Merge pull request #12795 from id/0327-cut-5.6.0
...
chore: 5.6.0
2024-03-28 09:30:08 +01:00
11bc1aeea1
fix: update hstreamdb driver
2024-03-28 10:30:48 +03:00
cf29f97440
revert: "test: comment out hstreamdb:stop_client"
...
This reverts commits:
3536b32ad2b126c27a1b0bd91e615069230931d7..
d63f10928d
2024-03-28 14:32:18 +08:00
34b5cd7bfd
chore: reuse hstream_erl `0.5.18+v0.18.1`
...
- change ezstd dep vsn for bulid
2024-03-28 14:31:10 +08:00
d63f10928d
test: comment out hstreamdb:stop_client
2024-03-28 13:25:01 +08:00
3536b32ad2
ci: print the failed stacktrace
2024-03-28 11:17:26 +08:00
d7917d05c6
chore: upgrade hstreamdb to 0.5.22+v0.18.1
2024-03-28 10:11:53 +08:00
a279a399db
ci: Override ezstd dependency
2024-03-27 20:54:46 +01:00
4b3c29ded8
fix(bpapi): Add 5.5 dump
2024-03-27 17:39:38 +01:00
beb9152d50
Merge pull request #12750 from zmstone/0316-introduce-client-attrs
...
0316 introduce client_attrs
2024-03-27 16:19:47 +01:00
5ed20e15cd
fix(bpapi): Add 5.5 bpapi file
2024-03-27 13:45:00 +01:00
df0c67fe50
Merge pull request #12787 from thalesmg/mv-api-examples-m-20240326
...
docs(message validation): add swagger API examples
2024-03-27 09:06:58 -03:00
33d9619271
chore: 5.6.0
2024-03-27 11:17:13 +01:00
a3b730ff10
fix(emqx_session_mem): keep msg inflight insert timestamp in Inflight after retry
2024-03-27 11:31:51 +02:00
4f9daa0d4a
chore: bump version to 5.6.0-rc.6
2024-03-27 14:04:27 +08:00
d82b6d216d
chore: remove the duplicated time field
...
Fixes https://emqx.atlassian.net/browse/EMQX-12104
2024-03-27 12:09:21 +08:00
f127bf53b3
chore: 5.6.0-rc.5
2024-03-26 17:52:57 +01:00
291cf689e2
Merge pull request #12786 from ieQu1/dev/check-core-emqx-vsn
...
fix(mria): Strictly check compatibility of the replicant with core
2024-03-26 17:49:37 +01:00
ba43da1b8e
fix(mria): Strictly check compatibility of the replicant with core
2024-03-26 16:32:01 +01:00
da4a04c411
Merge pull request #12777 from thalesmg/test-flaky-conf-copy-m-20240325
...
test: attempt to fix flaky test
2024-03-26 11:52:12 -03:00
78c5f8177c
test: fix test case
2024-03-26 14:51:21 +01:00
10a43db742
docs(message validation): add swagger API examples
...
Fixes https://emqx.atlassian.net/browse/EMQX-12102
2024-03-26 09:39:05 -03:00
067f018337
test: fix test cases
2024-03-26 13:36:12 +01:00
b1c9bb63d7
chore: delete dead code
...
hocon schema validator never validates 'undefined'
2024-03-26 13:20:48 +01:00
c42e980442
feat(log): configurable time format
...
now logs can be configured to use 'epoch' or 'rfc3339' time format
2024-03-26 13:09:32 +01:00
cc37030265
Merge pull request #12770 from thalesmg/mv-metrics-m-20240321
...
feat(message validation): implement metrics
2024-03-26 08:59:05 -03:00
56a40d61ae
fix: bpapi undef abort on old node
2024-03-26 13:06:56 +08:00
2ae973821f
chore: 5.6.0-rc.4
2024-03-25 20:00:10 +01:00
b06d3932af
Merge pull request #12768 from keynslug/fix/EMQX-12068/routing-schema-conflict
...
fix(routing): add heuristic for routing schema in empty cluster
2024-03-25 19:50:51 +01:00
6259f95aab
test: attempt to fix flaky test
...
```
expected: {ok,<<...>>}
got: {error,enoent}
line: 187
comment: #{node => 'emqx_conf_app_SUITE9@127.0.0.1'}
```
2024-03-25 14:36:22 -03:00
0f426e6e77
feat(ds): make durable sessions handle will messages
...
Fixes https://emqx.atlassian.net/browse/EMQX-10431
2024-03-25 14:18:25 -03:00
8c1a1d21a7
feat(session): pass will message down to session when creating/opening
2024-03-25 14:18:23 -03:00
64eeeb7494
chore: verify status code
2024-03-25 10:05:39 -07:00
8b963d5960
Merge pull request #12739 from ieQu1/dev/zone-durability
...
feat(sessds): Add zone overrides for session durability settings
2024-03-25 17:32:53 +01:00
b8b0b809b4
chore: add a warning log if conflict is detected
2024-03-25 17:07:56 +01:00
9411d6078e
docs: remove unnecessary restart instruction
2024-03-25 16:50:51 +01:00
d38545010a
fix: case_clause
2024-03-25 16:43:24 +01:00
c93145cb97
fix: function clause
2024-03-25 16:40:53 +01:00
3aff9eb2a4
fix(route schema): allow boot if all peer nodes agree on one version
2024-03-25 16:33:27 +01:00
fe3cc25855
chore: update instructions for routing schema conflict
2024-03-25 15:22:43 +01:00
59ff43ec87
feat(message validation): implement metrics
...
Fixes https://emqx.atlassian.net/browse/EMQX-12071
2024-03-25 10:55:01 -03:00
94c6e5bb38
Merge pull request #12727 from kjellwinblad/kjell/emqx_connector_info/EMQX-11427
...
refactor: add emqx_connector_info behavior
2024-03-25 14:31:40 +01:00
849fe0c2c8
feat(routing): add schema conflict resolution procedure
...
In the log message printed when a schema conflict in cluster routing
is detected.
2024-03-25 13:43:46 +01:00
f393d75a1e
Merge pull request #12771 from thalesmg/test-another-flaky-monitor-m-20240322
...
test: attempt to fix flaky test
2024-03-25 09:19:20 -03:00
e39fa5eaa9
docs(es_connector_info): add copyright header
2024-03-25 10:44:33 +01:00
3e97e5528e
Merge pull request #12765 from JimMoen/EMQX-12043/subscribers-count
...
fix(stats): `'subscribers.count'` contains shared-subscriber
2024-03-25 17:19:13 +08:00
5f7f9e43f9
fix(stats): `'subscribers.count'` contains shared-subscriber
2024-03-25 15:12:57 +08:00
2926518b3d
Merge pull request #12764 from zhongwencool/fix-bpapi-undef
...
fix: bpapi undef on old node
2024-03-25 13:37:37 +08:00
df913f2450
docs: refine changelog
2024-03-23 14:11:30 +01:00
22838f027a
fix: mountpoint template render should not replace unknown as undefined
...
For backward compatibility, the unknown vars used in mountpoint
is kept unchanged.
e.g. '${unknown}/foo/bar' should be rendered as '${unknown}/foo/bar'
but not 'undefined/foo/bar'
2024-03-23 10:16:05 +01:00
3136ec5958
feat: allow mountpoint to use client_attrs
2024-03-23 10:16:05 +01:00
8e8fc6a3d1
test: fix config test
2024-03-23 10:16:05 +01:00
5e9814d171
fix: add debug level logging for invalid client attributes
2024-03-23 10:16:05 +01:00
0cf61932b6
feat: allow using client_attrs in authentication templates
2024-03-23 10:16:05 +01:00
8254b801ae
feat: support initialize client attribute from user property
2024-03-23 10:16:02 +01:00
2fd0a2cd4d
feat: support extracting initial client attrs from clientinfo
2024-03-23 10:16:02 +01:00
c75840306b
fix: restrict client_attr key and value string format
...
The keys and values are used to render templates for
authz rules, such as topic names, and SQL statements etc.
2024-03-23 10:16:02 +01:00
9ec99fef4a
feat: allow client_attr used in authz rules
2024-03-23 10:16:02 +01:00
e5816f5a13
refactor: rename attr to client_attr
...
client_attr is unique enough for all contexts
so the name can be unified from external responses
to internal template rendering, and rule-engine template rendering
2024-03-23 10:16:02 +01:00
cc4805b1ac
feat: extract attrs field from http and jwt auth response
2024-03-23 10:16:02 +01:00
f9d51484cc
feat: introuce attrs field in clientinfo
...
the 'attrs' field is to be used as extra attributes extracted
from various client properties
for example, HTTP auth result can return an extra client_attributes
field in the result which will get merged into the original
attrs field in clientinfo
2024-03-23 10:16:02 +01:00
ef0fee52f4
fix: query_by_clientid with v0.7 sql api
2024-03-22 17:09:23 -07:00
1fcd06f4bc
feat: update greptimedb client lib and ci version
2024-03-22 15:33:15 -07:00
303d0ed81b
test: attempt to fix flaky test
2024-03-22 16:29:00 -03:00
70378052dd
Merge pull request #12769 from SergeTupchiy/EMQX-12058-handle-client-shutdown-errors
...
fix:(emqx_mgmt): handle client shutdown errors in call_client
2024-03-22 17:43:57 +02:00
cbc84900b2
chore: update bpapi versions dump
2024-03-22 16:24:52 +01:00
9986de49d1
chore: make dyalizer happy
2024-03-22 16:16:49 +01:00
be1886fb91
chore: make dyalizer happy and update changelog
2024-03-22 15:56:44 +01:00
ed0b5ac939
fix(emqx_mgmt): handle client shutdown errors in call_client
...
The client process may be shutdown normally during/shortly before the call.
2024-03-22 16:47:02 +02:00
b81a11b790
chore(routing): mark function that is now an RPC target
2024-03-22 15:14:53 +01:00
ba276d807f
fix(routing): add heuristic for routing schema in empty cluster
...
When the cluster is empty but comprised of different versions of EMQX,
try to additionally ask the cluster nodes for the routing schema in use,
to be able to make more informed decisions.
2024-03-22 15:14:32 +01:00
bc5775a988
fix(kafka_producer): handle ancient v1 config when migrating to actions
...
Fixes https://emqx.atlassian.net/browse/EMQX-12064
2024-03-22 09:45:27 -03:00
d2a1a7f7cf
chore: rename `message_queue_too_long` error reason to `mailbox_overflow`
...
`mailbox_overflow` is consistent with the corresponding config parameter:
'force_shutdown.max_mailbox_size'
2024-03-22 12:20:20 +02:00
42faffc320
refactor: uniform shared_sub table macros
2024-03-22 18:03:13 +08:00
45eedbb9c6
fix: bpapi undef on old node
2024-03-22 17:02:09 +08:00
2b684c7bcc
fix(emqx_connector_info): refactoring based on thalesmg's comments
2024-03-22 09:29:26 +01:00
5d77e1b01e
fix: return not_implemented error for mqueue/inflight msgs queries
2024-03-22 10:02:31 +02:00
9b085b5797
fix(api_schema): removed metrics schema in api spec
...
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622 ).
2024-03-22 10:57:14 +08:00
23ad37f566
Merge pull request #12762 from thalesmg/ds-fix-sync-egress-reply-m-20240321
...
fix(ds): add caller to pending replies before flushing and reply failures to sync callers
2024-03-21 16:42:42 -03:00
cfd4a1297f
Merge pull request #12761 from thalesmg/test-fix-flaky-dash-mon-m-20240321
...
test: fix flaky test
2024-03-21 16:04:08 -03:00
e17f663fa5
Merge pull request #12749 from thalesmg/mv-followup1-m-20240320
...
Follow up features and fixes for message validation - part 1
2024-03-21 16:03:49 -03:00
796c04e7a8
test: fix flaky test
...
We should emit the trace event before replying to callers.
Example failure:
https://github.com/emqx/emqx/actions/runs/8378977952/job/22946318696#step:6:182
```
=CRITICAL REPORT==== 21-Mar-2024::17:45:37.676024 ===
"check stage" failed: error
{assertMatch,[{module,emqx_ds_storage_bitfield_lts_SUITE},
{line,270},
{expression,"? of_kind ( emqx_ds_replication_layer_egress_flush , Trace )"},
{pattern,"[ # { batch := [ _ , _ , _ ] } ]"},
{value,[]}]}
Stacktrace: [{emqx_ds_storage_bitfield_lts_SUITE,
'-t_atomic_store_batch/1-fun-1-',1,
[{file,
"/__w/emqx/emqx/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl"},
{line,270}]},
{emqx_ds_storage_bitfield_lts_SUITE,t_atomic_store_batch,1,
[{file,
"/__w/emqx/emqx/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl"},
{line,249}]}]
```
2024-03-21 15:47:29 -03:00
68af211130
fix(ds): reply sync callers after raft store failure
2024-03-21 15:40:21 -03:00
70737a437a
fix(ds): add caller to pending replies before flushing
2024-03-21 14:39:21 -03:00
93c80bc912
chore: 5.6.0-rc.3
2024-03-21 18:10:59 +01:00
fa7ec231e3
test: fix flaky test
2024-03-21 14:07:09 -03:00
e2a2295c99
Merge pull request #12759 from SergeTupchiy/EMQX-11808-remove-uploaded-invalid-backups
...
fix(emqx_mgmt_data_backup): remove invalid uploaded backup files
2024-03-21 18:58:48 +02:00
00aa7b5621
feat: create new `message.validation_failed` hookpoint and rule engine event
2024-03-21 13:46:27 -03:00
7d7c6685d4
Merge pull request #12753 from thalesmg/test-flaky-retry-m-20240321
...
test: attempt to stabilize flaky test
2024-03-21 13:43:51 -03:00
40eccb10d6
fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid
2024-03-21 17:36:39 +02:00
4bfc51c052
refactor(emqx_connector_info): improve documentation and type specs
2024-03-21 16:08:31 +01:00
7d6fde5960
fix(topics-api): respond with correct totals to paged queries
2024-03-21 16:02:16 +01:00
e837791f94
fix(kafka-based bridges): avoid trying to get raw config for replayq dir
...
Fixes https://emqx.atlassian.net/browse/EMQX-12049
2024-03-21 11:49:00 -03:00
64399b6861
test: attempt to stabilize flaky test
2024-03-21 11:04:14 -03:00
89befa580e
refactor(emqx_connector_info): remove code that is no longer needed
2024-03-21 11:58:54 +01:00
cada944350
feat(sessds): Expose relevant durable session info in the REST API
2024-03-21 10:37:04 +01:00
a3b75eb39d
refactor(remaining connectors): to use emqx_connector_info
...
This commit refactors the sqlserver, iotdb, elasticsearch, opents,
greptimedb, pulsar, tdengine, rabbitmq and s3 connectors to use the
`emqx_connector_info` behavior.
2024-03-21 09:47:57 +01:00
62030e8942
feat(message validation): forbid repeated schema checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-12054
2024-03-20 15:26:16 -03:00
b8cd1c9020
feat(message validation api): add enable/disable HTTP API
2024-03-20 14:59:32 -03:00
4944cc080e
feat(message_validation): add `ignore` failure action
2024-03-20 14:31:43 -03:00
8753e3583f
feat(message_validation): add `none` log level
2024-03-20 14:31:26 -03:00
e767f01e0a
fix(message_validation): take `enable` into account
2024-03-20 13:57:12 -03:00
7aa287c6c1
fix: add message validation schema to `emqx_enterprise_schema`
2024-03-20 13:37:01 -03:00
49e0488498
chore: 5.6.0-rc.2
2024-03-20 17:11:37 +01:00
bede5a5b85
Merge pull request #12746 from zmstone/0320-add-throttle-log-for-auth-failure
...
feat: add authentication_failure throttled log
2024-03-20 17:06:36 +01:00
27c1f1a4e4
Merge pull request #12732 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API-updates
...
EMQX-11861 client mqueue inflight api updates
2024-03-20 16:27:44 +02:00
7257fe526b
fix(ci): add `ra` to emqx app dependencies as well
2024-03-20 14:46:53 +01:00
90fce21adc
docs: add change log for #12746
...
also updated #12520 to include newly added throttled log id
2024-03-20 14:43:28 +01:00
38e30622f3
chore(license): add LICENSE tag to logs
2024-03-20 14:42:01 +01:00
2c81b56377
feat: add authentication_failure throttled log
...
also included changes:
1. add 'username' to log metadata
2. add 'tag' to authz warning logs
2024-03-20 14:42:01 +01:00
e5f9e7631d
Merge pull request #12736 from zmstone/0319-json-schema-draft-06-as-default-version
...
chore: upgrade to jesse 1.8.0 for draft-06 as default version
2024-03-20 14:39:37 +01:00
27d109ce6d
refactor(syskeeper connector): to use emqx_connector_info
...
This commit refactors the syskeeper connector to use the
`emqx_connector_info` behavior.
2024-03-20 14:29:02 +01:00
cb5fdb3c79
fix: rework In-flight / Mqueue API
...
- use timestamp 'position' to find the next chunk of data
- add 'start' position to response meta
- sort In-flight messages by insertion time
- sort Mqueue messages by priority
2024-03-20 15:25:33 +02:00
188ab71673
refactor(rocketmq connector): to use emqx_connector_info
...
This commit refactors the rocketmq connector to use the
`emqx_connector_info` behavior.
2024-03-20 14:02:18 +01:00
aa368bb9d0
refactor(redis connector): to use emqx_connector_info
...
This commit refactors the redis connector to use the
`emqx_connector_info` behavior.
2024-03-20 13:49:28 +01:00
12717baa0b
refactor(mqtt connector): to use emqx_connector_info
...
This commit refactors the mqtt connector to use the
`emqx_connector_info` behavior.
2024-03-20 13:33:13 +01:00
e55e1dd1b2
chore: whitelist `ra` to make RPCs w/o BPAPIs
2024-03-20 13:20:25 +01:00
a8baff61ec
docs(dsrepl): describe briefly what `n_sites` is for
2024-03-20 13:20:25 +01:00
efac5c6197
test(ds): stabilize `t_message_gc` testcase
2024-03-20 13:20:25 +01:00
fe50a1711b
fix(ds-egress): drop pending batch on failures
...
Before this commit, messages in the current batch will be retried as
part of next batch. This could have led to message duplication which is
probably not what the user wants by default.
2024-03-20 13:20:25 +01:00
a1f5de3f5b
fix(dsrepl): turn memoize into a safer function
2024-03-20 13:20:24 +01:00
d39ca41070
chore(dsrepl): mark per-node `add_generation` RPC target obsolete
...
Also annotate internal exports with comments according with their
intended use.
2024-03-20 13:20:24 +01:00
35b18f9125
fix(dsrepl): properly handle error conditions in generation mgmt
...
Also update few outdated typespecs. Also make error reasons easier
to comprehend.
2024-03-20 13:20:24 +01:00
f2268aa69a
fix(dsrepl): use correct base dir for ra system stuff
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-20 13:20:24 +01:00
404e919494
refactor(dsrepl): make shard allocator more robust and consistent
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-20 13:20:24 +01:00
0e18bd6e80
fix(dsrepl): increase replication site id bitsize back
...
In order to minimize chances of site id collision to practically zero.
2024-03-20 13:20:24 +01:00
ac9700dd28
fix(dsrepl): split shard allocator into a separate module
2024-03-20 13:20:23 +01:00
1b647035d0
chore(dsrepl): make dialyzer a bit happier
2024-03-20 13:20:23 +01:00
611b3f0e07
feat(dsrepl): use more straightforward way to drop ra shards
2024-03-20 13:20:23 +01:00
74881e8706
feat(dsrepl): make storage layer unaware of granularity of time
...
Storage also becomes a bit more pure, depending on the upper layer to
provide the timestamps, which also makes it possible to handle more
operations idempotently.
2024-03-20 13:20:23 +01:00
783f769a98
fix(mgmt): Fix bulk kickout of durable sessions
2024-03-20 12:52:44 +01:00
223bbc05fe
refactor(mysql connector): to use emqx_connector_info
...
This commit refactors the mysql connector to use the
`emqx_connector_info` behavior.
2024-03-20 11:12:38 +01:00
c6d7c57bbd
refactor(clickhouse connector): to use emqx_connector_info
...
This commit refactors the clickhouse connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:59:58 +01:00
afa3da6d0d
refactor(cassandra connector): to use emqx_connector_info
...
This commit refactors the cassandra connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:50:44 +01:00
159ab81904
refactor(influxdb connector): to use emqx_connector_info
...
This commit refactors the influxdb connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:39:16 +01:00
c1e6092013
refactor(oracle connector): to use emqx_connector_info
...
This commit refactors the oracle connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:12:55 +01:00
808efe0f64
chore: rm outdated comment
2024-03-20 16:49:36 +08:00
b8b4c5a34a
fix(otel): cpu usage/idle metrics for opentelemetry
2024-03-20 16:48:21 +08:00
0547b32727
feat(sessds): Add zone overrides for session durability settings
2024-03-19 22:51:06 +01:00
611e4f6710
Merge pull request #12740 from ieQu1/dev/kick-durable-sessions
...
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 21:30:40 +01:00
e2db038e7c
Merge pull request #12734 from keynslug/fix/EMQX-12030/topics-api
...
fix(api-topics): expose persistent session topics in APIs
2024-03-19 21:08:23 +01:00
3cb36a5619
feat(ds-lts): extract timestamp from storage key itself
...
1. This avoids the need to deserialize the message to get the timestamp.
2. It also makes possible to decouple the storage key timestamp from the
message timestamp, which might be useful for replication purposes.
2024-03-19 20:21:56 +01:00
5cc0246351
feat(dsrepl): allow to tune select ra options
2024-03-19 20:21:55 +01:00
54b5adf868
feat(dsrepl): allocate shards predictably
...
To ensure strictly optimal and fair shard allocation across
cluster. Before this commit it was quite easy to end up with
an allocation significantly skewed towards some node, because
of the nature of randomness and relatively small number of
shards.
2024-03-19 20:21:55 +01:00
d30c99512a
feat(utils-stream): add a few more stream combinators
2024-03-19 20:21:54 +01:00
887e151be5
fix(dsrepl): handle errors gracefully in shard egress process
...
Also add cooldown on timeout / unavailability.
2024-03-19 20:21:53 +01:00
e16aee99b4
chore(dsrepl): clarify how to perform leadership transfer in runtime
2024-03-19 20:21:18 +01:00
00d509f27b
feat(dsrepl): prefer local replica in read path
...
To optimize out any unnecessary RPCs. Given the load should be
smoothed evenly across the cluster, choosing non-leader node is
not a priority.
2024-03-19 20:11:42 +01:00
19305c223c
fix(dsrepl): require majority for replication-related tables
2024-03-19 20:11:42 +01:00
f89909f60c
fix(dsrepl): tolerate trigger election timeouts for existing servers
2024-03-19 20:11:42 +01:00
6c6ea50e42
fix(boot): add `emqx_durable_storage` to the list of restarted apps
2024-03-19 20:11:42 +01:00
3b59cf2ebf
feat(dsrepl): move shard allocation to separate process
...
That starts shard and egress processes only when shards are fully
allocated.
2024-03-19 20:11:41 +01:00
4dafbf21f6
fix(dsrepl): make db + shard part of machine state
...
It doesn't feel right, but right now is the easiest way to have it
in the scope of `apply/3`, because `init/1` doesn't actually invoked
for ra machines recovered from the existing log / snapshot.
2024-03-19 20:11:41 +01:00
d19128ed65
feat(dsrepl): cache shard metadata in persistent terms
2024-03-19 20:11:41 +01:00
e6c2c2fb07
feat(dsrepl): manage generations / db config through ra machine
2024-03-19 20:11:39 +01:00
5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
...
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
be793e4735
fix(dsrepl): reassign timestamp at the time of submission
...
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
146f082fdc
feat(dsrepl): implement raft-based replication
...
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
...
sync release-56
2024-03-19 19:09:41 +01:00
f453bdf055
chore(topic-api): describe `emqx_persistent_session_ds_router:stream/1`
2024-03-19 18:51:06 +01:00
027b237f07
test(ds): stabilize `t_session_replay_retry` testcase
2024-03-19 18:33:38 +01:00
236685f714
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 17:49:04 +01:00
df8c7152fc
refactor(mongodb connector): to use emqx_connector_info
...
This commit refactors the mongodb connector to use the
`emqx_connector_info` behavior.
2024-03-19 16:09:30 +01:00
c3d7d68cfc
refactor(kinesis,pgsql,timescale,matrix connectors): emqx_connector_info
...
This commit refactors the kinesis, pgsql, timescale and matrix
connectors to use the `emqx_connector_info` behavior.
2024-03-19 15:57:23 +01:00
f2dc940436
Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56
2024-03-19 15:20:08 +01:00
a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
...
feat: implement message validation
2024-03-19 11:12:28 -03:00
10ea367030
test(client-api): attempt to get rid of inter-testcase side effects
2024-03-19 14:56:50 +01:00
3129a8fa42
test(client-api): fix snabbkaffe crash noise in the CT logs
2024-03-19 14:55:28 +01:00
b8505ebd54
chore: upgrade to jesse 1.8.0 for draft-06 as default version
2024-03-19 14:47:11 +01:00
a3e631cda2
refactor(kafka connectors): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_kafka` to use the
`emqx_connector_info` behavior. The `emqx_bridge_kafka` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
b8ef357fef
refactor(http connector): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_http` to use the
`emqx_connector_info` behavior. The `emqx_bridge_http` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
dc08fab0e8
refactor(hstreamdb connector): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_hstreamdb` to use the
`emqx_connector_info` behavior. The `emqx_bridge_hstreamdb` related
information can thus be removed from `emqx_connector_ee_schema` and
`emqx_connector_schema`.
2024-03-19 14:41:20 +01:00
d0bebe8b93
refactor(gcp_pubsub connectors): to use emqx_connector_info
...
This commit refactors the emqx_bridge_gcp_pubsub app to use the
emqx_connector_info behavior. The emqx_bridge_gcp_pubsub
related information can thus be removed from emqx_connector_ee_schema
and emqx_connector_schema.
2024-03-19 14:40:33 +01:00
76d32b6e97
refactor(confluent connector): to use emqx_connector_info
...
This commit refactors the emqx_bridge_confluent to use the
emqx_connector_info behavior. The emqx_bridge_confluent related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:43 +01:00
795b668c4f
refactor(azure_event_hub connector): to use emqx_connector_info
...
This commit refactors the emqx_bridge_azure_event_hub to use the
emqx_connector_info behavior. The emqx_bridge_azure_event_hub related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:17 +01:00
c478e375f6
refactor: change failure log configuration schema
2024-03-19 10:04:18 -03:00
c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
...
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
2eac54bf54
chore: rm `move` API
...
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
...
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
JianBo He