Andrew Mayorov
d12e907209
fix(dsrepl): correctly handle ra membership change command results
...
Before this change, results similar to `{error, {no_more_servers_to_try,
[{error, nodedown}, {error, not_member}]}}` were considered retryable
failures, which is incorrect.
2024-04-08 22:44:34 +02:00
Andrew Mayorov
3223797ae5
fix(dsrepl): attempt leadership transfer before server removal
...
This should make it much less likely to hit weird edge cases that lead
to duplicate Raft log entries because of client retries upon receiving
`shutdown` from the leader being removed.
2024-04-08 22:43:58 +02:00
Andrew Mayorov
1e95bd4da6
test(dsrepl): test unresponsive nodes removal / node restarts
2024-04-08 21:27:56 +02:00
Andrew Mayorov
7a836317ac
fix(dsrepl): trigger unfinished shard transition upon startup
...
Also provide a trivial API to trigger them by hand.
2024-04-08 16:12:42 +02:00
Andrew Mayorov
75bb7f5cdc
fix(dsrepl): retry only {add, Site} crashed membership transitions
...
To minimize the potential negative impact of removal transitions that
crash for some unknown and unusual reasons.
2024-04-08 16:04:33 +02:00
Andrew Mayorov
4c0cc079c2
fix(dsrepl): apply unnecessary rebalancing transitions cleanly
2024-04-08 13:25:45 +02:00
Andrew Mayorov
dcde30c38a
test(dsrepl): add two more testcases for rebalancing
2024-04-08 13:22:31 +02:00
Andrew Mayorov
2ace9bb893
chore(dsrepl): sprinkle few comments and typespecs for exports
2024-04-07 22:51:56 +02:00
Andrew Mayorov
ecaad348a7
chore(dsrepl): update few outdated comments / TODOs
2024-04-07 22:51:56 +02:00
Andrew Mayorov
6293efb995
fix(dsrepl): retry crashed membership transitions
2024-04-07 22:51:56 +02:00
Andrew Mayorov
826ce5806d
fix(dsrepl): ensure that new member UID matches server's UID
...
Before that change, UIDs supplied in the `ra:add_member/3` were not
the same as those servers were using. This haven't caused any issues
for some reason, but it's better to ensure that UIDs are the same.
2024-04-07 22:31:24 +02:00
Andrew Mayorov
556ffc78c9
feat(dsrepl): implement membership changes and rebalancing
2024-04-05 18:57:28 +02:00
Andrew Mayorov
d6058b7f51
feat(dsrepl): allow to subscribe to DB metadata changes
...
Currently, only shard metadata changes are announced to the
subscribers.
2024-04-05 17:40:55 +02:00
Andrew Mayorov
a07295d3bc
fix(ds): address shards in the supervisor properly
2024-04-05 17:40:38 +02:00
ieQu1
3f7b14c861
Merge pull request #12833 from ieQu1/dev/ds-cluster-api
...
feat(ds): Add REST API for durable storage
2024-04-05 17:33:20 +02:00
Ivan Dyachkov
b6dc52020d
Merge pull request #12837 from id/0405-ecql-gen_statem
...
chore: bump ecql version to 0.7.0
2024-04-05 15:37:37 +02:00
ieQu1
2504b8126b
feat(ds): Pass mgmt_ds REST API calls to the application
2024-04-05 15:22:06 +02:00
ieQu1
46261440cb
feat(ds): Add a CLI for managing DB replicas
2024-04-05 15:22:06 +02:00
ieQu1
a62db08676
feat(ds): Add REST API for durable storage
2024-04-05 15:22:06 +02:00
ieQu1
d09787d1a6
fix(ds): Fix return types in replication_layer_meta
2024-04-05 15:22:06 +02:00
Ivan Dyachkov
be47fe49ad
chore: bump ecql version to 0.7.0
...
PR: https://github.com/emqx/ecql/pull/13
No functional changes, just switch gen_fsm to gen_statem.
2024-04-05 13:31:33 +02:00
Ivan Dyachkov
7930136803
Merge pull request #12835 from id/0405-ci-update-codeql-workflow
...
ci: stop running codeql on release-55 and enable it for release-57
2024-04-05 13:27:55 +02:00
Ivan Dyachkov
ca0eecb9d0
ci: stop running codeql on release-55 and enable it for release-57
2024-04-05 11:09:02 +02:00
Andrew Mayorov
70396e9766
Merge pull request #12825 from keynslug/feat/EMQX-12110/repl-meta-api
...
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 22:32:03 +02:00
Andrew Mayorov
df6c5b35fe
feat(dsrepl): add more primitive operations to modify DB sites
2024-04-04 21:22:49 +02:00
Andrew Mayorov
bb8ffee18c
feat(dsrepl): add API to get current DB replication sites
2024-04-04 21:22:02 +02:00
Andrew Mayorov
ad52f7838e
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 21:22:01 +02:00
Thales Macedo Garitezi
8d58b40f33
Merge pull request #12831 from thalesmg/ds-checkpoint-clean-m-20240404
...
feat(ds): clear all checkpoints when (re)starting storage layer
2024-04-04 15:41:50 -03:00
Thales Macedo Garitezi
217b35bce5
Merge pull request #12798 from thalesmg/ds-client-api-v2-m-20240327
...
feat(client mgmt api): add cursor-based list API
2024-04-04 15:10:49 -03:00
Thales Macedo Garitezi
c57c36adb2
feat(ds): clear all checkpoints when (re)starting storage layer
...
Fixes https://emqx.atlassian.net/browse/EMQX-12143
2024-04-04 14:05:52 -03:00
Thales Macedo Garitezi
069cd4fbb4
Merge pull request #12812 from thalesmg/async-res-manager-health-check-m-20240328
...
feat(resource manager): perform non-blocking health checks
2024-04-04 09:07:02 -03:00
ieQu1
a0ad4fa35c
Merge pull request #12781 from ieQu1/dev/rocksdb-metrics
...
Add metrics for the builtin durable storage.
2024-04-04 11:11:58 +02:00
ieQu1
f37ed3a40a
fix(ds): Limit the number of retries in egress to 0
2024-04-03 16:38:49 +02:00
ieQu1
2bbfada7af
fix(ds): Make async batches truly async
2024-04-03 11:57:47 +02:00
ieQu1
92ca90c0ca
fix(ds): Improve egress logging
2024-04-03 11:57:47 +02:00
ieQu1
ae5935e7f7
test(ds): Attempt to stabilize metrics_worker tests in CI
2024-04-02 19:14:10 +02:00
ieQu1
4382971443
fix(ds): Preserve errors in the egress
2024-04-02 16:47:43 +02:00
ieQu1
94ca7ad0f8
feat(ds): Report counters for LTS storage layout
2024-04-02 16:47:43 +02:00
ieQu1
f14c253dea
fix(prometheus): Don't add DS metrics when feature is disabled
2024-04-02 16:47:43 +02:00
ieQu1
b379f331de
fix(sessds): Handle errors when storing messages
2024-04-02 16:47:41 +02:00
ieQu1
f41e538526
feat(sessds): Observe next time
2024-04-02 16:45:52 +02:00
ieQu1
b9ad241658
feat(sessds): Add metrics for the number of persisted messages
2024-04-02 16:45:52 +02:00
ieQu1
75b092bf0e
fix(ds): Actually retry sending batch
2024-04-02 16:45:49 +02:00
ieQu1
0de255cac8
feat(ds): Report egress flush time
2024-04-02 16:25:04 +02:00
ieQu1
044f3d4ef5
fix(ds): Don't reverse entries in the atomic batch
2024-04-02 16:25:04 +02:00
ieQu1
606f2a88cd
feat(ds): Add egress metrics
2024-04-02 16:25:04 +02:00
ieQu1
c9de336234
feat(ds): Add metrics worker to the builtin db supervision tree
2024-04-02 16:25:04 +02:00
ieQu1
d8204021dc
refactor(metrics): Move metrics worker to emqx_utils application
2024-04-02 16:25:04 +02:00
Andrew Mayorov
879709e686
Merge pull request #12780 from keynslug/ft/EMQX-11979/snapshot-transfer
...
feat(dsrepl): transfer storage snapshot during ra snapshot recovery
2024-04-02 16:07:46 +02:00
Thales Macedo Garitezi
2097e854fc
feat(client mgmt api): add cursor-based list API
...
Fixes https://emqx.atlassian.net/browse/EMQX-12028
2024-04-02 10:55:28 -03:00
