Andrew Mayorov
35e360fcbe
feat(api-ds): provide more information on nonexistent site leave
2024-05-14 15:57:41 +02:00
ieQu1
3da3a36863
test(ds): Add generation in the replication suite
2024-05-13 19:51:04 +02:00
ieQu1
9f7ef9f34f
fix(ds): Apply review remarks
2024-05-13 19:35:24 +02:00
ieQu1
07aa708894
test(ds): Refactor replication suite
2024-05-09 03:56:56 +02:00
ieQu1
63e51fca66
test(ds): Use streams to fill the storage
2024-05-09 02:46:57 +02:00
ieQu1
a0a3977043
feat(ds): Assign latest timestamp deterministically
2024-05-08 23:17:57 +02:00
ieQu1
2236af84ba
feat(ds): two-stage storage commit on the storage level
2024-05-08 23:17:57 +02:00
ieQu1
68ca891f41
test(ds): Use streams to create traffic
2024-05-08 23:17:57 +02:00
ieQu1
1ff2e02fd9
feat(ds): Pass current time to the storage layer via argument
2024-05-06 11:21:32 +02:00
Andrew Mayorov
c4d1360b96
fix(dsrepl): trigger election for new ra servers unconditionallly
...
Otherwise we might end up in a situation when there's no member online
yet at the time of the election trigger, and the election will never
happen.
2024-04-15 16:42:29 +02:00
Andrew Mayorov
1e95bd4da6
test(dsrepl): test unresponsive nodes removal / node restarts
2024-04-08 21:27:56 +02:00
Andrew Mayorov
dcde30c38a
test(dsrepl): add two more testcases for rebalancing
2024-04-08 13:22:31 +02:00
Andrew Mayorov
556ffc78c9
feat(dsrepl): implement membership changes and rebalancing
2024-04-05 18:57:28 +02:00
Andrew Mayorov
70396e9766
Merge pull request #12825 from keynslug/feat/EMQX-12110/repl-meta-api
...
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 22:32:03 +02:00
Andrew Mayorov
ad52f7838e
feat(dsrepl): add APIs to manage DB replication sites
2024-04-04 21:22:01 +02:00
ieQu1
f37ed3a40a
fix(ds): Limit the number of retries in egress to 0
2024-04-03 16:38:49 +02:00
ieQu1
ae5935e7f7
test(ds): Attempt to stabilize metrics_worker tests in CI
2024-04-02 19:14:10 +02:00
ieQu1
94ca7ad0f8
feat(ds): Report counters for LTS storage layout
2024-04-02 16:47:43 +02:00
ieQu1
75b092bf0e
fix(ds): Actually retry sending batch
2024-04-02 16:45:49 +02:00
Andrew Mayorov
c666c65c6a
test(ds): factor out storage iteration into helper module
2024-04-02 13:48:51 +02:00
Andrew Mayorov
e029b8f996
test(dsrepl): wait for whole cluster readiness
...
To minimize the chance of flaky tests due to the shards not being
completely online.
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-04-02 13:48:50 +02:00
Andrew Mayorov
2cd357a5bd
fix(ds): ensure store batch is idempotent wrt generations
2024-04-02 13:48:50 +02:00
Andrew Mayorov
77a022bd93
feat(dsrepl): transfer storage snapshot during ra snapshot recovery
2024-04-02 13:48:49 +02:00
Andrew Mayorov
74881e8706
feat(dsrepl): make storage layer unaware of granularity of time
...
Storage also becomes a bit more pure, depending on the upper layer to
provide the timestamps, which also makes it possible to handle more
operations idempotently.
2024-03-20 13:20:23 +01:00
Andrew Mayorov
5cc0246351
feat(dsrepl): allow to tune select ra options
2024-03-19 20:21:55 +01:00
Andrew Mayorov
e6c2c2fb07
feat(dsrepl): manage generations / db config through ra machine
2024-03-19 20:11:39 +01:00
Ivan Dyachkov
f2dc940436
Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56
2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi
11ae04d810
test(ds): rm unused var warning
2024-03-14 17:16:24 -03:00
Thales Macedo Garitezi
6af01b916e
feat(ds): implement get_delete_streams, make_delete_iterator and delete_next callbacks for builtin storage
...
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-08 09:56:46 -03:00
Andrew Mayorov
f7e3afde16
test(ds): avoid introducing new macros
2024-03-07 16:49:20 +01:00
Andrew Mayorov
69427dc42d
test(ds): add tests for error mapping and replay recovery
2024-03-07 12:59:58 +01:00
Andrew Mayorov
2146d9e1fe
feat(ds): introduce error classes in critical API functions
...
For now, only recoverable / unrecoverable errors are introduced.
2024-03-07 12:59:57 +01:00
Thales Macedo Garitezi
5d87d400f4
feat(ds): add atomic store API
...
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-06 15:24:14 -03:00
Zaiming (Stone) Shi
75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache
...
Relicense apps/emqx_durable_storage under Apache 2.0
2024-02-05 10:16:52 +01:00
ieQu1
b50d6bf1fd
chore(ds): Change the license to Apache 2.0
...
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
2024-02-01 00:10:48 +01:00
Thales Macedo Garitezi
f7b12470dd
test(ds): fix inter-suite flakiness
...
Attempt to mitigate this frequent source of flakiness:
```
=CRASH REPORT==== 31-Jan-2024::17:30:15.025404 ===
crasher:
initial call: emqx_ds_replication_layer_egress:init/1
pid: <0.11312.0>
registered_name: []
exception error: no match of right hand side value {error,
no_leader_for_shard}
in function emqx_ds_replication_layer_egress:init/1 (/emqx/apps/emqx_durable_storage/src/emqx_ds_replication_layer_egress.erl, line 93)
in call from gen_server:init_it/2 (gen_server.erl, line 980)
in call from gen_server:init_it/6 (gen_server.erl, line 935)
ancestors: [<0.11310.0>,<0.11304.0>,emqx_ds_builtin_databases_sup,
emqx_ds_builtin_sup,emqx_ds_sup,<0.11236.0>]
message_queue_len: 0
messages: []
links: [<0.11310.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 376
stack_size: 28
reductions: 231
neighbours:
```
2024-01-31 15:42:26 -03:00
ieQu1
137535a821
feat(ds): Introduce egress process for the builtin backend
2024-01-24 19:33:30 +01:00
Thales Macedo Garitezi
db710c4be5
feat(lts): inherit previous generation's lts when possible
2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi
75b08b525b
feat(ds): add list_generations and drop_generation APIs
2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi
756980837c
Merge branch 'release-54' into sync-r54-m-20240108
2024-01-08 17:39:42 -03:00
ieQu1
caf461fdf6
fix(ds): Don't start the supervision tree when feature is not in use
2024-01-08 17:59:19 +01:00
Ilya Averyanov
9e0f3ce53b
feat(ds): restore original add_generation/update_db_config callback semantics
2023-12-29 13:12:15 +03:00
firest
ed38ca67d5
fix(ds): Unified the names of the add_generation API
2023-12-20 18:58:03 +08:00
firest
31060733a5
feat(ds): add an API for making new generations
2023-12-15 16:08:52 +08:00
Zaiming (Stone) Shi
c1f2287b86
Merge remote-tracking branch 'origin/release-54'
2023-12-14 15:26:49 +01:00
Zaiming (Stone) Shi
d560366c14
test: fix some compile warnings
2023-12-11 09:43:13 +01:00
Thales Macedo Garitezi
66d043becd
feat(ds): introduce update_iterator callback
2023-12-08 15:04:18 -03:00
Thales Macedo Garitezi
2a6d72878f
chore(ds): return DS message key along with batch
2023-12-07 11:36:08 -03:00
ieQu1
2a1f7d946a
feat(ds): Shard messages by publisher client ID
2023-11-21 19:37:01 +01:00
ieQu1
62542e5844
feat(ds): Metadata storage for the replication layer
2023-11-21 19:37:01 +01:00
