19cc7c34a5
wip: disable rocksdb wal
2024-02-29 21:06:08 +01:00
475efc93b3
wip: allow to tune select ra options
2024-02-29 17:19:38 +01:00
da6844957e
wip: allocate shards predictably
2024-02-27 21:38:42 +01:00
574746b190
wip: tune egress draining strategy
2024-02-22 18:41:10 +01:00
1895b250e9
wip: provide persist feedback to broker + alternative egress impl
2024-02-20 20:33:57 +01:00
83dd6a2896
wip: handle errors gracefully in shard egress process
...
Also add cooldown on timeout / unavailability.
2024-02-19 19:09:22 +01:00
77c65266d0
wip: clarify how to perform leadership transfer in runtime
2024-02-19 19:07:59 +01:00
0e832db1d4
wip: catch and mask badrpcs in the read path
2024-02-15 16:51:52 +01:00
58bd42bfc1
wip: prefer local replica in read path
2024-02-12 19:24:42 +01:00
3a6b4b57d7
wip: require majority for replication-related tables
2024-02-12 19:24:42 +01:00
5ab036a5d2
wip: tolerate trigger election timeouts for existing servers
2024-02-12 19:24:42 +01:00
9d94ee73de
wip: move shard allocation to separate process
...
That starts shard and egress processes only when shards are fully
allocated.
2024-02-12 19:24:42 +01:00
1b107d578c
wip: make db + shard part of machine state
...
It doesn't feel right, but right now is the easiest way to have it
in the scope of `apply/3`, because `init/1` doesn't actually invoked
for ra machines recovered from the existing log / snapshot.
2024-02-12 19:24:42 +01:00
5e32fe4355
wip: cache shard metadata in persistent terms
2024-02-12 19:24:41 +01:00
7fa3bbf176
wip: manage generations / db config through ra machine
2024-02-12 19:24:41 +01:00
46e8118e1c
wip: allocate shards only when predefined number of sites online
2024-02-12 19:24:41 +01:00
fcb5ed346f
wip: reassign timestamp at the time of submission
2024-02-12 19:24:41 +01:00
bb0cf62879
wip: implement raft-based replication
2024-02-12 19:24:41 +01:00
280fcd8c52
Merge pull request #12437 from ieQu1/dev/optimize_make_filter
...
Optimize emqx_ds_bitmask_keymapper:make_filter function.
2024-02-05 17:32:28 +01:00
4665837cf0
fix(ds): Apply review remarks
2024-02-05 16:52:06 +01:00
c7888ad1f1
Merge pull request #12475 from ieQu1/dev/lean-stream
...
Use a more compact data structure to represent streams
2024-02-05 13:55:24 +01:00
698ba3f271
fix(ds): Optimize emqx_ds_bitmask_keymapper:make_filter
...
This optimization makes idle polling faster
2024-02-05 10:54:19 +01:00
8edbec5929
refactor(ds): Clarify the language used in ds_bitmapper
2024-02-05 10:54:18 +01:00
75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache
...
Relicense apps/emqx_durable_storage under Apache 2.0
2024-02-05 10:16:52 +01:00
2e56810ea2
refactor(ds): Use a simple improper list to represent the streams
2024-02-03 21:15:54 +01:00
3d2ac97c61
feat(ds): Add a CLI interface to inspect status of DS databases
2024-02-02 10:11:01 +01:00
139f5dc3bd
chore(ds): Remove an obsolete document; superceded by ./README.md
2024-02-01 00:14:08 +01:00
b50d6bf1fd
chore(ds): Change the license to Apache 2.0
...
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
2024-02-01 00:10:48 +01:00
f7b12470dd
test(ds): fix inter-suite flakiness
...
Attempt to mitigate this frequent source of flakiness:
```
=CRASH REPORT==== 31-Jan-2024::17:30:15.025404 ===
crasher:
initial call: emqx_ds_replication_layer_egress:init/1
pid: <0.11312.0>
registered_name: []
exception error: no match of right hand side value {error,
no_leader_for_shard}
in function emqx_ds_replication_layer_egress:init/1 (/emqx/apps/emqx_durable_storage/src/emqx_ds_replication_layer_egress.erl, line 93)
in call from gen_server:init_it/2 (gen_server.erl, line 980)
in call from gen_server:init_it/6 (gen_server.erl, line 935)
ancestors: [<0.11310.0>,<0.11304.0>,emqx_ds_builtin_databases_sup,
emqx_ds_builtin_sup,emqx_ds_sup,<0.11236.0>]
message_queue_len: 0
messages: []
links: [<0.11310.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 376
stack_size: 28
reductions: 231
neighbours:
```
2024-01-31 15:42:26 -03:00
d51deac222
fix(ds): use configured data dir for site storage
2024-01-31 15:42:26 -03:00
35c4ef2ee2
Merge pull request #12407 from zmstone/0126-script-to-update-bsl-license-convert-time
...
0126 script to update bsl license convert time
2024-01-30 09:54:34 +08:00
7ee0a6aaa4
Merge pull request #12414 from ieQu1/dev/ds-readme
...
doc(ds): Update README
2024-01-29 18:01:03 +01:00
82403167c2
chore: update BSL license change date
2024-01-29 16:47:31 +01:00
2479e1189a
fix(ds): Remove unused module
2024-01-29 00:36:13 +01:00
96c527541b
docs(ds): Update README
2024-01-28 20:27:20 +01:00
eec56b0d6b
fix(sessds): Improve comments
2024-01-26 17:49:33 +01:00
8e8d3af096
fix(sessds): Refactor emqx_persistent_session_ds to use CRUD module
2024-01-26 17:49:33 +01:00
8e31afe6c2
fix(ds): don't make data dir part of the schema
...
The data directory was ending up being persisted in the database schema. This led to
issues when opening the DB on different nodes.
2024-01-25 14:44:06 -03:00
305a54f646
chore(ds): Update BPAPI version
2024-01-24 19:33:30 +01:00
eee221f1d0
feat(ds): Make egress batching configurable
2024-01-24 19:33:30 +01:00
137535a821
feat(ds): Introduce egress process for the builtin backend
2024-01-24 19:33:30 +01:00
9b7df302e8
fix(ds): Cache database metadata in RAM
2024-01-24 18:43:48 +01:00
1eb47d0c16
perf(ds): inherit only LTS paths containing wildcards when adding a new generation
...
Fixes https://github.com/emqx/emqx/pull/12338#discussion_r1462139499
2024-01-23 09:20:28 -03:00
d122340c13
Merge pull request #12338 from thalesmg/ds-message-gc-20240115
...
feat(ds): add message GC
2024-01-22 16:57:26 -03:00
db710c4be5
feat(lts): inherit previous generation's lts when possible
2024-01-22 14:53:17 -03:00
75b08b525b
feat(ds): add `list_generations` and `drop_generation` APIs
2024-01-22 14:53:17 -03:00
57074015c6
feat(ds): allow customizing the data directory
...
The storage expectations for the RocksDB DB may be different from our usual data
directory. Also, it may consume a lot more storage than other data.
This allows customizing the data directory for the builtin DS storage backend.
Note: if the cluster was already initialized using a directory path, changing that config
will have no effect. This path is currently persisted in mnesia and used when reopening
the DB.
2024-01-19 13:07:24 -03:00
7763b0fd34
chore: bump app vsns
2024-01-08 17:42:56 -03:00
756980837c
Merge branch 'release-54' into sync-r54-m-20240108
2024-01-08 17:39:42 -03:00
caf461fdf6
fix(ds): Don't start the supervision tree when feature is not in use
2024-01-08 17:59:19 +01:00
Andrew Mayorov