ieQu1
b010efb647
fix(ds): Improve logic of skipstream LTS layout
...
Iterators:
Previously it used timestamp of the next message as a reference. This
won't work well for the upcoming beamformer/beamsplitter feature. This
commit changes the logic so iterators store timestamp of the last seen
message.
Cooked batches:
Cooked batches no longer store index entries. Creation of indexes has
been delegated to commit callback.
2024-07-24 10:32:06 +02:00
Andrew Mayorov
0e545ffcec
feat(ds): add dedicated #message_matcher{} for preconditions
2024-07-17 21:27:17 +02:00
Andrew Mayorov
0c05b3f019
fix(ds): make conditionals less confusing
2024-07-17 16:23:41 +02:00
Andrew Mayorov
14022aded1
feat(ds): allow isolated batches with preconditions
...
Namely, single message deletions and preconditions that can be used to
build complex "compare-and-swap"-style operations. Also allow user to
declare that atomic batches support is needed for a DB.
2024-07-17 15:57:17 +02:00
Andrew Mayorov
02e1007a16
feat(dslocal): implement force_monotonic_timestamps => false
2024-07-17 15:49:50 +02:00
Andrew Mayorov
ae3812da85
feat(ds): allow to turn monotonic timestamps off for DB
...
That tells implementation how to assign timestamps to messages. Current
implicit default is now `force_monotonic_timestamps => true`.
2024-07-17 14:40:23 +02:00
ieQu1
3721be65ee
fix(ds): Improve comments
2024-07-09 13:15:15 +02:00
ieQu1
661f79544b
fix(ds): Optimize hot loop of skipstream storage layout
2024-07-09 13:15:14 +02:00
ieQu1
afeb2ab8aa
feat(ds): Add metrics for skipstream layout
2024-07-09 13:15:14 +02:00
ieQu1
a4642d4d06
feat(ds): Add a new storage layout engine: "skipstream"
...
This layout is based on LTS as well, but it uses separate index
streams for constrained replay of streams with learned wildcards
2024-07-09 13:15:14 +02:00
ieQu1
210556e545
feat(ds): Generalize value serialization
...
- Add a new utility module for message serialization
- Add experimental serializer based on ASN.1
2024-07-09 13:15:14 +02:00
ieQu1
843973ef32
fix(ds): bitfield_lts: static_key_size -> static_key_bits
2024-07-09 13:15:14 +02:00
ieQu1
f84fb34692
feat(ds_lts): New APIs: info, reverse lookups and topic compression
2024-07-09 13:15:14 +02:00
ieQu1
eb80402ccb
fix(ds): Improve typespecs and descriptions in storage_layer
2024-07-09 13:15:14 +02:00
ieQu1
afe1c5617d
refactor(ds): Rename macros for bitfield_lts metrics
2024-07-09 13:15:14 +02:00
ieQu1
0f2c19b656
refactor(ds): Move end_of_stream detection logic for delete_next
2024-07-09 13:15:14 +02:00
ieQu1
b565976794
fix(ds): Fix hashing of empty wildcard topic levels in bitfield_lts
2024-07-09 13:15:13 +02:00
Andrew Mayorov
30efa1f57e
test(dsrepl): relax crash-recover testcase to tolerate message loss
...
Which is quite an expected occasion for this kind of stress test.
2024-06-26 20:44:34 +02:00
Andrew Mayorov
733751fadd
refactor(dsstore): keep passing Options to both prepare + commit
2024-06-26 20:44:34 +02:00
Andrew Mayorov
5b5f33c421
chore(dsstore): resurrect prepare_batch entry tracepoint
2024-06-26 20:44:34 +02:00
Andrew Mayorov
5fd5fc76e5
fix(dsstore): ensure backward compatibility
2024-06-26 20:44:33 +02:00
Andrew Mayorov
2180cc7c26
fix(dsstore): avoid storing cf_refs() in the RocksDB itself
...
This is both pointless and confusing.
2024-06-26 20:44:33 +02:00
Andrew Mayorov
7895e9cc45
feat(dsstore): make WAL-less mode optional
...
And make the upper layer choose when to use it.
2024-06-26 20:44:32 +02:00
Andrew Mayorov
0c0757b8c2
feat(dsrepl): enable WAL-less batch writes
2024-06-26 20:44:32 +02:00
Andrew Mayorov
b6894c18fa
chore(dsrepl): improve tracepoints usability a bit
2024-06-26 20:44:32 +02:00
ieQu1
d349f84f04
fix(ds): Apply remarks
2024-06-24 20:20:40 +02:00
ieQu1
9a58d71378
fix(ds): Move DS backend supervision trees to their own apps
2024-06-21 18:22:04 +02:00
ieQu1
3851fc189f
fix(ds): Avoid reverse dependencies from storage to repl. layer
2024-06-21 17:16:24 +02:00
ieQu1
bc915216a0
feat(ds): Support metrics for builtin_local backend
2024-06-21 17:16:24 +02:00
ieQu1
99c9b56cf3
feat(ds_buffer): Add `ls' function to list all local buffers
2024-06-21 17:16:24 +02:00
ieQu1
ecb172b07e
refactor(ds): Rename egress metrics to 'buffer'
2024-06-21 17:16:24 +02:00
ieQu1
09c3ae795d
refactor(ds_raft): Replace egress server with common emqx_ds_buffer
2024-06-21 17:16:24 +02:00
ieQu1
a0fbd37e58
refactor(emqx): Use emqx_ds_backends application
2024-06-21 17:16:24 +02:00
ieQu1
ef09cfcd71
feat(ds): Add emqx_ds_backends application
2024-06-21 17:16:24 +02:00
ieQu1
279619fc80
feat(ds): Add `builtin_local' backend
2024-06-21 17:16:24 +02:00
ieQu1
a8ea0ae4e5
refactor(ds): Extract DS replication layer to a separate application
2024-06-21 17:16:24 +02:00
ieQu1
63f1856a2c
feat(ds): Dynamic backend registration
2024-06-21 17:16:24 +02:00
ieQu1
83dc8f4d77
fix(ds): Fix return values of emqx_ds_storage_layer functions
2024-06-21 17:16:23 +02:00
zmstone
213e4785e7
Merge remote-tracking branch 'origin/release-57' into 0619-sync-release-57-to-master
2024-06-19 14:54:48 +02:00
Thales Macedo Garitezi
8ce16fd7d9
fix(dsreplmeta): check site status when fetching shard info
...
Fixes https://emqx.atlassian.net/browse/EMQX-12356
2024-06-18 10:40:09 -03:00
Thales Macedo Garitezi
20cffb54d4
Merge branch 'release-57' into sync-r57-m-20240617
2024-06-17 17:10:12 -03:00
Andrew Mayorov
82588fbc35
chore(dsstore): refine module and callback typespecs
2024-06-17 17:53:39 +02:00
Andrew Mayorov
05d97397d3
fix(dsstore): use correct comparison when walking over generations
2024-06-17 17:53:39 +02:00
Andrew Mayorov
68f6556856
fix(dsstore): persist inherited LTS trie
...
Before this commit, inherited trie was actually only kept in memory
cache.
Also simplify storage backend behaviour around inheriting previous
generation's legacy.
2024-06-17 17:53:38 +02:00
zmstone
41cbfcfaa1
Merge remote-tracking branch 'origin/release-57' into 0527-sync-5.7.0-to-master
2024-05-27 20:10:44 +02:00
ieQu1
830b62d899
fix(dsrepl): Retry sending ra commands to the leader
2024-05-24 19:09:34 +02:00
ieQu1
1ffc7d5d9e
fix(dsrepl): Treat all exceptions from storage layer as recoverable
2024-05-24 19:09:34 +02:00
ieQu1
25721bceba
fix(dsstor): Delete generation metadata before dropping it
2024-05-24 19:09:33 +02:00
ieQu1
c952e46f08
Merge remote-tracking branch 'origin/release-57'
2024-05-23 21:04:03 +02:00
Andrew Mayorov
ba6382adae
fix(dsrepl): trigger "last-resort" pending transitions handler when idle
...
This is a hack to work around the unintended issues causing shard
allocator to become idle even when there are pending transitions.
2024-05-23 14:54:01 +02:00
