382feab7d1
chore(dsraft): fix few spelling errors
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-08-05 10:55:49 +02:00
d631b5b296
feat(ds): support deletions + precondition-related API in bitfield-lts
2024-08-05 10:55:48 +02:00
26ec69d5f4
test(ds): verify deletions work predictably
2024-08-05 10:55:48 +02:00
7b243ef7ad
feat(ds): support operations + preconditions in skipstream-lts
2024-08-01 14:26:45 +02:00
fcf76d28ba
feat(dsraft): support atomic batches + preconditions
2024-08-01 14:26:45 +02:00
3b5d98c1d9
feat(ds): adopt buffer interface to `emqx_ds:operation()`
2024-08-01 14:26:45 +02:00
451b03ff99
feat(ds): add generic preconditions implementation
2024-08-01 14:26:45 +02:00
b010efb647
fix(ds): Improve logic of skipstream LTS layout
...
Iterators:
Previously it used timestamp of the next message as a reference. This
won't work well for the upcoming beamformer/beamsplitter feature. This
commit changes the logic so iterators store timestamp of the last seen
message.
Cooked batches:
Cooked batches no longer store index entries. Creation of indexes has
been delegated to commit callback.
2024-07-24 10:32:06 +02:00
3a893626b8
Merge pull request #13474 from keynslug/ft/EMQX-12309/ds-cas-api
...
feat(ds): allow isolated batches with preconditions
2024-07-18 12:52:13 +02:00
0e545ffcec
feat(ds): add dedicated `#message_matcher{}` for preconditions
2024-07-17 21:27:17 +02:00
0c05b3f019
fix(ds): make conditionals less confusing
2024-07-17 16:23:41 +02:00
14022aded1
feat(ds): allow isolated batches with preconditions
...
Namely, single message deletions and preconditions that can be used to
build complex "compare-and-swap"-style operations. Also allow user to
declare that atomic batches support is needed for a DB.
2024-07-17 15:57:17 +02:00
02e1007a16
feat(dslocal): implement `force_monotonic_timestamps => false`
2024-07-17 15:49:50 +02:00
ae3812da85
feat(ds): allow to turn monotonic timestamps off for DB
...
That tells implementation how to assign timestamps to messages. Current
implicit default is now `force_monotonic_timestamps => true`.
2024-07-17 14:40:23 +02:00
02a0ccfdd1
ci: preparations for new mix build
2024-07-15 09:26:04 -03:00
066fd0481b
feat(mix): compile asn1 files
2024-07-15 09:26:04 -03:00
3721be65ee
fix(ds): Improve comments
2024-07-09 13:15:15 +02:00
dc4ae82798
test(ds): Add message ID
2024-07-09 13:15:14 +02:00
661f79544b
fix(ds): Optimize hot loop of skipstream storage layout
2024-07-09 13:15:14 +02:00
23dafbb03b
feat(ds): Add a benchmarking tool for storage efficiency analysis
2024-07-09 13:15:14 +02:00
afeb2ab8aa
feat(ds): Add metrics for skipstream layout
2024-07-09 13:15:14 +02:00
8c5e4a2376
test(ds): Generalize storage layout test suite for different layouts
2024-07-09 13:15:14 +02:00
a4642d4d06
feat(ds): Add a new storage layout engine: "skipstream"
...
This layout is based on LTS as well, but it uses separate index
streams for constrained replay of streams with learned wildcards
2024-07-09 13:15:14 +02:00
de48077ac4
test(ds): Add new helper functions
...
- Improve message comparison
- Add set operations
2024-07-09 13:15:14 +02:00
210556e545
feat(ds): Generalize value serialization
...
- Add a new utility module for message serialization
- Add experimental serializer based on ASN.1
2024-07-09 13:15:14 +02:00
843973ef32
fix(ds): bitfield_lts: static_key_size -> static_key_bits
2024-07-09 13:15:14 +02:00
f84fb34692
feat(ds_lts): New APIs: info, reverse lookups and topic compression
2024-07-09 13:15:14 +02:00
eb80402ccb
fix(ds): Improve typespecs and descriptions in storage_layer
2024-07-09 13:15:14 +02:00
71dad0242e
docs(ds): Move Raft-related parts to emqx_ds_builtin_raft README
2024-07-09 13:15:14 +02:00
afe1c5617d
refactor(ds): Rename macros for bitfield_lts metrics
2024-07-09 13:15:14 +02:00
0f2c19b656
refactor(ds): Move end_of_stream detection logic for delete_next
2024-07-09 13:15:14 +02:00
b565976794
fix(ds): Fix hashing of empty wildcard topic levels in bitfield_lts
2024-07-09 13:15:13 +02:00
32ace85e1c
Merge branch 'release-57' into 20240703-m-sync-r57
2024-07-03 16:03:15 -03:00
afb65817c4
ci(mix): refactor to avoid conflicts
...
This introduces a refactoring that will hopefully help avoid conflicts with ongoing work
to migrate our build / CI tooling to Mix.
2024-07-03 14:37:25 -03:00
5532f40d83
Merge branch 'release-57' into 20240702-m-sync-r57-mix-umbrella
2024-07-02 11:52:36 -03:00
19f3b030f9
chore: preparing to run common tests / eunit with mix
2024-07-02 09:45:45 -03:00
b74828d7ea
perf: "mixify" project to improve release speed
...
Fixes https://emqx.atlassian.net/browse/EMQX-12527
2024-07-02 09:39:48 -03:00
30efa1f57e
test(dsrepl): relax crash-recover testcase to tolerate message loss
...
Which is quite an expected occasion for this kind of stress test.
2024-06-26 20:44:34 +02:00
733751fadd
refactor(dsstore): keep passing `Options` to both prepare + commit
2024-06-26 20:44:34 +02:00
5b5f33c421
chore(dsstore): resurrect `prepare_batch` entry tracepoint
2024-06-26 20:44:34 +02:00
8db70b5bbc
test(dsrepl): add crash-restart-recover testcase
...
That verifies nothing is lost in the event of abrupt node failures.
2024-06-26 20:44:33 +02:00
5fd5fc76e5
fix(dsstore): ensure backward compatibility
2024-06-26 20:44:33 +02:00
8538a5a5b6
test(dsrepl): anticipate transitionless membership changes
...
E.g. when a membership change is applied twice in a row.
2024-06-26 20:44:33 +02:00
2180cc7c26
fix(dsstore): avoid storing `cf_refs()` in the RocksDB itself
...
This is both pointless and confusing.
2024-06-26 20:44:33 +02:00
7895e9cc45
feat(dsstore): make WAL-less mode optional
...
And make the upper layer choose when to use it.
2024-06-26 20:44:32 +02:00
0c0757b8c2
feat(dsrepl): enable WAL-less batch writes
2024-06-26 20:44:32 +02:00
b6894c18fa
chore(dsrepl): improve tracepoints usability a bit
2024-06-26 20:44:32 +02:00
d349f84f04
fix(ds): Apply remarks
2024-06-24 20:20:40 +02:00
5d1608f34b
chore: minimize oss/platform diff
2024-06-24 15:30:03 +02:00
9a58d71378
fix(ds): Move DS backend supervision trees to their own apps
2024-06-21 18:22:04 +02:00
Andrew Mayorov