8e5dda40be
fix(ds): Fix static checks
2023-11-08 03:04:19 +01:00
99329e1243
refactor(ds): Address review remarks
2023-11-08 03:04:19 +01:00
2de79dd9ac
feat(ds): Replay QoS1 messages
2023-11-08 03:04:19 +01:00
e745e42093
test(ds): Explore full range of keys when testing ratchet function
2023-11-08 03:04:19 +01:00
87689890ff
chore(ds): Fix linter and compilation warnings
2023-11-08 03:04:19 +01:00
ef46c09caf
feat(ds): Implement ratchet function for bitmask keymapper
2023-11-08 03:04:19 +01:00
164ae9e94a
feat(ds): LTS bitfield storage passes all tests
2023-11-08 03:04:19 +01:00
56b6b176c2
fix(ds): LTS shall keeps the concrete topic indexes
2023-11-08 03:04:19 +01:00
7428e7037b
feat(ds): Bitfield + Learned Topic Structure
2023-11-08 03:04:19 +01:00
ac91dbc58f
feat(ds): Restore LTS trie from a dump
2023-11-08 03:04:19 +01:00
f1ab7c8a7c
feat(ds): Add persist callback to LTS trie
2023-11-08 03:04:19 +01:00
c149e0e2df
fix(ds): Pass topic filter to emqx_ds:make_iterator call
2023-11-08 03:04:19 +01:00
51a6f623fd
refactor(ds): Split out bitfield keymapper to a different module
2023-11-08 03:04:19 +01:00
903b3863d1
chore(ps_ds): make persistent session module use new `emqx_ds` APIs
2023-11-08 03:04:19 +01:00
2972bf14ee
refactor(ds): Implement create_generation gen_rpc storage layer call
2023-11-08 03:04:19 +01:00
6d65707d41
refactor(ds): Implement drop_db function
2023-11-08 03:04:19 +01:00
c6a721a7eb
refactor(ds): Passthrough open_db and get_channels to storage layer
2023-11-08 03:04:19 +01:00
59d01dc823
refactor(ds): Implement emqx_ds:open_db
2023-11-08 03:04:19 +01:00
7095cb8583
refactor(ds): Refactor storage layer
2023-11-08 03:04:19 +01:00
c91df2f5cd
refactor(ds): Create a prototype of replication layer
2023-11-08 03:04:19 +01:00
f1454bb57e
feat(ds): learned topic structure
2023-11-08 03:04:19 +01:00
7ab57824dc
chore(ds): change return type of `storage_layer:next/{1,2}`
...
Part of https://emqx.atlassian.net/browse/EMQX-10942
The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.
From:
```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```
To:
```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.
-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
...
0928 sync release 53
2023-10-02 21:57:32 +02:00
3f6e0e890b
chore: bump emqx_durable_storage and emqx_license app vsn
2023-10-02 20:13:09 +02:00
2358d67908
refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
...
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
dafd7c6085
chore: bump apps versions
2023-09-21 10:58:42 +02:00
105bebc250
chore: merge release-52 into master
2023-09-21 10:22:47 +02:00
a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
...
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
...
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session
2023-09-20 22:55:25 +04:00
045d8b7f10
refactor(ds): reorder functions to improve readability
2023-09-20 22:55:24 +04:00
7326ef550b
fix(sessds): make existing parts of persistent session impl work
2023-09-20 22:55:22 +04:00
5ed5ac48ee
refactor: combine shard id and keyspace into a single value
2023-09-19 12:31:32 -03:00
a511088fd4
refactor: address review comments
2023-09-18 17:47:56 -03:00
b30bcf32bd
feat(ds): introduce keyspace concept
...
Fixes https://emqx.atlassian.net/browse/EMQX-10579
This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.
We might want to tune the storage options differently for distinct sets of topics, the
keyspaces. The keyspace is composed by one or more shards.
- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
cae9ae1fab
fix(ps_router): use `disc_copies` for storing persistent session routes
2023-09-13 10:02:39 -03:00
ea73d674b9
chore: bump version to e5.2.1-alpha.2
2023-09-13 14:55:17 +02:00
f1294736b7
feat(ds): add filter for message persistence
...
Fixes https://emqx.atlassian.net/browse/EMQX-10520
2023-09-11 16:25:34 -03:00
fe4640922d
feat(ds): add delete callback
2023-09-11 16:25:33 -03:00
23d63f5e01
refactor: fork clientid type for `emqx_ds:session_id/0`
2023-09-04 15:38:25 -03:00
922ca5e141
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
...
Fixes https://emqx.atlassian.net/browse/EMQX-9742
2023-08-30 11:14:28 -03:00
33a0048155
refactor: move logic to `ensure_iterator`
2023-08-25 15:24:53 -03:00
ee2897e5de
test(refactor): move test to integration tests dir
2023-08-22 13:29:15 -03:00
dad27091be
test: rm custom option
2023-08-22 13:29:15 -03:00
dbfacae283
fix: reinstate transactions
2023-08-22 13:29:15 -03:00
3344bfb0bd
refactor: rm `emqx_ds_replay`
2023-08-22 13:29:15 -03:00
6de0bbe76a
test(refactor): always allocate listeners for emqx app
2023-08-22 13:29:15 -03:00
021755b82b
refactor: rm iterators from DS `#session{}` record
2023-08-22 13:29:15 -03:00
e4e88ebf36
test: add scenario for node stopping midway during subscribe
2023-08-22 13:29:15 -03:00
3239f5ac5b
feat: rm unnecessary transactions, use separate table for iterator references
2023-08-22 13:29:15 -03:00
8eab389ae1
perf: avoid unnecessary transaction
2023-08-22 13:29:15 -03:00
9463e271c0
feat(ds): open iterators when handling `SUBSCRIBE` packets
...
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
ca697a4e14
fix: rename `emqx_ds{,_replay}:replay_id()`
2023-08-22 13:29:15 -03:00
abf8d8b391
fix(ds): make dialyzer happy
2023-07-26 16:35:07 +08:00
14b76916df
fix(ds): avoid crashes when starting on Windows
2023-07-26 15:37:58 +08:00
8c883feb54
chore: bump `emqx_durable_storage` to 0.1.1
2023-07-07 21:56:21 +02:00
daf4e86da1
feat(ds): add `ensure_shard/2` shortcut
...
And use it in `emqx_persistent_session_ds:init()` backend initialization
function.
2023-07-07 21:56:20 +02:00
ac56de9fc5
feat(ds): allow to redefine directory for shard's rocksdb
2023-07-07 21:56:20 +02:00
940ad12108
docs(ds): Apply suggestions from code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-28 10:04:07 +02:00
e0c11f54aa
docs(ds): Update IMPLEMENTATION.md
2023-06-27 10:53:38 +02:00
9a81f22105
fix(ds): Apply review remarks
2023-06-22 17:07:30 +02:00
cfcd81e17b
fix(ds): Fix build
2023-06-22 11:07:52 +02:00
b29c5ad23c
feat(emqx_ds): Add API draft for logic layer
2023-06-22 10:20:18 +02:00
a343cdb1d5
refactor(ds): message_storage -> message_storage_bitmask
2023-06-22 10:20:18 +02:00
a4219db163
refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer
2023-06-22 10:20:18 +02:00
1159f99432
refactor(ds): emqx_replay -> emqx_ds
2023-06-22 10:20:18 +02:00
8d6bcc1414
refactor(ds): emqx_replay -> emqx_durable_storage
2023-06-22 10:20:18 +02:00
ieQu1