Commit Graph

31 Commits

Author SHA1 Message Date
ieQu1
8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
ieQu1
e745e42093 test(ds): Explore full range of keys when testing ratchet function 2023-11-08 03:04:19 +01:00
ieQu1
87689890ff chore(ds): Fix linter and compilation warnings 2023-11-08 03:04:19 +01:00
ieQu1
ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
ieQu1
164ae9e94a feat(ds): LTS bitfield storage passes all tests 2023-11-08 03:04:19 +01:00
ieQu1
56b6b176c2 fix(ds): LTS shall keeps the concrete topic indexes 2023-11-08 03:04:19 +01:00
ieQu1
7428e7037b feat(ds): Bitfield + Learned Topic Structure 2023-11-08 03:04:19 +01:00
ieQu1
c149e0e2df fix(ds): Pass topic filter to emqx_ds:make_iterator call 2023-11-08 03:04:19 +01:00
ieQu1
2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1
6d65707d41 refactor(ds): Implement drop_db function 2023-11-08 03:04:19 +01:00
ieQu1
c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1
59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1
7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
7ab57824dc chore(ds): change return type of storage_layer:next/{1,2}
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
Thales Macedo Garitezi
a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Thales Macedo Garitezi
b30bcf32bd feat(ds): introduce keyspace concept
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
Thales Macedo Garitezi
ee2897e5de test(refactor): move test to integration tests dir 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
dad27091be test: rm custom option 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
3344bfb0bd refactor: rm emqx_ds_replay 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
6de0bbe76a test(refactor): always allocate listeners for emqx app 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
021755b82b refactor: rm iterators from DS #session{} record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
9463e271c0 feat(ds): open iterators when handling SUBSCRIBE packets
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
ca697a4e14 fix: rename emqx_ds{,_replay}:replay_id() 2023-08-22 13:29:15 -03:00
Andrew Mayorov
ac56de9fc5
feat(ds): allow to redefine directory for shard's rocksdb 2023-07-07 21:56:20 +02:00
ieQu1
cfcd81e17b fix(ds): Fix build 2023-06-22 11:07:52 +02:00
ieQu1
a343cdb1d5 refactor(ds): message_storage -> message_storage_bitmask 2023-06-22 10:20:18 +02:00
ieQu1
a4219db163 refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer 2023-06-22 10:20:18 +02:00
ieQu1
1159f99432 refactor(ds): emqx_replay -> emqx_ds 2023-06-22 10:20:18 +02:00
ieQu1
8d6bcc1414 refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00