yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,425 Commits 393 Branches 684 Tags 74 MiB
28f6eb0200
Commit Graph

245 Commits

Author SHA1 Message Date
Ivan Dyachkov
f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi
2ebc8dcc55 fix(ds): use infinity timeout when storing batches 2024-03-14 10:17:18 -03:00
Thales Macedo Garitezi
6af01b916e feat(ds): implement get_delete_streams, make_delete_iterator and delete_next callbacks for builtin storage 
Part of https://emqx.atlassian.net/browse/EMQX-11841
 2024-03-08 09:56:46 -03:00
Andrew Mayorov
09905d78cd
chore(ds): make error handling slightly simpler 
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
 2024-03-07 12:59:57 +01:00
Andrew Mayorov
b39c710ec2
fix(ds): tidy up few typespecs 2024-03-07 12:59:57 +01:00
Andrew Mayorov
2146d9e1fe
feat(ds): introduce error classes in critical API functions 
For now, only recoverable / unrecoverable errors are introduced.
 2024-03-07 12:59:57 +01:00
Thales Macedo Garitezi
5d87d400f4 feat(ds): add atomic store API 
Part of https://emqx.atlassian.net/browse/EMQX-11841
 2024-03-06 15:24:14 -03:00
Thales Macedo Garitezi
06334798a5 fix(ds): fix drop_generation typespec 
This typespec fix will be used downstream by other backends.
 2024-03-04 14:15:59 -03:00
Ilya Averyanov
b706caf294 feat(ds): export types 2024-02-29 14:27:18 +03:00
Ilya Averyanov
d5ae0e5c53 feat(ds): update delete/count interface 2024-02-28 22:51:24 +03:00
Ilya Averyanov
b010d34640 chore(ds): add delete callbacks 2024-02-26 17:35:13 +03:00
Zaiming (Stone) Shi
46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Thales Macedo Garitezi
d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
ieQu1
8cfb22f0b8
fix(ds): Retry getting the shard leader 2024-02-16 12:42:48 +01:00
ieQu1
280fcd8c52
Merge pull request #12437 from ieQu1/dev/optimize_make_filter 
Optimize emqx_ds_bitmask_keymapper:make_filter function.
 2024-02-05 17:32:28 +01:00
ieQu1
4665837cf0 fix(ds): Apply review remarks 2024-02-05 16:52:06 +01:00
ieQu1
c7888ad1f1
Merge pull request #12475 from ieQu1/dev/lean-stream 
Use a more compact data structure to represent streams
 2024-02-05 13:55:24 +01:00
ieQu1
698ba3f271 fix(ds): Optimize emqx_ds_bitmask_keymapper:make_filter 
This optimization makes idle polling faster
 2024-02-05 10:54:19 +01:00
ieQu1
8edbec5929 refactor(ds): Clarify the language used in ds_bitmapper 2024-02-05 10:54:18 +01:00
Zaiming (Stone) Shi
75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache 
Relicense apps/emqx_durable_storage under Apache 2.0
 2024-02-05 10:16:52 +01:00
ieQu1
2e56810ea2 refactor(ds): Use a simple improper list to represent the streams 2024-02-03 21:15:54 +01:00
ieQu1
3d2ac97c61 feat(ds): Add a CLI interface to inspect status of DS databases 2024-02-02 10:11:01 +01:00
ieQu1
b50d6bf1fd chore(ds): Change the license to Apache 2.0 
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
 2024-02-01 00:10:48 +01:00
Thales Macedo Garitezi
d51deac222 fix(ds): use configured data dir for site storage 2024-01-31 15:42:26 -03:00
ieQu1
2479e1189a fix(ds): Remove unused module 2024-01-29 00:36:13 +01:00
ieQu1
eec56b0d6b fix(sessds): Improve comments 2024-01-26 17:49:33 +01:00
ieQu1
8e8d3af096 fix(sessds): Refactor emqx_persistent_session_ds to use CRUD module 2024-01-26 17:49:33 +01:00
Thales Macedo Garitezi
8e31afe6c2 fix(ds): don't make data dir part of the schema 
The data directory was ending up being persisted in the database schema.  This led to
issues when opening the DB on different nodes.
 2024-01-25 14:44:06 -03:00
ieQu1
305a54f646 chore(ds): Update BPAPI version 2024-01-24 19:33:30 +01:00
ieQu1
eee221f1d0 feat(ds): Make egress batching configurable 2024-01-24 19:33:30 +01:00
ieQu1
137535a821 feat(ds): Introduce egress process for the builtin backend 2024-01-24 19:33:30 +01:00
ieQu1
9b7df302e8 fix(ds): Cache database metadata in RAM 2024-01-24 18:43:48 +01:00
Thales Macedo Garitezi
1eb47d0c16 perf(ds): inherit only LTS paths containing wildcards when adding a new generation 
Fixes https://github.com/emqx/emqx/pull/12338#discussion_r1462139499
 2024-01-23 09:20:28 -03:00
Thales Macedo Garitezi
d122340c13
Merge pull request #12338 from thalesmg/ds-message-gc-20240115 
feat(ds): add message GC
 2024-01-22 16:57:26 -03:00
Thales Macedo Garitezi
db710c4be5 feat(lts): inherit previous generation's lts when possible 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi
75b08b525b feat(ds): add list_generations and drop_generation APIs 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi
57074015c6 feat(ds): allow customizing the data directory 
The storage expectations for the RocksDB DB may be different from our usual data
directory.  Also, it may consume a lot more storage than other data.

This allows customizing the data directory for the builtin DS storage backend.

Note: if the cluster was already initialized using a directory path, changing that config
will have no effect.  This path is currently persisted in mnesia and used when reopening
the DB.
 2024-01-19 13:07:24 -03:00
Thales Macedo Garitezi
7763b0fd34 chore: bump app vsns 2024-01-08 17:42:56 -03:00
Thales Macedo Garitezi
756980837c Merge branch 'release-54' into sync-r54-m-20240108 2024-01-08 17:39:42 -03:00
ieQu1
caf461fdf6 fix(ds): Don't start the supervision tree when feature is not in use 2024-01-08 17:59:19 +01:00
Ilya Averyanov
9e0f3ce53b feat(ds): restore original add_generation/update_db_config callback semantics 2023-12-29 13:12:15 +03:00
Zaiming (Stone) Shi
23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi
20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi
322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
firest
612be8e280 fix(ds): make dialyzer happy 2023-12-20 22:09:03 +08:00
firest
ed38ca67d5 fix(ds): Unified the names of the add_generation API 2023-12-20 18:58:03 +08:00
firest
31060733a5 feat(ds): add an API for making new generations 2023-12-15 16:08:52 +08:00
Zaiming (Stone) Shi
c22a3686ae fix(emqx_durable_storage): fix type specs 2023-12-14 17:35:32 +01:00
Zaiming (Stone) Shi
c1f2287b86 Merge remote-tracking branch 'origin/release-54' 2023-12-14 15:26:49 +01:00
Thales Macedo Garitezi
dbc8141930
Merge pull request #12125 from thalesmg/ds-cache-m-20231206 
chore(ds): return DS key from `next` and add `update_iterator` callback
 2023-12-14 10:23:25 -03:00
Zaiming (Stone) Shi
50f4aba5cd fix(dialyzer): batch 3 2023-12-09 15:50:09 +01:00
Thales Macedo Garitezi
66d043becd feat(ds): introduce update_iterator callback 2023-12-08 15:04:18 -03:00
Zaiming (Stone) Shi
ddbb8560fa fix(dialyzer): batch 2 2023-12-08 17:59:55 +01:00
Thales Macedo Garitezi
2a6d72878f chore(ds): return DS message key along with batch 2023-12-07 11:36:08 -03:00
ieQu1
c43b3eb535 fix(sessds): Add debug logs for the session garbage collection 2023-12-06 15:37:23 +01:00
Andrew Mayorov
130a5a5442
fix(ds): pass topics to emqx_topic:words/1 before feeding LTS tree 
So that empty levels in topics will be properly mapped into `''` atoms.
 2023-12-04 13:39:01 +03:00
ieQu1
0ae618d010 fix(ds): Use emqx_rpc for calls that work with large binaries 2023-12-01 08:27:05 +01:00
Ivan Dyachkov
9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
ieQu1
05e47254e2 fix(ds): Fixes related to the shards table 2023-11-28 12:59:09 +01:00
ieQu1
e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0 
Support QoS0 messages in persistent_session_ds
 2023-11-23 23:43:18 +01:00
ieQu1
449bafc27e fix(ds): LTS trie handles empty topic levels 2023-11-23 22:16:38 +01:00
ieQu1
c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1
1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message 
feat(ds): Don't store #message record in the DB
 2023-11-23 18:56:45 +01:00
ieQu1
c69b82455e feat(ds): Don't store #message record in the DB 2023-11-23 17:56:35 +01:00
Ilya Averyanov
29d42506d5 fix(ds): impose more flexible types for ds implementations 2023-11-23 17:16:11 +03:00
Ilya Averyanov
a1b9a14fa1 feat(ds): allow fdb implementation for durable storage 2023-11-22 14:49:38 +03:00
ieQu1
3165b4f645 fix(ds): Abort application startup when rocksdb is not avialable 2023-11-21 20:27:07 +01:00
ieQu1
3d823beb11 fix(ds): Apply review remarks 2023-11-21 19:37:01 +01:00
ieQu1
f5c71e8068 refactor(ds): Add a wrapper to the store batch API 2023-11-21 19:37:01 +01:00
ieQu1
2a1f7d946a feat(ds): Shard messages by publisher client ID 2023-11-21 19:37:01 +01:00
ieQu1
62542e5844 feat(ds): Metadata storage for the replication layer 2023-11-21 19:37:01 +01:00
Ivan Dyachkov
28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
Thales Macedo Garitezi
088de9476c fix(ds): use integer for tag values 
Follow up to https://github.com/emqx/emqx/pull/11906#discussion_r1389115973
 2023-11-10 11:26:32 -03:00
ieQu1
2ade6da7a6 fix(ds): Remove redundand type 2023-11-10 12:53:30 +01:00
ieQu1
feef23fc08 feat(ds): Pass store_batch through RPC 2023-11-10 12:16:39 +01:00
ieQu1
8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
Thales Macedo Garitezi
42536bb5f7 chore(ds): avoid using records in persistence / rpc 
Fixes https://emqx.atlassian.net/browse/EMQX-11279
 2023-11-09 12:00:25 -03:00
ieQu1
a1cdbaa76d fix(ds): Address code review remarks 2023-11-08 04:31:27 +01:00
ieQu1
7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
ieQu1
74cb43f8b1 fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
Andrew Mayorov
54951c273f feat(ds): mix safe cutoff into keymapper filter during iteration 2023-11-08 03:04:19 +01:00
Andrew Mayorov
7a94db25c3 fix(ds): don't iterate over incomplete epoch in bitmask lts storage 2023-11-08 03:04:19 +01:00
ieQu1
38b7f516bc fix(ds): Change the directory where message database is stored 2023-11-08 03:04:19 +01:00
ieQu1
8e5dda40be fix(ds): Fix static checks 2023-11-08 03:04:19 +01:00
ieQu1
99329e1243 refactor(ds): Address review remarks 2023-11-08 03:04:19 +01:00
ieQu1
2de79dd9ac feat(ds): Replay QoS1 messages 2023-11-08 03:04:19 +01:00
ieQu1
e745e42093 test(ds): Explore full range of keys when testing ratchet function 2023-11-08 03:04:19 +01:00
ieQu1
87689890ff chore(ds): Fix linter and compilation warnings 2023-11-08 03:04:19 +01:00
ieQu1
ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
ieQu1
164ae9e94a feat(ds): LTS bitfield storage passes all tests 2023-11-08 03:04:19 +01:00
ieQu1
56b6b176c2 fix(ds): LTS shall keeps the concrete topic indexes 2023-11-08 03:04:19 +01:00
ieQu1
7428e7037b feat(ds): Bitfield + Learned Topic Structure 2023-11-08 03:04:19 +01:00
ieQu1
ac91dbc58f feat(ds): Restore LTS trie from a dump 2023-11-08 03:04:19 +01:00
ieQu1
f1ab7c8a7c feat(ds): Add persist callback to LTS trie 2023-11-08 03:04:19 +01:00
ieQu1
c149e0e2df fix(ds): Pass topic filter to emqx_ds:make_iterator call 2023-11-08 03:04:19 +01:00
ieQu1
51a6f623fd refactor(ds): Split out bitfield keymapper to a different module 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
903b3863d1 chore(ps_ds): make persistent session module use new emqx_ds APIs 2023-11-08 03:04:19 +01:00
ieQu1
2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1
6d65707d41 refactor(ds): Implement drop_db function 2023-11-08 03:04:19 +01:00
ieQu1
c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1
59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1
7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
ieQu1
c91df2f5cd refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
ieQu1
f1454bb57e feat(ds): learned topic structure 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi
7ab57824dc chore(ds): change return type of storage_layer:next/{1,2} 
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
 2023-11-08 03:04:19 +01:00
Zaiming (Stone) Shi
ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53 
0928 sync release 53
 2023-10-02 21:57:32 +02:00
Zaiming (Stone) Shi
3f6e0e890b chore: bump emqx_durable_storage and emqx_license app vsn 2023-10-02 20:13:09 +02:00
Thales Macedo Garitezi
2358d67908 refactor: move session stuff from emqx_ds to emqx_persistent_session_ds 
Part of https://emqx.atlassian.net/browse/EMQX-10942
 2023-09-29 18:00:24 -03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
Ivan Dyachkov
105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
Andrew Mayorov
a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0 
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-20 22:55:54 +04:00
Andrew Mayorov
c1583f7f9d
fix(ds): refine topic() type to describe parsed topics 
And separate it from `topic_filter()` type, which describes parsed
topic filters.
 2023-09-20 22:55:52 +04:00
Andrew Mayorov
7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov
045d8b7f10
refactor(ds): reorder functions to improve readability 2023-09-20 22:55:24 +04:00
Andrew Mayorov
7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Thales Macedo Garitezi
5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
Thales Macedo Garitezi
a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Thales Macedo Garitezi
b30bcf32bd feat(ds): introduce keyspace concept 
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
 2023-09-14 16:39:00 -03:00
Thales Macedo Garitezi
cae9ae1fab fix(ps_router): use disc_copies for storing persistent session routes 2023-09-13 10:02:39 -03:00
William Yang
ea73d674b9 chore: bump version to e5.2.1-alpha.2 2023-09-13 14:55:17 +02:00
Thales Macedo Garitezi
f1294736b7 feat(ds): add filter for message persistence 
Fixes https://emqx.atlassian.net/browse/EMQX-10520
 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi
fe4640922d feat(ds): add delete callback 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi
23d63f5e01 refactor: fork clientid type for emqx_ds:session_id/0 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi
922ca5e141 feat(ds): close iterators when handling UNSUBSCRIBE packets 
Fixes https://emqx.atlassian.net/browse/EMQX-9742
 2023-08-30 11:14:28 -03:00
Thales Macedo Garitezi
33a0048155 refactor: move logic to ensure_iterator 2023-08-25 15:24:53 -03:00
Thales Macedo Garitezi
dbfacae283 fix: reinstate transactions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
3344bfb0bd refactor: rm emqx_ds_replay 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
021755b82b refactor: rm iterators from DS #session{} record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
3239f5ac5b feat: rm unnecessary transactions, use separate table for iterator references 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
8eab389ae1 perf: avoid unnecessary transaction 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
9463e271c0 feat(ds): open iterators when handling SUBSCRIBE packets 
Fixes https://emqx.atlassian.net/browse/EMQX-9741
 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
ca697a4e14 fix: rename emqx_ds{,_replay}:replay_id() 2023-08-22 13:29:15 -03:00
firest
abf8d8b391 fix(ds): make dialyzer happy 2023-07-26 16:35:07 +08:00
firest
14b76916df fix(ds): avoid crashes when starting on Windows 2023-07-26 15:37:58 +08:00
Andrew Mayorov
8c883feb54
chore: bump emqx_durable_storage to 0.1.1 2023-07-07 21:56:21 +02:00
Andrew Mayorov
daf4e86da1
feat(ds): add ensure_shard/2 shortcut 
And use it in `emqx_persistent_session_ds:init()` backend initialization
function.
 2023-07-07 21:56:20 +02:00
Andrew Mayorov
ac56de9fc5
feat(ds): allow to redefine directory for shard's rocksdb 2023-07-07 21:56:20 +02:00
ieQu1
9a81f22105 fix(ds): Apply review remarks 2023-06-22 17:07:30 +02:00
ieQu1
cfcd81e17b fix(ds): Fix build 2023-06-22 11:07:52 +02:00
ieQu1
b29c5ad23c feat(emqx_ds): Add API draft for logic layer 2023-06-22 10:20:18 +02:00
ieQu1
a343cdb1d5 refactor(ds): message_storage -> message_storage_bitmask 2023-06-22 10:20:18 +02:00
ieQu1
a4219db163 refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer 2023-06-22 10:20:18 +02:00
ieQu1
1159f99432 refactor(ds): emqx_replay -> emqx_ds 2023-06-22 10:20:18 +02:00
ieQu1
8d6bcc1414 refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00