yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
emqx/apps/emqx_durable_storage
History
Thales Macedo Garitezi 1eb47d0c16 perf(ds): inherit only LTS paths containing wildcards when adding a new generation 
Fixes https://github.com/emqx/emqx/pull/12338#discussion_r1462139499
 		2024-01-23 09:20:28 -03:00
..
include fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
src perf(ds): inherit only LTS paths containing wildcards when adding a new generation 2024-01-23 09:20:28 -03:00
test feat(lts): inherit previous generation's lts when possible 2024-01-22 14:53:17 -03:00
BSL.txt refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00
IMPLEMENTATION.md refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
README.md fix(ds): Update README 2023-12-01 08:27:05 +01:00
rebar.config style: erlfmt all `rebar.config` files and `bin/nodetool` 2023-12-29 09:08:03 +08:00

README.md

EMQX Replay

emqx_ds is an application implementing durable storage for MQTT messages within EMQX.

Features

  • Streams. Stream is an abstraction that encompasses topics, shards, different data layouts, etc. The client application must only aware of the streams.

  • Batching. All the API functions are batch-oriented.

  • Iterators. Iterators can be stored durably or transferred over network. They take relatively small space.

  • Support for various backends. Almost any DBMS that supports range queries can serve as a emqx_durable_storage backend.

  • Builtin backend based on RocksDB.

    • Changing storage layout on the fly: it's achieved by creating a new set of tables (known as "generation") and the schema.
    • Sharding based on publisher's client ID

Limitation

  • Builtin backend currently doesn't replicate data across different sites
  • There is no local cache of messages, which may result in transferring the same data multiple times

Documentation links

TBD

Usage

Currently it's only used to implement persistent sessions.

In the future it can serve as a storage for retained messages or as a generic message buffering layer for the bridges.

Configurations

emqx_durable_storage doesn't have any configurable parameters. Instead, it relies on the upper-level business applications to create a correct configuration and pass it to emqx_ds:open_db(DBName, Config) function according to its needs.

HTTP APIs

None

Other

TBD

Contributing

Please see our contributing.md.