a757afcb7e
Merge pull request #11122 from zmstone/0622-ci-split-emqx-app-suites
...
ci: split apps/emqx common test suites into chunks
2023-06-22 18:17:00 +02:00
9a81f22105
fix(ds): Apply review remarks
2023-06-22 17:07:30 +02:00
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
...
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
36ad8f25ac
Merge pull request #11125 from thalesmg/hide-gcp-consumer
...
chore: hide gcp consumer
2023-06-22 11:29:31 -03:00
22356b7c25
chore: hide gcp pubsub consumer until e5.2.0
2023-06-22 10:05:52 -03:00
f01de238b2
chore: bump app vsns
2023-06-22 10:01:12 -03:00
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
...
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
eec70b62e8
Merge pull request #11114 from zmstone/0621-merge-release-51-to-master
...
0621 merge release-51 to master
2023-06-22 13:35:05 +02:00
75eb18bc81
chore: bump app vsns
2023-06-22 11:09:09 +02:00
cfcd81e17b
fix(ds): Fix build
2023-06-22 11:07:52 +02:00
c58a98954b
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-22 11:05:51 +02:00
d6cd56558a
ci: split apps/emqx common test suites into chunks
2023-06-22 10:35:00 +02:00
b29c5ad23c
feat(emqx_ds): Add API draft for logic layer
2023-06-22 10:20:18 +02:00
a343cdb1d5
refactor(ds): message_storage -> message_storage_bitmask
2023-06-22 10:20:18 +02:00
a4219db163
refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer
2023-06-22 10:20:18 +02:00
1159f99432
refactor(ds): emqx_replay -> emqx_ds
2023-06-22 10:20:18 +02:00
8d6bcc1414
refactor(ds): emqx_replay -> emqx_durable_storage
2023-06-22 10:20:18 +02:00
c4f721f214
chore(ds): Add callback definitions to local_store
2023-06-22 10:20:18 +02:00
efed7cdc2f
chore(ds): behavior -> behaviour
2023-06-22 10:20:18 +02:00
0f6c81e42d
chore(ds): Change license to BSL
2023-06-22 10:20:18 +02:00
04adb65c09
refactor(ds): zone -> shard
...
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
8ac0bba958
test(ds): verify preserve / restore works with stored iterators
2023-06-22 10:20:18 +02:00
d2065e0c1b
feat(ds): allow to preserve / restore iterators in the db
...
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
cbded9af39
feat(ds): Make `create_generation` safer against bad input
2023-06-22 10:20:18 +02:00
83d1606d00
feat(ds): Make iteration fully generation-aware
2023-06-22 10:20:18 +02:00
bf9d57f6a9
chore(ds): Simplify iteration-related typespecs
2023-06-22 10:20:18 +02:00
cf6a5e1643
feat(ds): Allow to create new storage generations
2023-06-22 10:20:18 +02:00
f80e906553
fix(ds): Correct typespec
2023-06-22 10:20:18 +02:00
f5a7b49f57
feat(ds): Wire iteration options up to the app config
2023-06-22 10:20:18 +02:00
2bf8a07b05
test(ds): Split unit tests off into a full-fledged suite
2023-06-22 10:20:18 +02:00
cf70184ebd
test(ds): Provide more general `keymapper_info/1`
2023-06-22 10:20:18 +02:00
2f385022cd
chore(ds): Rename testsuite to reflect test subject better
2023-06-22 10:20:18 +02:00
d875fa49d3
feat(ds): Enable periodic iterator refresh
...
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
7f408da251
refactor(ds): Use inline functions instead of macros
2023-06-22 10:20:07 +02:00
a11e75d189
fix(ds): clear bitmask of topic filter tail containing wildcards
2023-06-22 10:16:55 +02:00
ac0935ef91
test(ds): Proptest that iteration is exhaustive
...
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
60e3070328
test(ds): Scale up number of messages per topic in proptests
2023-06-22 10:16:41 +02:00
82a7292851
test(ds): Use `_build/test/proper` as a scratch dir for testruns
2023-06-22 10:16:32 +02:00
440fecd268
test(ds): Add a proptest on iterator preserve / restore
...
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
1f5425975f
feat(ds): add an ability to preserve and restore iterators
...
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
94e8dbfe7f
test(ds): Move proptests into a separate module
...
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
4c643aa7d5
refactor(ds): Introduce keyspace filter concept
...
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
c5826cee56
refactor(ds): rename `compute_hash_*` → `compute_topic_*`
2023-06-22 10:16:05 +02:00
862e17cd78
chore(ds): assign CODEOWNER
2023-06-22 10:16:01 +02:00
85d3783ff1
chore(ds): Drop few unused macrodefinitions
2023-06-22 10:15:56 +02:00
a0bcdb5104
chore(ds): Attempt to make `compute_next_seek`'s logic clearer
2023-06-22 10:15:49 +02:00
f1e39d977a
fix(ds): Fix dialyzer warnings
2023-06-22 10:15:47 +02:00
adcbf40d27
refactor(ds): tau -> epoch
2023-06-22 10:15:44 +02:00
120d4e66ae
refactor(ds): Factor out configuration to a separate module
2023-06-22 10:15:42 +02:00
Zaiming (Stone) Shi