Andrew Mayorov
46d9adb926
fix(build): sync mix dependencies
2024-03-20 13:20:24 +01:00
Andrew Mayorov
ac9700dd28
fix(dsrepl): split shard allocator into a separate module
2024-03-20 13:20:23 +01:00
Andrew Mayorov
1b647035d0
chore(dsrepl): make dialyzer a bit happier
2024-03-20 13:20:23 +01:00
Andrew Mayorov
611b3f0e07
feat(dsrepl): use more straightforward way to drop ra shards
2024-03-20 13:20:23 +01:00
Andrew Mayorov
74881e8706
feat(dsrepl): make storage layer unaware of granularity of time
...
Storage also becomes a bit more pure, depending on the upper layer to
provide the timestamps, which also makes it possible to handle more
operations idempotently.
2024-03-20 13:20:23 +01:00
Andrew Mayorov
3cb36a5619
feat(ds-lts): extract timestamp from storage key itself
...
1. This avoids the need to deserialize the message to get the timestamp.
2. It also makes possible to decouple the storage key timestamp from the
message timestamp, which might be useful for replication purposes.
2024-03-19 20:21:56 +01:00
Andrew Mayorov
5cc0246351
feat(dsrepl): allow to tune select ra options
2024-03-19 20:21:55 +01:00
Andrew Mayorov
54b5adf868
feat(dsrepl): allocate shards predictably
...
To ensure strictly optimal and fair shard allocation across
cluster. Before this commit it was quite easy to end up with
an allocation significantly skewed towards some node, because
of the nature of randomness and relatively small number of
shards.
2024-03-19 20:21:55 +01:00
Andrew Mayorov
d30c99512a
feat(utils-stream): add a few more stream combinators
2024-03-19 20:21:54 +01:00
Andrew Mayorov
887e151be5
fix(dsrepl): handle errors gracefully in shard egress process
...
Also add cooldown on timeout / unavailability.
2024-03-19 20:21:53 +01:00
Andrew Mayorov
e16aee99b4
chore(dsrepl): clarify how to perform leadership transfer in runtime
2024-03-19 20:21:18 +01:00
Andrew Mayorov
00d509f27b
feat(dsrepl): prefer local replica in read path
...
To optimize out any unnecessary RPCs. Given the load should be
smoothed evenly across the cluster, choosing non-leader node is
not a priority.
2024-03-19 20:11:42 +01:00
Andrew Mayorov
19305c223c
fix(dsrepl): require majority for replication-related tables
2024-03-19 20:11:42 +01:00
Andrew Mayorov
f89909f60c
fix(dsrepl): tolerate trigger election timeouts for existing servers
2024-03-19 20:11:42 +01:00
Andrew Mayorov
6c6ea50e42
fix(boot): add emqx_durable_storage to the list of restarted apps
2024-03-19 20:11:42 +01:00
Andrew Mayorov
3b59cf2ebf
feat(dsrepl): move shard allocation to separate process
...
That starts shard and egress processes only when shards are fully
allocated.
2024-03-19 20:11:41 +01:00
Andrew Mayorov
4dafbf21f6
fix(dsrepl): make db + shard part of machine state
...
It doesn't feel right, but right now is the easiest way to have it
in the scope of `apply/3`, because `init/1` doesn't actually invoked
for ra machines recovered from the existing log / snapshot.
2024-03-19 20:11:41 +01:00
Andrew Mayorov
d19128ed65
feat(dsrepl): cache shard metadata in persistent terms
2024-03-19 20:11:41 +01:00
Andrew Mayorov
e6c2c2fb07
feat(dsrepl): manage generations / db config through ra machine
2024-03-19 20:11:39 +01:00
Andrew Mayorov
5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
...
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
Andrew Mayorov
be793e4735
fix(dsrepl): reassign timestamp at the time of submission
...
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
Andrew Mayorov
146f082fdc
feat(dsrepl): implement raft-based replication
...
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
Ivan Dyachkov
76d14de480
Merge pull request #12722 from emqx/dependabot/github_actions/dot-github/actions/prepare-jmeter/actions-prepare-jmeter-0101c7941f
...
chore(deps): bump the actions-prepare-jmeter group in /.github/actions/prepare-jmeter with 2 updates
2024-03-19 19:14:36 +01:00
Ivan Dyachkov
923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
...
sync release-56
2024-03-19 19:09:41 +01:00
Ivan Dyachkov
f2dc940436
Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56
2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi
a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
...
feat: implement message validation
2024-03-19 11:12:28 -03:00
Thales Macedo Garitezi
c478e375f6
refactor: change failure log configuration schema
2024-03-19 10:04:18 -03:00
JianBo He
c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
...
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
Thales Macedo Garitezi
2eac54bf54
chore: rm move API
...
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
SergeTupchiy
92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
...
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
JianBo He
24c04e715d
Merge pull request #12715 from zhongwencool/fix-source-replace-error
...
fix: cant replace source conf
2024-03-19 19:32:47 +08:00
zhongwencool
111cbbbe3c
fix: don't crash when ban ip error
2024-03-19 19:10:36 +08:00
zhongwencool
d7c01a4fe0
fix: cant replace source conf
2024-03-19 18:53:51 +08:00
JianBo He
92b53f23da
Merge pull request #12733 from zhongwencool/fix-otel-metrics-create-view-crash
...
fix: otel metrics create view crash
2024-03-19 16:10:58 +08:00
Ivan Dyachkov
0ce56fc397
Merge pull request #12728 from id/0318-optimize-docker-image-layers
...
build(docker): optimize docker image layers
2024-03-19 07:53:02 +01:00
zhongwencool
9a44c6712a
fix: otel metrics create view crash
2024-03-19 12:20:49 +08:00
JianBo He
8227e567a1
Merge pull request #12725 from HJianBo/export_the_source_types
...
Support to return the supported source types via `GET /source_types` endpoint
2024-03-19 09:07:51 +08:00
Thales Macedo Garitezi
e8a8416e44
feat(message_validation): impose restrictions on validation name
2024-03-18 15:57:39 -03:00
Thales Macedo Garitezi
74c03377f2
feat(message_validation_api): implement reorder API
2024-03-18 14:30:52 -03:00
Ivan Dyachkov
255d9e7d1e
ci(docker): return condition on when to push images
2024-03-18 17:23:21 +01:00
Ivan Dyachkov
58d0f04056
build(docker): optimize docker image layers
...
chown only mutable state directories
2024-03-18 17:23:19 +01:00
Thales Macedo Garitezi
bcb7fe96d5
refactor: remove ambiguity from api
...
In preparation for the "reorder" API
2024-03-18 13:20:05 -03:00
Thales Macedo Garitezi
96ffc9b174
refactor: export schema_check fn from schema registry
2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi
cde87bce66
chore: add missing traces
2024-03-18 13:11:39 -03:00
Thales Macedo Garitezi
f84a996671
feat: implement message validation
...
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
Serge Tupchii
8f8b023429
refactor: simplify match spec holder construction
2024-03-18 18:00:26 +02:00
Serge Tupchii
b3d44125f6
feat: support multiple clientid / username Qs params in "/clients" API
...
Additionally, add an option to specify which Client info fields to return in the response.
2024-03-18 18:00:26 +02:00
JianBo He
2a611e4508
chore: fix the error api desc
2024-03-18 18:27:28 +08:00
JianBo He
485d8dbbdb
feat: add the get /source_types endpoint to return all supported source
2024-03-18 18:11:06 +08:00
Ivan Dyachkov
f4e20f8fc9
Merge pull request #12724 from emqx/0318-fix-multiarch-docker-image-push
...
ci(docker): invoke buildx again to push multiarch images
2024-03-18 10:47:38 +01:00
