Commit Graph

21515 Commits

Author SHA1 Message Date
Kjell Winblad 159ab81904 refactor(influxdb connector): to use emqx_connector_info
This commit refactors the influxdb connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:39:16 +01:00
Kjell Winblad c1e6092013 refactor(oracle connector): to use emqx_connector_info
This commit refactors the oracle connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:12:55 +01:00
JimMoen 808efe0f64
chore: rm outdated comment 2024-03-20 16:49:36 +08:00
JimMoen b8b4c5a34a
fix(otel): cpu usage/idle metrics for opentelemetry 2024-03-20 16:48:21 +08:00
ieQu1 0547b32727
feat(sessds): Add zone overrides for session durability settings 2024-03-19 22:51:06 +01:00
ieQu1 611e4f6710
Merge pull request #12740 from ieQu1/dev/kick-durable-sessions
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 21:30:40 +01:00
Andrew Mayorov e2db038e7c
Merge pull request #12734 from keynslug/fix/EMQX-12030/topics-api
fix(api-topics): expose persistent session topics in APIs
2024-03-19 21:08:23 +01:00
Andrew Mayorov 3cb36a5619
feat(ds-lts): extract timestamp from storage key itself
1. This avoids the need to deserialize the message to get the timestamp.
2. It also makes possible to decouple the storage key timestamp from the
   message timestamp, which might be useful for replication purposes.
2024-03-19 20:21:56 +01:00
Andrew Mayorov 5cc0246351
feat(dsrepl): allow to tune select ra options 2024-03-19 20:21:55 +01:00
Andrew Mayorov 54b5adf868
feat(dsrepl): allocate shards predictably
To ensure strictly optimal and fair shard allocation across
cluster. Before this commit it was quite easy to end up with
an allocation significantly skewed towards some node, because
of the nature of randomness and relatively small number of
shards.
2024-03-19 20:21:55 +01:00
Andrew Mayorov d30c99512a
feat(utils-stream): add a few more stream combinators 2024-03-19 20:21:54 +01:00
Andrew Mayorov 887e151be5
fix(dsrepl): handle errors gracefully in shard egress process
Also add cooldown on timeout / unavailability.
2024-03-19 20:21:53 +01:00
Andrew Mayorov e16aee99b4
chore(dsrepl): clarify how to perform leadership transfer in runtime 2024-03-19 20:21:18 +01:00
Ivan Dyachkov 749ad73819
Merge pull request #12721 from emqx/dependabot/github_actions/actions-d8a5788e1a
chore(deps): bump the actions group with 3 updates
2024-03-19 20:16:03 +01:00
Andrew Mayorov 00d509f27b
feat(dsrepl): prefer local replica in read path
To optimize out any unnecessary RPCs. Given the load should be
smoothed evenly across the cluster, choosing non-leader node is
not a priority.
2024-03-19 20:11:42 +01:00
Andrew Mayorov 19305c223c
fix(dsrepl): require majority for replication-related tables 2024-03-19 20:11:42 +01:00
Andrew Mayorov f89909f60c
fix(dsrepl): tolerate trigger election timeouts for existing servers 2024-03-19 20:11:42 +01:00
Andrew Mayorov 6c6ea50e42
fix(boot): add `emqx_durable_storage` to the list of restarted apps 2024-03-19 20:11:42 +01:00
Andrew Mayorov 3b59cf2ebf
feat(dsrepl): move shard allocation to separate process
That starts shard and egress processes only when shards are fully
allocated.
2024-03-19 20:11:41 +01:00
Andrew Mayorov 4dafbf21f6
fix(dsrepl): make db + shard part of machine state
It doesn't feel right, but right now is the easiest way to have it
in the scope of `apply/3`, because `init/1` doesn't actually invoked
for ra machines recovered from the existing log / snapshot.
2024-03-19 20:11:41 +01:00
Andrew Mayorov d19128ed65
feat(dsrepl): cache shard metadata in persistent terms 2024-03-19 20:11:41 +01:00
Andrew Mayorov e6c2c2fb07
feat(dsrepl): manage generations / db config through ra machine 2024-03-19 20:11:39 +01:00
Andrew Mayorov 5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
Andrew Mayorov be793e4735
fix(dsrepl): reassign timestamp at the time of submission
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
Andrew Mayorov 146f082fdc
feat(dsrepl): implement raft-based replication
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
Ivan Dyachkov 76d14de480
Merge pull request #12722 from emqx/dependabot/github_actions/dot-github/actions/prepare-jmeter/actions-prepare-jmeter-0101c7941f
chore(deps): bump the actions-prepare-jmeter group in /.github/actions/prepare-jmeter with 2 updates
2024-03-19 19:14:36 +01:00
Ivan Dyachkov 923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
sync release-56
2024-03-19 19:09:41 +01:00
Andrew Mayorov 9d7d8edcc3
Merge pull request #12741 from keynslug/fix/flaky-ds-replay-testcase
test(ds): stabilize `t_session_replay_retry` testcase
2024-03-19 18:59:54 +01:00
Andrew Mayorov f453bdf055
chore(topic-api): describe `emqx_persistent_session_ds_router:stream/1` 2024-03-19 18:51:06 +01:00
Andrew Mayorov 027b237f07
test(ds): stabilize `t_session_replay_retry` testcase 2024-03-19 18:33:38 +01:00
Zaiming (Stone) Shi 73121f3ccc
Merge pull request #12730 from zmstone/0318-docs-fix-escape-in-i18n
docs: there is no need to escape quotes in triple-quotes
2024-03-19 18:30:14 +01:00
ieQu1 236685f714 fix(sessds): Channel must destroy sessions that are kicked 2024-03-19 17:49:04 +01:00
Kjell Winblad df8c7152fc refactor(mongodb connector): to use emqx_connector_info
This commit refactors the mongodb connector to use the
`emqx_connector_info` behavior.
2024-03-19 16:09:30 +01:00
Kjell Winblad c3d7d68cfc refactor(kinesis,pgsql,timescale,matrix connectors): emqx_connector_info
This commit refactors the kinesis, pgsql, timescale and matrix
connectors to use the `emqx_connector_info` behavior.
2024-03-19 15:57:23 +01:00
Ivan Dyachkov f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
feat: implement message validation
2024-03-19 11:12:28 -03:00
Andrew Mayorov 10ea367030
test(client-api): attempt to get rid of inter-testcase side effects 2024-03-19 14:56:50 +01:00
Andrew Mayorov 3129a8fa42
test(client-api): fix snabbkaffe crash noise in the CT logs 2024-03-19 14:55:28 +01:00
zmstone b8505ebd54 chore: upgrade to jesse 1.8.0 for draft-06 as default version 2024-03-19 14:47:11 +01:00
Kjell Winblad a3e631cda2 refactor(kafka connectors): to use emqx_connector_info
This commit refactors the `emqx_bridge_kafka` to use the
`emqx_connector_info` behavior. The `emqx_bridge_kafka` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
Kjell Winblad b8ef357fef refactor(http connector): to use emqx_connector_info
This commit refactors the `emqx_bridge_http` to use the
`emqx_connector_info` behavior. The `emqx_bridge_http` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
Kjell Winblad dc08fab0e8 refactor(hstreamdb connector): to use emqx_connector_info
This commit refactors the `emqx_bridge_hstreamdb` to use the
`emqx_connector_info` behavior. The `emqx_bridge_hstreamdb` related
information can thus be removed from `emqx_connector_ee_schema` and
`emqx_connector_schema`.
2024-03-19 14:41:20 +01:00
Kjell Winblad d0bebe8b93 refactor(gcp_pubsub connectors): to use emqx_connector_info
This commit refactors the emqx_bridge_gcp_pubsub app to use the
emqx_connector_info behavior. The emqx_bridge_gcp_pubsub
related information can thus be removed from emqx_connector_ee_schema
and emqx_connector_schema.
2024-03-19 14:40:33 +01:00
Kjell Winblad 76d32b6e97 refactor(confluent connector): to use emqx_connector_info
This commit refactors the emqx_bridge_confluent to use the
emqx_connector_info behavior. The emqx_bridge_confluent related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:43 +01:00
Kjell Winblad 795b668c4f refactor(azure_event_hub connector): to use emqx_connector_info
This commit refactors the emqx_bridge_azure_event_hub to use the
emqx_connector_info behavior. The emqx_bridge_azure_event_hub related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:17 +01:00
Thales Macedo Garitezi c478e375f6 refactor: change failure log configuration schema 2024-03-19 10:04:18 -03:00
JianBo He c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
Thales Macedo Garitezi 2eac54bf54 chore: rm `move` API
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
SergeTupchiy 92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
Andrew Mayorov e1b3263177
fix(sessds): make dialyzer a tad happier 2024-03-19 13:11:32 +01:00