Commit Graph

10876 Commits

Author SHA1 Message Date
Andrew Mayorov 2b1231b980
fix(s3-bridge): ensure bridge and connector names are identical 2024-02-12 13:46:54 +01:00
Andrew Mayorov 0edc5b5992
docs(s3): fix README example 2024-02-12 10:44:18 +01:00
Kjell Winblad 976099f5fb fix: cleanups due to problems found by @thalesmg 2024-02-12 10:16:17 +01:00
Zaiming (Stone) Shi d4fb3e83f6 chore: upgrade to wolff 1.10.2
Wolff 1.10.2 added an enhancement which makes Kafka resource health
checks more lightweight and lowers the chance of false resource alerts.
2024-02-12 09:05:44 +01:00
Ilya Averyanov 90fd2b26d3 feat(banned): allow ban by clientid/username regexps, peerhost cidrs 2024-02-10 17:59:22 +03:00
Andrew Mayorov c108262771
chore(s3-bridge): use `emqx_connector_schema` helper for brewity 2024-02-09 16:08:33 +01:00
Andrew Mayorov f476f4343b
fix(release): add `emqx_bridge_s3` to applications list 2024-02-09 14:16:32 +01:00
Andrew Mayorov 2a4e37869e
fix(dashboard): provide full context in startup errors 2024-02-09 14:16:32 +01:00
Andrew Mayorov 802c760406
feat(s3): introduce S3 connector and action
This is a trivial connector based on `emqx_s3` and simple action
that maps each incoming event into an S3 object upload. Due to
current `emqx_s3` limitation this bridge is compatible with
backends providing S3 API with path-style bucket access.
2024-02-09 14:16:32 +01:00
SergeTupchiy 755b59b7fe
Merge pull request #12494 from SergeTupchiy/EMQX-11825-mongodb-connector-perf
Improve mongodb connector performance
2024-02-09 13:12:12 +02:00
Zaiming (Stone) Shi cbebc59115
Merge pull request #12492 from zmstone/0208-fix-return-receive-maximum-in-connack
fix(emqx_channel): return Receive-Maximum in CONNACK when no error
2024-02-09 12:08:46 +01:00
Andrew Mayorov 8f66bd9ddf
fix(s3-client): make log levels saner 2024-02-09 12:03:41 +01:00
Andrew Mayorov 4ff04ab1f3
feat(s3): separate concepts to make app reusable in bridges 2024-02-09 12:03:41 +01:00
Zaiming (Stone) Shi a74c828e19 test: add test case to cover Receive-Maximum in CONNACK 2024-02-09 10:42:48 +01:00
Andrew Mayorov a5266f68ec
feat(s3): switch schema to use secrets with loader support
This will make applications using `emqx_s3` follow the same
conventions as bridges and support loading secrets from files
at runtime.
2024-02-08 22:43:20 +01:00
Serge Tupchii 1d9ce709f4 perf: un-deprecate mongodb topology pool_size and set it to 10 by default
Benchmarks proved that it effects performance:
10 ecpool workers / 10 topology pool workers outperform
100 ecpool / 1 topology worker.

Fixes: EMQX-11825
2024-02-08 20:39:19 +02:00
Serge Tupchii c9ee5addb4 perf: upgrade mongodb-erlang to v3.0.23
Fixes: EMQX-11825
2024-02-08 20:00:26 +02:00
Thales Macedo Garitezi 64367d834b test(bridge_v2): actually check schema 2024-02-08 13:39:31 -03:00
Zaiming (Stone) Shi cf22692c74 fix(emqx_channel): return Receive-Maximum in CONNACK when no error 2024-02-08 17:05:16 +01:00
Kjell Winblad e284a83f73 feat: refactor RocketMQ bridge to connector and action
Fixes:
https://emqx.atlassian.net/browse/EMQX-11467
2024-02-08 16:21:58 +01:00
ieQu1 811edb32a2
Merge pull request #12485 from ieQu1/dev/ds-fix-unsubscribe
fix(ds): Fix unsubscribe logic related to the replay
2024-02-07 21:38:25 +01:00
ieQu1 19c6d1127f
refactor(sessds): Extract subscription mgmt logic to separate module 2024-02-07 15:27:11 +01:00
ieQu1 3000a8f286
fix(sessds): Postpone deletion of the subscription until fully acked 2024-02-07 14:51:02 +01:00
lafirest 3f13aed3de
Merge pull request #12481 from lafirest/fix/action_info
fix(tdengine): remove redundant codes
2024-02-06 11:39:25 +08:00
zhongwencool eace8f2fcb
Merge pull request #12423 from zhongwencool/rabbitmq-e560
feat: rabbitmq bridge v2 integration
2024-02-06 09:55:05 +08:00
ieQu1 30eb54e86b fix(sessds): Delay unsubscribe until full ack of in-flight messages 2024-02-06 02:49:07 +01:00
ieQu1 a9c55f7568 feat(sessds): Consider #srs with only QoS0 messages fully acked 2024-02-06 02:49:07 +01:00
zhongwencool d3db53626b
Merge pull request #12483 from zmstone/0205-docs-fix-typos
Improve emqx ctl conf cluster_sync CLI and help text
2024-02-06 09:11:35 +08:00
zhongwencool 3e599e5d8e chore: add maybe feature enable on rabbitmq_bridge 2024-02-06 07:57:13 +08:00
zhongwencool e3cb49db8e chore: add copyright header 2024-02-06 07:57:13 +08:00
zhongwencool 688701eedb chore: add desc for rabbitmq source 2024-02-06 07:57:13 +08:00
zhongwencool b444c82a42 feat: rabbitmq bridge v2 integration 2024-02-06 07:57:12 +08:00
ieQu1 280fcd8c52
Merge pull request #12437 from ieQu1/dev/optimize_make_filter
Optimize emqx_ds_bitmask_keymapper:make_filter function.
2024-02-05 17:32:28 +01:00
ieQu1 4665837cf0 fix(ds): Apply review remarks 2024-02-05 16:52:06 +01:00
Zaiming (Stone) Shi d7b0456bb0 feat: rename emqx ctl cluster_sync 'tnxid' to 'inspect' 2024-02-05 16:50:55 +01:00
Thales Macedo Garitezi e5b14a38ed
Merge pull request #12472 from thalesmg/fix-bridgev2-api-multicall-m-20240202
fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
2024-02-05 10:28:28 -03:00
ieQu1 c7888ad1f1
Merge pull request #12475 from ieQu1/dev/lean-stream
Use a more compact data structure to represent streams
2024-02-05 13:55:24 +01:00
Thales Macedo Garitezi e1ff276d76
Merge pull request #12471 from thalesmg/fix-config-upgrade-m-20240202
fix(config): apply config upgrade to deprecated configs
2024-02-05 09:05:12 -03:00
firest b4a6be0e7c fix(tdengine): remove redundant codes 2024-02-05 18:50:31 +08:00
lafirest 571786b6c5
Merge pull request #12480 from lafirest/fix/td_v1_schema
fix(tdengine): keep the bridge schema still compatible with v1
2024-02-05 18:17:55 +08:00
ieQu1 698ba3f271 fix(ds): Optimize emqx_ds_bitmask_keymapper:make_filter
This optimization makes idle polling faster
2024-02-05 10:54:19 +01:00
ieQu1 8edbec5929 refactor(ds): Clarify the language used in ds_bitmapper 2024-02-05 10:54:18 +01:00
Zaiming (Stone) Shi 75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache
Relicense apps/emqx_durable_storage under Apache 2.0
2024-02-05 10:16:52 +01:00
firest 4a2c5a3161 fix(tdengine): keep the bridge schema still compatible with v1 2024-02-05 13:22:17 +08:00
zhongwencool f7b0d1bc9d chore: add duplicated key assert in swagger.json 2024-02-04 11:53:37 +08:00
JimMoen b1150a5f88
fix(schema): bridge oracle connector schema 2024-02-04 10:53:13 +08:00
ieQu1 2e2f3cb2aa fix(sessds): Avoid stream hash collisions 2024-02-03 21:15:54 +01:00
ieQu1 2e56810ea2 refactor(ds): Use a simple improper list to represent the streams 2024-02-03 21:15:54 +01:00
Thales Macedo Garitezi 76d242df9b fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
Fixes https://emqx.atlassian.net/browse/EMQX-11834
2024-02-02 17:36:08 -03:00
Thales Macedo Garitezi 76401a302a fix(config): apply config upgrade to deprecated configs
Fixes https://emqx.atlassian.net/browse/EMQX-11845

Fixes https://github.com/emqx/emqx/issues/12452
2024-02-02 15:12:08 -03:00
Ivan Dyachkov ed8660cb68
Merge pull request #12458 from id/0202-sync-release-55
sync release-55
2024-02-02 17:20:38 +01:00
SergeTupchiy 81433282be
Merge pull request #12467 from SergeTupchiy/autocluster-dns-aaaaa-record-type
feat(emqx_conf): support AAAA cluster dns record type
2024-02-02 18:13:06 +02:00
Zaiming (Stone) Shi b1a05c7b59
Merge pull request #12326 from zmstone/0111-unregister-session-with-timestamp
0111 unregister session with timestamp
2024-02-02 17:00:23 +01:00
ieQu1 e0b5d9fae7
Merge pull request #12464 from ieQu1/dev/feat-ds-cli
feat(ds): Add a CLI interface to inspect status of DS databases
2024-02-02 14:14:48 +01:00
Serge Tupchii 5e0c3d4051 feat(emqx_conf): support AAAA cluster dns record type
Ekka autocluster DNS already supports AAAA records, so it's only needed to update enum in schema.
2024-02-02 12:13:29 +02:00
Kjell Winblad fa0baf2f38
Merge pull request #12439 from kjellwinblad/kjell/refactor/oracle_bridge/EMQX-11464
feat: refactor Oracle bridge to connector and action
2024-02-02 10:53:15 +01:00
Ivan Dyachkov 84b6d7d720 fix: remove 5.5.bpapi2 file 2024-02-02 10:38:50 +01:00
ieQu1 3d2ac97c61 feat(ds): Add a CLI interface to inspect status of DS databases 2024-02-02 10:11:01 +01:00
Zaiming (Stone) Shi f0569d8ae8 refactor: use mria:async_dirty to group dirty ops 2024-02-02 09:26:43 +01:00
lafirest 24d8a9ab00
Merge pull request #12449 from lafirest/feat/td
feat(tdengine): improve the TDengine bridge to v2 style
2024-02-02 16:17:01 +08:00
Ivan Dyachkov dd490de2e1 chore: emqx 5.5.0 2024-02-02 08:48:56 +01:00
Ivan Dyachkov f1c7e716ce chore: 5.5.0-rc.2 2024-02-02 08:48:56 +01:00
JianBo He 9aad7997ca chore: compatible the contet-type sytanx 2024-02-02 08:48:56 +01:00
Ivan Dyachkov dae3a94670 chore: 5.5.0-rc.1 2024-02-02 08:48:56 +01:00
lafirest c724d2127e
Merge pull request #12457 from lafirest/fix/opentsdb
fix(opents): change the schema of tags to object style
2024-02-02 15:33:50 +08:00
JimMoen 17d79026cd
test: fix emqx_connector_schema eunit missing fields 2024-02-02 14:43:25 +08:00
firest 5291adf2c0 fix(opents): change the schema of tags to object style 2024-02-02 13:07:49 +08:00
JimMoen 1a372ce39a
fix(swagger): no-need call common_fields when `emqx_connector_schema:api_fields/3` used 2024-02-02 10:54:49 +08:00
firest 2241461acb chore: update change & bump version 2024-02-02 07:48:44 +08:00
Kjell Winblad af3f43ffd5
fix: use emqx_resource macro for status
Thanks for the suggestion @thalesmg

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 17:24:25 +01:00
ieQu1 98d1094d73 feat(sessds): Expose subscriptions in the REST API 2024-02-01 16:47:57 +01:00
Kjell Winblad 016fbd2c5c chore: remove commented out code 2024-02-01 16:18:37 +01:00
firest dfad020c49 feat(tdengine): improve the TDengine bridge to v2 style 2024-02-01 23:14:24 +08:00
Kjell Winblad a4272c71dc feat: refactor Oracle bridge to connector and action
Fixes:
https://emqx.atlassian.net/browse/EMQX-11464
2024-02-01 16:03:05 +01:00
Ilya Averyanov ae1ef9d1bb
Merge pull request #12428 from savonarola/0130-simplify-boot-machine-runtime-deps
chore(emqx_machine): refactor injecting runtime deps
2024-02-01 16:32:53 +02:00
William Yang 80ff1b1657
Merge pull request #12434 from qzhuyan/fix/william/kafka-dont-stacktrace-when-no-partiion-count
kafka-bridge: dont stacktrace when no partition count
2024-02-01 15:09:45 +01:00
Zaiming (Stone) Shi 3e518c1876
Merge pull request #12427 from zmstone/0130-limit-kafka-partitions
0130 limit kafka partitions
2024-02-01 14:39:52 +01:00
Ilya Averyanov 6a092aeb69 chore(emqx_machine): refactor injecting runtime deps
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 16:13:49 +03:00
William Yang 16444b7da9 fix(bridge-kafka): no stacktrace when fetch topic partition count fail 2024-02-01 14:09:15 +01:00
zhongwencool 42a4bc3382
Merge pull request #12435 from zhongwencool/dont-add-disable-resource-to-connector
fix: don't add disable bridge to connector's channel
2024-02-01 10:39:47 +08:00
JianBo He 0d5911d942
Merge pull request #12422 from zhongwencool/upgrade-erlfmt-to-v1.3.0
feat: upgrade erlfmt to support maybe syntax
2024-02-01 09:34:33 +08:00
zhongwencool f9d16340e0 fix: don't add disable bridge to connector's channel 2024-02-01 09:12:43 +08:00
ieQu1 139f5dc3bd chore(ds): Remove an obsolete document; superceded by ./README.md 2024-02-01 00:14:08 +01:00
ieQu1 b50d6bf1fd chore(ds): Change the license to Apache 2.0
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
2024-02-01 00:10:48 +01:00
Thales Macedo Garitezi f7b12470dd test(ds): fix inter-suite flakiness
Attempt to mitigate this frequent source of flakiness:

```
=CRASH REPORT==== 31-Jan-2024::17:30:15.025404 ===
  crasher:
    initial call: emqx_ds_replication_layer_egress:init/1
    pid: <0.11312.0>
    registered_name: []
    exception error: no match of right hand side value {error,
                                                        no_leader_for_shard}
      in function  emqx_ds_replication_layer_egress:init/1 (/emqx/apps/emqx_durable_storage/src/emqx_ds_replication_layer_egress.erl, line 93)
      in call from gen_server:init_it/2 (gen_server.erl, line 980)
      in call from gen_server:init_it/6 (gen_server.erl, line 935)
    ancestors: [<0.11310.0>,<0.11304.0>,emqx_ds_builtin_databases_sup,
                  emqx_ds_builtin_sup,emqx_ds_sup,<0.11236.0>]
    message_queue_len: 0
    messages: []
    links: [<0.11310.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 376
    stack_size: 28
    reductions: 231
  neighbours:
```
2024-01-31 15:42:26 -03:00
Thales Macedo Garitezi d51deac222 fix(ds): use configured data dir for site storage 2024-01-31 15:42:26 -03:00
Zaiming (Stone) Shi 3b6fbff9a5 test: fix compile warnings 2024-01-31 19:32:57 +01:00
Zaiming (Stone) Shi 3b42a7425b feat(kafka_producer): add partitions_limit 2024-01-31 19:26:01 +01:00
Zaiming (Stone) Shi 53c217c383 refactor: micro optimization 2024-01-31 19:25:57 +01:00
Ilya Averyanov b3657f1892
Merge pull request #12436 from savonarola/0131-fix-stream-replay
fix(sessds): fix renew stream logic
2024-01-31 18:25:17 +02:00
lafirest a60aeed539
Merge pull request #12398 from lafirest/feat/api-docs
feat(dashboard): expose the `swagger_support` option
2024-01-31 23:31:56 +08:00
Ilya Averyanov fe46434687 fix(sessds): fix renew stream logic 2024-01-31 18:02:29 +03:00
Kjell Winblad 2680fae9b6
Merge pull request #12420 from kjellwinblad/kjell/fix/kinesis/refactoring/problems/thanks_HJianBo
fix: kinesis schema problems found by @HJianBo
2024-01-31 15:52:45 +01:00
zhongwencool e9c8446d57 feat: upgrade erlfmt to support maybe syntax 2024-01-31 20:06:49 +08:00
Zaiming (Stone) Shi 38047108a4 test: add test coverage for emqx_cm_registry_keeper module 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 209331ad33 test: fix config test 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 87a2368e37 feat(metrics): add cluster_session guage 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 509ab6f35a feat(api): add /sessions_count api to count sessions 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 562a2736ae feat: add `broker` root to hot-config schema 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi e9318752e6 feat: store session unregistration timestamp in emqx_cm_registry table 2024-01-31 09:50:12 +01:00