Commit Graph

11596 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 415d27a4c9
Merge pull request #12544 from zmstone/0220-delete-non-prod-code
0220 delete non prod code in emqx_resource.erl
2024-02-21 14:08:08 +01:00
Thales Macedo Garitezi 4118fcfdc6
Merge pull request #12486 from thalesmg/gcp-consumer-source-m-20240205
feat: convert `gcp_pubsub_consumer` to connector/source
2024-02-21 10:06:26 -03:00
Thales Macedo Garitezi 25cdc2a04b chore: bump version to 5.5.1-rc.1 2024-02-21 09:57:05 -03:00
Zaiming (Stone) Shi b14e51d7c4 docs: delete useless comments 2024-02-21 13:33:29 +01:00
Thales Macedo Garitezi 03138f8345 refactor(mgmt): move persistent session counting function to mgmt module
Follow up to https://github.com/emqx/emqx/pull/12500#discussion_r1496634087
2024-02-21 09:12:56 -03:00
Thales Macedo Garitezi 529211b9ac
Merge pull request #12500 from thalesmg/ds-list-client-api-m-20240209
feat(ds): list disconnected persistent sessions in clients API
2024-02-21 09:02:27 -03:00
Andrew Mayorov 92c00ee21a
Merge pull request #12550 from keynslug/fix/bridge-s3-resopts
fix(bridge-s3): expose connector-level `resource_opts` properly
2024-02-21 11:37:55 +01:00
Zaiming (Stone) Shi 333eb34300 test: call emqx_resource:remove_local 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi de93d1c0a3 chore: add bpapi.versions 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 9c1d5e49dd chore: fix a typo 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi c8f9e12de1 refactor: deprecate emqx_resource_proto_v1 with v2 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 06e0a7f2ed refactor: delete emqx_resource:remove api
this api is only used in tests,
changed to call the _local flavor instead
2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 8541d2efc1 refactor: delete emqx_resource:recreate
this api is only used in tests, changed to call the _local flavor
instead
2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 1878fded5a refactor: delete emqx_resource:create/4 and /5
the functions are only used in tests,
now the tests are rewritten to call create_local/4 and /5 instead
2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 0cbe626aed
Merge pull request #12549 from zmstone/0221-sync-release-55
Sync `release-55` with i18n fixes to master
2024-02-21 10:52:36 +01:00
Andrew Mayorov d8b6ecd185
fix(bridge-s3): expose connector-level `resource_opts` properly 2024-02-21 10:49:04 +01:00
zhongwencool ee02079661
Merge pull request #12548 from zhongwencool/fix-rabbitmq-source-crash
fix: bridge_v1 http api crash when create rabbitmq source
2024-02-21 16:34:43 +08:00
SergeTupchiy f7a7b7c8e0
Merge pull request #12545 from SergeTupchiy/fix-typo-tls_gc
chore(emqx_tls_certfile_gc): fix typo
2024-02-21 10:11:13 +02:00
Zaiming (Stone) Shi da340c92a1 Merge remote-tracking branch 'origin/master' into 0221-sync-release-55 2024-02-21 09:11:12 +01:00
Zaiming (Stone) Shi 80be84c401
Merge pull request #12491 from zmstone/docs-sync-changes-for-i18n
docs: fix i18n
2024-02-21 09:10:25 +01:00
zhongwencool 2a7a662c03 fix: bridge_v1 http api crash when create rabbitmq source 2024-02-21 15:30:05 +08:00
Thales Macedo Garitezi 65cfece0a3 chore: delete 5.5.bpapi2
> This particular warning won't be seen in the real life. Also, we need to start storing
bpapi files for each minor release, since the plan to use hot upgrade has been abandoned.

> ...The original procedure for BPAPI life cycle was designed under the assumption that
we'll support hot upgrade between minor releases, so we'll only ever need to keep one
BPAPI file per major release, and rewrite it whenever a new minor version is
released. We'll need to revisit it.

https://emqxio.slack.com/archives/C04H4RHKKEY/p1706862980546309
2024-02-20 17:35:28 -03:00
Thales Macedo Garitezi d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
Thales Macedo Garitezi 74dbbf6d9d Merge branch 'release-55' into sync-r55-m-20240220 2024-02-20 16:33:34 -03:00
Serge Tupchii dd9f96211b chore(emqx_tls_certfile_gc): fix typo 2024-02-20 21:19:36 +02:00
Thales Macedo Garitezi 3a4c7f60e2 feat(ds): list disconnected persistent sessions in clients API
Fixes https://emqx.atlassian.net/browse/EMQX-11540

Note that not all information provided by disconnected in-memory sessions is available to
disconnected persistent sessions, nor does all of them make sense.
2024-02-20 14:52:55 -03:00
Zaiming (Stone) Shi 1de9e9dd01
Merge pull request #12541 from zmstone/0220-fail-fqdn-node-name-if-dns-discovery
Fail fast if node host is not IP address but cluster strategy is `dns` with `a` or `aaaa` record type
2024-02-20 17:48:02 +01:00
Thales Macedo Garitezi b6689d178f fix(http_bridge): redact headers
Fixes https://emqx.atlassian.net/browse/EMQX-11864

```
2024-02-20T15:51:34.744509+00:00 [debug] msg: http_connector_received, mfa: emqx_bridge_http_connector:on_query_async/4(471), tag: QUERY_ASYNC, request: {"/bah",[{<<"Authorization">>,<<"******">>},{<<"content-type">>,<<"application/json">>},{<<"authorization">>,<<"******">>},{<<"content-type">>,<<"application/json">>}],<<"******">>}, state: #{port => 18083,request => #{path => [<<>>],body => undefined,headers => [{[<<"content-type">>],[<<"application/json">>]},{[<<"Authorization">>],#Fun<emqx_secret.0.85191762>}],method => [<<"undefined">>],max_retries => 2,request_timeout => 30000},host => "localhost",connect_timeout => 15000,pool_type => random,pool_name => <<"connector:http:p1">>,base_path => "/",installed_actions => #{<<"action:http:p1:connector:http:p1">> => #{path => [<<"/bah">>],body => undefined,headers => [{[<<"authorization">>],#Fun<emqx_secret.0.85191762>},{[<<"content-type">>],[<<"application/json">>]}],method => [<<"post">>],max_retries => 2,request_timeout => 30000,render_template_func => fun emqx_bridge_http_connector:render_template/2}}}, connector: <<"connector:http:p1">>, note: the request body is redacted due to security reasons
```
2024-02-20 12:57:13 -03:00
Zaiming (Stone) Shi dc015e7a6f fix(schema): validate cluster strategy and node name
If cluster strategy is configured as `dns`, the node name
must be IP address
2024-02-20 16:09:54 +01:00
SergeTupchiy f72558737c
Merge pull request #12520 from SergeTupchiy/EMQX-11530-log-throttling
feat: implement log throttling
2024-02-20 15:52:46 +02:00
Kjell Winblad 86c85f3b45
Merge pull request #12512 from kjellwinblad/kjell/refactor/hstreamdb/EMQX-11458
feat: refactor HStreamDB bridge to connector and action
2024-02-20 13:30:29 +01:00
Serge Tupchii 9bd0d1ba1b feat: enable log throttling for potentially flooding log events 2024-02-20 14:20:21 +02:00
Serge Tupchii 98ba300f7c feat: implement log throttling 2024-02-20 13:23:17 +02:00
zhongwencool 9639a977a3 fix: hidden batch_time for mongodb 2024-02-20 09:42:49 +08:00
zhongwencool c7bfaf51fd chore: typo error (templete -> template) 2024-02-20 09:41:09 +08:00
zhongwencool 2561ff94ec fix: check channel id exist when get_channel_status 2024-02-20 09:38:33 +08:00
Thales Macedo Garitezi 146d89bd89 feat: convert `gcp_pubsub_consumer` to connector/source
Fixes https://emqx.atlassian.net/browse/EMQX-11471
2024-02-19 17:54:26 -03:00
Zaiming (Stone) Shi fd13a661cf
Merge pull request #12535 from zmstone/0219-refactor-delete-non-prod-code
refactor: delete non-prod code
2024-02-19 19:04:59 +01:00
Thales Macedo Garitezi 11e4a295ed fix: use connector type for constructing resource id 2024-02-19 14:42:35 -03:00
Thales Macedo Garitezi 9a32895a1a feat: convert `gcp_pubsub_consumer` to connector/source
Fixes https://emqx.atlassian.net/browse/EMQX-11471
2024-02-19 14:42:35 -03:00
Zaiming (Stone) Shi ad53523e6b refactor: delete non-prod code
So far the retainer backend type is always `built_in_database`.
The slightly over-engineered pre-implementation to support
another backend is likely not going to fly as the EMQX
resource frame work is mostly for auth and data integration.
i.e. not generic enough for retained messages.
2024-02-19 14:36:12 +01:00
Zaiming (Stone) Shi 68baa4dcaf docs: fix i18n 2024-02-19 13:57:44 +01:00
Thales Macedo Garitezi bee7fa630b chore: bump app vsns 2024-02-19 09:05:13 -03:00
Thales Macedo Garitezi dbb6725a47 fix(config): apply config upgrade to deprecated configs
Fixes https://emqx.atlassian.net/browse/EMQX-11845

Fixes https://github.com/emqx/emqx/issues/12452
2024-02-19 09:02:43 -03:00
Kjell Winblad def95aa22b docs(HStreamDB bridge): make pool size descriptions better 2024-02-19 11:33:37 +01:00
Zaiming (Stone) Shi 866fea7648
Merge pull request #12530 from zmstone/0217-improve-frame-too-large-logging-context
0217 improve frame too large logging context
2024-02-19 11:33:31 +01:00
zhongwencool 2ac4bde102
Merge pull request #12487 from zhongwencool/rabbitmq-source-test
feat: don't publish mqtt message in rabbitmq's source
2024-02-19 17:36:12 +08:00
Zaiming (Stone) Shi 8b0e15e402 refactor: rename 'hint' to 'cause' for MQTT fram parse failure reason
'reason' is maybe the wrapping field's name, so it was not used.
'hint' however, per our logging convention, is usually a free text
description for human to read.
change to 'cause' here because the field is always an atom and
it's use as shutdown counter in esockd_connection_sup
2024-02-19 09:38:35 +01:00
Xinyu Liu 0fef19f86f
Merge pull request #12425 from emqx/clickhouse-bridge-v2
refactor: split clickhouse bridges to actions and connectors
2024-02-19 09:20:59 +08:00
Zaiming (Stone) Shi 668d3be390 test: for better test coverage 2024-02-18 19:23:35 +01:00
Zaiming (Stone) Shi 22dabcb3ea feat(mqtt): add more logging context for frame_too_large error 2024-02-18 18:33:12 +01:00
Shawn ce0c27e802 fix: correct the swagger example for clickhouse connectors 2024-02-18 14:49:58 +08:00
JimMoen 6aae3ba2ed
refactor(prom_test): assert fun prefix 2024-02-18 10:42:40 +08:00
Shawn 8ae0e78786 refactor: split clickhouse bridges to actions and connectors 2024-02-18 10:01:53 +08:00
Shawn 451c8ecaf5 ci: add env vars to run clickhouse tests locally 2024-02-18 10:00:47 +08:00
JimMoen c61b558423
test(prom): start mock pushgateway 2024-02-18 03:39:18 +08:00
JimMoen ba1d24d054
test(prom_api): '/prometheus/auth' and '/prometheus/data_integration' 2024-02-18 02:32:25 +08:00
JimMoen 7bfabd7865
refactor: meck license checker for prom cases 2024-02-18 02:32:21 +08:00
JimMoen d45d925529
test(prom_api): `/prometheus/stats` in prom/json format 2024-02-18 02:14:36 +08:00
JimMoen 9e16f33c31
test: `/prometheus/stats` endpoint format in json 2024-02-18 02:14:34 +08:00
Zaiming (Stone) Shi 52cfeee2b1
Merge pull request #12526 from zmstone/0216-fix-flaky-tests
test(listeners): try to stop all listeners after each test
2024-02-16 15:33:04 +01:00
Zaiming (Stone) Shi 9ffda0187f
Merge pull request #12525 from zmstone/0216-docs-add-ref-to-bridge_mode
docs: add a reference to the origin of is_bridge and bridge_mode
2024-02-16 15:32:52 +01:00
ieQu1 b7f3f48d38
Merge pull request #12524 from emqx/fix/bridge-stop-snabbkaffe
test(bridge_mqtt): Stop snabbkaffe servers
2024-02-16 15:17:29 +01:00
Zaiming (Stone) Shi 2b46cbab7a
Merge pull request #12517 from zmstone/0213-hocon-multiline-string-with-indentation
0213 hocon multiline string with indentation
2024-02-16 15:12:46 +01:00
Zaiming (Stone) Shi f78c30c9ff test: reduce false warning messages in test logs 2024-02-16 14:38:51 +01:00
Zaiming (Stone) Shi 3fa262f9ca test(listeners): try to stop all listeners after each test
- Reduce the `eaddrinuse` flakiness
- Use ?FUNCTION_NAME as listener name to avoid name clashing between
  tests
- Call emqx:remove_config for adhoc listeners created for testing
  tombestone is designed for default listeners
2024-02-16 14:37:06 +01:00
ieQu1 8cfb22f0b8
fix(ds): Retry getting the shard leader 2024-02-16 12:42:48 +01:00
Zaiming (Stone) Shi 17a0513962 docs: add a reference to the origin of is_bridge and bridge_mode 2024-02-16 12:24:57 +01:00
ieQu1 59e4db98f7
test(bridge_mqtt): Stop snabbkaffe servers 2024-02-16 12:14:19 +01:00
Zaiming (Stone) Shi 497f631b7c
Merge pull request #12522 from zmstone/0216-upgrade-kafka-protocol
chore: upgrade kafka_protocol from 4.1.3 to 4.1.5
2024-02-16 11:47:31 +01:00
Zaiming (Stone) Shi f57f617ba3 refactor(schema): ensure roots/0 and namespace/0 for all schema modules 2024-02-16 11:35:32 +01:00
Andrew Mayorov a945892c52
Merge pull request #12508 from keynslug/fix/explicit-tab-mgmt
feat: manage mria tables explicitly during apps startup
2024-02-16 11:11:59 +01:00
Zaiming (Stone) Shi 242df0ffe9 chore: upgrade kafka_protocol from 4.1.3 to 4.1.5 2024-02-16 10:33:53 +01:00
Zaiming (Stone) Shi a389d78b42 chore: delete stale code 2024-02-16 10:32:13 +01:00
Zaiming (Stone) Shi b24321a8fc feat: upgrade to hocon-0.41.0
hocon 0.41.0 added support for multiline string indentation.
now there is no need to escape (unless there is backslash)
quotes and line feeds etc.
2024-02-16 10:32:13 +01:00
ieQu1 ab2a469aff
Merge pull request #12489 from ieQu1/dev/ds-retainer
Durable sessions support retained messages
2024-02-15 21:59:32 +01:00
Andrew Mayorov 4484578c1f
Merge pull request #12507 from keynslug/fix/gw-app-no-scan
fix(gateway): avoid scanning modules of known gateway apps
2024-02-15 17:11:49 +01:00
ieQu1 5f85105801 feat(sessds): Specialize the interval queue for positive numbers 2024-02-15 16:55:05 +01:00
ieQu1 3adbe65a58 refactor(sessds): Unify logic for QoS 1 and 2 2024-02-15 11:34:13 +01:00
ieQu1 c781240459 feat(sessds): Add support for the retainer
Note: this is currently not ideal. Retained messages won't be
redelivered.
2024-02-15 11:34:13 +01:00
Kjell Winblad 5e442fcbca fix: labels and descriptions 2024-02-15 11:08:08 +01:00
Andrew Mayorov 6514659733
test(mgmt): update and simplify testsuites setup 2024-02-15 11:04:58 +01:00
ieQu1 94254ec05b feat(sessds): Correct handling of gaps in the seqno series 2024-02-15 10:49:02 +01:00
Serge Tupchii 0d0e26d6af fix: set `info` level for potentially flooding log events 2024-02-15 11:02:03 +02:00
Andrew Mayorov cffb52ab28
test(plugins): update and simplify test suite 2024-02-14 19:00:04 +01:00
Kjell Winblad 4e8dfb48b7 fix: elvis problems 2024-02-14 17:54:14 +01:00
Kjell Winblad a153d758c3 feat: refactor HStreamDB bridge to connector and action
This commit also upgrades the hstreamdb_erl driver library and change
the action/bridge to use the new hstreamdb_erl.

Much of the code for the new API is copied from:
be1a1604dd/lib-ee/emqx_rule_actions/src/emqx_backend_hstreamdb_actions.erl

Fixes:
https://emqx.atlassian.net/browse/EMQX-11458
2024-02-14 17:34:35 +01:00
Serge Tupchii 7272ef25d4 feat(emqx_auth): implement API to re-order all authenticators/authz sources
Fixes: EMQX-11770
2024-02-14 14:35:46 +02:00
Andrew Mayorov 35bf805504
test(exhook): update and simplify testsuites setup 2024-02-13 21:02:39 +01:00
Andrew Mayorov ece1c6c6dc
test(prometheus): update and simplify testsuites setup 2024-02-13 20:36:23 +01:00
Andrew Mayorov a64850a84b
test(sso): update and simplify testsuites setup 2024-02-13 20:36:23 +01:00
Andrew Mayorov b15e81baf2
test(slowsub): update and simplify testsuite setup 2024-02-13 20:36:23 +01:00
Andrew Mayorov d0e507eba4
test(trie): fix testsuite setup 2024-02-13 20:36:22 +01:00
Andrew Mayorov 841fdea124
test(conf): update and simplify some testcases 2024-02-13 20:36:22 +01:00
Andrew Mayorov 0a9cbe3080
test(license): update and simplify tests 2024-02-13 20:36:22 +01:00
Andrew Mayorov 24dfa41722
feat(cth-suite): use cheaper heuristic for schema modules 2024-02-13 20:36:22 +01:00
Andrew Mayorov 8f2a4f7b19
fix(cth-suite): use cheaper check for loaded applications 2024-02-13 20:36:22 +01:00
Andrew Mayorov 82a4e6ef68
feat(psk): manage mria tables explicitly during startup
Also switch test suite to `emqx_cth_suite` tooling.
2024-02-13 20:36:22 +01:00
Andrew Mayorov b07df487f0
feat(modules): manage mria tables explicitly during startup 2024-02-13 20:36:22 +01:00
Andrew Mayorov b6d77c164e
feat(emqx-mgmt): manage mria tables explicitly during startup 2024-02-13 20:36:22 +01:00
Andrew Mayorov 21780e2126
feat(emqx-conf): manage mria tables explicitly during startup 2024-02-13 20:36:21 +01:00
Andrew Mayorov 8f7b0ac498
docs(jt808): fix doc references 2024-02-13 16:27:41 +01:00
Kjell Winblad 366827390e
Merge pull request #12488 from kjellwinblad/kjell/refactor/rocketmq_bridge/EMQX-11467
feat: refactor RocketMQ bridge to connector and action
2024-02-13 16:19:21 +01:00
Andrew Mayorov 0e1d27c836
fix(gateway): avoid scanning modules of known gateway apps
Since the set of known gateways is already predefined in `emqx_gateway_utils`
it makes no sense to scan over all modules just to find out the same set of
modules. Cutting out this scanning saves few seconds per each `emqx_conf`
application startup, which is especially noticeable when running tests.
2024-02-13 15:13:36 +01:00
Andrew Mayorov 6fbb6f6846
Merge pull request #12506 from keynslug/fix/EMQX-11830/recoverable
fix(s3-bridge): handle recoverable AWS errors
2024-02-13 12:44:23 +01:00
Andrew Mayorov eff149e676
feat(emqx-auth-mnesia): manage mria tables explicitly during startup 2024-02-12 19:17:22 +01:00
Andrew Mayorov 21a5751575
feat(emqx): manage mria tables explicitly during startup 2024-02-12 19:17:09 +01:00
Andrew Mayorov a4eac75b25
fix(s3-bridge): handle recoverable AWS errors 2024-02-12 18:20:03 +01:00
Ilya Averyanov edd28be090
Merge pull request #12499 from savonarola/0209-emqx-banned-extended-rules
feat(banned): allow ban by clientid/username regexps, peerhost cidrs
2024-02-12 17:12:47 +02:00
Andrew Mayorov a9fdf9f1f9
Merge pull request #12495 from keynslug/ft/EMQX-11830/s3-bridge-simple
feat(s3): introduce S3 connector and action
2024-02-12 16:05:22 +01:00
Andrew Mayorov 2b1231b980
fix(s3-bridge): ensure bridge and connector names are identical 2024-02-12 13:46:54 +01:00
Andrew Mayorov 0edc5b5992
docs(s3): fix README example 2024-02-12 10:44:18 +01:00
Kjell Winblad 976099f5fb fix: cleanups due to problems found by @thalesmg 2024-02-12 10:16:17 +01:00
Zaiming (Stone) Shi d4fb3e83f6 chore: upgrade to wolff 1.10.2
Wolff 1.10.2 added an enhancement which makes Kafka resource health
checks more lightweight and lowers the chance of false resource alerts.
2024-02-12 09:05:44 +01:00
Ilya Averyanov 90fd2b26d3 feat(banned): allow ban by clientid/username regexps, peerhost cidrs 2024-02-10 17:59:22 +03:00
Andrew Mayorov 0515c5528f test: add facility for starting `emqx_dashboard` with `emqx_cth_suite` 2024-02-09 14:29:04 -03:00
Andrew Mayorov c108262771
chore(s3-bridge): use `emqx_connector_schema` helper for brewity 2024-02-09 16:08:33 +01:00
Serge Tupchii 8ac8d14abd chore(emqx_mongodb): bump app vsn to 0.1.6 2024-02-09 16:29:42 +02:00
Serge Tupchii c3b7d76102 perf: un-deprecate mongodb topology pool_size and set it to 10 by default
Benchmarks proved that it effects performance:
10 ecpool workers / 10 topology pool workers outperform
100 ecpool / 1 topology worker.

Fixes: EMQX-11825
2024-02-09 16:28:53 +02:00
Serge Tupchii e248cd6c66 perf: upgrade mongodb-erlang to v3.0.23
Fixes: EMQX-11825
2024-02-09 16:18:32 +02:00
Andrew Mayorov f476f4343b
fix(release): add `emqx_bridge_s3` to applications list 2024-02-09 14:16:32 +01:00
Andrew Mayorov 2a4e37869e
fix(dashboard): provide full context in startup errors 2024-02-09 14:16:32 +01:00
Andrew Mayorov 802c760406
feat(s3): introduce S3 connector and action
This is a trivial connector based on `emqx_s3` and simple action
that maps each incoming event into an S3 object upload. Due to
current `emqx_s3` limitation this bridge is compatible with
backends providing S3 API with path-style bucket access.
2024-02-09 14:16:32 +01:00
SergeTupchiy 755b59b7fe
Merge pull request #12494 from SergeTupchiy/EMQX-11825-mongodb-connector-perf
Improve mongodb connector performance
2024-02-09 13:12:12 +02:00
Zaiming (Stone) Shi cbebc59115
Merge pull request #12492 from zmstone/0208-fix-return-receive-maximum-in-connack
fix(emqx_channel): return Receive-Maximum in CONNACK when no error
2024-02-09 12:08:46 +01:00
Andrew Mayorov 8f66bd9ddf
fix(s3-client): make log levels saner 2024-02-09 12:03:41 +01:00
Andrew Mayorov 4ff04ab1f3
feat(s3): separate concepts to make app reusable in bridges 2024-02-09 12:03:41 +01:00
Zaiming (Stone) Shi a74c828e19 test: add test case to cover Receive-Maximum in CONNACK 2024-02-09 10:42:48 +01:00
Andrew Mayorov a5266f68ec
feat(s3): switch schema to use secrets with loader support
This will make applications using `emqx_s3` follow the same
conventions as bridges and support loading secrets from files
at runtime.
2024-02-08 22:43:20 +01:00
Serge Tupchii 1d9ce709f4 perf: un-deprecate mongodb topology pool_size and set it to 10 by default
Benchmarks proved that it effects performance:
10 ecpool workers / 10 topology pool workers outperform
100 ecpool / 1 topology worker.

Fixes: EMQX-11825
2024-02-08 20:39:19 +02:00
Serge Tupchii c9ee5addb4 perf: upgrade mongodb-erlang to v3.0.23
Fixes: EMQX-11825
2024-02-08 20:00:26 +02:00
Thales Macedo Garitezi 64367d834b test(bridge_v2): actually check schema 2024-02-08 13:39:31 -03:00
Zaiming (Stone) Shi cf22692c74 fix(emqx_channel): return Receive-Maximum in CONNACK when no error 2024-02-08 17:05:16 +01:00
Kjell Winblad e284a83f73 feat: refactor RocketMQ bridge to connector and action
Fixes:
https://emqx.atlassian.net/browse/EMQX-11467
2024-02-08 16:21:58 +01:00
zhongwencool a0f8e4f328 test: rafator rabbitmq test SUITE, delete redundance code 2024-02-08 12:38:18 +08:00
zhongwencool 90ba2977fe feat: don't publish mqtt message in rabbitmq's source 2024-02-08 12:38:18 +08:00
ieQu1 811edb32a2
Merge pull request #12485 from ieQu1/dev/ds-fix-unsubscribe
fix(ds): Fix unsubscribe logic related to the replay
2024-02-07 21:38:25 +01:00
ieQu1 19c6d1127f
refactor(sessds): Extract subscription mgmt logic to separate module 2024-02-07 15:27:11 +01:00
ieQu1 3000a8f286
fix(sessds): Postpone deletion of the subscription until fully acked 2024-02-07 14:51:02 +01:00
lafirest 3f13aed3de
Merge pull request #12481 from lafirest/fix/action_info
fix(tdengine): remove redundant codes
2024-02-06 11:39:25 +08:00
zhongwencool eace8f2fcb
Merge pull request #12423 from zhongwencool/rabbitmq-e560
feat: rabbitmq bridge v2 integration
2024-02-06 09:55:05 +08:00
ieQu1 30eb54e86b fix(sessds): Delay unsubscribe until full ack of in-flight messages 2024-02-06 02:49:07 +01:00
ieQu1 a9c55f7568 feat(sessds): Consider #srs with only QoS0 messages fully acked 2024-02-06 02:49:07 +01:00
zhongwencool d3db53626b
Merge pull request #12483 from zmstone/0205-docs-fix-typos
Improve emqx ctl conf cluster_sync CLI and help text
2024-02-06 09:11:35 +08:00
zhongwencool 3e599e5d8e chore: add maybe feature enable on rabbitmq_bridge 2024-02-06 07:57:13 +08:00
zhongwencool e3cb49db8e chore: add copyright header 2024-02-06 07:57:13 +08:00
zhongwencool 688701eedb chore: add desc for rabbitmq source 2024-02-06 07:57:13 +08:00
zhongwencool b444c82a42 feat: rabbitmq bridge v2 integration 2024-02-06 07:57:12 +08:00
ieQu1 280fcd8c52
Merge pull request #12437 from ieQu1/dev/optimize_make_filter
Optimize emqx_ds_bitmask_keymapper:make_filter function.
2024-02-05 17:32:28 +01:00
ieQu1 4665837cf0 fix(ds): Apply review remarks 2024-02-05 16:52:06 +01:00
Zaiming (Stone) Shi d7b0456bb0 feat: rename emqx ctl cluster_sync 'tnxid' to 'inspect' 2024-02-05 16:50:55 +01:00
Thales Macedo Garitezi e5b14a38ed
Merge pull request #12472 from thalesmg/fix-bridgev2-api-multicall-m-20240202
fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
2024-02-05 10:28:28 -03:00
ieQu1 c7888ad1f1
Merge pull request #12475 from ieQu1/dev/lean-stream
Use a more compact data structure to represent streams
2024-02-05 13:55:24 +01:00
Thales Macedo Garitezi e1ff276d76
Merge pull request #12471 from thalesmg/fix-config-upgrade-m-20240202
fix(config): apply config upgrade to deprecated configs
2024-02-05 09:05:12 -03:00
firest b4a6be0e7c fix(tdengine): remove redundant codes 2024-02-05 18:50:31 +08:00
lafirest 571786b6c5
Merge pull request #12480 from lafirest/fix/td_v1_schema
fix(tdengine): keep the bridge schema still compatible with v1
2024-02-05 18:17:55 +08:00
ieQu1 698ba3f271 fix(ds): Optimize emqx_ds_bitmask_keymapper:make_filter
This optimization makes idle polling faster
2024-02-05 10:54:19 +01:00
ieQu1 8edbec5929 refactor(ds): Clarify the language used in ds_bitmapper 2024-02-05 10:54:18 +01:00
Zaiming (Stone) Shi 75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache
Relicense apps/emqx_durable_storage under Apache 2.0
2024-02-05 10:16:52 +01:00
firest 4a2c5a3161 fix(tdengine): keep the bridge schema still compatible with v1 2024-02-05 13:22:17 +08:00
zhongwencool f7b0d1bc9d chore: add duplicated key assert in swagger.json 2024-02-04 11:53:37 +08:00
JimMoen b1150a5f88
fix(schema): bridge oracle connector schema 2024-02-04 10:53:13 +08:00
ieQu1 2e2f3cb2aa fix(sessds): Avoid stream hash collisions 2024-02-03 21:15:54 +01:00
ieQu1 2e56810ea2 refactor(ds): Use a simple improper list to represent the streams 2024-02-03 21:15:54 +01:00
Thales Macedo Garitezi 76d242df9b fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
Fixes https://emqx.atlassian.net/browse/EMQX-11834
2024-02-02 17:36:08 -03:00
Thales Macedo Garitezi 76401a302a fix(config): apply config upgrade to deprecated configs
Fixes https://emqx.atlassian.net/browse/EMQX-11845

Fixes https://github.com/emqx/emqx/issues/12452
2024-02-02 15:12:08 -03:00
Ivan Dyachkov ed8660cb68
Merge pull request #12458 from id/0202-sync-release-55
sync release-55
2024-02-02 17:20:38 +01:00
SergeTupchiy 81433282be
Merge pull request #12467 from SergeTupchiy/autocluster-dns-aaaaa-record-type
feat(emqx_conf): support AAAA cluster dns record type
2024-02-02 18:13:06 +02:00
Zaiming (Stone) Shi b1a05c7b59
Merge pull request #12326 from zmstone/0111-unregister-session-with-timestamp
0111 unregister session with timestamp
2024-02-02 17:00:23 +01:00
ieQu1 e0b5d9fae7
Merge pull request #12464 from ieQu1/dev/feat-ds-cli
feat(ds): Add a CLI interface to inspect status of DS databases
2024-02-02 14:14:48 +01:00
Serge Tupchii 5e0c3d4051 feat(emqx_conf): support AAAA cluster dns record type
Ekka autocluster DNS already supports AAAA records, so it's only needed to update enum in schema.
2024-02-02 12:13:29 +02:00
Kjell Winblad fa0baf2f38
Merge pull request #12439 from kjellwinblad/kjell/refactor/oracle_bridge/EMQX-11464
feat: refactor Oracle bridge to connector and action
2024-02-02 10:53:15 +01:00
Ivan Dyachkov 84b6d7d720 fix: remove 5.5.bpapi2 file 2024-02-02 10:38:50 +01:00
ieQu1 3d2ac97c61 feat(ds): Add a CLI interface to inspect status of DS databases 2024-02-02 10:11:01 +01:00
Zaiming (Stone) Shi f0569d8ae8 refactor: use mria:async_dirty to group dirty ops 2024-02-02 09:26:43 +01:00
lafirest 24d8a9ab00
Merge pull request #12449 from lafirest/feat/td
feat(tdengine): improve the TDengine bridge to v2 style
2024-02-02 16:17:01 +08:00
Ivan Dyachkov dd490de2e1 chore: emqx 5.5.0 2024-02-02 08:48:56 +01:00
Ivan Dyachkov f1c7e716ce chore: 5.5.0-rc.2 2024-02-02 08:48:56 +01:00
JianBo He 9aad7997ca chore: compatible the contet-type sytanx 2024-02-02 08:48:56 +01:00
Ivan Dyachkov dae3a94670 chore: 5.5.0-rc.1 2024-02-02 08:48:56 +01:00
lafirest c724d2127e
Merge pull request #12457 from lafirest/fix/opentsdb
fix(opents): change the schema of tags to object style
2024-02-02 15:33:50 +08:00
JimMoen 17d79026cd
test: fix emqx_connector_schema eunit missing fields 2024-02-02 14:43:25 +08:00
firest 5291adf2c0 fix(opents): change the schema of tags to object style 2024-02-02 13:07:49 +08:00
JimMoen 1a372ce39a
fix(swagger): no-need call common_fields when `emqx_connector_schema:api_fields/3` used 2024-02-02 10:54:49 +08:00
firest 2241461acb chore: update change & bump version 2024-02-02 07:48:44 +08:00
Kjell Winblad af3f43ffd5
fix: use emqx_resource macro for status
Thanks for the suggestion @thalesmg

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 17:24:25 +01:00
ieQu1 98d1094d73 feat(sessds): Expose subscriptions in the REST API 2024-02-01 16:47:57 +01:00
Kjell Winblad 016fbd2c5c chore: remove commented out code 2024-02-01 16:18:37 +01:00
firest dfad020c49 feat(tdengine): improve the TDengine bridge to v2 style 2024-02-01 23:14:24 +08:00
Kjell Winblad a4272c71dc feat: refactor Oracle bridge to connector and action
Fixes:
https://emqx.atlassian.net/browse/EMQX-11464
2024-02-01 16:03:05 +01:00
Ilya Averyanov ae1ef9d1bb
Merge pull request #12428 from savonarola/0130-simplify-boot-machine-runtime-deps
chore(emqx_machine): refactor injecting runtime deps
2024-02-01 16:32:53 +02:00
William Yang 80ff1b1657
Merge pull request #12434 from qzhuyan/fix/william/kafka-dont-stacktrace-when-no-partiion-count
kafka-bridge: dont stacktrace when no partition count
2024-02-01 15:09:45 +01:00
Zaiming (Stone) Shi 3e518c1876
Merge pull request #12427 from zmstone/0130-limit-kafka-partitions
0130 limit kafka partitions
2024-02-01 14:39:52 +01:00
Ilya Averyanov 6a092aeb69 chore(emqx_machine): refactor injecting runtime deps
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 16:13:49 +03:00
William Yang 16444b7da9 fix(bridge-kafka): no stacktrace when fetch topic partition count fail 2024-02-01 14:09:15 +01:00
Ivan Dyachkov 3ad71f75e6 chore: emqx 5.5.0 2024-02-01 11:07:02 +01:00
zhongwencool 42a4bc3382
Merge pull request #12435 from zhongwencool/dont-add-disable-resource-to-connector
fix: don't add disable bridge to connector's channel
2024-02-01 10:39:47 +08:00
JianBo He 0d5911d942
Merge pull request #12422 from zhongwencool/upgrade-erlfmt-to-v1.3.0
feat: upgrade erlfmt to support maybe syntax
2024-02-01 09:34:33 +08:00
zhongwencool f9d16340e0 fix: don't add disable bridge to connector's channel 2024-02-01 09:12:43 +08:00
ieQu1 139f5dc3bd chore(ds): Remove an obsolete document; superceded by ./README.md 2024-02-01 00:14:08 +01:00
ieQu1 b50d6bf1fd chore(ds): Change the license to Apache 2.0
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
2024-02-01 00:10:48 +01:00
Thales Macedo Garitezi f7b12470dd test(ds): fix inter-suite flakiness
Attempt to mitigate this frequent source of flakiness:

```
=CRASH REPORT==== 31-Jan-2024::17:30:15.025404 ===
  crasher:
    initial call: emqx_ds_replication_layer_egress:init/1
    pid: <0.11312.0>
    registered_name: []
    exception error: no match of right hand side value {error,
                                                        no_leader_for_shard}
      in function  emqx_ds_replication_layer_egress:init/1 (/emqx/apps/emqx_durable_storage/src/emqx_ds_replication_layer_egress.erl, line 93)
      in call from gen_server:init_it/2 (gen_server.erl, line 980)
      in call from gen_server:init_it/6 (gen_server.erl, line 935)
    ancestors: [<0.11310.0>,<0.11304.0>,emqx_ds_builtin_databases_sup,
                  emqx_ds_builtin_sup,emqx_ds_sup,<0.11236.0>]
    message_queue_len: 0
    messages: []
    links: [<0.11310.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 376
    stack_size: 28
    reductions: 231
  neighbours:
```
2024-01-31 15:42:26 -03:00
Thales Macedo Garitezi d51deac222 fix(ds): use configured data dir for site storage 2024-01-31 15:42:26 -03:00
Zaiming (Stone) Shi 3b6fbff9a5 test: fix compile warnings 2024-01-31 19:32:57 +01:00
Zaiming (Stone) Shi 3b42a7425b feat(kafka_producer): add partitions_limit 2024-01-31 19:26:01 +01:00
Zaiming (Stone) Shi 53c217c383 refactor: micro optimization 2024-01-31 19:25:57 +01:00
Ilya Averyanov b3657f1892
Merge pull request #12436 from savonarola/0131-fix-stream-replay
fix(sessds): fix renew stream logic
2024-01-31 18:25:17 +02:00
Ivan Dyachkov 3f71a98e5e chore: 5.5.0-rc.2 2024-01-31 17:06:54 +01:00
lafirest a60aeed539
Merge pull request #12398 from lafirest/feat/api-docs
feat(dashboard): expose the `swagger_support` option
2024-01-31 23:31:56 +08:00
Ilya Averyanov fe46434687 fix(sessds): fix renew stream logic 2024-01-31 18:02:29 +03:00
Kjell Winblad 2680fae9b6
Merge pull request #12420 from kjellwinblad/kjell/fix/kinesis/refactoring/problems/thanks_HJianBo
fix: kinesis schema problems found by @HJianBo
2024-01-31 15:52:45 +01:00
zhongwencool e9c8446d57 feat: upgrade erlfmt to support maybe syntax 2024-01-31 20:06:49 +08:00
JianBo He b787f28d27 chore: compatible the contet-type sytanx 2024-01-31 17:49:46 +08:00
Zaiming (Stone) Shi 38047108a4 test: add test coverage for emqx_cm_registry_keeper module 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 209331ad33 test: fix config test 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 87a2368e37 feat(metrics): add cluster_session guage 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 509ab6f35a feat(api): add /sessions_count api to count sessions 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi 562a2736ae feat: add `broker` root to hot-config schema 2024-01-31 09:50:12 +01:00
Zaiming (Stone) Shi e9318752e6 feat: store session unregistration timestamp in emqx_cm_registry table 2024-01-31 09:50:12 +01:00
Ilya Averyanov 5bc67cb288
Merge pull request #12432 from ieQu1/ds_ignore_fully_replayed_streams
fix(sessds): Stream scheduler must ignore fully replayed streams
2024-01-31 09:51:49 +02:00
firest 13df7fa46e fix(dashboard): add test case & update change 2024-01-31 14:14:56 +08:00
ieQu1 e843d9fd91 fix(sessds): Stream scheduler must ignore fully replayed streams 2024-01-31 00:20:54 +01:00
Ivan Dyachkov a2ba8fc645 chore: 5.5.0-rc.1 2024-01-30 18:58:00 +01:00
Kjell Winblad 274f378c6e
fix: better delete of local_topic as suggested by @thalesmg
Co-authored-by:  Thales Macedo Garitezi <thalesmg@gmail.com>

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-01-30 17:53:32 +01:00
lafirest c7f1e44513
Merge pull request #12424 from lafirest/fix/opentsdb
feat(opentsdb): supports more flexible tags schema
2024-01-30 22:05:30 +08:00
Zaiming (Stone) Shi 55fdc83aa5 Merge remote-tracking branch 'origin/release-55' into 0130-syn-release-55-to-master 2024-01-30 12:06:43 +01:00
JianBo He c0674913e2
Merge pull request #12421 from HJianBo/fix-import-users
fix(user_import): ensure the last record overwrites previous one
2024-01-30 18:49:35 +08:00
firest 81f96f1a68 feat(opentsdb): supports more flexible tags schema 2024-01-30 18:43:39 +08:00
SergeTupchiy 03ff6f2ddf
Merge pull request #12410 from SergeTupchiy/EMQX-11812-subscribe-performance-degraded
perf(emqx_broker): pick broker pool worker by topic/shard pair to dis…
2024-01-30 11:48:59 +02:00
Xinyu Liu db3f285054
Merge pull request #12417 from emqx/port-msg-expiry-interval-from44
feat: configurable server side message_expiry_interval
2024-01-30 17:10:32 +08:00
Shawn 206af96a33 ci: update testcases for message-expiry-interval 2024-01-30 15:46:11 +08:00
JianBo He 4190682a16 chore: fix the data type and example value for cluster invitation result 2024-01-30 15:29:46 +08:00
JianBo He aedfc8e8c0 fix(user_import): ensure the last record overwrites previous one 2024-01-30 14:14:20 +08:00
JianBo He 35c4ef2ee2
Merge pull request #12407 from zmstone/0126-script-to-update-bsl-license-convert-time
0126 script to update bsl license convert time
2024-01-30 09:54:34 +08:00
JianBo He 4688b36cdf
Merge pull request #12396 from HJianBo/import_user_2
feat(import_users): support user's password in plain text
2024-01-30 09:00:00 +08:00
Serge Tupchii 5007650bd2 perf(emqx_broker): pick broker pool worker by topic/shard pair to distribute the load more evenly.
Fixes: EMQX-11812
2024-01-29 20:57:10 +02:00
Thales Macedo Garitezi cf0465dad4
Merge pull request #12411 from thalesmg/fix-cassandra-null-r55-20240126
fix(cassandra_bridge): correctly insert null values into columns
2024-01-29 14:09:12 -03:00
Kjell Winblad b849b08dbc fix: kinesis schema problems found by @HJianBo
This commit fixes problems in schema found by @HJianBo here:

https://github.com/emqx/emqx/pull/12376#pullrequestreview-1848306286
2024-01-29 18:08:53 +01:00
ieQu1 7ee0a6aaa4
Merge pull request #12414 from ieQu1/dev/ds-readme
doc(ds): Update README
2024-01-29 18:01:03 +01:00
Zaiming (Stone) Shi 82403167c2 chore: update BSL license change date 2024-01-29 16:47:31 +01:00
Thales Macedo Garitezi 47f61ba68a fix(cassandra_bridge): correctly insert null values into columns
Fixes https://emqx.atlassian.net/browse/EMQX-11822
2024-01-29 11:32:00 -03:00
ieQu1 cb90d3c90c
Merge pull request #12415 from ieQu1/dev/remove-ds-conf-module
fix(ds): Remove unused module
2024-01-29 12:56:04 +01:00
Shawn 9f22c2c455 ci: add some sleep and retry to emqx_persistent_session_ds_SUITE 2024-01-29 18:18:18 +08:00
JianBo He d12335c4c6 chore: add tests 2024-01-29 18:10:42 +08:00
JianBo He d286b7e28c
Merge pull request #12404 from thalesmg/fix-action-metric-handler-r55-20240126
fix(resource_metrics): avoid detaching handler on crashes
2024-01-29 17:51:04 +08:00
Shawn 12da3c0986 feat: configurable server side message_expiry_interval 2024-01-29 17:29:52 +08:00
Ivan Dyachkov fa6d65887d test(retainer): fix test cases 2024-01-29 08:42:36 +01:00
Ivan Dyachkov e4b8d79444 fix(retainer): add default delivery rate in the schema 2024-01-29 08:04:58 +01:00
JianBo He ab99a17c99 chore(utils_stream): simplfy the csv reader implementation 2024-01-29 11:10:50 +08:00
JianBo He 2e35024df1 test: update eunit tests 2024-01-29 10:49:07 +08:00
JianBo He 9915c85b0b chore(authn_mnesia): use emqx_utils_stream module to imporve reusability 2024-01-29 10:49:07 +08:00
JianBo He 8fc8106819 test: cover password_type and new data format 2024-01-29 10:49:07 +08:00
JianBo He 829887630d test: refine existed test cases 2024-01-29 10:49:07 +08:00
JianBo He fed512689a chore: make elvis checking happy 2024-01-29 10:49:07 +08:00
JianBo He e65cfb836c feat(import_users): support user's password in plain text 2024-01-29 10:49:07 +08:00
ieQu1 2479e1189a fix(ds): Remove unused module 2024-01-29 00:36:13 +01:00
ieQu1 96c527541b docs(ds): Update README 2024-01-28 20:27:20 +01:00
Zaiming (Stone) Shi dbe2baa405
Merge pull request #12408 from zmstone/sync-release-55
Sync release 55
2024-01-27 19:20:58 +00:00
ieQu1 2c6a776318 fix(sessds): Stricter checks for PacketIds 2024-01-26 17:49:33 +01:00
ieQu1 eec56b0d6b fix(sessds): Improve comments 2024-01-26 17:49:33 +01:00
ieQu1 3c451c6ae6 test(sessds): Fix unstable tests 2024-01-26 17:49:33 +01:00
ieQu1 2d23212792 refactor(sessds): #ifs -> #srs 2024-01-26 17:49:33 +01:00
ieQu1 974760d331 test(sessds): Don't trap exits in the test 2024-01-26 17:49:33 +01:00
ieQu1 ebe2339810 fix(sessds): Use mria:async_dirty instead of transaction 2024-01-26 17:49:33 +01:00
ieQu1 f5b9bd30aa fix(sessds): Apply review remarks 2024-01-26 17:49:33 +01:00
ieQu1 39857626ce test(sessds): Fix failing tests 2024-01-26 17:49:33 +01:00
ieQu1 e7b03cdc59 test(sessds): Create a property-based test for the session state 2024-01-26 17:49:33 +01:00
ieQu1 893656f092 fix(sessds): Fix session garbage collection after the refactoring 2024-01-26 17:49:33 +01:00
ieQu1 963df8f941 fix(sessds): Delete the routes when the session expires 2024-01-26 17:49:33 +01:00
ieQu1 cff6c15e13 fix(sessds): Store the QoS as the MSB of the packet ID 2024-01-26 17:49:33 +01:00
ieQu1 978a3bfef3 refactor(sessds): Simplify representation of QoS tracks 2024-01-26 17:49:33 +01:00
ieQu1 1b4f69b44d refactor(sessds): Simplify data structure of ds_state pmap datatype 2024-01-26 17:49:33 +01:00
ieQu1 4f4831fe7f refactor(sessds): Factor out stream scheduler into its own module 2024-01-26 17:49:33 +01:00
ieQu1 82ef34998a feat(sessds): Index streams by a unique subid 2024-01-26 17:49:33 +01:00
ieQu1 3fb2064ea4 test(sessds): Add property-based tests for seqno generator 2024-01-26 17:49:33 +01:00
ieQu1 8e8d3af096 fix(sessds): Refactor emqx_persistent_session_ds to use CRUD module 2024-01-26 17:49:33 +01:00
ieQu1 2d08aa88d8 refactor(ds): Create a CRUD module for the persistent session 2024-01-26 17:49:33 +01:00
Thales Macedo Garitezi 6410f5a717 fix(resource_metrics): avoid detaching handler on crashes
Fixes https://emqx.atlassian.net/browse/EMQX-11821
2024-01-26 13:05:41 -03:00
Zaiming (Stone) Shi 2a40152721 Merge remote-tracking branch 'origin/release-55' into sync-release-55 2024-01-26 16:51:13 +01:00
William Yang 933c00c7ad
Merge pull request #12392 from qzhuyan/perf/william/ws-no-utf8-validate
perf: new ws listener option to disable UTF-8 validation
2024-01-26 16:01:04 +01:00
ieQu1 0933dc818e chore(ekka): Bump version to 0.18.4
Don't escalate the errors in etcd node discovery to the node level.

Fixes: #12255
2024-01-26 14:25:16 +01:00
Serge Tupchii a56a5e9c3c chore: add copyright 2024-01-26 11:01:53 +02:00
Serge Tupchii 2d693402c5 refactor: split greptimedb bridge to actions and connectors 2024-01-26 11:01:53 +02:00
William Yang 726302ef6a perf: new ws listener option to disable UTF-8 validation 2024-01-26 09:55:17 +01:00
JimMoen 2e37f7514a
fix(prom_stats): compatibility with previous api format in json mode 2024-01-26 16:01:32 +08:00
JimMoen 308cde53b9
fix(prom_stats): merge cl_consistented data into per nodes data 2024-01-26 11:39:20 +08:00
JianBo He e29a80d568
Merge pull request #12388 from qzhuyan/dev/william/per-listn-conn-cnt
feat: quic per listener conn count
2024-01-26 09:01:39 +08:00
Thales Macedo Garitezi 590746ddfa
Merge pull request #12400 from thalesmg/ds-fix-data-dir-schema-m-20240125
fix(ds): don't make data dir part of the schema
2024-01-25 15:52:11 -03:00
Thales Macedo Garitezi 8e31afe6c2 fix(ds): don't make data dir part of the schema
The data directory was ending up being persisted in the database schema.  This led to
issues when opening the DB on different nodes.
2024-01-25 14:44:06 -03:00
lafirest 0b6aee0ae9
Merge pull request #12393 from lafirest/fix/iotdb
fix(iotdb): change the `data` field to optional
2024-01-25 23:00:36 +08:00
firest c37097a150 feat(dashboard): expose the `swagger_support` option 2024-01-25 22:58:16 +08:00
Kjell Winblad 5bf37cbe0a
Merge pull request #12376 from kjellwinblad/kjell/kinesis
feat: refactor kinesis bridge to connector and action
2024-01-25 14:58:40 +01:00
Thales Macedo Garitezi b5567d9d9f
Merge pull request #12389 from thalesmg/ci-peer-cover-m-20240124
ci: start/flush cover when using peer
2024-01-25 10:00:50 -03:00
Kjell Winblad 2887a05ba3 refactor: simplify emqx_bridge_kinesis (thanks to @thalesmg's tips) 2024-01-25 12:48:31 +01:00
JimMoen 4dca1ef848
fix(prom_di): compatibility with bridge_v1 2024-01-25 19:37:15 +08:00
JimMoen 7b041683bb
fix(prom_di): action enable and status 2024-01-25 19:36:40 +08:00
JimMoen beeedd33d1
fix(prom_mria): mria_data in init_acc 2024-01-25 19:36:39 +08:00
JimMoen 58b60181d7
fix(prom_mria): move mria callback into `emqx_prometheus.erl`
- they will return in same api endpoint.
2024-01-25 19:36:38 +08:00
Kjell Winblad cfad0923cf chore: use macros for connector and action status atoms 2024-01-25 11:50:01 +01:00
firest 01f374d8e5 fix(iotdb): change the `data` field to optional 2024-01-25 18:29:41 +08:00
zhongwencool 3b0736fc67
Merge pull request #12387 from thalesmg/sync-r55-m-20240124
sync release-55 to master
2024-01-25 17:35:41 +08:00
zhongwencool 4f8accc31b
Merge branch 'master' into sync-r55-m-20240124 2024-01-25 16:09:05 +08:00
ieQu1 aeefbe9544 fix(ds): Cache database metadata in RAM 2024-01-25 16:01:44 +08:00
Thales Macedo Garitezi 846ad42a65 fix(ds): don't use env var for data dir default value
Fixes https://github.com/emqx/emqx/pull/12380
2024-01-25 16:01:44 +08:00
Thales Macedo Garitezi 024ffe2909 test(ds): reduce flakiness 2024-01-25 16:01:44 +08:00
Thales Macedo Garitezi 9003bc5b72 perf(ds): inherit only LTS paths containing wildcards when adding a new generation
Fixes https://github.com/emqx/emqx/pull/12338#discussion_r1462139499
2024-01-25 16:01:44 +08:00
Shawn e4c683d6f8 chore: update ecql to 0.6.0 2024-01-25 16:01:44 +08:00
Shawn 5cd9f49558 ci: update generated connector name 2024-01-25 16:01:44 +08:00
Shawn 6a21766ce3 refactor: split cassandra bridges to actions and connectors 2024-01-25 16:01:44 +08:00
Shawn 497e735bf4 ci: add env vars to run cassandra tests locally 2024-01-25 16:01:43 +08:00
zhongwencool 186e1591df chore: document api-key path api only support bearerAuth 2024-01-25 16:01:43 +08:00
firest b44420c14f fix(opentsdb): Enhanced the type support for template data 2024-01-25 16:01:43 +08:00
firest 83a8822798 chore(opents): bump version && update changes 2024-01-25 16:01:43 +08:00
firest dad8a32e0b feat(opents): improve the OpentsDB bridge to v2 style 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi 878c9ee8b1 fix(ds): do not count persistent session-only routed messages as dropped
Fixes https://emqx.atlassian.net/browse/EMQX-11539
2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi d323fc7c27 feat(ps): add message gc 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi 7c0d37fdb9 feat(lts): inherit previous generation's lts when possible 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi 4a0fd756ae feat(ds): add `list_generations` and `drop_generation` APIs 2024-01-25 16:01:43 +08:00
Thales Macedo Garitezi fb39e1eacc feat(ds): allow customizing the data directory
The storage expectations for the RocksDB DB may be different from our usual data
directory.  Also, it may consume a lot more storage than other data.

This allows customizing the data directory for the builtin DS storage backend.

Note: if the cluster was already initialized using a directory path, changing that config
will have no effect.  This path is currently persisted in mnesia and used when reopening
the DB.
2024-01-25 16:01:42 +08:00
JianBo He 51d32bd620 chore: improve http connector logs format 2024-01-25 16:01:42 +08:00
Ilya Averyanov adf22f1f10 fix(mqtt_bridge): render valid messages from incomplete rule data 2024-01-25 16:01:42 +08:00
Serge Tupchii a8c6280a5e test(emqx_cm_SUITE): use one helper function: `emqx_pool:flush_async_tasks/1` 2024-01-25 16:01:42 +08:00
Serge Tupchii f52cc93d9d perf(emqx_cm): use a dedicated pool for channel cleanup
This is to isolate channels cleanup from other async tasks (like routes cleanup),
as channels cleanup can be quite slow under high network latency conditions.

Fixes: EMQX-11743
2024-01-25 16:01:42 +08:00
JianBo He 52a7b45d97 fix: upgrade ekka to 0.18.3 to ensure the leader correct 2024-01-25 14:47:59 +08:00
zhongwencool 3bc6e546e0 fix: reboot emqx_dashboard after emqx_licencse 2024-01-25 12:29:04 +08:00
Xinyu Liu 4eb0260eaf
Merge pull request #12381 from emqx/port-sql-funcs-from-emqx4
feat: port emqx/emqx-enterprise#1892, add some SQL functions
2024-01-25 11:12:58 +08:00
JianBo He ff0fd65f9d
Merge pull request #12373 from JimMoen/fix-prom-data-aggre
Follow up #12299
2024-01-25 10:37:47 +08:00
JimMoen a752119d05
fix(prom_di): use bridge_v2 metrics api 2024-01-25 09:22:22 +08:00
JimMoen 61dd362db0
fix(prom_di): api crash in `connecting`
- https://emqx.atlassian.net/browse/EMQX-11801
2024-01-25 09:22:21 +08:00
Thales Macedo Garitezi ae387d1812 ci: start/flush cover when using peer
`cover` is not automatically started by `peer`.  Without starting/flushing it, we don't
get coverage data from peer nodes.
2024-01-24 18:02:47 -03:00
William Yang 9fb6688801 fix(quic): Per listener conn count 2024-01-24 21:36:25 +01:00
Thales Macedo Garitezi a19cf253ae
Merge pull request #12359 from thalesmg/fix-connector-app-deps-r55-20240119
fix(machine_boot): inject connector/bridge apps as dependencies to `emqx_connector` app
2024-01-24 15:40:24 -03:00
ieQu1 7b5f2948fe test(ds): Fix flaky testcase 2024-01-24 19:33:30 +01:00
ieQu1 305a54f646 chore(ds): Update BPAPI version 2024-01-24 19:33:30 +01:00
ieQu1 eee221f1d0 feat(ds): Make egress batching configurable 2024-01-24 19:33:30 +01:00
ieQu1 137535a821 feat(ds): Introduce egress process for the builtin backend 2024-01-24 19:33:30 +01:00
ieQu1 ec93d8cf18
Merge pull request #12385 from ieQu1/ds-meta-ram-cache
fix(ds): Cache database metadata in RAM
2024-01-24 19:32:38 +01:00
ieQu1 9b7df302e8 fix(ds): Cache database metadata in RAM 2024-01-24 18:43:48 +01:00
Kjell Winblad fa842736d2 feat: refactor kinesis bridge to connector and action
Fixes:
https://emqx.atlassian.net/browse/EMQX-11461
2024-01-24 18:31:03 +01:00
Thales Macedo Garitezi 5547a40ceb fix(ds): don't use env var for data dir default value
Fixes https://github.com/emqx/emqx/pull/12380
2024-01-24 11:39:14 -03:00
JimMoen ace1303de2
fix(prom_auth): authz metric name: `allow` and `deny`
- https://emqx.atlassian.net/browse/EMQX-11799
2024-01-24 22:38:49 +08:00
JimMoen 5e02709de0
fix(prom_auth): authz_enable key from enable status
- https://emqx.atlassian.net/browse/EMQX-11800
2024-01-24 22:38:48 +08:00
Shawn 4afba8eb94 feat: port emqx/emqx-enterprise#1892, add some SQL functions 2024-01-24 18:50:57 +08:00
JimMoen 6e3ce4d644
fix(prom_stats): emqx_stats metric type should be gauge 2024-01-24 14:11:17 +08:00
JimMoen 4a1d1b6aeb
fix: topics and retained metrics are cluster consistented 2024-01-24 14:11:16 +08:00
JimMoen 731efd8b49
fix(prom): cluster aggre/unaggre labels 2024-01-24 14:11:15 +08:00
JimMoen 497ec04359
fix(prom): missing callback attribute 2024-01-24 14:11:14 +08:00
JianBo He 947a08d1d2
Merge pull request #12375 from thalesmg/fix-actions-reset-node-metrics-r55-20240123
fix(actions_api): reset metrics on all nodes
2024-01-24 09:17:09 +08:00
Thales Macedo Garitezi 77329209a2
Merge pull request #12368 from thalesmg/ds-inherit-wildcards-only-lts-m-20240122
perf(ds): inherit only LTS paths containing wildcards when adding a new generation
2024-01-23 17:14:51 -03:00
Thales Macedo Garitezi 67c9e9c8fc fix(actions_api): reset metrics on all nodes
Fixes https://emqx.atlassian.net/browse/EMQX-11796
2024-01-23 10:40:02 -03:00
Thales Macedo Garitezi eecd7e084c test(ds): reduce flakiness 2024-01-23 09:47:03 -03:00
Thales Macedo Garitezi 1be5b6aaf3 fix(mqtt_action): use overridden `resource_opts`
Fixes https://emqx.atlassian.net/browse/EMQX-11790
2024-01-23 09:41:28 -03:00
Thales Macedo Garitezi 1eb47d0c16 perf(ds): inherit only LTS paths containing wildcards when adding a new generation
Fixes https://github.com/emqx/emqx/pull/12338#discussion_r1462139499
2024-01-23 09:20:28 -03:00
zhongwencool a8eefe808f
Merge pull request #12371 from emqx/es-server-schema
chore: es's base_url to server
2024-01-23 17:17:19 +08:00
zhongwencool 6a9cbb3d97 test: add es's update test 2024-01-23 15:51:09 +08:00
Xinyu Liu aafb683ec7
Merge pull request #12330 from emqx/bridge-v2-cassandra
refactor: split cassandra bridges to actions and connectors
2024-01-23 14:45:32 +08:00
zhongwencool 94692162b5
Merge pull request #12370 from zhongwencool/api-key-auth
chore: document api-key path api only support bearerAuth
2024-01-23 14:22:35 +08:00
Shawn 218af3fef4 chore: update ecql to 0.6.0 2024-01-23 14:14:23 +08:00
lafirest 9d2a207184
Merge pull request #12353 from lafirest/feat/optsdb
feat(opents): improve the OpentsDB bridge to v2 style
2024-01-23 13:49:33 +08:00
lafirest dc458ff0db
Merge pull request #12366 from lafirest/fix/sys
fix(sysk): fix probe testing bugs for syskeeper
2024-01-23 13:49:28 +08:00
firest 51e8c955ff fix(connector): make the connector id unique when doing probe testing 2024-01-23 13:03:12 +08:00
zhongwencool ada2785b5d chore: es's base_url to server 2024-01-23 12:20:34 +08:00
zhongwencool 0dbaaa5d94 chore: document api-key path api only support bearerAuth 2024-01-23 10:31:43 +08:00
firest 28867d07e6 fix(opentsdb): Enhanced the type support for template data 2024-01-23 10:00:56 +08:00
Thales Macedo Garitezi 609ba7e332 fix(ds): do not count persistent session-only routed messages as dropped
Fixes https://emqx.atlassian.net/browse/EMQX-11539
2024-01-22 16:58:45 -03:00
Thales Macedo Garitezi d122340c13
Merge pull request #12338 from thalesmg/ds-message-gc-20240115
feat(ds): add message GC
2024-01-22 16:57:26 -03:00
Thales Macedo Garitezi 7035b4c8b3 feat(ps): add message gc 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi db710c4be5 feat(lts): inherit previous generation's lts when possible 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi 75b08b525b feat(ds): add `list_generations` and `drop_generation` APIs 2024-01-22 14:53:17 -03:00
Thales Macedo Garitezi 39090d3732
Merge pull request #12340 from thalesmg/ds-db-data-path-m-20240116
feat(ds): allow customizing the data directory
2024-01-22 13:30:59 -03:00
Thales Macedo Garitezi 3207f0ea80 fix(machine_boot): inject connector/bridge apps as dependencies to `emqx_connector` app
Fixes https://emqx.atlassian.net/browse/EMQX-11771

For the same reasons as we inject `emqx_bridge_*` applications as dependencies to
`emqx_bridge` when starting the node.  Already configured connectors are started when
`emqx_connector` application starts, and may lead to crashes and noise in the logs.  One
example is to configure a mongodb bridge and restart the node.
2024-01-22 11:13:34 -03:00
firest 3f7b913e88 chore(opents): bump version && update changes 2024-01-22 20:45:18 +08:00
firest e337e1dc40 feat(opents): improve the OpentsDB bridge to v2 style 2024-01-22 20:45:10 +08:00
firest cd90b93550 fix(sysk): fix probe testing bugs for syskeeper 2024-01-22 20:27:48 +08:00
JianBo He 2706e005ff
Merge pull request #12354 from SergeTupchiy/EMQX-11751-dashboard-import-timeout-rel55
Apply brdige post config changes in parallel
2024-01-22 17:24:55 +08:00
JimMoen 2061d75b50
docs: prometheus api `mode` field description 2024-01-22 10:39:56 +08:00
JimMoen 2263df0242
fix(prom_push_gw): use format mode `node` for prometheus push gateway 2024-01-22 10:39:55 +08:00
JimMoen b424f8ac12
feat(prom_stats): aggregated/unaggregated prometheus data 2024-01-22 10:39:54 +08:00
JimMoen 6b064dd8eb
fix(prom_stats): missing metric key
- follow https://github.com/emqx/emqx/pull/11497
2024-01-22 10:39:53 +08:00
JimMoen c6c1a7fc28
refactor(prom): prom_auth and prom_di as prom_cluster behaviour 2024-01-22 10:39:52 +08:00
JimMoen b480c5b371
fix(prom): use name `mode` and macros to put/get format mode 2024-01-22 10:39:51 +08:00
Shawn 0e1043f80c ci: update generated connector name 2024-01-22 10:34:40 +08:00
JianBo He f7ae9ab489
Merge pull request #12351 from HJianBo/improve-the-http-connector-logs
chore: improve http connector logs format
2024-01-22 10:27:35 +08:00
Shawn b32c0fb0d8 refactor: split cassandra bridges to actions and connectors 2024-01-22 10:16:46 +08:00
Shawn 29d767bd62 ci: add env vars to run cassandra tests locally 2024-01-22 10:16:46 +08:00
JimMoen 8cb12c6a74
refactor: use utils func for prom_auth and prom_di 2024-01-21 00:14:54 +08:00
JimMoen c3da792323
feat(prom_di): cluster aggregated/unaggregated metrics 2024-01-21 00:14:53 +08:00
JimMoen 9627124d67
fix(prom_di): metric `emqx_rule_enable` 2024-01-21 00:14:52 +08:00
JimMoen 5534d5e9de
fix(bpapi): make static_check happy 2024-01-21 00:14:51 +08:00
JimMoen 5914eb5ca5
fix(prom_cert): rm cacertfile expiry epoch 2024-01-21 00:14:50 +08:00
JimMoen 4fb1ff2f9d
fix(prom_api): format-mode example value 2024-01-21 00:14:49 +08:00
JimMoen 57f3efde63
feat(prom_auth): cluster metrics with different format-mode 2024-01-21 00:14:48 +08:00
JimMoen fb330f77e6
feat(prometheus): api `format_mode` parameter support
- node(default):
  The only supported format_mode for PushGateway.
  Return the current node's metrics.
  -  Without label `node_name` with content-type: `text/plain`.
  -  Without key `node_name` with content-type: `application/json`

- nodes_aggregated:
  Return all nodes metrics Arithmetic-Sum or Logical-Sum.
  See details in callback modules.
    - Logical-Sum for metrics named with `xxx_enable` or `xxx_status`.
    - Arithmetic-Sum for other metrics.
  `node_name` field:
    -  Without label `node_name` with content-type: `text/plain`.
    -  Without key `node_name` with content-type: `application/json`

- nodes_unaggregated:
  Return all nodes metrics without aggregated.
  `node_name` field:
    -  _With_ label `node_name` with content-type: `text/plain`.
    -  _With_ key `node_name` with content-type: `application/json`
2024-01-21 00:14:47 +08:00
JimMoen 94032aafb2
fix(prom_data_integration): fix metric type 2024-01-21 00:14:46 +08:00
JimMoen c3e9533260
fix: prometheus auth metrics fields and type
- rm rate fields
- fix few fields type to counter
2024-01-21 00:14:45 +08:00
JimMoen a18c4d193a
refactor: abstract function call 2024-01-21 00:14:44 +08:00
JimMoen 8f7964f435
feat(prom): data integration metrics in josn format 2024-01-21 00:14:42 +08:00
JimMoen 36f009b0c2
fix(prom): connectors specific data 2024-01-21 00:14:41 +08:00
JimMoen 76d9ace582
fix: connector_count contains bridge_v1 and bridge_v2 2024-01-21 00:14:40 +08:00
JimMoen f457def201
fix: use `id` uniformly as the label key for rules and actions 2024-01-21 00:14:39 +08:00
JimMoen 4b23930fce
fix(prometheus): license expiry and schema_registry only for ee 2024-01-21 00:14:38 +08:00
JimMoen 38a90bd273
fix(prom_auth): authn && authz data response by josn 2024-01-21 00:14:37 +08:00
JimMoen 0dca9905dd
test(prometheus): `{Registry, Collector}` tuple 2024-01-21 00:14:36 +08:00
JimMoen 7832bbc0a4
fix(prom): schema registry not in ce edition 2024-01-21 00:14:35 +08:00
JimMoen 5158395bcf
feat(prometheus): data integration prom data 2024-01-21 00:14:34 +08:00
JimMoen bf2e4d134a
refactor(prometheus): generic api response funcs 2024-01-21 00:14:33 +08:00
JimMoen e0feb580b6
feat(prometheus): auth metrics with text/plain 2024-01-21 00:14:32 +08:00
JimMoen 092159b071
feat(prometheus): cert expiry epoch in endpoint `/prometheus/stats` 2024-01-21 00:14:31 +08:00
JimMoen 558c471392
feat(prometheus): license expiry at epoch as gauge 2024-01-21 00:14:30 +08:00
JimMoen fae0cea17f
fix(license): make dialyzer happy 2024-01-21 00:14:29 +08:00
JimMoen 81ba166b49
feat(license): expiry epoch api 2024-01-21 00:14:28 +08:00
JimMoen 89128958ed
fix(dashboard): licence quota not provided to ce edition 2024-01-21 00:14:27 +08:00
JimMoen ecd0da9fde
feat(dashboard): `/monitor_current/nodes/{node}` provide `node_uptime` 2024-01-21 00:14:26 +08:00
JimMoen 7bc3a5090d
fix(dashboard): meck `emqx_retainer` in SUITE 2024-01-21 00:14:25 +08:00
JimMoen 3d6b65aced
chore: emqx_dashboard_monitor:current_rate/0 not exported anymore 2024-01-21 00:14:24 +08:00
JimMoen a35698009c
feat(dashboard): endpoint `/monitor_current` provides mor fields
- `retained_msg_count`
  Current retained_msg_count on each node and should be same on all nodes.

- `license_quota`
  Only for enterprise edition, provides the max limited connections num.
2024-01-21 00:14:23 +08:00
Serge Tupchii dc15d37dcc perf(emqx_bridge/connector): load and unload bridges/connectors in parallel
This should reduce app start/stop time, when a large number of bridges/connectors are not healthy.
2024-01-19 21:44:31 +02:00
Serge Tupchii d3a6870097 feat(emqx_utils): add pforeach/2,3 2024-01-19 21:44:31 +02:00
Serge Tupchii a6568dec75 perf(emqx_bridge/connector): apply post config bridge/connector changes in parallel
This can greatly improve the performance when many bridges/connectors are being changed,
e.g. when a backup file is being imported.

Fixes: EMQX-11751
2024-01-19 21:44:31 +02:00
Serge Tupchii 54457b7093 feat(emqx_utils): allow `infinity` timeout in `pmap/3` 2024-01-19 19:34:04 +02:00
Thales Macedo Garitezi a40a13b786
Merge pull request #12357 from thalesmg/fix-bridge-bpapi-interval-r55-20240119
fix(bridge_api): explicitly state bpapi version ranges
2024-01-19 13:08:44 -03:00
Thales Macedo Garitezi 57074015c6 feat(ds): allow customizing the data directory
The storage expectations for the RocksDB DB may be different from our usual data
directory.  Also, it may consume a lot more storage than other data.

This allows customizing the data directory for the builtin DS storage backend.

Note: if the cluster was already initialized using a directory path, changing that config
will have no effect.  This path is currently persisted in mnesia and used when reopening
the DB.
2024-01-19 13:07:24 -03:00
Thales Macedo Garitezi d22092e3da fix(mqtt_source): don't attempt local publish when legacy config is absent
Fixes https://emqx.atlassian.net/browse/EMQX-11765
2024-01-19 10:54:05 -03:00
Thales Macedo Garitezi e369c1b971 docs(mqtt_source): fix example qos 2024-01-19 10:54:05 -03:00
Thales Macedo Garitezi 4068258c78 fix(bridge_api): explicitly state bpapi version ranges
Fixes https://emqx.atlassian.net/browse/EMQX-11767

The currently supported version for a node may not be available at the time it's probed.
2024-01-19 10:20:57 -03:00
Zaiming (Stone) Shi 7d14a13b7b Merge remote-tracking branch 'origin/release-55' 2024-01-18 14:36:30 +01:00
Thales Macedo Garitezi e219c3eda1
Merge pull request #12166 from kjellwinblad/kjell/mqtt_conn_action_3/EMQX-11489
feat: refactor MQTT bridge to source, action, and connector
2024-01-18 09:28:14 -03:00
Zaiming (Stone) Shi 6b462a9100
Merge pull request #12349 from zmstone/0118-fix-authz-cache-excludes-type
0118 fix authz cache excludes type
2024-01-18 10:32:38 +00:00
zhongwencool 1b432eadd8
Merge pull request #12348 from emqx/elasticsearch-e550
Elasticsearch e550
2024-01-18 18:06:48 +08:00
JianBo He 5b064e399f chore: improve http connector logs format 2024-01-18 17:29:23 +08:00
zhongwencool dae835635c fix: don't crash in http SUITE 2024-01-18 16:45:46 +08:00
zhongwencool 59797cfea7 feat: es's update support doc_as_upsert 2024-01-18 16:44:56 +08:00
zhongwencool 91368a57ff test: add es docker CI test 2024-01-18 16:44:56 +08:00
zhongwencool 7f5fe91905 fix: es's action is atom not binary 2024-01-18 16:44:56 +08:00
aiotter 7802d6e018 chore: fix typos 2024-01-18 16:35:57 +08:00
Zaiming (Stone) Shi 556092b7d0 feat(authz/prometheus): add authz cache_miss counter 2024-01-18 09:01:17 +01:00
Zaiming (Stone) Shi 85b6a3454c fix(authz): use binary() type instead of string() for cache.excludes 2024-01-18 09:01:17 +01:00
lafirest 6d6242c27a
Merge pull request #12335 from lafirest/fix/type_clause
fix(iotdb): enhances type checking when converting value
2024-01-18 11:28:40 +08:00
Ilya Averyanov 74bf4042c5 fix(mqtt_bridge): render valid messages from incomplete rule data 2024-01-17 23:13:55 +03:00
firest b2af7fdd70 fix(sysk): fix a update issue for the Syskeeper forwarder 2024-01-17 12:51:10 +08:00
firest 238ecc68cf fix(iotdb): enhances type checking when converting value 2024-01-17 12:00:15 +08:00
Zaiming (Stone) Shi 4fa906731d
Merge pull request #12333 from zmstone/0116-add-tags-for-action-and-connectors
feat: add 'tags' field for action and connector
2024-01-16 18:56:57 +00:00
Serge Tupchii 80e82db282 test(emqx_cm_SUITE): use one helper function: `emqx_pool:flush_async_tasks/1` 2024-01-16 19:42:37 +02:00
Serge Tupchii b472b56883 perf(emqx_cm): use a dedicated pool for channel cleanup
This is to isolate channels cleanup from other async tasks (like routes cleanup),
as channels cleanup can be quite slow under high network latency conditions.

Fixes: EMQX-11743
2024-01-16 19:08:02 +02:00
Zaiming (Stone) Shi 01d52e37c4
fix: resource tag type should be binary string
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-01-16 17:35:22 +01:00
Thales Macedo Garitezi 2a41cad54f fix(sources): remote irrelevant `resource_opts` fields for sources
Since they don't use buffer workers, they shouldn't have buffer-related sub-fields.
2024-01-16 11:29:35 -03:00
Thales Macedo Garitezi a8f9e5676f docs(mqtt_bridge): add API examples 2024-01-16 10:42:52 -03:00
Kjell Winblad 60fab6ee45 refactor: attempt to improve function names 2024-01-16 10:42:52 -03:00
Kjell Winblad c6cd3adccb refactor: fix type upgrade calls and move compatiblitly logic
Some bridge V1 to V2 calls were wrong but did not seem to cause issues
(perhaps due to locking test coverage). This commit also move
compatibility logic from the API module to the emqx_bridge_v2 module
where most of the compatibility logic exists.
2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 440a543a85 docs: fix typo 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 938429f351 chore(mqtt_bridge): change schema to remote `remote` sub-fields and hide `local`
`local` is still needed for backwards compatibility
2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 007af20a30 test(bridge_v2_api): adapt more tests to sources 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi fc88a1ed1e test(sources_api): add some tests to cover `/sources` HTTP API
Also fixes a bug with `DELETE /sources/:id`
2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 8f304d3456 test(bridge_v2_api): refactor suite to use CT matrix 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi cc24fe6e93 feat(mqtt_consumer): add support for rule engine `FROM` 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 28de7c89c7 feat: add `/sources*` HTTP APIs 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi e6ccfa5b39 fix(mqtt_bridge): fixes after rebasing onto current `master`
Rebased on top of 7f57ec47d5
2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 6511693b2e refactor(action_api): prepare for `/sources` HTTP API 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 12dc9fbeb9 test(mqtt_bridge): fix assertion 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 862283ff7c test: fix expected connector name after name convention generation changed 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 7fc069da46 test: fix another broken test 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 1ad3100cad chore: add i18n 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 3597ee7c93 fix(mqtt_action): fix resource_opts schema 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi ab1b0dda67 refactor: fix typo 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 697c8f5ee1 test: fix broken tests 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi cc34660ab9 fix(actions): use backward-compatible ids 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 14b99737e9 fix(mqtt_bridge): add missing fields to POST api spec; fix test 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 7befe898d0 fix(mqtt_bridge): fix schema 2024-01-16 10:42:52 -03:00
Thales Macedo Garitezi 139da6d720 fix: don't double-write the transformed config; return a triplet in all cases 2024-01-16 10:42:52 -03:00
Kjell Winblad 2ecc775fb7 style: remove commented out code and fix copyright headers 2024-01-16 10:42:52 -03:00
Kjell Winblad 886ed55374 fix: don't call non-existing function 2024-01-16 10:42:52 -03:00
Kjell Winblad 145ed2e632 fix: elvis style error 2024-01-16 10:42:52 -03:00
Kjell Winblad f199a0f24a feat: refactor MQTT bridge to source, action, and connector
This commit:

* refactors the MQTT V1 bridge into connector, source and action
* Extends the compatibility layer so it works for sources
* Fixes the MQTT bridge test suite so that all test cases passes

We still need to add a HTTP API handling sources. Also, we still need to
add HTTP API example schemes and examples for the MQTT
connector/action/source.

We should also make sure that we handle the corner cases of the MQTT V1
bridge automatic upgrade downgrade in a sufficiently good way:

* An error is currently thrown when converting an MQTT V1 bridge without
  egress or ingress config.
* If there is a source and action with the same name we will currently
  throw an error in the compatibility layer.
* We will also throw an error when converting an MQTT V1 bridge with
  both ingress and egress.

The above is probably the right thing to do  but we have to make sure
that we return a reasonable error to the user when this happens.

(partly)
Fixes:
https://emqx.atlassian.net/browse/EMQX-11489
2024-01-16 10:42:52 -03:00
Zaiming (Stone) Shi 996a851cf6 chore: format username as string in log formatter 2024-01-16 14:42:10 +01:00
Zaiming (Stone) Shi 541525c50f fix(authz): fix authz result logs
prior to this fix, it's always the default authz result logged at
warning level
2024-01-16 14:40:53 +01:00
Zaiming (Stone) Shi 1fe1a62fe2 test: fix already exported function warning 2024-01-16 14:00:34 +01:00
Zaiming (Stone) Shi ab66986f16 feat: add 'tags' field for action and connector 2024-01-16 13:57:48 +01:00
JianBo He b6d0365027
Merge pull request #12267 from HJianBo/new-timeout-param-for-invite
feat(cluster): supports inviting nodes to join the cluster in an asynchronous manner
2024-01-16 10:11:20 +08:00
Zaiming (Stone) Shi 58a54adbb4 Merge remote-tracking branch 'origin/master' into release-55 2024-01-15 17:58:48 +01:00
lafirest 04d3e0335b
Merge pull request #12328 from lafirest/fix/iotdb
fix(iotdb): ensure the `data` field is `required`
2024-01-15 23:26:36 +08:00
firest 50ce2384d9 fix(iotdb): ensure the `data` field is `required` 2024-01-15 20:29:58 +08:00
William Yang d2d3ddb72a test(quic): listener port updates 2024-01-15 11:42:45 +01:00
William Yang c77837a9ea feat(quic): support reload with new binding port 2024-01-15 11:33:25 +01:00
zhongwencool 90c2a5445c
Merge pull request #12295 from zhongwencool/elasticsearch
feat: support elasticsearch bridge
2024-01-15 14:13:38 +08:00
zhongwencool ace443fc18 refactor: refactor es's action 2024-01-15 11:31:13 +08:00
JianBo He 4c40e754f4 chore: forward the async invite to leader node 2024-01-15 11:24:05 +08:00
zhongwencool e49d3ca50c feat: support elasticsearch bridge 2024-01-15 09:33:44 +08:00
Shawn e3fee93d9f fix: write quoted strings to influxdb failed 2024-01-15 09:29:09 +08:00
Shawn 1668e9ac7d fix: cannot write literal numbers to influxdb 2024-01-15 09:29:09 +08:00
JianBo He 93ef6766ef chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-13 15:29:58 +08:00
JianBo He d2991dae03 chore: make dialyzer happy 2024-01-13 15:29:58 +08:00
JianBo He 81209e0ded chore: update changes 2024-01-13 15:29:58 +08:00
JianBo He 944137ad45 chore: format error message 2024-01-13 15:29:58 +08:00
JianBo He 6ff4c560e4 feat: support invite node in async mananer 2024-01-13 15:29:58 +08:00
JianBo He a08f56db42 feat(cluster): expose the timeout parameter to invite node 2024-01-13 15:29:58 +08:00
Thales Macedo Garitezi 4736012d0a
Merge pull request #12317 from thalesmg/fix-mongo-batch-schema-m-20240112
fix(mongodb_action): make `batch_size` hidden and fixed in the API
2024-01-12 14:51:38 -03:00
Andrew Mayorov eca2f972bc
Merge pull request #12237 from keynslug/fix/EMQX-11483/broker
feat(routing): add route sync process pool
2024-01-12 14:44:31 +01:00
Thales Macedo Garitezi a5e7db793b fix(mongodb_action): make `batch_size` hidden and fixed in the API
Fixes https://emqx.atlassian.net/browse/ED-1171

The old bridge schema already had this override, but it had not been ported to actions.
2024-01-12 10:04:32 -03:00
lafirest 1c0ca72877
Merge pull request #12316 from lafirest/fix/iotdb
fix(iotdb): move the iot_version into IoTDB connector
2024-01-12 20:56:10 +08:00
Zaiming (Stone) Shi 61417f26d4 chore: bump versions to 5.5.x 2024-01-12 13:54:35 +01:00
Andrew Mayorov b4eac25678
chore(broker): fix code style issue
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 13:54:23 +01:00
Thales Macedo Garitezi d1ae6f5195
Merge pull request #12306 from thalesmg/fix-password-deobfuscate-m-20240111
fix(bridge_api): correctly deobfuscate secrets during dry run
2024-01-12 09:13:20 -03:00
firest b15106c753 fix(iotdb): robustify type verification
1. let the type is not case sensitive
2. return error if type is invalid
2024-01-12 19:22:06 +08:00
Andrew Mayorov 2f98f1faaf
fix(route-sync): ensure batch sync preserve idemopotency 2024-01-12 11:57:26 +01:00
Andrew Mayorov e21a3497c7
chore(route-sync): turn TODO into comment 2024-01-12 11:57:26 +01:00
Andrew Mayorov 8f4758d9d4
feat(route-sync): use the smallest possible min sync delay 2024-01-12 11:57:26 +01:00
Andrew Mayorov 2ac6cddf19
fix(route-sync): handle batch sync errors gracefully 2024-01-12 11:57:25 +01:00
Andrew Mayorov a28fc7bfa8
feat(route-sync): do not run empty batches 2024-01-12 11:57:25 +01:00
Andrew Mayorov 2dffd44985
feat(route-sync): allow to enable syncer pool only on cores/replicants
So we would able to roll it out and test more gradually.
2024-01-12 11:57:25 +01:00
Andrew Mayorov 884f784c1c
refactor(router): don't `emqx_broker_helper:monitor/1` in batches
As per out current understanding, this doesn't changes much in terms of
observability, since other nodes call `ekka:monitor(membership)` anyway,
so they will observe nodedowns without explicitly writing an entry into
the `?ROUTING_NODE` table.
2024-01-12 11:57:25 +01:00
Andrew Mayorov d6f731c4fc
fix(route-sync): use public function as mria activity target 2024-01-12 11:57:25 +01:00
Andrew Mayorov 7d037cfe91
chore(route-sync): clarify why wait indefinitely for reply 2024-01-12 11:57:24 +01:00
Andrew Mayorov 498b7a922d
chore(syncer): fix code style issues
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 11:57:24 +01:00
Andrew Mayorov 0b3f5f7c37
feat(syncer): allow to turn syncer pool on/off through config 2024-01-12 11:57:24 +01:00
Andrew Mayorov 38e13f2337
fix(syncer): start syncer pool before broker pool
As the latter depends on the former.
2024-01-12 11:57:24 +01:00
Andrew Mayorov a1ccf85c66
test(routesync): verify that syncer preserves consistency
Under a highly concurrent load. Be aware that this testcase is not
deterministic.
2024-01-12 11:57:24 +01:00
Andrew Mayorov 5aeff20f8b
fix(routesync): ensure causal relationships are preserved
At the cost of strict FIFO semantics though.
2024-01-12 11:57:23 +01:00
Andrew Mayorov f92b5b3f32
feat(stream): add simple stream over process message queue 2024-01-12 11:57:23 +01:00
Andrew Mayorov 54f8b47455
feat(routing): add route sync process pool
Dedicated to synchronizing local state updates with the global view
of the routing state.
2024-01-12 11:57:18 +01:00
firest 0f1aaa65bc fix(iotdb): move the `iot_version` into IoTDB connector 2024-01-12 17:52:31 +08:00
JianBo He 9e85b53c39
Merge pull request #12296 from HJianBo/fix-delayed-message-http-api-500
fix(delayed): fix http 500 error
2024-01-12 09:37:41 +08:00
Thales Macedo Garitezi 79a4a041e4 fix(bridge_api): correctly deobfuscate secrets during dry run
Fixes https://emqx.atlassian.net/browse/EMQX-11733
2024-01-11 15:42:14 -03:00
Ilya Averyanov ef0850c71f
Merge pull request #12305 from savonarola/1101-fix-channel-info-update
fix(conn): avoid storing incomplete channel info
2024-01-11 18:48:53 +02:00
Ilya Averyanov 4c66a1135b
Merge pull request #12303 from savonarola/0111-fix-indexing
fix(retainer): fix topic search by index
2024-01-11 18:44:56 +02:00
Ilya Averyanov 7b0b2a0527 fix(conn): avoid storing incomplete channel info 2024-01-11 18:08:13 +03:00
Ilya Averyanov 09d524144b fix(retainer): fix topic search by index 2024-01-11 16:00:58 +03:00
Thales Macedo Garitezi 6ddbfd7b43 fix(gcp_pubsub_producer): mark connector resource opts fields as deprecated
Fixes https://emqx.atlassian.net/browse/EMQX-11703
2024-01-11 09:53:36 -03:00
lafirest c026d3f18a
Merge pull request #12300 from lafirest/fix/clients_api
fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT
2024-01-11 19:10:54 +08:00
lafirest f042462d53
Merge pull request #12290 from lafirest/fix/retain_match
fix(retain): add `hasnext` into the meta data for the `GET /retain/messages`
2024-01-11 18:09:29 +08:00
firest aca9942fd4 fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT 2024-01-11 16:53:02 +08:00
firest f896cf2f4f fix(retain): add `hasnext` into the meta data for the `GET /retain/messages` 2024-01-11 16:14:40 +08:00
JianBo He 596607d549 chore: update bpapi.versions 2024-01-11 14:56:02 +08:00
JianBo He c4dd083fd9 fix(delayed): delete delayed messge on all nodes 2024-01-11 14:12:30 +08:00
JianBo He eea0ec135f fix(delayed): fix http 500 error 2024-01-11 10:36:55 +08:00
JianBo He b16920e796
Merge pull request #12285 from HJianBo/shorten-coap-params
feat(coap): support short param names
2024-01-11 09:09:09 +08:00
SergeTupchiy 8bea0711ac
Merge pull request #12196 from SergeTupchiy/EMQX-10891-route-cleanup-optimization
EMQX-10891 route cleanup optimization
2024-01-10 17:26:24 +02:00
Thales Macedo Garitezi 8532e0cf59
Merge pull request #12292 from thalesmg/fix-syskeeper-v2-only-m-20240110
fix(syskeeper_bridge): mark syskeeper forwarder as a v2-only action
2024-01-10 11:42:24 -03:00
Serge Tupchii 965ce5d446 perf: use mria:match_delete/2 to cleanup routes
This must be much more network efficient since both Mria and Mnesia
need only to replicate one op, e.g.: `{MatchPattern, clear_table}`
instead of replicating one per each key to be deleted.
2024-01-10 16:23:58 +02:00
Serge Tupchii febaaefc38 chore: upgrade ekka to 0.18.1 (mria 0.8.1)
Mria 0.8.1 release implements `mria:match_delete/2` function
2024-01-10 16:23:58 +02:00
Thales Macedo Garitezi b395924569
Merge pull request #12283 from thalesmg/fix-gcp-pubsub-produ-resopts-m-20240109
fix(gcp_pubsub_producer): fix connector `resource_opts` schema
2024-01-10 11:08:39 -03:00
Zaiming (Stone) Shi 7c8a36fc06
Merge pull request #12289 from zmstone/0108-support-acl-cache-masks
0108 support acl cache excludes
2024-01-10 13:40:29 +00:00
Andrew Mayorov 8d61f4665a
Merge pull request #12291 from keynslug/fix/bump-hocon-0.44.4
fix(conf): bump to hocon 0.40.4
2024-01-10 14:32:17 +01:00
Zaiming (Stone) Shi 1d46175a7d
Merge pull request #12286 from lafirest/fix/iotdb_enum
fix(iotdb): improve the schema by gathering atoms into enums
2024-01-10 13:25:53 +00:00
Thales Macedo Garitezi b69fc9af15 fix(gcp_pubsub_producer): fix connector `resource_opts` schema
Fixes https://emqx.atlassian.net/browse/EMQX-11703
2024-01-10 10:13:11 -03:00
Thales Macedo Garitezi 787f952ad8 fix(syskeeper_bridge): mark syskeeeper as a v2-only action
Fixes https://emqx.atlassian.net/browse/EMQX-11706
2024-01-10 10:09:51 -03:00
Zaiming (Stone) Shi 6fcb2fdb05 test: fix test case flakyness 2024-01-10 13:52:00 +01:00
Zaiming (Stone) Shi 9e8a67fd68 feat: support authz cache exclusion config
now one can configure a list of topic-filters to avoid
caching ACL check results

for example

authorization.cache.excludes = ["nocache/#"]

this means ACL check results for topics having 'nocache/' prefix
will not be cached
2024-01-10 13:52:00 +01:00
lafirest a1e86454e1
Merge pull request #12287 from lafirest/fix/sysk
fix(sysk): fix errors of proxy schema
2024-01-10 20:47:47 +08:00
Andrew Mayorov 8af9512a90
test(ft-conf): verify undefined password won't become `******` 2024-01-10 13:35:49 +01:00
Andrew Mayorov ff2fb95e5d
fix(conf): bump to hocon 0.40.4
Which includes a bugfix for undefined sensitive values obfuscation.
2024-01-10 13:09:08 +01:00
Thales Macedo Garitezi 6d3c397be8
Merge pull request #12282 from thalesmg/fix-mysql-del-action-m-20240109
fix(mysql_bridge): allow deleting bridge when sql contains undefined fields
2024-01-10 09:03:19 -03:00
William Yang 8cd99cb0c8
Merge pull request #12274 from qzhuyan/dev/william/quic-listn-conf-reload
feat: quic listener conf reload
2024-01-10 11:52:52 +01:00
firest 423607bf88 fix(sysk): fix errors of proxy schema 2024-01-10 17:53:55 +08:00
JianBo He 3038ded59e chore: make static_check happy 2024-01-10 17:53:39 +08:00
firest b29fd486af fix(iotdb): improve the schema by gathering atoms into enums 2024-01-10 15:12:42 +08:00
JianBo He 64613c6f0c feat(coap): support short param names
see: https://github.com/emqx/emqx/issues/10089
2024-01-10 14:52:59 +08:00
zhongwencool 3706c701fe
Merge pull request #12269 from HJianBo/return-detailed-error-message
chore: improve the error message if the parameter invalid
2024-01-10 14:02:59 +08:00
JianBo He 7b211b5a23
Merge pull request #12276 from lafirest/fix/iotdb_schema
fix(iotdb): improve the schema for IotDB
2024-01-10 10:21:01 +08:00
JianBo He 6ab8e31db0 test: fix failed tests 2024-01-10 09:53:22 +08:00
JianBo He da0307faa9 chore: update apps/emqx_management/src/emqx_mgmt_api_clients.erl
Co-authored-by: William Yang <mscame@gmail.com>
2024-01-10 09:52:27 +08:00
JianBo He f45298d14a chore: improve the error message if the parameter invalid 2024-01-10 09:52:27 +08:00
lafirest 3d7264c9ef
Merge pull request #12278 from lafirest/fix/limit
fix(api): adjust the maximum limit of page query to 10,000
2024-01-10 09:42:46 +08:00
JianBo He adcda51eff
Merge pull request #12277 from HJianBo/feat-delete-delayed-messages-via-topic
feat: support to delete delayed messages via topic name
2024-01-10 09:05:19 +08:00
firest 4e046eaa3d chore: bump version && update changes 2024-01-10 06:58:01 +08:00
Thales Macedo Garitezi abd4873091 fix(mysql_bridge): allow deleting bridge when sql contains undefined fields
Fixes https://emqx.atlassian.net/browse/EMQX-11655
2024-01-09 17:25:34 -03:00
Thales Macedo Garitezi d0588b86b5 fix(mysql_bridge): check prepare statement error messages
Fixes https://emqx.atlassian.net/browse/EMQX-11648
2024-01-09 17:25:34 -03:00
Zaiming (Stone) Shi 98e94d8619 docs: fix typo in comments 2024-01-09 19:16:55 +01:00
William Yang 2201dec2f4 feat(quic): 3s cutoff time for fast_close 2024-01-09 17:29:28 +01:00
William Yang 2483a4ecff feat(quic): support listener config reload 2024-01-09 17:29:25 +01:00
firest d1234e3f75 fix(api): adjust the maximum limit of page query to 10,000 2024-01-09 16:43:33 +08:00
William Yang 973f40041d chore: bump 0.0.311 2024-01-09 09:35:03 +01:00
firest 2bf7ac00a0 fix(iotdb): improve the schema for IotDB 2024-01-09 16:29:10 +08:00
lafirest cc00dd80ee
Merge pull request #12272 from lafirest/feat/retain
feat(retain): add two new endpoints for retain API
2024-01-09 15:59:42 +08:00
JianBo He 9ce96bafa3 feat: support to delete delayed messages via topic name 2024-01-09 15:06:06 +08:00
lafirest add1368f5d
Merge pull request #12270 from lafirest/fix/syskeeper
fix(sysk): improve the seakeeper schema
2024-01-09 14:41:42 +08:00
firest 23e9ed763e chore: bump version && update changes 2024-01-09 10:20:54 +08:00
zhongwencool 7365160eeb
Merge pull request #12275 from thalesmg/sync-r54-m-20240108
sync r54 into master
2024-01-09 10:05:06 +08:00
Thales Macedo Garitezi 7763b0fd34 chore: bump app vsns 2024-01-08 17:42:56 -03:00
Thales Macedo Garitezi 756980837c Merge branch 'release-54' into sync-r54-m-20240108 2024-01-08 17:39:42 -03:00
ieQu1 caf461fdf6 fix(ds): Don't start the supervision tree when feature is not in use 2024-01-08 17:59:19 +01:00
firest c180325e0a feat(retain): add two new endpoints for retain API 2024-01-08 19:11:43 +08:00
firest 714abaf73b chore: bump syskeeper version 2024-01-08 15:47:39 +08:00
firest 2fb1569363 fix(sysk): improve the seakeeper schema 2024-01-08 15:25:56 +08:00
zhongwencool ec34b6f41d refactor: remove redis_type from redis action 2024-01-08 15:09:37 +08:00
zhongwencool 3404f39fd2 chore: remove redundant redis action swagger example 2024-01-08 10:45:54 +08:00
lafirest 7f57ec47d5
Merge pull request #12261 from lafirest/fix/iotdb
feat(iotdb): improve the IoTDB bridge to v2 style
2024-01-08 09:46:07 +08:00
firest 6f4b22e376 test(iotdb): add template test case for IoTDB 2024-01-06 21:32:37 +08:00
firest d221f34ce8 chore: bump version && update changes 2024-01-06 13:34:07 +08:00
firest e20b024b6e fix(iotdb): fix iotdb testcases 2024-01-06 13:01:43 +08:00
Andrew Mayorov 803529968e
test(s3): fix httpc ssl opts 2024-01-05 19:35:13 +01:00
Ivan Dyachkov 06117c3a33 Merge remote-tracking branch 'upstream/release-54' into 0105-sync-r54 2024-01-05 14:20:38 +01:00
Ivan Dyachkov d04ac08d1d chore: release 5.4.1 2024-01-05 11:48:28 +01:00
firest d16458ccd0 feat(iotdb): improve the IoTDB bridge to v2 style 2024-01-05 16:54:31 +08:00
yr 37feeed560 fix: judgment condition always failure 2024-01-05 14:23:15 +08:00
Xinyu Liu da3937b80e
Merge pull request #12247 from emqx/split_influxdb_connector_actions
refactor: split influxdb bridges to actions and connectors
2024-01-05 14:07:35 +08:00
Shawn 2f08471303 fix: cannot found desc for influxdb_parameters 2024-01-05 11:59:13 +08:00
Shawn 9714092525 fix: fix: update more influxdb testcases 2024-01-05 11:20:27 +08:00
Andrew Mayorov 304fe84434
fix(mysql): expect password may be missing from config 2024-01-04 15:16:27 +01:00
Shawn 8ebd233a46 chore: update influxdb-erl to 1.1.12 2024-01-04 10:20:52 +08:00
zhongwencool 90404b62a4 fix: incorrectly attempt to update the field value when the previous field don't have value 2024-01-04 10:12:34 +08:00
Shawn 19e2ec9748 fix: update influxdb testcases 2024-01-03 18:39:29 +08:00
zhongwencool 00e8a36dbe fix: the API prompt garbled characters for updating the configuration 2024-01-03 15:34:45 +08:00
JianBo He 57c0a7c5ef
Merge pull request #12235 from zhongwencool/ft-config-update-api
fix: ft config update api return raw config
2024-01-03 14:07:22 +08:00
Ivan Dyachkov d4c83cf463 chore: tag 5.4.1-alpha.1 2024-01-02 14:37:45 +01:00
Shawn 904bd0270f chore: bump app versions 2024-01-02 18:04:06 +08:00
Shawn 7c7a0b2172 refactor: split influxdb bridges to actions and connectors 2024-01-02 18:01:22 +08:00
Andrew Mayorov a87df28dfc
Merge pull request #12243 from keynslug/fix/EMQX-11483/route-async-repl
fix(router): rely on local sync state updates to propagate routes
2023-12-29 16:31:00 +01:00
Andrew Mayorov 5c91984ad7
chore: leave comment describing need for tighter synchronization 2023-12-29 14:04:48 +01:00
Ilya Averyanov 9e0f3ce53b feat(ds): restore original add_generation/update_db_config callback semantics 2023-12-29 13:12:15 +03:00
JimMoen 5e100f52b8
style: erlfmt all `rebar.config` files and `bin/nodetool` 2023-12-29 09:08:03 +08:00
JimMoen a3f33cc8bc
chore: fix typos in code and suite 2023-12-29 09:07:51 +08:00
JimMoen ed10827343
docs: README spell 2023-12-29 09:07:25 +08:00
Andrew Mayorov b9627c420f
fix(router): rely on local sync state updates to propagate routes
Instead of (potentially lagging) global table state itself. Local
state should be the only source of truth anyway. Moreover,
synchronize parts of local state update logic through the broker
pool to ensure consistency.

The problem was that `emqx_router:has_route/2` check may observe a
stale route, which deletion is not yet replicated from the core node
to the local replicant node.

For example:
1. The only one subscriber per a given topic A unsubscribes from
   a replicant node.
2. The route to A is deleted by the emqx_broker.
3. Mria makes RPC to a core node, it succeeds and returns.
4. The client resubscribes or another client subscribes to the same
   topic A.
5. emqx_broker tries to add a route again:
   `emqx_router:do_add_route(Topic)`.
6. `emqx_router` checks if the route is present.
7. The stale route is present because deletion not replicated yet,
   so no route is being added.
8. Route deletion is replicated locally but it's too late:
   we already have a local subscriber without a route.

Co-Authored-By: Serge Tupchii <serge.tupchii@protonmail.com>
2023-12-28 19:16:31 +01:00
Andrew Mayorov 331bfaa535
fix(broker): avoid reclaiming per-topic counter twice 2023-12-28 17:43:50 +01:00
Andrew Mayorov a8cd609ff4
chore(broker): simplify subscribe code path 2023-12-28 17:43:42 +01:00
Zaiming (Stone) Shi 2b61f5290d
Merge pull request #12236 from zmstone/1227-ensure-short-mqtt-bridge-clientid
fix(bridge/mqtt): ensure short clientid
2023-12-28 16:06:09 +01:00
zhongwencool a6acebb4aa
Merge pull request #12240 from zhongwencool/ft-api-e-541
fix: return raw config when update ft's config
2023-12-28 22:25:22 +08:00
Andrew Mayorov 18dc9b2516
fix(s3): remove `emqx_bridge_http` from runtime dependencies
Otherwise it's impossible to test `emqx_s3` in isolation.
2023-12-28 14:13:22 +01:00
Andrew Mayorov 1d9374125c
chore: bump `emqx_s3` to 5.0.13 2023-12-28 14:12:33 +01:00
Andrew Mayorov 33ef964582
fix(s3): handle extra configured headers correctly
* Header names in config become atoms.
  (Probably because of blanket `unsafe_atom_key_map/1` in `emqx_config`)
* Header names should be normalized to lowercase.
  (Otherwise signature will be incorrect)
2023-12-28 13:55:05 +01:00
Zaiming (Stone) Shi 4669c4d552 fix(bridge/mqtt): ensure short clientid
Some mqtt brokers do not allow long client IDs.
To make it compatible with this limitation, this commit
tries to limit the number of bytes under 23 with a best-effort
attempt to derive it from the bridge name.
2023-12-28 11:53:27 +01:00
zhongwencool 1fd67564ce fix: return raw config when update ft's config 2023-12-28 14:49:36 +08:00
JianBo He b6391cf18c fix(bridge-http): compatible conf with the error format introduced in 5.3.2 2023-12-28 11:36:48 +08:00
zhongwencool e645fa4874 fix: ft config update api return raw config 2023-12-27 21:13:47 +08:00
Serge Tupchii 423b8b4274 fix(emqx_opentelemetry): use converter to convert legacy metrics config
HOCON converter is cleaner and safer option,
that will also work for configurations defined in emqx.conf.
Previously used `upgrade_raw_conf/1` is currently not supported in hocon_cli (used in bin/emqx).

Fixes: EMQX-11643
2023-12-27 14:15:12 +02:00
zhongwencool 711a9a06c8
Merge pull request #12221 from thalesmg/test-flaky-retry-m-20231222
test: reduce flakiness
2023-12-27 14:30:25 +08:00
zhongwencool 11feab983f fix: clean rpc_commit when force_leave cluster 2023-12-26 15:25:58 +08:00
Zaiming (Stone) Shi e421e9ce56 Merge remote-tracking branch 'origin/release-54' into 1222-sync-e5.4.0-build.2-to-master 2023-12-23 11:22:05 +01:00
Zaiming (Stone) Shi f36b842d94 chore: prepare for e5.4.0 retag 2023-12-23 11:18:19 +01:00
Zaiming (Stone) Shi 86a018d921 chore: bump app versions 2023-12-22 21:28:26 +01:00
Zaiming (Stone) Shi 9fdac4af0c Merge remote-tracking branch 'origin/master' into 1222-sync-e5.4.0-build.2-to-master 2023-12-22 21:27:27 +01:00
Thales Macedo Garitezi 890970345b test: reduce flakiness 2023-12-22 15:31:56 -03:00
Zaiming (Stone) Shi ac6e4cf2cc fix: config update for ft 2023-12-22 19:01:14 +01:00
Zaiming (Stone) Shi 733196aa83 chore: prepare for e5.4.0-build.2 2023-12-22 18:59:46 +01:00
Thales Macedo Garitezi 85963d3b45 test: restore connect timeout for test case 2023-12-22 14:10:24 -03:00
Zaiming (Stone) Shi 23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi 891fd972bd Merge remote-tracking branch 'origin/release-54' into 1221-prepare-bpapi-on-otp-26 2023-12-22 15:19:25 +01:00
Zaiming (Stone) Shi 5b5d4a6f68 chore: bump release version to e5.4.0-build.1 2023-12-22 14:00:12 +01:00
zhongwencool a16bce8c24 fix: deobfuscate ft's secret keys in api 2023-12-22 13:13:44 +01:00
Zaiming (Stone) Shi 20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi 322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Zaiming (Stone) Shi ecc31217fd Merge remote-tracking branch 'origin/master' into 1221-prepare-bpapi-on-otp-26 2023-12-22 12:48:06 +01:00
Zaiming (Stone) Shi 17d3037a13 chore: add 5.4.bpapi 2023-12-22 12:47:41 +01:00
ieQu1 ec0693861a fix(static_checks): Prepare for OTP26 2023-12-22 12:02:30 +01:00
zhongwencool 2d5f7e0a6d
Merge pull request #12215 from thalesmg/sync-r54-m-20231221
sync r54 to master
2023-12-22 10:24:29 +08:00
Thales Macedo Garitezi 035f5f977e Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231221 2023-12-21 17:53:05 -03:00
Zaiming (Stone) Shi f4286f3208 test(gcp_pubsub): increase wait timeout and fix falt injection 2023-12-21 20:59:51 +01:00
Zaiming (Stone) Shi 36d2929534 chore: prepare for release version 5.4.0 2023-12-21 19:51:12 +01:00
Zaiming (Stone) Shi a9b1b82546
Merge pull request #12210 from keynslug/feat/EMQX-11579/schema-type-doc
feat(schema): provide type-level documentation snippets
2023-12-21 18:46:22 +01:00
Zaiming (Stone) Shi d9b3280f30
Merge pull request #12214 from zhongwencool/update-actions-connectors
fix: emqx conf ctl load failed with connectors/actions
2023-12-21 17:39:44 +01:00
Andrew Mayorov aa3f8d6735
fix(typedoc): meld it into the field doc in the meantime 2023-12-21 17:15:54 +01:00
Andrew Mayorov 85d7518a7d
feat(schema): provide type-level documentation snippets
For stuff like `duration()`, `bytesize()` and `secret()` for now.
2023-12-21 17:15:54 +01:00
zhongwencool 37930f1d3c fix: emqx conf ctl load failed with connectors/actions 2023-12-21 22:09:53 +08:00
zhongwencool 74f1ce0955
Merge pull request #12211 from zhongwencool/redis-resource-opts
fix: redis resource_opts not working
2023-12-21 22:08:41 +08:00
Thales Macedo Garitezi 18b69809da Merge branch 'release-54' into sync-r54-m-20231221 2023-12-21 10:05:13 -03:00
Thales Macedo Garitezi d898573acf
Merge pull request #12208 from thalesmg/test-yet-another-attempt-r54-20231220
test(gcp_pubsub_consumer): yet another attempt to stabilize tests
2023-12-21 09:55:33 -03:00
zhongwencool 5d1a412d0c fix: redis resource_opts not working 2023-12-21 18:29:49 +08:00
lafirest de02dd21ca
Merge pull request #12157 from lafirest/feat/dsgen
feat(ds): add an API for making new generations
2023-12-21 09:30:17 +08:00
Thales Macedo Garitezi 471877598e
Merge pull request #12209 from thalesmg/fix-schema-file-typo-r54-20231220
ci: fix typo in file name
2023-12-20 17:49:07 -03:00
Thales Macedo Garitezi aeabff41c9 ci: fix typo in file name 2023-12-20 15:39:57 -03:00
Thales Macedo Garitezi b2c82ab052 test(gcp_pubsub_consumer): yet another attempt to stabilize tests
Hopefully, should work better after https://github.com/emqx/emqx/pull/12197
2023-12-20 13:21:13 -03:00
Thales Macedo Garitezi a43f49f8fa chore: prepare to tag `5.4.0-rc.1` 2023-12-20 12:43:08 -03:00
Thales Macedo Garitezi 2b03ae8daf
Merge pull request #12205 from thalesmg/fix-list-v2-actions-in-v1-bridges-r54-20231220
fix(bridges_v1): don't list v2-only bridges in API
2023-12-20 12:37:16 -03:00
Andrew Mayorov f39af14524
Merge pull request #12180 from keynslug/fix/gw-dtls-opts
fix(gw): use more conservative set of DTLS options
2023-12-20 15:24:56 +01:00
firest 612be8e280 fix(ds): make dialyzer happy 2023-12-20 22:09:03 +08:00
Thales Macedo Garitezi b61ccf3373 fix(bridges_v1): don't list v2-only bridges in API
Fixes https://emqx.atlassian.net/browse/EMQX-11614
2023-12-20 10:46:35 -03:00
Andrew Mayorov ccb9a977d6
test(gw-authn): switch to `emqx_cth_suite` + simplify testsuite 2023-12-20 14:38:18 +01:00
Andrew Mayorov 76c89ad372
test(gw-authz): switch to `emqx_cth_suite` + simplify testsuite
Also separate DTLS related MQTT-SN test config, so that it doesn't
leak into this test suite.
2023-12-20 14:37:49 +01:00
Serge Tupchii 333709164c refactor: change mria default shard transport from 'gen_rpc' to 'distr'
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-12-20 15:37:00 +02:00
Andrew Mayorov 32a64cf201
fix(gw): update DTLS listener setup according to esockd 5.11.x
Also drop unnecessary UDP default options.
2023-12-20 13:59:30 +01:00
Andrew Mayorov 62bb9938e7
test(exproto): switch to `emqx_cth_suite` + fix listener options 2023-12-20 13:59:30 +01:00
firest ed38ca67d5 fix(ds): Unified the names of the `add_generation` API 2023-12-20 18:58:03 +08:00
Andrew Mayorov 5c5ecbe3cf
fix(gw): unbreak schema + transform DTLS options properly
Thus ensuring full backward compatibility. Unsupported options
(`gc_after_handshake`, `ocsp`) are silently ignored now. Also make
sure that UDP configuration are part of DTLS option set, as expected
by `esockd`.
2023-12-20 11:46:08 +01:00
Andrew Mayorov 3ea2bbefaa
test(mqttsn): generate TLS certificates for tests on the fly 2023-12-20 11:46:07 +01:00
Andrew Mayorov 3d679f7e26
test(cth-tls): add test helpers for TLS related activities
Currently, only certificate issuing and related utilities are there.
2023-12-20 11:46:07 +01:00
Andrew Mayorov 78d3f49fe4
fix(gw): use more conservative set of DTLS options 2023-12-20 11:46:04 +01:00
Andrew Mayorov d3b32b64e1
Merge pull request #12201 from keynslug/feat/EMQX-11527/tls-hot-update
feat(listen): support hot config update of MQTT listeners
2023-12-20 09:45:12 +01:00
zhongwencool a4fe4ceb9e chore: add example for clients api 2023-12-20 14:53:05 +08:00
JianBo He cc15e3a03b
Merge pull request #12199 from zhongwencool/configs-put-bridge-v1-api
fix: support bridge v1 conf hocon-format in the put configs API.
2023-12-20 09:32:32 +08:00
JianBo He 1e288de1d9
Merge pull request #12203 from JimMoen/EMQX-11610-gw-OCPP-subscription-cnt
EMQX 11610 gw ocpp subscription cnt
2023-12-20 09:31:08 +08:00
Andrew Mayorov a6c26ce992
test(listen): stabilize flaky testcase 2023-12-19 23:36:52 +01:00
Andrew Mayorov baf46c9aa2
test(listen): also change CA on update and verify clients notice 2023-12-19 22:48:38 +01:00
JimMoen 465f71180a
fix(gw): return HTTP 400 for unimplemented handle call 2023-12-20 03:22:54 +08:00
Zaiming (Stone) Shi 9e24d08be3
Merge pull request #12202 from zmstone/1219-upgrade-gen_rpc-3.3.1
fix: upgrade to gen_rpc 3.3.1 which does not log cast args only arity
2023-12-19 19:41:46 +01:00
JimMoen 578a231a8d
fix(gw_ocpp): subscriptions cnt 2023-12-20 02:14:30 +08:00
JimMoen d37c5cc798
fix(gw_ocpp): frame outgoing badmatch 2023-12-20 01:27:12 +08:00
JimMoen af7b14ed3f
refactor(gw_ocpp): default conf macro readable 2023-12-20 01:27:12 +08:00
Thales Macedo Garitezi 55f0c1bbda
Merge pull request #12197 from thalesmg/test-more-flaky-fix-r54-20231218
test(gcp_pubsub_consumer): another attempt to stabilize flaky tests
2023-12-19 13:47:23 -03:00
Thales Macedo Garitezi 5b19e6b6f4
Merge pull request #12198 from thalesmg/sync-r54-m-20231218
sync r54 to master
2023-12-19 13:22:45 -03:00
Andrew Mayorov 7e4049620d
fix(listen): ensure limiter server state consistent with updates 2023-12-19 16:06:27 +01:00
Andrew Mayorov 4796f85dff
test(listen): verify SSL options of WSS listener are hot updateable 2023-12-19 16:06:27 +01:00
Andrew Mayorov 036048b9ab
feat(listen): support hot update of WS/WSS options as well
However, currently any transport option changes will cause listener to
restart, which slightly impacts the availability of the broker.
2023-12-19 16:06:27 +01:00
Andrew Mayorov 68bbbec556
chore: bump to esockd 5.11.1
With a typespec fix.
2023-12-19 16:06:27 +01:00
Andrew Mayorov a20ef0376b
test(exproto): fix too small buffer size for listener
These options have not been picked up by the listener before esockd
5.11.0, so it was not apparent that they are not working.
2023-12-19 16:06:27 +01:00
Andrew Mayorov fb2d4544b6
test(listen): verify that SSL options are hot-updateable 2023-12-19 16:06:26 +01:00
Andrew Mayorov da0f0f947e
feat(listen): support hot config update of esockd-based listeners 2023-12-19 16:06:26 +01:00
Zaiming (Stone) Shi 2b6176aa60 fix: upgrade to gen_rpc 3.3.1 which does not log cast args only arity 2023-12-19 12:46:42 +01:00
Andrew Mayorov cafd384466
chore: upgrade to esockd 5.11.0 2023-12-19 11:15:09 +01:00
zhongwencool 1c17121d44 chore: move redis_type into paramaters in actions 2023-12-19 17:09:08 +08:00
zhongwencool a46b415c77 fix: support bridge v1 conf hocon-format in the put configs API. 2023-12-19 16:40:45 +08:00