e5a3574d89
fix: maybe send willmsg
2024-02-22 17:12:16 +01:00
a30e8818fd
fix(s3-bridge): check for unobtainable credentials in healthcheck
2024-02-22 16:56:14 +01:00
69c7f858af
fix(s3): try to obtain credentials with default lhttpc
...
Otherwise `emqx_s3_client` will try to do it later by contacting the S3
endpoint instead of the metadata server.
2024-02-22 16:56:13 +01:00
88b1d9ba88
refactor: use atoms for root config fields and types
2024-02-22 16:51:40 +01:00
6311b582ec
test(willmsg): session taken over before willmsg delay /session expire
2024-02-22 16:39:27 +01:00
5397402396
test(willmsg): test will delay and session expires
...
will delay > session expire
will delay < session expire
timer triggered events are handled in seq, exclude the case of
(will delay == session expire)
2024-02-22 16:39:27 +01:00
dd62280e04
fix: handle delayed willmsg, part 1
2024-02-22 16:39:27 +01:00
b76c701b1c
test(takeover): add back the delay when takeover
2024-02-22 16:39:27 +01:00
6243cf0b0c
fix(kick): defer willmsg publish when conn terminates
...
because kick means shutdown connection AND delete session
2024-02-22 16:39:27 +01:00
9da4896f57
fix(mqtt): ensure publish willmsg when session expires
2024-02-22 16:39:24 +01:00
07eec31a8a
fix: willmsg not published in takeover
2024-02-22 16:35:59 +01:00
47757a0983
fix(mongodb_bridge): remove `local_topic` from v1 config transformation
...
Fixes https://emqx.atlassian.net/browse/EMQX-11893
2024-02-22 12:23:26 -03:00
205b97f732
fix(gcp_pubsub_consumer): fabricate topic mapping when producing v1 config
...
Fixes https://emqx.atlassian.net/browse/EMQX-11891
2024-02-22 10:27:28 -03:00
ae8f59979d
fix(utils): handle improper lists as well in `redact/1`
2024-02-22 13:35:11 +01:00
9893cc700b
Merge pull request #12565 from zhongwencool/fix-bpapi-bad-node
...
fix: bpapi announce's bad node name
2024-02-22 20:18:33 +08:00
f0aeaadc48
fix(api_key): enhanced bootstrap files for REST API keys
2024-02-22 18:40:24 +08:00
7bb9d5d8f6
Merge pull request #12543 from kjellwinblad/kjell/refactor/dynamodb/EMQX-11456
...
feat: refactor DynamoDB bridge to connector and action
2024-02-22 09:47:57 +01:00
ccf607ee2c
fix: bpapi announce's bad node name
2024-02-22 14:55:11 +08:00
985a3e3062
test: remove unused var warning
2024-02-21 18:14:46 -03:00
8e47503f7d
Merge pull request #12560 from SergeTupchiy/EMQX-11530-log-throttling-followup-fixes
...
fix: disable log throttling if primary log level is debug
2024-02-21 22:02:32 +02:00
f31e9e6a15
Merge pull request #12555 from thalesmg/ds-move-count-to-mgmt-m-20240221
...
refactor(mgmt): move persistent session counting function to mgmt module
2024-02-21 16:36:29 -03:00
ceb208f1eb
fix: disable log throttling if primary log level is debug
2024-02-21 21:19:15 +02:00
7feb444e31
Merge pull request #12557 from keynslug/fix/s3-bridge-startup
...
fix(s3-bridge): anticipate repeated connector start requests
2024-02-21 15:13:48 +01:00
9b6511b3e7
docs: fix spelling mistake in error message
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-21 14:57:55 +01:00
4678df2229
build: bump emqx_bridge_gcp_pubsub version
...
The CI check ./scripts/apps-version-check.sh requires that
emqx_bridge_gcp_pubsub's version number is bumped.
2024-02-21 14:59:03 +01:00
6561d989d6
feat: refactor DynamoDB bridge to connector and action
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11456
2024-02-21 14:58:48 +01:00
e98fb0db48
chore: bump `emqx_bridge_gcp_pubsub` version to 0.2.2
2024-02-21 14:28:58 +01:00
6b97983303
fix(s3-bridge): anticipate repeated connector start requests
...
Starting disconnected connector manually implies that there's no
attempt to stop it first.
2024-02-21 14:23:29 +01:00
191b435a48
chore: bump gcp bridge app version
2024-02-21 21:15:27 +08:00
ed23ad906a
fix: rabbitmq source metrics not working
2024-02-21 21:10:23 +08:00
415d27a4c9
Merge pull request #12544 from zmstone/0220-delete-non-prod-code
...
0220 delete non prod code in emqx_resource.erl
2024-02-21 14:08:08 +01:00
4118fcfdc6
Merge pull request #12486 from thalesmg/gcp-consumer-source-m-20240205
...
feat: convert `gcp_pubsub_consumer` to connector/source
2024-02-21 10:06:26 -03:00
25cdc2a04b
chore: bump version to 5.5.1-rc.1
2024-02-21 09:57:05 -03:00
b14e51d7c4
docs: delete useless comments
2024-02-21 13:33:29 +01:00
03138f8345
refactor(mgmt): move persistent session counting function to mgmt module
...
Follow up to https://github.com/emqx/emqx/pull/12500#discussion_r1496634087
2024-02-21 09:12:56 -03:00
529211b9ac
Merge pull request #12500 from thalesmg/ds-list-client-api-m-20240209
...
feat(ds): list disconnected persistent sessions in clients API
2024-02-21 09:02:27 -03:00
92c00ee21a
Merge pull request #12550 from keynslug/fix/bridge-s3-resopts
...
fix(bridge-s3): expose connector-level `resource_opts` properly
2024-02-21 11:37:55 +01:00
333eb34300
test: call emqx_resource:remove_local
2024-02-21 10:53:22 +01:00
de93d1c0a3
chore: add bpapi.versions
2024-02-21 10:53:22 +01:00
9c1d5e49dd
chore: fix a typo
2024-02-21 10:53:22 +01:00
c8f9e12de1
refactor: deprecate emqx_resource_proto_v1 with v2
2024-02-21 10:53:22 +01:00
06e0a7f2ed
refactor: delete emqx_resource:remove api
...
this api is only used in tests,
changed to call the _local flavor instead
2024-02-21 10:53:22 +01:00
8541d2efc1
refactor: delete emqx_resource:recreate
...
this api is only used in tests, changed to call the _local flavor
instead
2024-02-21 10:53:22 +01:00
1878fded5a
refactor: delete emqx_resource:create/4 and /5
...
the functions are only used in tests,
now the tests are rewritten to call create_local/4 and /5 instead
2024-02-21 10:53:22 +01:00
0cbe626aed
Merge pull request #12549 from zmstone/0221-sync-release-55
...
Sync `release-55` with i18n fixes to master
2024-02-21 10:52:36 +01:00
d8b6ecd185
fix(bridge-s3): expose connector-level `resource_opts` properly
2024-02-21 10:49:04 +01:00
ee02079661
Merge pull request #12548 from zhongwencool/fix-rabbitmq-source-crash
...
fix: bridge_v1 http api crash when create rabbitmq source
2024-02-21 16:34:43 +08:00
f7a7b7c8e0
Merge pull request #12545 from SergeTupchiy/fix-typo-tls_gc
...
chore(emqx_tls_certfile_gc): fix typo
2024-02-21 10:11:13 +02:00
da340c92a1
Merge remote-tracking branch 'origin/master' into 0221-sync-release-55
2024-02-21 09:11:12 +01:00
80be84c401
Merge pull request #12491 from zmstone/docs-sync-changes-for-i18n
...
docs: fix i18n
2024-02-21 09:10:25 +01:00
2a7a662c03
fix: bridge_v1 http api crash when create rabbitmq source
2024-02-21 15:30:05 +08:00
65cfece0a3
chore: delete 5.5.bpapi2
...
> This particular warning won't be seen in the real life. Also, we need to start storing
bpapi files for each minor release, since the plan to use hot upgrade has been abandoned.
> ...The original procedure for BPAPI life cycle was designed under the assumption that
we'll support hot upgrade between minor releases, so we'll only ever need to keep one
BPAPI file per major release, and rewrite it whenever a new minor version is
released. We'll need to revisit it.
https://emqxio.slack.com/archives/C04H4RHKKEY/p1706862980546309
2024-02-20 17:35:28 -03:00
d469f4158e
chore: bump app vsns
2024-02-20 16:53:57 -03:00
74dbbf6d9d
Merge branch 'release-55' into sync-r55-m-20240220
2024-02-20 16:33:34 -03:00
dd9f96211b
chore(emqx_tls_certfile_gc): fix typo
2024-02-20 21:19:36 +02:00
3a4c7f60e2
feat(ds): list disconnected persistent sessions in clients API
...
Fixes https://emqx.atlassian.net/browse/EMQX-11540
Note that not all information provided by disconnected in-memory sessions is available to
disconnected persistent sessions, nor does all of them make sense.
2024-02-20 14:52:55 -03:00
1de9e9dd01
Merge pull request #12541 from zmstone/0220-fail-fqdn-node-name-if-dns-discovery
...
Fail fast if node host is not IP address but cluster strategy is `dns` with `a` or `aaaa` record type
2024-02-20 17:48:02 +01:00
b6689d178f
fix(http_bridge): redact headers
...
Fixes https://emqx.atlassian.net/browse/EMQX-11864
```
2024-02-20T15:51:34.744509+00:00 [debug] msg: http_connector_received, mfa: emqx_bridge_http_connector:on_query_async/4(471), tag: QUERY_ASYNC, request: {"/bah",[{<<"Authorization">>,<<"******">>},{<<"content-type">>,<<"application/json">>},{<<"authorization">>,<<"******">>},{<<"content-type">>,<<"application/json">>}],<<"******">>}, state: #{port => 18083,request => #{path => [<<>>],body => undefined,headers => [{[<<"content-type">>],[<<"application/json">>]},{[<<"Authorization">>],#Fun<emqx_secret.0.85191762>}],method => [<<"undefined">>],max_retries => 2,request_timeout => 30000},host => "localhost",connect_timeout => 15000,pool_type => random,pool_name => <<"connector:http:p1">>,base_path => "/",installed_actions => #{<<"action:http:p1:connector:http:p1">> => #{path => [<<"/bah">>],body => undefined,headers => [{[<<"authorization">>],#Fun<emqx_secret.0.85191762>},{[<<"content-type">>],[<<"application/json">>]}],method => [<<"post">>],max_retries => 2,request_timeout => 30000,render_template_func => fun emqx_bridge_http_connector:render_template/2}}}, connector: <<"connector:http:p1">>, note: the request body is redacted due to security reasons
```
2024-02-20 12:57:13 -03:00
dc015e7a6f
fix(schema): validate cluster strategy and node name
...
If cluster strategy is configured as `dns`, the node name
must be IP address
2024-02-20 16:09:54 +01:00
f72558737c
Merge pull request #12520 from SergeTupchiy/EMQX-11530-log-throttling
...
feat: implement log throttling
2024-02-20 15:52:46 +02:00
86c85f3b45
Merge pull request #12512 from kjellwinblad/kjell/refactor/hstreamdb/EMQX-11458
...
feat: refactor HStreamDB bridge to connector and action
2024-02-20 13:30:29 +01:00
9bd0d1ba1b
feat: enable log throttling for potentially flooding log events
2024-02-20 14:20:21 +02:00
98ba300f7c
feat: implement log throttling
2024-02-20 13:23:17 +02:00
9639a977a3
fix: hidden batch_time for mongodb
2024-02-20 09:42:49 +08:00
c7bfaf51fd
chore: typo error (templete -> template)
2024-02-20 09:41:09 +08:00
2561ff94ec
fix: check channel id exist when get_channel_status
2024-02-20 09:38:33 +08:00
146d89bd89
feat: convert `gcp_pubsub_consumer` to connector/source
...
Fixes https://emqx.atlassian.net/browse/EMQX-11471
2024-02-19 17:54:26 -03:00
fd13a661cf
Merge pull request #12535 from zmstone/0219-refactor-delete-non-prod-code
...
refactor: delete non-prod code
2024-02-19 19:04:59 +01:00
11e4a295ed
fix: use connector type for constructing resource id
2024-02-19 14:42:35 -03:00
9a32895a1a
feat: convert `gcp_pubsub_consumer` to connector/source
...
Fixes https://emqx.atlassian.net/browse/EMQX-11471
2024-02-19 14:42:35 -03:00
ad53523e6b
refactor: delete non-prod code
...
So far the retainer backend type is always `built_in_database`.
The slightly over-engineered pre-implementation to support
another backend is likely not going to fly as the EMQX
resource frame work is mostly for auth and data integration.
i.e. not generic enough for retained messages.
2024-02-19 14:36:12 +01:00
68baa4dcaf
docs: fix i18n
2024-02-19 13:57:44 +01:00
bee7fa630b
chore: bump app vsns
2024-02-19 09:05:13 -03:00
dbb6725a47
fix(config): apply config upgrade to deprecated configs
...
Fixes https://emqx.atlassian.net/browse/EMQX-11845
Fixes https://github.com/emqx/emqx/issues/12452
2024-02-19 09:02:43 -03:00
def95aa22b
docs(HStreamDB bridge): make pool size descriptions better
2024-02-19 11:33:37 +01:00
866fea7648
Merge pull request #12530 from zmstone/0217-improve-frame-too-large-logging-context
...
0217 improve frame too large logging context
2024-02-19 11:33:31 +01:00
2ac4bde102
Merge pull request #12487 from zhongwencool/rabbitmq-source-test
...
feat: don't publish mqtt message in rabbitmq's source
2024-02-19 17:36:12 +08:00
8b0e15e402
refactor: rename 'hint' to 'cause' for MQTT fram parse failure reason
...
'reason' is maybe the wrapping field's name, so it was not used.
'hint' however, per our logging convention, is usually a free text
description for human to read.
change to 'cause' here because the field is always an atom and
it's use as shutdown counter in esockd_connection_sup
2024-02-19 09:38:35 +01:00
0fef19f86f
Merge pull request #12425 from emqx/clickhouse-bridge-v2
...
refactor: split clickhouse bridges to actions and connectors
2024-02-19 09:20:59 +08:00
668d3be390
test: for better test coverage
2024-02-18 19:23:35 +01:00
22dabcb3ea
feat(mqtt): add more logging context for frame_too_large error
2024-02-18 18:33:12 +01:00
ce0c27e802
fix: correct the swagger example for clickhouse connectors
2024-02-18 14:49:58 +08:00
6aae3ba2ed
refactor(prom_test): assert fun prefix
2024-02-18 10:42:40 +08:00
8ae0e78786
refactor: split clickhouse bridges to actions and connectors
2024-02-18 10:01:53 +08:00
451c8ecaf5
ci: add env vars to run clickhouse tests locally
2024-02-18 10:00:47 +08:00
c61b558423
test(prom): start mock pushgateway
2024-02-18 03:39:18 +08:00
ba1d24d054
test(prom_api): '/prometheus/auth' and '/prometheus/data_integration'
2024-02-18 02:32:25 +08:00
7bfabd7865
refactor: meck license checker for prom cases
2024-02-18 02:32:21 +08:00
d45d925529
test(prom_api): `/prometheus/stats` in prom/json format
2024-02-18 02:14:36 +08:00
9e16f33c31
test: `/prometheus/stats` endpoint format in json
2024-02-18 02:14:34 +08:00
52cfeee2b1
Merge pull request #12526 from zmstone/0216-fix-flaky-tests
...
test(listeners): try to stop all listeners after each test
2024-02-16 15:33:04 +01:00
9ffda0187f
Merge pull request #12525 from zmstone/0216-docs-add-ref-to-bridge_mode
...
docs: add a reference to the origin of is_bridge and bridge_mode
2024-02-16 15:32:52 +01:00
b7f3f48d38
Merge pull request #12524 from emqx/fix/bridge-stop-snabbkaffe
...
test(bridge_mqtt): Stop snabbkaffe servers
2024-02-16 15:17:29 +01:00
2b46cbab7a
Merge pull request #12517 from zmstone/0213-hocon-multiline-string-with-indentation
...
0213 hocon multiline string with indentation
2024-02-16 15:12:46 +01:00
f78c30c9ff
test: reduce false warning messages in test logs
2024-02-16 14:38:51 +01:00
3fa262f9ca
test(listeners): try to stop all listeners after each test
...
- Reduce the `eaddrinuse` flakiness
- Use ?FUNCTION_NAME as listener name to avoid name clashing between
tests
- Call emqx:remove_config for adhoc listeners created for testing
tombestone is designed for default listeners
2024-02-16 14:37:06 +01:00
8cfb22f0b8
fix(ds): Retry getting the shard leader
2024-02-16 12:42:48 +01:00
17a0513962
docs: add a reference to the origin of is_bridge and bridge_mode
2024-02-16 12:24:57 +01:00
59e4db98f7
test(bridge_mqtt): Stop snabbkaffe servers
2024-02-16 12:14:19 +01:00
497f631b7c
Merge pull request #12522 from zmstone/0216-upgrade-kafka-protocol
...
chore: upgrade kafka_protocol from 4.1.3 to 4.1.5
2024-02-16 11:47:31 +01:00
f57f617ba3
refactor(schema): ensure roots/0 and namespace/0 for all schema modules
2024-02-16 11:35:32 +01:00
a945892c52
Merge pull request #12508 from keynslug/fix/explicit-tab-mgmt
...
feat: manage mria tables explicitly during apps startup
2024-02-16 11:11:59 +01:00
242df0ffe9
chore: upgrade kafka_protocol from 4.1.3 to 4.1.5
2024-02-16 10:33:53 +01:00
a389d78b42
chore: delete stale code
2024-02-16 10:32:13 +01:00
b24321a8fc
feat: upgrade to hocon-0.41.0
...
hocon 0.41.0 added support for multiline string indentation.
now there is no need to escape (unless there is backslash)
quotes and line feeds etc.
2024-02-16 10:32:13 +01:00
ab2a469aff
Merge pull request #12489 from ieQu1/dev/ds-retainer
...
Durable sessions support retained messages
2024-02-15 21:59:32 +01:00
4484578c1f
Merge pull request #12507 from keynslug/fix/gw-app-no-scan
...
fix(gateway): avoid scanning modules of known gateway apps
2024-02-15 17:11:49 +01:00
5f85105801
feat(sessds): Specialize the interval queue for positive numbers
2024-02-15 16:55:05 +01:00
3adbe65a58
refactor(sessds): Unify logic for QoS 1 and 2
2024-02-15 11:34:13 +01:00
c781240459
feat(sessds): Add support for the retainer
...
Note: this is currently not ideal. Retained messages won't be
redelivered.
2024-02-15 11:34:13 +01:00
5e442fcbca
fix: labels and descriptions
2024-02-15 11:08:08 +01:00
6514659733
test(mgmt): update and simplify testsuites setup
2024-02-15 11:04:58 +01:00
94254ec05b
feat(sessds): Correct handling of gaps in the seqno series
2024-02-15 10:49:02 +01:00
0d0e26d6af
fix: set `info` level for potentially flooding log events
2024-02-15 11:02:03 +02:00
cffb52ab28
test(plugins): update and simplify test suite
2024-02-14 19:00:04 +01:00
4e8dfb48b7
fix: elvis problems
2024-02-14 17:54:14 +01:00
a153d758c3
feat: refactor HStreamDB bridge to connector and action
...
This commit also upgrades the hstreamdb_erl driver library and change
the action/bridge to use the new hstreamdb_erl.
Much of the code for the new API is copied from:
be1a1604dd/lib-ee/emqx_rule_actions/src/emqx_backend_hstreamdb_actions.erlhttps://emqx.atlassian.net/browse/EMQX-11458
2024-02-14 17:34:35 +01:00
7272ef25d4
feat(emqx_auth): implement API to re-order all authenticators/authz sources
...
Fixes: EMQX-11770
2024-02-14 14:35:46 +02:00
35bf805504
test(exhook): update and simplify testsuites setup
2024-02-13 21:02:39 +01:00
ece1c6c6dc
test(prometheus): update and simplify testsuites setup
2024-02-13 20:36:23 +01:00
a64850a84b
test(sso): update and simplify testsuites setup
2024-02-13 20:36:23 +01:00
b15e81baf2
test(slowsub): update and simplify testsuite setup
2024-02-13 20:36:23 +01:00
d0e507eba4
test(trie): fix testsuite setup
2024-02-13 20:36:22 +01:00
841fdea124
test(conf): update and simplify some testcases
2024-02-13 20:36:22 +01:00
0a9cbe3080
test(license): update and simplify tests
2024-02-13 20:36:22 +01:00
24dfa41722
feat(cth-suite): use cheaper heuristic for schema modules
2024-02-13 20:36:22 +01:00
8f2a4f7b19
fix(cth-suite): use cheaper check for loaded applications
2024-02-13 20:36:22 +01:00
82a4e6ef68
feat(psk): manage mria tables explicitly during startup
...
Also switch test suite to `emqx_cth_suite` tooling.
2024-02-13 20:36:22 +01:00
b07df487f0
feat(modules): manage mria tables explicitly during startup
2024-02-13 20:36:22 +01:00
b6d77c164e
feat(emqx-mgmt): manage mria tables explicitly during startup
2024-02-13 20:36:22 +01:00
21780e2126
feat(emqx-conf): manage mria tables explicitly during startup
2024-02-13 20:36:21 +01:00
8f7b0ac498
docs(jt808): fix doc references
2024-02-13 16:27:41 +01:00
366827390e
Merge pull request #12488 from kjellwinblad/kjell/refactor/rocketmq_bridge/EMQX-11467
...
feat: refactor RocketMQ bridge to connector and action
2024-02-13 16:19:21 +01:00
0e1d27c836
fix(gateway): avoid scanning modules of known gateway apps
...
Since the set of known gateways is already predefined in `emqx_gateway_utils`
it makes no sense to scan over all modules just to find out the same set of
modules. Cutting out this scanning saves few seconds per each `emqx_conf`
application startup, which is especially noticeable when running tests.
2024-02-13 15:13:36 +01:00
6fbb6f6846
Merge pull request #12506 from keynslug/fix/EMQX-11830/recoverable
...
fix(s3-bridge): handle recoverable AWS errors
2024-02-13 12:44:23 +01:00
eff149e676
feat(emqx-auth-mnesia): manage mria tables explicitly during startup
2024-02-12 19:17:22 +01:00
21a5751575
feat(emqx): manage mria tables explicitly during startup
2024-02-12 19:17:09 +01:00
a4eac75b25
fix(s3-bridge): handle recoverable AWS errors
2024-02-12 18:20:03 +01:00
edd28be090
Merge pull request #12499 from savonarola/0209-emqx-banned-extended-rules
...
feat(banned): allow ban by clientid/username regexps, peerhost cidrs
2024-02-12 17:12:47 +02:00
a9fdf9f1f9
Merge pull request #12495 from keynslug/ft/EMQX-11830/s3-bridge-simple
...
feat(s3): introduce S3 connector and action
2024-02-12 16:05:22 +01:00
2b1231b980
fix(s3-bridge): ensure bridge and connector names are identical
2024-02-12 13:46:54 +01:00
0edc5b5992
docs(s3): fix README example
2024-02-12 10:44:18 +01:00
976099f5fb
fix: cleanups due to problems found by @thalesmg
2024-02-12 10:16:17 +01:00
d4fb3e83f6
chore: upgrade to wolff 1.10.2
...
Wolff 1.10.2 added an enhancement which makes Kafka resource health
checks more lightweight and lowers the chance of false resource alerts.
2024-02-12 09:05:44 +01:00
90fd2b26d3
feat(banned): allow ban by clientid/username regexps, peerhost cidrs
2024-02-10 17:59:22 +03:00
0515c5528f
test: add facility for starting `emqx_dashboard` with `emqx_cth_suite`
2024-02-09 14:29:04 -03:00
c108262771
chore(s3-bridge): use `emqx_connector_schema` helper for brewity
2024-02-09 16:08:33 +01:00
8ac8d14abd
chore(emqx_mongodb): bump app vsn to 0.1.6
2024-02-09 16:29:42 +02:00
c3b7d76102
perf: un-deprecate mongodb topology pool_size and set it to 10 by default
...
Benchmarks proved that it effects performance:
10 ecpool workers / 10 topology pool workers outperform
100 ecpool / 1 topology worker.
Fixes: EMQX-11825
2024-02-09 16:28:53 +02:00
e248cd6c66
perf: upgrade mongodb-erlang to v3.0.23
...
Fixes: EMQX-11825
2024-02-09 16:18:32 +02:00
f476f4343b
fix(release): add `emqx_bridge_s3` to applications list
2024-02-09 14:16:32 +01:00
2a4e37869e
fix(dashboard): provide full context in startup errors
2024-02-09 14:16:32 +01:00
802c760406
feat(s3): introduce S3 connector and action
...
This is a trivial connector based on `emqx_s3` and simple action
that maps each incoming event into an S3 object upload. Due to
current `emqx_s3` limitation this bridge is compatible with
backends providing S3 API with path-style bucket access.
2024-02-09 14:16:32 +01:00
755b59b7fe
Merge pull request #12494 from SergeTupchiy/EMQX-11825-mongodb-connector-perf
...
Improve mongodb connector performance
2024-02-09 13:12:12 +02:00
cbebc59115
Merge pull request #12492 from zmstone/0208-fix-return-receive-maximum-in-connack
...
fix(emqx_channel): return Receive-Maximum in CONNACK when no error
2024-02-09 12:08:46 +01:00
8f66bd9ddf
fix(s3-client): make log levels saner
2024-02-09 12:03:41 +01:00
4ff04ab1f3
feat(s3): separate concepts to make app reusable in bridges
2024-02-09 12:03:41 +01:00
a74c828e19
test: add test case to cover Receive-Maximum in CONNACK
2024-02-09 10:42:48 +01:00
a5266f68ec
feat(s3): switch schema to use secrets with loader support
...
This will make applications using `emqx_s3` follow the same
conventions as bridges and support loading secrets from files
at runtime.
2024-02-08 22:43:20 +01:00
1d9ce709f4
perf: un-deprecate mongodb topology pool_size and set it to 10 by default
...
Benchmarks proved that it effects performance:
10 ecpool workers / 10 topology pool workers outperform
100 ecpool / 1 topology worker.
Fixes: EMQX-11825
2024-02-08 20:39:19 +02:00
c9ee5addb4
perf: upgrade mongodb-erlang to v3.0.23
...
Fixes: EMQX-11825
2024-02-08 20:00:26 +02:00
64367d834b
test(bridge_v2): actually check schema
2024-02-08 13:39:31 -03:00
cf22692c74
fix(emqx_channel): return Receive-Maximum in CONNACK when no error
2024-02-08 17:05:16 +01:00
e284a83f73
feat: refactor RocketMQ bridge to connector and action
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11467
2024-02-08 16:21:58 +01:00
a0f8e4f328
test: rafator rabbitmq test SUITE, delete redundance code
2024-02-08 12:38:18 +08:00
90ba2977fe
feat: don't publish mqtt message in rabbitmq's source
2024-02-08 12:38:18 +08:00
811edb32a2
Merge pull request #12485 from ieQu1/dev/ds-fix-unsubscribe
...
fix(ds): Fix unsubscribe logic related to the replay
2024-02-07 21:38:25 +01:00
19c6d1127f
refactor(sessds): Extract subscription mgmt logic to separate module
2024-02-07 15:27:11 +01:00
3000a8f286
fix(sessds): Postpone deletion of the subscription until fully acked
2024-02-07 14:51:02 +01:00
3f13aed3de
Merge pull request #12481 from lafirest/fix/action_info
...
fix(tdengine): remove redundant codes
2024-02-06 11:39:25 +08:00
eace8f2fcb
Merge pull request #12423 from zhongwencool/rabbitmq-e560
...
feat: rabbitmq bridge v2 integration
2024-02-06 09:55:05 +08:00
30eb54e86b
fix(sessds): Delay unsubscribe until full ack of in-flight messages
2024-02-06 02:49:07 +01:00
a9c55f7568
feat(sessds): Consider #srs with only QoS0 messages fully acked
2024-02-06 02:49:07 +01:00
d3db53626b
Merge pull request #12483 from zmstone/0205-docs-fix-typos
...
Improve emqx ctl conf cluster_sync CLI and help text
2024-02-06 09:11:35 +08:00
3e599e5d8e
chore: add maybe feature enable on rabbitmq_bridge
2024-02-06 07:57:13 +08:00
e3cb49db8e
chore: add copyright header
2024-02-06 07:57:13 +08:00
688701eedb
chore: add desc for rabbitmq source
2024-02-06 07:57:13 +08:00
b444c82a42
feat: rabbitmq bridge v2 integration
2024-02-06 07:57:12 +08:00
280fcd8c52
Merge pull request #12437 from ieQu1/dev/optimize_make_filter
...
Optimize emqx_ds_bitmask_keymapper:make_filter function.
2024-02-05 17:32:28 +01:00
4665837cf0
fix(ds): Apply review remarks
2024-02-05 16:52:06 +01:00
d7b0456bb0
feat: rename emqx ctl cluster_sync 'tnxid' to 'inspect'
2024-02-05 16:50:55 +01:00
e5b14a38ed
Merge pull request #12472 from thalesmg/fix-bridgev2-api-multicall-m-20240202
...
fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
2024-02-05 10:28:28 -03:00
c7888ad1f1
Merge pull request #12475 from ieQu1/dev/lean-stream
...
Use a more compact data structure to represent streams
2024-02-05 13:55:24 +01:00
e1ff276d76
Merge pull request #12471 from thalesmg/fix-config-upgrade-m-20240202
...
fix(config): apply config upgrade to deprecated configs
2024-02-05 09:05:12 -03:00
b4a6be0e7c
fix(tdengine): remove redundant codes
2024-02-05 18:50:31 +08:00
571786b6c5
Merge pull request #12480 from lafirest/fix/td_v1_schema
...
fix(tdengine): keep the bridge schema still compatible with v1
2024-02-05 18:17:55 +08:00
698ba3f271
fix(ds): Optimize emqx_ds_bitmask_keymapper:make_filter
...
This optimization makes idle polling faster
2024-02-05 10:54:19 +01:00
8edbec5929
refactor(ds): Clarify the language used in ds_bitmapper
2024-02-05 10:54:18 +01:00
75023f2ca3
Merge pull request #12442 from ieQu1/dev/ds-license-apache
...
Relicense apps/emqx_durable_storage under Apache 2.0
2024-02-05 10:16:52 +01:00
4a2c5a3161
fix(tdengine): keep the bridge schema still compatible with v1
2024-02-05 13:22:17 +08:00
f7b0d1bc9d
chore: add duplicated key assert in swagger.json
2024-02-04 11:53:37 +08:00
b1150a5f88
fix(schema): bridge oracle connector schema
2024-02-04 10:53:13 +08:00
2e2f3cb2aa
fix(sessds): Avoid stream hash collisions
2024-02-03 21:15:54 +01:00
2e56810ea2
refactor(ds): Use a simple improper list to represent the streams
2024-02-03 21:15:54 +01:00
76d242df9b
fix(bridge_v2_api): avoid calling nodes that do not support minimum bpapi
...
Fixes https://emqx.atlassian.net/browse/EMQX-11834
2024-02-02 17:36:08 -03:00
76401a302a
fix(config): apply config upgrade to deprecated configs
...
Fixes https://emqx.atlassian.net/browse/EMQX-11845
Fixes https://github.com/emqx/emqx/issues/12452
2024-02-02 15:12:08 -03:00
ed8660cb68
Merge pull request #12458 from id/0202-sync-release-55
...
sync release-55
2024-02-02 17:20:38 +01:00
81433282be
Merge pull request #12467 from SergeTupchiy/autocluster-dns-aaaaa-record-type
...
feat(emqx_conf): support AAAA cluster dns record type
2024-02-02 18:13:06 +02:00
b1a05c7b59
Merge pull request #12326 from zmstone/0111-unregister-session-with-timestamp
...
0111 unregister session with timestamp
2024-02-02 17:00:23 +01:00
e0b5d9fae7
Merge pull request #12464 from ieQu1/dev/feat-ds-cli
...
feat(ds): Add a CLI interface to inspect status of DS databases
2024-02-02 14:14:48 +01:00
5e0c3d4051
feat(emqx_conf): support AAAA cluster dns record type
...
Ekka autocluster DNS already supports AAAA records, so it's only needed to update enum in schema.
2024-02-02 12:13:29 +02:00
fa0baf2f38
Merge pull request #12439 from kjellwinblad/kjell/refactor/oracle_bridge/EMQX-11464
...
feat: refactor Oracle bridge to connector and action
2024-02-02 10:53:15 +01:00
84b6d7d720
fix: remove 5.5.bpapi2 file
2024-02-02 10:38:50 +01:00
3d2ac97c61
feat(ds): Add a CLI interface to inspect status of DS databases
2024-02-02 10:11:01 +01:00
f0569d8ae8
refactor: use mria:async_dirty to group dirty ops
2024-02-02 09:26:43 +01:00
24d8a9ab00
Merge pull request #12449 from lafirest/feat/td
...
feat(tdengine): improve the TDengine bridge to v2 style
2024-02-02 16:17:01 +08:00
dd490de2e1
chore: emqx 5.5.0
2024-02-02 08:48:56 +01:00
f1c7e716ce
chore: 5.5.0-rc.2
2024-02-02 08:48:56 +01:00
9aad7997ca
chore: compatible the contet-type sytanx
2024-02-02 08:48:56 +01:00
dae3a94670
chore: 5.5.0-rc.1
2024-02-02 08:48:56 +01:00
c724d2127e
Merge pull request #12457 from lafirest/fix/opentsdb
...
fix(opents): change the schema of tags to object style
2024-02-02 15:33:50 +08:00
17d79026cd
test: fix emqx_connector_schema eunit missing fields
2024-02-02 14:43:25 +08:00
5291adf2c0
fix(opents): change the schema of tags to object style
2024-02-02 13:07:49 +08:00
1a372ce39a
fix(swagger): no-need call common_fields when `emqx_connector_schema:api_fields/3` used
2024-02-02 10:54:49 +08:00
2241461acb
chore: update change & bump version
2024-02-02 07:48:44 +08:00
af3f43ffd5
fix: use emqx_resource macro for status
...
Thanks for the suggestion @thalesmg
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 17:24:25 +01:00
98d1094d73
feat(sessds): Expose subscriptions in the REST API
2024-02-01 16:47:57 +01:00
016fbd2c5c
chore: remove commented out code
2024-02-01 16:18:37 +01:00
dfad020c49
feat(tdengine): improve the TDengine bridge to v2 style
2024-02-01 23:14:24 +08:00
a4272c71dc
feat: refactor Oracle bridge to connector and action
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11464
2024-02-01 16:03:05 +01:00
ae1ef9d1bb
Merge pull request #12428 from savonarola/0130-simplify-boot-machine-runtime-deps
...
chore(emqx_machine): refactor injecting runtime deps
2024-02-01 16:32:53 +02:00
80ff1b1657
Merge pull request #12434 from qzhuyan/fix/william/kafka-dont-stacktrace-when-no-partiion-count
...
kafka-bridge: dont stacktrace when no partition count
2024-02-01 15:09:45 +01:00
3e518c1876
Merge pull request #12427 from zmstone/0130-limit-kafka-partitions
...
0130 limit kafka partitions
2024-02-01 14:39:52 +01:00
6a092aeb69
chore(emqx_machine): refactor injecting runtime deps
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-02-01 16:13:49 +03:00
16444b7da9
fix(bridge-kafka): no stacktrace when fetch topic partition count fail
2024-02-01 14:09:15 +01:00
3ad71f75e6
chore: emqx 5.5.0
2024-02-01 11:07:02 +01:00
42a4bc3382
Merge pull request #12435 from zhongwencool/dont-add-disable-resource-to-connector
...
fix: don't add disable bridge to connector's channel
2024-02-01 10:39:47 +08:00
0d5911d942
Merge pull request #12422 from zhongwencool/upgrade-erlfmt-to-v1.3.0
...
feat: upgrade erlfmt to support maybe syntax
2024-02-01 09:34:33 +08:00
f9d16340e0
fix: don't add disable bridge to connector's channel
2024-02-01 09:12:43 +08:00
139f5dc3bd
chore(ds): Remove an obsolete document; superceded by ./README.md
2024-02-01 00:14:08 +01:00
b50d6bf1fd
chore(ds): Change the license to Apache 2.0
...
Due to technicalities parts of the original code were licensed under
BSL.
In preparations for the public release of the feature, the license has
been changed to Apache 2.0
2024-02-01 00:10:48 +01:00
f7b12470dd
test(ds): fix inter-suite flakiness
...
Attempt to mitigate this frequent source of flakiness:
```
=CRASH REPORT==== 31-Jan-2024::17:30:15.025404 ===
crasher:
initial call: emqx_ds_replication_layer_egress:init/1
pid: <0.11312.0>
registered_name: []
exception error: no match of right hand side value {error,
no_leader_for_shard}
in function emqx_ds_replication_layer_egress:init/1 (/emqx/apps/emqx_durable_storage/src/emqx_ds_replication_layer_egress.erl, line 93)
in call from gen_server:init_it/2 (gen_server.erl, line 980)
in call from gen_server:init_it/6 (gen_server.erl, line 935)
ancestors: [<0.11310.0>,<0.11304.0>,emqx_ds_builtin_databases_sup,
emqx_ds_builtin_sup,emqx_ds_sup,<0.11236.0>]
message_queue_len: 0
messages: []
links: [<0.11310.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 376
stack_size: 28
reductions: 231
neighbours:
```
2024-01-31 15:42:26 -03:00
d51deac222
fix(ds): use configured data dir for site storage
2024-01-31 15:42:26 -03:00
3b6fbff9a5
test: fix compile warnings
2024-01-31 19:32:57 +01:00
3b42a7425b
feat(kafka_producer): add partitions_limit
2024-01-31 19:26:01 +01:00
53c217c383
refactor: micro optimization
2024-01-31 19:25:57 +01:00
b3657f1892
Merge pull request #12436 from savonarola/0131-fix-stream-replay
...
fix(sessds): fix renew stream logic
2024-01-31 18:25:17 +02:00
3f71a98e5e
chore: 5.5.0-rc.2
2024-01-31 17:06:54 +01:00
a60aeed539
Merge pull request #12398 from lafirest/feat/api-docs
...
feat(dashboard): expose the `swagger_support` option
2024-01-31 23:31:56 +08:00
fe46434687
fix(sessds): fix renew stream logic
2024-01-31 18:02:29 +03:00
2680fae9b6
Merge pull request #12420 from kjellwinblad/kjell/fix/kinesis/refactoring/problems/thanks_HJianBo
...
fix: kinesis schema problems found by @HJianBo
2024-01-31 15:52:45 +01:00
e9c8446d57
feat: upgrade erlfmt to support maybe syntax
2024-01-31 20:06:49 +08:00
b787f28d27
chore: compatible the contet-type sytanx
2024-01-31 17:49:46 +08:00
38047108a4
test: add test coverage for emqx_cm_registry_keeper module
2024-01-31 09:50:12 +01:00
209331ad33
test: fix config test
2024-01-31 09:50:12 +01:00
87a2368e37
feat(metrics): add cluster_session guage
2024-01-31 09:50:12 +01:00
509ab6f35a
feat(api): add /sessions_count api to count sessions
2024-01-31 09:50:12 +01:00
562a2736ae
feat: add `broker` root to hot-config schema
2024-01-31 09:50:12 +01:00
e9318752e6
feat: store session unregistration timestamp in emqx_cm_registry table
2024-01-31 09:50:12 +01:00
5bc67cb288
Merge pull request #12432 from ieQu1/ds_ignore_fully_replayed_streams
...
fix(sessds): Stream scheduler must ignore fully replayed streams
2024-01-31 09:51:49 +02:00
13df7fa46e
fix(dashboard): add test case & update change
2024-01-31 14:14:56 +08:00
e843d9fd91
fix(sessds): Stream scheduler must ignore fully replayed streams
2024-01-31 00:20:54 +01:00
a2ba8fc645
chore: 5.5.0-rc.1
2024-01-30 18:58:00 +01:00
274f378c6e
fix: better delete of local_topic as suggested by @thalesmg
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-01-30 17:53:32 +01:00
c7f1e44513
Merge pull request #12424 from lafirest/fix/opentsdb
...
feat(opentsdb): supports more flexible tags schema
2024-01-30 22:05:30 +08:00
55fdc83aa5
Merge remote-tracking branch 'origin/release-55' into 0130-syn-release-55-to-master
2024-01-30 12:06:43 +01:00
c0674913e2
Merge pull request #12421 from HJianBo/fix-import-users
...
fix(user_import): ensure the last record overwrites previous one
2024-01-30 18:49:35 +08:00
81f96f1a68
feat(opentsdb): supports more flexible tags schema
2024-01-30 18:43:39 +08:00
03ff6f2ddf
Merge pull request #12410 from SergeTupchiy/EMQX-11812-subscribe-performance-degraded
...
perf(emqx_broker): pick broker pool worker by topic/shard pair to dis…
2024-01-30 11:48:59 +02:00
db3f285054
Merge pull request #12417 from emqx/port-msg-expiry-interval-from44
...
feat: configurable server side message_expiry_interval
2024-01-30 17:10:32 +08:00
206af96a33
ci: update testcases for message-expiry-interval
2024-01-30 15:46:11 +08:00
4190682a16
chore: fix the data type and example value for cluster invitation result
2024-01-30 15:29:46 +08:00
aedfc8e8c0
fix(user_import): ensure the last record overwrites previous one
2024-01-30 14:14:20 +08:00
35c4ef2ee2
Merge pull request #12407 from zmstone/0126-script-to-update-bsl-license-convert-time
...
0126 script to update bsl license convert time
2024-01-30 09:54:34 +08:00
4688b36cdf
Merge pull request #12396 from HJianBo/import_user_2
...
feat(import_users): support user's password in plain text
2024-01-30 09:00:00 +08:00
5007650bd2
perf(emqx_broker): pick broker pool worker by topic/shard pair to distribute the load more evenly.
...
Fixes: EMQX-11812
2024-01-29 20:57:10 +02:00
cf0465dad4
Merge pull request #12411 from thalesmg/fix-cassandra-null-r55-20240126
...
fix(cassandra_bridge): correctly insert null values into columns
2024-01-29 14:09:12 -03:00
b849b08dbc
fix: kinesis schema problems found by @HJianBo
...
This commit fixes problems in schema found by @HJianBo here:
https://github.com/emqx/emqx/pull/12376#pullrequestreview-1848306286
2024-01-29 18:08:53 +01:00
7ee0a6aaa4
Merge pull request #12414 from ieQu1/dev/ds-readme
...
doc(ds): Update README
2024-01-29 18:01:03 +01:00
82403167c2
chore: update BSL license change date
2024-01-29 16:47:31 +01:00
47f61ba68a
fix(cassandra_bridge): correctly insert null values into columns
...
Fixes https://emqx.atlassian.net/browse/EMQX-11822
2024-01-29 11:32:00 -03:00
cb90d3c90c
Merge pull request #12415 from ieQu1/dev/remove-ds-conf-module
...
fix(ds): Remove unused module
2024-01-29 12:56:04 +01:00
9f22c2c455
ci: add some sleep and retry to emqx_persistent_session_ds_SUITE
2024-01-29 18:18:18 +08:00
d12335c4c6
chore: add tests
2024-01-29 18:10:42 +08:00
d286b7e28c
Merge pull request #12404 from thalesmg/fix-action-metric-handler-r55-20240126
...
fix(resource_metrics): avoid detaching handler on crashes
2024-01-29 17:51:04 +08:00
12da3c0986
feat: configurable server side message_expiry_interval
2024-01-29 17:29:52 +08:00
fa6d65887d
test(retainer): fix test cases
2024-01-29 08:42:36 +01:00
e4b8d79444
fix(retainer): add default delivery rate in the schema
2024-01-29 08:04:58 +01:00
ab99a17c99
chore(utils_stream): simplfy the csv reader implementation
2024-01-29 11:10:50 +08:00
2e35024df1
test: update eunit tests
2024-01-29 10:49:07 +08:00
9915c85b0b
chore(authn_mnesia): use emqx_utils_stream module to imporve reusability
2024-01-29 10:49:07 +08:00
8fc8106819
test: cover password_type and new data format
2024-01-29 10:49:07 +08:00
829887630d
test: refine existed test cases
2024-01-29 10:49:07 +08:00
fed512689a
chore: make elvis checking happy
2024-01-29 10:49:07 +08:00
e65cfb836c
feat(import_users): support user's password in plain text
2024-01-29 10:49:07 +08:00
2479e1189a
fix(ds): Remove unused module
2024-01-29 00:36:13 +01:00
96c527541b
docs(ds): Update README
2024-01-28 20:27:20 +01:00
dbe2baa405
Merge pull request #12408 from zmstone/sync-release-55
...
Sync release 55
2024-01-27 19:20:58 +00:00
2c6a776318
fix(sessds): Stricter checks for PacketIds
2024-01-26 17:49:33 +01:00
eec56b0d6b
fix(sessds): Improve comments
2024-01-26 17:49:33 +01:00
3c451c6ae6
test(sessds): Fix unstable tests
2024-01-26 17:49:33 +01:00
2d23212792
refactor(sessds): #ifs -> #srs
2024-01-26 17:49:33 +01:00
974760d331
test(sessds): Don't trap exits in the test
2024-01-26 17:49:33 +01:00
ebe2339810
fix(sessds): Use mria:async_dirty instead of transaction
2024-01-26 17:49:33 +01:00
f5b9bd30aa
fix(sessds): Apply review remarks
2024-01-26 17:49:33 +01:00
39857626ce
test(sessds): Fix failing tests
2024-01-26 17:49:33 +01:00
e7b03cdc59
test(sessds): Create a property-based test for the session state
2024-01-26 17:49:33 +01:00
893656f092
fix(sessds): Fix session garbage collection after the refactoring
2024-01-26 17:49:33 +01:00
963df8f941
fix(sessds): Delete the routes when the session expires
2024-01-26 17:49:33 +01:00
cff6c15e13
fix(sessds): Store the QoS as the MSB of the packet ID
2024-01-26 17:49:33 +01:00
William Yang