56c81c2c25
feat(ft-api): bump app versions
2023-07-07 19:17:22 +03:00
e0353ab750
fix(ft-api): update FT API tests to use emqx_cth_suite
2023-07-07 19:15:42 +03:00
fde506838a
fix(ft-api): implement import config behaviour
2023-07-07 19:15:42 +03:00
fe691e8330
fix(ft-api): fix swagger schema dump for ft schema
2023-07-07 19:15:42 +03:00
a2eb658cd9
feat(ft-api): do cleanup certs explicitly
2023-07-07 19:15:42 +03:00
a2b03716be
feat(ft-api): provide configuration API
...
To configure `emqx_ft` during the runtime.
2023-07-07 19:15:42 +03:00
34793c5ed0
feat(ft-conf): preprocess TLS configuration on updates
...
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
2023-07-07 19:15:42 +03:00
27ab9c62d8
fix(ft): unload conf hooks in `prep_stop`
...
In order to avoid situations when the root supervisor is stopped
already.
2023-07-07 19:15:42 +03:00
aba52c6a25
chore(rebalance test): fix API tests
2023-07-07 18:04:32 +03:00
9bb159cf1e
fix(rebalance): fix changelog and version
2023-07-07 18:04:11 +03:00
5b1c265485
Merge pull request #11224 from zhongwencool/fix-psk-conf-load
...
fix: psk_authentication is updated failed
2023-07-07 21:27:49 +08:00
a6019a2846
Merge pull request #11198 from savonarola/0705-fix-rebalance-swagger
...
fix(rebalance): fix rebalance swagger examples
2023-07-07 15:55:16 +03:00
70bd7a59e4
Merge pull request #11225 from lafirest/fix/pgsql_function_clause
...
fix(bridge): ensure the username of pgsql must exists
2023-07-07 20:39:07 +08:00
8b80a2f754
Merge pull request #11192 from zhongwencool/bump-hocon-to-0.39.13
...
chore: update hocon to 0.39.13
2023-07-07 20:02:46 +08:00
0ef00d5919
fix: psk_authentication is updated failed
2023-07-07 20:02:06 +08:00
97b6c43062
fix(ft-test): use new cth tooling in `emqx_ft_storage_fs_SUITE`
...
In attempt to battle test flakiness.
2023-07-07 14:00:33 +02:00
7e76914599
feat(ds): hardwire emqx_durable_storage message persistence
...
Only message persistence is currently implemented, irrespectively of
whether there are persistent sessions around or not.
2023-07-07 13:46:12 +02:00
d01eee7fe4
chore: update changes && app version
2023-07-07 18:22:54 +08:00
8cee75139d
fix(bridge): ensure the username of pgsql must exists
2023-07-07 18:06:28 +08:00
055a1e8d75
Merge pull request #11210 from sstrigler/EMQX-9569-provide-the-ce-ee-version-info-and-the-current-version-number-in-the-info-field
...
feat(emqx_dashboard): include edition and version in swagger.json info
2023-07-07 11:55:44 +02:00
b5cc8fb3c3
fix: start_after_created's default value
2023-07-07 16:39:26 +08:00
352d818e3b
chore: update hocon to 0.39.13
2023-07-07 16:39:26 +08:00
88fd8d0582
Merge pull request #11219 from JimMoen/fix-hstreamdb-deps-app
...
chore: hstreamdb_erl as deps for hstreamdb bridge
2023-07-07 16:39:12 +08:00
2f33fd9bac
Merge pull request #11208 from HJianBo/update-lwm2m-stats
...
fix(lwm2m): update the stats correctly and timely
2023-07-07 15:32:50 +08:00
daa6cf14fe
Merge pull request #11206 from HJianBo/coap-bugs
...
Fix(coap): Make the username and password params to optional
2023-07-07 15:32:00 +08:00
ab07fd0547
Merge pull request #11214 from zhongwencool/fix-cluster-rpc-call-tnx-id
...
fix: bad tnx-id when rejoin cluster
2023-07-07 15:08:06 +08:00
63c6427037
chore: hstreamdb_erl as deps for hstreamdb bridge
2023-07-07 14:50:44 +08:00
30c931ae62
fix: pulsar flaky cluster tests
2023-07-07 12:25:37 +08:00
4ee44972b2
feat: refactor hstreamdb connector to to avoid resources leaking
2023-07-07 12:25:37 +08:00
b089fba100
refactor: rm ee_bridge and ee_connector application
2023-07-07 12:25:37 +08:00
583ccfaafd
test(hstreamdb): create stream and wirte data
2023-07-07 12:25:36 +08:00
1587f038a5
feat: hstreamdb bridge with batch query
2023-07-07 12:25:36 +08:00
b9bfdfd583
chore: compatibility with hstreamdb v0.15.0
2023-07-07 12:25:36 +08:00
d31a305bcb
refactor(hstreamdb): HStreamDB bridge to its own application
2023-07-07 12:25:36 +08:00
f7513b900a
fix: set load config done after update tnx_id
2023-07-07 12:25:34 +08:00
9f57ba510e
chore: add 11214 changelog
2023-07-07 12:25:34 +08:00
221f6eba06
fix: bad tnx-id when rejoin cluster
2023-07-07 12:25:34 +08:00
d3d2c8a532
chore(lwm2m): bump version
2023-07-07 12:24:31 +08:00
31a240ba63
fix(lwm2m): update the stats correctly and timely
2023-07-07 12:24:31 +08:00
791b8ef671
chore: bump versions
2023-07-07 12:23:57 +08:00
6d2222318d
chore(coap): update the subscriptions_cnt stats in time
2023-07-07 12:23:57 +08:00
800b154582
test(coap): cover the params optional logic to create connection
2023-07-07 12:23:57 +08:00
5c901a52bd
fix(coap): make username/password optinal in connection
2023-07-07 12:23:57 +08:00
ab809c8b85
Merge pull request #11207 from lafirest/feat/keep_obfuscated
...
feat: ensure data backends don't leak sensitive data
2023-07-07 12:22:11 +08:00
b11fa5f3bf
Merge pull request #11195 from HJianBo/stomp-topic-id-unique
...
Fix(stomp): avoid to create duplicated subscription by HTTP or Client
2023-07-07 11:18:30 +08:00
a1d2b2ca51
chore: bump app versions
2023-07-07 11:17:54 +08:00
18dec53d8b
feat: ensure data backends don't leak sensitive data
2023-07-07 11:04:13 +08:00
05048dbf93
Merge pull request #11204 from HJianBo/avoid-case-clause
...
Allow the `*/*` or multiple `Accept` values in header for the `/configs` API
2023-07-07 09:46:28 +08:00
8faaa86c07
chore: bump vsn for emqx and emqx_bridge
2023-07-06 14:26:11 +02:00
22b25fbbb3
feat(emqx_dashboard): include edition and version in swagger.json info
2023-07-06 14:14:21 +02:00
353cc065db
Merge pull request #11175 from thalesmg/fix-mysql-start-error
...
fix(mysql_bridge): make nxdomain a 400 API error
2023-07-06 08:26:12 -03:00
9bcabe0a9e
Merge pull request #11184 from sstrigler/EMQX-10270-e-5-1-0-missing-value-range-validation-for-the-max-packet-size-of-mqtt-settings
...
fix: limit mqtt max_packet_size to 256MB
2023-07-06 10:20:24 +02:00
71d1f80530
test(api): cover the accept logic for `/configs` API
2023-07-06 11:30:38 +08:00
966b2affc2
chore: allow handle */* or multiple values in Accept headers
2023-07-06 10:53:07 +08:00
1a79e34510
Merge pull request #11202 from SergeTupchiy/fix-rule-type-acl-conf
...
chore(emqx_authz): fix `rule()` type example
2023-07-05 21:02:36 +03:00
4dfd3421e7
chore(emqx_authz): fix `rule()` type example
...
Use `action()` type which is a correct and defined type.
2023-07-05 20:24:26 +03:00
9af0030538
chore: bump to v5.1.1
2023-07-05 18:43:45 +03:00
f174cb656c
fix(rebalance): fix swagger examples in api-doc
2023-07-05 15:38:41 +03:00
dca6fe6200
fix(rebalance): fix global status evaluation on replicant nodes
2023-07-05 12:48:44 +03:00
aaf015c06f
test(stomp): cover subscription in use
2023-07-05 15:22:44 +08:00
bf5167a8bf
Merge pull request #11172 from lafirest/fix/dup_payload
...
fix(ruleengine): fix duplicate `payload` problems
2023-07-05 14:20:26 +08:00
0483c6afef
fix(stomp): not allow to create same subscriptions
2023-07-05 14:13:25 +08:00
263890be47
fix: limit mqtt max_packet_size to 256MB
2023-07-04 16:03:11 +02:00
8cc7022760
Merge pull request #11120 from ft/EMQX-10329/tooling
...
test: add tooling to make testruns more isolated and predictable
2023-07-04 14:36:39 +02:00
0bd0a16693
Merge pull request #11039 from savonarola/0613-fix-redis-db-schema
...
fix(redis): validate database number
2023-07-04 15:32:02 +03:00
f2c5f4932c
Merge pull request #11148 from lafirest/fix/mcall_error_report
...
fix(emqx_conf): don't sync the MFA to a leaved node
2023-07-04 19:56:52 +08:00
9ed75085b7
chore: drop dead commented out code
2023-07-04 13:20:04 +02:00
c7ba300efc
fix(cthsuite): use correct type for "smth serializable as HOCON"
2023-07-04 13:20:04 +02:00
d568d43fe4
test(bridge): switch `emqx_bridge_api_SUITE` to use new tooling
2023-07-04 13:20:03 +02:00
fc709a14cc
test: switch `emqx_authz_file_SUITE` to use new tooling
2023-07-04 13:20:03 +02:00
2c50e61cee
fix(cth): reduce amount of logging + streamline logging
...
Also make sure that code running on `emqx_cth_cluster` nodes don't
crash when using `ct:pal` and friends.
2023-07-04 13:20:03 +02:00
9bda4192e5
fix(cthsuite): disable default `acl.conf` authz source
...
So that users of `emqx_cth_suite` won't have to deal with providing
correct filepath each time they start `emqx_conf` and/or `emqx_authz`
up.
2023-07-04 13:20:03 +02:00
24a16ea247
fix(cthsuite): bump to hocon 0.39.11 with atom prettyprint fix
2023-07-04 13:20:03 +02:00
6a7254f55e
fix(cthcluster): match correct driver in `stop_node/2`
...
Co-authored-by: Ilya Averyanov <av@rubybox.dev>
2023-07-04 13:20:02 +02:00
fe0b8cfbaf
test(conf): respect `init_config_load_done` in emqx_conf itself
...
This should make task of configuring various applications (emqx_conf
/ emqx) during testsuites setup much simpler.
2023-07-04 13:20:02 +02:00
f76f3b77d7
test: add tooling to make testruns more isolated and predictable
...
Also showcase their usage in some of the existing testsuites.
2023-07-04 12:53:39 +02:00
afabdc3440
fix(emqx_conf): call clean callback before ekka stop
2023-07-04 18:04:33 +08:00
d509c47344
feat: add conf load api
2023-07-04 14:17:11 +08:00
33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
...
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
9701c3f7a1
fix(redis): validate database number
2023-07-03 22:44:54 +03:00
a127fd6571
Merge pull request #11183 from keynslug/perf/EMQX-9591/routing
...
perf(broker): avoid usorting aggregated routes in a loop
2023-07-03 19:16:25 +02:00
2cc0ba1217
Merge pull request #11174 from thalesmg/make-rule-server-a-binary
...
fix(mqtt_bridge): ensure `server` key is a binary
2023-07-03 10:38:39 -03:00
59d71a1147
Merge pull request #11173 from thalesmg/fix-flaky-gcp-consu-test
...
test(gcp_consumer): attempts to reduce flakiness
2023-07-03 10:04:31 -03:00
0f25df3aa8
perf(router): employ `maps:foreach/2` instead
2023-07-03 15:01:47 +02:00
57da71eb1d
perf(broker): avoid usorting aggregated routes in a loop
2023-07-03 15:01:33 +02:00
902603780b
test(ruleengine): add test for the fix of duplicate payload
2023-07-03 18:22:35 +08:00
bd4f207365
fix(ruleengine): fix duplicate `payload` problems
2023-07-03 14:21:56 +08:00
c216dfd96b
fix(mysql_bridge): make nxdomain a 400 API error
...
Fixes https://emqx.atlassian.net/browse/EMQX-10460
2023-06-30 16:10:09 -03:00
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
...
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
425eba8b13
fix(mqtt_bridge): ensure `server` key is a binary
...
Fixes https://emqx.atlassian.net/browse/EMQX-10461
So that it can be JSON encoded correctly.
```
2023-06-30T02:00:41.160110+00:00 [debug] msg: publish_to, mfa: emqx_trace:publish/1, line: 73, topic: b/$, payload: {"topic":"t/1","server":[49,48,46,52,50,46,51,46,49,56,48,58,49,56,56,51],"retain":false,"qos":1,"pub_props":{},"payload":"{\"msg\": \"hello\"}","message_received_at":1688090441159,"id":"0005FF4F2F181488103417000C2E0000","dup":false}, tag: PUBLISH
```
2023-06-30 13:46:24 -03:00
6fe6aa7997
Merge pull request #11162 from thalesmg/treat-404-as-failure-master
...
fix(webhook): treat 404 and other error replies as errors in async requests
2023-06-30 13:13:18 -03:00
7cf1dd7e86
Merge pull request #11163 from thalesmg/fix-rm-mongo-topology-pool-size
...
fix(mongodb): hide `topology.pool_size` and fix it at 1
2023-06-30 13:13:02 -03:00
0201481c70
Merge pull request #11164 from thalesmg/fix-placeholder-sql-nested
...
fix(placeholder): add back support for nested keys inside binary encoded json
2023-06-30 13:12:51 -03:00
1a058f6890
test(gcp_consumer): attempts to reduce flakiness
2023-06-30 12:49:23 -03:00
714363bd01
fix: decouple emqx_rule_engine application from emqx_ee_schema_registry
...
This commit decouples the emqx_rule_engine application from the
emqx_ee_schema_registry application by making it possible to register a
callback module that defines extra rule engine SQL functions instead of
calling a module in emqx_ee_schema_registry directly from the
emqx_rule_engine application.
2023-06-30 16:56:16 +02:00
724efcb557
Merge pull request #11168 from savonarola/0630-fix-rebalance-schema
...
Specify correct response schema for global rebalance status
2023-06-30 17:00:35 +03:00
3c8f591cc4
fix(mongodb): hide `topology.pool_size` and fix it at 1
...
Fixes https://emqx.atlassian.net/browse/EMQX-10408
From an old conversation with @kjellwinblad:
> There are 3 pool_sizes
> - The buffer workers pool size, just exposed here: https://github.com/emqx/emqx/pull/9742
> - The topology.pool_size, which controls the pool size for the poolboy_pool in Kjell's
diagram (on mongodb's side).
> - The pool_size from emqx_connector_mongo:mongo_fields that controls the ecpool pool
size (on EMQX's side).
> So we actually want to set topology.pool_size = 1 and hide it from users.
2023-06-30 10:17:33 -03:00
631f4ceac9
fix(placeholder): add back support for nested keys inside binary encoded json
...
Fixes https://emqx.atlassian.net/browse/EMQX-10459
2023-06-30 10:05:49 -03:00
ca435975de
fix(webhook): treat http status code 429 as recoverable
2023-06-30 09:46:03 -03:00
1ea92d4009
Merge pull request #11166 from lafirest/feat/porti_uuid_el_v4
...
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 18:36:45 +08:00
c8d1bd3bd5
Merge pull request #11138 from zhongwencool/cluster-conf-k8s-default
...
fix: update cluster.conf's k8s default api_server value
2023-06-30 17:01:52 +08:00
039e27a153
feat: add Sparkplug encode and decode functions to the rule engine
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
b5c0951584
fix(rebalance): specify correct response schema for global rebalance status
2023-06-30 10:53:08 +03:00
872698d19e
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 14:38:27 +08:00
bb1ae808b4
Merge pull request #11165 from zhongwencool/hide-global-limiter-from-swagger-api
...
chore: hide /configs/limiter from swagger api doc
2023-06-30 11:54:01 +08:00
3eefa1b2a3
chore: hide /configs/limiter from swagger api doc
2023-06-30 10:59:18 +08:00
8b679cf358
Merge pull request #11137 from zhongwencool/dashboard-https-ssl-options
...
feat: refactor dashboard https ssl_options
2023-06-30 09:57:07 +08:00
bd83365d45
fix: update cluster.conf's k8s default api_server value
2023-06-30 09:55:36 +08:00
59b109eb5c
fix(webhook): treat 404 and other error replies as errors in async requests
...
Fixes https://emqx.atlassian.net/browse/EMQX-10405
The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
4506eff3ff
Merge pull request #11140 from thalesmg/gcp-consumer-tests
...
test(gcp_pubsub_consumer): add more tests and improve bridge
2023-06-29 13:02:47 -03:00
14da61194e
fix: handle resource errors
2023-06-29 16:09:45 +02:00
321fd53132
fix: use ReplyTo in QUERY for async
2023-06-29 16:09:45 +02:00
d71d70c889
fix: allow to pass in Opts rather than just a ReplyTo
2023-06-29 16:09:45 +02:00
40dd34a704
fix: use reply_to instead of async_reply_fun
2023-06-29 16:09:45 +02:00
1363108678
fix: fix simple_sync_query
2023-06-29 16:09:45 +02:00
422c79583a
test(emqx_rule_engine): fix simple sync
2023-06-29 16:09:45 +02:00
837acd4234
test(emqx_rule_engine): test rule metrics
2023-06-29 16:09:45 +02:00
2274a192cc
fix(emqx_resource): call async reply fun in simple_aysnc_query
2023-06-29 16:09:45 +02:00
ae636a52d7
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-29 16:09:45 +02:00
9708a02680
Merge pull request #11134 from lafirest/fix/auth_header_sensitive
...
fix: mask the value of the authorization header whatever its case
2023-06-29 21:43:16 +08:00
f2074e9c9b
Merge pull request #11143 from ieQu1/master
...
docs(ds): Update IMPLEMENTATION.md
2023-06-29 11:14:55 +02:00
be1a4c89a5
Merge pull request #11158 from lafirest/fix/wait_table_in_retainer
...
fix(emqx_retainer): wait for Mria tables
2023-06-29 16:01:54 +08:00
31e914f6a2
fix(emqx_utils): improve the checking for the `authorization` key
2023-06-29 15:57:48 +08:00
6fedd0328f
fix(retainer): Fix missing changes in configuration
2023-06-29 14:06:17 +08:00
1202bef6db
fix(emqx_retainer): wait for Mria tables
2023-06-29 03:16:05 +00:00
9dae408b5a
Merge pull request #11142 from zhongwencool/list-listener-group-with-wrong-bind
...
List listener group with wrong bind
2023-06-29 11:04:35 +08:00
73487ecdd5
chore: add desc for ssl_options
2023-06-29 10:49:39 +08:00
82ea9e5a04
test: dashbaord api ct failed
2023-06-29 10:49:39 +08:00
e26c30ff0c
feat: refactor dashboard https ssl_options
2023-06-29 10:49:39 +08:00
71d34504bf
fix(cluster_rpc): clean self-data by mria stop callback
2023-06-29 01:45:47 +00:00
8e68af14ef
chore: bump ekka version
2023-06-29 01:45:47 +00:00
51ec8cb8fa
fix(emqx_conf): don't sync the MFA to a leaved node
2023-06-29 01:45:47 +00:00
30e0b4be54
test(gcp_pubsub_consumer): add more tests and improve bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
f9ea924cd0
Merge pull request #11151 from kjellwinblad/kjell/refactor/mysql_bridge/EMQX-9533
...
refactor: MySQL bridge and connector to separate applications
2023-06-28 15:15:55 +02:00
940ad12108
docs(ds): Apply suggestions from code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-28 10:04:07 +02:00
6d6287ef26
test: start emqx_mongodb when running the MongoDB tests
2023-06-28 08:03:41 +02:00
9cf874e37e
Merge pull request #11150 from SergeTupchiy/wait-for-emqx_psk_tab
...
fix(emqx_psk): wait for Mria table on app start
2023-06-28 08:57:32 +03:00
30b67add05
chore: add more example for clients api
2023-06-28 10:55:55 +08:00
93cbb5ebd8
fix: show the local listener's settings first
2023-06-28 10:55:55 +08:00
d556adf86c
Merge pull request #11146 from zhongwencool/hide-node-process-limit
...
feat: hide node.process_limit
2023-06-28 10:45:37 +08:00
c4fc0e767e
feat: allow specifying more helpful messages for unhealthy targets
2023-06-27 17:13:43 -03:00
8e877caf47
fix(emqx_psk): wait for Mria table on app start
...
This ensures emqx_psk table is synced to a replicant even if it doesn't have init psk file.
2023-06-27 20:22:40 +03:00
30d78aaac2
Merge pull request #11152 from keynslug/test/ft-prop-aggressive
...
test(ft): make proptests even less aggressive
2023-06-27 17:44:54 +02:00
7d171de02b
test(ft): make proptests even less aggressive
2023-06-27 17:17:56 +02:00
3d1712c00c
refactor: MySQL bridge to its own application
...
This commit refactor the MySQL bridge to its own application.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9533
2023-06-27 17:13:26 +02:00
ff327609db
test(ft): generate segments manually in proptests
...
In order to avoid spamming proptest logs with HUGE type instances,
and make proptests themselves simpler. Proper now should refuse
shrinking altogether, instances that cause prop function to timeout
do not shrink well for some reason, making proptests stuck.
2023-06-27 17:13:23 +02:00
85a130f68b
refactor: move MySQL open source connector to its own app
2023-06-27 16:15:18 +02:00
4ed422cb9a
Merge pull request #11127 from zmstone/0622-refine-examples
...
0622 refine examples
2023-06-27 15:30:30 +02:00
0e55aeada5
Merge pull request #11147 from keynslug/test/ft-timetrap
...
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 15:11:14 +02:00
d3a0b3e183
Merge pull request #11139 from kjellwinblad/kjell/refactor/redis/EMQX-9534
...
refactor: redis bridge to its own application
2023-06-27 14:55:55 +02:00
90c35c3ff8
chore(prometheus): change default vaue enum the first symbol
2023-06-27 14:23:18 +02:00
9b6ed09513
refactor: call hocon_tconf:check_plain/4 directly for example conf check
...
the attemp to re-use emqx_hocon:check failed because
hocon_tconf:check_plain/3 and check_plain/4 behaves slightly different
for unknown root fields.
To keep the old logic unchanged, for example config checks, we call
check_plain/4 directly
2023-06-27 14:23:18 +02:00
92ed7d7639
docs: move some prometheus docs from schema desc to code comments
2023-06-27 14:23:18 +02:00
41de679557
ci: run example config check in ci
2023-06-27 14:23:18 +02:00
a069b351fd
test: add test script to verify config example files
2023-06-27 14:23:18 +02:00
03d05825db
docs: hide core_nodes config doc
...
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
e0d05a02ff
refactor: refine examples
...
Also turns off vm_dist_collector metrics collector by default
2023-06-27 14:23:18 +02:00
31e08a6046
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 13:03:19 +02:00
116675e86f
chore: bump ekka to 0.15.4
2023-06-27 13:24:49 +03:00
b4ae4260c0
feat: hide node.process_limit
2023-06-27 18:09:26 +08:00
e0c11f54aa
docs(ds): Update IMPLEMENTATION.md
2023-06-27 10:53:38 +02:00
a7db85c1e6
style: fix code style issue
2023-06-27 10:52:24 +02:00
f99ccd4d76
fix: fix issues found in review by @paulozulato and @thalesmg
2023-06-27 10:49:19 +02:00
e76daa0f3b
refactor: redis bridge to its own application
...
This commit refactor the redis bridge to its own application.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9534
2023-06-27 10:24:44 +02:00
dd6aa96c99
refactor: move Redis open source connector to its own app
2023-06-27 10:24:40 +02:00
36b7aef04c
chore: bump versions
2023-06-26 14:35:29 +02:00
a91a478d42
build: fix order in mix.exs list due to CI script
2023-06-26 14:32:59 +02:00
a65d768873
style: fix format issues
2023-06-26 14:32:59 +02:00
fa2c2a300a
refactor: rename emqx_connector_mongo to emqx_mongodb
...
This renaming is consistent with the naming in the only other connector
only application emqx_oracle.
2023-06-26 14:32:59 +02:00
b78a75bb5c
refactor: the MongoDB connector into its own application
2023-06-26 14:32:59 +02:00
e3199ef813
refactor: MongoDB bridge to use the new app structure
2023-06-26 14:32:53 +02:00
f2e3446e29
Merge pull request #11135 from lafirest/pref/calendar_timeoffset
...
perf(rule_engine): improve the time offset parser
2023-06-26 18:54:00 +08:00
850d642ec2
Merge pull request #11108 from ieQu1/emqx-ds
...
Durable storage
2023-06-25 11:36:34 +08:00
c8c2793944
perf(rule_engine): improve the time offset parser
2023-06-24 13:05:27 +00:00
d485f8fe96
fix: mask the value of the authorization header whatever its case
2023-06-24 09:20:26 +00:00
60a1f5a38d
chore: update app version && changes
2023-06-23 17:01:22 +00:00
6bb8bf58ea
fix(retainer): rename deliver_rate to delivery_rate
2023-06-23 16:51:19 +00:00
639fe276e1
chore(emqx_authz): fix `ipaddrs` type and style in etc/acl.conf
2023-06-23 14:53:26 +03:00
9a81f22105
fix(ds): Apply review remarks
2023-06-22 17:07:30 +02:00
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
...
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
22356b7c25
chore: hide gcp pubsub consumer until e5.2.0
2023-06-22 10:05:52 -03:00
f01de238b2
chore: bump app vsns
2023-06-22 10:01:12 -03:00
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
...
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
75eb18bc81
chore: bump app vsns
2023-06-22 11:09:09 +02:00
cfcd81e17b
fix(ds): Fix build
2023-06-22 11:07:52 +02:00
c58a98954b
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-22 11:05:51 +02:00
b29c5ad23c
feat(emqx_ds): Add API draft for logic layer
2023-06-22 10:20:18 +02:00
a343cdb1d5
refactor(ds): message_storage -> message_storage_bitmask
2023-06-22 10:20:18 +02:00
a4219db163
refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer
2023-06-22 10:20:18 +02:00
1159f99432
refactor(ds): emqx_replay -> emqx_ds
2023-06-22 10:20:18 +02:00
8d6bcc1414
refactor(ds): emqx_replay -> emqx_durable_storage
2023-06-22 10:20:18 +02:00
c4f721f214
chore(ds): Add callback definitions to local_store
2023-06-22 10:20:18 +02:00
efed7cdc2f
chore(ds): behavior -> behaviour
2023-06-22 10:20:18 +02:00
0f6c81e42d
chore(ds): Change license to BSL
2023-06-22 10:20:18 +02:00
04adb65c09
refactor(ds): zone -> shard
...
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
8ac0bba958
test(ds): verify preserve / restore works with stored iterators
2023-06-22 10:20:18 +02:00
d2065e0c1b
feat(ds): allow to preserve / restore iterators in the db
...
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
cbded9af39
feat(ds): Make `create_generation` safer against bad input
2023-06-22 10:20:18 +02:00
83d1606d00
feat(ds): Make iteration fully generation-aware
2023-06-22 10:20:18 +02:00
bf9d57f6a9
chore(ds): Simplify iteration-related typespecs
2023-06-22 10:20:18 +02:00
cf6a5e1643
feat(ds): Allow to create new storage generations
2023-06-22 10:20:18 +02:00
f80e906553
fix(ds): Correct typespec
2023-06-22 10:20:18 +02:00
f5a7b49f57
feat(ds): Wire iteration options up to the app config
2023-06-22 10:20:18 +02:00
2bf8a07b05
test(ds): Split unit tests off into a full-fledged suite
2023-06-22 10:20:18 +02:00
cf70184ebd
test(ds): Provide more general `keymapper_info/1`
2023-06-22 10:20:18 +02:00
2f385022cd
chore(ds): Rename testsuite to reflect test subject better
2023-06-22 10:20:18 +02:00
d875fa49d3
feat(ds): Enable periodic iterator refresh
...
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
7f408da251
refactor(ds): Use inline functions instead of macros
2023-06-22 10:20:07 +02:00
a11e75d189
fix(ds): clear bitmask of topic filter tail containing wildcards
2023-06-22 10:16:55 +02:00
ac0935ef91
test(ds): Proptest that iteration is exhaustive
...
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
60e3070328
test(ds): Scale up number of messages per topic in proptests
2023-06-22 10:16:41 +02:00
82a7292851
test(ds): Use `_build/test/proper` as a scratch dir for testruns
2023-06-22 10:16:32 +02:00
440fecd268
test(ds): Add a proptest on iterator preserve / restore
...
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
1f5425975f
feat(ds): add an ability to preserve and restore iterators
...
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
94e8dbfe7f
test(ds): Move proptests into a separate module
...
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
4c643aa7d5
refactor(ds): Introduce keyspace filter concept
...
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
c5826cee56
refactor(ds): rename `compute_hash_*` → `compute_topic_*`
2023-06-22 10:16:05 +02:00
85d3783ff1
chore(ds): Drop few unused macrodefinitions
2023-06-22 10:15:56 +02:00
a0bcdb5104
chore(ds): Attempt to make `compute_next_seek`'s logic clearer
2023-06-22 10:15:49 +02:00
f1e39d977a
fix(ds): Fix dialyzer warnings
2023-06-22 10:15:47 +02:00
adcbf40d27
refactor(ds): tau -> epoch
2023-06-22 10:15:44 +02:00
120d4e66ae
refactor(ds): Factor out configuration to a separate module
2023-06-22 10:15:42 +02:00
0bbc5ecb32
refactor(ds): Introduce bitwise_concat function
2023-06-22 10:15:40 +02:00
b5bb77dd58
feat(ds): Add metadata storage and supervision tree
2023-06-22 10:15:37 +02:00
5e30a5d3dd
feat(ds): Implement keyspace partitioning across time
2023-06-22 10:15:32 +02:00
8c9c9cc669
feat(ds): Allow to specify message store options
...
* Keymapper
* Column family name + DB options
* DB write / read options
2023-06-22 10:15:25 +02:00
5b6c4d766b
test(ds): Add basic property tests
2023-06-22 10:15:20 +02:00
a1e29e4e24
refactor(ds): Improve function naming
2023-06-22 10:15:03 +02:00
cd12338c3f
feat(ds): Smoke tests for wildcard iterator scans
2023-06-22 10:14:46 +02:00
9c1cd4911d
feat(ds): Implement iterator next
2023-06-22 10:14:43 +02:00
27b925405b
test(ds): Add testcase
2023-06-22 10:14:41 +02:00
691a27883d
feat(ds): Initial commit
2023-06-22 10:14:37 +02:00
f47cc2a458
feat(kafka): add support for kafka headers and ext headers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
...
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
...
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00
86d787eced
chore: bump hocon to 0.39.10
...
Which comes with a fix for slightly more user-friendly validation error
messages.
2023-06-21 21:25:43 +02:00
18f0510353
fix(mongo): return health check failure reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
8430ec673c
feat(kafka): check whether target topic exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
decfd6df2b
feat(buffer_worker): log expired message count
...
Fixes https://emqx.atlassian.net/browse/EMQX-10165
```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
2023-06-21 14:38:51 -03:00
7cf8a6c892
chore: bump app vsns
2023-06-21 16:36:51 +02:00
5fa87091c4
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-21 16:08:41 +02:00
7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud
...
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-21 15:07:45 +03:00
f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type
2023-06-21 17:36:59 +08:00
5f47649d4d
chore: downgrade log severity level of 'not_ready' nodes to info
2023-06-21 09:45:45 +02:00
045ed340dd
fix(emqx_conf_app): fix release version detect during cluster conf sync
2023-06-21 09:28:46 +02:00
0463828e84
refactor(gcp_pubsub): transform connector into opaque client
2023-06-20 15:27:42 -03:00
1d791d7a8c
fix(resource): validate maximum worker pool size
...
Fixes https://emqx.atlassian.net/browse/EMQX-10297
2023-06-20 14:26:42 -03:00
ffce6fefa8
docs: note that connector state must be immutable to avoid errors
2023-06-20 13:50:53 -03:00
dddccfdc5c
chore: bump to v5.1.0 official cut
2023-06-20 17:09:17 +02:00
728656d4f6
chore: bump to e5.1.0 official cut
2023-06-20 17:08:18 +02:00
ba96c725e5
feat: add time-based pull for safety
2023-06-20 11:36:35 -03:00
57460e83b9
test(fix): rm unused var warnings
2023-06-20 11:15:37 -03:00
2ac2d4c037
refactor: addressing review comments
2023-06-20 11:15:13 -03:00
54d96488a0
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-20 16:46:23 +03:00
b588b88a60
chore: change autoclean interval default value to 24h
...
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).
To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.
This PR also has the autoclean config removed from examples.
2023-06-20 15:43:37 +02:00
8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example
...
chore: add examples of how to configure
2023-06-20 20:09:44 +08:00
f95787d229
test: make possible to test core/replica locally
2023-06-20 12:42:00 +02:00
e5803a61f1
fix(management): Don't wait for an empty shard
2023-06-20 10:08:41 +02:00
bb2bf059ac
chore: modify the build to copy examples into the release
2023-06-20 14:53:14 +08:00
36ee872361
chore: hide plugins from conf load cli
2023-06-20 10:34:09 +08:00
856de78698
chore: remove the is_list guard
2023-06-20 08:39:23 +08:00
093cdab838
chore: to_integer to make sure integer is converted
2023-06-20 08:39:23 +08:00
07172e42f0
test: integer CI check failed
2023-06-20 08:39:23 +08:00
a4be9b8281
chore: bump hocon to 0.39.9
2023-06-20 08:39:23 +08:00
45963b6a95
fix: ip_port schema type crash
2023-06-20 08:39:23 +08:00
ead511a128
chore: bump to 5.1.0-rc.1
2023-06-19 23:19:27 +02:00
86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5
...
chore(mria): Bump version to 0.5.5
2023-06-19 23:13:36 +02:00
13746c2cdf
fix(resource): check status when (re)starting a resource
...
Fixes https://emqx.atlassian.net/browse/EMQX-10290
2023-06-19 18:01:02 -03:00
01d758e4c0
fix(bpapi): fix `query_mode` type(spec)
...
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
2023-06-19 16:04:12 -03:00
b442910ff1
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
bb49482529
refactor(gcp_pubsub): move logging from connector to bridge
2023-06-19 15:59:00 -03:00
dabefdb01b
fix(gcp_pubsub): redact request when errors occur
2023-06-19 15:59:00 -03:00
7f850f7499
fix(resource): fix `query_mode/0` type and usage
2023-06-19 15:59:00 -03:00
a1690b3684
chore(gcp_pubsub): add stricter compile/xref flags
2023-06-19 15:59:00 -03:00
5375421954
refactor(gcp_pubsub): split connector into producer and reusable parts
2023-06-19 15:59:00 -03:00
5f10936091
feat(oracle): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
c9a2ddf98c
feat(mysql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
f82c56ae16
chore(mria): Bump version to 0.5.5
2023-06-19 19:28:17 +02:00
cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8
...
chore: bump to v5.1.0.alpha.8
2023-06-19 23:00:53 +08:00
b0a6c18917
chore: bump vsn to 5.1.0-alpha.8
2023-06-19 22:59:05 +08:00
9454af9a8b
feat(postgresql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags
...
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 16:00:04 +02:00
7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master
...
fix(nolink_apply): avoid sending late replies to caller
2023-06-19 10:32:43 -03:00
ae5846be29
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 13:35:21 +02:00
1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli
...
chore: hide plugins from conf load cli
2023-06-19 19:11:52 +08:00
64612cab14
chore: bump to v5.1.0.alpha.8
2023-06-19 18:22:51 +08:00
6efb07f6eb
fix: crash on emqx_ctl listeners
2023-06-19 18:12:33 +08:00
a578d6fa1d
chore: release e5.1.0-alpha.8
2023-06-19 17:19:03 +08:00
8b20a70344
chore: hide plugins from conf load cli
2023-06-19 15:57:40 +08:00
803cd6c812
Merge pull request #11082 from emqx/release-51
...
sync release-51 to master
2023-06-19 15:12:37 +08:00
30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error
...
fix: shared-sub with nl sub-option should cause protocol error
2023-06-19 10:39:08 +08:00
59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216
...
fix: rule engine different behavior for div and mod
2023-06-17 07:20:37 +02:00
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view
...
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
2023-06-17 00:54:24 +02:00
50e7d5d2ec
fix(nolink_apply): avoid sending late replies to caller
...
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.
e.g. when a timeout occurred while resource manager was checking a resource's health:
```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```
Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277)
2023-06-16 17:03:39 -03:00
8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard
...
fix(rewrite): avoid wildcards on destination topic on publish
2023-06-16 09:57:46 -03:00
178711f742
fix: shared-sub with nl sub-option should cause protocol error
2023-06-16 18:19:23 +08:00
5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean
...
fix: use mria.cluster_autoclean
2023-06-16 14:56:52 +08:00
c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal
2023-06-16 14:04:24 +08:00
b49b80d36c
fix(emqx_schema): use non negative integer type for 'depth' SSL option
...
Closes: EMQX-10276
2023-06-16 10:59:39 +08:00
Ilya Averyanov