24a16ea247
fix(cthsuite): bump to hocon 0.39.11 with atom prettyprint fix
2023-07-04 13:20:03 +02:00
6a7254f55e
fix(cthcluster): match correct driver in `stop_node/2`
...
Co-authored-by: Ilya Averyanov <av@rubybox.dev>
2023-07-04 13:20:02 +02:00
fe0b8cfbaf
test(conf): respect `init_config_load_done` in emqx_conf itself
...
This should make task of configuring various applications (emqx_conf
/ emqx) during testsuites setup much simpler.
2023-07-04 13:20:02 +02:00
f76f3b77d7
test: add tooling to make testruns more isolated and predictable
...
Also showcase their usage in some of the existing testsuites.
2023-07-04 12:53:39 +02:00
afabdc3440
fix(emqx_conf): call clean callback before ekka stop
2023-07-04 18:04:33 +08:00
d509c47344
feat: add conf load api
2023-07-04 14:17:11 +08:00
33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
...
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
9701c3f7a1
fix(redis): validate database number
2023-07-03 22:44:54 +03:00
a127fd6571
Merge pull request #11183 from keynslug/perf/EMQX-9591/routing
...
perf(broker): avoid usorting aggregated routes in a loop
2023-07-03 19:16:25 +02:00
2cc0ba1217
Merge pull request #11174 from thalesmg/make-rule-server-a-binary
...
fix(mqtt_bridge): ensure `server` key is a binary
2023-07-03 10:38:39 -03:00
59d71a1147
Merge pull request #11173 from thalesmg/fix-flaky-gcp-consu-test
...
test(gcp_consumer): attempts to reduce flakiness
2023-07-03 10:04:31 -03:00
0f25df3aa8
perf(router): employ `maps:foreach/2` instead
2023-07-03 15:01:47 +02:00
57da71eb1d
perf(broker): avoid usorting aggregated routes in a loop
2023-07-03 15:01:33 +02:00
902603780b
test(ruleengine): add test for the fix of duplicate payload
2023-07-03 18:22:35 +08:00
bd4f207365
fix(ruleengine): fix duplicate `payload` problems
2023-07-03 14:21:56 +08:00
c216dfd96b
fix(mysql_bridge): make nxdomain a 400 API error
...
Fixes https://emqx.atlassian.net/browse/EMQX-10460
2023-06-30 16:10:09 -03:00
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
...
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
425eba8b13
fix(mqtt_bridge): ensure `server` key is a binary
...
Fixes https://emqx.atlassian.net/browse/EMQX-10461
So that it can be JSON encoded correctly.
```
2023-06-30T02:00:41.160110+00:00 [debug] msg: publish_to, mfa: emqx_trace:publish/1, line: 73, topic: b/$, payload: {"topic":"t/1","server":[49,48,46,52,50,46,51,46,49,56,48,58,49,56,56,51],"retain":false,"qos":1,"pub_props":{},"payload":"{\"msg\": \"hello\"}","message_received_at":1688090441159,"id":"0005FF4F2F181488103417000C2E0000","dup":false}, tag: PUBLISH
```
2023-06-30 13:46:24 -03:00
6fe6aa7997
Merge pull request #11162 from thalesmg/treat-404-as-failure-master
...
fix(webhook): treat 404 and other error replies as errors in async requests
2023-06-30 13:13:18 -03:00
7cf1dd7e86
Merge pull request #11163 from thalesmg/fix-rm-mongo-topology-pool-size
...
fix(mongodb): hide `topology.pool_size` and fix it at 1
2023-06-30 13:13:02 -03:00
0201481c70
Merge pull request #11164 from thalesmg/fix-placeholder-sql-nested
...
fix(placeholder): add back support for nested keys inside binary encoded json
2023-06-30 13:12:51 -03:00
1a058f6890
test(gcp_consumer): attempts to reduce flakiness
2023-06-30 12:49:23 -03:00
714363bd01
fix: decouple emqx_rule_engine application from emqx_ee_schema_registry
...
This commit decouples the emqx_rule_engine application from the
emqx_ee_schema_registry application by making it possible to register a
callback module that defines extra rule engine SQL functions instead of
calling a module in emqx_ee_schema_registry directly from the
emqx_rule_engine application.
2023-06-30 16:56:16 +02:00
724efcb557
Merge pull request #11168 from savonarola/0630-fix-rebalance-schema
...
Specify correct response schema for global rebalance status
2023-06-30 17:00:35 +03:00
3c8f591cc4
fix(mongodb): hide `topology.pool_size` and fix it at 1
...
Fixes https://emqx.atlassian.net/browse/EMQX-10408
From an old conversation with @kjellwinblad:
> There are 3 pool_sizes
> - The buffer workers pool size, just exposed here: https://github.com/emqx/emqx/pull/9742
> - The topology.pool_size, which controls the pool size for the poolboy_pool in Kjell's
diagram (on mongodb's side).
> - The pool_size from emqx_connector_mongo:mongo_fields that controls the ecpool pool
size (on EMQX's side).
> So we actually want to set topology.pool_size = 1 and hide it from users.
2023-06-30 10:17:33 -03:00
631f4ceac9
fix(placeholder): add back support for nested keys inside binary encoded json
...
Fixes https://emqx.atlassian.net/browse/EMQX-10459
2023-06-30 10:05:49 -03:00
ca435975de
fix(webhook): treat http status code 429 as recoverable
2023-06-30 09:46:03 -03:00
1ea92d4009
Merge pull request #11166 from lafirest/feat/porti_uuid_el_v4
...
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 18:36:45 +08:00
c8d1bd3bd5
Merge pull request #11138 from zhongwencool/cluster-conf-k8s-default
...
fix: update cluster.conf's k8s default api_server value
2023-06-30 17:01:52 +08:00
039e27a153
feat: add Sparkplug encode and decode functions to the rule engine
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
b5c0951584
fix(rebalance): specify correct response schema for global rebalance status
2023-06-30 10:53:08 +03:00
872698d19e
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 14:38:27 +08:00
bb1ae808b4
Merge pull request #11165 from zhongwencool/hide-global-limiter-from-swagger-api
...
chore: hide /configs/limiter from swagger api doc
2023-06-30 11:54:01 +08:00
3eefa1b2a3
chore: hide /configs/limiter from swagger api doc
2023-06-30 10:59:18 +08:00
8b679cf358
Merge pull request #11137 from zhongwencool/dashboard-https-ssl-options
...
feat: refactor dashboard https ssl_options
2023-06-30 09:57:07 +08:00
bd83365d45
fix: update cluster.conf's k8s default api_server value
2023-06-30 09:55:36 +08:00
59b109eb5c
fix(webhook): treat 404 and other error replies as errors in async requests
...
Fixes https://emqx.atlassian.net/browse/EMQX-10405
The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
4506eff3ff
Merge pull request #11140 from thalesmg/gcp-consumer-tests
...
test(gcp_pubsub_consumer): add more tests and improve bridge
2023-06-29 13:02:47 -03:00
14da61194e
fix: handle resource errors
2023-06-29 16:09:45 +02:00
321fd53132
fix: use ReplyTo in QUERY for async
2023-06-29 16:09:45 +02:00
d71d70c889
fix: allow to pass in Opts rather than just a ReplyTo
2023-06-29 16:09:45 +02:00
40dd34a704
fix: use reply_to instead of async_reply_fun
2023-06-29 16:09:45 +02:00
1363108678
fix: fix simple_sync_query
2023-06-29 16:09:45 +02:00
422c79583a
test(emqx_rule_engine): fix simple sync
2023-06-29 16:09:45 +02:00
837acd4234
test(emqx_rule_engine): test rule metrics
2023-06-29 16:09:45 +02:00
2274a192cc
fix(emqx_resource): call async reply fun in simple_aysnc_query
2023-06-29 16:09:45 +02:00
ae636a52d7
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-29 16:09:45 +02:00
9708a02680
Merge pull request #11134 from lafirest/fix/auth_header_sensitive
...
fix: mask the value of the authorization header whatever its case
2023-06-29 21:43:16 +08:00
f2074e9c9b
Merge pull request #11143 from ieQu1/master
...
docs(ds): Update IMPLEMENTATION.md
2023-06-29 11:14:55 +02:00
be1a4c89a5
Merge pull request #11158 from lafirest/fix/wait_table_in_retainer
...
fix(emqx_retainer): wait for Mria tables
2023-06-29 16:01:54 +08:00
31e914f6a2
fix(emqx_utils): improve the checking for the `authorization` key
2023-06-29 15:57:48 +08:00
6fedd0328f
fix(retainer): Fix missing changes in configuration
2023-06-29 14:06:17 +08:00
1202bef6db
fix(emqx_retainer): wait for Mria tables
2023-06-29 03:16:05 +00:00
9dae408b5a
Merge pull request #11142 from zhongwencool/list-listener-group-with-wrong-bind
...
List listener group with wrong bind
2023-06-29 11:04:35 +08:00
73487ecdd5
chore: add desc for ssl_options
2023-06-29 10:49:39 +08:00
82ea9e5a04
test: dashbaord api ct failed
2023-06-29 10:49:39 +08:00
e26c30ff0c
feat: refactor dashboard https ssl_options
2023-06-29 10:49:39 +08:00
71d34504bf
fix(cluster_rpc): clean self-data by mria stop callback
2023-06-29 01:45:47 +00:00
8e68af14ef
chore: bump ekka version
2023-06-29 01:45:47 +00:00
51ec8cb8fa
fix(emqx_conf): don't sync the MFA to a leaved node
2023-06-29 01:45:47 +00:00
30e0b4be54
test(gcp_pubsub_consumer): add more tests and improve bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
f9ea924cd0
Merge pull request #11151 from kjellwinblad/kjell/refactor/mysql_bridge/EMQX-9533
...
refactor: MySQL bridge and connector to separate applications
2023-06-28 15:15:55 +02:00
940ad12108
docs(ds): Apply suggestions from code review
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-28 10:04:07 +02:00
6d6287ef26
test: start emqx_mongodb when running the MongoDB tests
2023-06-28 08:03:41 +02:00
9cf874e37e
Merge pull request #11150 from SergeTupchiy/wait-for-emqx_psk_tab
...
fix(emqx_psk): wait for Mria table on app start
2023-06-28 08:57:32 +03:00
30b67add05
chore: add more example for clients api
2023-06-28 10:55:55 +08:00
93cbb5ebd8
fix: show the local listener's settings first
2023-06-28 10:55:55 +08:00
d556adf86c
Merge pull request #11146 from zhongwencool/hide-node-process-limit
...
feat: hide node.process_limit
2023-06-28 10:45:37 +08:00
c4fc0e767e
feat: allow specifying more helpful messages for unhealthy targets
2023-06-27 17:13:43 -03:00
8e877caf47
fix(emqx_psk): wait for Mria table on app start
...
This ensures emqx_psk table is synced to a replicant even if it doesn't have init psk file.
2023-06-27 20:22:40 +03:00
30d78aaac2
Merge pull request #11152 from keynslug/test/ft-prop-aggressive
...
test(ft): make proptests even less aggressive
2023-06-27 17:44:54 +02:00
7d171de02b
test(ft): make proptests even less aggressive
2023-06-27 17:17:56 +02:00
3d1712c00c
refactor: MySQL bridge to its own application
...
This commit refactor the MySQL bridge to its own application.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9533
2023-06-27 17:13:26 +02:00
ff327609db
test(ft): generate segments manually in proptests
...
In order to avoid spamming proptest logs with HUGE type instances,
and make proptests themselves simpler. Proper now should refuse
shrinking altogether, instances that cause prop function to timeout
do not shrink well for some reason, making proptests stuck.
2023-06-27 17:13:23 +02:00
85a130f68b
refactor: move MySQL open source connector to its own app
2023-06-27 16:15:18 +02:00
4ed422cb9a
Merge pull request #11127 from zmstone/0622-refine-examples
...
0622 refine examples
2023-06-27 15:30:30 +02:00
0e55aeada5
Merge pull request #11147 from keynslug/test/ft-timetrap
...
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 15:11:14 +02:00
d3a0b3e183
Merge pull request #11139 from kjellwinblad/kjell/refactor/redis/EMQX-9534
...
refactor: redis bridge to its own application
2023-06-27 14:55:55 +02:00
90c35c3ff8
chore(prometheus): change default vaue enum the first symbol
2023-06-27 14:23:18 +02:00
9b6ed09513
refactor: call hocon_tconf:check_plain/4 directly for example conf check
...
the attemp to re-use emqx_hocon:check failed because
hocon_tconf:check_plain/3 and check_plain/4 behaves slightly different
for unknown root fields.
To keep the old logic unchanged, for example config checks, we call
check_plain/4 directly
2023-06-27 14:23:18 +02:00
92ed7d7639
docs: move some prometheus docs from schema desc to code comments
2023-06-27 14:23:18 +02:00
41de679557
ci: run example config check in ci
2023-06-27 14:23:18 +02:00
a069b351fd
test: add test script to verify config example files
2023-06-27 14:23:18 +02:00
03d05825db
docs: hide core_nodes config doc
...
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
e0d05a02ff
refactor: refine examples
...
Also turns off vm_dist_collector metrics collector by default
2023-06-27 14:23:18 +02:00
31e08a6046
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 13:03:19 +02:00
116675e86f
chore: bump ekka to 0.15.4
2023-06-27 13:24:49 +03:00
b4ae4260c0
feat: hide node.process_limit
2023-06-27 18:09:26 +08:00
e0c11f54aa
docs(ds): Update IMPLEMENTATION.md
2023-06-27 10:53:38 +02:00
a7db85c1e6
style: fix code style issue
2023-06-27 10:52:24 +02:00
f99ccd4d76
fix: fix issues found in review by @paulozulato and @thalesmg
2023-06-27 10:49:19 +02:00
e76daa0f3b
refactor: redis bridge to its own application
...
This commit refactor the redis bridge to its own application.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9534
2023-06-27 10:24:44 +02:00
dd6aa96c99
refactor: move Redis open source connector to its own app
2023-06-27 10:24:40 +02:00
36b7aef04c
chore: bump versions
2023-06-26 14:35:29 +02:00
a91a478d42
build: fix order in mix.exs list due to CI script
2023-06-26 14:32:59 +02:00
a65d768873
style: fix format issues
2023-06-26 14:32:59 +02:00
fa2c2a300a
refactor: rename emqx_connector_mongo to emqx_mongodb
...
This renaming is consistent with the naming in the only other connector
only application emqx_oracle.
2023-06-26 14:32:59 +02:00
b78a75bb5c
refactor: the MongoDB connector into its own application
2023-06-26 14:32:59 +02:00
e3199ef813
refactor: MongoDB bridge to use the new app structure
2023-06-26 14:32:53 +02:00
f2e3446e29
Merge pull request #11135 from lafirest/pref/calendar_timeoffset
...
perf(rule_engine): improve the time offset parser
2023-06-26 18:54:00 +08:00
850d642ec2
Merge pull request #11108 from ieQu1/emqx-ds
...
Durable storage
2023-06-25 11:36:34 +08:00
c8c2793944
perf(rule_engine): improve the time offset parser
2023-06-24 13:05:27 +00:00
d485f8fe96
fix: mask the value of the authorization header whatever its case
2023-06-24 09:20:26 +00:00
60a1f5a38d
chore: update app version && changes
2023-06-23 17:01:22 +00:00
6bb8bf58ea
fix(retainer): rename deliver_rate to delivery_rate
2023-06-23 16:51:19 +00:00
639fe276e1
chore(emqx_authz): fix `ipaddrs` type and style in etc/acl.conf
2023-06-23 14:53:26 +03:00
9a81f22105
fix(ds): Apply review remarks
2023-06-22 17:07:30 +02:00
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
...
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
22356b7c25
chore: hide gcp pubsub consumer until e5.2.0
2023-06-22 10:05:52 -03:00
f01de238b2
chore: bump app vsns
2023-06-22 10:01:12 -03:00
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
...
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
75eb18bc81
chore: bump app vsns
2023-06-22 11:09:09 +02:00
cfcd81e17b
fix(ds): Fix build
2023-06-22 11:07:52 +02:00
c58a98954b
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-22 11:05:51 +02:00
b29c5ad23c
feat(emqx_ds): Add API draft for logic layer
2023-06-22 10:20:18 +02:00
a343cdb1d5
refactor(ds): message_storage -> message_storage_bitmask
2023-06-22 10:20:18 +02:00
a4219db163
refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer
2023-06-22 10:20:18 +02:00
1159f99432
refactor(ds): emqx_replay -> emqx_ds
2023-06-22 10:20:18 +02:00
8d6bcc1414
refactor(ds): emqx_replay -> emqx_durable_storage
2023-06-22 10:20:18 +02:00
c4f721f214
chore(ds): Add callback definitions to local_store
2023-06-22 10:20:18 +02:00
efed7cdc2f
chore(ds): behavior -> behaviour
2023-06-22 10:20:18 +02:00
0f6c81e42d
chore(ds): Change license to BSL
2023-06-22 10:20:18 +02:00
04adb65c09
refactor(ds): zone -> shard
...
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
8ac0bba958
test(ds): verify preserve / restore works with stored iterators
2023-06-22 10:20:18 +02:00
d2065e0c1b
feat(ds): allow to preserve / restore iterators in the db
...
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
cbded9af39
feat(ds): Make `create_generation` safer against bad input
2023-06-22 10:20:18 +02:00
83d1606d00
feat(ds): Make iteration fully generation-aware
2023-06-22 10:20:18 +02:00
bf9d57f6a9
chore(ds): Simplify iteration-related typespecs
2023-06-22 10:20:18 +02:00
cf6a5e1643
feat(ds): Allow to create new storage generations
2023-06-22 10:20:18 +02:00
f80e906553
fix(ds): Correct typespec
2023-06-22 10:20:18 +02:00
f5a7b49f57
feat(ds): Wire iteration options up to the app config
2023-06-22 10:20:18 +02:00
2bf8a07b05
test(ds): Split unit tests off into a full-fledged suite
2023-06-22 10:20:18 +02:00
cf70184ebd
test(ds): Provide more general `keymapper_info/1`
2023-06-22 10:20:18 +02:00
2f385022cd
chore(ds): Rename testsuite to reflect test subject better
2023-06-22 10:20:18 +02:00
d875fa49d3
feat(ds): Enable periodic iterator refresh
...
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
7f408da251
refactor(ds): Use inline functions instead of macros
2023-06-22 10:20:07 +02:00
a11e75d189
fix(ds): clear bitmask of topic filter tail containing wildcards
2023-06-22 10:16:55 +02:00
ac0935ef91
test(ds): Proptest that iteration is exhaustive
...
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
60e3070328
test(ds): Scale up number of messages per topic in proptests
2023-06-22 10:16:41 +02:00
82a7292851
test(ds): Use `_build/test/proper` as a scratch dir for testruns
2023-06-22 10:16:32 +02:00
440fecd268
test(ds): Add a proptest on iterator preserve / restore
...
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
1f5425975f
feat(ds): add an ability to preserve and restore iterators
...
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
94e8dbfe7f
test(ds): Move proptests into a separate module
...
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
4c643aa7d5
refactor(ds): Introduce keyspace filter concept
...
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
c5826cee56
refactor(ds): rename `compute_hash_*` → `compute_topic_*`
2023-06-22 10:16:05 +02:00
85d3783ff1
chore(ds): Drop few unused macrodefinitions
2023-06-22 10:15:56 +02:00
a0bcdb5104
chore(ds): Attempt to make `compute_next_seek`'s logic clearer
2023-06-22 10:15:49 +02:00
f1e39d977a
fix(ds): Fix dialyzer warnings
2023-06-22 10:15:47 +02:00
adcbf40d27
refactor(ds): tau -> epoch
2023-06-22 10:15:44 +02:00
Andrew Mayorov