Commit Graph

8681 Commits

Author SHA1 Message Date
zhongwencool c8d1bd3bd5
Merge pull request #11138 from zhongwencool/cluster-conf-k8s-default
fix: update cluster.conf's k8s default api_server value
2023-06-30 17:01:52 +08:00
Kjell Winblad 039e27a153 feat: add Sparkplug encode and decode functions to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
Ilya Averyanov b5c0951584 fix(rebalance): specify correct response schema for global rebalance status 2023-06-30 10:53:08 +03:00
firest 872698d19e feat(ruleengine): port random && uuid_v4 functions from v4 2023-06-30 14:38:27 +08:00
zhongwencool bb1ae808b4
Merge pull request #11165 from zhongwencool/hide-global-limiter-from-swagger-api
chore: hide /configs/limiter from swagger api doc
2023-06-30 11:54:01 +08:00
zhongwencool 3eefa1b2a3 chore: hide /configs/limiter from swagger api doc 2023-06-30 10:59:18 +08:00
zhongwencool 8b679cf358
Merge pull request #11137 from zhongwencool/dashboard-https-ssl-options
feat: refactor dashboard https ssl_options
2023-06-30 09:57:07 +08:00
zhongwencool bd83365d45 fix: update cluster.conf's k8s default api_server value 2023-06-30 09:55:36 +08:00
Thales Macedo Garitezi 59b109eb5c fix(webhook): treat 404 and other error replies as errors in async requests
Fixes https://emqx.atlassian.net/browse/EMQX-10405

The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
Thales Macedo Garitezi 4506eff3ff
Merge pull request #11140 from thalesmg/gcp-consumer-tests
test(gcp_pubsub_consumer): add more tests and improve bridge
2023-06-29 13:02:47 -03:00
Stefan Strigler 14da61194e fix: handle resource errors 2023-06-29 16:09:45 +02:00
Stefan Strigler 321fd53132 fix: use ReplyTo in QUERY for async 2023-06-29 16:09:45 +02:00
Stefan Strigler d71d70c889 fix: allow to pass in Opts rather than just a ReplyTo 2023-06-29 16:09:45 +02:00
Stefan Strigler 40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler 1363108678 fix: fix simple_sync_query 2023-06-29 16:09:45 +02:00
Stefan Strigler 422c79583a test(emqx_rule_engine): fix simple sync 2023-06-29 16:09:45 +02:00
Stefan Strigler 837acd4234 test(emqx_rule_engine): test rule metrics 2023-06-29 16:09:45 +02:00
Stefan Strigler 2274a192cc fix(emqx_resource): call async reply fun in simple_aysnc_query 2023-06-29 16:09:45 +02:00
Stefan Strigler ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
lafirest 9708a02680
Merge pull request #11134 from lafirest/fix/auth_header_sensitive
fix: mask the value of the authorization header whatever its case
2023-06-29 21:43:16 +08:00
ieQu1 f2074e9c9b
Merge pull request #11143 from ieQu1/master
docs(ds): Update IMPLEMENTATION.md
2023-06-29 11:14:55 +02:00
lafirest be1a4c89a5
Merge pull request #11158 from lafirest/fix/wait_table_in_retainer
fix(emqx_retainer): wait for Mria tables
2023-06-29 16:01:54 +08:00
firest 31e914f6a2 fix(emqx_utils): improve the checking for the `authorization` key 2023-06-29 15:57:48 +08:00
firest 6fedd0328f fix(retainer): Fix missing changes in configuration 2023-06-29 14:06:17 +08:00
firest 1202bef6db fix(emqx_retainer): wait for Mria tables 2023-06-29 03:16:05 +00:00
zhongwencool 9dae408b5a
Merge pull request #11142 from zhongwencool/list-listener-group-with-wrong-bind
List listener group with wrong bind
2023-06-29 11:04:35 +08:00
zhongwencool 73487ecdd5 chore: add desc for ssl_options 2023-06-29 10:49:39 +08:00
zhongwencool 82ea9e5a04 test: dashbaord api ct failed 2023-06-29 10:49:39 +08:00
zhongwencool e26c30ff0c feat: refactor dashboard https ssl_options 2023-06-29 10:49:39 +08:00
firest 71d34504bf fix(cluster_rpc): clean self-data by mria stop callback 2023-06-29 01:45:47 +00:00
firest 8e68af14ef chore: bump ekka version 2023-06-29 01:45:47 +00:00
firest 51ec8cb8fa fix(emqx_conf): don't sync the MFA to a leaved node 2023-06-29 01:45:47 +00:00
Thales Macedo Garitezi 30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
Kjell Winblad f9ea924cd0
Merge pull request #11151 from kjellwinblad/kjell/refactor/mysql_bridge/EMQX-9533
refactor: MySQL bridge and connector to separate applications
2023-06-28 15:15:55 +02:00
ieQu1 940ad12108 docs(ds): Apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-28 10:04:07 +02:00
Kjell Winblad 6d6287ef26 test: start emqx_mongodb when running the MongoDB tests 2023-06-28 08:03:41 +02:00
SergeTupchiy 9cf874e37e
Merge pull request #11150 from SergeTupchiy/wait-for-emqx_psk_tab
fix(emqx_psk): wait for Mria table on app start
2023-06-28 08:57:32 +03:00
zhongwencool 30b67add05 chore: add more example for clients api 2023-06-28 10:55:55 +08:00
zhongwencool 93cbb5ebd8 fix: show the local listener's settings first 2023-06-28 10:55:55 +08:00
zhongwencool d556adf86c
Merge pull request #11146 from zhongwencool/hide-node-process-limit
feat: hide node.process_limit
2023-06-28 10:45:37 +08:00
Thales Macedo Garitezi c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Serge Tupchii 8e877caf47 fix(emqx_psk): wait for Mria table on app start
This ensures emqx_psk table is synced to a replicant even if it doesn't have init psk file.
2023-06-27 20:22:40 +03:00
Andrew Mayorov 30d78aaac2
Merge pull request #11152 from keynslug/test/ft-prop-aggressive
test(ft): make proptests even less aggressive
2023-06-27 17:44:54 +02:00
Andrew Mayorov 7d171de02b
test(ft): make proptests even less aggressive 2023-06-27 17:17:56 +02:00
Kjell Winblad 3d1712c00c refactor: MySQL bridge to its own application
This commit refactor the MySQL bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9533
2023-06-27 17:13:26 +02:00
Andrew Mayorov ff327609db
test(ft): generate segments manually in proptests
In order to avoid spamming proptest logs with HUGE type instances,
and make proptests themselves simpler. Proper now should refuse
shrinking altogether, instances that cause prop function to timeout
do not shrink well for some reason, making proptests stuck.
2023-06-27 17:13:23 +02:00
Kjell Winblad 85a130f68b refactor: move MySQL open source connector to its own app 2023-06-27 16:15:18 +02:00
Zaiming (Stone) Shi 4ed422cb9a
Merge pull request #11127 from zmstone/0622-refine-examples
0622 refine examples
2023-06-27 15:30:30 +02:00
Andrew Mayorov 0e55aeada5
Merge pull request #11147 from keynslug/test/ft-timetrap
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 15:11:14 +02:00
Kjell Winblad d3a0b3e183
Merge pull request #11139 from kjellwinblad/kjell/refactor/redis/EMQX-9534
refactor: redis bridge to its own application
2023-06-27 14:55:55 +02:00
Zaiming (Stone) Shi 90c35c3ff8 chore(prometheus): change default vaue enum the first symbol 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 9b6ed09513 refactor: call hocon_tconf:check_plain/4 directly for example conf check
the attemp to re-use emqx_hocon:check failed because
hocon_tconf:check_plain/3 and check_plain/4 behaves slightly different
for unknown root fields.

To keep the old logic unchanged, for example config checks, we call
check_plain/4 directly
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 92ed7d7639 docs: move some prometheus docs from schema desc to code comments 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 41de679557 ci: run example config check in ci 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi a069b351fd test: add test script to verify config example files 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 03d05825db docs: hide core_nodes config doc
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi e0d05a02ff refactor: refine examples
Also turns off vm_dist_collector metrics collector by default
2023-06-27 14:23:18 +02:00
Andrew Mayorov 31e08a6046
test(ft): set safer suite-wide timetraps in cluster-related suites 2023-06-27 13:03:19 +02:00
Serge Tupchii 116675e86f chore: bump ekka to 0.15.4 2023-06-27 13:24:49 +03:00
zhongwencool b4ae4260c0 feat: hide node.process_limit 2023-06-27 18:09:26 +08:00
ieQu1 e0c11f54aa docs(ds): Update IMPLEMENTATION.md 2023-06-27 10:53:38 +02:00
Kjell Winblad a7db85c1e6 style: fix code style issue 2023-06-27 10:52:24 +02:00
Kjell Winblad f99ccd4d76 fix: fix issues found in review by @paulozulato and @thalesmg 2023-06-27 10:49:19 +02:00
Kjell Winblad e76daa0f3b refactor: redis bridge to its own application
This commit refactor the redis bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9534
2023-06-27 10:24:44 +02:00
Kjell Winblad dd6aa96c99 refactor: move Redis open source connector to its own app 2023-06-27 10:24:40 +02:00
Kjell Winblad 36b7aef04c chore: bump versions 2023-06-26 14:35:29 +02:00
Kjell Winblad a91a478d42 build: fix order in mix.exs list due to CI script 2023-06-26 14:32:59 +02:00
Kjell Winblad a65d768873 style: fix format issues 2023-06-26 14:32:59 +02:00
Kjell Winblad fa2c2a300a refactor: rename emqx_connector_mongo to emqx_mongodb
This renaming is consistent with the naming in the only other connector
only application emqx_oracle.
2023-06-26 14:32:59 +02:00
Kjell Winblad b78a75bb5c refactor: the MongoDB connector into its own application 2023-06-26 14:32:59 +02:00
Kjell Winblad e3199ef813 refactor: MongoDB bridge to use the new app structure 2023-06-26 14:32:53 +02:00
lafirest f2e3446e29
Merge pull request #11135 from lafirest/pref/calendar_timeoffset
perf(rule_engine): improve the time offset parser
2023-06-26 18:54:00 +08:00
lafirest 850d642ec2
Merge pull request #11108 from ieQu1/emqx-ds
Durable storage
2023-06-25 11:36:34 +08:00
firest c8c2793944 perf(rule_engine): improve the time offset parser 2023-06-24 13:05:27 +00:00
firest d485f8fe96 fix: mask the value of the authorization header whatever its case 2023-06-24 09:20:26 +00:00
firest 60a1f5a38d chore: update app version && changes 2023-06-23 17:01:22 +00:00
firest 6bb8bf58ea fix(retainer): rename deliver_rate to delivery_rate 2023-06-23 16:51:19 +00:00
Serge Tupchii 639fe276e1 chore(emqx_authz): fix `ipaddrs` type and style in etc/acl.conf 2023-06-23 14:53:26 +03:00
ieQu1 9a81f22105 fix(ds): Apply review remarks 2023-06-22 17:07:30 +02:00
Paulo Zulato f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
Thales Macedo Garitezi 22356b7c25 chore: hide gcp pubsub consumer until e5.2.0 2023-06-22 10:05:52 -03:00
Thales Macedo Garitezi f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi 2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
Thales Macedo Garitezi 7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi 75eb18bc81 chore: bump app vsns 2023-06-22 11:09:09 +02:00
ieQu1 cfcd81e17b fix(ds): Fix build 2023-06-22 11:07:52 +02:00
Zaiming (Stone) Shi c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
ieQu1 b29c5ad23c feat(emqx_ds): Add API draft for logic layer 2023-06-22 10:20:18 +02:00
ieQu1 a343cdb1d5 refactor(ds): message_storage -> message_storage_bitmask 2023-06-22 10:20:18 +02:00
ieQu1 a4219db163 refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer 2023-06-22 10:20:18 +02:00
ieQu1 1159f99432 refactor(ds): emqx_replay -> emqx_ds 2023-06-22 10:20:18 +02:00
ieQu1 8d6bcc1414 refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00
ieQu1 c4f721f214 chore(ds): Add callback definitions to local_store 2023-06-22 10:20:18 +02:00
ieQu1 efed7cdc2f chore(ds): behavior -> behaviour 2023-06-22 10:20:18 +02:00
ieQu1 0f6c81e42d chore(ds): Change license to BSL 2023-06-22 10:20:18 +02:00
ieQu1 04adb65c09 refactor(ds): zone -> shard
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
Andrew Mayorov 8ac0bba958 test(ds): verify preserve / restore works with stored iterators 2023-06-22 10:20:18 +02:00
Andrew Mayorov d2065e0c1b feat(ds): allow to preserve / restore iterators in the db
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
Andrew Mayorov cbded9af39 feat(ds): Make `create_generation` safer against bad input 2023-06-22 10:20:18 +02:00
Andrew Mayorov 83d1606d00 feat(ds): Make iteration fully generation-aware 2023-06-22 10:20:18 +02:00
Andrew Mayorov bf9d57f6a9 chore(ds): Simplify iteration-related typespecs 2023-06-22 10:20:18 +02:00
Andrew Mayorov cf6a5e1643 feat(ds): Allow to create new storage generations 2023-06-22 10:20:18 +02:00
Andrew Mayorov f80e906553 fix(ds): Correct typespec 2023-06-22 10:20:18 +02:00
Andrew Mayorov f5a7b49f57 feat(ds): Wire iteration options up to the app config 2023-06-22 10:20:18 +02:00
Andrew Mayorov 2bf8a07b05 test(ds): Split unit tests off into a full-fledged suite 2023-06-22 10:20:18 +02:00
Andrew Mayorov cf70184ebd test(ds): Provide more general `keymapper_info/1` 2023-06-22 10:20:18 +02:00
Andrew Mayorov 2f385022cd chore(ds): Rename testsuite to reflect test subject better 2023-06-22 10:20:18 +02:00
Andrew Mayorov d875fa49d3 feat(ds): Enable periodic iterator refresh
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
Andrew Mayorov 7f408da251 refactor(ds): Use inline functions instead of macros 2023-06-22 10:20:07 +02:00
Andrew Mayorov a11e75d189 fix(ds): clear bitmask of topic filter tail containing wildcards 2023-06-22 10:16:55 +02:00
Andrew Mayorov ac0935ef91 test(ds): Proptest that iteration is exhaustive
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
Andrew Mayorov 60e3070328 test(ds): Scale up number of messages per topic in proptests 2023-06-22 10:16:41 +02:00
Andrew Mayorov 82a7292851 test(ds): Use `_build/test/proper` as a scratch dir for testruns 2023-06-22 10:16:32 +02:00
Andrew Mayorov 440fecd268 test(ds): Add a proptest on iterator preserve / restore
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
Andrew Mayorov 1f5425975f feat(ds): add an ability to preserve and restore iterators
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
Andrew Mayorov 94e8dbfe7f test(ds): Move proptests into a separate module
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
Andrew Mayorov 4c643aa7d5 refactor(ds): Introduce keyspace filter concept
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
Andrew Mayorov c5826cee56 refactor(ds): rename `compute_hash_*` → `compute_topic_*` 2023-06-22 10:16:05 +02:00
Andrew Mayorov 85d3783ff1 chore(ds): Drop few unused macrodefinitions 2023-06-22 10:15:56 +02:00
Andrew Mayorov a0bcdb5104 chore(ds): Attempt to make `compute_next_seek`'s logic clearer 2023-06-22 10:15:49 +02:00
ieQu1 f1e39d977a fix(ds): Fix dialyzer warnings 2023-06-22 10:15:47 +02:00
ieQu1 adcbf40d27 refactor(ds): tau -> epoch 2023-06-22 10:15:44 +02:00
ieQu1 120d4e66ae refactor(ds): Factor out configuration to a separate module 2023-06-22 10:15:42 +02:00
ieQu1 0bbc5ecb32 refactor(ds): Introduce bitwise_concat function 2023-06-22 10:15:40 +02:00
ieQu1 b5bb77dd58 feat(ds): Add metadata storage and supervision tree 2023-06-22 10:15:37 +02:00
Andrew Mayorov 5e30a5d3dd feat(ds): Implement keyspace partitioning across time 2023-06-22 10:15:32 +02:00
Andrew Mayorov 8c9c9cc669 feat(ds): Allow to specify message store options
* Keymapper
* Column family name + DB options
* DB write / read options
2023-06-22 10:15:25 +02:00
Andrew Mayorov 5b6c4d766b test(ds): Add basic property tests 2023-06-22 10:15:20 +02:00
Andrew Mayorov a1e29e4e24 refactor(ds): Improve function naming 2023-06-22 10:15:03 +02:00
Andrew Mayorov cd12338c3f feat(ds): Smoke tests for wildcard iterator scans 2023-06-22 10:14:46 +02:00
Andrew Mayorov 9c1cd4911d feat(ds): Implement iterator next 2023-06-22 10:14:43 +02:00
ieQu1 27b925405b test(ds): Add testcase 2023-06-22 10:14:41 +02:00
Andrew Mayorov 691a27883d feat(ds): Initial commit 2023-06-22 10:14:37 +02:00
Paulo Zulato f47cc2a458 feat(kafka): add support for kafka headers and ext headers
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
Paulo Zulato 62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
Andrew Mayorov 62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00
Andrew Mayorov 86d787eced
chore: bump hocon to 0.39.10
Which comes with a fix for slightly more user-friendly validation error
messages.
2023-06-21 21:25:43 +02:00
Thales Macedo Garitezi 18f0510353 fix(mongo): return health check failure reason
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
Paulo Zulato 8430ec673c feat(kafka): check whether target topic exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi decfd6df2b feat(buffer_worker): log expired message count
Fixes https://emqx.atlassian.net/browse/EMQX-10165

```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
2023-06-21 14:38:51 -03:00
Zaiming (Stone) Shi 7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Zaiming (Stone) Shi 5fa87091c4 Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-21 16:08:41 +02:00
Ilya Averyanov 7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-21 15:07:45 +03:00
zhongwencool f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type 2023-06-21 17:36:59 +08:00
Zaiming (Stone) Shi 5f47649d4d chore: downgrade log severity level of 'not_ready' nodes to info 2023-06-21 09:45:45 +02:00
Zaiming (Stone) Shi 045ed340dd fix(emqx_conf_app): fix release version detect during cluster conf sync 2023-06-21 09:28:46 +02:00
Thales Macedo Garitezi 0463828e84 refactor(gcp_pubsub): transform connector into opaque client 2023-06-20 15:27:42 -03:00
Thales Macedo Garitezi 1d791d7a8c fix(resource): validate maximum worker pool size
Fixes https://emqx.atlassian.net/browse/EMQX-10297
2023-06-20 14:26:42 -03:00
Thales Macedo Garitezi ffce6fefa8 docs: note that connector state must be immutable to avoid errors 2023-06-20 13:50:53 -03:00
Zaiming (Stone) Shi dddccfdc5c chore: bump to v5.1.0 official cut 2023-06-20 17:09:17 +02:00
Zaiming (Stone) Shi 728656d4f6 chore: bump to e5.1.0 official cut 2023-06-20 17:08:18 +02:00
Thales Macedo Garitezi ba96c725e5 feat: add time-based pull for safety 2023-06-20 11:36:35 -03:00
Thales Macedo Garitezi 57460e83b9 test(fix): rm unused var warnings 2023-06-20 11:15:37 -03:00
Thales Macedo Garitezi 2ac2d4c037 refactor: addressing review comments 2023-06-20 11:15:13 -03:00
Ilya Averyanov 54d96488a0 chore(deps): update erlcloud and remove unnecessary overrides 2023-06-20 16:46:23 +03:00
Zaiming (Stone) Shi b588b88a60 chore: change autoclean interval default value to 24h
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).

To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.

This PR also has the autoclean config removed from examples.
2023-06-20 15:43:37 +02:00
lafirest 8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example
chore: add examples of how to configure
2023-06-20 20:09:44 +08:00
Zaiming (Stone) Shi f95787d229 test: make possible to test core/replica locally 2023-06-20 12:42:00 +02:00
ieQu1 e5803a61f1 fix(management): Don't wait for an empty shard 2023-06-20 10:08:41 +02:00
firest bb2bf059ac chore: modify the build to copy examples into the release 2023-06-20 14:53:14 +08:00
zhongwencool 36ee872361 chore: hide plugins from conf load cli 2023-06-20 10:34:09 +08:00
zhongwencool 856de78698 chore: remove the is_list guard 2023-06-20 08:39:23 +08:00
zhongwencool 093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool 07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
zhongwencool a4be9b8281 chore: bump hocon to 0.39.9 2023-06-20 08:39:23 +08:00
zhongwencool 45963b6a95 fix: ip_port schema type crash 2023-06-20 08:39:23 +08:00
Zaiming (Stone) Shi ead511a128 chore: bump to 5.1.0-rc.1 2023-06-19 23:19:27 +02:00
ieQu1 86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5
chore(mria): Bump version to 0.5.5
2023-06-19 23:13:36 +02:00
Thales Macedo Garitezi 13746c2cdf fix(resource): check status when (re)starting a resource
Fixes https://emqx.atlassian.net/browse/EMQX-10290
2023-06-19 18:01:02 -03:00
Thales Macedo Garitezi 01d758e4c0 fix(bpapi): fix `query_mode` type(spec)
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi bb49482529 refactor(gcp_pubsub): move logging from connector to bridge 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi dabefdb01b fix(gcp_pubsub): redact request when errors occur 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi 7f850f7499 fix(resource): fix `query_mode/0` type and usage 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi a1690b3684 chore(gcp_pubsub): add stricter compile/xref flags 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi 5375421954 refactor(gcp_pubsub): split connector into producer and reusable parts 2023-06-19 15:59:00 -03:00
Paulo Zulato 5f10936091 feat(oracle): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
Paulo Zulato c9a2ddf98c feat(mysql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
ieQu1 f82c56ae16 chore(mria): Bump version to 0.5.5 2023-06-19 19:28:17 +02:00
JianBo He cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8
chore: bump to v5.1.0.alpha.8
2023-06-19 23:00:53 +08:00
JianBo He b0a6c18917 chore: bump vsn to 5.1.0-alpha.8 2023-06-19 22:59:05 +08:00
Paulo Zulato 9454af9a8b feat(postgresql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
Andrew Mayorov af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 16:00:04 +02:00
Thales Macedo Garitezi 7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master
fix(nolink_apply): avoid sending late replies to caller
2023-06-19 10:32:43 -03:00
Andrew Mayorov ae5846be29
fix(ft-api): make schema tags look nicer in the docs 2023-06-19 13:35:21 +02:00
zhongwencool 1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli
chore: hide plugins from conf load cli
2023-06-19 19:11:52 +08:00
zhongwencool 64612cab14 chore: bump to v5.1.0.alpha.8 2023-06-19 18:22:51 +08:00
zhongwencool 6efb07f6eb fix: crash on emqx_ctl listeners 2023-06-19 18:12:33 +08:00
zhongwencool a578d6fa1d chore: release e5.1.0-alpha.8 2023-06-19 17:19:03 +08:00
zhongwencool 8b20a70344 chore: hide plugins from conf load cli 2023-06-19 15:57:40 +08:00
zhongwencool 803cd6c812
Merge pull request #11082 from emqx/release-51
sync release-51 to master
2023-06-19 15:12:37 +08:00
JimMoen 30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error
fix: shared-sub with nl sub-option should cause protocol error
2023-06-19 10:39:08 +08:00
Kjell Winblad 59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216
fix: rule engine different behavior for div and mod
2023-06-17 07:20:37 +02:00
Andrew Mayorov 13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
2023-06-17 00:54:24 +02:00
Thales Macedo Garitezi 50e7d5d2ec fix(nolink_apply): avoid sending late replies to caller
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.

e.g. when a timeout occurred while resource manager was checking a resource's health:

```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```

Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277))
does), we avoid polluting the caller's mailbox with late replies.
2023-06-16 17:03:39 -03:00
Thales Macedo Garitezi 8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard
fix(rewrite): avoid wildcards on destination topic on publish
2023-06-16 09:57:46 -03:00
JimMoen 178711f742
fix: shared-sub with nl sub-option should cause protocol error 2023-06-16 18:19:23 +08:00
zhongwencool 5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean
fix: use mria.cluster_autoclean
2023-06-16 14:56:52 +08:00
JimMoen c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal 2023-06-16 14:04:24 +08:00
Serge Tupchii b49b80d36c fix(emqx_schema): use non negative integer type for 'depth' SSL option
Closes: EMQX-10276
2023-06-16 10:59:39 +08:00
Paulo Zulato b2f4dbd232
Merge pull request #11067 from paulozulato/chore-bump-influx
chore(influxdb): bump version for influxdb client
2023-06-15 20:16:28 -03:00
Paulo Zulato e127df3912 chore(influxdb): bump version for influxdb client 2023-06-15 17:53:08 -03:00
zhongwencool 9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename 2023-06-15 20:51:52 +08:00
zhongwencool 3406a5d916
Merge pull request #11063 from zhongwencool/bump-to-alpha.7
chore: bump to e5.1.0-alpha.7
2023-06-15 19:22:37 +08:00
zhongwencool fc1d78134a test: replace 'to' with 'path' 2023-06-15 19:13:51 +08:00
zhongwencool 9060a2dc05 chore: bump to e5.1.0-alpha.7 2023-06-15 19:05:13 +08:00
zhongwencool e6c2da8d19 feat: rename log.file.to to log.file.path 2023-06-15 18:24:21 +08:00
zhongwencool 84a5d0c3c1 fix: delete a default listener then recreate it, it's disabled 2023-06-15 17:19:11 +08:00
zhongwencool 0adbb0deec feat: try to save config when post_config_update failed not in init mfa status 2023-06-15 17:19:11 +08:00
zhongwencool c160301dbe fix: don't log enoent error 2023-06-15 17:06:09 +08:00
zhongwencool 7dade3a52c fix: tls_certfile_gc notice log don't print abspath 2023-06-15 17:06:09 +08:00
Serge Tupchii 64bbe21209 fix(emqx_schema): use non negative integer type for 'depth' SSL option
Closes: EMQX-10276
2023-06-15 11:22:58 +03:00
JimMoen 903d8d899e
fix: .js files are required by dashboard as static
revert parts of 8c1334c421
2023-06-15 16:10:16 +08:00
firest c28346d67b chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-15 10:35:18 +08:00
zhongwencool e42cc58694
Merge pull request #11050 from emqx/release-51
chore: sync release-51 back to master
2023-06-15 09:45:31 +08:00
Paulo Zulato 8c7c0877fd fix(rewrite): avoid wildcards on destination topic on publish
Fixes https://emqx.atlassian.net/browse/EMQX-9247
2023-06-14 20:02:03 -03:00
Andrew Mayorov 6dc281345c
test(ft-fs): run API tests with replicant nodes 2023-06-15 00:50:18 +03:00
Andrew Mayorov acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
2023-06-15 00:50:15 +03:00
Andrew Mayorov 52354bf58a
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
When iterating over complete exports cluster-wide.
2023-06-14 22:54:56 +03:00
Paulo Zulato 36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username
fix(influxdb): check authentication
2023-06-14 14:41:50 -03:00
zhongwencool 41004e86da
Merge pull request #10959 from qzhuyan/dev/william/remove-depre-quic-config
chore: remove deprecated QUIC listener config
2023-06-14 23:01:00 +08:00
zhongwencool bd269e259e
Merge pull request #11047 from zhongwencool/dashboard-listeners-deprecate-enable
feat: deprecated dashboard.listeners.name.enable
2023-06-14 22:58:39 +08:00
Serge Tupchii 7882755114 chore: bump cowboy to 2.9.2
cowboy 2.9.2 uses ranch 1.8.1-emqx that fixes incompatible TLS options
2023-06-14 17:35:17 +03:00
Serge Tupchii 4425942030 fix(emqx_schema): move tls version gap validation to emqx_schema 2023-06-14 17:29:56 +03:00
Ivan Dyachkov 6d88cd7a20 fix(tls): fix incompatible tls options and issue with tls version gap 2023-06-14 17:29:56 +03:00
Paulo Zulato a573b9b38c fix(influxdb): check authentication
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.

Fixes https://emqx.atlassian.net/browse/EMQX-10213
2023-06-14 11:27:39 -03:00
zhongwencool 9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
zhongwencool 4b82d4d927 feat: deprecated dashboard.listeners.name.enable 2023-06-14 20:51:17 +08:00
zhongwencool 267053cc35
Merge pull request #11045 from zhongwencool/hide-zones-authn-in-listeners
feat: hide zone/authn in listeners and remove listeners's authn api
2023-06-14 20:32:27 +08:00
Stefan Strigler bf62cdc3e3
Merge pull request #11037 from sstrigler/EMQX-10174-e-5-1-0-reconnect-does-not-show-any-errors-for-apache-io-tdb-or-webhook-or-kafka
fix(emqx_connector): report errors in on_start handler
2023-06-14 13:20:47 +02:00
zhongwencool f6df52020c test: conf_cli failed with replace 2023-06-14 18:43:41 +08:00
Andrew Mayorov 1beb588207
fix(ft): give more failproof example in README
Without that `ssl` defaults to being disabled, and interacting
with AWS S3 with such configuration consistently times out.
2023-06-14 13:15:00 +03:00
zhongwencool 9a0856fd6b fix: authn/authz's keys should be binary 2023-06-14 18:00:36 +08:00
zhongwencool bd33dcb049 feat: remove zone from /clients api 2023-06-14 17:11:20 +08:00
Stefan Strigler 0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
zhongwencool 121cd9124a chore: improve show_keys output 2023-06-14 15:53:55 +08:00
zhongwencool 7668753f50 feat: hide zone/authn in listeners and remove /listeners/:id/authentication api 2023-06-14 15:17:12 +08:00
JianBo He 80e2c2c955 fix(http): avoid crash if the old sample data dont contain `live_connections`
In version 5.1.0, a new metric called "live_connections" was added. However, we try to merge
some data from the Mria disk table, which may include old data formats.

Therefore, maps:get/3 is needed to avoid crashes.
2023-06-14 15:14:20 +08:00
JianBo He 8ce0380704 chore: bump vsn to e5.1.0-alpha.6 2023-06-14 12:49:02 +08:00
JianBo He 86d3b702d7
Merge pull request #11042 from zhongwencool/max-connections-string
fix: api_listener crash when setting max_connnections as string
2023-06-14 12:46:40 +08:00
zhongwencool feb08b1181 chore: update 11042 changelog 2023-06-14 11:40:20 +08:00
zhongwencool 091c49277f fix: api_listener crash when setting max_connnections as string 2023-06-14 11:26:41 +08:00
JianBo He 26ad06b9ca chore: update changes 2023-06-14 10:49:17 +08:00
JianBo He c4c32f5032 chore: cover stomp mountpoint tests 2023-06-14 10:49:17 +08:00
JianBo He 4212d90672 fix(stomp): unmount topic prefix in delivering proccess 2023-06-14 10:49:17 +08:00
JianBo He e3d208f678 fix(gw): close the stomp connections once an error frame occured
According to the Stomp v1.2 specification:

> The server MAY send ERROR frames if something goes wrong. In this case,
> it MUST then close the connection just after sending the ERROR frame

Additional, fixes the `is_superuser` is not working for all gateways
2023-06-14 10:49:17 +08:00
zhongwencool d41a23b7d0
Merge pull request #11036 from zhongwencool/conf-load-cli
feat: support --replace|merge with conf load
2023-06-14 10:48:13 +08:00
zhongwencool 7f0f40cb58
Merge pull request #11034 from zhongwencool/mqtt-conf
feat:  move shared_subscription_strategy from broker to mqtt
2023-06-14 10:00:38 +08:00
zhongwencool 9b5c7065f2 feat: conf reload support --replace|merge 2023-06-14 09:12:39 +08:00
zhongwencool 095fb040c0 chore: format conf cli usage text 2023-06-14 09:12:39 +08:00
zhongwencool a8959900d0 feat: support --replace|merge with conf load 2023-06-14 09:12:39 +08:00
zhongwencool f57ff802e4
Merge pull request #11038 from thalesmg/fix-pulsar-get-status-r51
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
2023-06-14 08:32:25 +08:00
Thales Macedo Garitezi 25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 13:54:57 -03:00
Stefan Strigler b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
Thales Macedo Garitezi b609792a90 fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10278

Since Pulsar client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 11:44:45 -03:00
zhongwencool 946de4a303 chore: dialyzer warning 2023-06-13 20:11:49 +08:00
zhongwencool 50041fdddf feat: hide broker rootkey and move broker.shared_subscription_strategy to mqtt.shared_subscription_strategy 2023-06-13 19:35:52 +08:00
zhongwencool beee35bdea
Merge pull request #11033 from HJianBo/fix-exproto-mountpoint
feat(exproto): abandon the mountpoint field in AuthenticateRequest
2023-06-13 19:18:17 +08:00
JianBo He 7afd9d2b06 chore: update changes 2023-06-13 16:38:23 +08:00
firest 536f7ba423 chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-13 16:06:28 +08:00
JianBo He 7684cefa86 feat(exproto): abandon the mountpoint field in AuthenticateRequest
This field was introduced in v4.x, but in fact, in 5.0 we have provided
`gateway.exproto.mountpoint` for configuration, so there is no need to override
it through the Authenticate request
2023-06-13 14:10:14 +08:00
William Yang 8fa743fa22 chore: mark deprecated QUIC listener config for 5.1 2023-06-13 06:00:57 +02:00
zhongwencool 3331af4757 feat: make gateway and slow_subs low level and hide rule_engine 2023-06-13 11:47:38 +08:00
Thales Macedo Garitezi 5ef7102cba fix(listener_mgmt_api): use union member selector fn for better error messages (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10247
2023-06-12 16:48:13 -03:00
Thales Macedo Garitezi 73fe671dba test: rm unused var warnings 2023-06-12 15:19:47 -03:00
zhongwencool 58dc042279 chore: bump to e/v5.1.0-alpha.5 2023-06-13 00:06:54 +08:00
Thales Macedo Garitezi c11011d857
Merge pull request #11024 from thalesmg/pulsar-cosmetic-connecting-check-r51
feat(pulsar): retry health check a bit before returning (r5.1)
2023-06-12 13:00:54 -03:00
Kjell Winblad cf31b65076 fix: rule engine different behavior for div and mod
Previously, the div operation could only be used as an infix operation
while mod could only be used as a function call. After this commit, one
can use both div and mod using function call syntax and infix syntax.

Fixes: https://emqx.atlassian.net/browse/EMQX-10216
2023-06-12 17:20:52 +02:00
SergeTupchiy 5ef19ecf58
Merge pull request #10993 from SergeTupchiy/improve_mnesia_backup_import
fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
2023-06-12 17:26:26 +03:00
Thales Macedo Garitezi 741c1f091e fix(pulsar): update pulsar -> 0.8.3
Fixes https://emqx.atlassian.net/browse/EMQX-10229

See https://github.com/emqx/pulsar-client-erl/pull/59

Fixes a case_clause error that could arise from race conditions.
2023-06-12 10:29:40 -03:00
Thales Macedo Garitezi db5d14d5bf feat(pulsar): retry health check a bit before returning (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10228

This is a cosmetic fix for the Pulsar Producer bridge health check status.

Pulsar connection process is asynchronous, therefore, when a bridge of this type is
created or updated (which is the same as stopping and re-creating it), the immediate
status will be connecting because it’s indeed still connecting.  The bridge will connect
very soon afterwards (assuming there are no true network/config issue), but having to
refresh the UI to see the new status and/or seeing the resource alarm might annoy users.

This workaround adds a few retries to account for that effect to reduce the probability of
seeing the `connecting` state on such happy-paths.
2023-06-12 10:26:07 -03:00
firest 0ed420f81f fix: obfuscated the value of JWT 2023-06-12 18:01:02 +08:00
Ilya Averyanov 93733c1ccc fix(docs): fix typo in ft README 2023-06-12 11:38:17 +02:00
zhongwencool 0fc10ad239
Merge pull request #10984 from zhongwencool/reload-cli
feat: support reload cli
2023-06-12 17:21:29 +08:00
Zaiming (Stone) Shi 12b73ba1c0 Merge remote-tracking branch 'origin/master' into release-51 2023-06-12 10:35:47 +02:00
firest 947d99218b chore: upgrade emqtt to avoid sensitive data leakage in the debug log 2023-06-12 16:32:24 +08:00
Ilya Averyanov bf2c768182
Merge pull request #11010 from zhongwencool/fs-delete-SUITE-data
chore: delete emqx_utils_fs_SUITE_data dir
2023-06-12 10:40:48 +03:00
JianBo He d14d87b443
Merge pull request #10955 from HJianBo/clean-mqttsn-topic-registry
Fix(mqttsn): clean predefined topics once gateway unload
2023-06-12 09:37:47 +08:00
Zaiming (Stone) Shi e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb
fix(mongodb): hide batch_size for mongodb resource
2023-06-11 21:26:12 +02:00
zhongwencool 89ddd54c68
Merge pull request #11013 from HJianBo/fix-max-conns-in-gateway-lists-api
fix(gateway): fix 500 crash for '/gateways' endpoint
2023-06-11 22:29:24 +08:00
JianBo He 57a3b6fdda fix(gateway): fix 500 crash for '/gateways' endpoint
It introduced by https://github.com/emqx/emqx/pull/10961
2023-06-11 18:43:38 +08:00
zhongwencool 1550158ddd fix: update emqx_flapping process's state when emqx:update_config([flapping_detect],Conf) 2023-06-11 18:23:15 +08:00
JianBo He 001e2976ec chore: fix typos 2023-06-11 18:13:25 +08:00
JianBo He 9c4a576fb3 feat: reserved 1-1024 topic id for predefined topics 2023-06-11 18:12:34 +08:00
JianBo He ac6693c8cc refactor(mqttsn): takeover can resume the registrations of session 2023-06-11 18:12:34 +08:00
JianBo He 3c6afee690 refactor(mqttsn): make the topic registration mechanism simpler 2023-06-11 18:12:34 +08:00
JianBo He 9760181ed9 test: cover mqttsn predefined topics 2023-06-11 18:12:34 +08:00
JianBo He 6fc50d11f8 fix(mqttsn): clean pre-defined topics once gateway reloaded 2023-06-11 18:12:34 +08:00
Zaiming (Stone) Shi 654a9dcab5
Merge pull request #11006 from zmstone/0610-fix-quic-default-certs-path
fix(quic): environment variables as cert file prefix for quic listener
2023-06-11 11:52:56 +02:00
zhongwencool c8cf8d1def
Merge pull request #11009 from zhongwencool/persistent_session_store-default-value
chore: correct persistent_session_store's default value type to boolean
2023-06-11 17:26:58 +08:00
Zaiming (Stone) Shi ddef751527 fix(mongodb): hide batch_size for mongodb resource
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
2023-06-11 11:08:58 +02:00
zhongwencool a10107045f test: delete emqx_ft_fs_utils_SUITE_data from source code 2023-06-11 15:50:45 +08:00
zhongwencool aed64aae60 chore: delete emqx_utils_fs_SUITE_data dir 2023-06-11 15:50:45 +08:00
zhongwencool 072f6fac8d chore: update hocon to 0.39.8 2023-06-11 09:11:17 +08:00
zhongwencool 346b363cb5 test: config SUITE failed 2023-06-11 09:11:17 +08:00
zhongwencool 505d677685 chore: move reload_local_etc_config from emqx_conf to emqx_conf_cli 2023-06-11 09:11:17 +08:00
zhongwencool 1ca1ba9e7a fix: get_sources return default authz when sources key not found 2023-06-11 09:11:17 +08:00
zhongwencool 01671d82c7 feat: delete authorization form etc/emqx.conf 2023-06-11 09:11:17 +08:00
zhongwencool 6c6ff4ea1f chore: rename {mod} to module 2023-06-11 09:11:17 +08:00
zhongwencool cf9a207743 feat: check all config schema before loading 2023-06-11 09:11:17 +08:00
zhongwencool b829d8edc2 feat: support --auth-chains merge|replace mod 2023-06-11 09:11:17 +08:00
zhongwencool 21ffd958b4 feat: support authn merge_authenticators 2023-06-11 09:11:17 +08:00
zhongwencool aac5b85d26 feat: authz support merge sources 2023-06-11 09:11:17 +08:00
zhongwencool 994db58cbf fix: load authz's default sources 2023-06-11 09:11:17 +08:00
zhongwencool c819ac27f4 feat: add conf reload cli 2023-06-11 09:11:17 +08:00
zhongwencool 61bbe19eba
Merge pull request #10961 from HJianBo/support-infinity-max-conns-in-gateway
Adds support for unlimited max connections for gateway listeners
2023-06-11 09:09:34 +08:00
zhongwencool 599eb9fef5 chore: correct persistent_session_store's default value type to boolean 2023-06-10 19:36:05 +08:00
Zaiming (Stone) Shi 97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi 9f135d1f2b fix(quic): environment variables as cert file prefix for quic listener 2023-06-10 12:23:20 +02:00
JianBo He aea3ae4024
Merge pull request #11005 from HJianBo/fix-tracing-logs-typos
fix(authn): fix the hardcode method in trace logs
2023-06-10 18:08:28 +08:00
Zaiming (Stone) Shi 3901ed3255
Merge pull request #10996 from sstrigler/EMQX-9435-request-the-non-existent-api-v-4-nodes-endpoint-in-emqx-e-5-0-1-returns-http-code-200
Return 404 for everything we don't know
2023-06-10 11:20:25 +02:00
Zaiming (Stone) Shi f98cdd4983
Merge pull request #10994 from sstrigler/EMQX-10003-e-5-0-4-auth-header-value-of-webhook-data-bridge-can-be-found-in-emqx-log
fix(emqx_utils): redact proxy-authorization headers
2023-06-10 11:18:58 +02:00
JianBo He fdaac9800c fix(authn): fix the hardcode method in trace logs 2023-06-10 13:04:24 +08:00
Serge Tupchii 48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option
Closes: EMQX-8725
2023-06-09 22:10:17 +03:00
Kjell Winblad 43292c06e4
Merge pull request #10997 from kjellwinblad/kjell/fix/clickhouse_lost_messages/EMQX-10217
fix: lost messages when clickhouse closes while sending messages
2023-06-09 21:08:25 +02:00
Serge Tupchii 87b57112df refactor: move tcp keepalive options helper to emqx_utils 2023-06-09 21:43:50 +03:00
Paulo Zulato 7c63f20aa8
Merge pull request #10999 from paulozulato/fix-kafka-time-field-too-large
fix(kafka): fix timeout field for some kafka fields
2023-06-09 15:29:59 -03:00
Paulo Zulato 720a84d48b fix(kafka): fix timeout field for some kafka fields
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
2023-06-09 13:38:04 -03:00
Kjell Winblad 1b3af2ac91 refactor: nicer handling of errors and warning log for recoverable errors 2023-06-09 17:22:16 +02:00
Stefan Strigler 57d72ed23e fix(emqx_utils): redact proxy-authorization headers 2023-06-09 17:13:53 +02:00
Stefan Strigler 8c1334c421 fix(emqx_dashboard): return 404 for everything we don't know 2023-06-09 17:13:18 +02:00
Andrew Mayorov b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder
refactor: tear `emqx_plugin_libs` application apart
2023-06-09 17:02:14 +02:00
Thales Macedo Garitezi 4a9f2247ca test(conf_app): fix bad merge conflict 2023-06-09 11:06:54 -03:00
Kjell Winblad 61488ec407 fix: lost messages when clickhouse closes while sending messages
Fixes:
https://emqx.atlassian.net/browse/EMQX-10217
2023-06-09 16:04:38 +02:00
Thales Macedo Garitezi 40675843a1 fix(iotdb): remove root `request_timeout` option
Fixes https://emqx.atlassian.net/browse/EMQX-10175

Since IoTDB bridge is not used as an authn/authz source and reuses the
webhook connector, we should have only one request timeout field (the
`resource_opts.request_ttl` one) like the webhook bridge does, to
avoid confusion and potentially bad configuration combinations.
2023-06-09 10:39:01 -03:00
Zaiming (Stone) Shi 643da0564b
Merge pull request #10990 from zmstone/0609-fix-schema-for-dashboard-use
0609 fix schema for dashboard use
2023-06-09 15:07:51 +02:00
Thales Macedo Garitezi 20a7ce64cd
Merge pull request #10919 from thalesmg/test-flakiness-20230601-a
test(pulsar_producer): attempt to fix flaky test
2023-06-09 09:30:30 -03:00
Andrew Mayorov c8360f86b7
chore: bump application versions 2023-06-09 14:44:38 +03:00
Andrew Mayorov 830ba54721
refactor(pluglib): remove `emqx_plugin_libs` application 2023-06-09 14:44:37 +03:00
Andrew Mayorov e6fb0203b4
refactor(pluglib): move connectivity checks to `emqx_connector_lib` 2023-06-09 14:44:37 +03:00
Andrew Mayorov 8919a6ef93
refactor(pluglib): provide SQL related utils in `emqx_utils_sql` 2023-06-09 14:44:37 +03:00
Andrew Mayorov a51baaa206
refactor(pluglib): move conversion utils to `emqx_utils_conv` 2023-06-09 14:44:37 +03:00
Andrew Mayorov 7d0abb6146
feat(emqx): add `emqx_topic:match_any/2` utility 2023-06-09 14:44:37 +03:00
Andrew Mayorov d6c1ee183f
refactor(pluglib): move `emqx_placeholder` to utils app
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
Serge Tupchii 2aad6671ac fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
As we use `keep_tables` option, we don't need to modify a backup schema before importing the backup file.
With this option, `mnesia:restore/2` ignores backup schema and keeps current table schema unchanged.
2023-06-09 14:41:21 +03:00
SergeTupchiy e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup
feat: implement configuration and user data export/import
2023-06-09 14:18:39 +03:00
Serge Tupchii e4d09d4ad4 feat: implement configuration and user data export/import CLI
Closes: EMQX-9203
2023-06-09 14:11:47 +03:00
Kjell Winblad 4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631
feat: add sync/async option to the Kafka producer bridge
2023-06-09 12:51:23 +02:00
Kjell Winblad 2671e8ecf9 fix: dialyzer type problem 2023-06-09 11:00:05 +02:00
Ilya Averyanov c2635e938b
Merge pull request #10974 from keynslug/ft/EMQX-9523/backend-enable-flag
fix(ft): add `enable` flag to every backend
2023-06-09 11:44:37 +03:00
JianBo He 53eb8d7f1b test: fix flaky tests introduced by https://github.com/emqx/emqx/pull/10948 2023-06-09 15:48:23 +08:00
JianBo He 72311a546b chore: more clear funcation name 2023-06-09 15:48:23 +08:00
JianBo He 6a05663bd5 chore: ensure default value's type be the first one in the union type
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-06-09 15:48:23 +08:00
JianBo He dbb8742e0e test: cover max_connection in infinity value 2023-06-09 15:48:23 +08:00
JianBo He 8df7b1a1be fix(gateway): fix http-api 500 issue if setting max_connections to infinity 2023-06-09 15:48:23 +08:00
JianBo He c450a1784a
Merge pull request #10950 from HJianBo/mqttsn-qos3
fix(mqttsn): checking enable_qos3 option
2023-06-09 15:46:55 +08:00
JianBo He 2153cd4317
Merge pull request #10979 from HJianBo/fix-coap-connectionless-pub
fix(coap): fix crash publish/subscribe in connectionless mode
2023-06-09 15:46:18 +08:00
Zaiming (Stone) Shi 38e767b808 fix(schema): map new duration types to dashboard's type name 2023-06-09 09:14:55 +02:00
Stefan Strigler 2e9ac8f1ec
Merge pull request #10980 from sstrigler/EMQX-10189-mark-support-for-json-output-format-in-prometheus-stats-as-deprecated
Mark support for json output format in prometheus stats as deprecated
2023-06-09 09:00:34 +02:00
Zaiming (Stone) Shi ee697831f6
Merge pull request #10926 from zmstone/0602-add-enable-as-an-alias-for-enabled
0602 add enable as an alias for enabled
2023-06-09 08:44:20 +02:00
Zaiming (Stone) Shi 9f6beee6b3
Merge pull request #10936 from zmstone/0603-imporove-desc
docs: refine some desc
2023-06-09 08:44:03 +02:00
Zaiming (Stone) Shi af5c6720de Merge remote-tracking branch 'origin/master' into release-51 2023-06-09 08:37:29 +02:00
lafirest 601db2b1d3
Merge pull request #10988 from lafirest/fix/konservi_sentema
fix(resource): improve log security when resource creation fails
2023-06-09 13:46:12 +08:00
firest 86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
zhongwencool daa5450218
Merge pull request #10985 from zmstone/0608-rename-cluster-call-cmd
0608 rename cluster call cmd
2023-06-09 10:56:18 +08:00
zhongwencool 4835c30f8c
Merge pull request #10975 from zhongwencool/conf-docs
fix: link different document between community and enterprise
2023-06-09 09:50:45 +08:00
JianBo He c733ddb166
Merge pull request #10977 from HJianBo/fix-stomp-bugs
Fix stomp gateway subscription_cnt is not updated timely and max_headers option is not working
2023-06-09 09:47:26 +08:00
zhongwencool c9cc06b6b1 chore: compile failed 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi 83c36a3c9f refactor(emqx_conf_cli): do not print hidden roots in 'conf show' cmd 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi 31ab973b30 refactor(emqx_conf_cli): rename cluster_call ctl command
Renamed 'clustedr_call' ctl command to 'cluster_sync' as a sub-command
of 'conf'
The old 'cluster_call' command is kept, but no usage prints
2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi 96a41ac6bd feat(emqx_ctl): support hidden commands
hidden commands do not deserve a place in the usage page
2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi 47f826f99a chore: fix typo 2023-06-09 09:31:32 +08:00
Serge Tupchii 53ec8326b0 fix(emqx_listeners): convert authentication certs 2023-06-08 20:44:18 +03:00
Serge Tupchii 33fa053b9b test(emqx_common_test_helpers): keep EMQX ENV vars on a child node for its lifetime 2023-06-08 20:42:51 +03:00
Serge Tupchii 5f526d548a style: use double percent (%%) comments 2023-06-08 20:42:51 +03:00
Kjell Winblad 1c7834e056 fix: fixes due to comments from @zmstone 2023-06-08 16:47:02 +02:00
Kjell Winblad cb3a5fdbd4 style: only callback modules should do dynamic calls 2023-06-08 16:27:04 +02:00
Kjell Winblad d524f8c805 refactor: rename config parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad 6f2271e9f0 test: add Kafka producer bridge test case for query_mode parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad ed9e29e769 refactor: refacor query_mode detection code
This commit refactor the query_mode resource detection code according to
a suggestion from @zmstone. This commit should not contain any
functional change except for a change of the Kafka producer bridge
config.
2023-06-08 16:26:55 +02:00
Kjell Winblad 273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609
fix: friendly error message when creating bridges with too long names
2023-06-08 15:46:42 +02:00
JianBo He 034caaf20f
Merge pull request #10941 from HJianBo/enhance-prometheus-sampe-perf
perf(prometheus): udpate the prometheus dep vsn
2023-06-08 21:34:50 +08:00
Ivan Dyachkov abbba71191 fix(tls): issue when ssl listner is configured to use tls v1.3 only
clients could not connect due to incompatible tls options if ssl listner is configured to use tls v1.3 only
2023-06-08 15:13:22 +02:00
Zaiming (Stone) Shi 7d2ea3dc76 docs: refine some desc 2023-06-08 13:58:31 +02:00
Andrew Mayorov ce8cc05cc8
chore(ftschema): mention `translate` is for tests purpose only 2023-06-08 14:54:14 +03:00
Andrew Mayorov e3c16be3c4
fix(ftschema): ensure `translate` follow `emqx_config` behavior
So that atom keys won't unexpectedly get to converters / validators.
2023-06-08 14:54:14 +03:00
Andrew Mayorov ae97d516b8
fix(ft): add `enable` flag to every backend
And tolerate multiple configured backends, as long as only one of
them is enabled.
2023-06-08 14:54:13 +03:00
Ivan Dyachkov b4d081f424
Merge pull request #10978 from id/0608-fix-404-in-rule-engine-api
fix: missing 404 response in rule engine api schema
2023-06-08 13:20:02 +02:00
Kjell Winblad 47fa17b3c1 feat: add sync/async option to the Kafka producer bridge
This commit makes it possible to configure if a Kafka bridge should work
in query mode sync or async by setting the resource_opts.query_mode
configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-8631
2023-06-08 13:16:06 +02:00
Stefan Strigler f5e93b1b2b docs(emqx_prometheus): leave remark about json being deprecated 2023-06-08 13:08:45 +02:00
Zaiming (Stone) Shi b481994e74 chore: bump to version 5.1.0-alpha.4 2023-06-08 12:30:35 +02:00
Zaiming (Stone) Shi c1cf2365c2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-08 12:30:02 +02:00
JianBo He 95f0745e31 fix(coap): fix crash publish/subscribe in connectionless mode
this bug was introduced by https://github.com/emqx/emqx/pull/10871
2023-06-08 18:08:11 +08:00
Ivan Dyachkov 711cedc282 fix: missing 404 response in rule engine api schema 2023-06-08 11:46:16 +02:00
JianBo He 4065f08083 fix(stomp): fix frame limitation is not working 2023-06-08 16:57:12 +08:00
JianBo He bad0c35bb9 fix(stomp): ensure the subscripton_cnt timely updated 2023-06-08 15:25:44 +08:00
Zaiming (Stone) Shi 641166a67a
Merge pull request #10971 from zmstone/0607-merge-master-to-release-51
0607 merge master to release 51
2023-06-08 09:22:21 +02:00
JianBo He dc01b7109e chore: update prometheus vsn and grafana template 2023-06-08 14:29:54 +08:00
JianBo He e0ec050e34 chore: update prometheus vsn 2023-06-08 14:29:54 +08:00
JianBo He f53e13b8b9 perf: update to our forked prometheus deps
it's based on v4.10.0 of upstream, and patched this PR:
https://github.com/emqx/prometheus.erl/pull/5
2023-06-08 14:29:54 +08:00
Ilya Averyanov b9f1a70214
Merge pull request #10967 from savonarola/0607-rebalance-fixes
Fix rebalance issues
2023-06-08 08:29:37 +03:00
zhongwencool 0116c53395 fix: link different document between community and enterprise 2023-06-08 11:33:39 +08:00
JianBo He d5ebf5650f test: improve the test 2023-06-08 11:24:07 +08:00
JianBo He fcef456174 feat: add `live_connections` field for some HTTP APIs
i.e:
- `/monitor_current`, `/monitor_current/nodes/{node}`
- `/monitor/nodes/{node}`, `/monitor`
- `/node/{node}`, `/nodes`
2023-06-08 11:24:06 +08:00
JianBo He 5d86443910 chore: fix dialyzer warnings 2023-06-08 11:06:25 +08:00
JianBo He d0222ae79d chore: ensure test 2023-06-08 10:23:59 +08:00
JianBo He 4e9eeb489d
Merge pull request #10969 from thalesmg/refactor-gcp-pubsub-on-stop-v50
refactor(gcp_pubsub): always cleanup resources when calling `on_stop`
2023-06-08 09:47:19 +08:00
JianBo He 0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple
feat: refactored some bridges to avoid leaking resources part.2
2023-06-08 09:37:31 +08:00
Zaiming (Stone) Shi 641aca00d8
Merge pull request #10954 from zhongwencool/flapping-detect-conf
feat: don't hide enable in flapping detect conf
2023-06-07 23:04:56 +02:00
Zaiming (Stone) Shi ccd2589ff2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-07 21:43:29 +02:00