yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,790 Commits 393 Branches 669 Tags 74 MiB
b1f144ab8b
Commit Graph

9147 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
75eb18bc81 chore: bump app vsns 2023-06-22 11:09:09 +02:00
ieQu1
cfcd81e17b fix(ds): Fix build 2023-06-22 11:07:52 +02:00
Zaiming (Stone) Shi
c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
ieQu1
b29c5ad23c feat(emqx_ds): Add API draft for logic layer 2023-06-22 10:20:18 +02:00
ieQu1
a343cdb1d5 refactor(ds): message_storage -> message_storage_bitmask 2023-06-22 10:20:18 +02:00
ieQu1
a4219db163 refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer 2023-06-22 10:20:18 +02:00
ieQu1
1159f99432 refactor(ds): emqx_replay -> emqx_ds 2023-06-22 10:20:18 +02:00
ieQu1
8d6bcc1414 refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00
ieQu1
c4f721f214 chore(ds): Add callback definitions to local_store 2023-06-22 10:20:18 +02:00
ieQu1
efed7cdc2f chore(ds): behavior -> behaviour 2023-06-22 10:20:18 +02:00
ieQu1
0f6c81e42d chore(ds): Change license to BSL 2023-06-22 10:20:18 +02:00
ieQu1
04adb65c09 refactor(ds): zone -> shard 
Also bump erlang-rocksdb version
 2023-06-22 10:20:18 +02:00
Andrew Mayorov
8ac0bba958 test(ds): verify preserve / restore works with stored iterators 2023-06-22 10:20:18 +02:00
Andrew Mayorov
d2065e0c1b feat(ds): allow to preserve / restore iterators in the db 
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
 2023-06-22 10:20:18 +02:00
Andrew Mayorov
cbded9af39 feat(ds): Make create_generation safer against bad input 2023-06-22 10:20:18 +02:00
Andrew Mayorov
83d1606d00 feat(ds): Make iteration fully generation-aware 2023-06-22 10:20:18 +02:00
Andrew Mayorov
bf9d57f6a9 chore(ds): Simplify iteration-related typespecs 2023-06-22 10:20:18 +02:00
Andrew Mayorov
cf6a5e1643 feat(ds): Allow to create new storage generations 2023-06-22 10:20:18 +02:00
Andrew Mayorov
f80e906553 fix(ds): Correct typespec 2023-06-22 10:20:18 +02:00
Andrew Mayorov
f5a7b49f57 feat(ds): Wire iteration options up to the app config 2023-06-22 10:20:18 +02:00
Andrew Mayorov
2bf8a07b05 test(ds): Split unit tests off into a full-fledged suite 2023-06-22 10:20:18 +02:00
Andrew Mayorov
cf70184ebd test(ds): Provide more general keymapper_info/1 2023-06-22 10:20:18 +02:00
Andrew Mayorov
2f385022cd chore(ds): Rename testsuite to reflect test subject better 2023-06-22 10:20:18 +02:00
Andrew Mayorov
d875fa49d3 feat(ds): Enable periodic iterator refresh 
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
 2023-06-22 10:20:18 +02:00
Andrew Mayorov
7f408da251 refactor(ds): Use inline functions instead of macros 2023-06-22 10:20:07 +02:00
Andrew Mayorov
a11e75d189 fix(ds): clear bitmask of topic filter tail containing wildcards 2023-06-22 10:16:55 +02:00
Andrew Mayorov
ac0935ef91 test(ds): Proptest that iteration is exhaustive 
Compare iteration results against what an extremely simplified model
produces.
 2023-06-22 10:16:45 +02:00
Andrew Mayorov
60e3070328 test(ds): Scale up number of messages per topic in proptests 2023-06-22 10:16:41 +02:00
Andrew Mayorov
82a7292851 test(ds): Use _build/test/proper as a scratch dir for testruns 2023-06-22 10:16:32 +02:00
Andrew Mayorov
440fecd268 test(ds): Add a proptest on iterator preserve / restore 
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
 2023-06-22 10:16:23 +02:00
Andrew Mayorov
1f5425975f feat(ds): add an ability to preserve and restore iterators 
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
 2023-06-22 10:16:20 +02:00
Andrew Mayorov
94e8dbfe7f test(ds): Move proptests into a separate module 
Following conventions. Also add few proptests on keyspace filters.
 2023-06-22 10:16:16 +02:00
Andrew Mayorov
4c643aa7d5 refactor(ds): Introduce keyspace filter concept 
So we could conveniently test it separately.
 2023-06-22 10:16:10 +02:00
Andrew Mayorov
c5826cee56 refactor(ds): rename compute_hash_* → compute_topic_* 2023-06-22 10:16:05 +02:00
Andrew Mayorov
85d3783ff1 chore(ds): Drop few unused macrodefinitions 2023-06-22 10:15:56 +02:00
Andrew Mayorov
a0bcdb5104 chore(ds): Attempt to make compute_next_seek's logic clearer 2023-06-22 10:15:49 +02:00
ieQu1
f1e39d977a fix(ds): Fix dialyzer warnings 2023-06-22 10:15:47 +02:00
ieQu1
adcbf40d27 refactor(ds): tau -> epoch 2023-06-22 10:15:44 +02:00
ieQu1
120d4e66ae refactor(ds): Factor out configuration to a separate module 2023-06-22 10:15:42 +02:00
ieQu1
0bbc5ecb32 refactor(ds): Introduce bitwise_concat function 2023-06-22 10:15:40 +02:00
ieQu1
b5bb77dd58 feat(ds): Add metadata storage and supervision tree 2023-06-22 10:15:37 +02:00
Andrew Mayorov
5e30a5d3dd feat(ds): Implement keyspace partitioning across time 2023-06-22 10:15:32 +02:00
Andrew Mayorov
8c9c9cc669 feat(ds): Allow to specify message store options 
* Keymapper
* Column family name + DB options
* DB write / read options
 2023-06-22 10:15:25 +02:00
Andrew Mayorov
5b6c4d766b test(ds): Add basic property tests 2023-06-22 10:15:20 +02:00
Andrew Mayorov
a1e29e4e24 refactor(ds): Improve function naming 2023-06-22 10:15:03 +02:00
Andrew Mayorov
cd12338c3f feat(ds): Smoke tests for wildcard iterator scans 2023-06-22 10:14:46 +02:00
Andrew Mayorov
9c1cd4911d feat(ds): Implement iterator next 2023-06-22 10:14:43 +02:00
ieQu1
27b925405b test(ds): Add testcase 2023-06-22 10:14:41 +02:00
Andrew Mayorov
691a27883d feat(ds): Initial commit 2023-06-22 10:14:37 +02:00
Paulo Zulato
f47cc2a458 feat(kafka): add support for kafka headers and ext headers 
Fixes https://emqx.atlassian.net/browse/EMQX-9175
 2023-06-21 18:27:39 -03:00
Paulo Zulato
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable 
Data bridge target unavailable
 2023-06-21 18:19:23 -03:00
Andrew Mayorov
62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon 
chore: bump hocon to 0.39.10
 2023-06-21 23:13:35 +02:00
Andrew Mayorov
86d787eced
chore: bump hocon to 0.39.10 
Which comes with a fix for slightly more user-friendly validation error
messages.
 2023-06-21 21:25:43 +02:00
Thales Macedo Garitezi
18f0510353 fix(mongo): return health check failure reason 
Fixes https://emqx.atlassian.net/browse/EMQX-10335
 2023-06-21 15:09:37 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi
decfd6df2b feat(buffer_worker): log expired message count 
Fixes https://emqx.atlassian.net/browse/EMQX-10165

```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
 2023-06-21 14:38:51 -03:00
Zaiming (Stone) Shi
7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Zaiming (Stone) Shi
5fa87091c4 Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-21 16:08:41 +02:00
Ilya Averyanov
7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud 
chore(deps): update erlcloud and remove unnecessary overrides
 2023-06-21 15:07:45 +03:00
zhongwencool
f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type 2023-06-21 17:36:59 +08:00
Zaiming (Stone) Shi
5f47649d4d chore: downgrade log severity level of 'not_ready' nodes to info 2023-06-21 09:45:45 +02:00
Zaiming (Stone) Shi
045ed340dd fix(emqx_conf_app): fix release version detect during cluster conf sync 2023-06-21 09:28:46 +02:00
Thales Macedo Garitezi
0463828e84 refactor(gcp_pubsub): transform connector into opaque client 2023-06-20 15:27:42 -03:00
Thales Macedo Garitezi
1d791d7a8c fix(resource): validate maximum worker pool size 
Fixes https://emqx.atlassian.net/browse/EMQX-10297
 2023-06-20 14:26:42 -03:00
Thales Macedo Garitezi
ffce6fefa8 docs: note that connector state must be immutable to avoid errors 2023-06-20 13:50:53 -03:00
Zaiming (Stone) Shi
dddccfdc5c chore: bump to v5.1.0 official cut 2023-06-20 17:09:17 +02:00
Zaiming (Stone) Shi
728656d4f6 chore: bump to e5.1.0 official cut 2023-06-20 17:08:18 +02:00
Thales Macedo Garitezi
ba96c725e5 feat: add time-based pull for safety 2023-06-20 11:36:35 -03:00
Thales Macedo Garitezi
57460e83b9 test(fix): rm unused var warnings 2023-06-20 11:15:37 -03:00
Thales Macedo Garitezi
2ac2d4c037 refactor: addressing review comments 2023-06-20 11:15:13 -03:00
Ilya Averyanov
54d96488a0 chore(deps): update erlcloud and remove unnecessary overrides 2023-06-20 16:46:23 +03:00
Zaiming (Stone) Shi
b588b88a60 chore: change autoclean interval default value to 24h 
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).

To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.

This PR also has the autoclean config removed from examples.
 2023-06-20 15:43:37 +02:00
lafirest
8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example 
chore: add examples of how to configure
 2023-06-20 20:09:44 +08:00
Zaiming (Stone) Shi
f95787d229 test: make possible to test core/replica locally 2023-06-20 12:42:00 +02:00
ieQu1
e5803a61f1 fix(management): Don't wait for an empty shard 2023-06-20 10:08:41 +02:00
firest
bb2bf059ac chore: modify the build to copy examples into the release 2023-06-20 14:53:14 +08:00
zhongwencool
36ee872361 chore: hide plugins from conf load cli 2023-06-20 10:34:09 +08:00
zhongwencool
856de78698 chore: remove the is_list guard 2023-06-20 08:39:23 +08:00
zhongwencool
093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
zhongwencool
a4be9b8281 chore: bump hocon to 0.39.9 2023-06-20 08:39:23 +08:00
zhongwencool
45963b6a95 fix: ip_port schema type crash 2023-06-20 08:39:23 +08:00
Zaiming (Stone) Shi
ead511a128 chore: bump to 5.1.0-rc.1 2023-06-19 23:19:27 +02:00
ieQu1
86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5 
chore(mria): Bump version to 0.5.5
 2023-06-19 23:13:36 +02:00
Thales Macedo Garitezi
13746c2cdf fix(resource): check status when (re)starting a resource 
Fixes https://emqx.atlassian.net/browse/EMQX-10290
 2023-06-19 18:01:02 -03:00
Thales Macedo Garitezi
01d758e4c0 fix(bpapi): fix query_mode type(spec) 
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
 2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi
bb49482529 refactor(gcp_pubsub): move logging from connector to bridge 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
dabefdb01b fix(gcp_pubsub): redact request when errors occur 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
7f850f7499 fix(resource): fix query_mode/0 type and usage 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
a1690b3684 chore(gcp_pubsub): add stricter compile/xref flags 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
5375421954 refactor(gcp_pubsub): split connector into producer and reusable parts 2023-06-19 15:59:00 -03:00
Paulo Zulato
5f10936091 feat(oracle): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 15:22:27 -03:00
Paulo Zulato
c9a2ddf98c feat(mysql): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 15:22:27 -03:00
ieQu1
f82c56ae16 chore(mria): Bump version to 0.5.5 2023-06-19 19:28:17 +02:00
JianBo He
cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8 
chore: bump to v5.1.0.alpha.8
 2023-06-19 23:00:53 +08:00
JianBo He
b0a6c18917 chore: bump vsn to 5.1.0-alpha.8 2023-06-19 22:59:05 +08:00
Paulo Zulato
9454af9a8b feat(postgresql): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 11:12:10 -03:00
Andrew Mayorov
af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags 
fix(ft-api): make schema tags look nicer in the docs
 2023-06-19 16:00:04 +02:00
Thales Macedo Garitezi
7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master 
fix(nolink_apply): avoid sending late replies to caller
 2023-06-19 10:32:43 -03:00
Andrew Mayorov
ae5846be29
fix(ft-api): make schema tags look nicer in the docs 2023-06-19 13:35:21 +02:00
zhongwencool
1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli 
chore: hide plugins from conf load cli
 2023-06-19 19:11:52 +08:00
zhongwencool
64612cab14 chore: bump to v5.1.0.alpha.8 2023-06-19 18:22:51 +08:00
zhongwencool
6efb07f6eb fix: crash on emqx_ctl listeners 2023-06-19 18:12:33 +08:00
zhongwencool
a578d6fa1d chore: release e5.1.0-alpha.8 2023-06-19 17:19:03 +08:00
zhongwencool
8b20a70344 chore: hide plugins from conf load cli 2023-06-19 15:57:40 +08:00
zhongwencool
803cd6c812
Merge pull request #11082 from emqx/release-51 
sync release-51 to master
 2023-06-19 15:12:37 +08:00
JimMoen
30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error 
fix: shared-sub with nl sub-option should cause protocol error
 2023-06-19 10:39:08 +08:00
Kjell Winblad
59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216 
fix: rule engine different behavior for div and mod
 2023-06-17 07:20:37 +02:00
Andrew Mayorov
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view 
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
 2023-06-17 00:54:24 +02:00
Thales Macedo Garitezi
50e7d5d2ec fix(nolink_apply): avoid sending late replies to caller 
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.

e.g. when a timeout occurred while resource manager was checking a resource's health:

```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```

Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277))
does), we avoid polluting the caller's mailbox with late replies.
 2023-06-16 17:03:39 -03:00
Thales Macedo Garitezi
8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard 
fix(rewrite): avoid wildcards on destination topic on publish
 2023-06-16 09:57:46 -03:00
JimMoen
178711f742
fix: shared-sub with nl sub-option should cause protocol error 2023-06-16 18:19:23 +08:00
zhongwencool
5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean 
fix: use mria.cluster_autoclean
 2023-06-16 14:56:52 +08:00
JimMoen
c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal 2023-06-16 14:04:24 +08:00
Serge Tupchii
b49b80d36c fix(emqx_schema): use non negative integer type for 'depth' SSL option 
Closes: EMQX-10276
 2023-06-16 10:59:39 +08:00
Paulo Zulato
b2f4dbd232
Merge pull request #11067 from paulozulato/chore-bump-influx 
chore(influxdb): bump version for influxdb client
 2023-06-15 20:16:28 -03:00
Paulo Zulato
e127df3912 chore(influxdb): bump version for influxdb client 2023-06-15 17:53:08 -03:00
zhongwencool
9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename 2023-06-15 20:51:52 +08:00
zhongwencool
3406a5d916
Merge pull request #11063 from zhongwencool/bump-to-alpha.7 
chore: bump to e5.1.0-alpha.7
 2023-06-15 19:22:37 +08:00
zhongwencool
fc1d78134a test: replace 'to' with 'path' 2023-06-15 19:13:51 +08:00
zhongwencool
9060a2dc05 chore: bump to e5.1.0-alpha.7 2023-06-15 19:05:13 +08:00
zhongwencool
e6c2da8d19 feat: rename log.file.to to log.file.path 2023-06-15 18:24:21 +08:00
zhongwencool
84a5d0c3c1 fix: delete a default listener then recreate it, it's disabled 2023-06-15 17:19:11 +08:00
zhongwencool
0adbb0deec feat: try to save config when post_config_update failed not in init mfa status 2023-06-15 17:19:11 +08:00
zhongwencool
c160301dbe fix: don't log enoent error 2023-06-15 17:06:09 +08:00
zhongwencool
7dade3a52c fix: tls_certfile_gc notice log don't print abspath 2023-06-15 17:06:09 +08:00
Serge Tupchii
64bbe21209 fix(emqx_schema): use non negative integer type for 'depth' SSL option 
Closes: EMQX-10276
 2023-06-15 11:22:58 +03:00
JimMoen
903d8d899e
fix: .js files are required by dashboard as static 
revert parts of 8c1334c421
 2023-06-15 16:10:16 +08:00
firest
c28346d67b chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-15 10:35:18 +08:00
zhongwencool
e42cc58694
Merge pull request #11050 from emqx/release-51 
chore: sync release-51 back to master
 2023-06-15 09:45:31 +08:00
Paulo Zulato
8c7c0877fd fix(rewrite): avoid wildcards on destination topic on publish 
Fixes https://emqx.atlassian.net/browse/EMQX-9247
 2023-06-14 20:02:03 -03:00
Andrew Mayorov
6dc281345c
test(ft-fs): run API tests with replicant nodes 2023-06-15 00:50:18 +03:00
Andrew Mayorov
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes 
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
 2023-06-15 00:50:15 +03:00
Andrew Mayorov
52354bf58a
fix(ft-fs): use emqx:running_nodes() as default cluster view 
When iterating over complete exports cluster-wide.
 2023-06-14 22:54:56 +03:00
Paulo Zulato
36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username 
fix(influxdb): check authentication
 2023-06-14 14:41:50 -03:00
zhongwencool
41004e86da
Merge pull request #10959 from qzhuyan/dev/william/remove-depre-quic-config 
chore: remove deprecated QUIC listener config
 2023-06-14 23:01:00 +08:00
zhongwencool
bd269e259e
Merge pull request #11047 from zhongwencool/dashboard-listeners-deprecate-enable 
feat: deprecated dashboard.listeners.name.enable
 2023-06-14 22:58:39 +08:00
Serge Tupchii
7882755114 chore: bump cowboy to 2.9.2 
cowboy 2.9.2 uses ranch 1.8.1-emqx that fixes incompatible TLS options
 2023-06-14 17:35:17 +03:00
Serge Tupchii
4425942030 fix(emqx_schema): move tls version gap validation to emqx_schema 2023-06-14 17:29:56 +03:00
Ivan Dyachkov
6d88cd7a20 fix(tls): fix incompatible tls options and issue with tls version gap 2023-06-14 17:29:56 +03:00
Paulo Zulato
a573b9b38c fix(influxdb): check authentication 
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.

Fixes https://emqx.atlassian.net/browse/EMQX-10213
 2023-06-14 11:27:39 -03:00
zhongwencool
9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
zhongwencool
4b82d4d927 feat: deprecated dashboard.listeners.name.enable 2023-06-14 20:51:17 +08:00
zhongwencool
267053cc35
Merge pull request #11045 from zhongwencool/hide-zones-authn-in-listeners 
feat: hide zone/authn in listeners and remove listeners's authn api
 2023-06-14 20:32:27 +08:00
Stefan Strigler
bf62cdc3e3
Merge pull request #11037 from sstrigler/EMQX-10174-e-5-1-0-reconnect-does-not-show-any-errors-for-apache-io-tdb-or-webhook-or-kafka 
fix(emqx_connector): report errors in on_start handler
 2023-06-14 13:20:47 +02:00
zhongwencool
f6df52020c test: conf_cli failed with replace 2023-06-14 18:43:41 +08:00
Andrew Mayorov
1beb588207
fix(ft): give more failproof example in README 
Without that `ssl` defaults to being disabled, and interacting
with AWS S3 with such configuration consistently times out.
 2023-06-14 13:15:00 +03:00
zhongwencool
9a0856fd6b fix: authn/authz's keys should be binary 2023-06-14 18:00:36 +08:00
zhongwencool
bd33dcb049 feat: remove zone from /clients api 2023-06-14 17:11:20 +08:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
zhongwencool
121cd9124a chore: improve show_keys output 2023-06-14 15:53:55 +08:00
zhongwencool
7668753f50 feat: hide zone/authn in listeners and remove /listeners/:id/authentication api 2023-06-14 15:17:12 +08:00
JianBo He
80e2c2c955 fix(http): avoid crash if the old sample data dont contain live_connections 
In version 5.1.0, a new metric called "live_connections" was added. However, we try to merge
some data from the Mria disk table, which may include old data formats.

Therefore, maps:get/3 is needed to avoid crashes.
 2023-06-14 15:14:20 +08:00
JianBo He
8ce0380704 chore: bump vsn to e5.1.0-alpha.6 2023-06-14 12:49:02 +08:00
JianBo He
86d3b702d7
Merge pull request #11042 from zhongwencool/max-connections-string 
fix: api_listener crash when setting max_connnections as string
 2023-06-14 12:46:40 +08:00
zhongwencool
feb08b1181 chore: update 11042 changelog 2023-06-14 11:40:20 +08:00
zhongwencool
091c49277f fix: api_listener crash when setting max_connnections as string 2023-06-14 11:26:41 +08:00
JianBo He
26ad06b9ca chore: update changes 2023-06-14 10:49:17 +08:00
JianBo He
c4c32f5032 chore: cover stomp mountpoint tests 2023-06-14 10:49:17 +08:00
JianBo He
4212d90672 fix(stomp): unmount topic prefix in delivering proccess 2023-06-14 10:49:17 +08:00
JianBo He
e3d208f678 fix(gw): close the stomp connections once an error frame occured 
According to the Stomp v1.2 specification:

> The server MAY send ERROR frames if something goes wrong. In this case,
> it MUST then close the connection just after sending the ERROR frame

Additional, fixes the `is_superuser` is not working for all gateways
 2023-06-14 10:49:17 +08:00
zhongwencool
d41a23b7d0
Merge pull request #11036 from zhongwencool/conf-load-cli 
feat: support --replace|merge with conf load
 2023-06-14 10:48:13 +08:00
zhongwencool
7f0f40cb58
Merge pull request #11034 from zhongwencool/mqtt-conf 
feat:  move shared_subscription_strategy from broker to mqtt
 2023-06-14 10:00:38 +08:00
zhongwencool
9b5c7065f2 feat: conf reload support --replace|merge 2023-06-14 09:12:39 +08:00
zhongwencool
095fb040c0 chore: format conf cli usage text 2023-06-14 09:12:39 +08:00
zhongwencool
a8959900d0 feat: support --replace|merge with conf load 2023-06-14 09:12:39 +08:00
zhongwencool
f57ff802e4
Merge pull request #11038 from thalesmg/fix-pulsar-get-status-r51 
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
 2023-06-14 08:32:25 +08:00
Thales Macedo Garitezi
25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 13:54:57 -03:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
Thales Macedo Garitezi
b609792a90 fix(pulsar_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10278

Since Pulsar client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 11:44:45 -03:00
zhongwencool
946de4a303 chore: dialyzer warning 2023-06-13 20:11:49 +08:00
zhongwencool
50041fdddf feat: hide broker rootkey and move broker.shared_subscription_strategy to mqtt.shared_subscription_strategy 2023-06-13 19:35:52 +08:00
zhongwencool
beee35bdea
Merge pull request #11033 from HJianBo/fix-exproto-mountpoint 
feat(exproto): abandon the mountpoint field in AuthenticateRequest
 2023-06-13 19:18:17 +08:00
JianBo He
7afd9d2b06 chore: update changes 2023-06-13 16:38:23 +08:00
firest
536f7ba423 chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-13 16:06:28 +08:00
JianBo He
7684cefa86 feat(exproto): abandon the mountpoint field in AuthenticateRequest 
This field was introduced in v4.x, but in fact, in 5.0 we have provided
`gateway.exproto.mountpoint` for configuration, so there is no need to override
it through the Authenticate request
 2023-06-13 14:10:14 +08:00
William Yang
8fa743fa22 chore: mark deprecated QUIC listener config for 5.1 2023-06-13 06:00:57 +02:00
zhongwencool
3331af4757 feat: make gateway and slow_subs low level and hide rule_engine 2023-06-13 11:47:38 +08:00
Thales Macedo Garitezi
5ef7102cba fix(listener_mgmt_api): use union member selector fn for better error messages (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10247
 2023-06-12 16:48:13 -03:00
Thales Macedo Garitezi
73fe671dba test: rm unused var warnings 2023-06-12 15:19:47 -03:00
zhongwencool
58dc042279 chore: bump to e/v5.1.0-alpha.5 2023-06-13 00:06:54 +08:00
Thales Macedo Garitezi
c11011d857
Merge pull request #11024 from thalesmg/pulsar-cosmetic-connecting-check-r51 
feat(pulsar): retry health check a bit before returning (r5.1)
 2023-06-12 13:00:54 -03:00
Kjell Winblad
cf31b65076 fix: rule engine different behavior for div and mod 
Previously, the div operation could only be used as an infix operation
while mod could only be used as a function call. After this commit, one
can use both div and mod using function call syntax and infix syntax.

Fixes: https://emqx.atlassian.net/browse/EMQX-10216
 2023-06-12 17:20:52 +02:00
SergeTupchiy
5ef19ecf58
Merge pull request #10993 from SergeTupchiy/improve_mnesia_backup_import 
fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
 2023-06-12 17:26:26 +03:00
Thales Macedo Garitezi
741c1f091e fix(pulsar): update pulsar -> 0.8.3 
Fixes https://emqx.atlassian.net/browse/EMQX-10229

See https://github.com/emqx/pulsar-client-erl/pull/59

Fixes a case_clause error that could arise from race conditions.
 2023-06-12 10:29:40 -03:00
Thales Macedo Garitezi
db5d14d5bf feat(pulsar): retry health check a bit before returning (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10228

This is a cosmetic fix for the Pulsar Producer bridge health check status.

Pulsar connection process is asynchronous, therefore, when a bridge of this type is
created or updated (which is the same as stopping and re-creating it), the immediate
status will be connecting because it’s indeed still connecting.  The bridge will connect
very soon afterwards (assuming there are no true network/config issue), but having to
refresh the UI to see the new status and/or seeing the resource alarm might annoy users.

This workaround adds a few retries to account for that effect to reduce the probability of
seeing the `connecting` state on such happy-paths.
 2023-06-12 10:26:07 -03:00
firest
0ed420f81f fix: obfuscated the value of JWT 2023-06-12 18:01:02 +08:00
Ilya Averyanov
93733c1ccc fix(docs): fix typo in ft README 2023-06-12 11:38:17 +02:00
zhongwencool
0fc10ad239
Merge pull request #10984 from zhongwencool/reload-cli 
feat: support reload cli
 2023-06-12 17:21:29 +08:00
Zaiming (Stone) Shi
12b73ba1c0 Merge remote-tracking branch 'origin/master' into release-51 2023-06-12 10:35:47 +02:00
firest
947d99218b chore: upgrade emqtt to avoid sensitive data leakage in the debug log 2023-06-12 16:32:24 +08:00
Ilya Averyanov
bf2c768182
Merge pull request #11010 from zhongwencool/fs-delete-SUITE-data 
chore: delete emqx_utils_fs_SUITE_data dir
 2023-06-12 10:40:48 +03:00
JianBo He
d14d87b443
Merge pull request #10955 from HJianBo/clean-mqttsn-topic-registry 
Fix(mqttsn): clean predefined topics once gateway unload
 2023-06-12 09:37:47 +08:00
Zaiming (Stone) Shi
e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb 
fix(mongodb): hide batch_size for mongodb resource
 2023-06-11 21:26:12 +02:00
zhongwencool
89ddd54c68
Merge pull request #11013 from HJianBo/fix-max-conns-in-gateway-lists-api 
fix(gateway): fix 500 crash for '/gateways' endpoint
 2023-06-11 22:29:24 +08:00
JianBo He
57a3b6fdda fix(gateway): fix 500 crash for '/gateways' endpoint 
It introduced by https://github.com/emqx/emqx/pull/10961
 2023-06-11 18:43:38 +08:00
zhongwencool
1550158ddd fix: update emqx_flapping process's state when emqx:update_config([flapping_detect],Conf) 2023-06-11 18:23:15 +08:00
JianBo He
001e2976ec chore: fix typos 2023-06-11 18:13:25 +08:00
JianBo He
9c4a576fb3 feat: reserved 1-1024 topic id for predefined topics 2023-06-11 18:12:34 +08:00
JianBo He
ac6693c8cc refactor(mqttsn): takeover can resume the registrations of session 2023-06-11 18:12:34 +08:00
JianBo He
3c6afee690 refactor(mqttsn): make the topic registration mechanism simpler 2023-06-11 18:12:34 +08:00
JianBo He
9760181ed9 test: cover mqttsn predefined topics 2023-06-11 18:12:34 +08:00
JianBo He
6fc50d11f8 fix(mqttsn): clean pre-defined topics once gateway reloaded 2023-06-11 18:12:34 +08:00
Zaiming (Stone) Shi
654a9dcab5
Merge pull request #11006 from zmstone/0610-fix-quic-default-certs-path 
fix(quic): environment variables as cert file prefix for quic listener
 2023-06-11 11:52:56 +02:00
zhongwencool
c8cf8d1def
Merge pull request #11009 from zhongwencool/persistent_session_store-default-value 
chore: correct persistent_session_store's default value type to boolean
 2023-06-11 17:26:58 +08:00
Zaiming (Stone) Shi
ddef751527 fix(mongodb): hide batch_size for mongodb resource 
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
 2023-06-11 11:08:58 +02:00
zhongwencool
a10107045f test: delete emqx_ft_fs_utils_SUITE_data from source code 2023-06-11 15:50:45 +08:00
zhongwencool
aed64aae60 chore: delete emqx_utils_fs_SUITE_data dir 2023-06-11 15:50:45 +08:00
zhongwencool
072f6fac8d chore: update hocon to 0.39.8 2023-06-11 09:11:17 +08:00
zhongwencool
346b363cb5 test: config SUITE failed 2023-06-11 09:11:17 +08:00
zhongwencool
505d677685 chore: move reload_local_etc_config from emqx_conf to emqx_conf_cli 2023-06-11 09:11:17 +08:00
zhongwencool
1ca1ba9e7a fix: get_sources return default authz when sources key not found 2023-06-11 09:11:17 +08:00
zhongwencool
01671d82c7 feat: delete authorization form etc/emqx.conf 2023-06-11 09:11:17 +08:00
zhongwencool
6c6ff4ea1f chore: rename {mod} to module 2023-06-11 09:11:17 +08:00
zhongwencool
cf9a207743 feat: check all config schema before loading 2023-06-11 09:11:17 +08:00
zhongwencool
b829d8edc2 feat: support --auth-chains merge|replace mod 2023-06-11 09:11:17 +08:00
zhongwencool
21ffd958b4 feat: support authn merge_authenticators 2023-06-11 09:11:17 +08:00
zhongwencool
aac5b85d26 feat: authz support merge sources 2023-06-11 09:11:17 +08:00
zhongwencool
994db58cbf fix: load authz's default sources 2023-06-11 09:11:17 +08:00
zhongwencool
c819ac27f4 feat: add conf reload cli 2023-06-11 09:11:17 +08:00
zhongwencool
61bbe19eba
Merge pull request #10961 from HJianBo/support-infinity-max-conns-in-gateway 
Adds support for unlimited max connections for gateway listeners
 2023-06-11 09:09:34 +08:00
zhongwencool
599eb9fef5 chore: correct persistent_session_store's default value type to boolean 2023-06-10 19:36:05 +08:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi
9f135d1f2b fix(quic): environment variables as cert file prefix for quic listener 2023-06-10 12:23:20 +02:00
JianBo He
aea3ae4024
Merge pull request #11005 from HJianBo/fix-tracing-logs-typos 
fix(authn): fix the hardcode method in trace logs
 2023-06-10 18:08:28 +08:00
Zaiming (Stone) Shi
3901ed3255
Merge pull request #10996 from sstrigler/EMQX-9435-request-the-non-existent-api-v-4-nodes-endpoint-in-emqx-e-5-0-1-returns-http-code-200 
Return 404 for everything we don't know
 2023-06-10 11:20:25 +02:00
Zaiming (Stone) Shi
f98cdd4983
Merge pull request #10994 from sstrigler/EMQX-10003-e-5-0-4-auth-header-value-of-webhook-data-bridge-can-be-found-in-emqx-log 
fix(emqx_utils): redact proxy-authorization headers
 2023-06-10 11:18:58 +02:00
JianBo He
fdaac9800c fix(authn): fix the hardcode method in trace logs 2023-06-10 13:04:24 +08:00
Serge Tupchii
48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option 
Closes: EMQX-8725
 2023-06-09 22:10:17 +03:00
Kjell Winblad
43292c06e4
Merge pull request #10997 from kjellwinblad/kjell/fix/clickhouse_lost_messages/EMQX-10217 
fix: lost messages when clickhouse closes while sending messages
 2023-06-09 21:08:25 +02:00
Serge Tupchii
87b57112df refactor: move tcp keepalive options helper to emqx_utils 2023-06-09 21:43:50 +03:00
Paulo Zulato
7c63f20aa8
Merge pull request #10999 from paulozulato/fix-kafka-time-field-too-large 
fix(kafka): fix timeout field for some kafka fields
 2023-06-09 15:29:59 -03:00
Paulo Zulato
720a84d48b fix(kafka): fix timeout field for some kafka fields 
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
 2023-06-09 13:38:04 -03:00
Kjell Winblad
1b3af2ac91 refactor: nicer handling of errors and warning log for recoverable errors 2023-06-09 17:22:16 +02:00
Stefan Strigler
57d72ed23e fix(emqx_utils): redact proxy-authorization headers 2023-06-09 17:13:53 +02:00
Stefan Strigler
8c1334c421 fix(emqx_dashboard): return 404 for everything we don't know 2023-06-09 17:13:18 +02:00
Andrew Mayorov
b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder 
refactor: tear `emqx_plugin_libs` application apart
 2023-06-09 17:02:14 +02:00
Thales Macedo Garitezi
4a9f2247ca test(conf_app): fix bad merge conflict 2023-06-09 11:06:54 -03:00
Kjell Winblad
61488ec407 fix: lost messages when clickhouse closes while sending messages 
Fixes:
https://emqx.atlassian.net/browse/EMQX-10217
 2023-06-09 16:04:38 +02:00
Thales Macedo Garitezi
40675843a1 fix(iotdb): remove root request_timeout option 
Fixes https://emqx.atlassian.net/browse/EMQX-10175

Since IoTDB bridge is not used as an authn/authz source and reuses the
webhook connector, we should have only one request timeout field (the
`resource_opts.request_ttl` one) like the webhook bridge does, to
avoid confusion and potentially bad configuration combinations.
 2023-06-09 10:39:01 -03:00
Zaiming (Stone) Shi
643da0564b
Merge pull request #10990 from zmstone/0609-fix-schema-for-dashboard-use 
0609 fix schema for dashboard use
 2023-06-09 15:07:51 +02:00
Thales Macedo Garitezi
20a7ce64cd
Merge pull request #10919 from thalesmg/test-flakiness-20230601-a 
test(pulsar_producer): attempt to fix flaky test
 2023-06-09 09:30:30 -03:00
Andrew Mayorov
c8360f86b7
chore: bump application versions 2023-06-09 14:44:38 +03:00
Andrew Mayorov
830ba54721
refactor(pluglib): remove emqx_plugin_libs application 2023-06-09 14:44:37 +03:00
Andrew Mayorov
e6fb0203b4
refactor(pluglib): move connectivity checks to emqx_connector_lib 2023-06-09 14:44:37 +03:00
Andrew Mayorov
8919a6ef93
refactor(pluglib): provide SQL related utils in emqx_utils_sql 2023-06-09 14:44:37 +03:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Andrew Mayorov
7d0abb6146
feat(emqx): add emqx_topic:match_any/2 utility 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app 
Also make user that existing code calls it directly.
 2023-06-09 14:44:36 +03:00
Serge Tupchii
2aad6671ac fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file 
As we use `keep_tables` option, we don't need to modify a backup schema before importing the backup file.
With this option, `mnesia:restore/2` ignores backup schema and keeps current table schema unchanged.
 2023-06-09 14:41:21 +03:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup 
feat: implement configuration and user data export/import
 2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
Kjell Winblad
4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631 
feat: add sync/async option to the Kafka producer bridge
 2023-06-09 12:51:23 +02:00
Kjell Winblad
2671e8ecf9 fix: dialyzer type problem 2023-06-09 11:00:05 +02:00
Ilya Averyanov
c2635e938b
Merge pull request #10974 from keynslug/ft/EMQX-9523/backend-enable-flag 
fix(ft): add `enable` flag to every backend
 2023-06-09 11:44:37 +03:00
JianBo He
53eb8d7f1b test: fix flaky tests introduced by https://github.com/emqx/emqx/pull/10948 2023-06-09 15:48:23 +08:00
JianBo He
72311a546b chore: more clear funcation name 2023-06-09 15:48:23 +08:00
JianBo He
6a05663bd5 chore: ensure default value's type be the first one in the union type 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-06-09 15:48:23 +08:00
JianBo He
dbb8742e0e test: cover max_connection in infinity value 2023-06-09 15:48:23 +08:00
JianBo He
8df7b1a1be fix(gateway): fix http-api 500 issue if setting max_connections to infinity 2023-06-09 15:48:23 +08:00
JianBo He
c450a1784a
Merge pull request #10950 from HJianBo/mqttsn-qos3 
fix(mqttsn): checking enable_qos3 option
 2023-06-09 15:46:55 +08:00
JianBo He
2153cd4317
Merge pull request #10979 from HJianBo/fix-coap-connectionless-pub 
fix(coap): fix crash publish/subscribe in connectionless mode
 2023-06-09 15:46:18 +08:00
Zaiming (Stone) Shi
38e767b808 fix(schema): map new duration types to dashboard's type name 2023-06-09 09:14:55 +02:00
Stefan Strigler
2e9ac8f1ec
Merge pull request #10980 from sstrigler/EMQX-10189-mark-support-for-json-output-format-in-prometheus-stats-as-deprecated 
Mark support for json output format in prometheus stats as deprecated
 2023-06-09 09:00:34 +02:00
Zaiming (Stone) Shi
ee697831f6
Merge pull request #10926 from zmstone/0602-add-enable-as-an-alias-for-enabled 
0602 add enable as an alias for enabled
 2023-06-09 08:44:20 +02:00
Zaiming (Stone) Shi
9f6beee6b3
Merge pull request #10936 from zmstone/0603-imporove-desc 
docs: refine some desc
 2023-06-09 08:44:03 +02:00
Zaiming (Stone) Shi
af5c6720de Merge remote-tracking branch 'origin/master' into release-51 2023-06-09 08:37:29 +02:00
lafirest
601db2b1d3
Merge pull request #10988 from lafirest/fix/konservi_sentema 
fix(resource): improve log security when resource creation fails
 2023-06-09 13:46:12 +08:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
zhongwencool
daa5450218
Merge pull request #10985 from zmstone/0608-rename-cluster-call-cmd 
0608 rename cluster call cmd
 2023-06-09 10:56:18 +08:00
zhongwencool
4835c30f8c
Merge pull request #10975 from zhongwencool/conf-docs 
fix: link different document between community and enterprise
 2023-06-09 09:50:45 +08:00
JianBo He
c733ddb166
Merge pull request #10977 from HJianBo/fix-stomp-bugs 
Fix stomp gateway subscription_cnt is not updated timely and max_headers option is not working
 2023-06-09 09:47:26 +08:00
zhongwencool
c9cc06b6b1 chore: compile failed 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi
83c36a3c9f refactor(emqx_conf_cli): do not print hidden roots in 'conf show' cmd 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi
31ab973b30 refactor(emqx_conf_cli): rename cluster_call ctl command 
Renamed 'clustedr_call' ctl command to 'cluster_sync' as a sub-command
of 'conf'
The old 'cluster_call' command is kept, but no usage prints
 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi
96a41ac6bd feat(emqx_ctl): support hidden commands 
hidden commands do not deserve a place in the usage page
 2023-06-09 09:31:32 +08:00
Zaiming (Stone) Shi
47f826f99a chore: fix typo 2023-06-09 09:31:32 +08:00
Serge Tupchii
53ec8326b0 fix(emqx_listeners): convert authentication certs 2023-06-08 20:44:18 +03:00
Serge Tupchii
33fa053b9b test(emqx_common_test_helpers): keep EMQX ENV vars on a child node for its lifetime 2023-06-08 20:42:51 +03:00
Serge Tupchii
5f526d548a style: use double percent (%%) comments 2023-06-08 20:42:51 +03:00
Kjell Winblad
1c7834e056 fix: fixes due to comments from @zmstone 2023-06-08 16:47:02 +02:00
Kjell Winblad
cb3a5fdbd4 style: only callback modules should do dynamic calls 2023-06-08 16:27:04 +02:00
Kjell Winblad
d524f8c805 refactor: rename config parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad
6f2271e9f0 test: add Kafka producer bridge test case for query_mode parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad
ed9e29e769 refactor: refacor query_mode detection code 
This commit refactor the query_mode resource detection code according to
a suggestion from @zmstone. This commit should not contain any
functional change except for a change of the Kafka producer bridge
config.
 2023-06-08 16:26:55 +02:00
Kjell Winblad
273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609 
fix: friendly error message when creating bridges with too long names
 2023-06-08 15:46:42 +02:00
JianBo He
034caaf20f
Merge pull request #10941 from HJianBo/enhance-prometheus-sampe-perf 
perf(prometheus): udpate the prometheus dep vsn
 2023-06-08 21:34:50 +08:00
Ivan Dyachkov
abbba71191 fix(tls): issue when ssl listner is configured to use tls v1.3 only 
clients could not connect due to incompatible tls options if ssl listner is configured to use tls v1.3 only
 2023-06-08 15:13:22 +02:00
Zaiming (Stone) Shi
7d2ea3dc76 docs: refine some desc 2023-06-08 13:58:31 +02:00
Andrew Mayorov
ce8cc05cc8
chore(ftschema): mention translate is for tests purpose only 2023-06-08 14:54:14 +03:00
Andrew Mayorov
e3c16be3c4
fix(ftschema): ensure translate follow emqx_config behavior 
So that atom keys won't unexpectedly get to converters / validators.
 2023-06-08 14:54:14 +03:00
Andrew Mayorov
ae97d516b8
fix(ft): add enable flag to every backend 
And tolerate multiple configured backends, as long as only one of
them is enabled.
 2023-06-08 14:54:13 +03:00
Ivan Dyachkov
b4d081f424
Merge pull request #10978 from id/0608-fix-404-in-rule-engine-api 
fix: missing 404 response in rule engine api schema
 2023-06-08 13:20:02 +02:00
Kjell Winblad
47fa17b3c1 feat: add sync/async option to the Kafka producer bridge 
This commit makes it possible to configure if a Kafka bridge should work
in query mode sync or async by setting the resource_opts.query_mode
configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-8631
 2023-06-08 13:16:06 +02:00
Stefan Strigler
f5e93b1b2b docs(emqx_prometheus): leave remark about json being deprecated 2023-06-08 13:08:45 +02:00
Zaiming (Stone) Shi
b481994e74 chore: bump to version 5.1.0-alpha.4 2023-06-08 12:30:35 +02:00
Zaiming (Stone) Shi
c1cf2365c2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-08 12:30:02 +02:00
JianBo He
95f0745e31 fix(coap): fix crash publish/subscribe in connectionless mode 
this bug was introduced by https://github.com/emqx/emqx/pull/10871
 2023-06-08 18:08:11 +08:00
Ivan Dyachkov
711cedc282 fix: missing 404 response in rule engine api schema 2023-06-08 11:46:16 +02:00
JianBo He
4065f08083 fix(stomp): fix frame limitation is not working 2023-06-08 16:57:12 +08:00
JianBo He
bad0c35bb9 fix(stomp): ensure the subscripton_cnt timely updated 2023-06-08 15:25:44 +08:00
Zaiming (Stone) Shi
641166a67a
Merge pull request #10971 from zmstone/0607-merge-master-to-release-51 
0607 merge master to release 51
 2023-06-08 09:22:21 +02:00
JianBo He
dc01b7109e chore: update prometheus vsn and grafana template 2023-06-08 14:29:54 +08:00
JianBo He
e0ec050e34 chore: update prometheus vsn 2023-06-08 14:29:54 +08:00
JianBo He
f53e13b8b9 perf: update to our forked prometheus deps 
it's based on v4.10.0 of upstream, and patched this PR:
https://github.com/emqx/prometheus.erl/pull/5
 2023-06-08 14:29:54 +08:00
Ilya Averyanov
b9f1a70214
Merge pull request #10967 from savonarola/0607-rebalance-fixes 
Fix rebalance issues
 2023-06-08 08:29:37 +03:00
zhongwencool
0116c53395 fix: link different document between community and enterprise 2023-06-08 11:33:39 +08:00
JianBo He
d5ebf5650f test: improve the test 2023-06-08 11:24:07 +08:00
JianBo He
fcef456174 feat: add live_connections field for some HTTP APIs 
i.e:
- `/monitor_current`, `/monitor_current/nodes/{node}`
- `/monitor/nodes/{node}`, `/monitor`
- `/node/{node}`, `/nodes`
 2023-06-08 11:24:06 +08:00
JianBo He
5d86443910 chore: fix dialyzer warnings 2023-06-08 11:06:25 +08:00
JianBo He
d0222ae79d chore: ensure test 2023-06-08 10:23:59 +08:00
JianBo He
4e9eeb489d
Merge pull request #10969 from thalesmg/refactor-gcp-pubsub-on-stop-v50 
refactor(gcp_pubsub): always cleanup resources when calling `on_stop`
 2023-06-08 09:47:19 +08:00
JianBo He
0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple 
feat: refactored some bridges to avoid leaking resources part.2
 2023-06-08 09:37:31 +08:00
Zaiming (Stone) Shi
641aca00d8
Merge pull request #10954 from zhongwencool/flapping-detect-conf 
feat: don't hide enable in flapping detect conf
 2023-06-07 23:04:56 +02:00
Zaiming (Stone) Shi
ccd2589ff2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-07 21:43:29 +02:00
Ilya Averyanov
7f2de66dab fix(rebalance): add wait_health_check timeout to node evacuation 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-06-07 21:37:01 +03:00
Andrew Mayorov
39b0d41527
fix(bpapi): ignore dropped BPAPI module 
It doesn't really breaks interoperability.
 2023-06-07 21:33:47 +03:00
Zaiming (Stone) Shi
aecea50564
Merge pull request #10902 from zmstone/0529-donot-copy-cluster-conf-from-newer-version 
0529 donot copy cluster conf from newer version
 2023-06-07 20:32:11 +02:00
Zaiming (Stone) Shi
1e0b2de0ea chore: refine data sync debug log messages 2023-06-07 20:19:48 +02:00
Zaiming (Stone) Shi
ed1ad4e684 chore(emqx_schema): add 'eanble' as an alias to 'enabled' fields 2023-06-07 20:19:48 +02:00
Thales Macedo Garitezi
d9aac63f82 refactor(gcp_pubsub): always cleanup resources when calling on_stop 
Fixes https://emqx.atlassian.net/browse/EMQX-9934

The way that this connector was structured already depended only on the resource ID.  This
PR just makes it completely independent of the resource state term, and also treats an
already absent pool as a success when stopping it.
 2023-06-07 14:34:22 -03:00
Thales Macedo Garitezi
105c26afb5
Merge pull request #10964 from thalesmg/fix-hocon-compact-errors-v50 
fix(schema): avoid `function_clause` error when compacting errors
 2023-06-07 14:00:24 -03:00
Andrew Mayorov
79d430cf03
refactor(pluglib): move emqx_plugin_libs_proto_v1 into emqx app 
And rename it to `emqx_metrics_proto_v1` in the process.
 2023-06-07 18:30:12 +03:00
Zaiming (Stone) Shi
622ac4d195
fix(emqx_schema): binary for string default value 2023-06-07 17:14:10 +02:00
JimMoen
4ea3483083
perf(topic): use tail recursive to joining a topic 2023-06-07 22:59:53 +08:00
JimMoen
b0e18f5e75
fix(shared-sub): validate share topic error send DISCONNECT 2023-06-07 22:59:52 +08:00
JimMoen
070e410c69
refactor: variable and func rename 2023-06-07 22:59:52 +08:00
JimMoen
ea81a924f1
refactor: move topic() types def in emqx_types.erl 2023-06-07 22:59:52 +08:00
JimMoen
65483e972e
refactor: use unified table define 2023-06-07 22:51:44 +08:00
zhongwencool
70e800f8a4 chore: delete unuse default value and make flapping_detect middle improtance 2023-06-07 22:50:11 +08:00
zhongwencool
5659cc2b47 feat: add converter to flapping_detect 2023-06-07 21:54:32 +08:00
zhongwencool
1d33b7dbd8 chore: make static check happy 2023-06-07 21:54:32 +08:00
zhongwencool
535870386a fix: zones update api failed 2023-06-07 21:54:32 +08:00
zhongwencool
d58506a7c3 feat: always update default zones 2023-06-07 21:54:32 +08:00
zhongwencool
48381d4c86 feat: refactor flapping detect conf 2023-06-07 21:54:32 +08:00
Thales Macedo Garitezi
cc8631223e fix(schema): avoid function_clause error when compacting errors 
Fixes https://emqx.atlassian.net/browse/EMQX-10168

The bridge probe API displayed the typecheck errors for the new
timeout duration types correctly, but when an user tried to create the
bridge anyway a `function_clause` error was raised when trying to
compact hocon errors:

```
09:47:19.045 [warning] [exception: :error, path: '/bridges', reason: {:case_clause, {:error, {:config_update_crashed, :function_clause}}}, stacktrace: [{:emqx_bridge_api, :create_or_update_bridge, 4, [file: '/home/thales/dev/emqx/emqx/apps/emqx_bridge/src/emqx_bridge_api.erl', line: 602]}, {:minirest_handler, :apply_callback, 3, [file: '/home/thales/dev/emqx/emqx/deps/minirest/src/minirest_handler.erl', line: 111]}, {:minirest_handler, :handle, 2, [file: '/home/thales/dev/emqx/emqx/deps/minirest/src/minirest_handler.erl', line: 44]}, {:minirest_handler, :init, 2, [file: '/home/thales/dev/emqx/emqx/deps/minirest/src/minirest_handler.erl', line: 27]}, {:cowboy_handler, :execute, 2, [file: '/home/thales/dev/emqx/emqx/deps/cowboy/src/cowboy_handler.erl', line: 41]}, {:cowboy_stream_h, :execute, 3, [file: '/home/thales/dev/emqx/emqx/deps/cowboy/src/cowboy_stream_h.erl', line: 318]}, {:cowboy_stream_h, :request_process, 3, [file: '/home/thales/dev/emqx/emqx/deps/cowboy/src/cowboy_stream_h.erl', line: 302]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]]
```

This fixes the issue so that both APIs return more friendly error messages.
 2023-06-07 10:34:58 -03:00
Paulo Zulato
a4407764f3 feat(connection): configurable TCP keepalive 
Fixes https://emqx.atlassian.net/browse/EMQX-9852
 2023-06-07 10:23:38 -03:00
Thales Macedo Garitezi
f9ff1007a0
Merge pull request #10944 from thalesmg/pubsub-jwt-on-demand-v50 
feat(gcp_pubsub): generate jwt tokens on demand without workers (5.1)
 2023-06-07 09:22:09 -03:00
Ilya Averyanov
d1a1e8041d fix(rebalance api): fix error message formatting 2023-06-07 13:57:16 +03:00
Andrew Mayorov
c05f462d0a
fix(tlslib): bring back deterministic filenames for managed certs 2023-06-07 13:32:35 +03:00
William Yang
c1be86c5ec
Merge pull request #10931 from qzhuyan/perf/william/save-times-of-maps-build 
Perf/william/save times of maps build
 2023-06-07 10:32:45 +02:00
SergeTupchiy
3c482e9aa6
Merge pull request #10953 from SergeTupchiy/fix-listener-authn-update 
fix(emqx_listeners): fix listener authentication create/update
 2023-06-07 11:12:41 +03:00
Zaiming (Stone) Shi
cb3e787cb1 Merge remote-tracking branch 'origin/master' into 0529-donot-copy-cluster-conf-from-newer-version 2023-06-07 09:14:04 +02:00
Zaiming (Stone) Shi
ce4e49cd9e
Merge pull request #10943 from zmstone/0605-hide-mcast-cluster-discovery 
chore: start deprecating mcast cluster discovery
 2023-06-07 08:32:33 +02:00
JianBo He
81387800b5
Merge branch 'master' into mqttsn-qos3 2023-06-07 09:49:29 +08:00
JianBo He
21b451a4c2
Merge pull request #10951 from HJianBo/mqttsn-mountpoint-pub 
fix(mqttsn): make mountpoint works for publish
 2023-06-07 09:39:46 +08:00
Andrew Mayorov
32a8963582
Merge pull request #10927 from keynslug/fix/EMQX-9930/tls-cert-gc 
feat(tlslib): add separate managed certfiles GC process
 2023-06-06 20:12:18 +03:00
Thales Macedo Garitezi
260fae296b feat(gcp_pubsub): generate jwt tokens on demand without workers (5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-9603

Rather than relying on a JWT worker to produce and refresh tokens, we
could just produce then on demand when pushing the messages to GCP
PubSub.  That can generate a bit of extra work (as multiple processes
might realize it’s time to refresh the JWT and do so), but that
shouldn’t be much.  In return, we avoid any possibility of not having
a fresh JWT when pushing messages.
 2023-06-06 13:19:24 -03:00
Serge Tupchii
bf2b5a9f24 fix(emqx_listeners): fix listener authentication create/update 
Listener authentication is a list of authenticators, so each of them must be created
or updated individually.
 2023-06-06 19:00:35 +03:00
Serge Tupchii
1968589f81 fix(emqx_schema): don't allow enabling fail_if_no_peer_cert if verify_none is set 
Setting `fail_if_no_peer_cert = true` and `verify = verify_none` causes connection errors.

Closes: EMQX-9586
 2023-06-06 17:20:22 +03:00
Thales Macedo Garitezi
2c97ee1b3f test(conf_app): fix test after common test helpers changes 2023-06-06 09:32:38 -03:00
Thales Macedo Garitezi
97a9bb484a test(pulsar_producer): attempt to fix flaky test 2023-06-06 09:32:38 -03:00
Thales Macedo Garitezi
770dd188b1
Merge pull request #10947 from emqx/release-51 
Release 51 to master
 2023-06-06 09:32:16 -03:00
JianBo He
bcc47442eb fix(mqttsn): make mountpoint works for publish 2023-06-06 18:44:27 +08:00
Andrew Mayorov
68cd0c47f5
Merge pull request #10945 from fix/EMQX-9967/tmpfn-enametoolong 
fix(ft): ensure temp filenames are under 255 bytes long
 2023-06-06 13:34:47 +03:00
JianBo He
26748ef242 fix(mqttsn): checking enable_qos3 option 2023-06-06 18:21:26 +08:00
Andrew Mayorov
d5aedaac7d
fix(tlslib): append random suffix to managed certfiles 
In order to lessen the chance of ambiguity in determining collectable
certfiles during GC.
 2023-06-06 11:21:03 +03:00
Andrew Mayorov
dcd59e4f1b
fix(ft): set more conservative filename length limit 
Otherwise, local fs exporter will have a hard time preserving the
filemeta, because its filename is even 13 bytes longer.
 2023-06-06 10:40:20 +03:00
Andrew Mayorov
69b98c1830
fix(ft): ensure temp filenames are under 255 bytes long 2023-06-06 10:38:31 +03:00
某文
47d7e6ce01 feat: update gateway from cli 2023-06-06 08:12:47 +08:00
zhongwencool
ddb4a09a3d
Merge pull request #10935 from zhongwencool/listeners-conf-update 
feat: update listeners from cli
 2023-06-06 08:10:45 +08:00
Zaiming (Stone) Shi
3caf114f68 test: fix test case is_list(Config) function guard 2023-06-05 23:00:41 +02:00
Zaiming (Stone) Shi
bb1cf9beaa Merge remote-tracking branch 'origin/master' into 0529-donot-copy-cluster-conf-from-newer-version 2023-06-05 22:56:58 +02:00
Andrew Mayorov
f7a6648103
fix(mqttconn): no warn if ingress poolsize is same as config 2023-06-05 23:32:13 +03:00
Andrew Mayorov
750b7158d4
fix(tlsgc): make computing orphans safe against symlinking 
Before this commit, `orphans/1` considered managed file an orphan based
on its absolute filename, which is not safe when, for example, emqx has
a symlink configured as `data_dir` while config references certfiles
through realpaths.
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
31d38e18c9
refactor(tlsgc): simplify fold_config/4 2023-06-05 22:41:08 +03:00
Andrew Mayorov
f8e8b7993b
fix(tlsgc): anticipate only binaries as string in raw config 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
468fd98173
test(utilfs): add few more traverse_dir/3 testcases 2023-06-05 22:41:08 +03:00
Andrew Mayorov
3c2a7dbadc
fix(tlsgc): consolidate conf keypaths knowledge in emqx_tls_lib 
So that this GC mechanism will be easier to maintain.
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
ec06850bd0
test(auth): drop obsolete testcase altogether 
Most of the stuff that was tested by this test case is now covered
by `emqx_tls_certfile_gc_SUITE`.
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
206d0472e0
fix(emqx): avoid interference in testsuites 
Also remove that `?wait_async_action` in `emqx_trace_SUITE` which
consistently becomes stuck for 100 seconds. It's not clear why it
was there in the first place because listeners should start
synchronously, so `emqx_common_test_helpers:start_apps/1` won't return
until required listeners are started.
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
8a31e5639b
fix(emqx): ensure that standalone build works 
Turns out rebar3 `path` resource has troubles with dangling
symlinks and non-ASCII filenames. 🥲
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
1cb226dffb
chore: bump applications versions 
* emqx_authz 0.1.22
* emqx_exhook 5.0.13
* emqx_utils 5.0.3
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
afbf8cb32f
chore(util): drop obsolete stuff from emqx_utils_maps 2023-06-05 22:41:07 +03:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process 
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys 
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
 2023-06-05 22:41:07 +03:00
Zaiming (Stone) Shi
67db9d6fe9 chore: bump version to 5.1.0-alpha.3 2023-06-05 20:25:48 +02:00
Thales Macedo Garitezi
75df622426
Merge pull request #10930 from thalesmg/max-int-timeout-v50 
check maximum timeout values in schema (5.1)
 2023-06-05 12:51:36 -03:00
Kjell Winblad
375661c6a1 test: add test case for error when bridge name is too long 2023-06-05 17:02:08 +02:00
Thales Macedo Garitezi
46393343e2 chore: use timeout_duration types for timer fields 
Fixes https://emqx.atlassian.net/browse/EMQX-10020
 2023-06-05 11:46:38 -03:00
Thales Macedo Garitezi
9bd37937f1 feat(types): add timeout_duration{,_ms,_s} types 2023-06-05 11:46:32 -03:00
Paulo Zulato
8d97a85b3c
Merge pull request #10893 from paulozulato/fix-oracle-error-handling 
fix(oracle): fix error handling on sync query
 2023-06-05 10:48:22 -03:00
Zaiming (Stone) Shi
bdcc069aac chore: start deprecating mcast cluster discovery 2023-06-05 14:58:53 +02:00
JimMoen
0f808345e6
Merge pull request #10924 from JimMoen/refactor_influxdb_on_stop 
feat: refactor influxdb connector to to avoid resources leaking
 2023-06-05 18:02:15 +08:00
lafirest
d51c658a30
Merge pull request #10908 from lafirest/feat/rocketmq_on_stop 
feat(rocketmq): refactored bridge to avoid leaking resources during crashes at creation
 2023-06-05 15:00:32 +08:00
JianBo He
dbc0cdce67 chore: dont allocate resource for simple connectiors 2023-06-05 10:59:00 +08:00
firest
5921ed3d2e fix(rocketmq): improve function name 2023-06-05 10:43:28 +08:00
JianBo He
e717ddafd7 chore: update changes 2023-06-05 10:19:46 +08:00
JianBo He
3739230435 feat: refactor connectors on_stop function to avoid resources leaking 
Supplement to https://github.com/emqx/emqx/pull/10895
 2023-06-05 10:19:46 +08:00
JianBo He
e07c86b6a8
Merge pull request #10909 from HJianBo/remove-deprecated-gateways 
feat: remove the deprecated '/gateway/*' HTTP APIs
 2023-06-05 09:59:50 +08:00
某文
d0d6992a14 feat: update listeners from cli 2023-06-04 20:07:33 +08:00
某文
03160ef599 test: add more test for update_config 2023-06-04 12:02:02 +08:00
某文
bd29433997 feat: support emqx_conf:update([exhook],Conf) 2023-06-04 09:30:17 +08:00
JimMoen
a174d5741b
feat: refactor influxdb connector to to avoid resources leaking 2023-06-04 02:21:49 +08:00
Zaiming (Stone) Shi
f469b31fee Merge remote-tracking branch 'origin/master' into 0529-donot-copy-cluster-conf-from-newer-version 2023-06-03 11:21:22 +02:00
Paulo Zulato
c1d935fd34
Merge pull request #10917 from paulozulato/block-oracle-default-database-connection 
test(oracle): add match for reason field in a test case
 2023-06-02 18:16:12 -03:00
Andrew Mayorov
5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests 
chore(bridge): drop deprecated `mode` setting from examples / tests
 2023-06-02 22:46:31 +03:00
Zaiming (Stone) Shi
0b87ea4cdc
Merge pull request #10929 from zmstone/0602-ci-pin-otp-25.3.2-1-elixir-1.14.5 
ci: upgrade to otp 25.3.2-1 and elixir 1.14.5
 2023-06-02 21:46:08 +02:00
Zaiming (Stone) Shi
aa46c0a484
Merge pull request #10928 from zmstone/0602-sync-release-51-to-master 
0602 sync release 51 to master
 2023-06-02 21:45:21 +02:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50 
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
 2023-06-02 16:20:36 -03:00
Andrew Mayorov
09290f8d61
chore(bridge): drop deprecated mode setting from examples / tests 2023-06-02 20:31:31 +03:00
Zaiming (Stone) Shi
e6fb27b3c7 fix(emqx_telemetry): make dialyzer happy 2023-06-02 18:02:46 +02:00
Zaiming (Stone) Shi
1ba07e1040 Merge remote-tracking branch 'origin/release-51' into 0529-donot-copy-cluster-conf-from-newer-version 2023-06-02 16:30:58 +02:00
Zaiming (Stone) Shi
28c564d15b Merge remote-tracking branch 'origin/master' into release-51 2023-06-02 16:25:48 +02:00
William Yang
b1c188f9b1 perf(mqtt-caps): dont filter cap keys 
save map builds
 2023-06-02 16:24:52 +02:00
William Yang
4961aca3d0 perf(mqtt-caps): save some map builds 2023-06-02 16:19:58 +02:00
Thales Macedo Garitezi
74ffd9ef96
Merge pull request #10918 from thalesmg/fix-ecpool-empty-unrecoverable-v50 
fix(ecpool,bridge): treat `{error, ecpool_empty}` as a retriable error
 2023-06-02 09:21:00 -03:00
Zaiming (Stone) Shi
367fc95976
Merge pull request #10923 from zmstone/0602-fix-channel-data-registration-race-condition 
fix(emqx_cm): fix channel data registration race-condition
 2023-06-02 14:20:18 +02:00
Thales Macedo Garitezi
0790c88aaf
refactor: use default's type as first union member 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-06-02 09:08:11 -03:00
Thales Macedo Garitezi
2d7c1da901
Merge pull request #10913 from thalesmg/fix-plugin-proto-multicall-v50 
fix(plugins): use `emqx:running_nodes` for multicall operations
 2023-06-02 09:03:42 -03:00
Zaiming (Stone) Shi
c75e9bbe0d fix(emqx_cm): fix channel data registration race-condition 
when clustered, there are chances the a mqtt client process
get killed (e.g. holding the channel registeration lock for too long),
if the channel data inserts happen before casting out the message
for channel process monitoring, there is a chance for the
stale message left in the ets tables indefinitely.

this commit changes the order of the non-atomic operations:
it casts out the monitor request message before inserting
channel data.
 2023-06-02 11:43:35 +02:00
Stefan Strigler
c9ef1ccacd fix: headers is array not object 2023-06-02 10:24:33 +02:00
Stefan Strigler
c7f177054c fix(emqx_authn): verify_claims is list(), therefor default must match 2023-06-02 10:24:33 +02:00
Stefan Strigler
15c7c281ca fix: don't convert empty list default 2023-06-02 10:03:01 +02:00
Stefan Strigler
190e44f8be fix: brackets not allowed in swagger schema for component names 2023-06-02 10:03:01 +02:00
Stefan Strigler
8fdc9de9e9 fix: empty properties as empty map so it ends up as a json object 2023-06-02 10:03:01 +02:00
Stefan Strigler
e62bd63199 fix: don't add summary, remove it instead 2023-06-02 10:03:01 +02:00
Stefan Strigler
ef2581d13b fix: remove unsupported schema properties 2023-06-02 10:03:01 +02:00
Stefan Strigler
12a4e48727 fix: add definition for bearerAuth 2023-06-02 10:03:01 +02:00
zhongwencool
8d8efe449e
Merge pull request #10896 from zhongwencool/api-keys-hot-conf-update 
feat: api_key conf support hot conf
 2023-06-02 15:58:45 +08:00
zhongwencool
dd85c981cd
Merge pull request #10783 from zhongwencool/improve-authn 
feat: load changes to the authentication configuration from command line
 2023-06-02 15:58:13 +08:00
zhongwencool
9d8a5716ec
Merge pull request #10822 from zhongwencool/authz-improve 
feat: support emqx_config:update([authrization],Conf) update
 2023-06-02 15:43:09 +08:00
lafirest
0f8824b40f
Merge pull request #10921 from lafirest/test/rocketmq_acl 
test(rocketmq): add an ACL test case
 2023-06-02 15:37:46 +08:00
JianBo He
5e0a3606ad chore: fix outdate tests 2023-06-02 13:30:03 +08:00
JianBo He
0633d894b8 test: fix failed tests 2023-06-02 13:30:03 +08:00
JianBo He
ec0049af7b feat: remove the deprecated '/gateway/*' HTTP APIs 2023-06-02 13:30:03 +08:00
firest
ea9b1e13d5 test(rocketmq): add an ACL test case 2023-06-02 11:19:06 +08:00
Paulo Zulato
fb455d68a3 fix(oracle): fix error handling on sync query 
Fixes https://emqx.atlassian.net/browse/EMQX-10075
 2023-06-01 20:50:29 -03:00
某文
367a0e312a chore: remove config_path() function 2023-06-02 07:09:08 +08:00
Andrew Mayorov
76e5243211
Merge pull request #10901 from fix/EMQX-9985/ft-fin-checksum 
fix(ft): respect checksum in `fin` packets
 2023-06-01 23:49:55 +03:00
Thales Macedo Garitezi
0072f4a598 fix(ecpool,bridge): treat {error, ecpool_empty} as a retriable error 
Example from flaky CI test:

```
/emqx/apps/emqx_resource/src/emqx_resource_buffer_worker.erl:716 2023-06-01T19:30:37.119862+00:00 [buffer_worker_flush_ack] #{batch_or_query => [{query,undefined,{send_message,#{payload => <<"-576460752303420927">>,timestamp => 1668602148000,topic => <<"t_write_failure">>}},false,-576460614954536809}],queue_count => 0,result => {async_return,{error,{unrecoverable_error,ecpool_empty}}},'~meta' => #{gl => <0.3883.5>,node => 'test@127.0.0.1',pid => <0.4724.5>}}.
```
 2023-06-01 17:13:06 -03:00
Paulo Zulato
92c72c6994 test(oracle): add match for reason field in a test case 
Fixes https://emqx.atlassian.net/browse/EMQX-9980
 2023-06-01 15:12:09 -03:00
Paulo Zulato
364601c3aa
Merge pull request #10892 from paulozulato/block-oracle-default-database-connection 
fix(oracle): require either sid or service name
 2023-06-01 15:04:03 -03:00
Thales Macedo Garitezi
7d3ce14835 test: rm unused var warning 2023-06-01 14:00:43 -03:00
Thales Macedo Garitezi
648b8231ef fix(plugins): avoid crashing when fetching error reason 2023-06-01 14:00:43 -03:00
Thales Macedo Garitezi
64258120d7 fix(plugins): use emqx:running_nodes for multicall operations 
Fixes https://emqx.atlassian.net/browse/EMQX-10079
 2023-06-01 14:00:43 -03:00
某文
2e5401f3cb fix: delete rule_engine failed 2023-06-02 00:30:07 +08:00
Thales Macedo Garitezi
3e4790edd4 test(pulsar_producer): fix flaky test 2023-06-01 13:01:58 -03:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl 
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
 2023-06-01 13:01:53 -03:00
某文
c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Stefan Strigler
9c76aa901e fix(emqx_mgmt_api_clients): hide set_keepalive_seconds from swagger 2023-06-01 16:57:19 +02:00
某文
94b86f01a5 chore: reformat SUITE case 2023-06-01 22:32:31 +08:00
zhongwencool
21df321d89
Merge branch 'master' into improve-authn 2023-06-01 22:24:59 +08:00
Thales Macedo Garitezi
f42ccb6262 feat(resource): increase default request timeout to 45 s 
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
 2023-06-01 11:20:06 -03:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval 
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
 2023-06-01 11:20:06 -03:00
Stefan Strigler
e011b5532f
Merge pull request #10881 from sstrigler/EMQX-9355-5-0-x-test-coverage-80 
IoTDB test coverage
 2023-06-01 15:48:56 +02:00
Thales Macedo Garitezi
ecbb6e0ecb test(global_gc): fix flaky test 2023-06-01 10:48:02 -03:00
Kjell Winblad
75ff76a16b fix: friendly error message when creating bridges with too long names 
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
 2023-06-01 15:30:06 +02:00
Thales Macedo Garitezi
7740987a17
Merge pull request #10905 from thalesmg/fix-enforce-binary-rootkey-config-v50 
fix(config): enforce root key name to be a binary (raw config)
 2023-06-01 10:27:48 -03:00
William Yang
c1d9bc8033 perf(emqx_mqtt_caps): direct read mqtt from zone conf 2023-06-01 14:54:44 +02:00
William Yang
923913e15c feat(config): merge with global defaults when put new zone 2023-06-01 14:54:44 +02:00
William Yang
a2f8e87389 test(config): default zone and global defaults 2023-06-01 14:54:14 +02:00
William Yang
e693c6ec54 test: fix emqx_takeover_SUITE 2023-06-01 14:52:33 +02:00
William Yang
8c26ee75db test: remove hardcoded configs 
test with schema defaults
 2023-06-01 14:52:33 +02:00
William Yang
4416122cf3 test: fix longname/shortname 2023-06-01 14:52:33 +02:00
William Yang
8ee78ad6b4 test: fix emqx_mgmt_api_configs_SUITE 2023-06-01 14:52:33 +02:00
William Yang
f621bf1f8e test(config): update emqx_client_SUITE 2023-06-01 14:52:33 +02:00
William Yang
98d6ed4251 test: fix emqx_session_SUITE 2023-06-01 14:52:33 +02:00
William Yang
6c1671651c test: use zone default in emqx_eviction_agent_channel_SUITE 2023-06-01 14:52:33 +02:00
William Yang
88f9782db0 perf(config): read zone conf once 2023-06-01 14:52:30 +02:00
firest
232ef23a48 feat(rocketmq): refactored bridge to avoid leaking resources during crashes at creation 2023-06-01 18:49:45 +08:00
Zhongwen Deng
7eea693422 fix: reorder authn when updating 2023-06-01 17:24:39 +08:00
Zhongwen Deng
96ee38ce15 chore: bump up authn/ctl app version 2023-06-01 17:24:39 +08:00
Zhongwen Deng
4f5d5216bb test: bad authn handler callback module in SUITE 2023-06-01 17:24:39 +08:00
Zhongwen Deng
8f12d81878 feat: support emqx_ctl conf command 2023-06-01 17:24:39 +08:00
Zhongwen Deng
b290d2543b fix: don't crash when command_ctl table not init 2023-06-01 17:24:39 +08:00
Zaiming (Stone) Shi
feecd53436 chore: bump to version 5.1.0-alpha.2 for both ce and ee 2023-06-01 10:13:56 +02:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id 
fix(ruleeng): parse bridge id on create / update
 2023-06-01 10:21:26 +03:00
JianBo He
bcc5e30f22
Merge pull request #10895 from lafirest/fix/refactor_on_stop 
feat: refactored some bridges to avoid leaking resources during crashes at creation
 2023-06-01 15:17:37 +08:00
Zaiming (Stone) Shi
760de7becf chore: imporve logging message 2023-06-01 08:26:53 +02:00
Zaiming (Stone) Shi
15ffa98bc5 chore: fix a typo in test case function name 2023-06-01 08:18:26 +02:00
Zaiming (Stone) Shi
cb2be1811e docs: fix function doc annotation 2023-06-01 08:17:36 +02:00
firest
cb31032e6d chore: make dialyzer && test happy 2023-06-01 11:15:51 +08:00
Thales Macedo Garitezi
f83f112189 fix(config): enforce root key name to be a binary (raw config) 
Fixes this flaky test:
https://github.com/emqx/emqx/actions/runs/5134910014/jobs/9239826162#step:8:1299

```
=ERROR REPORT==== 31-May-2023::16:16:10.710969 ===
    exception: error
    key_path: [listeners,tcp,default,authentication]
    module: emqx_schema
    msg: change_config_crashed
    reason: {config_not_found,[<<"listeners">>,<<"tcp">>,<<"default">>,
                               <<"authentication">>]}
    stacktrace: [{emqx_utils_maps,deep_get,2,
                     [{file,
                          "/__w/emqx/emqx/source/apps/emqx_utils/src/emqx_utils_maps.erl"},
                      {line,49}]},
                 {emqx_config_handler,return_change_result,2,
                     [{file,
                          "/__w/emqx/emqx/source/apps/emqx/src/emqx_config_handler.erl"},
                      {line,456}]},
                 {emqx_config_handler,check_and_save_configs,7,
                     [{file,
                          "/__w/emqx/emqx/source/apps/emqx/src/emqx_config_handler.erl"},
                      {line,284}]},
                 {emqx_config_handler,handle_update_request,4,
                     [{file,
                          "/__w/emqx/emqx/source/apps/emqx/src/emqx_config_handler.erl"},
                      {line,195}]},
                 {emqx_config_handler,handle_call,3,
                     [{file,
                          "/__w/emqx/emqx/source/apps/emqx/src/emqx_config_handler.erl"},
                      {line,124}]},
                 {gen_server,try_handle_call,4,
                     [{file,"gen_server.erl"},{line,1149}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1178}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,240}]}]
    update_req: {{update,
                     {create_authenticator,'tcp:default',
                         #{backend => built_in_database,enable => true,
                           mechanism => password_based}}},
                 #{rawconf_with_defaults => true}}
```

Which was cause because there were both binary and atom `listeners`
keys in the `persistent_term` storage:

```erlang
%% persistent_term:get()
{{emqx_config,raw_conf,<<"listeners">>},
 #{<<"ssl">> =>
       #{<<"default">> => ...
...
{{emqx_config,raw_conf,listeners},#{}},
```

... and the atom one was winning when fetching the whole config with
`emqx_config:get_raw([])`.
 2023-05-31 18:03:58 -03:00
Paulo Zulato
3a6ad3b049 fix(oracle): require either sid or service name 
Fixes https://emqx.atlassian.net/browse/EMQX-9980
 2023-05-31 17:18:45 -03:00
Thales Macedo Garitezi
b7c72e6cfb
Merge pull request #10890 from thalesmg/test-flakiness-20230530-e 
more attempts to fix more test flakiness
 2023-05-31 16:07:18 -03:00
Zaiming (Stone) Shi
5146de5b1c feat: add a backup copies for cluster.hocon 2023-05-31 20:34:25 +02:00
Zaiming (Stone) Shi
332daabcc5 fix(config): Do not sync cluster config from nodes running new version 2023-05-31 20:34:25 +02:00
Zaiming (Stone) Shi
39d6f612ca feat(emqx_release): add new APIs to parse/compare release versions 2023-05-31 20:34:25 +02:00
Andrew Mayorov
aa98740280
Merge pull request #10903 from fix/ci-flaky-dashboard 
test(dashboard): fix inter-suite test flakiness
 2023-05-31 20:28:39 +03:00
Andrew Mayorov
f90ce9bbfe
test(dashboard): fix inter-suite test flakiness 2023-05-31 19:09:37 +03:00
Andrew Mayorov
0293b54211
fix(ft): respect checksum in fin packets 2023-05-31 18:50:38 +03:00
Thales Macedo Garitezi
189e2c87be test(channel): fix flaky test 
```
%%% emqx_channel_SUITE ==> t_handle_kicked_publish_will_msg: FAILED
%%% emqx_channel_SUITE ==> will_message_not_published
```

The problem was that the whole generated message was compared for
equality when receiving the will message, but the timestamp may be
different at publishing time...
 2023-05-31 12:42:24 -03:00
JianBo He
a3a6480f00
Merge pull request #10880 from HJianBo/batch-kickout-clients 
feat: support kickout clients in batch
 2023-05-31 22:36:35 +08:00
JianBo He
8ce0132569
Merge pull request #10871 from HJianBo/fix-coap-bugs 
fix(coap): to better handle coap requests in connection mode
 2023-05-31 22:36:15 +08:00
JianBo He
c67135dd4b
Merge pull request #10598 from HJianBo/refactor_exproto_stream 
refactor(exproto): support unary handler
 2023-05-31 22:36:01 +08:00
Kjell Winblad
036f180c27
Merge pull request #10878 from kjellwinblad/kjell/bridge/rabbitmq_password_log_leak/EMQX-9976 
fix: password log leak in RabbitMQ bridge
 2023-05-31 16:17:19 +02:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update 
Instead of parsing it every time in the `send_message/2` hot path.
 2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
77c9eda036 test: rm stray clear_screen calls 2023-05-31 11:07:44 -03:00
Stefan Strigler
1e4cee05df fix(emqx_bridge_iotdb): handle rule engine passed payload 
also remove topic logic as it's duplicated functionality via rule engine
 2023-05-31 15:57:22 +02:00
Thales Macedo Garitezi
83a7c800cc test(kafka_consumer): reduce flakiness 
The crash reason can also be `{error, not_found}` depending on
timing...

```
=ERROR REPORT==== 31-May-2023::12:17:31.461490 ===
    exception: error
    key_path: [bridges,kafka_consumer,
               't_resource_manager_crash_before_subscriber_started-576460752303419630']
    module: emqx_conf_schema
    msg: change_config_crashed
    reason: {badmatch,{error,not_found}}
    stacktrace: [{emqx_resource_manager,create_and_return_data,5,
                     [{file,
                          "/emqx/apps/emqx_resource/src/emqx_resource_manager.erl"},
                      {line,119}]},
                 {emqx_bridge_resource,create,4,
                     [{file,
                          "/emqx/apps/emqx_bridge/src/emqx_bridge_resource.erl"},
                      {line,171}]},
                 {emqx_bridge,'-perform_bridge_changes/2-fun-1-',5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,367}]},
                 {maps,fold_1,3,[{file,"maps.erl"},{line,411}]},
                 {emqx_bridge,perform_bridge_changes,2,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,354}]},
                 {emqx_bridge,post_config_update,5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,229}]},
                 {emqx_config_handler,call_post_config_update,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,398}]},
                 {emqx_config_handler,do_post_config_update,8,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,336}]},
                 {emqx_config_handler,check_and_save_configs,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,281}]},
                 {emqx_config_handler,handle_update_request,4,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,195}]},
                 {emqx_config_handler,handle_call,3,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,124}]},
                 {gen_server,try_handle_call,4,
                     [{file,"gen_server.erl"},{line,1149}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1178}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,240}]}]
    update_req: {{update,#{<<"authentication">> => <<"none">>,
                           <<"bootstrap_hosts">> =>
                               <<"toxiproxy.emqx.net:9292">>,
                           <<"connect_timeout">> => <<"5s">>,
                           <<"enable">> => true,
                           <<"kafka">> =>
                               #{<<"max_batch_bytes">> => <<"896KB">>,
                                 <<"max_rejoin_attempts">> => 5,
                                 <<"offset_commit_interval_seconds">> => 3,
                                 <<"offset_reset_policy">> => <<"latest">>},
                           <<"key_encoding_mode">> => <<"none">>,
                           <<"metadata_request_timeout">> => <<"5s">>,
                           <<"min_metadata_refresh_interval">> => <<"3s">>,
                           <<"ssl">> =>
                               #{<<"enable">> => false,
                                 <<"server_name_indication">> => <<"auto">>,
                                 <<"verify">> => <<"verify_none">>},
                           <<"topic_mapping">> =>
                               [#{<<"kafka_topic">> =>
                                      <<"t_resource_manager_crash_before_subscriber_started-576460752303419662">>,
                                  <<"mqtt_topic">> =>
                                      <<"mqtt/topic/-576460752303422911">>,
                                  <<"payload_template">> => <<"${.}">>,
                                  <<"qos">> => 0}],
                           <<"value_encoding_mode">> => <<"none">>}},
                 #{override_to => cluster}}
```
 2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
e7b35011e6 test(oracle): attempt to fix flaky test 
https://github.com/emqx/emqx/actions/runs/5126014459/jobs/9220155908#step:7:754
```
=CRITICAL REPORT==== 30-May-2023::20:50:26.267103 ===
Run stage failed: error:{assertMatch,
                         [{module,emqx_bridge_oracle_SUITE},
                          {line,394},
                          {expression,
                           "emqx_resource : simple_sync_query ( ResourceId , { query , \"SELECT COUNT(*) FROM mqtt_test\" } )"},
                          {pattern,
                           "{ ok , [ { result_set , _ , _ , [ [ { 3 } ] ] } ] }"},
                          {value,
                           {ok,[{result_set,[<<"COUNT(*)">>],[],[[{0}]]}]}}]}
Stacktrace: [{emqx_bridge_oracle_SUITE,'-t_batch_sync_query/1-fun-5-',1,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,394}]},
             {emqx_bridge_oracle_SUITE,'-t_batch_sync_query/1-fun-6-',5,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,390}]},
             {emqx_bridge_oracle_SUITE,t_batch_sync_query,1,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,364}]}]
```
 2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test 
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
 2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
1e25ebb64c test(gcp_pubsub): attempt to fix flakiness 
https://github.com/emqx/emqx/actions/runs/5125118728/jobs/9218520994?pr=10887#step:8:309
```
  =CRITICAL REPORT==== 30-May-2023::19:19:34.887082 ===
  "check stage" failed: error
  {assertMatch,[{module,emqx_bridge_gcp_pubsub_SUITE},
                {line,1066},
                {expression,"? of_kind ( gcp_pubsub_request_failed , Trace )"},
                {pattern,"[ # { reason := Error , connector := ResourceId } | _ ]"},
                {value,[#{connector =>
                              <<"bridge:gcp_pubsub:emqx_bridge_gcp_pubsub_SUITE0005FCEE15534E9CD4CD02004CF10000">>,
                          msg => gcp_pubsub_request_failed,query_mode => async,
                          reason => {closed,"The connection was lost."},
                          recoverable_error => true,
                          '~meta' =>
                              #{gl => <0.17903.2>,
                                location =>
                                    #Fun<emqx_bridge_gcp_pubsub_connector.19.19548918>,
                                node => 'test@127.0.0.1',pid => <0.19724.2>,
                                time => -576460610660164}}]}]}
  Stacktrace: [{emqx_bridge_gcp_pubsub_SUITE,
                   '-do_econnrefused_or_timeout_test/2-fun-2-',3,
                   [{file,
                        "/__w/emqx/emqx/source/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_SUITE.erl"},
                    {line,1066}]},
               {emqx_bridge_gcp_pubsub_SUITE,do_econnrefused_or_timeout_test,2,
                   [{file,
                        "/__w/emqx/emqx/source/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_SUITE.erl"},
                    {line,1022}]}]
```
 2023-05-31 10:19:55 -03:00
lafirest
502b24bbe2
Merge pull request #10877 from lafirest/fix/rocketmq_sensitive_data 
fix(rocketmq): bump RocketMQ driver to enhance security for sensitive data
 2023-05-31 18:54:44 +08:00
Zhongwen Deng
5729ea0426 feat: api_key conf support hot conf 2023-05-31 18:23:58 +08:00
zhongwencool
26c2ab0bef
Merge pull request #10894 from zhongwencool/crl-cache-conf-hot-update 
feat: update crl_cache conf at runtime
 2023-05-31 17:59:09 +08:00
firest
ce32526c96 chore: bump version && update changes 2023-05-31 17:31:56 +08:00
Stefan Strigler
6c0fb0e2ea test: run tests against iotdb013 container 2023-05-31 11:30:13 +02:00
William Yang
2250ed7052
Merge pull request #10823 from qzhuyan/dev/william/meck-olp 
test: use meck in olp tests
 2023-05-31 11:01:56 +02:00
firest
6081ce8b00 feat: refactored some bridges to avoid leaking resources during crashes at creation 2023-05-31 16:48:13 +08:00
Zhongwen Deng
38e57f511c feat: update crl_cache conf at runtime 2023-05-31 16:18:03 +08:00
JianBo He
837dfeb46f feat(exproto): add raw_publish function 2023-05-31 13:44:51 +08:00
JianBo He
5256d8092c fix: keep the real closed_reason 2023-05-31 13:44:51 +08:00
JianBo He
2e4ec4888d fix(exproto): ensure the on_socket_closed event delivering correctly 2023-05-31 13:44:51 +08:00
JianBo He
9dff026f50 test: fix failed test cases 2023-05-31 13:44:51 +08:00
JianBo He
b6bc3cd921 refactor(exproto): support unary handler 2023-05-31 13:44:51 +08:00
JianBo He
afcd9fcb5e chore: update bpapi.versions 2023-05-31 11:29:55 +08:00
lafirest
5968c71d0c
Merge pull request #10782 from lafirest/feat/retainer_flow_control 
feat(retainer): add a field `deliver_rate` to limit the maximum delivery rate
 2023-05-31 11:01:10 +08:00
zhongwencool
6202cefd5a
Merge pull request #10870 from zhongwencool/stop-dashboard-listener-waiting 
fix: flaky test when stopping dashboard listener
 2023-05-31 10:54:12 +08:00
zhongwencool
ec580f0454
Merge pull request #10866 from zhongwencool/add-delayed-test 
test: add test for delayed update
 2023-05-31 10:53:55 +08:00
firest
a725ff0440 chore: bump version && update changes 2023-05-31 10:18:00 +08:00
firest
a258ef32e5 chore: update retainer app version && changes 2023-05-31 09:44:25 +08:00
firest
03c7b84e89 feat(retainer): add a field deliver_rate to limit the maximum delivery rate 2023-05-31 09:44:25 +08:00
firest
6f19cc394e fix(rocketmq): bump RocketMQ driver to enhance security for sensitive data 2023-05-31 09:44:06 +08:00
firest
79d08e4d26 fix(limiter): temporarily hide the node-level settings 2023-05-31 09:43:14 +08:00
firest
55809f801c test: fix test case error 2023-05-31 09:43:14 +08:00
firest
be2dfa51df fix(limiter): improve the default configuration 2023-05-31 09:43:14 +08:00
Zhongwen Deng
c0b15c402b test: add test for delayed update 2023-05-31 08:17:29 +08:00
Zhongwen Deng
12ef951b8e test: add test for emqx_conf:update([log], Conf) 2023-05-31 07:54:55 +08:00
Zhongwen Deng
5690469896 fix: flaky test when stopping dashboard listener 2023-05-31 07:53:10 +08:00
Thales Macedo Garitezi
a7f4f81c38
Merge pull request #10887 from thalesmg/fix-async-worker-down-buffer-worker-20230530-v50 
fix: block buffer workers so they may retry requests
 2023-05-30 17:39:18 -03:00
Thales Macedo Garitezi
299a67656e
Merge pull request #10886 from thalesmg/test-flakiness-20230530-d 
test(quic_multistreams): attempt to fix flaky tests
 2023-05-30 17:34:23 -03:00
Andrew Mayorov
a2688325e5
Merge pull request #10754 from fix/EMQX-10056/mqtt 
feat(mqttconn): employ ecpool instead of single worker
 2023-05-30 23:28:10 +03:00
Andrew Mayorov
26819a647c
fix(mqttbridge): clarify schema descriptions + log messages 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-05-30 22:21:22 +03:00
Thales Macedo Garitezi
4f1c55f3fb
Merge pull request #10885 from thalesmg/test-flakiness-20230530-c 
test(olp): fix inter-suite flakiness
 2023-05-30 16:09:17 -03:00
Thales Macedo Garitezi
596f05f981 test(channel): attempt to reduce test flakiness 
https://github.com/emqx/emqx/actions/runs/5124277038/jobs/9217362541?pr=10886#step:5:1582
```
%%% emqx_channel_SUITE ==> t_handle_kicked_publish_will_msg: FAILED
%%% emqx_channel_SUITE ==> will_message_not_published
```
 2023-05-30 15:48:48 -03:00
Thales Macedo Garitezi
c43e5b8406
Merge pull request #10883 from thalesmg/test-flakiness-20230530-b 
test(exclusive_sub): fix test flakiness
 2023-05-30 15:46:14 -03:00
Thales Macedo Garitezi
8c565abc84 test(cassandra): fix flaky test 2023-05-30 15:42:53 -03:00
Thales Macedo Garitezi
6be8ff378e fix(buffer_worker): make buffer worker enter blocked state when async worker dies 
Fixes https://emqx.atlassian.net/browse/EMQX-10074

Otherwise, requests from those async workers, now retriable, might not
be retried until the buffer worker blocks for other reasons, which
might take a long time.
 2023-05-30 15:34:22 -03:00
Thales Macedo Garitezi
0f802775c3
Merge pull request #10884 from thalesmg/fix-rule-metrics-empty-map-v50 
fix(rule_engine_api): don't crash when formatting empty metrics
 2023-05-30 15:09:04 -03:00
William Yang
0d6465b016 chore: bump app vsn 2023-05-30 19:41:05 +02:00
Thales Macedo Garitezi
98a99d47cb test(shared_sub): attempt to reduce test flakiness 
https://github.com/emqx/emqx/actions/runs/5122717037/jobs/9215439237#step:8:565
```
%%% emqx_shared_sub_SUITE ==> t_local_fallback: FAILED
%%% emqx_shared_sub_SUITE ==> {{badmatch,<<"not yet?">>},
 [{emqx_shared_sub_SUITE,t_local_fallback,1,
                         [{file,"/__w/emqx/emqx/source/apps/emqx/test/emqx_shared_sub_SUITE.erl"},
                          {line,669}]},
```
 2023-05-30 14:03:14 -03:00
Thales Macedo Garitezi
fa9afcd7e4 test(quic_multistreams): attempt to fix flaky tests 
https://github.com/emqx/emqx/actions/runs/5122717037/jobs/9214950882#step:8:1757
```
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
test_server:ts_tc failed on line 1782
Reason: {test_case_failed,Client <0.6075.5> did not die: stacktrace: {current_stacktrace,
                                            [{gen_statem,loop_receive,3,
                                              [{file,"gen_statem.erl"},
                                               {line,1281}]},
                                             {proc_lib,init_p_do_apply,3,
                                              [{file,"proc_lib.erl"},
                                               {line,240}]}]}}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Testing lib.emqx.emqx_quic_multistreams_SUITE: *** FAILED test case 335 of 364 ***
%%% emqx_quic_multistreams_SUITE ==> shutdown.abort_recv_shutdown.ctrl_stream_shutdown.t_multi_streams_remote_shutdown: FAILED
%%% emqx_quic_multistreams_SUITE ==> {test_case_failed,"Client <0.6075.5> did not die: stacktrace: {current_stacktrace,\n                                            [{gen_statem,loop_receive,3,\n                                              [{file,\"gen_statem.erl\"},\n                                               {line,1281}]},\n                                             {proc_lib,init_p_do_apply,3,\n                                              [{file,\"proc_lib.erl\"},\n                                               {line,240}]}]}"}
```

```
  %%% emqx_quic_multistreams_SUITE ==> mstream.profiles.profile_max_throughput.pub_qos2.sub_qos2.qos.t_multi_streams_unsub: FAILED
  %%% emqx_quic_multistreams_SUITE ==> {{case_clause,
       {ok,#{packet_id => 8,properties => #{},reason_code => 0,
             reason_code_name => success,
             via =>
                 {quic,#Ref<0.686875463.2755788802.201649>,
                     #Ref<0.686875463.2755788804.193527>}}}},
   [{emqx_quic_multistreams_SUITE,t_multi_streams_unsub,1,
        [{file,
             "/home/thales/dev/emqx/emqx/apps/emqx/test/emqx_quic_multistreams_SUITE.erl"},
         {line,1094}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}
```
 2023-05-30 13:56:05 -03:00
Thales Macedo Garitezi
37b4a44e23 test(olp): fix inter-suite flakiness 
https://github.com/emqx/emqx/actions/runs/5122717037/jobs/9212645120?pr=10883#step:8:2538
```
   emqx_olp_SUITE:t_overload_cooldown_conn failed on line 96
  Reason: {assertMatch,[{module,...},{...}|...]}
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  Testing lib.emqx.emqx_olp_SUITE: *** FAILED test case 3 of 4 ***
  %%% emqx_olp_SUITE ==> t_overload_cooldown_conn: FAILED
  %%% emqx_olp_SUITE ==>
  Failure/Error: ?assertMatch({ ok , _Pid }, emqtt : connect ( C ))
    expected: = { ok , _Pid }
         got: {error,econnrefused}
        line: 96
  %%% emqx_olp_SUITE ==> t_overloaded_conn: OK
  Testing lib.emqx.emqx_olp_SUITE: TEST COMPLETE, 3 ok, 1 failed of 4 test cases
```

```sh
make ct-suite SUITE=apps/emqx/test/emqx_access_control_SUITE.erl,apps/emqx/test/emqx_olp_SUITE.erl
```
 2023-05-30 13:39:52 -03:00
William Yang
f2e54c9a6e test: use meck in olp tests 
avoid flaky test results by using mecks.
 2023-05-30 17:57:09 +02:00
Andrew Mayorov
95cc9b9b72
fix(mqttbridge): ensure elixir release assembles successfully 2023-05-30 17:56:35 +03:00
Andrew Mayorov
1c2719236c
chore(mqttbridge): add README 2023-05-30 17:56:35 +03:00
Andrew Mayorov
7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Thales Macedo Garitezi
57aacb471c fix(rule_engine_api): don't crash when formatting empty metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-10073
Fixes https://github.com/emqx/emqx/issues/10714#issuecomment-1567987664

Similar issue to https://github.com/emqx/emqx/pull/10743, but on the
rule engine API.
 2023-05-30 11:54:27 -03:00
Thales Macedo Garitezi
a28580248a test(exclusive_sub): fix test flakiness 
https://github.com/emqx/emqx/actions/runs/5115608275/jobs/9208594481?pr=10864#step:8:735

```
  Testing lib.emqx.emqx_access_control_SUITE: TEST COMPLETE, 4 ok, 0 failed of 4 test cases

  Testing lib.emqx.emqx_exclusive_sub_SUITE: Starting test, 5 test cases

  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  Error detected: {'EXIT',{shutdown,econnrefused}}
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  Testing lib.emqx.emqx_exclusive_sub_SUITE: *** FAILED test case 1 of 5 ***
  %%% emqx_exclusive_sub_SUITE ==> t_allow_normal_sub: FAILED
  %%% emqx_exclusive_sub_SUITE ==> {'EXIT',{shutdown,econnrefused}}

  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  emqx_exclusive_sub_SUITE:t_clean_session failed on line 128
  Reason: {badmatch,{error,econnrefused}}
  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  Testing lib.emqx.emqx_exclusive_sub_SUITE: *** FAILED test case 2 of 5 ***
  %%% emqx_exclusive_sub_SUITE ==> t_clean_session: FAILED
```

To reproduce:

```
make ct-suite SUITE=apps/emqx/test/emqx_access_control_SUITE.erl,apps/emqx/test/emqx_exclusive_sub_SUITE.erl
```
 2023-05-30 11:23:17 -03:00
Andrew Mayorov
ebd612b194
chore: bump applications versions 
* emqx_connector 0.1.25
* emqx_rule_engine 5.0.19
* emqx_ee_bridge 0.1.15
 2023-05-30 17:21:45 +03:00
Andrew Mayorov
eed9358abd
chore: bump ecpool to 0.5.4 
With fixed typings and empty pool handling.
 2023-05-30 17:21:45 +03:00
Andrew Mayorov
a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools 
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
 2023-05-30 17:21:44 +03:00
Andrew Mayorov
c7528e9b35
feat(mqttconn): add pool_size config parameter 
That currently tunes the number of MQTT clients employed both for
subscriptions (if shared subscription is used) and for publishing to
a remote broker.
 2023-05-30 17:21:44 +03:00
Andrew Mayorov
6e97dffdb8
feat(mqttconn): deprecate mode config parameter 
It adds no value: the only mode was `cluster_shareload` and we just as
well can decide to "share" the load across cluster just by looking if
the remote topic is shared subcription filter or not.
 2023-05-30 17:21:44 +03:00
Thales Macedo Garitezi
cb7929a2fc test(broker): attempt to reduce flakiness 2023-05-30 11:12:18 -03:00
Thales Macedo Garitezi
26992e9d98 test(banned): attempt to improve flaky test 2023-05-30 11:08:31 -03:00
Thales Macedo Garitezi
04bd39861d chore: bump app vsns 2023-05-30 11:08:20 -03:00
Stefan Strigler
767f7c57e7 test: check device_id is used from topic or config 2023-05-30 15:55:21 +02:00
Stefan Strigler
938d62a666 test: handle bad payload 2023-05-30 15:55:21 +02:00
Stefan Strigler
860d7b169a test: check data at iotdb instance 2023-05-30 15:55:21 +02:00
Stefan Strigler
a3021c58f1 fix(emqx_bridge_iotdb): allow integer timestamp 2023-05-30 15:55:21 +02:00
Stefan Strigler
bd92116cee test: fix data in calls to look like real world 2023-05-30 15:55:21 +02:00
Stefan Strigler
1381b54a8d fix(emqx_bridge_iotdb): allow non-binary values 2023-05-30 15:55:21 +02:00
Stefan Strigler
64d582770d test: add tracepoints 2023-05-30 15:55:21 +02:00
Stefan Strigler
3d3f2a223c test: test more value conversions 2023-05-30 15:55:21 +02:00
Stefan Strigler
6f54220a51 feat(emqx_bridge_iotdb): handle bad message format gracefully 2023-05-30 15:55:20 +02:00
JianBo He
a132df5568 fix(coap): to better handle coap requests in connection mode 
Fixes for connection deletion and message publishing requests not taking effect
issues once the connection has been created in a different UDP port
first.
 2023-05-30 21:48:57 +08:00
JianBo He
48c53d8b32 chore: add emqx_management_proto_v4 2023-05-30 21:46:33 +08:00
Thales Macedo Garitezi
6b5b9fc8b8
Merge pull request #10864 from thalesmg/fix-flaky-channel-test-20230529 
test(channel): fix flaky test
 2023-05-30 10:18:47 -03:00
Thales Macedo Garitezi
5f6ff9144b
Merge pull request #10862 from thalesmg/fix-flaky-cassandra-test-v50 
test(cassandra): attempt to fix flaky test
 2023-05-30 10:02:14 -03:00
Thales Macedo Garitezi
9c3f838e14
Merge pull request #10841 from thalesmg/kafka-validate-key-v50 
feat({kafka,pulsar}_producer): add validation for empty message key when strategy = key_dispatch
 2023-05-30 09:37:15 -03:00
Kjell Winblad
a4fcc0dcb3 fix: password log leak in RabbitMQ bridge 
This fixes a vulnerability in the RabbitMQ bridge, which could
potentially expose passwords to log files. This was accomplished by
initializing the encryption library specifically designed for RabbitMQ's
passwords. Consequently, passwords are no longer stored in unencrypted
format. As a result, they will no longer be visible as plain text in log
messages, thereby enhancing the system's security.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9976
 2023-05-30 14:36:02 +02:00
JianBo He
fc806c4acd test: fix failed tests 2023-05-30 20:15:49 +08:00
Thales Macedo Garitezi
75fcac9711
Merge pull request #10826 from thalesmg/test-partial-batch-expired-inflight-v50 
test(buffer_worker): add assertion for inflight count after batch expiration
 2023-05-30 09:05:59 -03:00
Thales Macedo Garitezi
5bcc28b8df
Merge pull request #10861 from thalesmg/fix-flaky-telemetry-test-v50 
test(telemetry): fix flaky test
 2023-05-30 09:05:28 -03:00
JianBo He
1e7872c319 feat: support kickout clients in batch 2023-05-30 19:58:40 +08:00
Zaiming (Stone) Shi
0f080cda66 chore: bump emqx_rule_engine app vsn 2023-05-30 13:50:36 +02:00
Andrew Mayorov
4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription 
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
81e78516aa
feat(mqttconn): employ ecpool instead of a single worker 2023-05-30 14:50:05 +03:00
Andrew Mayorov
6967f621d8
fix(mqttconn): unify error interpretation in sync/async modes 
Also move this logic to the mqtt connector itself, in order to avoid
dealing with extra callback layer.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
67d703f8c5
refactor(mqttconn): simplify mqtt connector 
Inline `emqx_connector_mqtt_msg` module code into
`emqx_connector_mqtt_worker` module, since it's not really used
anywhere else and does not provide any reusable abstractions.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
bd956d00b6
feat(mqttconn): stop using gproc in hot path 
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
4da0d83faf
chore(mqttconn): remove dead code 2023-05-30 14:50:04 +03:00
Zaiming (Stone) Shi
aa096f496f
Merge pull request #10876 from zmstone/merge-release-51-back-to-master 
Merge release 51 back to master
 2023-05-30 13:49:21 +02:00
Kjell Winblad
5cfe04956d
Merge pull request #10858 from kjellwinblad/kjell/rule_engine_funcs/add_timezone_to_second/EMQX-10058 
feat: add timezone_to_second/1 function to rule engine
 2023-05-30 12:29:40 +02:00
Zaiming (Stone) Shi
d2ce7c22fd chore: bump app vsn, core app 'emqx' is now at 5.1 2023-05-30 11:32:18 +02:00
Zaiming (Stone) Shi
1e8ed89db8 Merge remote-tracking branch 'origin/master' into release-51 2023-05-30 11:29:44 +02:00
Zaiming (Stone) Shi
91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master 
0530 merge release 50 to master
 2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi
367de2a2eb docs: update bpapi readme 2023-05-30 08:29:43 +02:00
Zaiming (Stone) Shi
941939f7d7 build: finalize 5.0.bpapi 2023-05-30 08:28:45 +02:00
Zaiming (Stone) Shi
3b8cba6ba0 chore: bump rlease versions 2023-05-30 08:28:45 +02:00
Zaiming (Stone) Shi
9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
JimMoen
8569fac6eb
fix: make static_check happy 2023-05-30 14:00:20 +08:00
Zaiming (Stone) Shi
747eb6fcff Merge remote-tracking branches 'origin/release-50' and 'origin/master' into release-51 2023-05-30 07:54:34 +02:00
Zaiming (Stone) Shi
286eefb563 Merge remote-tracking branch 'origin/release-50' 2023-05-30 07:52:54 +02:00
JimMoen
909d348570
chore: fix spec typo 2023-05-30 11:31:37 +08:00
JimMoen
bb363e9b9e
chore: bump dependency vsn 
* bump `snappyer` ~> 1.2.9
  - as dependency of `kafka_protocol` ~> 4.1.3
  - as dependency of `brod`

* bump `rocksdb` ~> 1.7.2-emqx-11
 2023-05-30 11:11:30 +08:00
Thales Macedo Garitezi
1aff46b7c9 test(channel): fix flaky test 2023-05-29 19:04:48 -03:00
Thales Macedo Garitezi
bae00c909f test(cassandra): attempt to fix flaky test 
```
=ERROR REPORT==== 29-May-2023::20:35:20.457796 ===
    id: <<"bridge:cassandra:emqx_bridge_cassandra_SUITE">>
    msg: unrecoverable_error
    reason: {8704,<<"Expected 8 or 0 byte long for date (4)">>}
=CRITICAL REPORT==== 29-May-2023::20:35:20.458850 ===
"check stage" failed: error
{assertMatch,[{module,emqx_bridge_cassandra_SUITE},
              {line,619},
              {expression,"Trace"},
              {pattern,"[ # { result := { ok , _Pid } } ]"},
              {value,[]}]}
Stacktrace: [{emqx_bridge_cassandra_SUITE,'-t_missing_data/1-fun-3-',1,
                 [{file,
                      "/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                  {line,619}]},
             {emqx_bridge_cassandra_SUITE,t_missing_data,1,
                 [{file,
                      "/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                  {line,607}]}]

=CRITICAL REPORT==== 29-May-2023::20:35:20.471872 ===
Check stage failed. Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-05-29_20.34.32/snabbkaffe/1685392520460040358.log"

%%% emqx_bridge_cassandra_SUITE ==> tcp.sync.with_batch.t_missing_data: FAILED
%%% emqx_bridge_cassandra_SUITE ==> {{panic,#{msg => "Unexpected result",result => {error,check_stage_failed}}},
 [{emqx_bridge_cassandra_SUITE,t_missing_data,1,
                               [{file,"/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                                {line,624}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1291}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1223}]}]}
```
 2023-05-29 18:19:26 -03:00
Thales Macedo Garitezi
fa508a95c1 test(telemetry): fix flaky test 
```
%%% emqx_telemetry_SUITE ==>
Failure/Error: ?assertEqual(0, get_value ( num_clients , TelemetryData1 ))
  expected: 0
       got: 1
      line: 454
```
 2023-05-29 18:09:35 -03:00
Thales Macedo Garitezi
67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50 
feat(kafka): ensure allocated resources are removed on failures
 2023-05-29 16:49:29 -03:00
Kjell Winblad
354603da01 test: helper function was interpreted as a test function 2023-05-29 17:20:37 +02:00
Thales Macedo Garitezi
3edbad9f56 feat(pulsar_producer): add validation for empty message key when strategy = key_dispatch 2023-05-29 10:04:19 -03:00
Kjell Winblad
995025b572 feat: add timezone_to_second/1 function to rule engine 
This commit adds the functions timezone_to_offset_seconds and its alias
timezone_to_second to the rule engine. As the names suggests, these
functions convert a timzone offset string such as "+02:00", "Z", "local"
to an integer representing how many seconds that the given timezone
differs from UTC. timezone_to_offset_seconds is the one of the two
functions that should be advertised while timezone_to_second in kept for
backwards compatibility with 4.X.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10058
 2023-05-29 14:30:43 +02:00
Zaiming (Stone) Shi
08c1d5e5ae test: stop testing i18n.conf 2023-05-29 14:22:23 +02:00
Zaiming (Stone) Shi
d014b80834 build: download zh translation from emqx-i18n.git 2023-05-29 14:22:23 +02:00
Thales Macedo Garitezi
79f6861499 chore: prepare to tag v5.0.26 2023-05-29 09:17:59 -03:00
Thales Macedo Garitezi
639feefab6 Merge remote-tracking branch 'origin/master' into tag-v5026 2023-05-29 09:15:48 -03:00
JimMoen
9526d080af
fix: override emqx_license is unnecessary 
* continue #10820
 2023-05-29 19:10:31 +08:00
JimMoen
adae1aec96
chore: fix typo for enterprise readme 2023-05-29 19:08:15 +08:00
JimMoen
6551eb21f2
Merge pull request #10851 from JimMoen/fix-bad-api-logging 
fix: redact api request to hide auth token
 2023-05-29 17:40:44 +08:00
JimMoen
e287cf9a09
Merge pull request #10820 from JimMoen/fix-license-checker-update 
fix: reboot `emqx_license` app for join cluster
 2023-05-29 17:40:29 +08:00
Andrew Mayorov
daa7eebc0f
Merge pull request #10843 from keynslug/fix/ft-prop-zerodiv 
test(ft): ensure no division by zero in coverage proptest
 2023-05-29 10:23:52 +03:00
zhongwencool
2b6d3bb748
Merge pull request #10833 from zhongwencool/telemtry-report 
fix: only report enable authn/authz in telemetry
 2023-05-29 15:14:36 +08:00
zhongwencool
1e986a7b93
Merge pull request #10847 from zhongwencool/telemetry-conf-update 
feat: support emqx_conf:update([telemetry],Conf).
 2023-05-29 15:14:16 +08:00
Zaiming (Stone) Shi
e6869a2904 chore: bump app vsn for emqx_machine 2023-05-29 09:04:17 +02:00
Zaiming (Stone) Shi
8e6dc29637 chore: bump release version to v5.0.26-alpha.1 2023-05-29 08:57:28 +02:00
Zaiming (Stone) Shi
25319c3151
Merge pull request #10839 from zmstone/0526-merge-release-50-to-master 
0526 merge release 50 to master
 2023-05-29 08:56:20 +02:00
Zhongwen Deng
8c814bc861 fix: only report enable authn/authz in telemetry 2023-05-29 14:17:46 +08:00
Zhongwen Deng
4dee1a2429 test: add more test for emqx_telemetry 2023-05-29 14:08:59 +08:00
JimMoen
6f90369429
fix: make dialyzer happy 2023-05-29 13:48:00 +08:00
JimMoen
cb05406a00
fix: reboot emqx_license app for join cluster 2023-05-29 13:47:46 +08:00
JimMoen
dcfe985ee9
fix: redact api request to hide auth token 2023-05-29 11:02:02 +08:00
某文
8ccee19aa5 feat: support emqx_conf:update([telemetry],Conf). 2023-05-28 22:54:35 +08:00
Zaiming (Stone) Shi
c2450d230d
Merge pull request #10605 from zmstone/0503-no-telemetry-app-for-ee 
0503 refactoring: no telemetry app for ee
 2023-05-27 10:57:15 +02:00
Thales Macedo Garitezi
d27f593309 test: fix flaky authn test 2023-05-26 16:44:00 -03:00
Andrew Mayorov
98ce7e8bbd
test(ft): ensure no division by zero in coverage proptest 2023-05-26 21:53:01 +03:00
Thales Macedo Garitezi
9b7e473cf6 feat(kafka_producer): add validation for empty message key when strategy = key_dispatch 
Fixes https://emqx.atlassian.net/browse/EMQX-9979
 2023-05-26 12:02:58 -03:00
Zaiming (Stone) Shi
fe81e9521a
Merge pull request #10600 from zmstone/0504-delete-statsd-all-together 
0504 delete statsd all together
 2023-05-26 16:29:59 +02:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi
cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi
3595a86724 chore(emqx_telemetry): add registered process names to app info 2023-05-26 15:28:06 +02:00
Zaiming (Stone) Shi
d167d6dd41 chore(emqx_telemetry): update supervisor's child restart strategy 2023-05-26 15:28:06 +02:00
Zaiming (Stone) Shi
256e92f4e9 refactor: call emqx_telemetry_config:on_server_stop 2023-05-26 15:28:06 +02:00
Zaiming (Stone) Shi
6e2cde8224 refactor: delete emqx_statsd all together 2023-05-26 15:17:42 +02:00
Andrew Mayorov
45b128946a
Merge pull request #10825 from fix/EMQX-9991/bugs 
fix(ft): avoid leaking secrets
 2023-05-26 16:08:14 +03:00
Zaiming (Stone) Shi
7eb22e27c4 chore: bump release version to e5.0.4 2023-05-26 14:34:01 +02:00
Zhongwen Deng
9b6da3ff10 chore: bump hocon to 0.39.7 2023-05-26 19:00:58 +08:00
Zhongwen Deng
df4a0afdf5 fix: zones api default value 2023-05-26 18:59:38 +08:00
Zaiming (Stone) Shi
2d986140df chore: bump version to e5.0.4-rc.2 2023-05-26 12:01:10 +02:00
Andrew Mayorov
48858dee33
fix(s3): wrap S3 secrets during config load 2023-05-26 12:59:21 +03:00
Andrew Mayorov
3d428a2e0e
fix(ft-s3): rely on asynchronous upload abort 
So that upload aborts would not block assemblers. This should not
affect the expected behavior since S3 API usually allows having
concurrent uploads, plus we couldn't really guarantee successful aborts
anyway.
 2023-05-26 12:59:20 +03:00
Andrew Mayorov
cc3275b389
fix(ft): use conservative timeouts for S3 uploader calls 2023-05-26 12:59:20 +03:00
Andrew Mayorov
d36c2c2928
fix(ft-gc): ensure GC of already complete transfers 2023-05-26 12:59:20 +03:00
Andrew Mayorov
75cf562c90
fix(ft): tune some logging levels 2023-05-26 12:59:20 +03:00
Andrew Mayorov
3278158c66
fix(ft): disable ipv6_probe by default for S3 clients 
It's often demonstrates degraded performance on common setups, e.g.
containers with virtualized network interfaces.
 2023-05-26 12:59:20 +03:00
Zaiming (Stone) Shi
17f6c58f7b fix(telemetry): no default value for telemetry.enable in schema 
the default value is computed from version string
 2023-05-26 11:58:58 +02:00
Zaiming (Stone) Shi
dfc11e57d3 chore: delete telemetry.enable from enterprise.conf 
telemetry is now not included in enterprise edition
 2023-05-26 11:58:58 +02:00
Zaiming (Stone) Shi
ee122155f1 docs: fix spellcheck for telemetry app 2023-05-26 08:44:19 +02:00
Zaiming (Stone) Shi
772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50 
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
 2023-05-26 08:39:00 +02:00
JimMoen
a3b77563ba
fix: make static check happy 2023-05-26 10:34:15 +08:00
JimMoen
1ba65539c6
chore: refine traces command usage 2023-05-26 09:48:56 +08:00
JimMoen
0788f2f197
fix: emqx_ctl traces ... command args 
* use second time as trace `start_at` and `end_at`
* default trace time duration 30min to same as api created
* prevent format line broken
 2023-05-26 09:48:56 +08:00
Zaiming (Stone) Shi
407cab220b
Merge pull request #10760 from paulozulato/fix-failure-joining 
fix: avoid error 500 when node is re-joining cluster
 2023-05-26 03:08:17 +02:00
Thales Macedo Garitezi
32e6213ce3 fix(resource_manager_sup): use one_for_one instead of simple_one_for_one 
Using `simple_one_for_one` has a potential race condition issue where
we read the PID of the resource manager before trying to remove a
resource, and then that PID changes because it was either dead at
first, or it crashed and changed, and later we use this stale PID to
try to remove it from the supervisor.  Under such circumstances, the
restarting child might linger in the supervisor, leaking resources.

By using the resource ID itself as a child ID (and using `one_for_one`
restart strategy), we ensure the child is truly removed.
 2023-05-25 18:07:43 -03:00
Thales Macedo Garitezi
e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
cb34bc5c46 test(kafka_consumer): attempt to stabilize cluster tests 
Example failure:
https://github.com/emqx/emqx/actions/runs/5070096710/jobs/9105822319#step:7:515

The attempt here is to setup the spy as early as possible, before the
bridge starts, so we avoid missing rebalancing events.
 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
0ca3f51503 fix(kafka): improve shutdown and health check logs during shutdown 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
5df7314255 feat(kafka): ensure allocated resources are removed on failures 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
42b37690c7 refactor(pulsar): use macros for allocatable resources 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
db60dcbada test(buffer_worker): add assertion for inflight count after batch expiration 
Fixes https://emqx.atlassian.net/browse/EMQX-9829
 2023-05-25 16:11:37 -03:00
Paulo Zulato
ea86f4442b fix: avoid error 500 when node is re-joining cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-9899
 2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi
194faefc8a fix(webhook): keep resource_opts.request_timeout for webhook bridge (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
 2023-05-25 13:19:27 -03:00
Zaiming (Stone) Shi
e8a4df96bc Merge remote-tracking branch 'origin/master' into 0503-no-telemetry-app-for-ee 2023-05-25 16:43:20 +02:00
Thales Macedo Garitezi
18d57ba3eb
Merge pull request #10812 from thalesmg/test-flakiness-20230524 
test: attempts to reduce flakiness (pgsql, cassandra)
 2023-05-25 09:29:13 -03:00
JianBo He
933357483f
Merge pull request #10817 from HJianBo/fix-restart-timer-interval 
fix: fix auto_restart_interval checker
 2023-05-25 20:23:53 +08:00
JianBo He
6b5ff24952 test: assert the block_until result 2023-05-25 18:12:38 +08:00
JianBo He
fdec86e29c
Merge pull request #10802 from Kinplemelon/kinple/set-base-url-required 
fix: set the base_url in IoTDB bridge as a required field
 2023-05-25 18:12:05 +08:00
JianBo He
de7f1c8aec test: add tests for auto_restart_interval 2023-05-25 17:15:19 +08:00
Zaiming (Stone) Shi
ba41c23b7c
Merge pull request #10794 from paulozulato/fix-oracle-database-field 
Fix oracle database field
 2023-05-25 08:14:13 +02:00
JianBo He
71b636e321 fix: fix auto_restart_interval checker 2023-05-25 12:04:23 +08:00
Paulo Zulato
1f36726cab
Merge pull request #10780 from paulozulato/fix-large-health-check-interval 
Add user-friendly message when interval is out of range
 2023-05-24 18:41:42 -03:00
Thales Macedo Garitezi
658160f09a test(cassandra): attempt to reduce flakiness 2023-05-24 16:57:51 -03:00
Thales Macedo Garitezi
37061b484a
Merge pull request #10778 from thalesmg/refactor-pulsar-on-stop-v50 
feat(pulsar): ensure allocated resources are removed on failures (v5.0)
 2023-05-24 16:00:50 -03:00
Thales Macedo Garitezi
cc8cebe950
Merge pull request #10811 from thalesmg/fix-flaky-kconsumer-balance-test-r50 
test(kafka_consumer): attempt to stabilize cluster tests
 2023-05-24 15:56:28 -03:00
Thales Macedo Garitezi
3cb12d7603
Merge pull request #10810 from thalesmg/merge-r50-into-v50-20230524 
Merge `release-50` into `master`
 2023-05-24 15:55:09 -03:00
Paulo Zulato
122ebcac24 fix: add user-friendly message when interval is out of range 2023-05-24 15:46:00 -03:00
Thales Macedo Garitezi
7f88521836 test(pgsql): reduce flakiness 
Depending on timing, `t_write_timeout` was getting stuck while
checking the resource health, and the previous request timeout options
were making a response to never be sent if that process took too long.
 2023-05-24 15:41:25 -03:00
Paulo Zulato
8d3d3e523b fix(oracle): replace 'database' field by 'service name' 
Fixes https://emqx.atlassian.net/browse/EMQX-9980
 2023-05-24 15:40:56 -03:00
Paulo Zulato
50ff6c4aa4
Merge pull request #10741 from paulozulato/fix-oracle-passwd-leaking 
fix(oracle): hide password on stacktrace
 2023-05-24 15:31:00 -03:00
Kjell Winblad
0e873731a1
Merge pull request #10804 from kjellwinblad/kjell/fix/iotdb_rest_api_v1/EMQX-9920 
fix: REST API version 2 not available in IoTDB 0.13 and 1.0
 2023-05-24 18:34:45 +02:00
Thales Macedo Garitezi
7374e00a02 test(kafka_consumer): attempt to stabilize cluster tests 
Example failure:
https://github.com/emqx/emqx/actions/runs/5070096710/jobs/9105822319#step:7:515

The attempt here is to setup the spy as early as possible, before the
bridge starts, so we avoid missing rebalancing events.
 2023-05-24 13:13:54 -03:00
ieQu1
2b79196a03
Merge pull request #10809 from ieQu1/ekka-0.15.2 
chore(ekka): Bump version to 0.15.2
 2023-05-24 18:13:04 +02:00
Thales Macedo Garitezi
324459990f Merge branch 'release-50' into merge-r50-into-v50-20230524 2023-05-24 12:54:15 -03:00
Thales Macedo Garitezi
ad93af2853
Merge pull request #10808 from thalesmg/fix-ehttpc-infinity-timeout-r50 
fix: handle `infinity` timeout option in `ehttpc` (r5.0)
 2023-05-24 12:53:05 -03:00
Thales Macedo Garitezi
fd2940cd77 feat(pulsar): ensure allocated resources are removed on failures (v5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9937
 2023-05-24 12:29:00 -03:00
zhongwencool
14a6b36899
Merge pull request #10803 from JimMoen/merge-release-50 
Merge release-50 into master
 2023-05-24 23:08:46 +08:00
zhongwencool
96e4b5914a
Merge pull request #10797 from zhongwencool/bump-minirest 
chore: upgrade minirest to 1.3.10 for more clear error msg
 2023-05-24 22:41:42 +08:00
Thales Macedo Garitezi
a172a6bc33
Merge pull request #10795 from thalesmg/fix-schema-registry-rule-iolist-r50 
fix(schema_registry): ensure `schema_encode` output in rule engine is a binary (r5.0)
 2023-05-24 11:02:05 -03:00
ieQu1
3d3dacfcf6 chore(ekka): Bump version to 0.15.2 2023-05-24 15:43:49 +02:00
Thales Macedo Garitezi
4565acc600 fix: handle infinity timeout option in ehttpc (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9987
 2023-05-24 10:33:54 -03:00
Andrew Mayorov
3fd2887921
Merge pull request #10793 from fix/EMQX-9965/bugs 
fix(ft): handle wider class of bad input in APIs
 2023-05-24 16:14:11 +03:00
某文
8816da41a6 test: log test failed 2023-05-24 21:06:15 +08:00
Thales Macedo Garitezi
6c414ab991 fix(schema_registry): ensure schema_encode output in rule engine is a binary (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9981
 2023-05-24 09:13:38 -03:00
JimMoen
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad
4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0 
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
 2023-05-24 13:09:47 +02:00
JimMoen
cc2bb87b82
Merge pull request #10801 from JimMoen/fix-topic-lookup-percent-decode 
fix: lookup topic without force percent decode
 2023-05-24 18:51:16 +08:00
Kinplemelon
7cad1cb93a fix: set the base_url in IoTDB bridge 
as a required field
 2023-05-24 18:18:32 +08:00
JimMoen
1ac140312d
test: lookup topic with percent encoded topic name 2023-05-24 17:39:25 +08:00
JimMoen
e38645aa28
fix: lookup topic without force percent decode 
* the minirest handler would do it
 2023-05-24 17:28:32 +08:00
Zaiming (Stone) Shi
2fdf4b5dac
Merge pull request #10744 from savonarola/0518-fix-eviction 
fix(evacuation): handle expire interval correctly
 2023-05-24 09:20:53 +02:00
Zaiming (Stone) Shi
256cb73e43
Merge pull request #10792 from thalesmg/fix-webhook-bridge-health-check-clause-r50 
fix(webhook): fix empty ehttpc worker list clause on health check (r5.0)
 2023-05-24 08:46:59 +02:00
Zaiming (Stone) Shi
9363ca5846
Merge pull request #10791 from thalesmg/fix-bridge-infinity-timeout-r50 
fix(bridge): pass resource option `request_timeout = infinity` along to buffer workers (r5.0)
 2023-05-24 08:46:43 +02:00
某文
7da0860d6c chore: upgrade minirest to 1.3.10 for more clear error msg 2023-05-24 10:44:52 +08:00
JianBo He
0d3de36f4d
Merge pull request #10770 from kjellwinblad/kjell/fix/iotdb_need_to_parse_input_bug/EMQX-9854 
fix: IoTDB bridge incoming payload needs to be parsed as JSON
 2023-05-24 10:17:23 +08:00
Thales Macedo Garitezi
8e6da40a31 test: fix flaky ocsp test setup (v5.0) 2023-05-23 17:52:02 -03:00
Zaiming (Stone) Shi
2de2a979f5
Merge pull request #10779 from zmstone/0522-remove-log-file-paht-validator 
chore: allow special chars in log dir
 2023-05-23 22:42:12 +02:00
Zaiming (Stone) Shi
b8b1779627 Merge remote-tracking branch 'origin/master' into 0503-no-telemetry-app-for-ee 2023-05-23 21:42:07 +02:00
Zaiming (Stone) Shi
f450f51e2c
Merge pull request #10789 from kjellwinblad/kjell/bridge/rabbitmq_password_not_required_filed/EMQX-9974 
fix: RabbitMQ field not marked as required
 2023-05-23 20:59:49 +02:00
Thales Macedo Garitezi
44ed6a19ee fix(bridge): pass resource option request_timeout = infinity along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00
Thales Macedo Garitezi
b5f63f4151 test: fix wrong test listeners option 2023-05-23 14:50:43 -03:00
Andrew Mayorov
f4047d3946
test(ft): add testcase for nasty tranfer filenames 
* Emoji
* Chinese characters
 2023-05-23 20:46:47 +03:00
Thales Macedo Garitezi
25437adf02 fix(webhook): fix empty ehttpc worker list clause on health check (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9970
 2023-05-23 14:06:40 -03:00
Andrew Mayorov
2dbf84479c
fix(ft): handle wider class of jiffy decode errors 
With malformed cursors in the File listing API.

Fixes EMQX-9965
 2023-05-23 20:00:00 +03:00
Andrew Mayorov
cb14a3e08b
fix(ft): handle empty filepath in fs exporter API 
Fixes EMQX-9973
 2023-05-23 19:58:05 +03:00
Kjell Winblad
63180c87be
style: simplify code for better readability 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-05-23 18:53:14 +02:00
Zaiming (Stone) Shi
4ada938b82
Merge pull request #10757 from thalesmg/perf-pulsar-producer-status-check-r50 
feat(pulsar): improve authn error check time and add connect timeout
 2023-05-23 16:37:19 +02:00
Kjell Winblad
ea2fea77d7 fix: RabbitMQ field not marked as required 
This commit makes sure that the RabbitMQ password filed is marked as
required. This ensures that the user provides a password and that the
bridge does not throw a function clause exception if the password filed
is not set.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9974
 2023-05-23 15:50:18 +02:00
Zaiming (Stone) Shi
47a3096776 test: add test case for file path validation 2023-05-23 14:22:50 +02:00
Zaiming (Stone) Shi
2fa5b511bf chore: hide stale config 2023-05-23 14:20:49 +02:00
Zaiming (Stone) Shi
91150e6d83 chore: allow special chars in log dir 
the current valdiator does not allow space and ':' in log file path
which is an unresonable restriction
 2023-05-23 14:20:49 +02:00
Zaiming (Stone) Shi
e78ffb2dd4 chore: bump apps version 2023-05-23 13:04:28 +02:00
Zaiming (Stone) Shi
3bb1f7ab2b Merge remote-tracking branch 'origin/master' into 0503-no-telemetry-app-for-ee 2023-05-23 13:03:55 +02:00
Kjell Winblad
89ea1646be fix: IoTDB name for version option 
The previous name for the version option was v1.0.x which is clearly
wrong since this option was tested against IoTDB version v1.1.0. This
commit fixes this by renaming the option to v1.x.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9926
 2023-05-23 10:57:05 +02:00
Kjell Winblad
d19ddb1832 fix: IoTDB bridge incoming payload needs to be parsed as JSON 
There was an incorrect assumption that the data incoming to the IoTDB
bridge has already been parsed. This is fixed by parsing the payload as
JSON data if the payload is not already a map.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9854
 2023-05-23 10:14:45 +02:00
zhongwencool
3b6295a085
Merge pull request #10781 from zhongwencool/fix-bridge-test-failed 
test: bridge test init dashboard listener failed
 2023-05-23 11:16:57 +08:00
某文
1deb3c50c8 test: bridge test init dashboard listener failed 2023-05-23 10:28:01 +08:00
某文
67a30ee976 test: bridge test init dashboard listener failed 2023-05-23 10:16:43 +08:00
某文
ed1268ed30 Merge branch 'release-50' into 0522-merge-release-50-to-master 2023-05-23 07:56:35 +08:00
zhongwencool
26e31137c7
Merge pull request #10764 from HJianBo/remove-unused-comments 
chore: remove unused conf comments
 2023-05-23 07:53:25 +08:00
Zaiming (Stone) Shi
68f4ae70b2 chore: bump ee release version to e5.0.4-alpha.2 2023-05-22 22:41:19 +02:00
Zaiming (Stone) Shi
732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
Zaiming (Stone) Shi
1f44dd4579
Merge pull request #10756 from zmstone/0519-refactor-move-lib-ee-emqx_ee_conf-to-apps-emqx_enterprise 
0519 refactor move lib ee emqx ee conf to apps emqx enterprise
 2023-05-22 16:56:51 +02:00
Thales Macedo Garitezi
1c4c7fad92
Merge pull request #10740 from thalesmg/perf-buffer-worker-periodic-metrics-v50 
perf(buffer_worker): update metrics periodically rather than immediately (v5.0)
 2023-05-22 11:41:41 -03:00
Thales Macedo Garitezi
0877e4296a fix(dashboard): add missing function clause 
Example error:
https://github.com/emqx/emqx/actions/runs/5045715277/jobs/9052482682#step:8:294

The previous change was not equivalent to the previous `maps:with/2`
behavior.
 2023-05-22 11:35:04 -03:00
Thales Macedo Garitezi
65f973044f feat(pulsar): improve authn error check time and add connect timeout 
Fixes https://emqx.atlassian.net/browse/EMQX-9910
 2023-05-22 11:33:16 -03:00
Andrew Mayorov
3f882ddcc6
Merge pull request #10751 from ci/ct-flap-listener-api 
fix(listen): wait until port is free when stopping ranch listeners
 2023-05-22 17:00:45 +03:00
lafirest
88e4078885
Merge pull request #10747 from lafirest/fix/rule_funs_time_offset 
fix: port the `emqx_calendar` from v4.4
 2023-05-22 20:52:21 +08:00
Zaiming (Stone) Shi
0b9a32f49a chore: add BSL in apps/emqx_enterprise 2023-05-22 14:51:27 +02:00
Zaiming (Stone) Shi
40e8d5d039 refactor: rename lib-ee/emqx_ee_conf to apps/emqx_enterprise 2023-05-22 14:51:27 +02:00
Zaiming (Stone) Shi
3482c2c849 refactor: redefine default value for 'node.applications' in enterprise 2023-05-22 14:51:26 +02:00
lafirest
42f9c9aa9a
Merge pull request #10746 from lafirest/fix/events_api 
fix: supports test the `$events/delivery_dropped` event by API
 2023-05-22 20:50:03 +08:00
Andrew Mayorov
d22541e8b3
fix(ft): correct mistyped option in README 2023-05-22 15:32:08 +03:00
Thales Macedo Garitezi
0559d6f639 refactor(buffer_worker): use static fn for bumping counters 2023-05-22 09:12:08 -03:00
Thales Macedo Garitezi
9c71f4ecbd test: fix flaky test 2023-05-22 09:12:00 -03:00
Thales Macedo Garitezi
c74c93388e refactor: rename some variables and sum type constructors for clarity 2023-05-22 09:11:23 -03:00
Thales Macedo Garitezi
7d798c10e9 perf(buffer_worker): flush metrics periodically inside buffer worker process 
Fixes https://emqx.atlassian.net/browse/EMQX-9905

Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
 2023-05-22 09:11:23 -03:00
Andrew Mayorov
092e112d62
Merge pull request #10767 from ci/ft-prop-flaky 
test(ft): make proptest less aggressive
 2023-05-22 12:45:24 +03:00
Andrew Mayorov
23542d1262
fix(listen): wait until port is free when stopping ranch listeners 
Simple `cowboy:stop_listener/1` will not close the listening socket
explicitly, it was the source of test flaps before this fix.
 2023-05-22 12:29:40 +03:00
Zhongwen Deng
cb5a596c57 fix: make sure swagger.json is fully generated 2023-05-22 16:58:13 +08:00
Andrew Mayorov
5b312a415b
Merge pull request #10759 from ci/ct-flap-clickhouse 
fix(clickhouse): start app in tests
 2023-05-22 11:11:15 +03:00
Andrew Mayorov
966f748551
test(ft): make proptest less aggressive 
So that the chance of getting huge coverage and as a result, high
chance of running out of memory, is reduced.
 2023-05-22 10:45:27 +03:00
William Yang
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path 
perf(config): enforcing atom key  path in hotcode path
 2023-05-22 09:38:09 +02:00
firest
087dc59115 test: add test cases for the rule_test API 2023-05-22 15:25:35 +08:00
Zhongwen Deng
218fc4a839 refactor: add emqx_authz_file validate function 2023-05-22 14:54:23 +08:00
某文
082214d039 feat: add authz file rule validator 2023-05-22 14:25:05 +08:00
某文
27fca0ef3c fix: check authz's file rule before save to file 2023-05-22 14:22:20 +08:00
zhongwencool
c65e3a46b1
Merge pull request #10742 from zhongwencool/fix-authz-file-rule-check 
fix: check authz's file source rules in pre_config_update
 2023-05-22 14:15:18 +08:00
JianBo He
ce3ec351c7 chore: remove unused conf comments 2023-05-22 14:01:49 +08:00
Zhongwen Deng
96e7005de8 refactor: add emqx_authz_file validate function 2023-05-22 11:03:23 +08:00
firest
56a6b699ac fix: port the emqx_calendar from v4.4 2023-05-22 09:55:31 +08:00
某文
7c2aac64bc fix: bad cert file path in dashboard https listener 2023-05-21 07:41:46 +08:00
Zaiming (Stone) Shi
3e98b3b050
Merge pull request #10755 from zmstone/0517-fix-bridge-update-timeout-issue 
0517 fix bridge update timeout issue
 2023-05-20 06:19:26 +02:00
Andrew Mayorov
ba6b208df2
fix(clickhouse): start app in tests 
Otherwise, depending on the test execution order, tests might
sometimes fail.

Moreover, ensure that applications describe their dependecies
correctly and avoid starting irrelevant apps in tests.
 2023-05-19 23:08:40 +03:00
Thales Macedo Garitezi
bb4fef3cfd
Merge pull request #10752 from thalesmg/fix-pulsar-redact-auth-r50 
fix(pulsar): redact error reason
 2023-05-19 16:47:05 -03:00
Zaiming (Stone) Shi
cb76e5a241 docs: add changelog for 10755 2023-05-19 20:41:26 +02:00
Zaiming (Stone) Shi
0d8ffc0d59 fix(resource-manager): ensure no false creation 
Update is implemented as remove + create.
If a dleete call is made while the create is in progress
the remove call is likely to timeout too.

This causes the follwing creation to falsely succeed,
because there is alreay a running child under the supervisor.

As a result, the resource is permanently removed after
resource_manager eventually handles the remove call.
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
f5e5c59763 refactor(resource-manager-sup): do not force kill resource manager 
the shutdown timeout is now set to infinity so it will never
force kill a resource manager, otherwise there will be
resource leaks
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
21de0f8274 fix(buffer-worker-sup): fast stop 
the timeout shutdown in child spec may
significantly slow down the deletion of a resource

this commit chagnes the shutdown to brutal kill

also, the pool worker removal code has been delete
because it's not necessary since the entier pool is
going to be force-delete later anyway
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
be90c63c78 chore(mqtt-connector): refine logging level 
connect failure should be at warning level but not error,
the connecting state is visiable from dashbaord
also the resource manager logs connection failures in general
at warning level
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
5bbcf4b712 fix(mqtt-connector): faster shutdown 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
7eef86363a test: make erlfmt happy 2023-05-19 17:58:14 +02:00
Andrew Mayorov
9aaa0b6fe1
Merge pull request #9927 from emqx/file-transfer 
feat: implement file transfer over mqtt
 2023-05-19 18:35:13 +03:00
Thales Macedo Garitezi
4327e40740 fix(pulsar): redact error reason 
Fixes https://emqx.atlassian.net/browse/EMQX-9940

Error log after fix:

```
2023-05-19T13:09:26.304769+00:00 [error] msg: failed_to_start_pulsar_client, mfa: emqx_bridge_pulsar_impl_producer:on_start/2, line: 104, instance_id: <<"bridge:pulsar_producer:pprodu">>, pulsar_hosts: pulsar://pulsar:6652, reason: {#{{"pulsar",6652} => #{error => 'AuthenticationError',message => "Unable to authenticate",request_id => 18446744073709551615}},{child,undefined,'pulsar_producer:pprodu:emqx@127.0.0.1',{pulsar_client,start_link,['pulsar_producer:pprodu:emqx@127.0.0.1',["pulsar://pulsar:6652"],#{conn_opts => #{auth_data => <<"******">>,auth_method_name => <<"basic">>},ssl_opts => []}]},transient,false,5000,worker,[pulsar_client]}}
```
 2023-05-19 10:09:57 -03:00
lafirest
264404e180
Merge pull request #10511 from lafirest/fix/resource_safe_log 
fix(resource): make some logging of the resource manager more secure
 2023-05-19 20:42:21 +08:00
lafirest
1aa904d019
Merge pull request #10738 from lafirest/fix/tdengine_template 
fix(tdengine): add supports for the `super table` feature in the SQL template
 2023-05-19 20:41:50 +08:00
Andrew Mayorov
c440cd77b0
fix(test): prefer emqx_connector_http:join_paths/2 in api tests 
It is more consistent with what HTTP servers are expecting, plus
not OS-specific as `filename:join/2`.
 2023-05-19 15:37:01 +03:00
Thales Macedo Garitezi
8cfcc8fbaf
Merge pull request #10743 from thalesmg/fix-bridge-metric-aggregation-r50 
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
 2023-05-19 09:07:22 -03:00
某文
6cb9efd7d3 feat: add authz file rule validator 2023-05-19 18:41:59 +08:00
Andrew Mayorov
5c112602b8
test(evict): fix flapping testcase 
Postpone enabling eviction agent so that target connection will not
get evicted too early.
 2023-05-19 13:37:36 +03:00
Kjell Winblad
28dda39e39
Merge pull request #10724 from kjellwinblad/kjell/docs/missing_http_api_endpoint_descriptions/EMQX-9767 
docs: make sure there is a summary for all API endpoints
 2023-05-19 12:05:41 +02:00
firest
baeb96a6e4 chore: update changes 2023-05-19 15:36:18 +08:00
firest
0eea8438bf fix(resource): make some logging of the resource manager more secure 2023-05-19 15:28:19 +08:00
firest
fa799e95a4 fix: supports test the $events/delivery_dropped event by API 2023-05-19 14:20:26 +08:00
firest
5a08a7b9de fix(tdengine): minor improvement of code and changes 2023-05-19 11:14:31 +08:00
JianBo He
48ceb14755
Merge pull request #10737 from HJianBo/fix-mqtt-sn-bugs 
Avoid urldecode clientid twice
 2023-05-19 09:19:12 +08:00
Paulo Zulato
9c15720fee
Merge pull request #10726 from paulozulato/fix-large-health-check-interval 
fix: validate range for some bridge options
 2023-05-18 19:25:24 -03:00
Ilya Averyanov
9c48b016a9 fix(evacuation): handle expire interval correctly 2023-05-18 22:34:54 +03:00
Paulo Zulato
f9a75cb2a9 fix(oracle): hide password on stacktrace 2023-05-18 16:00:02 -03:00
Thales Macedo Garitezi
09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-9872
 2023-05-18 14:25:13 -03:00
某文
52e2caa671 fix: check authz's file rule before save to file 2023-05-18 22:33:12 +08:00
Thales Macedo Garitezi
56634e5238
Merge pull request #10725 from thalesmg/bump-gproc-091-r50 
chore: bump gproc -> 0.9.0.1 (r5.0)
 2023-05-18 10:38:06 -03:00
Thales Macedo Garitezi
49424cc9a2
Merge pull request #10733 from thalesmg/fix-pulsar-sensitive-struct-r50 
fix(pulsar): mark whole auth struct as sensitive, update dashboard version and improve bridge type matching errors (r5.0)
 2023-05-18 10:15:10 -03:00
Paulo Zulato
5d289ade56 fix: validate range for some bridge options 
Fixes https://emqx.atlassian.net/browse/EMQX-9864

Setting a very large interval can cause `erlang:start_timer` to crash.
Also, setting auto_restart_interval or health_check_interval to "0s"
causes the state machine to be in loop as time 0 is handled separately:

| state_timeout() = timeout() | integer()
| (...)
| If Time is relative and 0 no timer is actually started, instead the the
| time-out event is enqueued to ensure that it gets processed before any
| not yet received external event.
from "https://www.erlang.org/doc/man/gen_statem.html#type-state_timeout"

Therefore, both fields are now validated against the range [1ms, 1h],
which doesn't cause above issues.
 2023-05-18 10:10:58 -03:00
firest
f1a3e5965e chore: update apps version && changes 2023-05-18 17:34:28 +08:00
firest
142125b9e4 test(tdengine): add test cases to cover the super table feature 2023-05-18 17:01:27 +08:00
JianBo He
26b69a5591 test(mqttsn): add test cases 2023-05-18 16:16:09 +08:00
firest
6ff77b221b fix(tdengine): add supports for the automatically create feature in the SQL template 2023-05-18 15:13:08 +08:00
JianBo He
c5398ab651 fix(gateway): avoid uri_decode twice 
The url path has beed decoded in
0ebceb432f/src/cowboy_router.erl (L324-L330).
So, we don't need uri_decode in minirest callback again.
 2023-05-18 14:55:12 +08:00
JianBo He
d77d5e33bc fix(mqttsn): Instantly refresh client info after subscribed/unsubscribed 2023-05-18 14:48:37 +08:00
zhongwencool
ba43a0e30f
Merge pull request #10734 from thalesmg/merge-r50-into-v50-a 
merge `release-50` into `master`
 2023-05-18 14:24:53 +08:00
zhongwencool
bf5ee41009
Merge pull request #10702 from zhongwencool/keepalive-backoff-rename 
feat: deprecated keepalive_backoff, introduce keepalive_multiplier
 2023-05-18 10:00:17 +08:00
某文
d4d25d2660 chore: don't lost previous's statval 2023-05-18 09:49:11 +08:00
Thales Macedo Garitezi
5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi
659cf64ad7 feat(pulsar): use an union member selector for better error messages 2023-05-17 17:56:53 -03:00
Thales Macedo Garitezi
f83f9a2d79 chore: bump app vsns 2023-05-17 14:56:11 -03:00
Thales Macedo Garitezi
447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Thales Macedo Garitezi
10f6edd6ea
Merge pull request #10717 from thalesmg/fix-bw-flush-timer-full-infl-r50 
fix(buffer_worker): avoid setting flush timer when inflight is full
 2023-05-17 14:49:27 -03:00
William Yang
727ad59995 chore(test): improve coverage 2023-05-17 16:58:49 +02:00
Thales Macedo Garitezi
dcccc0910a fix(pulsar): mark whole auth struct as sensitive (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9900

I tried to patch hocon itself to filter the sensitive data, but the
way it's currently structured doesn't seem to keep that field
metadata.  So, for now, we can just mark the whole auth union as
sensitive.
 2023-05-17 11:50:22 -03:00
Thales Macedo Garitezi
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50 
perf(webhook): add async retries and evaluate reply callback in fresh process
 2023-05-17 11:06:48 -03:00
Thales Macedo Garitezi
060efd6964 chore: bump gproc -> 0.9.0.1 (r5.0) 
Includes this fix: https://github.com/uwiger/gproc/pull/193
 2023-05-17 11:05:44 -03:00
Zhongwen Deng
e26ce5816e feat: hide keepalive_backoff, introduce keepalive_multiplier 2023-05-17 21:25:10 +08:00
zhongwencool
ea8ac877b0
Merge pull request #10713 from zhongwencool/put-webhook-request-timeout-into-resource-opts 
feat: update wehbook's request_timeout into resource_opts
 2023-05-17 21:20:42 +08:00
Thales Macedo Garitezi
bc7d0d5fd6 perf(webhook): evaluate replies in fresh process rather than in ehttpc 
This surprisingly simple change yields a big performance improvement
in throughput.

While the previous commit achieves ~ 55 k messages / s
in throughput under some test conditions (100 k concurrent publishers
publishing 1 QoS 1 message per second), the simple change in this
commit improves it further to ~ 63 k messages / s.

Benchmarks indicated that the evaluating one reply function is
consistently quite fast (~ 20 µs), which makes this performance gain
counterintuitive.  Perhaps, although each call is cheap, `ehttpc`
calls several of these in a row when there are several sent requests,
and those costs might add up in latency.
 2023-05-17 09:20:50 -03:00
Thales Macedo Garitezi
a7b41e1cdf perf(webhook): add retry attempts for async 
This is a performance improvement for webhook bridge.

Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well.  Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.

This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
 2023-05-17 09:20:50 -03:00
Thales Macedo Garitezi
01cf7fc703 feat(webhook): use clientid for hash-loadbalancing ehttpc workers 2023-05-17 09:20:50 -03:00
Andrew Mayorov
c95ef71fb5
chore(ft): provide more details in README.md 2023-05-17 12:49:20 +03:00
Kjell Winblad
397dce0d78
Merge pull request #10654 from kjellwinblad/kjell/refactor/clickhouse_bridge/EMQX-9538 
refactor: the clickhouse bridge to use the new bridge app structure
 2023-05-17 11:19:33 +02:00
William Yang
cdf42760fa chore: little change avoid atom leak 2023-05-17 11:01:03 +02:00
Kjell Winblad
bce43978a2 refactor: the clickhouse bridge to use the new bridge app structure 
Fixes:
https://emqx.atlassian.net/browse/EMQX-9538
 2023-05-17 10:12:43 +02:00
Kjell Winblad
ab8b8ccaad docs: make sure there is a summary for all API endpoints 
Many HTTP API endpoints did not have a summary when looking at the API
docs at "http://emqx_host_name:18083/api-docs". This has been fixed by
making sure there is a summary corresponding to all endpoints.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9767
 2023-05-17 09:58:56 +02:00
Ilya Averyanov
c54d04450c
Merge pull request #10723 from savonarola/0516-improve-rebalance-logging 
Improve rebalance logging
 2023-05-17 09:21:10 +03:00
某文
2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
lafirest
b5da9eb9ad
Merge pull request #10728 from lafirest/fix/port_ruleengine_export_colums 
fix: cannot access columns exported by FOREACH in DO clause
 2023-05-17 11:46:17 +08:00
firest
1f7ede90a4 chore: update app version && changes 2023-05-17 11:01:31 +08:00
firest
18043150be fix: cannot access columns exported by FOREACH in DO clause 2023-05-17 10:55:47 +08:00
JianBo He
0dd3325d70
Merge pull request #10711 from lafirest/fix/fix_infinity_bucket 
fix(limiter): a bucket with an infinity rate shouldn't be added to limiter server
 2023-05-17 10:40:59 +08:00
Ilya Averyanov
b71955e368 fix(ft): bump application versions 2023-05-17 00:29:15 +03:00
Ilya Averyanov
1a8cf0e392 Merge branch 'release-50' into file-transfer 
* release-50:
  fix(pulsar): use a binary duration as default `health_check_interval`
  docs: add changelog entry
  docs: clarify description of bridge username and password
  chore: bump to v5.0.25
  fix(limiter): adjust type for compatibility
  fix(limiter): fix that update node-level limiter config will not working
  chore: upgrade dashboard to v1.2.4-1 for ce
  chore: upgarde rulesql to 0.1.6 to fix invaid utf8 input
  chore: add changelog for 10659
  fix: crash when sysmon.os.mem_check_interval = disabled
  chore: bump influxdb version && update changes
  refactor(influxdb): move influxdb bridge into its own app
  chore: add listener default changelog
  fix: ocsp cache SUITE failed
  fix: ensure atom key for emqx_config:get
  fix: only fill cerf_file default in server side
  fix: authn init is empty
  fix: bad listeners default ssl_options
 2023-05-17 00:27:55 +03:00
Zaiming (Stone) Shi
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field 
0421 refactor delete unused schema field
 2023-05-16 22:22:03 +02:00
Thales Macedo Garitezi
85089a3210 fix(buffer_worker): correctly flush the buffer workers when inflight table room is made 
The previous commit uncovered another bug that was hidden by it:
`maybe_flush_after_async_reply` was sending a message to the wrong
PID.  It was sending a message to `self()` meaning to target a buffer
worker, but `self()` in that context is never the buffer worker, it's
the connector's worker.

This change also revealed a race condition where the buffer workers
could stop flushing messages.  So we piggy-backed on the atomic update
of the table size count to check if the buffer worker should be poked
to continue flushing.  This allows us to get rid of
`maybe_flush_after_async_reply` altogether.
 2023-05-16 17:15:42 -03:00
Andrew Mayorov
b2ecbef0f1
Merge pull request #10715 from fix/EMQX-9897/preserve-peercert-until-connected 
fix(chan): postpone trimming conninfo after `connected` hook run
 2023-05-16 22:56:41 +03:00
Andrew Mayorov
3cd95f40e5
test(chan): verify hooks receive peercert until connected 2023-05-16 19:35:56 +03:00
Andrew Mayorov
967b2e72e0
test(emqx): remove peercert from clientinfo fixtures 
According to typespec, there's no place for `peercert` in
`clientinfo()`.
 2023-05-16 19:35:55 +03:00
Andrew Mayorov
b341a04955
fix(chan): postpone trimming conninfo until connected hooks run 
Some users expect to get the peer certificate in `connected` hooks, but
the `conninfo` was trimmed before `connected` hooks run.
 2023-05-16 19:35:55 +03:00
Ilya Averyanov
d4b60c5610 chore(rebalance): improve debug logging 2023-05-16 18:59:12 +03:00
Thales Macedo Garitezi
cebde87114 fix(pulsar): use a binary duration as default health_check_interval 
Fixes https://emqx.atlassian.net/browse/EMQX-9885

The frontend needs the default value to match the duration (binary)
type to display correctly.
 2023-05-16 11:29:29 -03:00
Thales Macedo Garitezi
657df05ad9 fix(buffer_worker): avoid setting flush timer when inflight is full 
Fixes https://emqx.atlassian.net/browse/EMQX-9902

When the buffer worker inflight window is full, we don’t need to set a
timer to flush the messages again because there’s no more room, and
one of the inflight windows will flush the buffer worker by calling
`flush_worker`.

Currently, we do set the timer on such situation, and this fact
combined with the default batch time of 0 yields a busy loop situation
where the CPU spins a lot while inflight messages do not return.
 2023-05-16 11:28:58 -03:00
Ilya Averyanov
c113a8ac6a
Merge pull request #10677 from savonarola/0511-fix-rule-api 
fix(api): respond 404 on the deletion of nonexistent rule
 2023-05-16 16:50:30 +03:00
Andrew Mayorov
6f8f21106b
fix(assert): use unpredictable binding names in macros 
Also translate macro vars to ALLCAPS for consistency.
 2023-05-16 16:46:05 +03:00
Andrew Mayorov
e3b822c1a0
chore: remove empty header 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
 2023-05-16 16:45:41 +03:00
Andrew Mayorov
a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
某文
a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
Andrew Mayorov
be3a0ce974
fix(maybe): correct apply/2 typespec 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
 2023-05-16 16:26:26 +03:00
Andrew Mayorov
6813ea8e7a
test(client): dedicate separate testcase to peercert cleaning 2023-05-16 13:25:33 +03:00
Zaiming (Stone) Shi
a0454b9417
Merge pull request #10674 from zmstone/0511-feat-support-rule-engine-eval 
Add an experimental feature: support external rule-engine SQL functions
 2023-05-16 10:55:00 +02:00
firest
255f616d26 chore: bump emqx app version 2023-05-16 16:36:36 +08:00
William Yang
1c746ed289 perf(config): enforcing atom key path in hotcode path 2023-05-16 09:57:04 +02:00
某文
7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
firest
50e7de9db2 fix(limiter): a bucket with an infinity rate shouldn't be added to limiter server 2023-05-16 13:30:41 +08:00
Thales Macedo Garitezi
5960cc530a fix(pulsar): use a binary duration as default health_check_interval 
Fixes https://emqx.atlassian.net/browse/EMQX-9885

The frontend needs the default value to match the duration (binary)
type to display correctly.
 2023-05-15 09:05:21 -03:00
Zaiming (Stone) Shi
3c64735b87 feat(rule_engine): support external SQL function provider 2023-05-15 08:58:40 +02:00
JianBo He
38fcb7a097 docs: hide the not-ready document links 2023-05-15 11:20:23 +08:00
JianBo He
383fec9dfc docs: fix invalid links 2023-05-15 11:18:09 +08:00
lafirest
67ada52808
Merge pull request #10682 from lafirest/fix/will_msg_timestamp 
fix: update the will message timestamp when it is ready to publish
 2023-05-12 22:25:32 +08:00
某文
5ef2a603a1 chore: bump to v5.0.25 2023-05-12 20:27:36 +08:00
firest
95a67f390f fix(limiter): adjust type for compatibility 2023-05-12 18:41:41 +08:00
zhongwencool
a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
firest
a38b270b03 fix(limiter): fix that update node-level limiter config will not working 2023-05-12 17:39:49 +08:00
Zhongwen Deng
90549abec7 fix: crash when sysmon.os.mem_check_interval = disabled 2023-05-12 14:42:23 +08:00
lafirest
33d1872e8f
Merge pull request #10679 from lafirest/refactor/influxdb_dir 
refactor(influxdb): move influxdb bridge into its own app
 2023-05-12 14:31:29 +08:00
firest
dcd4640a57 fix: update the will message timestamp when it is ready to publish 2023-05-12 14:28:26 +08:00
zhongwencool
1aceed7b7c
Merge pull request #10672 from zhongwencool/fix-listener-default 
fix: bad listeners default ssl_options
 2023-05-12 14:28:23 +08:00
firest
7d46fe8390 chore: bump influxdb version && update changes 2023-05-12 11:48:56 +08:00
firest
6497bcae46 refactor(influxdb): move influxdb bridge into its own app 2023-05-12 11:48:49 +08:00
Thales Macedo Garitezi
64dc9ed46a perf(metrics): avoid increasing counters by 0 
Some performance tests indicate that calling `telemetry` is costly in
hot paths.  Since increasing a counter by 0 is a no-op, we should
avoid calling `telemetry` if the amount to increase is 0.
 2023-05-11 15:13:37 -03:00
Ilya Averyanov
49e9ace1c1 fix(api): respond 404 on the deletion of nonexistent rule 2023-05-11 23:12:59 +05:00
Thales Macedo Garitezi
c4f2dba9c5
Merge pull request #10643 from thalesmg/test-sanity-checks-v50 
test: perform sanity checks when starting apps
 2023-05-11 14:00:22 -03:00
某文
91f97f6c29 fix: ocsp cache SUITE failed 2023-05-11 21:26:23 +08:00
Ilya Averyanov
5fdcfad60c fix(ft): synchronize erlcloud deps 2023-05-11 16:18:45 +05:00
Ilya Averyanov
5b5d7ceac5 Merge branch 'release-50' into file-transfer 
* release-50:
  fix(limiter): fix an error when setting `max_conn_rate` in a listener
  chore: bump erlcloud dependencies vsns
  chore: rename dynamo template files
  refactor(dynamo): move dynamo bridge into its own app
  chore: update changes && bump app versions
  fix: issues with the RabbitMQ config
  refactor(pgsql): move pgsql && matrix && timescale bridges into their own app
  fix: the iotdb password field so it has the password format
  chore: update changes
  refactor(tdengine): move tdengine bridge into its own app
  feat: deprecate listeners's authn http api
 2023-05-11 16:15:38 +05:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zhongwen Deng
3d41449fde fix: only fill cerf_file default in server side 2023-05-11 17:33:18 +08:00
SergeTupchiy
0617a9b11c
Merge pull request #10653 from SergeTupchiy/convert-gateway-authn-certs 
fix(emqx_gateway): convert and clear authentication certificates
 2023-05-11 11:35:55 +03:00
Zhongwen Deng
78c2c53daa fix: authn init is empty 2023-05-11 16:26:47 +08:00
Zhongwen Deng
d9f9e951ec fix: bad listeners default ssl_options 2023-05-11 16:25:25 +08:00
zhongwencool
a736b633b0
Merge pull request #10607 from zhongwencool/log-conf-refactor 
feat: simplify log configuration
 2023-05-11 16:12:27 +08:00
lafirest
d3a7d6d9d8
Merge pull request #10668 from lafirest/fix/max_conn_rate 
fix(limiter): fix an error when setting `max_conn_rate` in a listener
 2023-05-11 15:49:44 +08:00
Zaiming (Stone) Shi
7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Zaiming (Stone) Shi
b3e35dac1c
Merge pull request #10584 from paulozulato/debug-ssl-handshake 
feat: add log level configuration to ssl communication
 2023-05-11 08:59:13 +02:00
firest
b7126257a5 fix(limiter): fix an error when setting max_conn_rate in a listener 2023-05-11 13:52:13 +08:00
JimMoen
fbd516d740
Merge pull request #10622 from JimMoen/refactor-dynamo-bridge 
refactor(dynamo): move dynamo bridge into its own app
 2023-05-11 11:00:50 +08:00
lafirest
c8758190b5
Merge pull request #10662 from lafirest/refactor/pgsql_dir 
refactor(pgsql): move pgsql && matrix && timescale bridges into their own app
 2023-05-11 10:05:51 +08:00
Andrew Mayorov
6b688d6646
fix(ft): anticipate repeated kickoffs + fix testcase 2023-05-10 17:35:53 +03:00
Kjell Winblad
296c870896
Merge pull request #10664 from kjellwinblad/kjell/fix/rabbitmq_and_iotdb_issues 
Fix small RabbitMQ and IotDB issues
 2023-05-10 14:48:20 +02:00
William Yang
ec6cac88c9
Merge pull request #10663 from qzhuyan/dev/william/fix-gateway-mqttsn-nolocal 
chore(gateway-mqttsn): fix a minor bug
 2023-05-10 13:13:48 +02:00
JimMoen
e32ab10d01
chore: rename dynamo template files 2023-05-10 18:49:21 +08:00
JimMoen
8914e006c2
refactor(dynamo): move dynamo bridge into its own app 2023-05-10 18:49:21 +08:00
firest
f6a2f752ff chore: update changes && bump app versions 2023-05-10 17:23:26 +08:00
Kjell Winblad
41f5eff014 fix: issues with the RabbitMQ config 2023-05-10 10:53:40 +02:00
William Yang
614f9bda51 chore(gateway-mqttsn): fix a minor bug 2023-05-10 10:53:00 +02:00
firest
066ed5c6ec refactor(pgsql): move pgsql && matrix && timescale bridges into their own app 2023-05-10 16:51:33 +08:00
lafirest
594ceaf9e6
Merge pull request #10650 from lafirest/refactor/tdengine_dir 
refactor(tdengine): move tdengine bridge into its own app
 2023-05-10 16:42:35 +08:00
Kjell Winblad
08af90daa9 fix: the iotdb password field so it has the password format 2023-05-10 10:38:40 +02:00
zhongwencool
7e720ca260
Merge pull request #10641 from thalesmg/bump-gproc-091-v50 
chore: bump gproc -> 0.9.0.1
 2023-05-10 16:16:26 +08:00
zhongwencool
48db98d75b
Merge pull request #10657 from zhongwencool/deprecated-listeners-authn 
feat: deprecate listeners authn
 2023-05-10 15:35:49 +08:00
Ilya Averyanov
8d9b785bd7 Merge branch 'release-50' into file-transfer 
* release-50:
  chore(rebalance): fix app metadata
  chore(rebalance): move apps from lib-ee, add READMEs
  docs: refine zh tr
  docs: delete APL header from ee file
  docs: delete zh changelog
  chore(rebalance): review fixes
  chore(rebalance): rebase and review fixes
  feat(rebalance): port apps from 4.x
 2023-05-10 11:55:23 +05:00
Zhongwen Deng
b80227a02a fix: crash when sysmon.os.mem_check_interval = disabled 2023-05-10 14:49:34 +08:00
Ilya Averyanov
42f5433aaf
Merge pull request #10655 from savonarola/0510-idempotent-fin 
Make FT fin command idempotent
 2023-05-10 11:39:43 +05:00
firest
0f979b3d24 refactor(tdengine): move tdengine bridge into its own app 2023-05-10 14:12:57 +08:00
Zhongwen Deng
83e7b30a80 feat: deprecate listeners's authn http api 2023-05-10 09:50:53 +08:00
Zhongwen Deng
1d609dacf7 test: update api_config test 2023-05-10 09:21:42 +08:00
Ilya Averyanov
244188982b fix(ft): add missing translations 2023-05-09 23:32:17 +05:00
Ilya Averyanov
45875bfee5 fix(ft): update s3 exporter tests to the new config 2023-05-09 23:32:17 +05:00
Ilya Averyanov
9381c895bb fix(ft): fix dialyzer issues 2023-05-09 23:32:16 +05:00
Andrew Mayorov
079b8e9476 fix(ft): silence warnings when some root is not yet mkdired 2023-05-09 23:32:16 +05:00
Andrew Mayorov
5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends 
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
 2023-05-09 23:32:16 +05:00
Ilya Averyanov
8ac881a140 chore(ft): handle multiple/concurrent fins more gracefully 2023-05-09 23:31:39 +05:00
Ilya Averyanov
7fa166f034 Merge branch 'release-50' into file-transfer 
* release-50: (73 commits)
  feat: add RabbitMQ bridge
  docs: improve rule engine labels and descriptions
  chore: bump version && update changes
  refactor(rocketmq): move rocketmq bridge into its own app
  test: dashboard_listener_test crash
  chore: bump chart versions
  chore: bump ee version to e5.0.4-alpha.1
  test: fix inter-suite flakiness
  build: compatibility to make 4.4+
  feat: add IotDB bridge
  ci: ensure git safe dir in build_packages
  ci: ensure git safe dir
  test: check_oom's max_mailbox_size
  feat: rename max_message_queue_len to max_mailbox_size
  fix(buffer_worker): fix inflight count when updating inflight item
  chore: prepare for v5.0.25-rc.1 release
  docs: add change log entry
  fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
  chore: `MQTT X` -> `MQTTX`
  chore: make sure brod_gssapi app is included in relese package
  ...
 2023-05-09 23:30:22 +05:00
Ilya Averyanov
61deda3ea6 chore(rebalance): fix app metadata 2023-05-09 21:05:20 +05:00
Ilya Averyanov
8d83dc12e7 chore(rebalance): move apps from lib-ee, add READMEs 2023-05-09 20:51:24 +05:00
Ilya Averyanov
e683d28973 chore(rebalance): rebase and review fixes 2023-05-09 20:51:24 +05:00
Ilya Averyanov
609f7bd8fd feat(rebalance): port apps from 4.x 2023-05-09 20:51:22 +05:00
Serge Tupchii
b98a7168c4 fix(emqx_gateway): convert and clear authentication certificates 2023-05-09 17:32:44 +03:00
Thales Macedo Garitezi
5643c12930 chore: bump gproc -> 0.9.0.1 
Includes fix: https://github.com/uwiger/gproc/pull/193

Prior to the fix, when using the `random` pool strategy, one of the
workers receives about double the load of other workers, which
decreases throughput of bridges like webhook.
 2023-05-09 11:27:25 -03:00
William Yang
328add3a02
Merge pull request #10563 from qzhuyan/fix/william/no_local_filter_all 
fix(mqtt): drop all local messages in session deliver
 2023-05-09 15:29:12 +02:00
Kjell Winblad
70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00
lafirest
5ebb96fcc8
Merge pull request #10648 from lafirest/refactor/rocketmq_dir 
refactor(rocketmq): move rocketmq bridge into its own app
 2023-05-09 17:55:10 +08:00
zhongwencool
05c71cece5
Merge pull request #10646 from zhongwencool/dashboard-test 
test: dashboard_listener_test crash
 2023-05-09 17:52:04 +08:00
Zhongwen Deng
428cc45c69 fix: filter disable file handler 2023-05-09 17:50:06 +08:00
firest
12491e08db chore: bump version && update changes 2023-05-09 15:58:02 +08:00
firest
a119100a0e refactor(rocketmq): move rocketmq bridge into its own app 2023-05-09 15:57:56 +08:00
Zhongwen Deng
ac6c8d840f test: dashboard_listener_test crash 2023-05-09 12:16:45 +08:00
Zhongwen Deng
c901f3a9d4 test: nodes's log test failed 2023-05-09 11:56:24 +08:00
lafirest
b94290db58
Merge pull request #10625 from lafirest/refactor/limiter_cfg 
refactor(limiter): simplify limiter configuration
 2023-05-09 11:40:05 +08:00
Zhongwen Deng
ad111a27f9 feat: alias log.file.to to log.file.file 2023-05-09 11:24:51 +08:00
Zhongwen Deng
14952658d5 chore: convert rotation struct to roation_count 2023-05-09 11:24:51 +08:00
Zhongwen Deng
bf87aebbba chore: upgrade hocon to 0.39.5 2023-05-09 11:24:51 +08:00
Zhongwen Deng
d31a44f96b feat: convert rotation from map to integer 2023-05-09 11:24:51 +08:00
某文
7b000157d0 feat: refactor log configuration 2023-05-09 11:24:51 +08:00
Zaiming (Stone) Shi
4456a32465 chore: bump ee version to e5.0.4-alpha.1 2023-05-08 22:08:51 +02:00
Zaiming (Stone) Shi
13dcb5732f Merge remote-tracking branch 'origin/release-50' into 0508-prepare-for-e5.0.4 2023-05-08 21:29:35 +02:00
Zaiming (Stone) Shi
a22dd399b6 Merge remote-tracking branch 'origin/master' into 0508-prepare-for-e5.0.4 2023-05-08 21:16:12 +02:00
Stefan Strigler
1ade7ce9a3
Merge pull request #10560 from sstrigler/EMQX-8390-5-0-x-io-tdb-bridge-port-conf-and-api-to-5-0 
IoTDB Bridge
 2023-05-08 20:27:01 +02:00
Paulo Zulato
77176787ca feat: add log level configuration to ssl communication 
Fixes https://emqx.atlassian.net/browse/EMQX-9781
 2023-05-08 15:11:49 -03:00
Paulo Zulato
83c4b2f2d4
Merge pull request #10568 from paulozulato/feat-shutdown-count 
feat: add shutdown count
 2023-05-08 15:07:51 -03:00
Thales Macedo Garitezi
306a732e5e test: perform sanity checks when starting apps 
These are checks to detect inter-suite or inter-testcase flakiness
early.  One suite might forget one application running
and stop others, and then the `application:start/2' callback is never
called again for this application.

One example of this was that: i) `emqx_rule_engine` was left running by
one suite; ii) `emqx` app was stopped, taking `emqx_config_handler`
down with it and losing the rule engine handler; iii) another suite
that uses rule engine "started" it (a no-op) and then the config
handler was never installed again.
 2023-05-08 14:00:23 -03:00
zhongwencool
f6c5644448
Merge pull request #10623 from zhongwencool/rename-max-message-queue-len 
feat: rename max_message_queue_len to max_mailbox_size
 2023-05-08 23:08:06 +08:00
Stefan Strigler
d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zaiming (Stone) Shi
ddd68eff22 docs: add emqx_telemetry README 2023-05-08 15:16:27 +02:00
Zaiming (Stone) Shi
4a2e583e3f refactor: move telemetry to its own app 2023-05-08 15:16:27 +02:00
zhongwencool
74d0436fbf
Merge pull request #10636 from kjellwinblad/kjell/fix/mongo/max_overflow/EMQX-9714 
fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
 2023-05-08 20:45:57 +08:00
William Yang
8545d3d4a7 test: subscribe with no_local, mixed pub from different clients 2023-05-08 14:40:37 +02:00
Zhongwen Deng
04e62f6a2d test: check_oom's max_mailbox_size 2023-05-08 20:27:52 +08:00
Zhongwen Deng
eaa129d0d7 feat: rename max_message_queue_len to max_mailbox_size 2023-05-08 20:27:52 +08:00
Thales Macedo Garitezi
eba627b365 fix(buffer_worker): fix inflight count when updating inflight item 2023-05-08 09:27:51 -03:00
Zaiming (Stone) Shi
0250718910 chore: prepare for v5.0.25-rc.1 release 2023-05-08 14:04:22 +02:00
Zaiming (Stone) Shi
f450dc49ba Merge remote-tracking branch 'origin/release-50' into 0508-prepare-for-e5.0.4 2023-05-08 14:03:42 +02:00
Zaiming (Stone) Shi
83257617cd
Merge pull request #10630 from zhongwencool/sync-release-50-to-master 
Sync release 50 to master
 2023-05-08 14:02:27 +02:00
Zaiming (Stone) Shi
3dba862f85
Merge pull request #10629 from zmstone/0506-add-kafka-gssapi-kerberos-auth 
fix(kafka): ensure brod_gssapi and sasl_auth included in release
 2023-05-08 13:57:33 +02:00
Zaiming (Stone) Shi
30da70692c
Merge pull request #10556 from sstrigler/wrap_auth_headers 
fix(emqx_connector_http): wrap and unwrap auth headers
 2023-05-08 13:43:24 +02:00
Kjell Winblad
a711ee21f3 fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json 
The schema type non_neg_integer() should be translated to minimum 0 and
not 1 when generating the bridge-api-en.json file.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9714
 2023-05-08 12:38:16 +02:00
zhongwencool
1d64d343e0
Merge pull request #10340 from zhongwencool/systemd-stop-crash-log-v50 
fix: systemctl stop don't stop port in the desired order
 2023-05-08 17:05:35 +08:00
Zaiming (Stone) Shi
8803c9b326 chore: make sure brod_gssapi app is included in relese package 2023-05-08 10:59:38 +02:00
Zaiming (Stone) Shi
f2a223c1e9 chore: prepare for e5.0.3 release 2023-05-08 10:59:04 +02:00
Stefan Strigler
d920f415cd fix(emqx_logger): fix spec for set_log_level 2023-05-08 09:57:26 +02:00