yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
16,714 Commits 393 Branches 684 Tags 74 MiB
d661b10355
Commit Graph

8426 Commits

Author SHA1 Message Date
Andrew Mayorov
cf70184ebd test(ds): Provide more general keymapper_info/1 2023-06-22 10:20:18 +02:00
Andrew Mayorov
2f385022cd chore(ds): Rename testsuite to reflect test subject better 2023-06-22 10:20:18 +02:00
Andrew Mayorov
d875fa49d3 feat(ds): Enable periodic iterator refresh 
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
 2023-06-22 10:20:18 +02:00
Andrew Mayorov
7f408da251 refactor(ds): Use inline functions instead of macros 2023-06-22 10:20:07 +02:00
Andrew Mayorov
a11e75d189 fix(ds): clear bitmask of topic filter tail containing wildcards 2023-06-22 10:16:55 +02:00
Andrew Mayorov
ac0935ef91 test(ds): Proptest that iteration is exhaustive 
Compare iteration results against what an extremely simplified model
produces.
 2023-06-22 10:16:45 +02:00
Andrew Mayorov
60e3070328 test(ds): Scale up number of messages per topic in proptests 2023-06-22 10:16:41 +02:00
Andrew Mayorov
82a7292851 test(ds): Use _build/test/proper as a scratch dir for testruns 2023-06-22 10:16:32 +02:00
Andrew Mayorov
440fecd268 test(ds): Add a proptest on iterator preserve / restore 
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
 2023-06-22 10:16:23 +02:00
Andrew Mayorov
1f5425975f feat(ds): add an ability to preserve and restore iterators 
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
 2023-06-22 10:16:20 +02:00
Andrew Mayorov
94e8dbfe7f test(ds): Move proptests into a separate module 
Following conventions. Also add few proptests on keyspace filters.
 2023-06-22 10:16:16 +02:00
Andrew Mayorov
4c643aa7d5 refactor(ds): Introduce keyspace filter concept 
So we could conveniently test it separately.
 2023-06-22 10:16:10 +02:00
Andrew Mayorov
c5826cee56 refactor(ds): rename compute_hash_* → compute_topic_* 2023-06-22 10:16:05 +02:00
Andrew Mayorov
85d3783ff1 chore(ds): Drop few unused macrodefinitions 2023-06-22 10:15:56 +02:00
Andrew Mayorov
a0bcdb5104 chore(ds): Attempt to make compute_next_seek's logic clearer 2023-06-22 10:15:49 +02:00
ieQu1
f1e39d977a fix(ds): Fix dialyzer warnings 2023-06-22 10:15:47 +02:00
ieQu1
adcbf40d27 refactor(ds): tau -> epoch 2023-06-22 10:15:44 +02:00
ieQu1
120d4e66ae refactor(ds): Factor out configuration to a separate module 2023-06-22 10:15:42 +02:00
ieQu1
0bbc5ecb32 refactor(ds): Introduce bitwise_concat function 2023-06-22 10:15:40 +02:00
ieQu1
b5bb77dd58 feat(ds): Add metadata storage and supervision tree 2023-06-22 10:15:37 +02:00
Andrew Mayorov
5e30a5d3dd feat(ds): Implement keyspace partitioning across time 2023-06-22 10:15:32 +02:00
Andrew Mayorov
8c9c9cc669 feat(ds): Allow to specify message store options 
* Keymapper
* Column family name + DB options
* DB write / read options
 2023-06-22 10:15:25 +02:00
Andrew Mayorov
5b6c4d766b test(ds): Add basic property tests 2023-06-22 10:15:20 +02:00
Andrew Mayorov
a1e29e4e24 refactor(ds): Improve function naming 2023-06-22 10:15:03 +02:00
Andrew Mayorov
cd12338c3f feat(ds): Smoke tests for wildcard iterator scans 2023-06-22 10:14:46 +02:00
Andrew Mayorov
9c1cd4911d feat(ds): Implement iterator next 2023-06-22 10:14:43 +02:00
ieQu1
27b925405b test(ds): Add testcase 2023-06-22 10:14:41 +02:00
Andrew Mayorov
691a27883d feat(ds): Initial commit 2023-06-22 10:14:37 +02:00
Paulo Zulato
f47cc2a458 feat(kafka): add support for kafka headers and ext headers 
Fixes https://emqx.atlassian.net/browse/EMQX-9175
 2023-06-21 18:27:39 -03:00
Paulo Zulato
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable 
Data bridge target unavailable
 2023-06-21 18:19:23 -03:00
Andrew Mayorov
62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon 
chore: bump hocon to 0.39.10
 2023-06-21 23:13:35 +02:00
Andrew Mayorov
86d787eced
chore: bump hocon to 0.39.10 
Which comes with a fix for slightly more user-friendly validation error
messages.
 2023-06-21 21:25:43 +02:00
Thales Macedo Garitezi
18f0510353 fix(mongo): return health check failure reason 
Fixes https://emqx.atlassian.net/browse/EMQX-10335
 2023-06-21 15:09:37 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi
decfd6df2b feat(buffer_worker): log expired message count 
Fixes https://emqx.atlassian.net/browse/EMQX-10165

```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
 2023-06-21 14:38:51 -03:00
Zaiming (Stone) Shi
7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Zaiming (Stone) Shi
5fa87091c4 Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-21 16:08:41 +02:00
Ilya Averyanov
7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud 
chore(deps): update erlcloud and remove unnecessary overrides
 2023-06-21 15:07:45 +03:00
zhongwencool
f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type 2023-06-21 17:36:59 +08:00
Zaiming (Stone) Shi
5f47649d4d chore: downgrade log severity level of 'not_ready' nodes to info 2023-06-21 09:45:45 +02:00
Zaiming (Stone) Shi
045ed340dd fix(emqx_conf_app): fix release version detect during cluster conf sync 2023-06-21 09:28:46 +02:00
Thales Macedo Garitezi
0463828e84 refactor(gcp_pubsub): transform connector into opaque client 2023-06-20 15:27:42 -03:00
Thales Macedo Garitezi
1d791d7a8c fix(resource): validate maximum worker pool size 
Fixes https://emqx.atlassian.net/browse/EMQX-10297
 2023-06-20 14:26:42 -03:00
Thales Macedo Garitezi
ffce6fefa8 docs: note that connector state must be immutable to avoid errors 2023-06-20 13:50:53 -03:00
Zaiming (Stone) Shi
dddccfdc5c chore: bump to v5.1.0 official cut 2023-06-20 17:09:17 +02:00
Zaiming (Stone) Shi
728656d4f6 chore: bump to e5.1.0 official cut 2023-06-20 17:08:18 +02:00
Thales Macedo Garitezi
ba96c725e5 feat: add time-based pull for safety 2023-06-20 11:36:35 -03:00
Thales Macedo Garitezi
57460e83b9 test(fix): rm unused var warnings 2023-06-20 11:15:37 -03:00
Thales Macedo Garitezi
2ac2d4c037 refactor: addressing review comments 2023-06-20 11:15:13 -03:00
Ilya Averyanov
54d96488a0 chore(deps): update erlcloud and remove unnecessary overrides 2023-06-20 16:46:23 +03:00
Zaiming (Stone) Shi
b588b88a60 chore: change autoclean interval default value to 24h 
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).

To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.

This PR also has the autoclean config removed from examples.
 2023-06-20 15:43:37 +02:00
lafirest
8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example 
chore: add examples of how to configure
 2023-06-20 20:09:44 +08:00
Zaiming (Stone) Shi
f95787d229 test: make possible to test core/replica locally 2023-06-20 12:42:00 +02:00
ieQu1
e5803a61f1 fix(management): Don't wait for an empty shard 2023-06-20 10:08:41 +02:00
firest
bb2bf059ac chore: modify the build to copy examples into the release 2023-06-20 14:53:14 +08:00
zhongwencool
36ee872361 chore: hide plugins from conf load cli 2023-06-20 10:34:09 +08:00
zhongwencool
856de78698 chore: remove the is_list guard 2023-06-20 08:39:23 +08:00
zhongwencool
093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
zhongwencool
a4be9b8281 chore: bump hocon to 0.39.9 2023-06-20 08:39:23 +08:00
zhongwencool
45963b6a95 fix: ip_port schema type crash 2023-06-20 08:39:23 +08:00
Zaiming (Stone) Shi
ead511a128 chore: bump to 5.1.0-rc.1 2023-06-19 23:19:27 +02:00
ieQu1
86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5 
chore(mria): Bump version to 0.5.5
 2023-06-19 23:13:36 +02:00
Thales Macedo Garitezi
13746c2cdf fix(resource): check status when (re)starting a resource 
Fixes https://emqx.atlassian.net/browse/EMQX-10290
 2023-06-19 18:01:02 -03:00
Thales Macedo Garitezi
01d758e4c0 fix(bpapi): fix query_mode type(spec) 
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
 2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi
bb49482529 refactor(gcp_pubsub): move logging from connector to bridge 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
dabefdb01b fix(gcp_pubsub): redact request when errors occur 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
7f850f7499 fix(resource): fix query_mode/0 type and usage 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
a1690b3684 chore(gcp_pubsub): add stricter compile/xref flags 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi
5375421954 refactor(gcp_pubsub): split connector into producer and reusable parts 2023-06-19 15:59:00 -03:00
Paulo Zulato
5f10936091 feat(oracle): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 15:22:27 -03:00
Paulo Zulato
c9a2ddf98c feat(mysql): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 15:22:27 -03:00
ieQu1
f82c56ae16 chore(mria): Bump version to 0.5.5 2023-06-19 19:28:17 +02:00
JianBo He
cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8 
chore: bump to v5.1.0.alpha.8
 2023-06-19 23:00:53 +08:00
JianBo He
b0a6c18917 chore: bump vsn to 5.1.0-alpha.8 2023-06-19 22:59:05 +08:00
Paulo Zulato
9454af9a8b feat(postgresql): check whether target table exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-19 11:12:10 -03:00
Andrew Mayorov
af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags 
fix(ft-api): make schema tags look nicer in the docs
 2023-06-19 16:00:04 +02:00
Thales Macedo Garitezi
7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master 
fix(nolink_apply): avoid sending late replies to caller
 2023-06-19 10:32:43 -03:00
Andrew Mayorov
ae5846be29
fix(ft-api): make schema tags look nicer in the docs 2023-06-19 13:35:21 +02:00
zhongwencool
1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli 
chore: hide plugins from conf load cli
 2023-06-19 19:11:52 +08:00
zhongwencool
64612cab14 chore: bump to v5.1.0.alpha.8 2023-06-19 18:22:51 +08:00
zhongwencool
6efb07f6eb fix: crash on emqx_ctl listeners 2023-06-19 18:12:33 +08:00
zhongwencool
a578d6fa1d chore: release e5.1.0-alpha.8 2023-06-19 17:19:03 +08:00
zhongwencool
8b20a70344 chore: hide plugins from conf load cli 2023-06-19 15:57:40 +08:00
zhongwencool
803cd6c812
Merge pull request #11082 from emqx/release-51 
sync release-51 to master
 2023-06-19 15:12:37 +08:00
JimMoen
30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error 
fix: shared-sub with nl sub-option should cause protocol error
 2023-06-19 10:39:08 +08:00
Kjell Winblad
59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216 
fix: rule engine different behavior for div and mod
 2023-06-17 07:20:37 +02:00
Andrew Mayorov
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view 
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
 2023-06-17 00:54:24 +02:00
Thales Macedo Garitezi
50e7d5d2ec fix(nolink_apply): avoid sending late replies to caller 
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.

e.g. when a timeout occurred while resource manager was checking a resource's health:

```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```

Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277))
does), we avoid polluting the caller's mailbox with late replies.
 2023-06-16 17:03:39 -03:00
Thales Macedo Garitezi
8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard 
fix(rewrite): avoid wildcards on destination topic on publish
 2023-06-16 09:57:46 -03:00
JimMoen
178711f742
fix: shared-sub with nl sub-option should cause protocol error 2023-06-16 18:19:23 +08:00
zhongwencool
5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean 
fix: use mria.cluster_autoclean
 2023-06-16 14:56:52 +08:00
JimMoen
c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal 2023-06-16 14:04:24 +08:00
Serge Tupchii
b49b80d36c fix(emqx_schema): use non negative integer type for 'depth' SSL option 
Closes: EMQX-10276
 2023-06-16 10:59:39 +08:00
Paulo Zulato
b2f4dbd232
Merge pull request #11067 from paulozulato/chore-bump-influx 
chore(influxdb): bump version for influxdb client
 2023-06-15 20:16:28 -03:00
Paulo Zulato
e127df3912 chore(influxdb): bump version for influxdb client 2023-06-15 17:53:08 -03:00
zhongwencool
9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename 2023-06-15 20:51:52 +08:00
zhongwencool
3406a5d916
Merge pull request #11063 from zhongwencool/bump-to-alpha.7 
chore: bump to e5.1.0-alpha.7
 2023-06-15 19:22:37 +08:00
zhongwencool
fc1d78134a test: replace 'to' with 'path' 2023-06-15 19:13:51 +08:00
zhongwencool
9060a2dc05 chore: bump to e5.1.0-alpha.7 2023-06-15 19:05:13 +08:00
zhongwencool
e6c2da8d19 feat: rename log.file.to to log.file.path 2023-06-15 18:24:21 +08:00
zhongwencool
84a5d0c3c1 fix: delete a default listener then recreate it, it's disabled 2023-06-15 17:19:11 +08:00
zhongwencool
0adbb0deec feat: try to save config when post_config_update failed not in init mfa status 2023-06-15 17:19:11 +08:00
zhongwencool
c160301dbe fix: don't log enoent error 2023-06-15 17:06:09 +08:00
zhongwencool
7dade3a52c fix: tls_certfile_gc notice log don't print abspath 2023-06-15 17:06:09 +08:00
Serge Tupchii
64bbe21209 fix(emqx_schema): use non negative integer type for 'depth' SSL option 
Closes: EMQX-10276
 2023-06-15 11:22:58 +03:00
JimMoen
903d8d899e
fix: .js files are required by dashboard as static 
revert parts of 8c1334c421
 2023-06-15 16:10:16 +08:00
firest
c28346d67b chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-15 10:35:18 +08:00
zhongwencool
e42cc58694
Merge pull request #11050 from emqx/release-51 
chore: sync release-51 back to master
 2023-06-15 09:45:31 +08:00
Paulo Zulato
8c7c0877fd fix(rewrite): avoid wildcards on destination topic on publish 
Fixes https://emqx.atlassian.net/browse/EMQX-9247
 2023-06-14 20:02:03 -03:00
Andrew Mayorov
6dc281345c
test(ft-fs): run API tests with replicant nodes 2023-06-15 00:50:18 +03:00
Andrew Mayorov
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes 
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
 2023-06-15 00:50:15 +03:00
Andrew Mayorov
52354bf58a
fix(ft-fs): use emqx:running_nodes() as default cluster view 
When iterating over complete exports cluster-wide.
 2023-06-14 22:54:56 +03:00
Paulo Zulato
36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username 
fix(influxdb): check authentication
 2023-06-14 14:41:50 -03:00
zhongwencool
41004e86da
Merge pull request #10959 from qzhuyan/dev/william/remove-depre-quic-config 
chore: remove deprecated QUIC listener config
 2023-06-14 23:01:00 +08:00
zhongwencool
bd269e259e
Merge pull request #11047 from zhongwencool/dashboard-listeners-deprecate-enable 
feat: deprecated dashboard.listeners.name.enable
 2023-06-14 22:58:39 +08:00
Serge Tupchii
7882755114 chore: bump cowboy to 2.9.2 
cowboy 2.9.2 uses ranch 1.8.1-emqx that fixes incompatible TLS options
 2023-06-14 17:35:17 +03:00
Serge Tupchii
4425942030 fix(emqx_schema): move tls version gap validation to emqx_schema 2023-06-14 17:29:56 +03:00
Ivan Dyachkov
6d88cd7a20 fix(tls): fix incompatible tls options and issue with tls version gap 2023-06-14 17:29:56 +03:00
Paulo Zulato
a573b9b38c fix(influxdb): check authentication 
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.

Fixes https://emqx.atlassian.net/browse/EMQX-10213
 2023-06-14 11:27:39 -03:00
zhongwencool
9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
zhongwencool
4b82d4d927 feat: deprecated dashboard.listeners.name.enable 2023-06-14 20:51:17 +08:00
zhongwencool
267053cc35
Merge pull request #11045 from zhongwencool/hide-zones-authn-in-listeners 
feat: hide zone/authn in listeners and remove listeners's authn api
 2023-06-14 20:32:27 +08:00
Stefan Strigler
bf62cdc3e3
Merge pull request #11037 from sstrigler/EMQX-10174-e-5-1-0-reconnect-does-not-show-any-errors-for-apache-io-tdb-or-webhook-or-kafka 
fix(emqx_connector): report errors in on_start handler
 2023-06-14 13:20:47 +02:00
zhongwencool
f6df52020c test: conf_cli failed with replace 2023-06-14 18:43:41 +08:00
Andrew Mayorov
1beb588207
fix(ft): give more failproof example in README 
Without that `ssl` defaults to being disabled, and interacting
with AWS S3 with such configuration consistently times out.
 2023-06-14 13:15:00 +03:00
zhongwencool
9a0856fd6b fix: authn/authz's keys should be binary 2023-06-14 18:00:36 +08:00
zhongwencool
bd33dcb049 feat: remove zone from /clients api 2023-06-14 17:11:20 +08:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
zhongwencool
121cd9124a chore: improve show_keys output 2023-06-14 15:53:55 +08:00
zhongwencool
7668753f50 feat: hide zone/authn in listeners and remove /listeners/:id/authentication api 2023-06-14 15:17:12 +08:00
JianBo He
80e2c2c955 fix(http): avoid crash if the old sample data dont contain live_connections 
In version 5.1.0, a new metric called "live_connections" was added. However, we try to merge
some data from the Mria disk table, which may include old data formats.

Therefore, maps:get/3 is needed to avoid crashes.
 2023-06-14 15:14:20 +08:00
JianBo He
8ce0380704 chore: bump vsn to e5.1.0-alpha.6 2023-06-14 12:49:02 +08:00
JianBo He
86d3b702d7
Merge pull request #11042 from zhongwencool/max-connections-string 
fix: api_listener crash when setting max_connnections as string
 2023-06-14 12:46:40 +08:00
zhongwencool
feb08b1181 chore: update 11042 changelog 2023-06-14 11:40:20 +08:00
zhongwencool
091c49277f fix: api_listener crash when setting max_connnections as string 2023-06-14 11:26:41 +08:00
JianBo He
26ad06b9ca chore: update changes 2023-06-14 10:49:17 +08:00
JianBo He
c4c32f5032 chore: cover stomp mountpoint tests 2023-06-14 10:49:17 +08:00
JianBo He
4212d90672 fix(stomp): unmount topic prefix in delivering proccess 2023-06-14 10:49:17 +08:00
JianBo He
e3d208f678 fix(gw): close the stomp connections once an error frame occured 
According to the Stomp v1.2 specification:

> The server MAY send ERROR frames if something goes wrong. In this case,
> it MUST then close the connection just after sending the ERROR frame

Additional, fixes the `is_superuser` is not working for all gateways
 2023-06-14 10:49:17 +08:00
zhongwencool
d41a23b7d0
Merge pull request #11036 from zhongwencool/conf-load-cli 
feat: support --replace|merge with conf load
 2023-06-14 10:48:13 +08:00
zhongwencool
7f0f40cb58
Merge pull request #11034 from zhongwencool/mqtt-conf 
feat:  move shared_subscription_strategy from broker to mqtt
 2023-06-14 10:00:38 +08:00
zhongwencool
9b5c7065f2 feat: conf reload support --replace|merge 2023-06-14 09:12:39 +08:00
zhongwencool
095fb040c0 chore: format conf cli usage text 2023-06-14 09:12:39 +08:00
zhongwencool
a8959900d0 feat: support --replace|merge with conf load 2023-06-14 09:12:39 +08:00
zhongwencool
f57ff802e4
Merge pull request #11038 from thalesmg/fix-pulsar-get-status-r51 
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
 2023-06-14 08:32:25 +08:00
Thales Macedo Garitezi
25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 13:54:57 -03:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
Thales Macedo Garitezi
b609792a90 fix(pulsar_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10278

Since Pulsar client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 11:44:45 -03:00
zhongwencool
946de4a303 chore: dialyzer warning 2023-06-13 20:11:49 +08:00
zhongwencool
50041fdddf feat: hide broker rootkey and move broker.shared_subscription_strategy to mqtt.shared_subscription_strategy 2023-06-13 19:35:52 +08:00
zhongwencool
beee35bdea
Merge pull request #11033 from HJianBo/fix-exproto-mountpoint 
feat(exproto): abandon the mountpoint field in AuthenticateRequest
 2023-06-13 19:18:17 +08:00
JianBo He
7afd9d2b06 chore: update changes 2023-06-13 16:38:23 +08:00
firest
536f7ba423 chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-13 16:06:28 +08:00
JianBo He
7684cefa86 feat(exproto): abandon the mountpoint field in AuthenticateRequest 
This field was introduced in v4.x, but in fact, in 5.0 we have provided
`gateway.exproto.mountpoint` for configuration, so there is no need to override
it through the Authenticate request
 2023-06-13 14:10:14 +08:00
William Yang
8fa743fa22 chore: mark deprecated QUIC listener config for 5.1 2023-06-13 06:00:57 +02:00
zhongwencool
3331af4757 feat: make gateway and slow_subs low level and hide rule_engine 2023-06-13 11:47:38 +08:00
Thales Macedo Garitezi
5ef7102cba fix(listener_mgmt_api): use union member selector fn for better error messages (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10247
 2023-06-12 16:48:13 -03:00
Thales Macedo Garitezi
73fe671dba test: rm unused var warnings 2023-06-12 15:19:47 -03:00
zhongwencool
58dc042279 chore: bump to e/v5.1.0-alpha.5 2023-06-13 00:06:54 +08:00
Thales Macedo Garitezi
c11011d857
Merge pull request #11024 from thalesmg/pulsar-cosmetic-connecting-check-r51 
feat(pulsar): retry health check a bit before returning (r5.1)
 2023-06-12 13:00:54 -03:00
Kjell Winblad
cf31b65076 fix: rule engine different behavior for div and mod 
Previously, the div operation could only be used as an infix operation
while mod could only be used as a function call. After this commit, one
can use both div and mod using function call syntax and infix syntax.

Fixes: https://emqx.atlassian.net/browse/EMQX-10216
 2023-06-12 17:20:52 +02:00
SergeTupchiy
5ef19ecf58
Merge pull request #10993 from SergeTupchiy/improve_mnesia_backup_import 
fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
 2023-06-12 17:26:26 +03:00
Thales Macedo Garitezi
741c1f091e fix(pulsar): update pulsar -> 0.8.3 
Fixes https://emqx.atlassian.net/browse/EMQX-10229

See https://github.com/emqx/pulsar-client-erl/pull/59

Fixes a case_clause error that could arise from race conditions.
 2023-06-12 10:29:40 -03:00
Thales Macedo Garitezi
db5d14d5bf feat(pulsar): retry health check a bit before returning (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10228

This is a cosmetic fix for the Pulsar Producer bridge health check status.

Pulsar connection process is asynchronous, therefore, when a bridge of this type is
created or updated (which is the same as stopping and re-creating it), the immediate
status will be connecting because it’s indeed still connecting.  The bridge will connect
very soon afterwards (assuming there are no true network/config issue), but having to
refresh the UI to see the new status and/or seeing the resource alarm might annoy users.

This workaround adds a few retries to account for that effect to reduce the probability of
seeing the `connecting` state on such happy-paths.
 2023-06-12 10:26:07 -03:00
firest
0ed420f81f fix: obfuscated the value of JWT 2023-06-12 18:01:02 +08:00
Ilya Averyanov
93733c1ccc fix(docs): fix typo in ft README 2023-06-12 11:38:17 +02:00
zhongwencool
0fc10ad239
Merge pull request #10984 from zhongwencool/reload-cli 
feat: support reload cli
 2023-06-12 17:21:29 +08:00
Zaiming (Stone) Shi
12b73ba1c0 Merge remote-tracking branch 'origin/master' into release-51 2023-06-12 10:35:47 +02:00
firest
947d99218b chore: upgrade emqtt to avoid sensitive data leakage in the debug log 2023-06-12 16:32:24 +08:00
Ilya Averyanov
bf2c768182
Merge pull request #11010 from zhongwencool/fs-delete-SUITE-data 
chore: delete emqx_utils_fs_SUITE_data dir
 2023-06-12 10:40:48 +03:00
JianBo He
d14d87b443
Merge pull request #10955 from HJianBo/clean-mqttsn-topic-registry 
Fix(mqttsn): clean predefined topics once gateway unload
 2023-06-12 09:37:47 +08:00
Zaiming (Stone) Shi
e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb 
fix(mongodb): hide batch_size for mongodb resource
 2023-06-11 21:26:12 +02:00
zhongwencool
89ddd54c68
Merge pull request #11013 from HJianBo/fix-max-conns-in-gateway-lists-api 
fix(gateway): fix 500 crash for '/gateways' endpoint
 2023-06-11 22:29:24 +08:00
JianBo He
57a3b6fdda fix(gateway): fix 500 crash for '/gateways' endpoint 
It introduced by https://github.com/emqx/emqx/pull/10961
 2023-06-11 18:43:38 +08:00
zhongwencool
1550158ddd fix: update emqx_flapping process's state when emqx:update_config([flapping_detect],Conf) 2023-06-11 18:23:15 +08:00
JianBo He
001e2976ec chore: fix typos 2023-06-11 18:13:25 +08:00
JianBo He
9c4a576fb3 feat: reserved 1-1024 topic id for predefined topics 2023-06-11 18:12:34 +08:00
JianBo He
ac6693c8cc refactor(mqttsn): takeover can resume the registrations of session 2023-06-11 18:12:34 +08:00
JianBo He
3c6afee690 refactor(mqttsn): make the topic registration mechanism simpler 2023-06-11 18:12:34 +08:00
JianBo He
9760181ed9 test: cover mqttsn predefined topics 2023-06-11 18:12:34 +08:00
JianBo He
6fc50d11f8 fix(mqttsn): clean pre-defined topics once gateway reloaded 2023-06-11 18:12:34 +08:00
Zaiming (Stone) Shi
654a9dcab5
Merge pull request #11006 from zmstone/0610-fix-quic-default-certs-path 
fix(quic): environment variables as cert file prefix for quic listener
 2023-06-11 11:52:56 +02:00
zhongwencool
c8cf8d1def
Merge pull request #11009 from zhongwencool/persistent_session_store-default-value 
chore: correct persistent_session_store's default value type to boolean
 2023-06-11 17:26:58 +08:00
Zaiming (Stone) Shi
ddef751527 fix(mongodb): hide batch_size for mongodb resource 
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
 2023-06-11 11:08:58 +02:00
zhongwencool
a10107045f test: delete emqx_ft_fs_utils_SUITE_data from source code 2023-06-11 15:50:45 +08:00
zhongwencool
aed64aae60 chore: delete emqx_utils_fs_SUITE_data dir 2023-06-11 15:50:45 +08:00
zhongwencool
072f6fac8d chore: update hocon to 0.39.8 2023-06-11 09:11:17 +08:00
zhongwencool
346b363cb5 test: config SUITE failed 2023-06-11 09:11:17 +08:00
zhongwencool
505d677685 chore: move reload_local_etc_config from emqx_conf to emqx_conf_cli 2023-06-11 09:11:17 +08:00
zhongwencool
1ca1ba9e7a fix: get_sources return default authz when sources key not found 2023-06-11 09:11:17 +08:00
zhongwencool
01671d82c7 feat: delete authorization form etc/emqx.conf 2023-06-11 09:11:17 +08:00
zhongwencool
6c6ff4ea1f chore: rename {mod} to module 2023-06-11 09:11:17 +08:00
zhongwencool
cf9a207743 feat: check all config schema before loading 2023-06-11 09:11:17 +08:00
zhongwencool
b829d8edc2 feat: support --auth-chains merge|replace mod 2023-06-11 09:11:17 +08:00
zhongwencool
21ffd958b4 feat: support authn merge_authenticators 2023-06-11 09:11:17 +08:00
zhongwencool
aac5b85d26 feat: authz support merge sources 2023-06-11 09:11:17 +08:00
zhongwencool
994db58cbf fix: load authz's default sources 2023-06-11 09:11:17 +08:00
zhongwencool
c819ac27f4 feat: add conf reload cli 2023-06-11 09:11:17 +08:00
zhongwencool
61bbe19eba
Merge pull request #10961 from HJianBo/support-infinity-max-conns-in-gateway 
Adds support for unlimited max connections for gateway listeners
 2023-06-11 09:09:34 +08:00
zhongwencool
599eb9fef5 chore: correct persistent_session_store's default value type to boolean 2023-06-10 19:36:05 +08:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi
9f135d1f2b fix(quic): environment variables as cert file prefix for quic listener 2023-06-10 12:23:20 +02:00
JianBo He
aea3ae4024
Merge pull request #11005 from HJianBo/fix-tracing-logs-typos 
fix(authn): fix the hardcode method in trace logs
 2023-06-10 18:08:28 +08:00
Zaiming (Stone) Shi
3901ed3255
Merge pull request #10996 from sstrigler/EMQX-9435-request-the-non-existent-api-v-4-nodes-endpoint-in-emqx-e-5-0-1-returns-http-code-200 
Return 404 for everything we don't know
 2023-06-10 11:20:25 +02:00
Zaiming (Stone) Shi
f98cdd4983
Merge pull request #10994 from sstrigler/EMQX-10003-e-5-0-4-auth-header-value-of-webhook-data-bridge-can-be-found-in-emqx-log 
fix(emqx_utils): redact proxy-authorization headers
 2023-06-10 11:18:58 +02:00
JianBo He
fdaac9800c fix(authn): fix the hardcode method in trace logs 2023-06-10 13:04:24 +08:00
Serge Tupchii
48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option 
Closes: EMQX-8725
 2023-06-09 22:10:17 +03:00
Kjell Winblad
43292c06e4
Merge pull request #10997 from kjellwinblad/kjell/fix/clickhouse_lost_messages/EMQX-10217 
fix: lost messages when clickhouse closes while sending messages
 2023-06-09 21:08:25 +02:00
Serge Tupchii
87b57112df refactor: move tcp keepalive options helper to emqx_utils 2023-06-09 21:43:50 +03:00
Paulo Zulato
7c63f20aa8
Merge pull request #10999 from paulozulato/fix-kafka-time-field-too-large 
fix(kafka): fix timeout field for some kafka fields
 2023-06-09 15:29:59 -03:00
Paulo Zulato
720a84d48b fix(kafka): fix timeout field for some kafka fields 
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
 2023-06-09 13:38:04 -03:00
Kjell Winblad
1b3af2ac91 refactor: nicer handling of errors and warning log for recoverable errors 2023-06-09 17:22:16 +02:00
Stefan Strigler
57d72ed23e fix(emqx_utils): redact proxy-authorization headers 2023-06-09 17:13:53 +02:00
Stefan Strigler
8c1334c421 fix(emqx_dashboard): return 404 for everything we don't know 2023-06-09 17:13:18 +02:00
Andrew Mayorov
b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder 
refactor: tear `emqx_plugin_libs` application apart
 2023-06-09 17:02:14 +02:00
Thales Macedo Garitezi
4a9f2247ca test(conf_app): fix bad merge conflict 2023-06-09 11:06:54 -03:00
Kjell Winblad
61488ec407 fix: lost messages when clickhouse closes while sending messages 
Fixes:
https://emqx.atlassian.net/browse/EMQX-10217
 2023-06-09 16:04:38 +02:00
Thales Macedo Garitezi
40675843a1 fix(iotdb): remove root request_timeout option 
Fixes https://emqx.atlassian.net/browse/EMQX-10175

Since IoTDB bridge is not used as an authn/authz source and reuses the
webhook connector, we should have only one request timeout field (the
`resource_opts.request_ttl` one) like the webhook bridge does, to
avoid confusion and potentially bad configuration combinations.
 2023-06-09 10:39:01 -03:00
Zaiming (Stone) Shi
643da0564b
Merge pull request #10990 from zmstone/0609-fix-schema-for-dashboard-use 
0609 fix schema for dashboard use
 2023-06-09 15:07:51 +02:00
Thales Macedo Garitezi
20a7ce64cd
Merge pull request #10919 from thalesmg/test-flakiness-20230601-a 
test(pulsar_producer): attempt to fix flaky test
 2023-06-09 09:30:30 -03:00
Andrew Mayorov
c8360f86b7
chore: bump application versions 2023-06-09 14:44:38 +03:00
Andrew Mayorov
830ba54721
refactor(pluglib): remove emqx_plugin_libs application 2023-06-09 14:44:37 +03:00
Andrew Mayorov
e6fb0203b4
refactor(pluglib): move connectivity checks to emqx_connector_lib 2023-06-09 14:44:37 +03:00
Andrew Mayorov
8919a6ef93
refactor(pluglib): provide SQL related utils in emqx_utils_sql 2023-06-09 14:44:37 +03:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Andrew Mayorov
7d0abb6146
feat(emqx): add emqx_topic:match_any/2 utility 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app 
Also make user that existing code calls it directly.
 2023-06-09 14:44:36 +03:00
Serge Tupchii
2aad6671ac fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file 
As we use `keep_tables` option, we don't need to modify a backup schema before importing the backup file.
With this option, `mnesia:restore/2` ignores backup schema and keeps current table schema unchanged.
 2023-06-09 14:41:21 +03:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup 
feat: implement configuration and user data export/import
 2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
Kjell Winblad
4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631 
feat: add sync/async option to the Kafka producer bridge
 2023-06-09 12:51:23 +02:00
Kjell Winblad
2671e8ecf9 fix: dialyzer type problem 2023-06-09 11:00:05 +02:00
Ilya Averyanov
c2635e938b
Merge pull request #10974 from keynslug/ft/EMQX-9523/backend-enable-flag 
fix(ft): add `enable` flag to every backend
 2023-06-09 11:44:37 +03:00
JianBo He
53eb8d7f1b test: fix flaky tests introduced by https://github.com/emqx/emqx/pull/10948 2023-06-09 15:48:23 +08:00
JianBo He
72311a546b chore: more clear funcation name 2023-06-09 15:48:23 +08:00
JianBo He
6a05663bd5 chore: ensure default value's type be the first one in the union type 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-06-09 15:48:23 +08:00
JianBo He
dbb8742e0e test: cover max_connection in infinity value 2023-06-09 15:48:23 +08:00
JianBo He
8df7b1a1be fix(gateway): fix http-api 500 issue if setting max_connections to infinity 2023-06-09 15:48:23 +08:00
JianBo He
c450a1784a
Merge pull request #10950 from HJianBo/mqttsn-qos3 
fix(mqttsn): checking enable_qos3 option
 2023-06-09 15:46:55 +08:00
JianBo He
2153cd4317
Merge pull request #10979 from HJianBo/fix-coap-connectionless-pub 
fix(coap): fix crash publish/subscribe in connectionless mode
 2023-06-09 15:46:18 +08:00
Zaiming (Stone) Shi
38e767b808 fix(schema): map new duration types to dashboard's type name 2023-06-09 09:14:55 +02:00
Stefan Strigler
2e9ac8f1ec
Merge pull request #10980 from sstrigler/EMQX-10189-mark-support-for-json-output-format-in-prometheus-stats-as-deprecated 
Mark support for json output format in prometheus stats as deprecated
 2023-06-09 09:00:34 +02:00
Zaiming (Stone) Shi
ee697831f6
Merge pull request #10926 from zmstone/0602-add-enable-as-an-alias-for-enabled 
0602 add enable as an alias for enabled
 2023-06-09 08:44:20 +02:00
Zaiming (Stone) Shi
9f6beee6b3
Merge pull request #10936 from zmstone/0603-imporove-desc 
docs: refine some desc
 2023-06-09 08:44:03 +02:00
Zaiming (Stone) Shi
af5c6720de Merge remote-tracking branch 'origin/master' into release-51 2023-06-09 08:37:29 +02:00
lafirest
601db2b1d3
Merge pull request #10988 from lafirest/fix/konservi_sentema 
fix(resource): improve log security when resource creation fails
 2023-06-09 13:46:12 +08:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
zhongwencool
daa5450218
Merge pull request #10985 from zmstone/0608-rename-cluster-call-cmd 
0608 rename cluster call cmd
 2023-06-09 10:56:18 +08:00