0877e4296a
fix(dashboard): add missing function clause
...
Example error:
https://github.com/emqx/emqx/actions/runs/5045715277/jobs/9052482682#step:8:294
The previous change was not equivalent to the previous `maps:with/2`
behavior.
2023-05-22 11:35:04 -03:00
65f973044f
feat(pulsar): improve authn error check time and add connect timeout
...
Fixes https://emqx.atlassian.net/browse/EMQX-9910
2023-05-22 11:33:16 -03:00
3f882ddcc6
Merge pull request #10751 from ci/ct-flap-listener-api
...
fix(listen): wait until port is free when stopping ranch listeners
2023-05-22 17:00:45 +03:00
88e4078885
Merge pull request #10747 from lafirest/fix/rule_funs_time_offset
...
fix: port the `emqx_calendar` from v4.4
2023-05-22 20:52:21 +08:00
0b9a32f49a
chore: add BSL in apps/emqx_enterprise
2023-05-22 14:51:27 +02:00
40e8d5d039
refactor: rename lib-ee/emqx_ee_conf to apps/emqx_enterprise
2023-05-22 14:51:27 +02:00
3482c2c849
refactor: redefine default value for 'node.applications' in enterprise
2023-05-22 14:51:26 +02:00
42f9c9aa9a
Merge pull request #10746 from lafirest/fix/events_api
...
fix: supports test the `$events/delivery_dropped` event by API
2023-05-22 20:50:03 +08:00
d22541e8b3
fix(ft): correct mistyped option in README
2023-05-22 15:32:08 +03:00
0559d6f639
refactor(buffer_worker): use static fn for bumping counters
2023-05-22 09:12:08 -03:00
9c71f4ecbd
test: fix flaky test
2023-05-22 09:12:00 -03:00
c74c93388e
refactor: rename some variables and sum type constructors for clarity
2023-05-22 09:11:23 -03:00
7d798c10e9
perf(buffer_worker): flush metrics periodically inside buffer worker process
...
Fixes https://emqx.atlassian.net/browse/EMQX-9905
Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
2023-05-22 09:11:23 -03:00
092e112d62
Merge pull request #10767 from ci/ft-prop-flaky
...
test(ft): make proptest less aggressive
2023-05-22 12:45:24 +03:00
23542d1262
fix(listen): wait until port is free when stopping ranch listeners
...
Simple `cowboy:stop_listener/1` will not close the listening socket
explicitly, it was the source of test flaps before this fix.
2023-05-22 12:29:40 +03:00
cb5a596c57
fix: make sure swagger.json is fully generated
2023-05-22 16:58:13 +08:00
5b312a415b
Merge pull request #10759 from ci/ct-flap-clickhouse
...
fix(clickhouse): start app in tests
2023-05-22 11:11:15 +03:00
966f748551
test(ft): make proptest less aggressive
...
So that the chance of getting huge coverage and as a result, high
chance of running out of memory, is reduced.
2023-05-22 10:45:27 +03:00
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path
...
perf(config): enforcing atom key path in hotcode path
2023-05-22 09:38:09 +02:00
087dc59115
test: add test cases for the `rule_test` API
2023-05-22 15:25:35 +08:00
218fc4a839
refactor: add emqx_authz_file validate function
2023-05-22 14:54:23 +08:00
082214d039
feat: add authz file rule validator
2023-05-22 14:25:05 +08:00
27fca0ef3c
fix: check authz's file rule before save to file
2023-05-22 14:22:20 +08:00
c65e3a46b1
Merge pull request #10742 from zhongwencool/fix-authz-file-rule-check
...
fix: check authz's file source rules in pre_config_update
2023-05-22 14:15:18 +08:00
ce3ec351c7
chore: remove unused conf comments
2023-05-22 14:01:49 +08:00
96e7005de8
refactor: add emqx_authz_file validate function
2023-05-22 11:03:23 +08:00
56a6b699ac
fix: port the `emqx_calendar` from v4.4
2023-05-22 09:55:31 +08:00
7c2aac64bc
fix: bad cert file path in dashboard https listener
2023-05-21 07:41:46 +08:00
3e98b3b050
Merge pull request #10755 from zmstone/0517-fix-bridge-update-timeout-issue
...
0517 fix bridge update timeout issue
2023-05-20 06:19:26 +02:00
ba6b208df2
fix(clickhouse): start app in tests
...
Otherwise, depending on the test execution order, tests might
sometimes fail.
Moreover, ensure that applications describe their dependecies
correctly and avoid starting irrelevant apps in tests.
2023-05-19 23:08:40 +03:00
bb4fef3cfd
Merge pull request #10752 from thalesmg/fix-pulsar-redact-auth-r50
...
fix(pulsar): redact error reason
2023-05-19 16:47:05 -03:00
cb76e5a241
docs: add changelog for 10755
2023-05-19 20:41:26 +02:00
0d8ffc0d59
fix(resource-manager): ensure no false creation
...
Update is implemented as remove + create.
If a dleete call is made while the create is in progress
the remove call is likely to timeout too.
This causes the follwing creation to falsely succeed,
because there is alreay a running child under the supervisor.
As a result, the resource is permanently removed after
resource_manager eventually handles the remove call.
2023-05-19 18:55:16 +02:00
f5e5c59763
refactor(resource-manager-sup): do not force kill resource manager
...
the shutdown timeout is now set to infinity so it will never
force kill a resource manager, otherwise there will be
resource leaks
2023-05-19 18:55:16 +02:00
21de0f8274
fix(buffer-worker-sup): fast stop
...
the timeout shutdown in child spec may
significantly slow down the deletion of a resource
this commit chagnes the shutdown to brutal kill
also, the pool worker removal code has been delete
because it's not necessary since the entier pool is
going to be force-delete later anyway
2023-05-19 18:55:16 +02:00
be90c63c78
chore(mqtt-connector): refine logging level
...
connect failure should be at warning level but not error,
the connecting state is visiable from dashbaord
also the resource manager logs connection failures in general
at warning level
2023-05-19 18:55:16 +02:00
5bbcf4b712
fix(mqtt-connector): faster shutdown
2023-05-19 18:55:16 +02:00
7eef86363a
test: make erlfmt happy
2023-05-19 17:58:14 +02:00
9aaa0b6fe1
Merge pull request #9927 from emqx/file-transfer
...
feat: implement file transfer over mqtt
2023-05-19 18:35:13 +03:00
4327e40740
fix(pulsar): redact error reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-9940
Error log after fix:
```
2023-05-19T13:09:26.304769+00:00 [error] msg: failed_to_start_pulsar_client, mfa: emqx_bridge_pulsar_impl_producer:on_start/2, line: 104, instance_id: <<"bridge:pulsar_producer:pprodu">>, pulsar_hosts: pulsar://pulsar:6652, reason: {#{{"pulsar",6652} => #{error => 'AuthenticationError',message => "Unable to authenticate",request_id => 18446744073709551615}},{child,undefined,'pulsar_producer:pprodu:emqx@127.0.0.1',{pulsar_client,start_link,['pulsar_producer:pprodu:emqx@127.0.0.1',["pulsar://pulsar:6652"],#{conn_opts => #{auth_data => <<"******">>,auth_method_name => <<"basic">>},ssl_opts => []}]},transient,false,5000,worker,[pulsar_client]}}
```
2023-05-19 10:09:57 -03:00
264404e180
Merge pull request #10511 from lafirest/fix/resource_safe_log
...
fix(resource): make some logging of the resource manager more secure
2023-05-19 20:42:21 +08:00
1aa904d019
Merge pull request #10738 from lafirest/fix/tdengine_template
...
fix(tdengine): add supports for the `super table` feature in the SQL template
2023-05-19 20:41:50 +08:00
c440cd77b0
fix(test): prefer `emqx_connector_http:join_paths/2` in api tests
...
It is more consistent with what HTTP servers are expecting, plus
not OS-specific as `filename:join/2`.
2023-05-19 15:37:01 +03:00
8cfcc8fbaf
Merge pull request #10743 from thalesmg/fix-bridge-metric-aggregation-r50
...
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
2023-05-19 09:07:22 -03:00
6cb9efd7d3
feat: add authz file rule validator
2023-05-19 18:41:59 +08:00
5c112602b8
test(evict): fix flapping testcase
...
Postpone enabling eviction agent so that target connection will not
get evicted too early.
2023-05-19 13:37:36 +03:00
28dda39e39
Merge pull request #10724 from kjellwinblad/kjell/docs/missing_http_api_endpoint_descriptions/EMQX-9767
...
docs: make sure there is a summary for all API endpoints
2023-05-19 12:05:41 +02:00
baeb96a6e4
chore: update changes
2023-05-19 15:36:18 +08:00
0eea8438bf
fix(resource): make some logging of the resource manager more secure
2023-05-19 15:28:19 +08:00
fa799e95a4
fix: supports test the `$events/delivery_dropped` event by API
2023-05-19 14:20:26 +08:00
5a08a7b9de
fix(tdengine): minor improvement of code and changes
2023-05-19 11:14:31 +08:00
48ceb14755
Merge pull request #10737 from HJianBo/fix-mqtt-sn-bugs
...
Avoid urldecode clientid twice
2023-05-19 09:19:12 +08:00
9c15720fee
Merge pull request #10726 from paulozulato/fix-large-health-check-interval
...
fix: validate range for some bridge options
2023-05-18 19:25:24 -03:00
9c48b016a9
fix(evacuation): handle expire interval correctly
2023-05-18 22:34:54 +03:00
09ea2e2224
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
...
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
52e2caa671
fix: check authz's file rule before save to file
2023-05-18 22:33:12 +08:00
56634e5238
Merge pull request #10725 from thalesmg/bump-gproc-091-r50
...
chore: bump gproc -> 0.9.0.1 (r5.0)
2023-05-18 10:38:06 -03:00
49424cc9a2
Merge pull request #10733 from thalesmg/fix-pulsar-sensitive-struct-r50
...
fix(pulsar): mark whole auth struct as sensitive, update dashboard version and improve bridge type matching errors (r5.0)
2023-05-18 10:15:10 -03:00
5d289ade56
fix: validate range for some bridge options
...
Fixes https://emqx.atlassian.net/browse/EMQX-9864
Setting a very large interval can cause `erlang:start_timer` to crash.
Also, setting auto_restart_interval or health_check_interval to "0s"
causes the state machine to be in loop as time 0 is handled separately:
| state_timeout() = timeout() | integer()
| (...)
| If Time is relative and 0 no timer is actually started, instead the the
| time-out event is enqueued to ensure that it gets processed before any
| not yet received external event.
from "https://www.erlang.org/doc/man/gen_statem.html#type-state_timeout "
Therefore, both fields are now validated against the range [1ms, 1h],
which doesn't cause above issues.
2023-05-18 10:10:58 -03:00
f1a3e5965e
chore: update apps version && changes
2023-05-18 17:34:28 +08:00
142125b9e4
test(tdengine): add test cases to cover the super table feature
2023-05-18 17:01:27 +08:00
26b69a5591
test(mqttsn): add test cases
2023-05-18 16:16:09 +08:00
6ff77b221b
fix(tdengine): add supports for the `automatically create` feature in the SQL template
2023-05-18 15:13:08 +08:00
c5398ab651
fix(gateway): avoid uri_decode twice
...
The url path has beed decoded in
0ebceb432f/src/cowboy_router.erl (L324-L330)
2023-05-18 14:55:12 +08:00
d77d5e33bc
fix(mqttsn): Instantly refresh client info after subscribed/unsubscribed
2023-05-18 14:48:37 +08:00
ba43a0e30f
Merge pull request #10734 from thalesmg/merge-r50-into-v50-a
...
merge `release-50` into `master`
2023-05-18 14:24:53 +08:00
bf5ee41009
Merge pull request #10702 from zhongwencool/keepalive-backoff-rename
...
feat: deprecated keepalive_backoff, introduce keepalive_multiplier
2023-05-18 10:00:17 +08:00
d4d25d2660
chore: don't lost previous's statval
2023-05-18 09:49:11 +08:00
5d5c16a56d
feat(bridges): use union member selector function for better error messages
2023-05-17 17:57:29 -03:00
659cf64ad7
feat(pulsar): use an union member selector for better error messages
2023-05-17 17:56:53 -03:00
f83f9a2d79
chore: bump app vsns
2023-05-17 14:56:11 -03:00
447b76464b
Merge branch 'release-50' into merge-r50-into-v50-a
2023-05-17 14:50:18 -03:00
10f6edd6ea
Merge pull request #10717 from thalesmg/fix-bw-flush-timer-full-infl-r50
...
fix(buffer_worker): avoid setting flush timer when inflight is full
2023-05-17 14:49:27 -03:00
727ad59995
chore(test): improve coverage
2023-05-17 16:58:49 +02:00
dcccc0910a
fix(pulsar): mark whole auth struct as sensitive (r5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9900
I tried to patch hocon itself to filter the sensitive data, but the
way it's currently structured doesn't seem to keep that field
metadata. So, for now, we can just mark the whole auth union as
sensitive.
2023-05-17 11:50:22 -03:00
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
...
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
060efd6964
chore: bump gproc -> 0.9.0.1 (r5.0)
...
Includes this fix: https://github.com/uwiger/gproc/pull/193
2023-05-17 11:05:44 -03:00
e26ce5816e
feat: hide keepalive_backoff, introduce keepalive_multiplier
2023-05-17 21:25:10 +08:00
ea8ac877b0
Merge pull request #10713 from zhongwencool/put-webhook-request-timeout-into-resource-opts
...
feat: update wehbook's request_timeout into resource_opts
2023-05-17 21:20:42 +08:00
bc7d0d5fd6
perf(webhook): evaluate replies in fresh process rather than in ehttpc
...
This surprisingly simple change yields a big performance improvement
in throughput.
While the previous commit achieves ~ 55 k messages / s
in throughput under some test conditions (100 k concurrent publishers
publishing 1 QoS 1 message per second), the simple change in this
commit improves it further to ~ 63 k messages / s.
Benchmarks indicated that the evaluating one reply function is
consistently quite fast (~ 20 µs), which makes this performance gain
counterintuitive. Perhaps, although each call is cheap, `ehttpc`
calls several of these in a row when there are several sent requests,
and those costs might add up in latency.
2023-05-17 09:20:50 -03:00
a7b41e1cdf
perf(webhook): add retry attempts for async
...
This is a performance improvement for webhook bridge.
Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well. Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.
This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
01cf7fc703
feat(webhook): use clientid for hash-loadbalancing ehttpc workers
2023-05-17 09:20:50 -03:00
c95ef71fb5
chore(ft): provide more details in README.md
2023-05-17 12:49:20 +03:00
397dce0d78
Merge pull request #10654 from kjellwinblad/kjell/refactor/clickhouse_bridge/EMQX-9538
...
refactor: the clickhouse bridge to use the new bridge app structure
2023-05-17 11:19:33 +02:00
cdf42760fa
chore: little change avoid atom leak
2023-05-17 11:01:03 +02:00
bce43978a2
refactor: the clickhouse bridge to use the new bridge app structure
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-9538
2023-05-17 10:12:43 +02:00
ab8b8ccaad
docs: make sure there is a summary for all API endpoints
...
Many HTTP API endpoints did not have a summary when looking at the API
docs at "http://emqx_host_name:18083/api-docs ". This has been fixed by
making sure there is a summary corresponding to all endpoints.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9767
2023-05-17 09:58:56 +02:00
c54d04450c
Merge pull request #10723 from savonarola/0516-improve-rebalance-logging
...
Improve rebalance logging
2023-05-17 09:21:10 +03:00
2b99a9b988
feat: hide resource_opts's request_timeout
2023-05-17 13:41:11 +08:00
b5da9eb9ad
Merge pull request #10728 from lafirest/fix/port_ruleengine_export_colums
...
fix: cannot access columns exported by FOREACH in DO clause
2023-05-17 11:46:17 +08:00
1f7ede90a4
chore: update app version && changes
2023-05-17 11:01:31 +08:00
18043150be
fix: cannot access columns exported by FOREACH in DO clause
2023-05-17 10:55:47 +08:00
0dd3325d70
Merge pull request #10711 from lafirest/fix/fix_infinity_bucket
...
fix(limiter): a bucket with an infinity rate shouldn't be added to limiter server
2023-05-17 10:40:59 +08:00
b71955e368
fix(ft): bump application versions
2023-05-17 00:29:15 +03:00
1a8cf0e392
Merge branch 'release-50' into file-transfer
...
* release-50:
fix(pulsar): use a binary duration as default `health_check_interval`
docs: add changelog entry
docs: clarify description of bridge username and password
chore: bump to v5.0.25
fix(limiter): adjust type for compatibility
fix(limiter): fix that update node-level limiter config will not working
chore: upgrade dashboard to v1.2.4-1 for ce
chore: upgarde rulesql to 0.1.6 to fix invaid utf8 input
chore: add changelog for 10659
fix: crash when sysmon.os.mem_check_interval = disabled
chore: bump influxdb version && update changes
refactor(influxdb): move influxdb bridge into its own app
chore: add listener default changelog
fix: ocsp cache SUITE failed
fix: ensure atom key for emqx_config:get
fix: only fill cerf_file default in server side
fix: authn init is empty
fix: bad listeners default ssl_options
2023-05-17 00:27:55 +03:00
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field
...
0421 refactor delete unused schema field
2023-05-16 22:22:03 +02:00
85089a3210
fix(buffer_worker): correctly flush the buffer workers when inflight table room is made
...
The previous commit uncovered another bug that was hidden by it:
`maybe_flush_after_async_reply` was sending a message to the wrong
PID. It was sending a message to `self()` meaning to target a buffer
worker, but `self()` in that context is never the buffer worker, it's
the connector's worker.
This change also revealed a race condition where the buffer workers
could stop flushing messages. So we piggy-backed on the atomic update
of the table size count to check if the buffer worker should be poked
to continue flushing. This allows us to get rid of
`maybe_flush_after_async_reply` altogether.
2023-05-16 17:15:42 -03:00
b2ecbef0f1
Merge pull request #10715 from fix/EMQX-9897/preserve-peercert-until-connected
...
fix(chan): postpone trimming conninfo after `connected` hook run
2023-05-16 22:56:41 +03:00
3cd95f40e5
test(chan): verify hooks receive peercert until connected
2023-05-16 19:35:56 +03:00
967b2e72e0
test(emqx): remove `peercert` from clientinfo fixtures
...
According to typespec, there's no place for `peercert` in
`clientinfo()`.
2023-05-16 19:35:55 +03:00
Thales Macedo Garitezi