Commit Graph

381 Commits

Author SHA1 Message Date
firest baeb96a6e4 chore: update changes 2023-05-19 15:36:18 +08:00
JianBo He 48ceb14755
Merge pull request #10737 from HJianBo/fix-mqtt-sn-bugs
Avoid urldecode clientid twice
2023-05-19 09:19:12 +08:00
Thales Macedo Garitezi 09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
某文 52e2caa671 fix: check authz's file rule before save to file 2023-05-18 22:33:12 +08:00
Paulo Zulato 5d289ade56 fix: validate range for some bridge options
Fixes https://emqx.atlassian.net/browse/EMQX-9864

Setting a very large interval can cause `erlang:start_timer` to crash.
Also, setting auto_restart_interval or health_check_interval to "0s"
causes the state machine to be in loop as time 0 is handled separately:

| state_timeout() = timeout() | integer()
| (...)
| If Time is relative and 0 no timer is actually started, instead the the
| time-out event is enqueued to ensure that it gets processed before any
| not yet received external event.
from "https://www.erlang.org/doc/man/gen_statem.html#type-state_timeout"

Therefore, both fields are now validated against the range [1ms, 1h],
which doesn't cause above issues.
2023-05-18 10:10:58 -03:00
JianBo He f42cab9c56 chore: update changes 2023-05-18 16:24:37 +08:00
zhongwencool ba43a0e30f
Merge pull request #10734 from thalesmg/merge-r50-into-v50-a
merge `release-50` into `master`
2023-05-18 14:24:53 +08:00
zhongwencool bf5ee41009
Merge pull request #10702 from zhongwencool/keepalive-backoff-rename
feat: deprecated keepalive_backoff, introduce keepalive_multiplier
2023-05-18 10:00:17 +08:00
Thales Macedo Garitezi 447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Thales Macedo Garitezi 10f6edd6ea
Merge pull request #10717 from thalesmg/fix-bw-flush-timer-full-infl-r50
fix(buffer_worker): avoid setting flush timer when inflight is full
2023-05-17 14:49:27 -03:00
Thales Macedo Garitezi b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
Zhongwen Deng e26ce5816e feat: hide keepalive_backoff, introduce keepalive_multiplier 2023-05-17 21:25:10 +08:00
zhongwencool ea8ac877b0
Merge pull request #10713 from zhongwencool/put-webhook-request-timeout-into-resource-opts
feat: update wehbook's request_timeout into resource_opts
2023-05-17 21:20:42 +08:00
William Yang ea4fbb8b6e chore: add changelog 2023-05-17 14:46:07 +02:00
Thales Macedo Garitezi a7b41e1cdf perf(webhook): add retry attempts for async
This is a performance improvement for webhook bridge.

Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well.  Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.

This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
Kjell Winblad ab8b8ccaad docs: make sure there is a summary for all API endpoints
Many HTTP API endpoints did not have a summary when looking at the API
docs at "http://emqx_host_name:18083/api-docs". This has been fixed by
making sure there is a summary corresponding to all endpoints.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9767
2023-05-17 09:58:56 +02:00
某文 2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
firest 1f7ede90a4 chore: update app version && changes 2023-05-17 11:01:31 +08:00
Andrew Mayorov b2ecbef0f1
Merge pull request #10715 from fix/EMQX-9897/preserve-peercert-until-connected
fix(chan): postpone trimming conninfo after `connected` hook run
2023-05-16 22:56:41 +03:00
Andrew Mayorov 74c04b847c
chore: add changelog entry
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-05-16 19:35:56 +03:00
某文 77cf19c96c chore: update 10340's changelog 2023-05-16 22:59:08 +08:00
Thales Macedo Garitezi 657df05ad9 fix(buffer_worker): avoid setting flush timer when inflight is full
Fixes https://emqx.atlassian.net/browse/EMQX-9902

When the buffer worker inflight window is full, we don’t need to set a
timer to flush the messages again because there’s no more room, and
one of the inflight windows will flush the buffer worker by calling
`flush_worker`.

Currently, we do set the timer on such situation, and this fact
combined with the default batch time of 0 yields a busy loop situation
where the CPU spins a lot while inflight messages do not return.
2023-05-16 11:28:58 -03:00
Ilya Averyanov c113a8ac6a
Merge pull request #10677 from savonarola/0511-fix-rule-api
fix(api): respond 404 on the deletion of nonexistent rule
2023-05-16 16:50:30 +03:00
某文 a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
Kjell Winblad b1a4f6ea05 docs: add changelog entry 2023-05-15 16:37:54 +02:00
Ivan Dyachkov bde086f549
Merge pull request #10701 from id/0510-bump-builder-version
ci: emqx-builder 5.0-35
2023-05-15 14:48:56 +02:00
firest 34b09e071f fix(limiter): fix a format error in the changelog 2023-05-15 18:08:33 +08:00
Ivan Dyachkov 8d8cf23fad chore: add changelog 2023-05-15 08:17:30 +02:00
lafirest 67ada52808
Merge pull request #10682 from lafirest/fix/will_msg_timestamp
fix: update the will message timestamp when it is ready to publish
2023-05-12 22:25:32 +08:00
zhongwencool a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
Zhongwen Deng 1141499b1b chore: add changelog for 10659 2023-05-12 14:42:33 +08:00
firest 2404d167c8 chore: update changes 2023-05-12 14:39:18 +08:00
Thales Macedo Garitezi 64dc9ed46a perf(metrics): avoid increasing counters by 0
Some performance tests indicate that calling `telemetry` is costly in
hot paths.  Since increasing a counter by 0 is a no-op, we should
avoid calling `telemetry` if the amount to increase is 0.
2023-05-11 15:13:37 -03:00
Ilya Averyanov 49e9ace1c1 fix(api): respond 404 on the deletion of nonexistent rule 2023-05-11 23:12:59 +05:00
SergeTupchiy 0617a9b11c
Merge pull request #10653 from SergeTupchiy/convert-gateway-authn-certs
fix(emqx_gateway): convert and clear authentication certificates
2023-05-11 11:35:55 +03:00
Zaiming (Stone) Shi b3e35dac1c
Merge pull request #10584 from paulozulato/debug-ssl-handshake
feat: add log level configuration to ssl communication
2023-05-11 08:59:13 +02:00
Zhongwen Deng 898826f15f chore: add changelog for 10659 2023-05-10 14:56:56 +08:00
Serge Tupchii b98a7168c4 fix(emqx_gateway): convert and clear authentication certificates 2023-05-09 17:32:44 +03:00
William Yang 328add3a02
Merge pull request #10563 from qzhuyan/fix/william/no_local_filter_all
fix(mqtt): drop all local messages in session deliver
2023-05-09 15:29:12 +02:00
lafirest b94290db58
Merge pull request #10625 from lafirest/refactor/limiter_cfg
refactor(limiter): simplify limiter configuration
2023-05-09 11:40:05 +08:00
Paulo Zulato 77176787ca feat: add log level configuration to ssl communication
Fixes https://emqx.atlassian.net/browse/EMQX-9781
2023-05-08 15:11:49 -03:00
Paulo Zulato 83c4b2f2d4
Merge pull request #10568 from paulozulato/feat-shutdown-count
feat: add shutdown count
2023-05-08 15:07:51 -03:00
zhongwencool f6c5644448
Merge pull request #10623 from zhongwencool/rename-max-message-queue-len
feat: rename max_message_queue_len to max_mailbox_size
2023-05-08 23:08:06 +08:00
zhongwencool 74d0436fbf
Merge pull request #10636 from kjellwinblad/kjell/fix/mongo/max_overflow/EMQX-9714
fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
2023-05-08 20:45:57 +08:00
William Yang fc46b81fb1 docs: change log for #10563 2023-05-08 14:40:40 +02:00
Zhongwen Deng eaa129d0d7 feat: rename max_message_queue_len to max_mailbox_size 2023-05-08 20:27:52 +08:00
Zaiming (Stone) Shi 83257617cd
Merge pull request #10630 from zhongwencool/sync-release-50-to-master
Sync release 50 to master
2023-05-08 14:02:27 +02:00
Kjell Winblad 62570d076b docs: add change log entry 2023-05-08 12:50:31 +02:00
Stefan Strigler b639353cf4 fix(emqx_connector_http): wrap and unwrap auth headers
This so we don't have sensitive information in a process' state
2023-05-08 09:56:43 +02:00
William Yang b6c7e55348
Merge pull request #10528 from qzhuyan/perf/william/force-config-atom-path
perf(config): ensure root keys of 'conf' config is atom
2023-05-08 09:37:18 +02:00
Zhongwen Deng 4f396a36a9 Merge remote-tracking branch 'upstream/master' into release-50 2023-05-08 14:58:03 +08:00
firest 197ebccf95 chore: update changes 2023-05-08 11:45:55 +08:00
lafirest 335d948bce
Merge pull request #10591 from lafirest/fix/simplify_limiter_client_cfg
perf(limiter): simplify the memory represent of limiter configuration
2023-05-05 16:59:04 +08:00
firest 7a96a9772d fix(limiter): update change && fix deprecated version 2023-05-05 16:26:06 +08:00
William Yang e4f501417b
Merge pull request #10525 from qzhuyan/perf/william/avoid-new-map-when-get-mqtt-caps
perf(config): avoid build new map in emqx_mgmt_caps:get_caps
2023-05-05 10:22:08 +02:00
firest 1b066fc0b6 chore: update changes 2023-05-05 16:21:21 +08:00
Zhongwen Deng e42e73e104 chore: add changelog for trace timestrap 2023-05-04 14:44:56 +08:00
Paulo Zulato f965fa7b9a feat: add shutdown count
Fixes https://emqx.atlassian.net/browse/EMQX-9008
Fixes https://emqx.atlassian.net/browse/EMQX-9219
2023-05-03 20:00:38 -03:00
William Yang c38cec77a7
Merge pull request #10417 from qzhuyan/perf/william/no-make-ref-in-get-config
perf(config): eliminate make_ref() calls in config get calls
2023-05-03 16:56:44 +02:00
William Yang 421efd561a
docs: Update changes/ce/perf-10417.en.md
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-05-03 13:50:48 +02:00
William Yang 0428766a6e docs: changelog for forcing atom conf path 2023-05-03 09:59:18 +02:00
Zaiming (Stone) Shi 66112fceed
Merge pull request #10572 from zmstone/0502-merge-release-50-back-to-master
0502 merge release 50 back to master
2023-05-03 09:01:35 +02:00
Thales Macedo Garitezi 8aa7c014e7 perf(buffer_worker): avoid calling `ets:info/2`
(Almost?) fixes https://emqx.atlassian.net/browse/EMQX-9637

During the course of performance tests comparing the performance of
e5.0.3 and e4.4.16 regarding the webhook bridge in sync mode, we
observed that the throughput in e5.0.3 (sync) was much lower than in
e4.4.16: ~ 9 k msgs / s vs. ~ 50 k msgs / s, respectively.

Analyzing `observer_cli` output, we noticed that a lot of the time
both buffer workers and ehttpc processes was spent in `ets:info/2`.
That function was called to check the size of the inflight table when
updating metrics and checking if the inflight table was full.  Other
uses of `ets:info/2` were contained inside the arguments to some
`?tp/2` macro usages (https://github.com/kafka4beam/snabbkaffe/pull/60).

By using a specific record to track the size of the table, we managed
to improve the bridge performance to ~ 45 k msgs / s in sync mode.
2023-05-02 17:05:32 -03:00
William Yang 3d7201502b perf(config): avoid build new map in emqx_mqtt_caps:get_caps 2023-05-02 21:50:36 +02:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Ilya Averyanov 791010a65c chore(topic_metrics): do not complain on nonexisting ets while topic metrics stop 2023-05-02 15:59:50 +03:00
Thales Macedo Garitezi 77f5e461a3 chore: bump ehttpc -> 0.4.8
Fixes https://emqx.atlassian.net/browse/EMQX-9656

See also https://github.com/emqx/ehttpc/pull/45

This fixes a race condition where the remote server would close the
connection before or during requests, and, depending on timing, an
`{error, normal}` response would be returned.  In those cases, we
should just retry the request without using up "retry credits".
2023-04-27 14:10:37 -03:00
Thales Macedo Garitezi c53741a08c fix(buffer_worker): avoid sending late reply messages to callers
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
2023-04-26 13:18:28 -03:00
William Yang 5ed3c3a92c perf(config): eliminate make_ref() calls in config get calls 2023-04-26 10:58:08 +02:00
SergeTupchiy d398276852
Merge pull request #10518 from SergeTupchiy/upgrade-ekka-to-0.15.1-mria-0.5.2
chore: bump ekka to 0.15.1
2023-04-26 11:03:23 +03:00
zhongwencool 341024f0bc
Merge pull request #10497 from HJianBo/improve-chages-sentence
chore: improve changes
2023-04-26 14:56:29 +08:00
Serge Tupchii 19b5ebff81 chore: bump ekka to 0.15.1
ekka 0.15.1 uses mria 0.5.2, which includes the following changes:
  - fix(mria_membership): call `mria_rlog:role/1` safely
  - feat: add extra field to ?rlog_sync table (for future use)
2023-04-25 23:21:07 +03:00
Zhongwen Deng 308056f0fc feat: improved the storage format of Unicode characters in data files 2023-04-25 18:08:34 +08:00
zhongwencool 79a235f8b5
Merge pull request #10491 from zhongwencool/rename-etcd-ssl-to-ssl_options
feat: rename etcd.ssl to etcd.ssl_options
2023-04-25 10:49:09 +08:00
JianBo He 09b17000c8
chore: update changes/ce/feat-10457.en.md
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-04-25 09:56:40 +08:00
lafirest 29d3afd692
Merge pull request #10487 from lafirest/fix/limiter_instance
fix(limiter): optimize the instance of limiter
2023-04-25 09:43:04 +08:00
Serge Tupchii 99e892b5c4 chore: bump ekka to 0.15.0
ekka 0.15.0 uses mria 0.5.0, which adds several fixes, enhancements and features:
  - protect `mria:join/1,2` with a global lock
  - implement new function `mria:sync_transaction/4,3,2`, which waits for
    a transaction replication to be ready on the local node
    (if the local node is a replicant)
  - optimize `mria:running_nodes/0`
  - optimize `mria:ro_transaction/2` when called on a replicant node.

Fixes: EMQX-9588 (#10380), EMQX-9102, EMQX-9152, EMQX-9213
2023-04-24 20:52:20 +03:00
lafirest 2845469c48
Merge pull request #10490 from lafirest/fix/rmv_def_conn_limit
fix(limiter): remove the default limit of connect rate
2023-04-24 22:04:29 +08:00
JianBo He 4aad5c7454 chore: improve changes 2023-04-24 21:44:21 +08:00
firest 6110aad23f chore: bump version && update changes 2023-04-24 18:17:23 +08:00
zhongwencool 22d6a2eaee
Merge pull request #10484 from zhongwencool/fix-copy-conf
fix: copy cluster-override.conf from old version
2023-04-24 17:06:07 +08:00
Zaiming (Stone) Shi 366aa53c80
Merge pull request #10462 from zmstone/0430-EMQX-8434-shared-dispatch-ack-deprecated
chore: Hide config shared_dispatch_ack_enabled
2023-04-24 09:42:09 +02:00
Zhongwen Deng dacf92c4ab chore: rename etcd.ssl changelog 2023-04-24 14:46:06 +08:00
firest 24cecae1f8 chore: update changes 2023-04-24 14:15:45 +08:00
firest 2f2f32ac7b chore: update changes 2023-04-24 10:52:30 +08:00
某文 38cebf2fdc chore: add changelog for 10484 2023-04-23 15:53:17 +08:00
Zaiming (Stone) Shi 81340edbca docs: add changelog 2023-04-22 09:08:07 +02:00
zhongwencool f831a0b827 chore: update changes/ce/fix-10449.en.md
Co-authored-by: JianBo He <heeejianbo@163.com>
2023-04-21 15:30:27 +08:00
Zhongwen Deng 397e28f5a4 chore: add changlog for authn_http validation 2023-04-21 15:30:05 +08:00
zhongwencool ad6090a778 chore: update changes/ce/fix-10449.en.md
Co-authored-by: JianBo He <heeejianbo@163.com>
2023-04-21 11:54:45 +08:00
Zhongwen Deng 2aef9ca215 chore: add changlog for authn_http validation 2023-04-21 11:54:45 +08:00
SergeTupchiy 0b105dcb8d
Merge pull request #10463 from SergeTupchiy/EMQX-9310-webhook-port-validation
fix(emqx_bridge): validate Webhook bad URL
2023-04-20 20:48:12 +03:00
Ilya Averyanov 8c4b32580e
Merge pull request #10420 from savonarola/0417-authn-authz-fix-path-quoting
Fix path joining and quoting in authn/authz
2023-04-20 18:03:04 +03:00
Serge Tupchii 45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
Fixes: EMQX-9310
2023-04-20 17:21:43 +03:00
Serge Tupchii 423a30fbb3 fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager
Don't let 'emqx_resource_manager' crash because of emqx_alarm timeouts.

Fixes: EMQX-9529/#10357
2023-04-20 17:15:13 +03:00
Thales Macedo Garitezi 3f18c5e2e3
Merge pull request #10455 from thalesmg/fix-late-gen-server-replies-buf-worker-v50
fix(buffer_worker): avoid sending late reply messages to callers
2023-04-20 10:19:06 -03:00
Zhongwen Deng bcc8f4313b chore: make plugins config to low level 2023-04-20 18:00:19 +08:00
JianBo He 794ddd8d73 chore: update changes 2023-04-20 14:23:55 +08:00
Thales Macedo Garitezi cb995e2033 fix(buffer_worker): avoid sending late reply messages to callers
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
2023-04-19 18:27:10 -03:00
firest 8219af2fd4 chore: improve changelog 2023-04-19 16:45:53 +08:00