Commit Graph

508 Commits

Author SHA1 Message Date
zhongwencool 9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
SergeTupchiy e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup
feat: implement configuration and user data export/import
2023-06-09 14:18:39 +03:00
Serge Tupchii e4d09d4ad4 feat: implement configuration and user data export/import CLI
Closes: EMQX-9203
2023-06-09 14:11:47 +03:00
firest 86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
Kjell Winblad 273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609
fix: friendly error message when creating bridges with too long names
2023-06-08 15:46:42 +02:00
Andrew Mayorov 99ea9b86c2
feat(tlslib): add separate managed certfiles GC process
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
2023-06-05 22:41:07 +03:00
Andrew Mayorov 95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
2023-06-05 22:41:07 +03:00
Kjell Winblad 375661c6a1 test: add test case for error when bridge name is too long 2023-06-05 17:02:08 +02:00
Andrew Mayorov 5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests
chore(bridge): drop deprecated `mode` setting from examples / tests
2023-06-02 22:46:31 +03:00
Zaiming (Stone) Shi aa46c0a484
Merge pull request #10928 from zmstone/0602-sync-release-51-to-master
0602 sync release 51 to master
2023-06-02 21:45:21 +02:00
Thales Macedo Garitezi 33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
2023-06-02 16:20:36 -03:00
Andrew Mayorov 09290f8d61
chore(bridge): drop deprecated `mode` setting from examples / tests 2023-06-02 20:31:31 +03:00
Zaiming (Stone) Shi 28c564d15b Merge remote-tracking branch 'origin/master' into release-51 2023-06-02 16:25:48 +02:00
Thales Macedo Garitezi 99796224d8 refactor(resource): rename `request_timeout` -> `request_ttl`
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
某文 c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Stefan Strigler e011b5532f
Merge pull request #10881 from sstrigler/EMQX-9355-5-0-x-test-coverage-80
IoTDB test coverage
2023-06-01 15:48:56 +02:00
Kjell Winblad 75ff76a16b fix: friendly error message when creating bridges with too long names
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
2023-06-01 15:30:06 +02:00
Andrew Mayorov b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id
fix(ruleeng): parse bridge id on create / update
2023-06-01 10:21:26 +03:00
Andrew Mayorov 29fe201676
fix(ruleeng): parse bridge id on create / update
Instead of parsing it every time in the `send_message/2` hot path.
2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi 0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
2023-05-31 10:19:55 -03:00
Andrew Mayorov 7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Andrew Mayorov a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
2023-05-30 17:21:44 +03:00
Stefan Strigler 767f7c57e7 test: check device_id is used from topic or config 2023-05-30 15:55:21 +02:00
Stefan Strigler 938d62a666 test: handle bad payload 2023-05-30 15:55:21 +02:00
Stefan Strigler 64d582770d test: add tracepoints 2023-05-30 15:55:21 +02:00
Andrew Mayorov 4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
2023-05-30 14:50:05 +03:00
Andrew Mayorov 81e78516aa
feat(mqttconn): employ ecpool instead of a single worker 2023-05-30 14:50:05 +03:00
Andrew Mayorov bd956d00b6
feat(mqttconn): stop using gproc in hot path
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
2023-05-30 14:50:05 +03:00
Zaiming (Stone) Shi 91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master
0530 merge release 50 to master
2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi 9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
Thales Macedo Garitezi 67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50
feat(kafka): ensure allocated resources are removed on failures
2023-05-29 16:49:29 -03:00
Zaiming (Stone) Shi 36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi 772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
2023-05-26 08:39:00 +02:00
Zaiming (Stone) Shi 407cab220b
Merge pull request #10760 from paulozulato/fix-failure-joining
fix: avoid error 500 when node is re-joining cluster
2023-05-26 03:08:17 +02:00
Thales Macedo Garitezi e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Paulo Zulato ea86f4442b fix: avoid error 500 when node is re-joining cluster
Fixes https://emqx.atlassian.net/browse/EMQX-9899
2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi 194faefc8a fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
2023-05-25 13:19:27 -03:00
JianBo He 6b5ff24952 test: assert the block_until result 2023-05-25 18:12:38 +08:00
JianBo He de7f1c8aec test: add tests for auto_restart_interval 2023-05-25 17:15:19 +08:00
zhongwencool 14a6b36899
Merge pull request #10803 from JimMoen/merge-release-50
Merge release-50 into master
2023-05-24 23:08:46 +08:00
JimMoen 28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad 4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
2023-05-24 13:09:47 +02:00
某文 7da0860d6c chore: upgrade minirest to 1.3.10 for more clear error msg 2023-05-24 10:44:52 +08:00
Thales Macedo Garitezi 44ed6a19ee fix(bridge): pass resource option `request_timeout = infinity` along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00
Thales Macedo Garitezi b5f63f4151 test: fix wrong test listeners option 2023-05-23 14:50:43 -03:00
某文 1deb3c50c8 test: bridge test init dashboard listener failed 2023-05-23 10:28:01 +08:00
某文 67a30ee976 test: bridge test init dashboard listener failed 2023-05-23 10:16:43 +08:00
Zaiming (Stone) Shi 732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00