194faefc8a
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-10001
Recently, we unified request_timeout in a single field located at the
webhook connector schema. However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
2023-05-25 13:19:27 -03:00
6b5ff24952
test: assert the block_until result
2023-05-25 18:12:38 +08:00
de7f1c8aec
test: add tests for auto_restart_interval
2023-05-25 17:15:19 +08:00
7da0860d6c
chore: upgrade minirest to 1.3.10 for more clear error msg
2023-05-24 10:44:52 +08:00
b5f63f4151
test: fix wrong test listeners option
2023-05-23 14:50:43 -03:00
1deb3c50c8
test: bridge test init dashboard listener failed
2023-05-23 10:28:01 +08:00
67a30ee976
test: bridge test init dashboard listener failed
2023-05-23 10:16:43 +08:00
732a7be187
Merge remote-tracking branch 'origin/release-50'
2023-05-22 17:46:54 +02:00
7d798c10e9
perf(buffer_worker): flush metrics periodically inside buffer worker process
...
Fixes https://emqx.atlassian.net/browse/EMQX-9905
Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
2023-05-22 09:11:23 -03:00
09ea2e2224
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
...
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
447b76464b
Merge branch 'release-50' into merge-r50-into-v50-a
2023-05-17 14:50:18 -03:00
a7b41e1cdf
perf(webhook): add retry attempts for async
...
This is a performance improvement for webhook bridge.
Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well. Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.
This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
2b99a9b988
feat: hide resource_opts's request_timeout
2023-05-17 13:41:11 +08:00
a2aa6b4666
chore: make ci happy again
2023-05-16 21:29:52 +08:00
14952658d5
chore: convert rotation struct to roation_count
2023-05-09 11:24:51 +08:00
d231e708cb
feat: add IotDB bridge
2023-05-08 16:19:01 +02:00
856129984b
refactor: remove raw_with_default config load option
...
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.
After a03f2dd64b
2023-05-04 09:48:28 +02:00
45254d7d85
fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
...
Fixes: EMQX-9310
2023-04-20 17:21:43 +03:00
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
...
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
e073bc90bc
refactor(buffer_worker): rename `s/queue/buffer/g`
2023-04-14 11:37:19 -03:00
4f80690162
fix: byebye jsx
2023-04-14 13:41:34 +02:00
062ce5f819
refactor: rename emqx_map_lib to emqx_utils_maps
2023-04-14 13:41:34 +02:00
9c11bfce80
refactor: rename emqx_misc to emqx_utils
2023-04-14 13:41:27 +02:00
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
e70deae1c3
feat(resource): ask for metrics only when needed
2023-04-11 12:00:19 +03:00
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
1824e7efcc
fix(rule_engine): count referenced bridges in `from` clauses as dependencies (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9325
Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.
When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
2023-03-28 10:31:10 -03:00
d942a531e6
fix(test): avoid unnecessary cleanups
...
This might make some noise in the logs, depending on the testsuite
configuration, because some test setup might be missing required
applications.
2023-03-27 23:41:19 +03:00
11b5b7b638
test(bridge-api): also run testcases in cluster environment
...
Excluding a couple of testcases which does not make much sense running
in the cluster. Also try to reduce amount of "noise" in the testcases,
making them easier to comprehend.
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 23:41:08 +03:00
fbfdaf44e0
fix(emqx_bridge): don't crash on validation error
2023-03-24 11:46:41 +01:00
35474578ca
refactor: rename async_inflight_window to inflight_window everywhere
2023-03-23 14:21:57 +01:00
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
3880862c81
fix(emqx_bridge): return 503 for inconsistency in bridge setup
2023-03-21 15:07:43 +01:00
84fc64822e
style: fix wording for nxdomain and malformed_username_or_password
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 15:02:29 +01:00
53825b9aba
fix(emqx_bridge): propagate connection error to resource status
2023-03-21 15:02:29 +01:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
66eb4ef069
test: fix inter-suite flakiness
2023-03-16 13:43:01 -03:00
91a57faa95
Merge pull request #10128 from thalesmg/ocsp-v50-mkII
...
feat: add ocsp stapling support to mqtt ssl listener (5.0)
2023-03-16 13:10:48 -03:00
816667d85c
Merge pull request #10123 from fix/EMQX-9136/lookup-busy-resources
...
perf(bridge-api): ask nodes' bridge listings in parallel
2023-03-16 00:07:12 +03:00
03b95073fc
test: fix inter-suite flakiness
2023-03-15 14:25:41 -03:00
b3e7e51094
test(bridge): drop unnecessary cleanup routines
...
Since `end_per_testcase` cleans out all the resources anyway.
2023-03-15 19:17:29 +03:00
c1adf0de1f
test(emqx_bridge): increase coverage by adding common edge cases
2023-03-13 10:40:47 +01:00
71ec77a2f2
fix(emqx_bridge): for node operation check if node is part of cluster
...
This fixes the case where we returned `501 NOT IMPLEMENTED` in the past.
2023-03-13 10:40:47 +01:00
1bcc5623ed
fix(emqx_bridge): check if bridge enabled before calling op
2023-03-13 10:38:49 +01:00
80b81748df
fix(emqx_bridge): handle bridge not found in call_operation
2023-03-13 10:37:28 +01:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
26b29185b2
test(emqx_bridge_webhook_SUITE): fix flakyness in test web server
2023-03-07 20:57:38 +01:00
163b33ab28
test: remove unnecessary dependencies of ee apps
2023-03-07 20:57:38 +01:00
ca947e3e70
fix: lost messages when HTTP connection times out
...
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.
Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test
...
test: fix flaky mqtt bridge test
2023-03-06 13:06:37 -03:00
f95a30ae89
fix(webhook): convert `request_timeout`s in root and resource_opts
2023-03-06 10:12:38 -03:00
8fbb948b6f
test: fix flaky mqtt bridge test
...
Sometimes, this test fails because the metrics are still in the
inflight phase.
2023-03-06 09:09:33 -03:00
570bf165af
fix: return human readable error message for most common cases
2023-03-03 16:49:09 +01:00
4c23ab097d
fix(emqx_bridge): return 400 if operation not possible
2023-03-02 11:23:07 +01:00
3f324f85bc
fix: consistently return 404 in case bridge is not found or invalid
...
Also: fix some typos and be more verbose
2023-03-01 18:03:49 +01:00
7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics
...
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
2023-02-27 14:55:41 +03:00
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start
...
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 20:40:37 +04:00
cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses
2023-02-10 19:15:48 +03:00
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 17:17:59 +03:00
e196d6f42a
fix(emqx_bridge): unwrap rpc multicall result
2023-02-10 12:00:20 +01:00
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
...
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
5328955ff4
fix(config): add back reconnect_interval field as deprecated
2023-02-09 17:30:09 +01:00
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events
2023-02-09 14:31:42 +03:00
a3fd0897bc
refactor: less code duplication
2023-02-08 14:04:09 +01:00
86f3f5787f
feat: allow to manually re-connect disconected bridge
2023-02-07 11:58:30 +01:00
a71d983ff8
test: fix assertMatch patter to make rebar3 fmt work
2023-02-02 16:37:45 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly
2023-02-01 16:28:16 +03:00
5ebceb20d2
test(mqtt-bridge): also test reconfiguration
2023-02-01 16:23:58 +03:00
ad88938d34
refactor: reuse some parts of test code for brewity
2023-02-01 16:22:19 +03:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
7005b71ddf
style: fix typo in comment
2023-01-27 11:43:51 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
...
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
478fcc6ffd
test: fix flaky test
2023-01-17 16:48:48 -03:00
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
d164e5bc0f
fix: don't require ssl conf
2023-01-17 15:29:19 +01:00
c3d5c25c26
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-17 17:42:10 +08:00
67909f0b40
fix: testing metrics for emqx_bridge_mqtt_SUITE
2023-01-16 12:10:06 +01:00
a19226326c
fix: don't test for strict values as that makes test flaky
2023-01-16 12:10:06 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
8f5881d1a5
fix: remove stale request/3 from merge error
2023-01-13 16:19:35 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
c3133fb6a2
fix(emqx_bridge): small fixes from review
2023-01-13 14:54:54 +01:00
f1c58c34ed
test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE
2023-01-13 14:19:23 +01:00
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge
2023-01-13 18:12:53 +08:00
42f42de4d9
feat(emqx_bridge): add separate endpoint for enable/disable of bridge
...
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
c2fd1a4482
feat(emqx_bridge): shorten operation endpoint URLs
...
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
c5f557e315
fix: disable basic auth for HTTP API
2023-01-12 21:35:49 +08:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
...
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
Thales Macedo Garitezi