1bcc5623ed
fix(emqx_bridge): check if bridge enabled before calling op
2023-03-13 10:38:49 +01:00
80b81748df
fix(emqx_bridge): handle bridge not found in call_operation
2023-03-13 10:37:28 +01:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
26b29185b2
test(emqx_bridge_webhook_SUITE): fix flakyness in test web server
2023-03-07 20:57:38 +01:00
163b33ab28
test: remove unnecessary dependencies of ee apps
2023-03-07 20:57:38 +01:00
ca947e3e70
fix: lost messages when HTTP connection times out
...
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.
Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test
...
test: fix flaky mqtt bridge test
2023-03-06 13:06:37 -03:00
f95a30ae89
fix(webhook): convert `request_timeout`s in root and resource_opts
2023-03-06 10:12:38 -03:00
8fbb948b6f
test: fix flaky mqtt bridge test
...
Sometimes, this test fails because the metrics are still in the
inflight phase.
2023-03-06 09:09:33 -03:00
570bf165af
fix: return human readable error message for most common cases
2023-03-03 16:49:09 +01:00
4c23ab097d
fix(emqx_bridge): return 400 if operation not possible
2023-03-02 11:23:07 +01:00
3f324f85bc
fix: consistently return 404 in case bridge is not found or invalid
...
Also: fix some typos and be more verbose
2023-03-01 18:03:49 +01:00
7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics
...
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
2023-02-27 14:55:41 +03:00
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start
...
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 20:40:37 +04:00
cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses
2023-02-10 19:15:48 +03:00
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 17:17:59 +03:00
e196d6f42a
fix(emqx_bridge): unwrap rpc multicall result
2023-02-10 12:00:20 +01:00
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
...
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
5328955ff4
fix(config): add back reconnect_interval field as deprecated
2023-02-09 17:30:09 +01:00
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events
2023-02-09 14:31:42 +03:00
a3fd0897bc
refactor: less code duplication
2023-02-08 14:04:09 +01:00
86f3f5787f
feat: allow to manually re-connect disconected bridge
2023-02-07 11:58:30 +01:00
a71d983ff8
test: fix assertMatch patter to make rebar3 fmt work
2023-02-02 16:37:45 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly
2023-02-01 16:28:16 +03:00
5ebceb20d2
test(mqtt-bridge): also test reconfiguration
2023-02-01 16:23:58 +03:00
ad88938d34
refactor: reuse some parts of test code for brewity
2023-02-01 16:22:19 +03:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
7005b71ddf
style: fix typo in comment
2023-01-27 11:43:51 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
...
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
478fcc6ffd
test: fix flaky test
2023-01-17 16:48:48 -03:00
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
d164e5bc0f
fix: don't require ssl conf
2023-01-17 15:29:19 +01:00
c3d5c25c26
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-17 17:42:10 +08:00
67909f0b40
fix: testing metrics for emqx_bridge_mqtt_SUITE
2023-01-16 12:10:06 +01:00
a19226326c
fix: don't test for strict values as that makes test flaky
2023-01-16 12:10:06 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
8f5881d1a5
fix: remove stale request/3 from merge error
2023-01-13 16:19:35 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
c3133fb6a2
fix(emqx_bridge): small fixes from review
2023-01-13 14:54:54 +01:00
f1c58c34ed
test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE
2023-01-13 14:19:23 +01:00
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge
2023-01-13 18:12:53 +08:00
42f42de4d9
feat(emqx_bridge): add separate endpoint for enable/disable of bridge
...
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
c2fd1a4482
feat(emqx_bridge): shorten operation endpoint URLs
...
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
c5f557e315
fix: disable basic auth for HTTP API
2023-01-12 21:35:49 +08:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
...
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
96ca0d9f49
feat(emqx_bridge): add /bridges_probe API endpoint
2023-01-09 10:01:33 +01:00
13b2f45405
fix: function_clause when sending messages to bridges
2023-01-06 09:24:38 +08:00
b59c4c34c5
fix(Bridge REST API): no feedback when deleting bridge
...
This fixes https://emqx.atlassian.net/browse/EMQX-8648 . The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:
```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist '
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```
After the fix, deleting a non existing bridge will give the following
response:
```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```
Closes: EMQX-8648
2023-01-05 16:19:46 +01:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
b43be50a98
test: add the `redacted` test case for bridges api
2023-01-03 22:16:34 +08:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
0b43ae621d
ci: dump docker-compose log if failed to run ct
2022-12-29 09:23:11 +01:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
6b6bb09d4a
fix: ingress only bridge causes egress bridge traffic stop
2022-12-09 15:36:20 +01:00
31098d6c67
test: fix bridge api tests, metrics is now a separate api
2022-12-06 19:03:32 +01:00
eb017ab034
test: add test case to cover mqtt bridge config upgrade
2022-12-01 16:50:10 +01:00
983e904858
fix(connector): fix ssl clear
2022-11-30 17:03:41 +01:00
7ee53e5319
Merge tag 'v5.0.11' into dev/ee5.0
2022-11-28 21:02:21 +01:00
6dc5078a96
fix: fix obsolete SSL files aren't deleted after the bridge configuration update
2022-11-23 23:49:43 +01:00
2d01726b22
fix: account calls when resource is not connected as matched
2022-10-13 15:32:04 -03:00
f0ff32c031
test: fix tests after counter changes
2022-10-11 17:45:48 -03:00
9aa7e826cb
refactor(resource): fast resume resource worker if inflight msgs are ACKed
2022-09-17 00:34:30 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
b9ae4ea276
refactor: rename some metrics for emqx_resource
2022-09-13 14:04:25 +08:00
60a90858f8
feat: check dependent actions before removing the bridges
2022-09-06 15:34:23 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
55c18c0b5f
fix(bridges): update the test cases for new config structure
2022-08-22 18:24:59 +08:00
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
69fba6958b
refactor: remove the emqx_bridge_monitor module
2022-06-01 15:31:48 +08:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
d61b44ef97
fix: ct failed again
2022-05-31 19:20:27 +08:00
a157539710
refactor(rule): rename http_bridge -> webhook
2022-05-25 18:56:12 +08:00
d89d692d3e
refactor: move some APIs to new module emqx_bridge_resource
2022-05-18 15:12:34 +08:00
47ed9cbc76
fix(telemetry): count bridges referenced in `from` rule clauses
2022-05-11 15:15:17 -03:00
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
6edb453ce9
fix: the field 'enable' is missing for GET /bridges/:id
2022-04-28 18:13:19 +08:00
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
eb3870b538
fix: when enable is false, restart and stop cannot be performed.
2022-04-25 18:48:12 +08:00
5d9df74be7
test: fix required fields
2022-04-21 18:47:02 +08:00
5aeb3c2dc2
feat: impl bridge reset_metrics api
2022-04-11 10:25:48 +08:00
2a7b777a2a
fix(test): fix inter-suite test coupling
2022-04-07 12:21:15 -03:00
911e02f626
feat(bridge): export basic usage info for telemetry
2022-04-06 10:20:23 -03:00
0728b1e3f4
chore(emqx_modules): add emqx_topic_metrics_api tests
2022-03-30 21:18:27 +03:00
1d023b541f
refactor(connnector): rename waiting_connect_complete -> wait_for_resource_ready
...
Rename the option to wait_for_resource_ready and defaults to 5s.
2022-03-10 10:46:57 +08:00
50d0a3271b
fix(test): add testcases for enable/disbale bridges
2022-03-09 18:49:30 +08:00
36e068d00d
feat(bridge): add APIs for restart/stop bridges on one node
2022-03-09 18:42:19 +08:00
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
8a0565a53b
chore(elvis): please the elvis
2022-03-08 11:11:29 +08:00
9a9c92ae88
fix: update testcases for emqx_bridge
2022-03-07 16:48:36 +08:00
fa21fa2432
fix(emqx_bridge): fix bridge status
2022-02-25 15:26:40 +08:00
d0b0b05aa4
fix: decouple suite tests
2022-02-08 14:53:01 +01:00
2105d1f22b
test: call ct helpers to load config
2022-01-25 11:50:18 +01:00
63167cea70
chore: update copyright
2022-01-05 20:55:00 +01:00
e95445728c
fix(test): wait until the bridge ready
2022-01-02 23:59:56 +08:00
d11cf6ad64
fix(bridges): store connector name and bridge name to config files
2021-12-30 23:48:41 +08:00
eb992ad2ad
fix(bridge): add test cases for sending msgs via http bridge
2021-12-28 11:35:47 +08:00
0699682f38
refactor(bridges): rename some config entries for MQTT bridge
2021-12-17 21:22:04 +08:00
d76f82d3d2
fix(bridges): add new option 'enable' for bridge configs
2021-12-08 18:16:38 +08:00
9b4fe87ed0
fix(bridges): update testcases for bridge APIs
2021-12-07 09:56:57 +08:00
37aeb28a79
fix(connector): test case conflict
2021-11-22 18:52:43 +08:00
f0c61068e8
fix(authn): test cases issues
2021-11-22 17:35:33 +08:00
9c93ea0338
feat(connector): add API for /connectors_test
2021-11-22 17:35:33 +08:00
4dac90f4a7
refactor(bridges): add POST /bridges for creating
2021-11-22 17:35:32 +08:00
c4668b4047
fix(bridge): load emqx_conf in testcases
2021-11-22 17:35:32 +08:00
339749764d
fix(bridge): update the testcases for HTTP bridges
2021-11-22 17:35:32 +08:00
7d64013edd
refactor(connector): add connector configs
2021-11-22 17:35:32 +08:00
e41736f5cb
fix(bridge): update testcases for bridge APIs
2021-11-22 17:35:32 +08:00
1e6884ee7d
fix(bridge): convert confs for http bridge when creating
2021-11-22 17:35:32 +08:00
63f942a1b8
feat(bridge): add descriptions to the mqtt bridge schema
2021-11-22 17:35:32 +08:00
dd9e2c4b24
refactor(bridge): the configs for http bridges
2021-11-22 17:35:32 +08:00
Stefan Strigler