yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,108 Commits 393 Branches 669 Tags 74 MiB
52f4519eeb
Commit Graph

214 Commits

Author SHA1 Message Date
Stefan Strigler
570bf165af fix: return human readable error message for most common cases 2023-03-03 16:49:09 +01:00
Stefan Strigler
4c23ab097d fix(emqx_bridge): return 400 if operation not possible 2023-03-02 11:23:07 +01:00
Stefan Strigler
3f324f85bc fix: consistently return 404 in case bridge is not found or invalid 
Also: fix some typos and be more verbose
 2023-03-01 18:03:49 +01:00
Andrew Mayorov
7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics 
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
 2023-02-27 14:55:41 +03:00
Andrew Mayorov
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start 
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
 2023-02-10 20:40:37 +04:00
Andrew Mayorov
cbb2885499
fix(mqtt-bridge): allow to configure clean_start for ingresses 2023-02-10 19:15:48 +03:00
Andrew Mayorov
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges 2023-02-10 17:17:59 +03:00
Stefan Strigler
e196d6f42a fix(emqx_bridge): unwrap rpc multicall result 2023-02-10 12:00:20 +01:00
Zaiming (Stone) Shi
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field 
fix(config): add back reconnect_interval field as deprecated
 2023-02-10 09:19:12 +01:00
Zaiming (Stone) Shi
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour 
EMQX 8861 improve bridge restart button behaviour
 2023-02-09 18:00:48 +01:00
Zaiming (Stone) Shi
5328955ff4 fix(config): add back reconnect_interval field as deprecated 2023-02-09 17:30:09 +01:00
Andrew Mayorov
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events 2023-02-09 14:31:42 +03:00
Stefan Strigler
a3fd0897bc refactor: less code duplication 2023-02-08 14:04:09 +01:00
Stefan Strigler
86f3f5787f feat: allow to manually re-connect disconected bridge 2023-02-07 11:58:30 +01:00
Zaiming (Stone) Shi
a71d983ff8 test: fix assertMatch patter to make rebar3 fmt work 2023-02-02 16:37:45 +01:00
Andrew Mayorov
0912f13c1f
fix(mqtt-bridge): stop respecting clean_start config parameter 
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
 2023-02-02 12:55:10 +03:00
Andrew Mayorov
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly 2023-02-01 16:28:16 +03:00
Andrew Mayorov
5ebceb20d2
test(mqtt-bridge): also test reconfiguration 2023-02-01 16:23:58 +03:00
Andrew Mayorov
ad88938d34
refactor: reuse some parts of test code for brewity 2023-02-01 16:22:19 +03:00
Andrew Mayorov
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter 2023-01-31 18:30:42 +03:00
Andrew Mayorov
d0c10b59aa
feat(mqtt-bridge): avoid middleman process 
Instead, supervise `emqtt` client process directly.
 2023-01-31 17:59:03 +03:00
Stefan Strigler
7005b71ddf style: fix typo in comment 2023-01-27 11:43:51 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Stefan Strigler
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub 
EMQX 8754 test function return 500 of data integration google pubsub
 2023-01-17 22:49:28 +01:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
478fcc6ffd test: fix flaky test 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Stefan Strigler
d164e5bc0f fix: don't require ssl conf 2023-01-17 15:29:19 +01:00
firest
c3d5c25c26 fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12 2023-01-17 17:42:10 +08:00
Stefan Strigler
67909f0b40 fix: testing metrics for emqx_bridge_mqtt_SUITE 2023-01-16 12:10:06 +01:00
Stefan Strigler
a19226326c fix: don't test for strict values as that makes test flaky 2023-01-16 12:10:06 +01:00
JimMoen
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge 
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
 2023-01-16 09:57:20 +08:00
Stefan Strigler
8f5881d1a5 fix: remove stale request/3 from merge error 2023-01-13 16:19:35 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Stefan Strigler
1690a6dcfc
Merge branch 'master' into dev/api-refactor 2023-01-13 15:34:13 +01:00
Erik Timan
c3133fb6a2 fix(emqx_bridge): small fixes from review 2023-01-13 14:54:54 +01:00
Erik Timan
f1c58c34ed test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE 2023-01-13 14:19:23 +01:00
JimMoen
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge 2023-01-13 18:12:53 +08:00
Erik Timan
42f42de4d9 feat(emqx_bridge): add separate endpoint for enable/disable of bridge 
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
 2023-01-12 19:39:43 +01:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Erik Timan
c2fd1a4482 feat(emqx_bridge): shorten operation endpoint URLs 
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
 2023-01-12 19:36:22 +01:00
firest
c5f557e315 fix: disable basic auth for HTTP API 2023-01-12 21:35:49 +08:00
Stefan Strigler
8ad8288195 feat: report error in create_dry_run 2023-01-11 14:22:37 +01:00
Kjell Winblad
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648 
fix: no feedback when deleting non-existing configuration
 2023-01-09 13:52:40 +01:00
Stefan Strigler
96ca0d9f49 feat(emqx_bridge): add /bridges_probe API endpoint 2023-01-09 10:01:33 +01:00
Shawn
13b2f45405 fix: function_clause when sending messages to bridges 2023-01-06 09:24:38 +08:00
Kjell Winblad
b59c4c34c5 fix(Bridge REST API): no feedback when deleting bridge 
This fixes https://emqx.atlassian.net/browse/EMQX-8648. The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:

```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist'
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```

After the fix, deleting a non existing bridge will give the following
response:

```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```

Closes: EMQX-8648
 2023-01-05 16:19:46 +01:00
Thales Macedo Garitezi
fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00
firest
b43be50a98 test: add the redacted test case for bridges api 2023-01-03 22:16:34 +08:00
Thales Macedo Garitezi
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50 
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
 2023-01-02 10:56:47 -03:00
Zaiming (Stone) Shi
dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Thales Macedo Garitezi
8b060a75f1 refactor(metrics): use absolute gauge values rather than deltas 
https://emqx.atlassian.net/browse/EMQX-8548

Currently, we face several issues trying to keep resource metrics
reasonable.  For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.

Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID.  This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
 2022-12-30 16:51:24 -03:00
Zaiming (Stone) Shi
0b43ae621d ci: dump docker-compose log if failed to run ct 2022-12-29 09:23:11 +01:00
Thales Macedo Garitezi
35dc75b7ed feat(mqtt): add option to customize clientid prefix for egress bridges 
https://emqx.atlassian.net/browse/EMQX-8445

Currently the bridge client’s client ID is prefixed with the resource
ID.

Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
 2022-12-23 09:50:26 -03:00
William Yang
6b6bb09d4a fix: ingress only bridge causes egress bridge traffic stop 2022-12-09 15:36:20 +01:00
Zaiming (Stone) Shi
31098d6c67 test: fix bridge api tests, metrics is now a separate api 2022-12-06 19:03:32 +01:00
Zaiming (Stone) Shi
eb017ab034 test: add test case to cover mqtt bridge config upgrade 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi
983e904858 fix(connector): fix ssl clear 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi
7ee53e5319 Merge tag 'v5.0.11' into dev/ee5.0 2022-11-28 21:02:21 +01:00
firest
6dc5078a96 fix: fix obsolete SSL files aren't deleted after the bridge configuration update 2022-11-23 23:49:43 +01:00
Thales Macedo Garitezi
2d01726b22 fix: account calls when resource is not connected as matched 2022-10-13 15:32:04 -03:00
Thales Macedo Garitezi
f0ff32c031 test: fix tests after counter changes 2022-10-11 17:45:48 -03:00
Shawn
9aa7e826cb refactor(resource): fast resume resource worker if inflight msgs are ACKed 2022-09-17 00:34:30 +08:00
Shawn
d5d3972ff5 chore: add test cases for MQTT Bridge reconnecting 2022-09-15 10:19:33 +08:00
Shawn
b9ae4ea276 refactor: rename some metrics for emqx_resource 2022-09-13 14:04:25 +08:00
Shawn
60a90858f8 feat: check dependent actions before removing the bridges 2022-09-06 15:34:23 +08:00
Shawn
73e19d84ee feat: use the new metrics to bridge APIs 2022-08-30 23:47:58 +08:00
Shawn
55c18c0b5f fix(bridges): update the test cases for new config structure 2022-08-22 18:24:59 +08:00
Shawn
0377d3cf61 fix: update existing testcases for new emqx_resource 2022-08-10 00:34:35 +08:00
Shawn
69fba6958b refactor: remove the emqx_bridge_monitor module 2022-06-01 15:31:48 +08:00
Shawn
88ca25c60c fix(resource): fast return when starting a unavailable resource 2022-06-01 08:24:53 +08:00
Zhongwen Deng
d61b44ef97 fix: ct failed again 2022-05-31 19:20:27 +08:00
Shawn
a157539710 refactor(rule): rename http_bridge -> webhook 2022-05-25 18:56:12 +08:00
Shawn
d89d692d3e refactor: move some APIs to new module emqx_bridge_resource 2022-05-18 15:12:34 +08:00
Thales Macedo Garitezi
47ed9cbc76
fix(telemetry): count bridges referenced in from rule clauses 2022-05-11 15:15:17 -03:00
Chris
0b3e30e813 feat: isolate resource manager processes 2022-05-09 13:24:34 +02:00
Shawn
6edb453ce9 fix: the field 'enable' is missing for GET /bridges/:id 2022-04-28 18:13:19 +08:00
Zaiming (Stone) Shi
02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
EMQ-YangM
eb3870b538 fix: when enable is false, restart and stop cannot be performed. 2022-04-25 18:48:12 +08:00
EMQ-YangM
5d9df74be7 test: fix required fields 2022-04-21 18:47:02 +08:00
EMQ-YangM
5aeb3c2dc2 feat: impl bridge reset_metrics api 2022-04-11 10:25:48 +08:00
Thales Macedo Garitezi
2a7b777a2a
fix(test): fix inter-suite test coupling 2022-04-07 12:21:15 -03:00
Thales Macedo Garitezi
911e02f626
feat(bridge): export basic usage info for telemetry 2022-04-06 10:20:23 -03:00
Ilya Averyanov
0728b1e3f4 chore(emqx_modules): add emqx_topic_metrics_api tests 2022-03-30 21:18:27 +03:00
Shawn
1d023b541f refactor(connnector): rename waiting_connect_complete -> wait_for_resource_ready 
Rename the option to wait_for_resource_ready and defaults to 5s.
 2022-03-10 10:46:57 +08:00
Shawn
50d0a3271b fix(test): add testcases for enable/disbale bridges 2022-03-09 18:49:30 +08:00
Shawn
36e068d00d feat(bridge): add APIs for restart/stop bridges on one node 2022-03-09 18:42:19 +08:00
EMQ-YangM
f29877bb6a fix(emqx_resource): remove create_opts async_create 2022-03-08 14:09:39 +08:00
Shawn
8a0565a53b chore(elvis): please the elvis 2022-03-08 11:11:29 +08:00
Shawn
9a9c92ae88 fix: update testcases for emqx_bridge 2022-03-07 16:48:36 +08:00
EMQ-YangM
fa21fa2432 fix(emqx_bridge): fix bridge status 2022-02-25 15:26:40 +08:00
Chris
d0b0b05aa4 fix: decouple suite tests 2022-02-08 14:53:01 +01:00
Zaiming (Stone) Shi
2105d1f22b test: call ct helpers to load config 2022-01-25 11:50:18 +01:00
Zaiming (Stone) Shi
63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
Shawn
e95445728c fix(test): wait until the bridge ready 2022-01-02 23:59:56 +08:00
Shawn
d11cf6ad64 fix(bridges): store connector name and bridge name to config files 2021-12-30 23:48:41 +08:00
Shawn
eb992ad2ad fix(bridge): add test cases for sending msgs via http bridge 2021-12-28 11:35:47 +08:00
Shawn
0699682f38 refactor(bridges): rename some config entries for MQTT bridge 2021-12-17 21:22:04 +08:00
Shawn
d76f82d3d2 fix(bridges): add new option 'enable' for bridge configs 2021-12-08 18:16:38 +08:00
Shawn
9b4fe87ed0 fix(bridges): update testcases for bridge APIs 2021-12-07 09:56:57 +08:00
Shawn
37aeb28a79 fix(connector): test case conflict 2021-11-22 18:52:43 +08:00
Shawn
f0c61068e8 fix(authn): test cases issues 2021-11-22 17:35:33 +08:00
Shawn
9c93ea0338 feat(connector): add API for /connectors_test 2021-11-22 17:35:33 +08:00
Shawn
4dac90f4a7 refactor(bridges): add POST /bridges for creating 2021-11-22 17:35:32 +08:00
Shawn
c4668b4047 fix(bridge): load emqx_conf in testcases 2021-11-22 17:35:32 +08:00
Shawn
339749764d fix(bridge): update the testcases for HTTP bridges 2021-11-22 17:35:32 +08:00
Shawn
7d64013edd refactor(connector): add connector configs 2021-11-22 17:35:32 +08:00
Shawn
e41736f5cb fix(bridge): update testcases for bridge APIs 2021-11-22 17:35:32 +08:00
Shawn
1e6884ee7d fix(bridge): convert confs for http bridge when creating 2021-11-22 17:35:32 +08:00
Shawn
63f942a1b8 feat(bridge): add descriptions to the mqtt bridge schema 2021-11-22 17:35:32 +08:00
Shawn
dd9e2c4b24 refactor(bridge): the configs for http bridges 2021-11-22 17:35:32 +08:00