7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics
...
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
2023-02-27 14:55:41 +03:00
72fe10f18b
chore: fix spell check errors
2023-02-10 19:41:14 +01:00
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start
...
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 20:40:37 +04:00
cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses
2023-02-10 19:15:48 +03:00
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 17:17:59 +03:00
e196d6f42a
fix(emqx_bridge): unwrap rpc multicall result
2023-02-10 12:00:20 +01:00
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
...
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
4e71f1cbc5
Merge pull request #9940 from zmstone/0208-ensure-no-leading-br-in-docs
...
Add style checks for i18n.conf files
2023-02-09 17:37:44 +01:00
5328955ff4
fix(config): add back reconnect_interval field as deprecated
2023-02-09 17:30:09 +01:00
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events
2023-02-09 14:31:42 +03:00
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
f89ac54b17
style: remove stale comment
2023-02-08 14:04:09 +01:00
a3fd0897bc
refactor: less code duplication
2023-02-08 14:04:09 +01:00
502f62e18d
fix: try old 'restart' behavior if 'start' is not implemented
2023-02-08 13:19:41 +01:00
86f3f5787f
feat: allow to manually re-connect disconected bridge
2023-02-07 11:58:30 +01:00
4e12a44ee6
fix: return 204 instead of 200 if there's no body content
2023-02-06 09:47:07 +01:00
a71d983ff8
test: fix assertMatch patter to make rebar3 fmt work
2023-02-02 16:37:45 +01:00
85fb53dcf0
refactor: use compile switch to diff functions between ce and ee
2023-02-02 15:15:31 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly
2023-02-01 16:28:16 +03:00
5ebceb20d2
test(mqtt-bridge): also test reconfiguration
2023-02-01 16:23:58 +03:00
ad88938d34
refactor: reuse some parts of test code for brewity
2023-02-01 16:22:19 +03:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
...
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker
...
EMQX 8774 failure to handle timeout error in resource worker
2023-01-27 14:36:44 +01:00
c47be57c59
fix(bridge): ensure all bridge resources are stopped before app stop
2023-01-27 12:39:05 +01:00
7005b71ddf
style: fix typo in comment
2023-01-27 11:43:51 +01:00
8836494542
fix: redact influxdb tokens in a few logs
2023-01-25 14:48:32 +01:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames
...
fix: ensure no colon in filenames
2023-01-18 09:36:16 +01:00
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
...
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
478fcc6ffd
test: fix flaky test
2023-01-17 16:48:48 -03:00
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
d164e5bc0f
fix: don't require ssl conf
2023-01-17 15:29:19 +01:00
c3d5c25c26
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-17 17:42:10 +08:00
676f017ec0
fix: ensure no colon in filenames
2023-01-16 21:27:01 +01:00
67909f0b40
fix: testing metrics for emqx_bridge_mqtt_SUITE
2023-01-16 12:10:06 +01:00
a19226326c
fix: don't test for strict values as that makes test flaky
2023-01-16 12:10:06 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
8f5881d1a5
fix: remove stale request/3 from merge error
2023-01-13 16:19:35 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
c3133fb6a2
fix(emqx_bridge): small fixes from review
2023-01-13 14:54:54 +01:00
8dd52e5a6e
chore: add translations to schemas
2023-01-13 14:19:23 +01:00
f1c58c34ed
test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE
2023-01-13 14:19:23 +01:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge
2023-01-13 18:12:53 +08:00
42f42de4d9
feat(emqx_bridge): add separate endpoint for enable/disable of bridge
...
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
c2fd1a4482
feat(emqx_bridge): shorten operation endpoint URLs
...
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
c5f557e315
fix: disable basic auth for HTTP API
2023-01-12 21:35:49 +08:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
48e1ba4832
feat(docs): add tags to schemas
...
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
2023-01-11 09:10:03 -03:00
6f0e228a9b
Merge pull request #9717 from zmstone/0110-do-not-start-resource-after-recreate-if-disabled
...
fix(emqx_resource_manager): do not start when disabled
2023-01-11 16:12:53 +08:00
85a8eff90b
fix(emqx_resource_manager): do not start when disabled
2023-01-11 08:33:48 +01:00
ee89de816c
fix: upgrade bridge config converter
2023-01-10 20:42:30 +01:00
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
...
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
724015af47
fix: remove the default value of webhook body field
2023-01-10 09:45:25 +08:00
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
...
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
c85a988a43
fix: split 'server' param into host and port for mqtt bridge
2023-01-09 13:27:42 +01:00
c42c99f94f
fix: set a timeout for tcp connect
2023-01-09 10:01:35 +01:00
96ca0d9f49
feat(emqx_bridge): add /bridges_probe API endpoint
2023-01-09 10:01:33 +01:00
13b2f45405
fix: function_clause when sending messages to bridges
2023-01-06 09:24:38 +08:00
b59c4c34c5
fix(Bridge REST API): no feedback when deleting bridge
...
This fixes https://emqx.atlassian.net/browse/EMQX-8648 . The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:
```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist '
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```
After the fix, deleting a non existing bridge will give the following
response:
```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```
Closes: EMQX-8648
2023-01-05 16:19:46 +01:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
b43be50a98
test: add the `redacted` test case for bridges api
2023-01-03 22:16:34 +08:00
b016695cb1
fix(bridges): obfuscate the password in bridges API responses
2023-01-03 22:09:29 +08:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
305ed68916
chore: bump app vsns
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
0b43ae621d
ci: dump docker-compose log if failed to run ct
2022-12-29 09:23:11 +01:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
c085ffa0fe
refactor: default mqtt bridgge buffer pool size down to 4
2022-12-19 23:59:46 +01:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
56066a03b5
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 20:04:20 +01:00
2d7099e3ae
refactor(emqx_bridge_resource): rename a variable
2022-12-14 15:53:42 +01:00
42c58e2a91
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 15:29:13 +01:00
b9bc82f87a
feat(gcp_pubsub): add `local_topic` config
...
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
2022-12-12 17:18:19 -03:00
c69022f3c4
test(ci): clean config after test
...
Trying to fix this error for profile `emqx`:
```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
crasher:
initial call: application_master:init/4
pid: <0.9682.1>
registered_name: []
exception exit: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
in function application_master:init/4 (application_master.erl, line 142)
ancestors: [<0.9681.1>]
message_queue_len: 1
messages: [{'EXIT',<0.9683.1>,normal}]
links: [<0.9681.1>,<0.44.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 29
reductions: 195
neighbours:
=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
application: emqx_conf
exited: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
type: temporary
%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}
%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
{failed,
{emqx_plugins_SUITE,init_per_suite,
{'EXIT',
{{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,
path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1224}]}]}}}}}
```
2022-12-12 17:18:13 -03:00
6b6bb09d4a
fix: ingress only bridge causes egress bridge traffic stop
2022-12-09 15:36:20 +01:00
40809b2ad0
Merge remote-tracking branch 'origin/dev/ee5.0' into release-50
2022-12-09 11:45:52 +01:00
6692b0c895
feat(bridge): add Redis bridge
2022-12-06 23:15:42 +03:00
31098d6c67
test: fix bridge api tests, metrics is now a separate api
2022-12-06 19:03:32 +01:00
eb017ab034
test: add test case to cover mqtt bridge config upgrade
2022-12-01 16:50:10 +01:00
9ea22d062d
refactor: make all bridges optional (required = false)
2022-12-01 16:50:10 +01:00
9a6901987f
feat: add mqtt bridge config upgrade converter
2022-11-30 21:01:59 +01:00
983e904858
fix(connector): fix ssl clear
2022-11-30 17:03:41 +01:00
b398617614
chore: bump app versions
2022-11-28 21:12:43 +01:00
7ee53e5319
Merge tag 'v5.0.11' into dev/ee5.0
2022-11-28 21:02:21 +01:00
6dc5078a96
fix: fix obsolete SSL files aren't deleted after the bridge configuration update
2022-11-23 23:49:43 +01:00
1719414c04
chore: update app versions
2022-11-23 11:42:49 +08:00
Andrew Mayorov