yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,108 Commits 393 Branches 669 Tags 74 MiB
52f4519eeb
Commit Graph

574 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
91a57faa95
Merge pull request #10128 from thalesmg/ocsp-v50-mkII 
feat: add ocsp stapling support to mqtt ssl listener (5.0)
 2023-03-16 13:10:48 -03:00
Andrew Mayorov
816667d85c
Merge pull request #10123 from fix/EMQX-9136/lookup-busy-resources 
perf(bridge-api): ask nodes' bridge listings in parallel
 2023-03-16 00:07:12 +03:00
Thales Macedo Garitezi
03b95073fc test: fix inter-suite flakiness 2023-03-15 14:25:41 -03:00
Andrew Mayorov
b3e7e51094
test(bridge): drop unnecessary cleanup routines 
Since `end_per_testcase` cleans out all the resources anyway.
 2023-03-15 19:17:29 +03:00
Andrew Mayorov
cad6492c99
perf(bridge-api): ask bridge listings in parallel 
Also rename response formatting functions to better clarify their
purpose.
 2023-03-15 19:17:29 +03:00
Andrew Mayorov
53bc27e0f4
refactor(bridge): avoid unnecessary maps:to_list/1 when listing 2023-03-13 14:49:38 +03:00
Stefan Strigler
c1adf0de1f test(emqx_bridge): increase coverage by adding common edge cases 2023-03-13 10:40:47 +01:00
Stefan Strigler
478601ee41 fix(emqx_bridge): remove unreachable code paths 2023-03-13 10:40:47 +01:00
Stefan Strigler
dba95ec0fa style(emqx_bridge): fix wording 2023-03-13 10:40:47 +01:00
Stefan Strigler
71ec77a2f2 fix(emqx_bridge): for node operation check if node is part of cluster 
This fixes the case where we returned `501 NOT IMPLEMENTED` in the past.
 2023-03-13 10:40:47 +01:00
Stefan Strigler
7124600a71 docs(emqx_bridge): add start as operation to bridge 2023-03-13 10:38:51 +01:00
Stefan Strigler
1bcc5623ed fix(emqx_bridge): check if bridge enabled before calling op 2023-03-13 10:38:49 +01:00
Stefan Strigler
a325133391 fix(emqx_bridge): don't crash checking if bridge enabled 2023-03-13 10:37:28 +01:00
Stefan Strigler
80b81748df fix(emqx_bridge): handle bridge not found in call_operation 2023-03-13 10:37:28 +01:00
Stefan Strigler
fb3d101b3a refactor(emqx_bridge): fix var names 2023-03-13 10:37:28 +01:00
Stefan Strigler
e31f4d6091 refactor(emqx_bridge): add BAD_REQUEST macro and minor cleanups 2023-03-13 10:37:28 +01:00
ieQu1
e3595f2e79 chore(mria): Bump version to 0.4.0 2023-03-11 00:37:25 +01:00
Andrew Mayorov
a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi
fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
firest
b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Zaiming (Stone) Shi
26b29185b2 test(emqx_bridge_webhook_SUITE): fix flakyness in test web server 2023-03-07 20:57:38 +01:00
Kjell Winblad
163b33ab28 test: remove unnecessary dependencies of ee apps 2023-03-07 20:57:38 +01:00
Kjell Winblad
ca947e3e70 fix: lost messages when HTTP connection times out 
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.

Fixes: https://emqx.atlassian.net/browse/EMQX-8974
 2023-03-07 20:57:19 +01:00
Thales Macedo Garitezi
e17ad320ee fix(bridge): do not log in converter 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi
2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test 
test: fix flaky mqtt bridge test
 2023-03-06 13:06:37 -03:00
Stefan Strigler
a81d1a4618
Merge pull request #10066 from sstrigler/EMQX-8932-bridge-api-reports-500-for-most-errors-when-starting-a-resource 
fix: return human readable error message for most common cases
 2023-03-06 14:13:34 +01:00
Thales Macedo Garitezi
f95a30ae89 fix(webhook): convert request_timeouts in root and resource_opts 2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi
8fbb948b6f test: fix flaky mqtt bridge test 
Sometimes, this test fails because the metrics are still in the
inflight phase.
 2023-03-06 09:09:33 -03:00
Kjell Winblad
a638cc6566
Merge pull request #9985 from kjellwinblad/kjell/feat/clickhouse_bridge_2/EMQX-8391 
feat: add clickhouse database bridge
 2023-03-04 07:05:05 +01:00
Stefan Strigler
570bf165af fix: return human readable error message for most common cases 2023-03-03 16:49:09 +01:00
Ivan Dyachkov
c01f62a1c1 chore: bump apps versions 2023-03-03 13:58:32 +01:00
Kjell Winblad
67acdf0888 feat: add clickhouse database bridge 
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
 2023-03-02 12:22:11 +01:00
Stefan Strigler
39e1cf9502 fix(emqx_bridge): let it crash instead of 400 on failed reset 2023-03-02 11:23:07 +01:00
Stefan Strigler
4c23ab097d fix(emqx_bridge): return 400 if operation not possible 2023-03-02 11:23:07 +01:00
Stefan Strigler
680a4362d1 fix: match macro variables 2023-03-02 10:12:56 +01:00
Stefan Strigler
76f97010bb style: change wording 2023-03-02 09:14:52 +01:00
Serge Tupchii
77dea0c77a fix: Handle obfuscated fields in bridges_probe API 2023-03-01 22:05:23 +02:00
Stefan Strigler
3f324f85bc fix: consistently return 404 in case bridge is not found or invalid 
Also: fix some typos and be more verbose
 2023-03-01 18:03:49 +01:00
Zaiming (Stone) Shi
56279f6305 docs: fix bridge status report 2023-02-28 11:56:46 +01:00
Andrew Mayorov
7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics 
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
 2023-02-27 14:55:41 +03:00
Zaiming (Stone) Shi
72fe10f18b chore: fix spell check errors 2023-02-10 19:41:14 +01:00
Andrew Mayorov
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start 
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
 2023-02-10 20:40:37 +04:00
Andrew Mayorov
cbb2885499
fix(mqtt-bridge): allow to configure clean_start for ingresses 2023-02-10 19:15:48 +03:00
Andrew Mayorov
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges 2023-02-10 17:17:59 +03:00
Stefan Strigler
e196d6f42a fix(emqx_bridge): unwrap rpc multicall result 2023-02-10 12:00:20 +01:00
Zaiming (Stone) Shi
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field 
fix(config): add back reconnect_interval field as deprecated
 2023-02-10 09:19:12 +01:00
Zaiming (Stone) Shi
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour 
EMQX 8861 improve bridge restart button behaviour
 2023-02-09 18:00:48 +01:00
Zaiming (Stone) Shi
4e71f1cbc5
Merge pull request #9940 from zmstone/0208-ensure-no-leading-br-in-docs 
Add style checks for i18n.conf files
 2023-02-09 17:37:44 +01:00
Zaiming (Stone) Shi
5328955ff4 fix(config): add back reconnect_interval field as deprecated 2023-02-09 17:30:09 +01:00
Andrew Mayorov
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events 2023-02-09 14:31:42 +03:00
Zaiming (Stone) Shi
157c919ba1 ci: add i18n style check script 2023-02-09 11:41:52 +01:00
Stefan Strigler
f89ac54b17 style: remove stale comment 2023-02-08 14:04:09 +01:00
Stefan Strigler
a3fd0897bc refactor: less code duplication 2023-02-08 14:04:09 +01:00
Stefan Strigler
502f62e18d fix: try old 'restart' behavior if 'start' is not implemented 2023-02-08 13:19:41 +01:00
Stefan Strigler
86f3f5787f feat: allow to manually re-connect disconected bridge 2023-02-07 11:58:30 +01:00
Stefan Strigler
4e12a44ee6 fix: return 204 instead of 200 if there's no body content 2023-02-06 09:47:07 +01:00
Zaiming (Stone) Shi
a71d983ff8 test: fix assertMatch patter to make rebar3 fmt work 2023-02-02 16:37:45 +01:00
Zaiming (Stone) Shi
85fb53dcf0 refactor: use compile switch to diff functions between ce and ee 2023-02-02 15:15:31 +01:00
Andrew Mayorov
0912f13c1f
fix(mqtt-bridge): stop respecting clean_start config parameter 
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
 2023-02-02 12:55:10 +03:00
Andrew Mayorov
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly 2023-02-01 16:28:16 +03:00
Andrew Mayorov
5ebceb20d2
test(mqtt-bridge): also test reconfiguration 2023-02-01 16:23:58 +03:00
Andrew Mayorov
ad88938d34
refactor: reuse some parts of test code for brewity 2023-02-01 16:22:19 +03:00
Andrew Mayorov
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter 2023-01-31 18:30:42 +03:00
Andrew Mayorov
d0c10b59aa
feat(mqtt-bridge): avoid middleman process 
Instead, supervise `emqtt` client process directly.
 2023-01-31 17:59:03 +03:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi
c47be57c59 fix(bridge): ensure all bridge resources are stopped before app stop 2023-01-27 12:39:05 +01:00
Stefan Strigler
7005b71ddf style: fix typo in comment 2023-01-27 11:43:51 +01:00
Erik Timan
8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Ivan Dyachkov
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames 
fix: ensure no colon in filenames
 2023-01-18 09:36:16 +01:00
Stefan Strigler
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub 
EMQX 8754 test function return 500 of data integration google pubsub
 2023-01-17 22:49:28 +01:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
478fcc6ffd test: fix flaky test 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Stefan Strigler
d164e5bc0f fix: don't require ssl conf 2023-01-17 15:29:19 +01:00
firest
c3d5c25c26 fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12 2023-01-17 17:42:10 +08:00
Ivan Dyachkov
676f017ec0 fix: ensure no colon in filenames 2023-01-16 21:27:01 +01:00
Stefan Strigler
67909f0b40 fix: testing metrics for emqx_bridge_mqtt_SUITE 2023-01-16 12:10:06 +01:00
Stefan Strigler
a19226326c fix: don't test for strict values as that makes test flaky 2023-01-16 12:10:06 +01:00
JimMoen
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge 
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
 2023-01-16 09:57:20 +08:00
Stefan Strigler
8f5881d1a5 fix: remove stale request/3 from merge error 2023-01-13 16:19:35 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Stefan Strigler
1690a6dcfc
Merge branch 'master' into dev/api-refactor 2023-01-13 15:34:13 +01:00
Erik Timan
c3133fb6a2 fix(emqx_bridge): small fixes from review 2023-01-13 14:54:54 +01:00
Erik Timan
8dd52e5a6e chore: add translations to schemas 2023-01-13 14:19:23 +01:00
Erik Timan
f1c58c34ed test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE 2023-01-13 14:19:23 +01:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
JimMoen
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge 2023-01-13 18:12:53 +08:00
Erik Timan
42f42de4d9 feat(emqx_bridge): add separate endpoint for enable/disable of bridge 
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
 2023-01-12 19:39:43 +01:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Erik Timan
c2fd1a4482 feat(emqx_bridge): shorten operation endpoint URLs 
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
 2023-01-12 19:36:22 +01:00
firest
c5f557e315 fix: disable basic auth for HTTP API 2023-01-12 21:35:49 +08:00
Stefan Strigler
8ad8288195 feat: report error in create_dry_run 2023-01-11 14:22:37 +01:00
Thales Macedo Garitezi
48e1ba4832 feat(docs): add tags to schemas 
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
 2023-01-11 09:10:03 -03:00
zhongwencool
6f0e228a9b
Merge pull request #9717 from zmstone/0110-do-not-start-resource-after-recreate-if-disabled 
fix(emqx_resource_manager): do not start when disabled
 2023-01-11 16:12:53 +08:00
Zaiming (Stone) Shi
85a8eff90b fix(emqx_resource_manager): do not start when disabled 2023-01-11 08:33:48 +01:00
Zaiming (Stone) Shi
ee89de816c fix: upgrade bridge config converter 2023-01-10 20:42:30 +01:00
Xinyu Liu
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body 
fix: remove the default value of webhook body field
 2023-01-10 18:23:50 +08:00
Shawn
724015af47 fix: remove the default value of webhook body field 2023-01-10 09:45:25 +08:00
Kjell Winblad
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648 
fix: no feedback when deleting non-existing configuration
 2023-01-09 13:52:40 +01:00
Stefan Strigler
c85a988a43 fix: split 'server' param into host and port for mqtt bridge 2023-01-09 13:27:42 +01:00
Stefan Strigler
c42c99f94f fix: set a timeout for tcp connect 2023-01-09 10:01:35 +01:00
Stefan Strigler
96ca0d9f49 feat(emqx_bridge): add /bridges_probe API endpoint 2023-01-09 10:01:33 +01:00
Shawn
13b2f45405 fix: function_clause when sending messages to bridges 2023-01-06 09:24:38 +08:00
Kjell Winblad
b59c4c34c5 fix(Bridge REST API): no feedback when deleting bridge 
This fixes https://emqx.atlassian.net/browse/EMQX-8648. The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:

```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist'
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```

After the fix, deleting a non existing bridge will give the following
response:

```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```

Closes: EMQX-8648
 2023-01-05 16:19:46 +01:00
Thales Macedo Garitezi
fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00
firest
b43be50a98 test: add the redacted test case for bridges api 2023-01-03 22:16:34 +08:00
firest
b016695cb1 fix(bridges): obfuscate the password in bridges API responses 2023-01-03 22:09:29 +08:00
Thales Macedo Garitezi
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50 
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
 2023-01-02 10:56:47 -03:00
Zaiming (Stone) Shi
dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Thales Macedo Garitezi
305ed68916 chore: bump app vsns 2022-12-30 16:51:24 -03:00
Thales Macedo Garitezi
8b060a75f1 refactor(metrics): use absolute gauge values rather than deltas 
https://emqx.atlassian.net/browse/EMQX-8548

Currently, we face several issues trying to keep resource metrics
reasonable.  For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.

Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID.  This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
 2022-12-30 16:51:24 -03:00
Zaiming (Stone) Shi
0b43ae621d ci: dump docker-compose log if failed to run ct 2022-12-29 09:23:11 +01:00
Thales Macedo Garitezi
35dc75b7ed feat(mqtt): add option to customize clientid prefix for egress bridges 
https://emqx.atlassian.net/browse/EMQX-8445

Currently the bridge client’s client ID is prefixed with the resource
ID.

Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
 2022-12-23 09:50:26 -03:00
Zaiming (Stone) Shi
c085ffa0fe refactor: default mqtt bridgge buffer pool size down to 4 2022-12-19 23:59:46 +01:00
Zaiming (Stone) Shi
9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi
56066a03b5 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 20:04:20 +01:00
Zaiming (Stone) Shi
2d7099e3ae refactor(emqx_bridge_resource): rename a variable 2022-12-14 15:53:42 +01:00
Zaiming (Stone) Shi
42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi
b9bc82f87a feat(gcp_pubsub): add local_topic config 
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi
c69022f3c4 test(ci): clean config after test 
Trying to fix this error for profile `emqx`:

```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.9682.1>
    registered_name: []
    exception exit: {bad_return,
                        {{emqx_conf_app,start,[normal,[]]},
                         {emqx_conf_schema,
                             [#{kind => validation_error,path => "bridges",
                                reason => unknown_fields,
                                unknown => <<"influxdb_api_v1">>,
                                unmatched => <<"mqtt,webhook">>}]}}}
      in function  application_master:init/4 (application_master.erl, line 142)
    ancestors: [<0.9681.1>]
    message_queue_len: 1
    messages: [{'EXIT',<0.9683.1>,normal}]
    links: [<0.9681.1>,<0.44.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 29
    reductions: 195
  neighbours:

=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
    application: emqx_conf
    exited: {bad_return,
                {{emqx_conf_app,start,[normal,[]]},
                 {emqx_conf_schema,
                     [#{kind => validation_error,path => "bridges",
                        reason => unknown_fields,
                        unknown => <<"influxdb_api_v1">>,
                        unmatched => <<"mqtt,webhook">>}]}}}
    type: temporary

%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
     {emqx_conf,
         {bad_return,
             {{emqx_conf_app,start,[normal,[]]},
              {emqx_conf_schema,
                  [#{kind => validation_error,path => "bridges",
                     reason => unknown_fields,
                     unknown => <<"influxdb_api_v1">>,
                     unmatched => <<"mqtt,webhook">>}]}}}}},
 [{emqx_common_test_helpers,start_app,4,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
       {line,227}]},
  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
  {emqx_plugins_SUITE,init_per_suite,1,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
       {line,34}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
    {failed,
        {emqx_plugins_SUITE,init_per_suite,
            {'EXIT',
                {{failed_to_start_app,emqx_conf,
                     {emqx_conf,
                         {bad_return,
                             {{emqx_conf_app,start,[normal,[]]},
                              {emqx_conf_schema,
                                  [#{kind => validation_error,
                                     path => "bridges",
                                     reason => unknown_fields,
                                     unknown => <<"influxdb_api_v1">>,
                                     unmatched => <<"mqtt,webhook">>}]}}}}},
                 [{emqx_common_test_helpers,start_app,4,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                       {line,227}]},
                  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
                  {emqx_plugins_SUITE,init_per_suite,1,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
                       {line,34}]},
                  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
                  {test_server,run_test_case_eval1,6,
                      [{file,"test_server.erl"},{line,1380}]},
                  {test_server,run_test_case_eval,9,
                      [{file,"test_server.erl"},{line,1224}]}]}}}}}
```
 2022-12-12 17:18:13 -03:00
William Yang
6b6bb09d4a fix: ingress only bridge causes egress bridge traffic stop 2022-12-09 15:36:20 +01:00
Zaiming (Stone) Shi
40809b2ad0 Merge remote-tracking branch 'origin/dev/ee5.0' into release-50 2022-12-09 11:45:52 +01:00
Ilya Averyanov
6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi
31098d6c67 test: fix bridge api tests, metrics is now a separate api 2022-12-06 19:03:32 +01:00
Zaiming (Stone) Shi
eb017ab034 test: add test case to cover mqtt bridge config upgrade 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi
9ea22d062d refactor: make all bridges optional (required = false) 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi
9a6901987f feat: add mqtt bridge config upgrade converter 2022-11-30 21:01:59 +01:00
Zaiming (Stone) Shi
983e904858 fix(connector): fix ssl clear 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi
b398617614 chore: bump app versions 2022-11-28 21:12:43 +01:00
Zaiming (Stone) Shi
7ee53e5319 Merge tag 'v5.0.11' into dev/ee5.0 2022-11-28 21:02:21 +01:00
firest
6dc5078a96 fix: fix obsolete SSL files aren't deleted after the bridge configuration update 2022-11-23 23:49:43 +01:00
firest
1719414c04 chore: update app versions 2022-11-23 11:42:49 +08:00
firest
19405114e2 fix: fix newly found unsafe binary_to_atom 2022-11-23 11:07:20 +08:00
Zaiming (Stone) Shi
e5ced07665 fix: add no_return function spec to make dialyzer happy 2022-11-16 19:07:51 +01:00
Zaiming (Stone) Shi
7305923d12 fix: bridge name parser should not leak atom 2022-11-16 18:25:42 +01:00
Zaiming (Stone) Shi
c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Zaiming (Stone) Shi
09455edae8 Merge tag 'v5.0.10' into dev/ee5.0 2022-11-16 16:20:30 +01:00
firest
6c52d5de1f fix(bridge): Ensure that the node name is known 2022-11-01 15:36:10 +08:00
firest
73b4ac9f65 fix(bridge): keep bridge name type as binary don't convert it to atom 
After investigation, it was confirmed that there was no need to convert the bridge name to atom
 2022-11-01 11:24:24 +08:00
Zaiming (Stone) Shi
c157392452 docs: fix self-closing html tag, change </br> to <br/> 2022-10-27 13:57:18 +02:00
Thales Macedo Garitezi
2d01726b22 fix: account calls when resource is not connected as matched 2022-10-13 15:32:04 -03:00
Thales Macedo Garitezi
f0ff32c031 test: fix tests after counter changes 2022-10-11 17:45:48 -03:00
Shawn
dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Shawn
4135910b42 chore: merge master into dev/ee5.0 2022-09-26 09:52:33 +08:00
Zaiming (Stone) Shi
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector 
feat: Add Kafka connector
 2022-09-24 22:57:50 +02:00
Shawn
d24441b204 fix(bridges): some issues found in code review 2022-09-23 18:32:39 +08:00
Zaiming (Stone) Shi
7b601bf970 chore: delete bad parse_bridge function clause 2022-09-23 11:53:02 +02:00
Kjell Winblad
adc67b165b test: test cases for Kafka bridge REST API 2022-09-23 10:09:07 +02:00
JimMoen
6788881642 fix: hide influxdb udp config 2022-09-20 14:46:59 +08:00
Xinyu Liu
cc327629c3
Merge pull request #8992 from terry-xiaoyu/fast_resume_from_inflight_full 
Fast resume from inflight full
 2022-09-18 17:49:04 +08:00
Shawn
9aa7e826cb refactor(resource): fast resume resource worker if inflight msgs are ACKed 2022-09-17 00:34:30 +08:00
JimMoen
0336b935a0
Merge pull request #8901 from JimMoen/fix-bridge-api 
- for connectors using ecpool, parse connect error `Reason` when start pool failed.
- Refine InfluxDB bridge desc.
 2022-09-16 10:22:06 +08:00
Shawn
d5d3972ff5 chore: add test cases for MQTT Bridge reconnecting 2022-09-15 10:19:33 +08:00
JimMoen
0390a5e547 fix(bridge): mysql bridge error case 2022-09-15 09:36:52 +08:00
Shawn
f41adb0997 refactor: change some default values of resource_opts 2022-09-14 15:18:07 +08:00
JimMoen
f018b8ab49 fix(bridge/authz/exhook): no need to drop invalid certs 2022-09-14 14:08:55 +08:00
Zaiming (Stone) Shi
0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Shawn
b9ae4ea276 refactor: rename some metrics for emqx_resource 2022-09-13 14:04:25 +08:00
Zaiming (Stone) Shi
81ec4c4403
Merge pull request #8906 from zmstone/merge-master-to-ee50-a 
Merge master to ee50
 2022-09-07 11:19:41 +02:00
Zaiming (Stone) Shi
befc4acced Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a 2022-09-06 20:31:38 +02:00
Shawn
60a90858f8 feat: check dependent actions before removing the bridges 2022-09-06 15:34:23 +08:00
Xinyu Liu
23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge 
feat: add mongodb bridge (e5.0)
 2022-09-03 12:27:01 +08:00
Xinyu Liu
7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources 
Some improvements and bug fixes to emqx_resources
 2022-09-03 12:16:49 +08:00
JimMoen
d533e35df0 chore: update app vsn 2022-09-02 22:26:21 +08:00
JimMoen
2693fc263d fix(bridge): show file paths for SSL certs in dashboard for bridge 2022-09-02 14:12:03 +08:00
Shawn
83f21b4c65 refactor(resource): remove metrics 'sent.exception' 2022-09-02 12:46:53 +08:00
Shawn
b45f3de8db refactor(resource): rename metrics batched,queued -> batching,queuing 2022-09-02 12:41:14 +08:00
Thales Macedo Garitezi
3d4afd65df feat: add mongodb bridge (e5.0) 2022-09-01 14:47:14 -03:00
Zaiming (Stone) Shi
8a5a9ca2bf
Merge pull request #8854 from zmstone/0831-chore-merge-main-to-dev-ee50 
0831 chore merge main to dev ee50
 2022-09-01 11:35:23 +01:00
JianBo He
c1afb34a86 test: fix failed tests 2022-09-01 11:34:31 +08:00
JianBo He
a5ac5b6f3a
Merge branch 'dev/ee5.0' into refactor-mqtt-bridge 2022-09-01 09:30:37 +08:00
Zaiming (Stone) Shi
dac178cbaf chore: ensure version bumps 2022-08-31 17:23:47 +02:00
Shawn
ba1f5eecd3 fix: update the swagger for new resource metrics 2022-08-31 11:14:36 +08:00
Shawn
73e19d84ee feat: use the new metrics to bridge APIs 2022-08-30 23:47:58 +08:00
Shawn
65dfa63324 fix: update the counters for data bridges 2022-08-30 12:28:01 +08:00
JianBo He
a6eff81163 chore: update emqtt to 1.7.0-rc.1 2022-08-30 09:50:36 +08:00
JianBo He
0aa10702db feat(bridge): support async mode resource options 2022-08-30 09:34:56 +08:00
JimMoen
ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
JimMoen
43c964c87e fix(docs): ee bridge api docs generation 
Use try catch. Because function was unexported before called.
 2022-08-24 09:52:38 +08:00
Shawn
45352206a3 fix(auth): remove emqx_connector from testcases of authz/authn 2022-08-22 20:20:45 +08:00
Shawn
55c18c0b5f fix(bridges): update the test cases for new config structure 2022-08-22 18:24:59 +08:00
Shawn
aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
Shawn
9e35032d78 fix: make resume_interval defaults to health_check_interval 2022-08-16 10:09:02 +08:00
Xinyu Liu
2898966439
Merge branch 'dev/ee5.0' into resource_opts 2022-08-15 21:43:22 +08:00
Shawn
19d85d485b refactor(resource): add resource_opts level into config structure 2022-08-15 21:40:10 +08:00
Shawn
b3162fe5ff fix: conflicts between master and ee5.0 2022-08-14 22:24:09 +08:00
Shawn
3cb79e811f fix: remove the extra fields max_retries 2022-08-12 18:05:03 +08:00
Shawn
0cdf4b47f1 feat: add more resource creation opts 2022-08-12 13:47:45 +08:00
Xinyu Liu
a05cc20db7
Merge pull request #8693 from JimMoen/feat-influxdb-batch-query 
Feat influxdb batch query
 2022-08-12 00:22:23 +08:00
JimMoen
83746daad5 fix: update bridge config badmap error 2022-08-11 23:15:53 +08:00
JimMoen
458dab53c5 fix: undefined_functions dialyzer warning 2022-08-11 22:55:22 +08:00
JimMoen
ed796acb95 fix: fetch resource options after reboot 2022-08-11 22:11:28 +08:00
Shawn
88388b0c54 feat: http connector support async sending 2022-08-11 21:44:22 +08:00
JimMoen
3a76a50382 fix: syntax error and compile error 2022-08-11 20:58:43 +08:00
Shawn
2872f0b668 fix(bridges): support create resources with options 2022-08-11 19:11:44 +08:00
Shawn
6203a01320 feat: add inflight window to emqx_resource 2022-08-11 08:36:35 +08:00
Shawn
0377d3cf61 fix: update existing testcases for new emqx_resource 2022-08-10 00:34:35 +08:00
JimMoen
2e336fbc01 fix: bridge and connector namespace contains DB name 
Contains Detailed DB name for identifier hocon schema refs.
 2022-08-07 19:37:44 +08:00
DDDHuang
55b9684541 fix: influxdb bridge api available 2022-08-01 10:28:07 +08:00
DDDHuang
3fe76446bc Merge branch 'master' into dev/ee5.0 2022-08-01 10:14:21 +08:00
DDDHuang
a4992ef1b5 fix: hstreamdb connector conf & api 2022-07-28 16:47:30 +08:00
Shawn
23eeb21b13 fix(webhook): the 'max_retries' opt means request retry times 2022-07-27 17:06:14 +08:00
DDDHuang
9ae7c62656 fix: exs deps & bad suites 2022-07-27 16:24:13 +08:00
DDDHuang
be6de4aad0 fix: ee bridge & connector remove app mod 2022-07-27 11:38:45 +08:00
DDDHuang
fa4bc921ac fix: hstream db connector & bridge, TODO: SUITE 2022-07-27 11:38:45 +08:00
DDDHuang
98b36c4681 fix: hstream db connector , TODO: start apps 2022-07-27 11:38:45 +08:00
DDDHuang
f00a7417bb feat: hstream bridge & connector 2022-07-27 11:38:45 +08:00
Shawn
6685a3c5a8 fix: remove the 'headers' field from the rule events 2022-06-17 15:57:10 +08:00
Shawn
d6ef2f7502 refactor: graceful recreate resources 2022-06-17 05:29:18 +08:00
Xinyu Liu
3b00b16abe
Merge pull request #8221 from terry-xiaoyu/fix_delayed_module_disbled_after_emqx_stop 
feat: fix the hook priorities
 2022-06-16 09:42:42 +08:00
Zaiming (Stone) Shi
738a551550 refactor: best-effort json for hocon validation errors 2022-06-15 15:01:15 +02:00
Shawn
39b1b20506 feat: fix the hook priorities 2022-06-15 19:03:40 +08:00
Shawn
dd099137f0 fix: catch errors when load bridges 2022-06-13 22:31:58 +08:00
Shawn
9a0fdc4b11 feat: support configure connector configs directly in bridges 2022-06-10 23:06:20 +08:00
Shawn
69fba6958b refactor: remove the emqx_bridge_monitor module 2022-06-01 15:31:48 +08:00
Shawn
88ca25c60c fix(resource): fast return when starting a unavailable resource 2022-06-01 08:24:53 +08:00
JianBo He
efc0ca2b62
Merge branch 'master' into changes-enable_pipelining-type 2022-05-31 20:49:10 +08:00
JianBo He
f0ddac63cb chore: change the enable_pipelining default value 2022-05-31 19:32:53 +08:00
Zhongwen Deng
d61b44ef97 fix: ct failed again 2022-05-31 19:20:27 +08:00
Zhongwen Deng
8aa60cc0a5 feat: generate a minimized emqx.conf 2022-05-31 19:20:27 +08:00
Shawn
1054c364ad refactor(resource): improve health check and alarm it if resource down 2022-05-31 01:40:40 +08:00
Shawn
a157539710 refactor(rule): rename http_bridge -> webhook 2022-05-25 18:56:12 +08:00
Shawn
71de9616d3 refactor(rule): rename outputs -> actions 2022-05-25 17:02:37 +08:00
Shawn
d89d692d3e refactor: move some APIs to new module emqx_bridge_resource 2022-05-18 15:12:34 +08:00
DDDHuang
8b0adf69d2
Merge pull request #7926 from DDDHuang/rule_api_errormsg 
fix: better error message for rule engine
 2022-05-12 19:07:47 +08:00
DDDHuang
2de69c97ba fix: ruleengine & connector & bridge api, better error message 2022-05-12 16:51:36 +08:00
Thales Macedo Garitezi
47ed9cbc76
fix(telemetry): count bridges referenced in from rule clauses 2022-05-11 15:15:17 -03:00
Chris Hicks
841acb7828
Merge pull request #7783 from emqx/EMQX-4199-introduce-a-new-emqx-resource-manager-module 
feat: isolate resource manager processes
 2022-05-09 18:33:01 +02:00
Chris
0b3e30e813 feat: isolate resource manager processes 2022-05-09 13:24:34 +02:00
Shawn
9a033af2e8 fix: restart bridges via HTTP APIs to wrong node 2022-05-05 18:47:19 +08:00
Shawn
1ce8a8c886 fix: testing connector removes ssl cert files that are in use 2022-05-05 14:55:40 +08:00
zhongwencool
c7241f2abc
Merge pull request #7813 from DDDHuang/bridge_api_bad_format 
fix: bridge api response bad node name format
 2022-04-29 09:12:22 +08:00
Zhongwen Deng
03d2a1ebb6 chore: reformat emqx_bridge_api 2022-04-29 09:08:45 +08:00
Shawn
6edb453ce9 fix: the field 'enable' is missing for GET /bridges/:id 2022-04-28 18:13:19 +08:00
Shawn
e18c2a6de2 fix: convert ssl certfiles for dry-run creating bridges 2022-04-28 16:17:55 +08:00
DDDHuang
a26b0f79f2 fix: bridge api response bad node name format 2022-04-28 16:06:57 +08:00
Zaiming (Stone) Shi
02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
Shawn
94e24c2621 refactor: move ssl file handling from resources to bridges 2022-04-27 11:59:15 +08:00
Thales Macedo Garitezi
f8f97d39d3
Merge pull request #7766 from thalesmg/bugfix-telemetry-bridge 
fix: avoid crashing telemetry if bridge app is not ready
 2022-04-26 09:03:29 -03:00
Thales Macedo Garitezi
ada3b03f6e
fix(telemetry): do not crash if bridge app is not ready yet 2022-04-25 11:23:13 -03:00
EMQ-YangM
eb3870b538 fix: when enable is false, restart and stop cannot be performed. 2022-04-25 18:48:12 +08:00
JimMoen
e9843aa225 fix(doc): html line break use </br> 2022-04-24 18:56:21 +08:00
Zhongwen Deng
5223c3ee61 feat: obfuscate sensitive values default_password 2022-04-24 09:41:29 +08:00