Commit Graph

758 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 7d798c10e9 perf(buffer_worker): flush metrics periodically inside buffer worker process
Fixes https://emqx.atlassian.net/browse/EMQX-9905

Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
2023-05-22 09:11:23 -03:00
William Yang e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path
perf(config): enforcing atom key  path in hotcode path
2023-05-22 09:38:09 +02:00
Thales Macedo Garitezi 09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
Thales Macedo Garitezi 5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi 447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Thales Macedo Garitezi b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
Thales Macedo Garitezi a7b41e1cdf perf(webhook): add retry attempts for async
This is a performance improvement for webhook bridge.

Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well.  Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.

This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
William Yang cdf42760fa chore: little change avoid atom leak 2023-05-17 11:01:03 +02:00
某文 2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
Zaiming (Stone) Shi 9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field
0421 refactor delete unused schema field
2023-05-16 22:22:03 +02:00
某文 a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
William Yang 1c746ed289 perf(config): enforcing atom key path in hotcode path 2023-05-16 09:57:04 +02:00
某文 7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
zhongwencool a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
某文 6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zaiming (Stone) Shi 7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Zhongwen Deng 14952658d5 chore: convert rotation struct to roation_count 2023-05-09 11:24:51 +08:00
Stefan Strigler d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zaiming (Stone) Shi 856129984b refactor: remove raw_with_default config load option
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
2023-05-04 09:48:28 +02:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi 57cc854a4a test(bridge): fix bridge map type filed converters
now the converters on map type fields only work at the wrapping map
level but not the values
2023-04-30 10:45:11 +02:00
Paulo Zulato dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-8398
2023-04-24 10:28:26 -03:00
Serge Tupchii 45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
Fixes: EMQX-9310
2023-04-20 17:21:43 +03:00
Zaiming (Stone) Shi 30fd9b10f7 refactor: change bridges config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Ivan Dyachkov dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
某文 fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
Ivan Dyachkov 9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Stefan Strigler 7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
Thales Macedo Garitezi e073bc90bc refactor(buffer_worker): rename `s/queue/buffer/g` 2023-04-14 11:37:19 -03:00
Stefan Strigler 4f80690162 fix: byebye jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler 062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler 9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen 790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 17:55:48 +08:00
Andrew Mayorov 5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics
feat(resource): ask for metrics only when needed
2023-04-14 12:28:52 +03:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
Zhongwen Deng cc2beda37a feat: hiden rule_engine/bridge/authz/authn from doc/example 2023-04-13 14:42:21 +08:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
Stefan Strigler 04626ce9cc fix: create consistent interface 'with_node' for API access 2023-04-04 16:54:14 +02:00
firest 11b3264251 fix: redact the password to `******` in API examples
fix #10222
2023-04-04 16:02:45 +08:00
Zaiming (Stone) Shi 36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi 80eb9d7542
Merge pull request #10252 from emqx/release-50
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
Kjell Winblad 8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
Thales Macedo Garitezi 64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi 1824e7efcc fix(rule_engine): count referenced bridges in `from` clauses as dependencies (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9325

Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.

When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
2023-03-28 10:31:10 -03:00
Stefan Strigler 193905c7b9
Merge pull request #10237 from sstrigler/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent
fix: create consistent interface 'with_node' for API access
2023-03-28 10:41:10 +02:00
Andrew Mayorov d942a531e6
fix(test): avoid unnecessary cleanups
This might make some noise in the logs, depending on the testsuite
configuration, because some test setup might be missing required
applications.
2023-03-27 23:41:19 +03:00
Andrew Mayorov 11b5b7b638
test(bridge-api): also run testcases in cluster environment
Excluding a couple of testcases which does not make much sense running
in the cluster. Also try to reduce amount of "noise" in the testcases,
making them easier to comprehend.

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 23:41:08 +03:00
Stefan Strigler 64a1d84a44 fix: create consistent interface 'with_node' for API access 2023-03-27 12:14:48 +02:00
JianBo He bfa5922209
Merge pull request #10140 from HJianBo/cassa
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
Zaiming (Stone) Shi dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
Stefan Strigler 8d084ed13b
Merge pull request #10226 from sstrigler/EMQX-9273-e-5-0-2-creating-a-data-bridge-the-name-verification-returned-an-incorrect-error-code-of-500
fix(emqx_bridge): don't crash on validation error
2023-03-24 14:48:03 +01:00
Stefan Strigler fbfdaf44e0 fix(emqx_bridge): don't crash on validation error 2023-03-24 11:46:41 +01:00
Stefan Strigler a8cfc1c7a1 fix(emqx_bridge): filter status_reason in config update 2023-03-24 11:46:41 +01:00
Stefan Strigler ba49316ec5 fix: use lower-case 2023-03-23 16:08:27 +01:00
Kjell Winblad 35474578ca refactor: rename async_inflight_window to inflight_window everywhere 2023-03-23 14:21:57 +01:00
JianBo He 8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
lafirest 84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
Zaiming (Stone) Shi e43aa5d841
Merge pull request #10199 from zmstone/0321-merge-release-50-to-master
0321 merge release 50 to master
2023-03-22 10:20:48 +01:00
Andrew Mayorov af7b2215ca
fix(bridge-api): unflatten nodes' bridge lists back
Bridge lists were erroneously flattened in cad6492c. This causes
bridge listing fail in emqx clusters consisting of more than 1 node.
2023-03-22 11:35:09 +03:00
firest 9853d00cad feat(bridges): integrate RocketMQ into data bridges 2023-03-22 10:36:52 +08:00
JianBo He 65c2da7ef5 Merge remote-tracking branch 'ce/master' into cassa 2023-03-22 09:30:50 +08:00
Andrew Mayorov bf35419b52
Merge pull request #10190 from keynslug/fix/EMQX-9269/bridge-list
fix(bridge-api): unflatten nodes' bridge lists back
2023-03-22 00:53:03 +03:00
Zaiming (Stone) Shi e6091db351 Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master 2023-03-21 22:03:31 +01:00
Stefan Strigler 6c24b3a77d
Merge pull request #10196 from sstrigler/EMQX-9201-the-case-is-not-correct-for-some-of-menu
Emqx 9201 the case is not correct for some of menu
2023-03-21 19:42:10 +01:00
Andrew Mayorov d9b1b1c802
fix(bridge-api): unflatten nodes' bridge lists back
Bridge lists were erroneously flattened in cad6492c. This causes
bridge listing fail in emqx clusters consisting of more than 1 node.
2023-03-21 18:59:27 +03:00
Stefan Strigler 981268911a style: make summary lower-case 2023-03-21 16:56:37 +01:00
Stefan Strigler 3880862c81 fix(emqx_bridge): return 503 for inconsistency in bridge setup 2023-03-21 15:07:43 +01:00
Stefan Strigler 4b0ea562a2 refactor(emqx_bridge): consistently use macros for http response 2023-03-21 15:07:41 +01:00
Stefan Strigler 84fc64822e style: fix wording for nxdomain and malformed_username_or_password
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 15:02:29 +01:00
Stefan Strigler 8af3fb4ee7 feat: move human readable error translations to emqx_misc 2023-03-21 15:02:29 +01:00
Stefan Strigler 53825b9aba fix(emqx_bridge): propagate connection error to resource status 2023-03-21 15:02:29 +01:00
Erik Timan 2d75c7d6d9 fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
JianBo He 12942b676d Merge remote-tracking branch 'upstream/master' into cassa 2023-03-20 09:50:27 +08:00
JianBo He c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Thales Macedo Garitezi 3954b7bde2 fix(bridges): function clause when a non-ingress bridge coexists with an egress bridge
This was not caught by our tests because we always test bridge types
in isolation.  So, if the config only contains ingress-only bridges,
the `on_message_publish` hook is never installed.

In a real system, if there are bridges of mixed types in the config,
the hook might be installed, and `emqx_bridge:get_matched_bridge_id`
would crash when iterating over the ingress bridges.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 66eb4ef069 test: fix inter-suite flakiness 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi f31f15e44e chore(kafka_producer): make schema changes more backwards compatible
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 03342923b9 fix(bridge): use the same dry run prefix
Kafka Producer and Consumer bridges rely on this prefix for detecting
a dry run and avoid leaking atoms.  At some point, this prefix was
changed, effectively disabling the check in Kafka Producer.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 1f31a87974 fix(bridge): improve macro var usage
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 969fa03ecc feat: implement kafka consumer 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 91a57faa95
Merge pull request #10128 from thalesmg/ocsp-v50-mkII
feat: add ocsp stapling support to mqtt ssl listener (5.0)
2023-03-16 13:10:48 -03:00
Andrew Mayorov 816667d85c
Merge pull request #10123 from fix/EMQX-9136/lookup-busy-resources
perf(bridge-api): ask nodes' bridge listings in parallel
2023-03-16 00:07:12 +03:00
Thales Macedo Garitezi 03b95073fc test: fix inter-suite flakiness 2023-03-15 14:25:41 -03:00
Andrew Mayorov b3e7e51094
test(bridge): drop unnecessary cleanup routines
Since `end_per_testcase` cleans out all the resources anyway.
2023-03-15 19:17:29 +03:00
Andrew Mayorov cad6492c99
perf(bridge-api): ask bridge listings in parallel
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
Andrew Mayorov 53bc27e0f4
refactor(bridge): avoid unnecessary `maps:to_list/1` when listing 2023-03-13 14:49:38 +03:00
Stefan Strigler c1adf0de1f test(emqx_bridge): increase coverage by adding common edge cases 2023-03-13 10:40:47 +01:00
Stefan Strigler 478601ee41 fix(emqx_bridge): remove unreachable code paths 2023-03-13 10:40:47 +01:00
Stefan Strigler dba95ec0fa style(emqx_bridge): fix wording 2023-03-13 10:40:47 +01:00
Stefan Strigler 71ec77a2f2 fix(emqx_bridge): for node operation check if node is part of cluster
This fixes the case where we returned `501 NOT IMPLEMENTED` in the past.
2023-03-13 10:40:47 +01:00
Stefan Strigler 7124600a71 docs(emqx_bridge): add `start` as operation to bridge 2023-03-13 10:38:51 +01:00
Stefan Strigler 1bcc5623ed fix(emqx_bridge): check if bridge enabled before calling op 2023-03-13 10:38:49 +01:00
Stefan Strigler a325133391 fix(emqx_bridge): don't crash checking if bridge enabled 2023-03-13 10:37:28 +01:00
Stefan Strigler 80b81748df fix(emqx_bridge): handle bridge not found in call_operation 2023-03-13 10:37:28 +01:00
Stefan Strigler fb3d101b3a refactor(emqx_bridge): fix var names 2023-03-13 10:37:28 +01:00
Stefan Strigler e31f4d6091 refactor(emqx_bridge): add BAD_REQUEST macro and minor cleanups 2023-03-13 10:37:28 +01:00
ieQu1 e3595f2e79 chore(mria): Bump version to 0.4.0 2023-03-11 00:37:25 +01:00
Andrew Mayorov a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Zaiming (Stone) Shi 26b29185b2 test(emqx_bridge_webhook_SUITE): fix flakyness in test web server 2023-03-07 20:57:38 +01:00
Kjell Winblad 163b33ab28 test: remove unnecessary dependencies of ee apps 2023-03-07 20:57:38 +01:00
Kjell Winblad ca947e3e70 fix: lost messages when HTTP connection times out
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.

Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
Thales Macedo Garitezi e17ad320ee fix(bridge): do not log in converter 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi 2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test
test: fix flaky mqtt bridge test
2023-03-06 13:06:37 -03:00
Stefan Strigler a81d1a4618
Merge pull request #10066 from sstrigler/EMQX-8932-bridge-api-reports-500-for-most-errors-when-starting-a-resource
fix: return human readable error message for most common cases
2023-03-06 14:13:34 +01:00
Thales Macedo Garitezi f95a30ae89 fix(webhook): convert `request_timeout`s in root and resource_opts 2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi 8fbb948b6f test: fix flaky mqtt bridge test
Sometimes, this test fails because the metrics are still in the
inflight phase.
2023-03-06 09:09:33 -03:00
Kjell Winblad a638cc6566
Merge pull request #9985 from kjellwinblad/kjell/feat/clickhouse_bridge_2/EMQX-8391
feat: add clickhouse database bridge
2023-03-04 07:05:05 +01:00
Stefan Strigler 570bf165af fix: return human readable error message for most common cases 2023-03-03 16:49:09 +01:00
Ivan Dyachkov c01f62a1c1 chore: bump apps versions 2023-03-03 13:58:32 +01:00
Kjell Winblad 67acdf0888 feat: add clickhouse database bridge
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
Stefan Strigler 39e1cf9502 fix(emqx_bridge): let it crash instead of 400 on failed reset 2023-03-02 11:23:07 +01:00
Stefan Strigler 4c23ab097d fix(emqx_bridge): return 400 if operation not possible 2023-03-02 11:23:07 +01:00
Stefan Strigler 680a4362d1 fix: match macro variables 2023-03-02 10:12:56 +01:00
Stefan Strigler 76f97010bb style: change wording 2023-03-02 09:14:52 +01:00
Serge Tupchii 77dea0c77a fix: Handle obfuscated fields in bridges_probe API 2023-03-01 22:05:23 +02:00
Stefan Strigler 3f324f85bc fix: consistently return 404 in case bridge is not found or invalid
Also: fix some typos and be more verbose
2023-03-01 18:03:49 +01:00
Zaiming (Stone) Shi 56279f6305 docs: fix bridge status report 2023-02-28 11:56:46 +01:00
Andrew Mayorov 7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
2023-02-27 14:55:41 +03:00
Zaiming (Stone) Shi 72fe10f18b chore: fix spell check errors 2023-02-10 19:41:14 +01:00
Andrew Mayorov 2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 20:40:37 +04:00
Andrew Mayorov cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses 2023-02-10 19:15:48 +03:00
Andrew Mayorov 7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges 2023-02-10 17:17:59 +03:00
Stefan Strigler e196d6f42a fix(emqx_bridge): unwrap rpc multicall result 2023-02-10 12:00:20 +01:00
Zaiming (Stone) Shi 3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
Zaiming (Stone) Shi 42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
Zaiming (Stone) Shi 4e71f1cbc5
Merge pull request #9940 from zmstone/0208-ensure-no-leading-br-in-docs
Add style checks for i18n.conf files
2023-02-09 17:37:44 +01:00
Zaiming (Stone) Shi 5328955ff4 fix(config): add back reconnect_interval field as deprecated 2023-02-09 17:30:09 +01:00
Andrew Mayorov 48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events 2023-02-09 14:31:42 +03:00
Zaiming (Stone) Shi 157c919ba1 ci: add i18n style check script 2023-02-09 11:41:52 +01:00
Stefan Strigler f89ac54b17 style: remove stale comment 2023-02-08 14:04:09 +01:00
Stefan Strigler a3fd0897bc refactor: less code duplication 2023-02-08 14:04:09 +01:00
Stefan Strigler 502f62e18d fix: try old 'restart' behavior if 'start' is not implemented 2023-02-08 13:19:41 +01:00
Stefan Strigler 86f3f5787f feat: allow to manually re-connect disconected bridge 2023-02-07 11:58:30 +01:00
Stefan Strigler 4e12a44ee6 fix: return 204 instead of 200 if there's no body content 2023-02-06 09:47:07 +01:00
Zaiming (Stone) Shi a71d983ff8 test: fix assertMatch patter to make rebar3 fmt work 2023-02-02 16:37:45 +01:00
Zaiming (Stone) Shi 85fb53dcf0 refactor: use compile switch to diff functions between ce and ee 2023-02-02 15:15:31 +01:00
Andrew Mayorov 0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
Andrew Mayorov 8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly 2023-02-01 16:28:16 +03:00
Andrew Mayorov 5ebceb20d2
test(mqtt-bridge): also test reconfiguration 2023-02-01 16:23:58 +03:00
Andrew Mayorov ad88938d34
refactor: reuse some parts of test code for brewity 2023-02-01 16:22:19 +03:00
Andrew Mayorov c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter 2023-01-31 18:30:42 +03:00
Andrew Mayorov d0c10b59aa
feat(mqtt-bridge): avoid middleman process
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
Zaiming (Stone) Shi 7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi 52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker
EMQX 8774 failure to handle timeout error in resource worker
2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi c47be57c59 fix(bridge): ensure all bridge resources are stopped before app stop 2023-01-27 12:39:05 +01:00
Stefan Strigler 7005b71ddf style: fix typo in comment 2023-01-27 11:43:51 +01:00
Erik Timan 8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi 5fdf7fd24c fix(kafka): use async callback to bump success counters
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
Erik Timan 28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi 6fa6c679bb feat(buffer_worker): add expiration time to requests
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
2023-01-20 11:36:52 -03:00
Thales Macedo Garitezi 47f796dd12 refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
Zaiming (Stone) Shi d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Ivan Dyachkov 430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames
fix: ensure no colon in filenames
2023-01-18 09:36:16 +01:00
Stefan Strigler f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
Thales Macedo Garitezi fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi 478fcc6ffd test: fix flaky test 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 32a9e60313 feat(buffer_worker): also use the inflight table for sync requests
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
Stefan Strigler d164e5bc0f fix: don't require ssl conf 2023-01-17 15:29:19 +01:00
firest c3d5c25c26 fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12 2023-01-17 17:42:10 +08:00
Ivan Dyachkov 676f017ec0 fix: ensure no colon in filenames 2023-01-16 21:27:01 +01:00
Stefan Strigler 67909f0b40 fix: testing metrics for emqx_bridge_mqtt_SUITE 2023-01-16 12:10:06 +01:00
Stefan Strigler a19226326c fix: don't test for strict values as that makes test flaky 2023-01-16 12:10:06 +01:00
JimMoen 54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
Stefan Strigler 8f5881d1a5 fix: remove stale request/3 from merge error 2023-01-13 16:19:35 +01:00
Stefan Strigler e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Stefan Strigler 1690a6dcfc
Merge branch 'master' into dev/api-refactor 2023-01-13 15:34:13 +01:00
Erik Timan c3133fb6a2 fix(emqx_bridge): small fixes from review 2023-01-13 14:54:54 +01:00
Erik Timan 8dd52e5a6e chore: add translations to schemas 2023-01-13 14:19:23 +01:00
Erik Timan f1c58c34ed test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE 2023-01-13 14:19:23 +01:00
Zhongwen Deng f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
JimMoen b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge 2023-01-13 18:12:53 +08:00
Erik Timan 42f42de4d9 feat(emqx_bridge): add separate endpoint for enable/disable of bridge
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
Erik Timan 860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
Erik Timan c2fd1a4482 feat(emqx_bridge): shorten operation endpoint URLs
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
firest c5f557e315 fix: disable basic auth for HTTP API 2023-01-12 21:35:49 +08:00
Stefan Strigler 8ad8288195 feat: report error in create_dry_run 2023-01-11 14:22:37 +01:00
Thales Macedo Garitezi 48e1ba4832 feat(docs): add tags to schemas
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
2023-01-11 09:10:03 -03:00
zhongwencool 6f0e228a9b
Merge pull request #9717 from zmstone/0110-do-not-start-resource-after-recreate-if-disabled
fix(emqx_resource_manager): do not start when disabled
2023-01-11 16:12:53 +08:00
Zaiming (Stone) Shi 85a8eff90b fix(emqx_resource_manager): do not start when disabled 2023-01-11 08:33:48 +01:00
Zaiming (Stone) Shi ee89de816c fix: upgrade bridge config converter 2023-01-10 20:42:30 +01:00
Xinyu Liu 0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
Shawn 724015af47 fix: remove the default value of webhook body field 2023-01-10 09:45:25 +08:00
Kjell Winblad 692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
Stefan Strigler c85a988a43 fix: split 'server' param into host and port for mqtt bridge 2023-01-09 13:27:42 +01:00
Stefan Strigler c42c99f94f fix: set a timeout for tcp connect 2023-01-09 10:01:35 +01:00
Stefan Strigler 96ca0d9f49 feat(emqx_bridge): add /bridges_probe API endpoint 2023-01-09 10:01:33 +01:00
Shawn 13b2f45405 fix: function_clause when sending messages to bridges 2023-01-06 09:24:38 +08:00
Kjell Winblad b59c4c34c5 fix(Bridge REST API): no feedback when deleting bridge
This fixes https://emqx.atlassian.net/browse/EMQX-8648. The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:

```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist'
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```

After the fix, deleting a non existing bridge will give the following
response:

```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```

Closes: EMQX-8648
2023-01-05 16:19:46 +01:00
Thales Macedo Garitezi fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
firest b43be50a98 test: add the `redacted` test case for bridges api 2023-01-03 22:16:34 +08:00
firest b016695cb1 fix(bridges): obfuscate the password in bridges API responses 2023-01-03 22:09:29 +08:00
Thales Macedo Garitezi 7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Thales Macedo Garitezi 305ed68916 chore: bump app vsns 2022-12-30 16:51:24 -03:00
Thales Macedo Garitezi 8b060a75f1 refactor(metrics): use absolute gauge values rather than deltas
https://emqx.atlassian.net/browse/EMQX-8548

Currently, we face several issues trying to keep resource metrics
reasonable.  For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.

Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID.  This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
Zaiming (Stone) Shi 0b43ae621d ci: dump docker-compose log if failed to run ct 2022-12-29 09:23:11 +01:00
Thales Macedo Garitezi 35dc75b7ed feat(mqtt): add option to customize clientid prefix for egress bridges
https://emqx.atlassian.net/browse/EMQX-8445

Currently the bridge client’s client ID is prefixed with the resource
ID.

Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
Zaiming (Stone) Shi c085ffa0fe refactor: default mqtt bridgge buffer pool size down to 4 2022-12-19 23:59:46 +01:00
Zaiming (Stone) Shi 9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi 56066a03b5 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 20:04:20 +01:00
Zaiming (Stone) Shi 2d7099e3ae refactor(emqx_bridge_resource): rename a variable 2022-12-14 15:53:42 +01:00
Zaiming (Stone) Shi 42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi b9bc82f87a feat(gcp_pubsub): add `local_topic` config
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi c69022f3c4 test(ci): clean config after test
Trying to fix this error for profile `emqx`:

```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.9682.1>
    registered_name: []
    exception exit: {bad_return,
                        {{emqx_conf_app,start,[normal,[]]},
                         {emqx_conf_schema,
                             [#{kind => validation_error,path => "bridges",
                                reason => unknown_fields,
                                unknown => <<"influxdb_api_v1">>,
                                unmatched => <<"mqtt,webhook">>}]}}}
      in function  application_master:init/4 (application_master.erl, line 142)
    ancestors: [<0.9681.1>]
    message_queue_len: 1
    messages: [{'EXIT',<0.9683.1>,normal}]
    links: [<0.9681.1>,<0.44.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 29
    reductions: 195
  neighbours:

=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
    application: emqx_conf
    exited: {bad_return,
                {{emqx_conf_app,start,[normal,[]]},
                 {emqx_conf_schema,
                     [#{kind => validation_error,path => "bridges",
                        reason => unknown_fields,
                        unknown => <<"influxdb_api_v1">>,
                        unmatched => <<"mqtt,webhook">>}]}}}
    type: temporary

%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
     {emqx_conf,
         {bad_return,
             {{emqx_conf_app,start,[normal,[]]},
              {emqx_conf_schema,
                  [#{kind => validation_error,path => "bridges",
                     reason => unknown_fields,
                     unknown => <<"influxdb_api_v1">>,
                     unmatched => <<"mqtt,webhook">>}]}}}}},
 [{emqx_common_test_helpers,start_app,4,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
       {line,227}]},
  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
  {emqx_plugins_SUITE,init_per_suite,1,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
       {line,34}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
    {failed,
        {emqx_plugins_SUITE,init_per_suite,
            {'EXIT',
                {{failed_to_start_app,emqx_conf,
                     {emqx_conf,
                         {bad_return,
                             {{emqx_conf_app,start,[normal,[]]},
                              {emqx_conf_schema,
                                  [#{kind => validation_error,
                                     path => "bridges",
                                     reason => unknown_fields,
                                     unknown => <<"influxdb_api_v1">>,
                                     unmatched => <<"mqtt,webhook">>}]}}}}},
                 [{emqx_common_test_helpers,start_app,4,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                       {line,227}]},
                  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
                  {emqx_plugins_SUITE,init_per_suite,1,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
                       {line,34}]},
                  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
                  {test_server,run_test_case_eval1,6,
                      [{file,"test_server.erl"},{line,1380}]},
                  {test_server,run_test_case_eval,9,
                      [{file,"test_server.erl"},{line,1224}]}]}}}}}
```
2022-12-12 17:18:13 -03:00
William Yang 6b6bb09d4a fix: ingress only bridge causes egress bridge traffic stop 2022-12-09 15:36:20 +01:00
Zaiming (Stone) Shi 40809b2ad0 Merge remote-tracking branch 'origin/dev/ee5.0' into release-50 2022-12-09 11:45:52 +01:00
Ilya Averyanov 6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi 31098d6c67 test: fix bridge api tests, metrics is now a separate api 2022-12-06 19:03:32 +01:00
Zaiming (Stone) Shi eb017ab034 test: add test case to cover mqtt bridge config upgrade 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi 9ea22d062d refactor: make all bridges optional (required = false) 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi 9a6901987f feat: add mqtt bridge config upgrade converter 2022-11-30 21:01:59 +01:00
Zaiming (Stone) Shi 983e904858 fix(connector): fix ssl clear 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi b398617614 chore: bump app versions 2022-11-28 21:12:43 +01:00
Zaiming (Stone) Shi 7ee53e5319 Merge tag 'v5.0.11' into dev/ee5.0 2022-11-28 21:02:21 +01:00
firest 6dc5078a96 fix: fix obsolete SSL files aren't deleted after the bridge configuration update 2022-11-23 23:49:43 +01:00
firest 1719414c04 chore: update app versions 2022-11-23 11:42:49 +08:00
firest 19405114e2 fix: fix newly found unsafe `binary_to_atom` 2022-11-23 11:07:20 +08:00
Zaiming (Stone) Shi e5ced07665 fix: add no_return function spec to make dialyzer happy 2022-11-16 19:07:51 +01:00
Zaiming (Stone) Shi 7305923d12 fix: bridge name parser should not leak atom 2022-11-16 18:25:42 +01:00
Zaiming (Stone) Shi c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Zaiming (Stone) Shi 09455edae8 Merge tag 'v5.0.10' into dev/ee5.0 2022-11-16 16:20:30 +01:00
firest 6c52d5de1f fix(bridge): Ensure that the node name is known 2022-11-01 15:36:10 +08:00
firest 73b4ac9f65 fix(bridge): keep bridge name type as binary don't convert it to atom
After investigation, it was confirmed that there was no need to convert the bridge name to atom
2022-11-01 11:24:24 +08:00
Zaiming (Stone) Shi c157392452 docs: fix self-closing html tag, change </br> to <br/> 2022-10-27 13:57:18 +02:00
Thales Macedo Garitezi 2d01726b22 fix: account calls when resource is not connected as matched 2022-10-13 15:32:04 -03:00
Thales Macedo Garitezi f0ff32c031 test: fix tests after counter changes 2022-10-11 17:45:48 -03:00
Shawn dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Shawn 4135910b42 chore: merge master into dev/ee5.0 2022-09-26 09:52:33 +08:00
Zaiming (Stone) Shi f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
Shawn d24441b204 fix(bridges): some issues found in code review 2022-09-23 18:32:39 +08:00
Zaiming (Stone) Shi 7b601bf970 chore: delete bad parse_bridge function clause 2022-09-23 11:53:02 +02:00
Kjell Winblad adc67b165b test: test cases for Kafka bridge REST API 2022-09-23 10:09:07 +02:00
JimMoen 6788881642 fix: hide influxdb udp config 2022-09-20 14:46:59 +08:00
Xinyu Liu cc327629c3
Merge pull request #8992 from terry-xiaoyu/fast_resume_from_inflight_full
Fast resume from inflight full
2022-09-18 17:49:04 +08:00
Shawn 9aa7e826cb refactor(resource): fast resume resource worker if inflight msgs are ACKed 2022-09-17 00:34:30 +08:00
JimMoen 0336b935a0
Merge pull request #8901 from JimMoen/fix-bridge-api
- for connectors using ecpool, parse connect error `Reason` when start pool failed.
- Refine InfluxDB bridge desc.
2022-09-16 10:22:06 +08:00
Shawn d5d3972ff5 chore: add test cases for MQTT Bridge reconnecting 2022-09-15 10:19:33 +08:00
JimMoen 0390a5e547 fix(bridge): mysql bridge error case 2022-09-15 09:36:52 +08:00
Shawn f41adb0997 refactor: change some default values of resource_opts 2022-09-14 15:18:07 +08:00
JimMoen f018b8ab49 fix(bridge/authz/exhook): no need to drop invalid certs 2022-09-14 14:08:55 +08:00
Zaiming (Stone) Shi 0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Shawn b9ae4ea276 refactor: rename some metrics for emqx_resource 2022-09-13 14:04:25 +08:00
Zaiming (Stone) Shi 81ec4c4403
Merge pull request #8906 from zmstone/merge-master-to-ee50-a
Merge master to ee50
2022-09-07 11:19:41 +02:00
Zaiming (Stone) Shi befc4acced Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a 2022-09-06 20:31:38 +02:00
Shawn 60a90858f8 feat: check dependent actions before removing the bridges 2022-09-06 15:34:23 +08:00
Xinyu Liu 23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge
feat: add mongodb bridge (e5.0)
2022-09-03 12:27:01 +08:00
Xinyu Liu 7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources
Some improvements and bug fixes to emqx_resources
2022-09-03 12:16:49 +08:00
JimMoen d533e35df0 chore: update app vsn 2022-09-02 22:26:21 +08:00
JimMoen 2693fc263d fix(bridge): show file paths for SSL certs in dashboard for bridge 2022-09-02 14:12:03 +08:00
Shawn 83f21b4c65 refactor(resource): remove metrics 'sent.exception' 2022-09-02 12:46:53 +08:00
Shawn b45f3de8db refactor(resource): rename metrics batched,queued -> batching,queuing 2022-09-02 12:41:14 +08:00
Thales Macedo Garitezi 3d4afd65df feat: add mongodb bridge (e5.0) 2022-09-01 14:47:14 -03:00
Zaiming (Stone) Shi 8a5a9ca2bf
Merge pull request #8854 from zmstone/0831-chore-merge-main-to-dev-ee50
0831 chore merge main to dev ee50
2022-09-01 11:35:23 +01:00
JianBo He c1afb34a86 test: fix failed tests 2022-09-01 11:34:31 +08:00
JianBo He a5ac5b6f3a
Merge branch 'dev/ee5.0' into refactor-mqtt-bridge 2022-09-01 09:30:37 +08:00
Zaiming (Stone) Shi dac178cbaf chore: ensure version bumps 2022-08-31 17:23:47 +02:00
Shawn ba1f5eecd3 fix: update the swagger for new resource metrics 2022-08-31 11:14:36 +08:00
Shawn 73e19d84ee feat: use the new metrics to bridge APIs 2022-08-30 23:47:58 +08:00
Shawn 65dfa63324 fix: update the counters for data bridges 2022-08-30 12:28:01 +08:00
JianBo He a6eff81163 chore: update emqtt to 1.7.0-rc.1 2022-08-30 09:50:36 +08:00
JianBo He 0aa10702db feat(bridge): support async mode resource options 2022-08-30 09:34:56 +08:00
JimMoen ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
JimMoen 43c964c87e fix(docs): ee bridge api docs generation
Use try catch. Because function was unexported before called.
2022-08-24 09:52:38 +08:00
Shawn 45352206a3 fix(auth): remove emqx_connector from testcases of authz/authn 2022-08-22 20:20:45 +08:00
Shawn 55c18c0b5f fix(bridges): update the test cases for new config structure 2022-08-22 18:24:59 +08:00
Shawn aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
Shawn 9e35032d78 fix: make resume_interval defaults to health_check_interval 2022-08-16 10:09:02 +08:00
Xinyu Liu 2898966439
Merge branch 'dev/ee5.0' into resource_opts 2022-08-15 21:43:22 +08:00
Shawn 19d85d485b refactor(resource): add resource_opts level into config structure 2022-08-15 21:40:10 +08:00
Shawn b3162fe5ff fix: conflicts between master and ee5.0 2022-08-14 22:24:09 +08:00
Shawn 3cb79e811f fix: remove the extra fields max_retries 2022-08-12 18:05:03 +08:00
Shawn 0cdf4b47f1 feat: add more resource creation opts 2022-08-12 13:47:45 +08:00
Xinyu Liu a05cc20db7
Merge pull request #8693 from JimMoen/feat-influxdb-batch-query
Feat influxdb batch query
2022-08-12 00:22:23 +08:00
JimMoen 83746daad5 fix: update bridge config badmap error 2022-08-11 23:15:53 +08:00
JimMoen 458dab53c5 fix: undefined_functions dialyzer warning 2022-08-11 22:55:22 +08:00
JimMoen ed796acb95 fix: fetch resource options after reboot 2022-08-11 22:11:28 +08:00
Shawn 88388b0c54 feat: http connector support async sending 2022-08-11 21:44:22 +08:00
JimMoen 3a76a50382 fix: syntax error and compile error 2022-08-11 20:58:43 +08:00
Shawn 2872f0b668 fix(bridges): support create resources with options 2022-08-11 19:11:44 +08:00
Shawn 6203a01320 feat: add inflight window to emqx_resource 2022-08-11 08:36:35 +08:00
Shawn 0377d3cf61 fix: update existing testcases for new emqx_resource 2022-08-10 00:34:35 +08:00
JimMoen 2e336fbc01 fix: bridge and connector namespace contains DB name
Contains Detailed DB name for identifier hocon schema refs.
2022-08-07 19:37:44 +08:00
DDDHuang 55b9684541 fix: influxdb bridge api available 2022-08-01 10:28:07 +08:00
DDDHuang 3fe76446bc Merge branch 'master' into dev/ee5.0 2022-08-01 10:14:21 +08:00
DDDHuang a4992ef1b5 fix: hstreamdb connector conf & api 2022-07-28 16:47:30 +08:00
Shawn 23eeb21b13 fix(webhook): the 'max_retries' opt means request retry times 2022-07-27 17:06:14 +08:00
DDDHuang 9ae7c62656 fix: exs deps & bad suites 2022-07-27 16:24:13 +08:00
DDDHuang be6de4aad0 fix: ee bridge & connector remove app mod 2022-07-27 11:38:45 +08:00
DDDHuang fa4bc921ac fix: hstream db connector & bridge, TODO: SUITE 2022-07-27 11:38:45 +08:00
DDDHuang 98b36c4681 fix: hstream db connector , TODO: start apps 2022-07-27 11:38:45 +08:00
DDDHuang f00a7417bb feat: hstream bridge & connector 2022-07-27 11:38:45 +08:00
Shawn 6685a3c5a8 fix: remove the 'headers' field from the rule events 2022-06-17 15:57:10 +08:00
Shawn d6ef2f7502 refactor: graceful recreate resources 2022-06-17 05:29:18 +08:00
Xinyu Liu 3b00b16abe
Merge pull request #8221 from terry-xiaoyu/fix_delayed_module_disbled_after_emqx_stop
feat: fix the hook priorities
2022-06-16 09:42:42 +08:00
Zaiming (Stone) Shi 738a551550 refactor: best-effort json for hocon validation errors 2022-06-15 15:01:15 +02:00
Shawn 39b1b20506 feat: fix the hook priorities 2022-06-15 19:03:40 +08:00
Shawn dd099137f0 fix: catch errors when load bridges 2022-06-13 22:31:58 +08:00
Shawn 9a0fdc4b11 feat: support configure connector configs directly in bridges 2022-06-10 23:06:20 +08:00
Shawn 69fba6958b refactor: remove the emqx_bridge_monitor module 2022-06-01 15:31:48 +08:00
Shawn 88ca25c60c fix(resource): fast return when starting a unavailable resource 2022-06-01 08:24:53 +08:00
JianBo He efc0ca2b62
Merge branch 'master' into changes-enable_pipelining-type 2022-05-31 20:49:10 +08:00
JianBo He f0ddac63cb chore: change the enable_pipelining default value 2022-05-31 19:32:53 +08:00
Zhongwen Deng d61b44ef97 fix: ct failed again 2022-05-31 19:20:27 +08:00
Zhongwen Deng 8aa60cc0a5 feat: generate a minimized emqx.conf 2022-05-31 19:20:27 +08:00
Shawn 1054c364ad refactor(resource): improve health check and alarm it if resource down 2022-05-31 01:40:40 +08:00
Shawn a157539710 refactor(rule): rename http_bridge -> webhook 2022-05-25 18:56:12 +08:00
Shawn 71de9616d3 refactor(rule): rename outputs -> actions 2022-05-25 17:02:37 +08:00
Shawn d89d692d3e refactor: move some APIs to new module emqx_bridge_resource 2022-05-18 15:12:34 +08:00
DDDHuang 8b0adf69d2
Merge pull request #7926 from DDDHuang/rule_api_errormsg
fix: better error message for rule engine
2022-05-12 19:07:47 +08:00
DDDHuang 2de69c97ba fix: ruleengine & connector & bridge api, better error message 2022-05-12 16:51:36 +08:00
Thales Macedo Garitezi 47ed9cbc76
fix(telemetry): count bridges referenced in `from` rule clauses 2022-05-11 15:15:17 -03:00
Chris Hicks 841acb7828
Merge pull request #7783 from emqx/EMQX-4199-introduce-a-new-emqx-resource-manager-module
feat: isolate resource manager processes
2022-05-09 18:33:01 +02:00
Chris 0b3e30e813 feat: isolate resource manager processes 2022-05-09 13:24:34 +02:00
Shawn 9a033af2e8 fix: restart bridges via HTTP APIs to wrong node 2022-05-05 18:47:19 +08:00
Shawn 1ce8a8c886 fix: testing connector removes ssl cert files that are in use 2022-05-05 14:55:40 +08:00
zhongwencool c7241f2abc
Merge pull request #7813 from DDDHuang/bridge_api_bad_format
fix: bridge api response bad node name format
2022-04-29 09:12:22 +08:00
Zhongwen Deng 03d2a1ebb6 chore: reformat emqx_bridge_api 2022-04-29 09:08:45 +08:00
Shawn 6edb453ce9 fix: the field 'enable' is missing for GET /bridges/:id 2022-04-28 18:13:19 +08:00
Shawn e18c2a6de2 fix: convert ssl certfiles for dry-run creating bridges 2022-04-28 16:17:55 +08:00
DDDHuang a26b0f79f2 fix: bridge api response bad node name format 2022-04-28 16:06:57 +08:00
Zaiming (Stone) Shi 02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
Shawn 94e24c2621 refactor: move ssl file handling from resources to bridges 2022-04-27 11:59:15 +08:00
Thales Macedo Garitezi f8f97d39d3
Merge pull request #7766 from thalesmg/bugfix-telemetry-bridge
fix: avoid crashing telemetry if bridge app is not ready
2022-04-26 09:03:29 -03:00
Thales Macedo Garitezi ada3b03f6e
fix(telemetry): do not crash if bridge app is not ready yet 2022-04-25 11:23:13 -03:00
EMQ-YangM eb3870b538 fix: when enable is false, restart and stop cannot be performed. 2022-04-25 18:48:12 +08:00
JimMoen e9843aa225 fix(doc): html line break use `</br>` 2022-04-24 18:56:21 +08:00
Zhongwen Deng 5223c3ee61 feat: obfuscate sensitive values default_password 2022-04-24 09:41:29 +08:00
Yang Miao 4a43a10296
Merge pull request #7701 from EMQ-YangM/improve_docs
fix: improve document quality
2022-04-22 16:52:09 +08:00
EMQ-YangM f00854aa9d fix(emqx_bridge): add required fields 2022-04-22 14:22:22 +08:00
EMQ-YangM 739fe9b71e feat: emqx_bridge i18n support 2022-04-22 10:53:06 +08:00
EMQ-YangM 5d9df74be7 test: fix required fields 2022-04-21 18:47:02 +08:00
Shawn 19630e9a99 feat: save ssl cert files for data bridges 2022-04-21 09:00:06 +08:00
Shawn 5f74db6d88 fix: treat all returns except {error, Reason} as ok 2022-04-18 22:24:39 +08:00
EMQ-YangM a7ee4f9fea fix: fix bridge resource Id 2022-04-12 14:10:35 +08:00
EMQ-YangM 5aeb3c2dc2 feat: impl bridge reset_metrics api 2022-04-11 10:25:48 +08:00
Thales Macedo Garitezi 2a7b777a2a
fix(test): fix inter-suite test coupling 2022-04-07 12:21:15 -03:00
Thales Macedo Garitezi 911e02f626
feat(bridge): export basic usage info for telemetry 2022-04-06 10:20:23 -03:00
ieQu1 99b3f2f83e docs(schema): Document missing records and fields 2022-04-04 23:58:43 +02:00
Ilya Averyanov 0728b1e3f4 chore(emqx_modules): add emqx_topic_metrics_api tests 2022-03-30 21:18:27 +03:00
Zaiming (Stone) Shi 5458490bc3 chore: add namespace to bridge schema 2022-03-23 21:12:55 +01:00
DDDHuang 74e6fa5a94 fix: generate bridge api response code
fix: generate connector api response code SUITE

fix: generate exhook api response code
2022-03-21 14:12:19 +08:00
Shawn 1d023b541f refactor(connnector): rename waiting_connect_complete -> wait_for_resource_ready
Rename the option to wait_for_resource_ready and defaults to 5s.
2022-03-10 10:46:57 +08:00
Shawn 50d0a3271b fix(test): add testcases for enable/disbale bridges 2022-03-09 18:49:30 +08:00