7da0860d6c
chore: upgrade minirest to 1.3.10 for more clear error msg
2023-05-24 10:44:52 +08:00
44ed6a19ee
fix(bridge): pass resource option `request_timeout = infinity` along to buffer workers (r5.0)
2023-05-23 14:50:43 -03:00
b5f63f4151
test: fix wrong test listeners option
2023-05-23 14:50:43 -03:00
1deb3c50c8
test: bridge test init dashboard listener failed
2023-05-23 10:28:01 +08:00
67a30ee976
test: bridge test init dashboard listener failed
2023-05-23 10:16:43 +08:00
732a7be187
Merge remote-tracking branch 'origin/release-50'
2023-05-22 17:46:54 +02:00
7d798c10e9
perf(buffer_worker): flush metrics periodically inside buffer worker process
...
Fixes https://emqx.atlassian.net/browse/EMQX-9905
Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
2023-05-22 09:11:23 -03:00
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path
...
perf(config): enforcing atom key path in hotcode path
2023-05-22 09:38:09 +02:00
09ea2e2224
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
...
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
5d5c16a56d
feat(bridges): use union member selector function for better error messages
2023-05-17 17:57:29 -03:00
447b76464b
Merge branch 'release-50' into merge-r50-into-v50-a
2023-05-17 14:50:18 -03:00
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
...
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
a7b41e1cdf
perf(webhook): add retry attempts for async
...
This is a performance improvement for webhook bridge.
Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well. Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.
This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
cdf42760fa
chore: little change avoid atom leak
2023-05-17 11:01:03 +02:00
2b99a9b988
feat: hide resource_opts's request_timeout
2023-05-17 13:41:11 +08:00
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field
...
0421 refactor delete unused schema field
2023-05-16 22:22:03 +02:00
a2aa6b4666
chore: make ci happy again
2023-05-16 21:29:52 +08:00
1c746ed289
perf(config): enforcing atom key path in hotcode path
2023-05-16 09:57:04 +02:00
7d7c069257
feat: update wehbook's request_timeout into resource_opts
2023-05-16 15:32:43 +08:00
a953b951fe
Merge branch 'master' into sync-release-50-to-master
2023-05-12 18:01:58 +08:00
6056b0e7a8
fix: ensure atom key for emqx_config:get
2023-05-11 17:40:03 +08:00
7566c25199
refactor: delete unused connector field 'base_url'
2023-05-11 09:04:55 +02:00
14952658d5
chore: convert rotation struct to roation_count
2023-05-09 11:24:51 +08:00
d231e708cb
feat: add IotDB bridge
2023-05-08 16:19:01 +02:00
856129984b
refactor: remove raw_with_default config load option
...
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.
After a03f2dd64b
2023-05-04 09:48:28 +02:00
d5f5f35787
Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master
2023-05-02 18:08:04 +02:00
57cc854a4a
test(bridge): fix bridge map type filed converters
...
now the converters on map type fields only work at the wrapping map
level but not the values
2023-04-30 10:45:11 +02:00
dd90b2f498
feat(oracle): Oracle Database integration
2023-04-27 13:07:20 -03:00
ad4be08bb2
feat: implement Pulsar Producer bridge (e5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-8398
2023-04-24 10:28:26 -03:00
45254d7d85
fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
...
Fixes: EMQX-9310
2023-04-20 17:21:43 +03:00
30fd9b10f7
refactor: change bridges config importance level from hidden to low
2023-04-18 19:17:03 +02:00
dc78ecb41c
chore: merge upstream/master
2023-04-18 17:33:32 +02:00
fbadfc06e4
feat: change exhook, rule_engine, bridge to low importance level instead of hidden
2023-04-18 20:15:19 +08:00
9fc8a498f8
chore: bump apps versions
2023-04-17 09:09:08 +02:00
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
...
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
e073bc90bc
refactor(buffer_worker): rename `s/queue/buffer/g`
2023-04-14 11:37:19 -03:00
4f80690162
fix: byebye jsx
2023-04-14 13:41:34 +02:00
062ce5f819
refactor: rename emqx_map_lib to emqx_utils_maps
2023-04-14 13:41:34 +02:00
d0df086c80
refactor: rename emqx_api_lib to emqx_utils_api
2023-04-14 13:41:33 +02:00
9c11bfce80
refactor: rename emqx_misc to emqx_utils
2023-04-14 13:41:27 +02:00
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge
...
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 17:55:48 +08:00
5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics
...
feat(resource): ask for metrics only when needed
2023-04-14 12:28:52 +03:00
5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
cc2beda37a
feat: hiden rule_engine/bridge/authz/authn from doc/example
2023-04-13 14:42:21 +08:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
e70deae1c3
feat(resource): ask for metrics only when needed
2023-04-11 12:00:19 +03:00
04626ce9cc
fix: create consistent interface 'with_node' for API access
2023-04-04 16:54:14 +02:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
64faccf50b
test: fix flaky kafka consumer test
2023-03-28 14:50:55 -03:00
1824e7efcc
fix(rule_engine): count referenced bridges in `from` clauses as dependencies (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9325
Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.
When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
2023-03-28 10:31:10 -03:00
193905c7b9
Merge pull request #10237 from sstrigler/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent
...
fix: create consistent interface 'with_node' for API access
2023-03-28 10:41:10 +02:00
d942a531e6
fix(test): avoid unnecessary cleanups
...
This might make some noise in the logs, depending on the testsuite
configuration, because some test setup might be missing required
applications.
2023-03-27 23:41:19 +03:00
11b5b7b638
test(bridge-api): also run testcases in cluster environment
...
Excluding a couple of testcases which does not make much sense running
in the cluster. Also try to reduce amount of "noise" in the testcases,
making them easier to comprehend.
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 23:41:08 +03:00
64a1d84a44
fix: create consistent interface 'with_node' for API access
2023-03-27 12:14:48 +02:00
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
8d084ed13b
Merge pull request #10226 from sstrigler/EMQX-9273-e-5-0-2-creating-a-data-bridge-the-name-verification-returned-an-incorrect-error-code-of-500
...
fix(emqx_bridge): don't crash on validation error
2023-03-24 14:48:03 +01:00
fbfdaf44e0
fix(emqx_bridge): don't crash on validation error
2023-03-24 11:46:41 +01:00
a8cfc1c7a1
fix(emqx_bridge): filter status_reason in config update
2023-03-24 11:46:41 +01:00
ba49316ec5
fix: use lower-case
2023-03-23 16:08:27 +01:00
35474578ca
refactor: rename async_inflight_window to inflight_window everywhere
2023-03-23 14:21:57 +01:00
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
...
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
e43aa5d841
Merge pull request #10199 from zmstone/0321-merge-release-50-to-master
...
0321 merge release 50 to master
2023-03-22 10:20:48 +01:00
af7b2215ca
fix(bridge-api): unflatten nodes' bridge lists back
...
Bridge lists were erroneously flattened in cad6492c
2023-03-22 11:35:09 +03:00
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
65c2da7ef5
Merge remote-tracking branch 'ce/master' into cassa
2023-03-22 09:30:50 +08:00
bf35419b52
Merge pull request #10190 from keynslug/fix/EMQX-9269/bridge-list
...
fix(bridge-api): unflatten nodes' bridge lists back
2023-03-22 00:53:03 +03:00
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
6c24b3a77d
Merge pull request #10196 from sstrigler/EMQX-9201-the-case-is-not-correct-for-some-of-menu
...
Emqx 9201 the case is not correct for some of menu
2023-03-21 19:42:10 +01:00
d9b1b1c802
fix(bridge-api): unflatten nodes' bridge lists back
...
Bridge lists were erroneously flattened in cad6492c
2023-03-21 18:59:27 +03:00
981268911a
style: make summary lower-case
2023-03-21 16:56:37 +01:00
3880862c81
fix(emqx_bridge): return 503 for inconsistency in bridge setup
2023-03-21 15:07:43 +01:00
4b0ea562a2
refactor(emqx_bridge): consistently use macros for http response
2023-03-21 15:07:41 +01:00
84fc64822e
style: fix wording for nxdomain and malformed_username_or_password
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 15:02:29 +01:00
8af3fb4ee7
feat: move human readable error translations to emqx_misc
2023-03-21 15:02:29 +01:00
53825b9aba
fix(emqx_bridge): propagate connection error to resource status
2023-03-21 15:02:29 +01:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
12942b676d
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-20 09:50:27 +08:00
c0a216a740
feat(bridge): support cassandra bridge
2023-03-17 11:34:48 +08:00
3954b7bde2
fix(bridges): function clause when a non-ingress bridge coexists with an egress bridge
...
This was not caught by our tests because we always test bridge types
in isolation. So, if the config only contains ingress-only bridges,
the `on_message_publish` hook is never installed.
In a real system, if there are bridges of mixed types in the config,
the hook might be installed, and `emqx_bridge:get_matched_bridge_id`
would crash when iterating over the ingress bridges.
2023-03-16 13:43:01 -03:00
66eb4ef069
test: fix inter-suite flakiness
2023-03-16 13:43:01 -03:00
f31f15e44e
chore(kafka_producer): make schema changes more backwards compatible
...
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
03342923b9
fix(bridge): use the same dry run prefix
...
Kafka Producer and Consumer bridges rely on this prefix for detecting
a dry run and avoid leaking atoms. At some point, this prefix was
changed, effectively disabling the check in Kafka Producer.
2023-03-16 13:43:01 -03:00
1f31a87974
fix(bridge): improve macro var usage
...
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-03-16 13:43:01 -03:00
969fa03ecc
feat: implement kafka consumer
2023-03-16 13:43:01 -03:00
91a57faa95
Merge pull request #10128 from thalesmg/ocsp-v50-mkII
...
feat: add ocsp stapling support to mqtt ssl listener (5.0)
2023-03-16 13:10:48 -03:00
816667d85c
Merge pull request #10123 from fix/EMQX-9136/lookup-busy-resources
...
perf(bridge-api): ask nodes' bridge listings in parallel
2023-03-16 00:07:12 +03:00
03b95073fc
test: fix inter-suite flakiness
2023-03-15 14:25:41 -03:00
b3e7e51094
test(bridge): drop unnecessary cleanup routines
...
Since `end_per_testcase` cleans out all the resources anyway.
2023-03-15 19:17:29 +03:00
cad6492c99
perf(bridge-api): ask bridge listings in parallel
...
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
53bc27e0f4
refactor(bridge): avoid unnecessary `maps:to_list/1` when listing
2023-03-13 14:49:38 +03:00
c1adf0de1f
test(emqx_bridge): increase coverage by adding common edge cases
2023-03-13 10:40:47 +01:00
478601ee41
fix(emqx_bridge): remove unreachable code paths
2023-03-13 10:40:47 +01:00
dba95ec0fa
style(emqx_bridge): fix wording
2023-03-13 10:40:47 +01:00
71ec77a2f2
fix(emqx_bridge): for node operation check if node is part of cluster
...
This fixes the case where we returned `501 NOT IMPLEMENTED` in the past.
2023-03-13 10:40:47 +01:00
7124600a71
docs(emqx_bridge): add `start` as operation to bridge
2023-03-13 10:38:51 +01:00
1bcc5623ed
fix(emqx_bridge): check if bridge enabled before calling op
2023-03-13 10:38:49 +01:00
a325133391
fix(emqx_bridge): don't crash checking if bridge enabled
2023-03-13 10:37:28 +01:00
80b81748df
fix(emqx_bridge): handle bridge not found in call_operation
2023-03-13 10:37:28 +01:00
fb3d101b3a
refactor(emqx_bridge): fix var names
2023-03-13 10:37:28 +01:00
e31f4d6091
refactor(emqx_bridge): add BAD_REQUEST macro and minor cleanups
2023-03-13 10:37:28 +01:00
e3595f2e79
chore(mria): Bump version to 0.4.0
2023-03-11 00:37:25 +01:00
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
b8b1a78025
feat(bridges): integrate the DynamoDB into data bridges
2023-03-08 11:13:38 +08:00
26b29185b2
test(emqx_bridge_webhook_SUITE): fix flakyness in test web server
2023-03-07 20:57:38 +01:00
163b33ab28
test: remove unnecessary dependencies of ee apps
2023-03-07 20:57:38 +01:00
ca947e3e70
fix: lost messages when HTTP connection times out
...
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.
Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
e17ad320ee
fix(bridge): do not log in converter
2023-03-06 15:31:28 -03:00
2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test
...
test: fix flaky mqtt bridge test
2023-03-06 13:06:37 -03:00
a81d1a4618
Merge pull request #10066 from sstrigler/EMQX-8932-bridge-api-reports-500-for-most-errors-when-starting-a-resource
...
fix: return human readable error message for most common cases
2023-03-06 14:13:34 +01:00
f95a30ae89
fix(webhook): convert `request_timeout`s in root and resource_opts
2023-03-06 10:12:38 -03:00
8fbb948b6f
test: fix flaky mqtt bridge test
...
Sometimes, this test fails because the metrics are still in the
inflight phase.
2023-03-06 09:09:33 -03:00
a638cc6566
Merge pull request #9985 from kjellwinblad/kjell/feat/clickhouse_bridge_2/EMQX-8391
...
feat: add clickhouse database bridge
2023-03-04 07:05:05 +01:00
570bf165af
fix: return human readable error message for most common cases
2023-03-03 16:49:09 +01:00
c01f62a1c1
chore: bump apps versions
2023-03-03 13:58:32 +01:00
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
39e1cf9502
fix(emqx_bridge): let it crash instead of 400 on failed reset
2023-03-02 11:23:07 +01:00
4c23ab097d
fix(emqx_bridge): return 400 if operation not possible
2023-03-02 11:23:07 +01:00
680a4362d1
fix: match macro variables
2023-03-02 10:12:56 +01:00
76f97010bb
style: change wording
2023-03-02 09:14:52 +01:00
77dea0c77a
fix: Handle obfuscated fields in bridges_probe API
2023-03-01 22:05:23 +02:00
3f324f85bc
fix: consistently return 404 in case bridge is not found or invalid
...
Also: fix some typos and be more verbose
2023-03-01 18:03:49 +01:00
56279f6305
docs: fix bridge status report
2023-02-28 11:56:46 +01:00
7c573d9365
fix(bridgeapi): anticipate node responses w/o metrics
...
When some resource manager is busy with trying to estabilish a
connection with remote, we hit the "read-from-cache" codepath so the
resource data will not contain any metrics.
2023-02-27 14:55:41 +03:00
72fe10f18b
chore: fix spell check errors
2023-02-10 19:41:14 +01:00
2ed54e560f
Merge pull request #9952 from keynslug/fix/mqtt-bridge-no-qos2-clean-start
...
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 20:40:37 +04:00
cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses
2023-02-10 19:15:48 +03:00
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 17:17:59 +03:00
e196d6f42a
fix(emqx_bridge): unwrap rpc multicall result
2023-02-10 12:00:20 +01:00
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
...
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
4e71f1cbc5
Merge pull request #9940 from zmstone/0208-ensure-no-leading-br-in-docs
...
Add style checks for i18n.conf files
2023-02-09 17:37:44 +01:00
5328955ff4
fix(config): add back reconnect_interval field as deprecated
2023-02-09 17:30:09 +01:00
48efe552d1
test(mqtt-bridge): manage test flaps by waiting for flush events
2023-02-09 14:31:42 +03:00
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
f89ac54b17
style: remove stale comment
2023-02-08 14:04:09 +01:00
a3fd0897bc
refactor: less code duplication
2023-02-08 14:04:09 +01:00
502f62e18d
fix: try old 'restart' behavior if 'start' is not implemented
2023-02-08 13:19:41 +01:00
86f3f5787f
feat: allow to manually re-connect disconected bridge
2023-02-07 11:58:30 +01:00
4e12a44ee6
fix: return 204 instead of 200 if there's no body content
2023-02-06 09:47:07 +01:00
a71d983ff8
test: fix assertMatch patter to make rebar3 fmt work
2023-02-02 16:37:45 +01:00
85fb53dcf0
refactor: use compile switch to diff functions between ce and ee
2023-02-02 15:15:31 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
8a46cb974e
test(mqtt-bridge): test async bridge reconnects seamlessly
2023-02-01 16:28:16 +03:00
某文