yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,458 Commits 393 Branches 669 Tags 74 MiB
4f341bef8b
Commit Graph

660 Commits

Author SHA1 Message Date
Kjell Winblad
9feba802e9 chore: add convenience function for creating action schemas 2023-11-17 16:10:08 -03:00
Thales Macedo Garitezi
b92821188b fix(kafka_producer): make status connecting while the client fails to connect 
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
 2023-11-16 14:50:23 -03:00
Ivan Dyachkov
65ba381fd8 Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53 2023-11-15 09:25:03 +01:00
Thales Macedo Garitezi
5275a01727 fix(syskeeper): refactor action to new action info scheme 2023-11-14 19:49:28 +01:00
Ivan Dyachkov
7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Thales Macedo Garitezi
2f1d88d414 fix(bridges_v1): avoid merging action examples for non-v1 bridges 
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
 2023-11-14 13:56:50 -03:00
Thales Macedo Garitezi
d6e9bbb95c fix(connector): validate connector name before converting ssl certs 
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
 2023-11-14 09:29:59 -03:00
Ivan Dyachkov
1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Andrew Mayorov
7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Ivan Dyachkov
a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad
d682e6e23c test: fix test mock by calling exported function 2023-11-14 09:20:46 +01:00
Kjell Winblad
3bea3496af test: fix test case mock that broke due to removal 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca3e5eab11 refactor: remove emqx_bridge_v2_enterprise.erl 2023-11-14 09:20:46 +01:00
Kjell Winblad
e93b71d8d5 fix: problems found by @thalesmg in code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-11-14 09:20:46 +01:00
Kjell Winblad
49fdfef8c3 fix: azure event hub names are defined in one place 2023-11-14 09:20:46 +01:00
Kjell Winblad
5e8e407017 refactor: action schema retrival after PR feedback 2023-11-14 09:20:46 +01:00
Kjell Winblad
bdab421885 fix(bridge_v2): name and copyright header 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca2cdbc08d fix: bapi static check error 2023-11-14 09:20:46 +01:00
Kjell Winblad
7839f0cbc8 chore(bridge_v2): make dialyzer and bpapi happy 2023-11-14 09:20:46 +01:00
Kjell Winblad
d26a1b9afb fix(bridge_v1): no hard coded downgrade and upgrade type functions 2023-11-14 09:20:46 +01:00
Kjell Winblad
a6aa81b548 fix(emqx_bridge_v2): dialyzer found bad format parameter error 2023-11-14 09:20:46 +01:00
Kjell Winblad
14e305e2a6 fix(emqx_bridge_v2): xref warning 2023-11-14 09:20:46 +01:00
Kjell Winblad
9eaee8f333 refactor(emqx_bridge_v2): make independent of Kafka 
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
 2023-11-14 09:20:46 +01:00
Kjell Winblad
cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Kjell Winblad
99031f0dae refactor(bridge_v2): prefix compatibility functions with bridge_v1 2023-11-14 09:20:46 +01:00
Kjell Winblad
729e7df0d5 docs(bridge_v2): add specs for CRUD functions 2023-11-14 09:20:46 +01:00
Stefan Strigler
0b3645057d fix(emqx_bridge): don't try to handle what we don't understand 2023-11-13 14:08:54 +01:00
Stefan Strigler
0d33284672 fix(emqx_bridge): don't crash uninstallling broken action 2023-11-13 14:08:54 +01:00
Stefan Strigler
4fe1d332b3 fix(emqx_bridge): don't crash if there's no status field 2023-11-13 14:08:54 +01:00
Zaiming (Stone) Shi
9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109 
fix(actions_api): change bridge v2 occurrences to action
 2023-11-10 07:35:35 +01:00
firest
447c933ff8 fix(syskeeper): homogenize the Syskeeper bridge and update license data 2023-11-10 10:42:53 +08:00
firest
548e39e799 refactor(sysk): refactor the syskeeper to v2 style 2023-11-10 09:52:38 +08:00
firest
ce83079c6b feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend 2023-11-10 09:51:43 +08:00
Thales Macedo Garitezi
371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
Thales Macedo Garitezi
7977ae95c0 fix(actions_api): change bridge v2 occurrences to action 
Fixes https://emqx.atlassian.net/browse/EMQX-11338
 2023-11-09 14:20:08 -03:00
Thales Macedo Garitezi
eabd09051a feat(actions_api): add /action_types API 2023-11-09 13:54:57 -03:00
Thales Macedo Garitezi
f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
 2023-11-08 09:53:42 -03:00
Ivan Dyachkov
0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
Thales Macedo Garitezi
b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106 
fix(resource): take error from action/connector before attempting query
 2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi
7dcdbc9e51 fix(resource): take error from action/connector before attempting query 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
 2023-11-07 10:04:04 -03:00
Stefan Strigler
fa740a1927 docs: fix inconsistencies for bridge and connector operations 
It's only `start` allowed for now. Also fixed some wordings.
 2023-11-07 13:58:23 +01:00
Zaiming (Stone) Shi
f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config 
Rename connector channel related configs in bridge_v2 to 'parameters'
 2023-11-07 13:41:55 +01:00
Zaiming (Stone) Shi
9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi
2b8cf50a1d chore: rename bridges_v2 -> actions in the public facing APIs 
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
 2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi
600747b7e5 fix(bridge): do not allow dot in bridge name 
also validate name at the API entry
 2023-11-03 20:44:57 +01:00
Stefan Strigler
8a1b370a5e
Merge pull request #11876 from sstrigler/EMQX-11317-creating-a-bridge-for-a-non-existent-or-non-matching-connector-results-in-a-crash 
handle errors from pre/post_config_update
 2023-11-03 16:26:21 +01:00
Stefan Strigler
dc11008993 fix(emqx_bridge): handle errors from pre/post_config_update 2023-11-03 14:11:25 +01:00
Kjell Winblad
65ba0b9de4
Merge pull request #11869 from kjellwinblad/kjell/shared_con/start_feedback/EMQX-11304 
fix(bridge_v2): start operation should return an error when unsuccessful
 2023-11-03 13:16:32 +01:00
Zaiming (Stone) Shi
afe6f79a66 feat(bridge): add description field to bridge and connector 2023-11-03 06:53:42 +01:00
Kjell Winblad
d14d0fbcb1 fix(bridge_v2 operations): better error message 2023-11-03 06:35:42 +01:00
Kjell Winblad
36640263ba fix(bridge_v2): start operation should return an error when unsuccessful 
The bridge V2 HTTP API start operation should return a 400 error if the
start is unsuccessful.

The bridge V1 HTTP API compatibility layer for Bridge V2 should return a
400 error if the start or restart operation is unsuccessful.

This commit fixes the above and adds tests that checks this for the V2
HTTP API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11304
 2023-11-02 18:28:22 +01:00
Zaiming (Stone) Shi
f5818a86e3
Merge pull request #11860 from thalesmg/fix-delete-shared-bridgev2-r53-20231101 
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
 2023-11-02 10:30:47 +01:00
Zaiming (Stone) Shi
34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293 
fix(bridge_v2 API): optional cascading delete operation
 2023-11-02 10:28:43 +01:00
Thales Macedo Garitezi
4bea65bf97 fix(bridge_v2_api): don't disable schema check 2023-11-01 12:44:40 -03:00
Kjell Winblad
95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting 
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
 2023-11-01 15:27:53 +01:00
Kjell Winblad
1e935e9eb4 fix(bridge_v2 API): optional cascading delete operation 
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
 2023-11-01 15:03:18 +01:00
Thales Macedo Garitezi
c03979a48a fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1 2023-11-01 09:51:26 -03:00
Kjell Winblad
10b2894b6e chore(bridge_v2_api): remove misplaced handling code 2023-11-01 11:31:17 +01:00
Zaiming (Stone) Shi
ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched 
perf: no need to format event message if no bridge matched
 2023-11-01 10:22:06 +01:00
Thales Macedo Garitezi
45a39d97c6 fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1 
Fixes https://emqx.atlassian.net/browse/EMQX-11291
 2023-10-31 16:39:47 -03:00
Thales Macedo Garitezi
b420b53075 fix(bridges_v2): check bridge name length before attempting atom conversion 
Fixes https://emqx.atlassian.net/browse/EMQX-11289
 2023-10-31 13:21:26 -03:00
Thales Macedo Garitezi
8eb822d898 test(bridges): add bridge v1 compatibility layer test suite 2023-10-31 13:21:01 -03:00
Zaiming (Stone) Shi
db5cdb6742 perf: no need to format event message if no bridge matched 2023-10-31 16:48:10 +01:00
Thales Macedo Garitezi
84e78f5d2e fix(bridge_api): fix response status code for some operations 2023-10-31 10:40:56 -03:00
Thales Macedo Garitezi
c90ca1ea53 fix(bridge_v1): always delete connector when deleting v1 bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-11287
 2023-10-31 10:40:06 -03:00
Ivan Dyachkov
b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Stefan Strigler
63606922f2 refactor(emqx_bridge): only one place to generate same error 2023-10-31 08:34:05 +01:00
Stefan Strigler
176bbe88bc fix: use _producer for AEH bridge type 2023-10-31 08:34:03 +01:00
Kjell Winblad
aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import 
EMQX-11272 fix kafka bridge v1 import
 2023-10-31 07:58:54 +01:00
Kjell Winblad
9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271 
fix: crash when listing non bridge_v1 compatible bridge_v2
 2023-10-30 20:29:36 +01:00
Zaiming (Stone) Shi
350e8433ab fix(bridge_v2): downgrade bridge type for GET api results 2023-10-30 18:17:35 +01:00
Serge Tupchii
bc8c291820 fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2 2023-10-30 19:14:58 +02:00
Kjell Winblad
8fb426e57f fix: crash when listing non bridge_v1 compatible bridge_v2 
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
 2023-10-30 16:32:04 +01:00
Kjell Winblad
9dc3a169b3 feat: split bridges into a connector part and a bridge part 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
 2023-10-30 14:48:47 +01:00
Ilya Averyanov
4f14e8df22 chore(source dir): split out postgresql connector 2023-10-13 16:58:57 +03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool
123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource 
fix: always return ok when remove local resource
 2023-09-21 09:21:45 +08:00
zhongwencool
dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Paulo Zulato
7b607c3300 refactor: remove BridgeName from connector_config callback 
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.

Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
 2023-08-31 11:01:49 -03:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-30 14:48:10 -03:00
Thales Macedo Garitezi
0f297ffef4 fix(bridge): validate bridge name before attempting to convert certificates 
Fixes https://emqx.atlassian.net/browse/EMQX-10865
 2023-08-29 16:32:01 -03:00
JimMoen
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
Thales Macedo Garitezi
29e706c83d refactor: move catch to dry run fn 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
4b8cea4498 fix: make error handling more flexible 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
963d63b899 refactor: rename temporary name 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms 
Fixes https://emqx.atlassian.net/browse/EMQX-10653
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
a3be1529c6 fix(schema): ensure enterprise module is correctly loaded 
Fixes https://emqx.atlassian.net/browse/EMQX-10654

Despite loading the application in `nodetool`, we need to invoke `:module_info()` to force
the module to be actually loaded, especially when it's called in `call_hocon` to generate
the initial configurations.  Otherwise, it'll fail to list all the bridge schemas.
 2023-07-28 16:27:43 -03:00
Ivan Dyachkov
5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master 2023-07-27 16:58:09 +02:00
Ivan Dyachkov
cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10338
 2023-07-27 09:22:39 -03:00
JianBo He
511d1b6ca1 chore: hide the hstreamdb http api 2023-07-24 20:11:44 +08:00
firest
8439ce0e84 chore: update app version && reboot_lists 2023-07-24 19:08:41 +08:00
Dennis Zhuang
c9550cc2e5 refactor: rename bridge greptimedb_grpc_v1 to greptimedb 2023-07-24 17:58:48 +08:00
Dennis Zhuang
6f7fbcf693 fix: compile error 2023-07-24 17:58:48 +08:00
Dennis Zhuang
975795a6e0 feat: add ci test 2023-07-24 17:58:48 +08:00
Dennis Zhuang
89bce99870 test: greptimedb data brige 2023-07-24 17:58:48 +08:00
Dennis Zhuang
c5078980f3 feat: adds the greptimedb bridge to emqx modules 2023-07-24 17:58:48 +08:00
JimMoen
613dc1646c
chore: hidden hstreamdb bridge schema 2023-07-24 17:22:29 +08:00
Thales Macedo Garitezi
41b780f7f2
Merge pull request #11280 from thalesmg/unhide-gcp-consumer-master-20230717 
chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
 2023-07-19 10:26:42 -03:00
Ivan Dyachkov
b7f0f7b32d chore: merge master into release-51 2023-07-19 09:28:05 +02:00
Paulo Zulato
484519dcf9 feat(kinesis): implement Amazon Kinesis Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10474
Fixes https://emqx.atlassian.net/browse/EMQX-10475
 2023-07-17 11:51:42 -03:00
Thales Macedo Garitezi
05c3e023a9 chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0 
Fixes https://emqx.atlassian.net/browse/EMQX-10506
 2023-07-17 11:24:21 -03:00
Thales Macedo Garitezi
01b143c5ad fix(resource): don't destruct error tuple 
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
 2023-07-13 16:12:33 -03:00
Thales Macedo Garitezi
924ed06760
Merge pull request #11256 from thalesmg/bridge-api-ecpool-400-errors-20230712-r51 
fix(bridge_api): return 400 for all ecpool start errors
 2023-07-13 10:05:07 -03:00
Thales Macedo Garitezi
bbff1d6ae9 fix(bridge_api): return 400 for all ecpool start errors 
Fixes https://emqx.atlassian.net/browse/EMQX-10460
(https://emqx.atlassian.net/browse/EMQX-10460?focusedCommentId=30126)
 2023-07-12 15:32:34 -03:00
Kjell Winblad
6ce47614d2 refactor: HTTP bridge into the emqx_bridge_http app 
Fixes:
https://emqx.atlassian.net/browse/EMQX-9568
 2023-07-12 14:47:02 +02:00
Kjell Winblad
f28510b3ad refactor: HTTP connector into emqx_bridge_http app 2023-07-12 14:46:43 +02:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
firest
a1d2b2ca51 chore: bump app versions 2023-07-07 11:17:54 +08:00
Thales Macedo Garitezi
c216dfd96b fix(mysql_bridge): make nxdomain a 400 API error 
Fixes https://emqx.atlassian.net/browse/EMQX-10460
 2023-06-30 16:10:09 -03:00
Stefan Strigler
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics 
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
 2023-06-30 20:07:23 +02:00
Stefan Strigler
d71d70c889 fix: allow to pass in Opts rather than just a ReplyTo 2023-06-29 16:09:45 +02:00
Stefan Strigler
40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler
ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi
c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi
f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup 
feat: implement configuration and user data export/import
 2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
Kjell Winblad
273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609 
fix: friendly error message when creating bridges with too long names
 2023-06-08 15:46:42 +02:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process 
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys 
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests 
chore(bridge): drop deprecated `mode` setting from examples / tests
 2023-06-02 22:46:31 +03:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50 
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
 2023-06-02 16:20:36 -03:00
Andrew Mayorov
09290f8d61
chore(bridge): drop deprecated mode setting from examples / tests 2023-06-02 20:31:31 +03:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl 
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
 2023-06-01 13:01:53 -03:00
某文
c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval 
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
 2023-06-01 11:20:06 -03:00
Kjell Winblad
75ff76a16b fix: friendly error message when creating bridges with too long names 
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
 2023-06-01 15:30:06 +02:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id 
fix(ruleeng): parse bridge id on create / update
 2023-06-01 10:21:26 +03:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update 
Instead of parsing it every time in the `send_message/2` hot path.
 2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test 
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
 2023-05-31 10:19:55 -03:00
Andrew Mayorov
7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Zaiming (Stone) Shi
91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master 
0530 merge release 50 to master
 2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi
9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
Thales Macedo Garitezi
67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50 
feat(kafka): ensure allocated resources are removed on failures
 2023-05-29 16:49:29 -03:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi
cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi
772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50 
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
 2023-05-26 08:39:00 +02:00
Thales Macedo Garitezi
e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Paulo Zulato
ea86f4442b fix: avoid error 500 when node is re-joining cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-9899
 2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi
194faefc8a fix(webhook): keep resource_opts.request_timeout for webhook bridge (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
 2023-05-25 13:19:27 -03:00
JimMoen
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad
4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0 
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
 2023-05-24 13:09:47 +02:00
Thales Macedo Garitezi
44ed6a19ee fix(bridge): pass resource option request_timeout = infinity along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00
Zaiming (Stone) Shi
732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
William Yang
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path 
perf(config): enforcing atom key  path in hotcode path
 2023-05-22 09:38:09 +02:00
Thales Macedo Garitezi
09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-9872
 2023-05-18 14:25:13 -03:00
Thales Macedo Garitezi
5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi
447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
William Yang
cdf42760fa chore: little change avoid atom leak 2023-05-17 11:01:03 +02:00
某文
2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
Zaiming (Stone) Shi
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field 
0421 refactor delete unused schema field
 2023-05-16 22:22:03 +02:00
William Yang
1c746ed289 perf(config): enforcing atom key path in hotcode path 2023-05-16 09:57:04 +02:00
某文
7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zaiming (Stone) Shi
7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Stefan Strigler
d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zaiming (Stone) Shi
d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi
57cc854a4a test(bridge): fix bridge map type filed converters 
now the converters on map type fields only work at the wrapping map
level but not the values
 2023-04-30 10:45:11 +02:00
Paulo Zulato
dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi
ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-8398
 2023-04-24 10:28:26 -03:00
Serge Tupchii
45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid 
Fixes: EMQX-9310
 2023-04-20 17:21:43 +03:00
Zaiming (Stone) Shi
30fd9b10f7 refactor: change bridges config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Ivan Dyachkov
dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
某文
fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
Ivan Dyachkov
9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Stefan Strigler
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules 
New emqx_utils app to collect utility modules
 2023-04-14 20:33:11 +02:00
Thales Macedo Garitezi
e073bc90bc refactor(buffer_worker): rename s/queue/buffer/g 2023-04-14 11:37:19 -03:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen
790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge 
feat: implement Microsoft SQL Server bridge (e5.0)
 2023-04-14 17:55:48 +08:00
Andrew Mayorov
5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics 
feat(resource): ask for metrics only when needed
 2023-04-14 12:28:52 +03:00
JimMoen
5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
Zhongwen Deng
cc2beda37a feat: hiden rule_engine/bridge/authz/authn from doc/example 2023-04-13 14:42:21 +08:00
Ivan Dyachkov
bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov
e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
Stefan Strigler
04626ce9cc fix: create consistent interface 'with_node' for API access 2023-04-04 16:54:14 +02:00
firest
11b3264251 fix: redact the password to ****** in API examples 
fix #10222
 2023-04-04 16:02:45 +08:00
Zaiming (Stone) Shi
80eb9d7542
Merge pull request #10252 from emqx/release-50 
0327 merge release-50 to master
 2023-03-29 12:33:17 +02:00
Kjell Winblad
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell 
fix: add inflight window setting to the clickhouse bridge
 2023-03-29 09:38:24 +02:00
Thales Macedo Garitezi
64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi
1824e7efcc fix(rule_engine): count referenced bridges in from clauses as dependencies (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9325

Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.

When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
 2023-03-28 10:31:10 -03:00
Stefan Strigler
193905c7b9
Merge pull request #10237 from sstrigler/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent 
fix: create consistent interface 'with_node' for API access
 2023-03-28 10:41:10 +02:00
Stefan Strigler
64a1d84a44 fix: create consistent interface 'with_node' for API access 2023-03-27 12:14:48 +02:00
JianBo He
bfa5922209
Merge pull request #10140 from HJianBo/cassa 
feat: support cassandra data bridge
 2023-03-27 10:23:02 +08:00
Zaiming (Stone) Shi
dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
Stefan Strigler
8d084ed13b
Merge pull request #10226 from sstrigler/EMQX-9273-e-5-0-2-creating-a-data-bridge-the-name-verification-returned-an-incorrect-error-code-of-500 
fix(emqx_bridge): don't crash on validation error
 2023-03-24 14:48:03 +01:00
Stefan Strigler
fbfdaf44e0 fix(emqx_bridge): don't crash on validation error 2023-03-24 11:46:41 +01:00
Stefan Strigler
a8cfc1c7a1 fix(emqx_bridge): filter status_reason in config update 2023-03-24 11:46:41 +01:00
Stefan Strigler
ba49316ec5 fix: use lower-case 2023-03-23 16:08:27 +01:00
Kjell Winblad
35474578ca refactor: rename async_inflight_window to inflight_window everywhere 2023-03-23 14:21:57 +01:00
JianBo He
8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
lafirest
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq 
feat(bridges): integrate RocketMQ into data bridges
 2023-03-22 20:43:22 +08:00
Zaiming (Stone) Shi
e43aa5d841
Merge pull request #10199 from zmstone/0321-merge-release-50-to-master 
0321 merge release 50 to master
 2023-03-22 10:20:48 +01:00
Andrew Mayorov
af7b2215ca
fix(bridge-api): unflatten nodes' bridge lists back 
Bridge lists were erroneously flattened in cad6492c. This causes
bridge listing fail in emqx clusters consisting of more than 1 node.
 2023-03-22 11:35:09 +03:00