Commit Graph

867 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
f40f6bc5dd refactor: split resource_opts fields between connector and actions 2023-11-20 12:37:36 -03:00
Thales Macedo Garitezi
53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Thales Macedo Garitezi
b42d4e5ab0
Merge pull request #11950 from thalesmg/fix-bridge-v1-examples-r53-20231114
fix(bridges_v1): avoid merging action examples for non-v1 bridges
2023-11-20 09:12:23 -03:00
Stefan Strigler
84ff7b0b38 feat(emqx_bridge): action_info with dynamic lookup
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
Thales Macedo Garitezi
eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Kjell Winblad
86c126ffcd feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
2023-11-17 16:10:08 -03:00
Kjell Winblad
9feba802e9 chore: add convenience function for creating action schemas 2023-11-17 16:10:08 -03:00
Thales Macedo Garitezi
b92821188b fix(kafka_producer): make status connecting while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Ivan Dyachkov
65ba381fd8 Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53 2023-11-15 09:25:03 +01:00
Thales Macedo Garitezi
5275a01727 fix(syskeeper): refactor action to new action info scheme 2023-11-14 19:49:28 +01:00
Ivan Dyachkov
7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Thales Macedo Garitezi
2f1d88d414 fix(bridges_v1): avoid merging action examples for non-v1 bridges
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
2023-11-14 13:56:50 -03:00
Thales Macedo Garitezi
d6e9bbb95c fix(connector): validate connector name before converting ssl certs
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
2023-11-14 09:29:59 -03:00
Ivan Dyachkov
1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Andrew Mayorov
7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Ivan Dyachkov
a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad
d682e6e23c test: fix test mock by calling exported function 2023-11-14 09:20:46 +01:00
Kjell Winblad
3bea3496af test: fix test case mock that broke due to removal 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca3e5eab11 refactor: remove emqx_bridge_v2_enterprise.erl 2023-11-14 09:20:46 +01:00
Kjell Winblad
e93b71d8d5 fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 09:20:46 +01:00
Kjell Winblad
49fdfef8c3 fix: azure event hub names are defined in one place 2023-11-14 09:20:46 +01:00
Kjell Winblad
5e8e407017 refactor: action schema retrival after PR feedback 2023-11-14 09:20:46 +01:00
Kjell Winblad
ab078647a5 chore: fix formatting problem 2023-11-14 09:20:46 +01:00
Kjell Winblad
bdab421885 fix(bridge_v2): name and copyright header 2023-11-14 09:20:46 +01:00
Kjell Winblad
ca2cdbc08d fix: bapi static check error 2023-11-14 09:20:46 +01:00
Kjell Winblad
7839f0cbc8 chore(bridge_v2): make dialyzer and bpapi happy 2023-11-14 09:20:46 +01:00
Kjell Winblad
d26a1b9afb fix(bridge_v1): no hard coded downgrade and upgrade type functions 2023-11-14 09:20:46 +01:00
Kjell Winblad
a6aa81b548 fix(emqx_bridge_v2): dialyzer found bad format parameter error 2023-11-14 09:20:46 +01:00
Kjell Winblad
3c778121a5 fix: bridge V1 type lookup issue 2023-11-14 09:20:46 +01:00
Kjell Winblad
77aaff137a test(emqx_bridge_v2_SUITE): fix incorrect assumption about return type
This commit fixes an incorrect assumption about the return type of
emqx_bridge_v2:lookup/2 in emqx_bridge_v2_SUITE:t_load_config_success/1.
2023-11-14 09:20:46 +01:00
Kjell Winblad
14e305e2a6 fix(emqx_bridge_v2): xref warning 2023-11-14 09:20:46 +01:00
Kjell Winblad
9eaee8f333 refactor(emqx_bridge_v2): make independent of Kafka
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
2023-11-14 09:20:46 +01:00
Kjell Winblad
cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Kjell Winblad
99031f0dae refactor(bridge_v2): prefix compatibility functions with bridge_v1 2023-11-14 09:20:46 +01:00
Kjell Winblad
729e7df0d5 docs(bridge_v2): add specs for CRUD functions 2023-11-14 09:20:46 +01:00
Stefan Strigler
17b18849de test(emqx_bridge): test broken config 2023-11-13 14:52:06 +01:00
Stefan Strigler
0b3645057d fix(emqx_bridge): don't try to handle what we don't understand 2023-11-13 14:08:54 +01:00
Stefan Strigler
0d33284672 fix(emqx_bridge): don't crash uninstallling broken action 2023-11-13 14:08:54 +01:00
Stefan Strigler
4fe1d332b3 fix(emqx_bridge): don't crash if there's no status field 2023-11-13 14:08:54 +01:00
Zaiming (Stone) Shi
9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109
fix(actions_api): change bridge v2 occurrences to action
2023-11-10 07:35:35 +01:00
firest
447c933ff8 fix(syskeeper): homogenize the Syskeeper bridge and update license data 2023-11-10 10:42:53 +08:00
firest
548e39e799 refactor(sysk): refactor the syskeeper to v2 style 2023-11-10 09:52:38 +08:00
firest
ce83079c6b feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend 2023-11-10 09:51:43 +08:00
Thales Macedo Garitezi
371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
Thales Macedo Garitezi
7977ae95c0 fix(actions_api): change bridge v2 occurrences to action
Fixes https://emqx.atlassian.net/browse/EMQX-11338
2023-11-09 14:20:08 -03:00
Thales Macedo Garitezi
eabd09051a feat(actions_api): add /action_types API 2023-11-09 13:54:57 -03:00
Thales Macedo Garitezi
f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-08 09:53:42 -03:00
Ivan Dyachkov
0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
Thales Macedo Garitezi
b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi
7dcdbc9e51 fix(resource): take error from action/connector before attempting query
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
Stefan Strigler
fa740a1927 docs: fix inconsistencies for bridge and connector operations
It's only `start` allowed for now. Also fixed some wordings.
2023-11-07 13:58:23 +01:00
Zaiming (Stone) Shi
f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config
Rename connector channel related configs in bridge_v2 to 'parameters'
2023-11-07 13:41:55 +01:00
Thales Macedo Garitezi
9742dae30d
Merge pull request #11889 from thalesmg/rename-bridgev2-to-action-apis-r53-20231106
chore: rename `bridges_v2` -> `actions` in the public facing APIs
2023-11-07 09:29:06 -03:00
Zaiming (Stone) Shi
9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi
2b8cf50a1d chore: rename bridges_v2 -> actions in the public facing APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
2023-11-06 15:37:07 -03:00
Thales Macedo Garitezi
f17b762596 chore: don't disable rule that references non-existent bridge
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
2023-11-06 10:09:14 -03:00
Zaiming (Stone) Shi
600747b7e5 fix(bridge): do not allow dot in bridge name
also validate name at the API entry
2023-11-03 20:44:57 +01:00
Kjell Winblad
dd8085d3a4 test(emqx_bridge_v2_api_SUITE): fix broken test case 2023-11-03 17:49:25 +01:00
Stefan Strigler
8a1b370a5e
Merge pull request #11876 from sstrigler/EMQX-11317-creating-a-bridge-for-a-non-existent-or-non-matching-connector-results-in-a-crash
handle errors from pre/post_config_update
2023-11-03 16:26:21 +01:00
Thales Macedo Garitezi
953d483c24
Merge pull request #11864 from thalesmg/test-enable-rule-check-deps-r53-20231101
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 11:01:38 -03:00
Stefan Strigler
dc11008993 fix(emqx_bridge): handle errors from pre/post_config_update 2023-11-03 14:11:25 +01:00
Thales Macedo Garitezi
c84e4a4187 fix(rule_engine): don't enable a rule that references non-existent bridge 2023-11-03 09:24:53 -03:00
Kjell Winblad
65ba0b9de4
Merge pull request #11869 from kjellwinblad/kjell/shared_con/start_feedback/EMQX-11304
fix(bridge_v2): start operation should return an error when unsuccessful
2023-11-03 13:16:32 +01:00
Kjell Winblad
ebb5997a8c test(emqx_bridge_v2_api_SUITE): fix cluster test 2023-11-03 11:11:27 +01:00
Zaiming (Stone) Shi
afe6f79a66 feat(bridge): add description field to bridge and connector 2023-11-03 06:53:42 +01:00
Kjell Winblad
d14d0fbcb1 fix(bridge_v2 operations): better error message 2023-11-03 06:35:42 +01:00
Kjell Winblad
36640263ba fix(bridge_v2): start operation should return an error when unsuccessful
The bridge V2 HTTP API start operation should return a 400 error if the
start is unsuccessful.

The bridge V1 HTTP API compatibility layer for Bridge V2 should return a
400 error if the start or restart operation is unsuccessful.

This commit fixes the above and adds tests that checks this for the V2
HTTP API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11304
2023-11-02 18:28:22 +01:00
Zaiming (Stone) Shi
f5818a86e3
Merge pull request #11860 from thalesmg/fix-delete-shared-bridgev2-r53-20231101
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-02 10:30:47 +01:00
Zaiming (Stone) Shi
34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293
fix(bridge_v2 API): optional cascading delete operation
2023-11-02 10:28:43 +01:00
Thales Macedo Garitezi
4bea65bf97 fix(bridge_v2_api): don't disable schema check 2023-11-01 12:44:40 -03:00
Kjell Winblad
357b664c8d fix(bridge_v2): more fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:54 +01:00
Kjell Winblad
95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
Kjell Winblad
1e935e9eb4 fix(bridge_v2 API): optional cascading delete operation
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
2023-11-01 15:03:18 +01:00
Thales Macedo Garitezi
c03979a48a fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1 2023-11-01 09:51:26 -03:00
Kjell Winblad
10b2894b6e chore(bridge_v2_api): remove misplaced handling code 2023-11-01 11:31:17 +01:00
Zaiming (Stone) Shi
ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched
perf: no need to format event message if no bridge matched
2023-11-01 10:22:06 +01:00
Thales Macedo Garitezi
45a39d97c6 fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1
Fixes https://emqx.atlassian.net/browse/EMQX-11291
2023-10-31 16:39:47 -03:00
Thales Macedo Garitezi
b420b53075 fix(bridges_v2): check bridge name length before attempting atom conversion
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
Thales Macedo Garitezi
8eb822d898 test(bridges): add bridge v1 compatibility layer test suite 2023-10-31 13:21:01 -03:00
Zaiming (Stone) Shi
db5cdb6742 perf: no need to format event message if no bridge matched 2023-10-31 16:48:10 +01:00
Thales Macedo Garitezi
84e78f5d2e fix(bridge_api): fix response status code for some operations 2023-10-31 10:40:56 -03:00
Thales Macedo Garitezi
c90ca1ea53 fix(bridge_v1): always delete connector when deleting v1 bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11287
2023-10-31 10:40:06 -03:00
Ivan Dyachkov
b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Stefan Strigler
39f49d425c test(emqx_bridge): fix test for bridge_v2_type_to_connector_type 2023-10-31 08:34:05 +01:00
Stefan Strigler
63606922f2 refactor(emqx_bridge): only one place to generate same error 2023-10-31 08:34:05 +01:00
Stefan Strigler
b256241650 test(emqx_bridge_azure_event_hub): fix legacy v1 bridge API tests 2023-10-31 08:34:05 +01:00
Stefan Strigler
176bbe88bc fix: use _producer for AEH bridge type 2023-10-31 08:34:03 +01:00
Kjell Winblad
aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import
EMQX-11272 fix kafka bridge v1 import
2023-10-31 07:58:54 +01:00
Kjell Winblad
9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271
fix: crash when listing non bridge_v1 compatible bridge_v2
2023-10-30 20:29:36 +01:00
Zaiming (Stone) Shi
350e8433ab fix(bridge_v2): downgrade bridge type for GET api results 2023-10-30 18:17:35 +01:00
Serge Tupchii
bc8c291820 fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2 2023-10-30 19:14:58 +02:00
Kjell Winblad
8fb426e57f fix: crash when listing non bridge_v1 compatible bridge_v2
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
2023-10-30 16:32:04 +01:00
Kjell Winblad
9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
Ilya Averyanov
4f14e8df22 chore(source dir): split out postgresql connector 2023-10-13 16:58:57 +03:00
Ilya Averyanov
c2c56ba481 chore(auth): update tests 2023-10-05 13:41:50 +03:00
Ilya Averyanov
1eb75b43c4 chore(auth): split emqx_authn and emqx_authz apps 2023-10-05 13:41:50 +03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool
123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool
dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Zaiming (Stone) Shi
e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
Paulo Zulato
7b607c3300 refactor: remove BridgeName from connector_config callback
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.

Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
2023-08-31 11:01:49 -03:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-30 14:48:10 -03:00
Thales Macedo Garitezi
0f297ffef4 fix(bridge): validate bridge name before attempting to convert certificates
Fixes https://emqx.atlassian.net/browse/EMQX-10865
2023-08-29 16:32:01 -03:00
Andrew Mayorov
f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
JimMoen
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
Thales Macedo Garitezi
29e706c83d refactor: move catch to dry run fn 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
4b8cea4498 fix: make error handling more flexible 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
963d63b899 refactor: rename temporary name
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms
Fixes https://emqx.atlassian.net/browse/EMQX-10653
2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
a3be1529c6 fix(schema): ensure enterprise module is correctly loaded
Fixes https://emqx.atlassian.net/browse/EMQX-10654

Despite loading the application in `nodetool`, we need to invoke `:module_info()` to force
the module to be actually loaded, especially when it's called in `call_hocon` to generate
the initial configurations.  Otherwise, it'll fail to list all the bridge schemas.
2023-07-28 16:27:43 -03:00
Ivan Dyachkov
5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master 2023-07-27 16:58:09 +02:00
Ivan Dyachkov
cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10338
2023-07-27 09:22:39 -03:00
JianBo He
511d1b6ca1 chore: hide the hstreamdb http api 2023-07-24 20:11:44 +08:00
firest
8439ce0e84 chore: update app version && reboot_lists 2023-07-24 19:08:41 +08:00
Dennis Zhuang
c9550cc2e5 refactor: rename bridge greptimedb_grpc_v1 to greptimedb 2023-07-24 17:58:48 +08:00
Dennis Zhuang
6f7fbcf693 fix: compile error 2023-07-24 17:58:48 +08:00
Dennis Zhuang
975795a6e0 feat: add ci test 2023-07-24 17:58:48 +08:00
Dennis Zhuang
89bce99870 test: greptimedb data brige 2023-07-24 17:58:48 +08:00
Dennis Zhuang
c5078980f3 feat: adds the greptimedb bridge to emqx modules 2023-07-24 17:58:48 +08:00
JimMoen
613dc1646c
chore: hidden hstreamdb bridge schema 2023-07-24 17:22:29 +08:00
Thales Macedo Garitezi
41b780f7f2
Merge pull request #11280 from thalesmg/unhide-gcp-consumer-master-20230717
chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
2023-07-19 10:26:42 -03:00
Ivan Dyachkov
b7f0f7b32d chore: merge master into release-51 2023-07-19 09:28:05 +02:00
Paulo Zulato
484519dcf9 feat(kinesis): implement Amazon Kinesis Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10474
Fixes https://emqx.atlassian.net/browse/EMQX-10475
2023-07-17 11:51:42 -03:00
Thales Macedo Garitezi
05c3e023a9 chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
Fixes https://emqx.atlassian.net/browse/EMQX-10506
2023-07-17 11:24:21 -03:00
Thales Macedo Garitezi
ab5fd1e5c3
Merge pull request #11260 from thalesmg/fix-rule-maps-nested-put-r51
fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
2023-07-14 15:02:24 -03:00
Thales Macedo Garitezi
01b143c5ad fix(resource): don't destruct error tuple
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
2023-07-13 16:12:33 -03:00
Thales Macedo Garitezi
924ed06760
Merge pull request #11256 from thalesmg/bridge-api-ecpool-400-errors-20230712-r51
fix(bridge_api): return 400 for all ecpool start errors
2023-07-13 10:05:07 -03:00
Thales Macedo Garitezi
bbff1d6ae9 fix(bridge_api): return 400 for all ecpool start errors
Fixes https://emqx.atlassian.net/browse/EMQX-10460
(https://emqx.atlassian.net/browse/EMQX-10460?focusedCommentId=30126)
2023-07-12 15:32:34 -03:00
Thales Macedo Garitezi
bffef386c1 fix(rule_maps): avoid losing data when using emqx_rule_maps:nested_put
Fixes https://emqx.atlassian.net/browse/EMQX-10541
2023-07-12 14:28:18 -03:00
Kjell Winblad
6ce47614d2 refactor: HTTP bridge into the emqx_bridge_http app
Fixes:
https://emqx.atlassian.net/browse/EMQX-9568
2023-07-12 14:47:02 +02:00
Kjell Winblad
f28510b3ad refactor: HTTP connector into emqx_bridge_http app 2023-07-12 14:46:43 +02:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
firest
a1d2b2ca51 chore: bump app versions 2023-07-07 11:17:54 +08:00
Thales Macedo Garitezi
353cc065db
Merge pull request #11175 from thalesmg/fix-mysql-start-error
fix(mysql_bridge): make nxdomain a 400 API error
2023-07-06 08:26:12 -03:00
Andrew Mayorov
d568d43fe4
test(bridge): switch emqx_bridge_api_SUITE to use new tooling 2023-07-04 13:20:03 +02:00
Andrew Mayorov
f76f3b77d7
test: add tooling to make testruns more isolated and predictable
Also showcase their usage in some of the existing testsuites.
2023-07-04 12:53:39 +02:00
Thales Macedo Garitezi
c216dfd96b fix(mysql_bridge): make nxdomain a 400 API error
Fixes https://emqx.atlassian.net/browse/EMQX-10460
2023-06-30 16:10:09 -03:00
Stefan Strigler
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
Thales Macedo Garitezi
6fe6aa7997
Merge pull request #11162 from thalesmg/treat-404-as-failure-master
fix(webhook): treat 404 and other error replies as errors in async requests
2023-06-30 13:13:18 -03:00
Thales Macedo Garitezi
ca435975de fix(webhook): treat http status code 429 as recoverable 2023-06-30 09:46:03 -03:00
Thales Macedo Garitezi
59b109eb5c fix(webhook): treat 404 and other error replies as errors in async requests
Fixes https://emqx.atlassian.net/browse/EMQX-10405

The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
Stefan Strigler
d71d70c889 fix: allow to pass in Opts rather than just a ReplyTo 2023-06-29 16:09:45 +02:00
Stefan Strigler
40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler
837acd4234 test(emqx_rule_engine): test rule metrics 2023-06-29 16:09:45 +02:00
Stefan Strigler
ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi
30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
Thales Macedo Garitezi
c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi
f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
zhongwencool
093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
Andrew Mayorov
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
2023-06-17 00:54:24 +02:00
zhongwencool
e42cc58694
Merge pull request #11050 from emqx/release-51
chore: sync release-51 back to master
2023-06-15 09:45:31 +08:00
Andrew Mayorov
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
2023-06-15 00:50:15 +03:00
zhongwencool
9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup
feat: implement configuration and user data export/import
2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI
Closes: EMQX-9203
2023-06-09 14:11:47 +03:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
Kjell Winblad
273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609
fix: friendly error message when creating bridges with too long names
2023-06-08 15:46:42 +02:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
2023-06-05 22:41:07 +03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
2023-06-05 22:41:07 +03:00
Kjell Winblad
375661c6a1 test: add test case for error when bridge name is too long 2023-06-05 17:02:08 +02:00
Andrew Mayorov
5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests
chore(bridge): drop deprecated `mode` setting from examples / tests
2023-06-02 22:46:31 +03:00
Zaiming (Stone) Shi
aa46c0a484
Merge pull request #10928 from zmstone/0602-sync-release-51-to-master
0602 sync release 51 to master
2023-06-02 21:45:21 +02:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
2023-06-02 16:20:36 -03:00
Andrew Mayorov
09290f8d61
chore(bridge): drop deprecated mode setting from examples / tests 2023-06-02 20:31:31 +03:00
Zaiming (Stone) Shi
28c564d15b Merge remote-tracking branch 'origin/master' into release-51 2023-06-02 16:25:48 +02:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
某文
c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Stefan Strigler
e011b5532f
Merge pull request #10881 from sstrigler/EMQX-9355-5-0-x-test-coverage-80
IoTDB test coverage
2023-06-01 15:48:56 +02:00
Kjell Winblad
75ff76a16b fix: friendly error message when creating bridges with too long names
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
2023-06-01 15:30:06 +02:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id
fix(ruleeng): parse bridge id on create / update
2023-06-01 10:21:26 +03:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update
Instead of parsing it every time in the `send_message/2` hot path.
2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
2023-05-31 10:19:55 -03:00
Andrew Mayorov
7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Andrew Mayorov
a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
2023-05-30 17:21:44 +03:00
Stefan Strigler
767f7c57e7 test: check device_id is used from topic or config 2023-05-30 15:55:21 +02:00
Stefan Strigler
938d62a666 test: handle bad payload 2023-05-30 15:55:21 +02:00
Stefan Strigler
64d582770d test: add tracepoints 2023-05-30 15:55:21 +02:00
Andrew Mayorov
4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
2023-05-30 14:50:05 +03:00
Andrew Mayorov
81e78516aa
feat(mqttconn): employ ecpool instead of a single worker 2023-05-30 14:50:05 +03:00
Andrew Mayorov
bd956d00b6
feat(mqttconn): stop using gproc in hot path
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
2023-05-30 14:50:05 +03:00
Zaiming (Stone) Shi
91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master
0530 merge release 50 to master
2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi
9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
Thales Macedo Garitezi
67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50
feat(kafka): ensure allocated resources are removed on failures
2023-05-29 16:49:29 -03:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi
cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi
772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
2023-05-26 08:39:00 +02:00
Zaiming (Stone) Shi
407cab220b
Merge pull request #10760 from paulozulato/fix-failure-joining
fix: avoid error 500 when node is re-joining cluster
2023-05-26 03:08:17 +02:00
Thales Macedo Garitezi
e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Paulo Zulato
ea86f4442b fix: avoid error 500 when node is re-joining cluster
Fixes https://emqx.atlassian.net/browse/EMQX-9899
2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi
194faefc8a fix(webhook): keep resource_opts.request_timeout for webhook bridge (r5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
2023-05-25 13:19:27 -03:00
JianBo He
6b5ff24952 test: assert the block_until result 2023-05-25 18:12:38 +08:00
JianBo He
de7f1c8aec test: add tests for auto_restart_interval 2023-05-25 17:15:19 +08:00
zhongwencool
14a6b36899
Merge pull request #10803 from JimMoen/merge-release-50
Merge release-50 into master
2023-05-24 23:08:46 +08:00
JimMoen
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad
4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
2023-05-24 13:09:47 +02:00
某文
7da0860d6c chore: upgrade minirest to 1.3.10 for more clear error msg 2023-05-24 10:44:52 +08:00
Thales Macedo Garitezi
44ed6a19ee fix(bridge): pass resource option request_timeout = infinity along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00
Thales Macedo Garitezi
b5f63f4151 test: fix wrong test listeners option 2023-05-23 14:50:43 -03:00
某文
1deb3c50c8 test: bridge test init dashboard listener failed 2023-05-23 10:28:01 +08:00
某文
67a30ee976 test: bridge test init dashboard listener failed 2023-05-23 10:16:43 +08:00
Zaiming (Stone) Shi
732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
Thales Macedo Garitezi
7d798c10e9 perf(buffer_worker): flush metrics periodically inside buffer worker process
Fixes https://emqx.atlassian.net/browse/EMQX-9905

Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
2023-05-22 09:11:23 -03:00
William Yang
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path
perf(config): enforcing atom key  path in hotcode path
2023-05-22 09:38:09 +02:00
Thales Macedo Garitezi
09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
Fixes https://emqx.atlassian.net/browse/EMQX-9872
2023-05-18 14:25:13 -03:00
Thales Macedo Garitezi
5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi
447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Thales Macedo Garitezi
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
Thales Macedo Garitezi
a7b41e1cdf perf(webhook): add retry attempts for async
This is a performance improvement for webhook bridge.

Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well.  Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.

This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
William Yang
cdf42760fa chore: little change avoid atom leak 2023-05-17 11:01:03 +02:00
某文
2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
Zaiming (Stone) Shi
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field
0421 refactor delete unused schema field
2023-05-16 22:22:03 +02:00
某文
a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
William Yang
1c746ed289 perf(config): enforcing atom key path in hotcode path 2023-05-16 09:57:04 +02:00
某文
7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
zhongwencool
a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zaiming (Stone) Shi
7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Zhongwen Deng
14952658d5 chore: convert rotation struct to roation_count 2023-05-09 11:24:51 +08:00
Stefan Strigler
d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zaiming (Stone) Shi
856129984b refactor: remove raw_with_default config load option
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
2023-05-04 09:48:28 +02:00
Zaiming (Stone) Shi
d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi
57cc854a4a test(bridge): fix bridge map type filed converters
now the converters on map type fields only work at the wrapping map
level but not the values
2023-04-30 10:45:11 +02:00
Paulo Zulato
dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi
ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-8398
2023-04-24 10:28:26 -03:00
Serge Tupchii
45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
Fixes: EMQX-9310
2023-04-20 17:21:43 +03:00
Zaiming (Stone) Shi
30fd9b10f7 refactor: change bridges config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Ivan Dyachkov
dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
某文
fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
Ivan Dyachkov
9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Stefan Strigler
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
Thales Macedo Garitezi
e073bc90bc refactor(buffer_worker): rename s/queue/buffer/g 2023-04-14 11:37:19 -03:00
Stefan Strigler
4f80690162 fix: byebye jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen
790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 17:55:48 +08:00
Andrew Mayorov
5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics
feat(resource): ask for metrics only when needed
2023-04-14 12:28:52 +03:00
JimMoen
5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
Zhongwen Deng
cc2beda37a feat: hiden rule_engine/bridge/authz/authn from doc/example 2023-04-13 14:42:21 +08:00
Ivan Dyachkov
bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov
e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00