Commit Graph

99 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 20cffb54d4 Merge branch 'release-57' into sync-r57-m-20240617 2024-06-17 17:10:12 -03:00
zmstone 3ac4ddcbe3 fix(kafka): handle message_too_large
bump 'failed' counter
2024-06-17 17:27:25 +02:00
Thales Macedo Garitezi 3818b75188
Merge pull request #12959 from thalesmg/kprodu-connector-hc-m-20240429
feat(kafka producer): add health check topic option
2024-05-03 12:48:48 -03:00
Thales Macedo Garitezi 6f3da6b131 feat(kafka producer): add health check topic option
Fixes https://emqx.atlassian.net/browse/EMQX-12241

This allows more accurate health checking for Kafka Producers.  Without a topic, it's not
possible to actually probe the connection to partition leaders, so the connector might not
be reported as `disconnected` without testing a concrete topic.
2024-05-02 17:07:44 -03:00
Thales Macedo Garitezi eb113fa578 fix: add non-empty validator 2024-05-02 11:19:00 -03:00
Thales Macedo Garitezi 3942b371d7 feat(kafka consumer): allow custom group id
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
2024-05-02 08:59:40 -03:00
Thales Macedo Garitezi 14ef0b1e51 feat(kafka consumer): allow custom group id
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
2024-05-01 13:53:29 -03:00
zmstone dfd13b4ab5 test: fix dashboard schema json test cases 2024-04-11 15:12:01 +02:00
Thales Macedo Garitezi bc5775a988 fix(kafka_producer): handle ancient v1 config when migrating to actions
Fixes https://emqx.atlassian.net/browse/EMQX-12064
2024-03-22 09:45:27 -03:00
Thales Macedo Garitezi a852695950 fix(kafka_consumer): validate topic mapping in v2 schema
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
Thales Macedo Garitezi 963e0de0c3 fix(kafka_consumer): check client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11945
2024-03-07 14:22:03 -03:00
Thales Macedo Garitezi 5d94a910b9 test: reduce flakiness 2024-02-29 13:59:43 -03:00
Thales Macedo Garitezi 16b3dc1166 refactor: use individual keys for subscriber id resources 2024-02-29 10:39:37 -03:00
Thales Macedo Garitezi 6b9844ae82 feat: migrate kafka consumer bridge to source + connector
Fixes https://emqx.atlassian.net/browse/EMQX-11848
2024-02-27 17:52:33 -03:00
zhongwencool f2619ce17a
Merge pull request #12571 from zhongwencool/delete-repeat-ci-test
test(bridge_v1): delete repeat probe_bridge_api code
2024-02-23 17:17:39 +08:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
zhongwencool d7faab4195 test(bridge_v1): delete repeat probe_bridge_api code 2024-02-23 14:43:42 +08:00
Zaiming (Stone) Shi 333eb34300 test: call emqx_resource:remove_local 2024-02-21 10:53:22 +01:00
Thales Macedo Garitezi 432ddc5a3b fix(bridge_v1_schema): undo changing v1 schema to avoid confusion 2023-12-15 12:55:34 -03:00
Zaiming (Stone) Shi 55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 21:35:40 +01:00
Zaiming (Stone) Shi 22f7cc1622 test: replace 'slave' and 'ct_slave' with 'peer' 2023-12-01 08:07:09 +01:00
Thales Macedo Garitezi 0388e1c1c4 fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-11-30 17:51:32 -03:00
Stefan Strigler c1ef773e77 fix: check for sane state after regular shutdown 2023-11-24 15:09:35 +01:00
Thales Macedo Garitezi fdfa3213cc fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 11:41:26 -03:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Thales Macedo Garitezi b3dffa4390 fix(kafka): don't return `parameters` from `/bridges` API
Fixes https://emqx.atlassian.net/browse/EMQX-11412
2023-11-21 16:00:15 -03:00
Thales Macedo Garitezi 53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Thales Macedo Garitezi 36b5d58957 test: reorganize test suite a bit 2023-11-16 13:16:02 -03:00
Ivan Dyachkov 7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Andrew Mayorov fda395014c
test(kafka): verify file-based secrets work 2023-11-14 16:05:25 +07:00
Andrew Mayorov aa458b65d6
test(kafka): simplify consumer testsuite matrix setup 2023-11-14 16:05:25 +07:00
Andrew Mayorov 18cd98def6
chore(test): fix formatting quirks 2023-11-14 16:05:25 +07:00
Thales Macedo Garitezi 371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
Thales Macedo Garitezi f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-08 09:53:42 -03:00
Ivan Dyachkov 0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
Thales Macedo Garitezi b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi 7dcdbc9e51 fix(resource): take error from action/connector before attempting query
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
Zaiming (Stone) Shi bb8f80f256 test: add test case to pin bridge schema json version 0.1.0 2023-11-06 21:59:36 +01:00
Zaiming (Stone) Shi 9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi 4265ef66cc fix(kafka_producer): don't return `disconnected` when there are connection issues while starting the bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixex https://emqx.atlassian.net/browse/EMQX-11298

We don't enforce the connection to be up when starting/creating the bridge, otherwise the
status will be `disconnected` for a possibly transient reason such as network issues or
Kafka broker restart.

Same applies for Azure Event Hub Producer bridge, as they share the same module.
2023-11-03 15:49:36 -03:00
Zaiming (Stone) Shi 4dd054b0a2 test(emqx_bridge_kafka_impl_consumer_SUITE): fix flaky 2023-11-03 11:48:50 +01:00
Kjell Winblad b06d05eaac test(bridge_v2): fix test case after new API 2023-11-01 15:27:54 +01:00
Kjell Winblad 96d6c6db49 test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change 2023-11-01 15:27:53 +01:00
Ivan Dyachkov b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Zaiming (Stone) Shi c07cf9051e
Merge pull request #11844 from zmstone/1030-downgrade-bridge-type-for-old-api
1030 downgrade bridge type for old api
2023-10-30 19:47:24 +01:00
Zaiming (Stone) Shi 350e8433ab fix(bridge_v2): downgrade bridge type for GET api results 2023-10-30 18:17:35 +01:00
Thales Macedo Garitezi a60b96c5fd test(kafka_producer): minor adjustments to test suite
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
Zaiming (Stone) Shi 6eb3bb7cff Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53 2023-10-14 10:16:38 +02:00