Commit Graph

165 Commits

Author SHA1 Message Date
Ivan Dyachkov cc3b26a3ac Merge remote-tracking branch 'upstream/release-58' into 20240807-sync-release-branches 2024-08-07 09:48:38 +02:00
Thales Macedo Garitezi 85cff5e7eb fix: merge conflicts 2024-07-31 09:14:29 -03:00
Ivan Dyachkov 74cef7937d Merge remote-tracking branch 'upstream/release-57' into 20240731-sync-release-57 2024-07-31 11:31:29 +02:00
zmstone 2e39c4ad5e
Merge pull request #13495 from thalesmg/20240719-m-hide-enable-fields-mkII
chore: hide enable flags from schema and config examples
2024-07-29 21:31:20 +02:00
Thales Macedo Garitezi 693d5dd394 feat: attempt to automatically decode `payload` similar to key and message templates 2024-07-29 13:01:06 -03:00
Thales Macedo Garitezi 6786c9b517
refactor: improve descriptions and identifiers
Co-authored-by: zmstone <zmstone@gmail.com>
2024-07-29 09:45:52 -03:00
Thales Macedo Garitezi 1d56ac6e5e refactor: change topic schema type 2024-07-26 14:26:21 -03:00
Thales Macedo Garitezi 4e0742c66f feat: make kafka producer freely dynamic 2024-07-26 14:25:20 -03:00
Thales Macedo Garitezi df1f4fad70 feat(kafka producer): allow dynamic topics from pre-configured topics
Fixes https://emqx.atlassian.net/browse/EMQX-12656
2024-07-25 15:33:12 -03:00
Ivan Dyachkov 606d829256 Merge remote-tracking branch 'upstream/release-57' into 20240724-sync-release-57 2024-07-24 10:28:00 +02:00
zhongwencool e7d07ea17c feat: add resource_type to emqx_resource behaviour 2024-07-23 18:24:28 +08:00
zhongwencool cba3f532f8 feat: don't record dry_run log 2024-07-23 15:14:41 +08:00
Thales Macedo Garitezi 69f5b6fa6c chore: hide `enable` fields from docgen
Fixes https://emqx.atlassian.net/browse/EMQX-12730
2024-07-22 13:26:53 -03:00
Ivan Dyachkov e28750b522 Merge remote-tracking branch 'upstream/release-57' into 0701-sync-release-57 2024-07-01 13:52:14 +02:00
Thales Macedo Garitezi 067beece75 chore: add count to thrown error 2024-06-28 12:06:47 -03:00
Thales Macedo Garitezi 4bd0abc93f chore: bump app vsns 2024-06-27 09:22:06 -03:00
Thales Macedo Garitezi ed5e6599d9 fix(buffer worker, kafka): send reply when async call fails immediately
Fixes https://emqx.atlassian.net/browse/EMQX-12585
2024-06-27 09:20:00 -03:00
Thales Macedo Garitezi 6023012f8b fix(kafka and derivatives): add `alias` config to avoid clashes with same topic
Fixes https://emqx.atlassian.net/browse/EMQX-12592
2024-06-27 09:20:00 -03:00
zmstone 5d1608f34b chore: minimize oss/platform diff 2024-06-24 15:30:03 +02:00
Thales Macedo Garitezi 20cffb54d4 Merge branch 'release-57' into sync-r57-m-20240617 2024-06-17 17:10:12 -03:00
zmstone 3ac4ddcbe3 fix(kafka): handle message_too_large
bump 'failed' counter
2024-06-17 17:27:25 +02:00
Ivan Dyachkov 29ad07ce29 Merge remote-tracking branch 'upstream/release-57' into 0531-sync-release-57 2024-05-31 07:32:26 +02:00
zmstone a00d9f17b1 chore: upgrade kafka producer lib wolff to 1.10.4
wolff-1.10.4 handles message_too_large error code from Kafka.
It tries to split the too-large batch into single-call batches
(for EMQX, one call batch is always one message), and retry.
In case a single-call batch is too large, EMQX should increment
a failure counter.
2024-05-30 15:40:10 +02:00
zmstone 9fe29bf5f7 chore: bump kafka bridge app vsn 2024-05-29 21:57:14 +02:00
zmstone 238c207b09 chore: bump app versions 2024-05-27 20:26:52 +02:00
ieQu1 acb19a06cf Merge remote-tracking branch 'origin/release-57' 2024-05-21 15:04:46 +02:00
zmstone 331d44a78a chore: improve Kafka producer health-check error logs
The Kafka producer lib wolff returns detailed error logs
about which host:port had issue, EMQX should log them to help
troubleshooting
2024-05-18 19:12:51 +02:00
Kjell Winblad 72b0c63ed2
Merge pull request #13043 from kjellwinblad/kjell/kafka_incorrect_query_type/EMQX-12363
fix(kafka trace): do not include query type in trace entry
2024-05-16 10:08:16 +02:00
firest 437a0c0d55 fix(rocketmq): fix a typo and a RocketMQ encoding error 2024-05-14 23:01:31 +08:00
Kjell Winblad 121084b2ff fix(kafka trace): do not include query type in trace entry
For Kafka that has its internal buffering the on_query_async callback is
used both for sync and async queries (for sync queries the caller waits
for a response directly after calling on_query_async). Therefore, before
this commit, the trace included incorrect query type information. This
has been fixed by removing the query type information from the trace.
This should be okay since the query type is not essential information
and can be derived from the configuration.

Fixes:
https://emqx.atlassian.net/browse/EMQX-12363
2024-05-14 12:24:30 +02:00
Thales Macedo Garitezi 3818b75188
Merge pull request #12959 from thalesmg/kprodu-connector-hc-m-20240429
feat(kafka producer): add health check topic option
2024-05-03 12:48:48 -03:00
Thales Macedo Garitezi 6f3da6b131 feat(kafka producer): add health check topic option
Fixes https://emqx.atlassian.net/browse/EMQX-12241

This allows more accurate health checking for Kafka Producers.  Without a topic, it's not
possible to actually probe the connection to partition leaders, so the connector might not
be reported as `disconnected` without testing a concrete topic.
2024-05-02 17:07:44 -03:00
Thales Macedo Garitezi eb113fa578 fix: add non-empty validator 2024-05-02 11:19:00 -03:00
Thales Macedo Garitezi 14ef0b1e51 feat(kafka consumer): allow custom group id
Fixes https://emqx.atlassian.net/browse/EMQX-12273
Fixes EMQX-12273

When consuming messages in Kafka in Alibaba Cloud, the group needs to be configured in
advance, and then the consumer can use the group to consume messages. Automatic group
creation is generally not allowed online.
2024-05-01 13:53:29 -03:00
Kjell Winblad 32c27f1711 feat: add stop after render and after render trace to kafka action 2024-04-25 11:51:46 +02:00
zmstone ca56e7e8d7 fix(kafka): headers are template fields 2024-04-18 13:04:36 +02:00
zmstone 5c014f4c29 test: fix test cases 2024-04-17 20:34:33 +02:00
zmstone 51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
Ivan Dyachkov db9efb9317 chore: bump apps versions 2024-03-28 10:19:09 +01:00
Ivan Dyachkov f4446ec680 Merge remote-tracking branch 'upstream/release-56' into 0328-sync-release-56 2024-03-28 09:59:54 +01:00
Kjell Winblad 94c6e5bb38
Merge pull request #12727 from kjellwinblad/kjell/emqx_connector_info/EMQX-11427
refactor: add emqx_connector_info behavior
2024-03-25 14:31:40 +01:00
Thales Macedo Garitezi bc5775a988 fix(kafka_producer): handle ancient v1 config when migrating to actions
Fixes https://emqx.atlassian.net/browse/EMQX-12064
2024-03-22 09:45:27 -03:00
Kjell Winblad 2b684c7bcc fix(emqx_connector_info): refactoring based on thalesmg's comments 2024-03-22 09:29:26 +01:00
Kjell Winblad 4bfc51c052 refactor(emqx_connector_info): improve documentation and type specs 2024-03-21 16:08:31 +01:00
Thales Macedo Garitezi e837791f94 fix(kafka-based bridges): avoid trying to get raw config for replayq dir
Fixes https://emqx.atlassian.net/browse/EMQX-12049
2024-03-21 11:49:00 -03:00
Kjell Winblad a3e631cda2 refactor(kafka connectors): to use emqx_connector_info
This commit refactors the `emqx_bridge_kafka` to use the
`emqx_connector_info` behavior. The `emqx_bridge_kafka` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
Thales Macedo Garitezi a852695950 fix(kafka_consumer): validate topic mapping in v2 schema
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
Thales Macedo Garitezi 963e0de0c3 fix(kafka_consumer): check client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11945
2024-03-07 14:22:03 -03:00
Thales Macedo Garitezi 16b3dc1166 refactor: use individual keys for subscriber id resources 2024-02-29 10:39:37 -03:00
Thales Macedo Garitezi 0aa9a872a3 feat: check `brod_sup` for client id 2024-02-29 09:53:23 -03:00