Commit Graph

121 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 04bf763890 fix(kafka-based bridges): avoid trying to get raw config for replayq dir
Fixes https://emqx.atlassian.net/browse/EMQX-12049
2024-03-28 09:13:34 -03:00
Thales Macedo Garitezi bc5775a988 fix(kafka_producer): handle ancient v1 config when migrating to actions
Fixes https://emqx.atlassian.net/browse/EMQX-12064
2024-03-22 09:45:27 -03:00
Thales Macedo Garitezi a852695950 fix(kafka_consumer): validate topic mapping in v2 schema
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
Thales Macedo Garitezi 963e0de0c3 fix(kafka_consumer): check client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11945
2024-03-07 14:22:03 -03:00
Thales Macedo Garitezi 16b3dc1166 refactor: use individual keys for subscriber id resources 2024-02-29 10:39:37 -03:00
Thales Macedo Garitezi 0aa9a872a3 feat: check `brod_sup` for client id 2024-02-29 09:53:23 -03:00
Thales Macedo Garitezi 6b9844ae82 feat: migrate kafka consumer bridge to source + connector
Fixes https://emqx.atlassian.net/browse/EMQX-11848
2024-02-27 17:52:33 -03:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Thales Macedo Garitezi d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
William Yang 80ff1b1657
Merge pull request #12434 from qzhuyan/fix/william/kafka-dont-stacktrace-when-no-partiion-count
kafka-bridge: dont stacktrace when no partition count
2024-02-01 15:09:45 +01:00
William Yang 16444b7da9 fix(bridge-kafka): no stacktrace when fetch topic partition count fail 2024-02-01 14:09:15 +01:00
Zaiming (Stone) Shi 3b42a7425b feat(kafka_producer): add partitions_limit 2024-01-31 19:26:01 +01:00
Thales Macedo Garitezi e219c3eda1
Merge pull request #12166 from kjellwinblad/kjell/mqtt_conn_action_3/EMQX-11489
feat: refactor MQTT bridge to source, action, and connector
2024-01-18 09:28:14 -03:00
Thales Macedo Garitezi 2a41cad54f fix(sources): remote irrelevant `resource_opts` fields for sources
Since they don't use buffer workers, they shouldn't have buffer-related sub-fields.
2024-01-16 11:29:35 -03:00
Thales Macedo Garitezi cc24fe6e93 feat(mqtt_consumer): add support for rule engine `FROM` 2024-01-16 10:42:52 -03:00
Zaiming (Stone) Shi ab66986f16 feat: add 'tags' field for action and connector 2024-01-16 13:57:48 +01:00
Zaiming (Stone) Shi 23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi 322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Zaiming (Stone) Shi 7338e394c8 perf: upgrade to wolff 1.9.1 for better performance
Upgrade from 1.8.0:
- wollf-1.9.0 has the global stats disabled by default
- wolff-1.9.1 improved client pid lookup performance
2023-12-18 18:54:21 +01:00
Thales Macedo Garitezi 5bdfac52b8 fix(bridge_kafka_producer): add missing `resource_opts` sub-fields to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi cb60880bab
Merge pull request #12067 from thalesmg/fix-kafka-check-client-m-20231130
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
2023-12-01 09:55:12 -03:00
Thales Macedo Garitezi 0388e1c1c4 fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-11-30 17:51:32 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Stefan Strigler 048f4724a9 feat(emqx_connector): add field 'actions' in API response
Also unify schemas, use emqx_connector_schema for the generic parts.
2023-11-27 16:09:17 +01:00
zhongwencool b5a00ec6b2
Merge pull request #12023 from emqx/master
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
Stefan Strigler d2901afd1b fix(emqx_bridge_kafka): match example in api schema 2023-11-24 10:07:11 +01:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Thales Macedo Garitezi bb549cdf8b feat(actions): allow multiple action info modules per application 2023-11-22 09:33:01 -03:00
Thales Macedo Garitezi 38d3a1d7d0 feat(actions): allow multiple action info modules per application 2023-11-22 09:30:37 -03:00
Thales Macedo Garitezi 39791511fc chore: remove obsolete workaround 2023-11-21 17:38:13 -03:00
Thales Macedo Garitezi b3dffa4390 fix(kafka): don't return `parameters` from `/bridges` API
Fixes https://emqx.atlassian.net/browse/EMQX-11412
2023-11-21 16:00:15 -03:00
Kjell Winblad b0d670aaa9
Merge pull request #11992 from kjellwinblad/kjell/backport/action_upgrade_downgrade_hooks
fix(action): upgrade and downgrade strategy
2023-11-21 19:14:37 +01:00
Thales Macedo Garitezi 068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120
sync `release-53` to `master`
2023-11-21 09:02:51 -03:00
Kjell Winblad 6030bf6fa5 fix(action): upgrade and downgrade strategy
This commit adds upgrade and downgrade hooks that are called when
upgrading from a bridge V1 to connector and action or the other way
around. The automatic translation is used if the callback is not
defined.

NOTE: Backported from master
2023-11-21 11:18:21 +01:00
Thales Macedo Garitezi 6f8630304d
Merge pull request #11980 from kjellwinblad/kjell/fix_fixup_callbacks/EMQX-11428
fix: bridge to action upgrade fixup hook should run after upgrade
2023-11-20 16:19:43 -03:00
Kjell Winblad 7fb5ade832 fix(kafka_producer): hocon renames kafka field 2023-11-20 19:20:13 +01:00
Kjell Winblad d214ae8772
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-20 18:59:57 +01:00
Kjell Winblad 3aa8044475 fix(action): upgrade and downgrade strategy
This commit fixes the upgrade and downgrade strategy when upgrading
from a bridge V1 to connector and action or the other way around so that
the custom callbacks get the complete unchanged input instead of the
result of the automatic translation. The automatic translation is used
if the callback is not defined.
2023-11-20 16:40:27 +01:00
Thales Macedo Garitezi f40f6bc5dd refactor: split `resource_opts` fields between connector and actions 2023-11-20 12:37:36 -03:00
Thales Macedo Garitezi 53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Kjell Winblad 79a764f117 fix: bridge to action upgrade fix up hook should run after upgrade
This commit changes how the `emqx_action_info` callback
`bridge_v1_to_action_fixup/1` works. It is now called after the
automatic upgrade instead of before. Since the full Bridge V1 config
might be needed to do the fixup, it is provided in a special field
`<<"__bridge_v1_conf__">>`. The `<<"__bridge_v1_conf__">>` field is
removed after the callback is called and can thus be ignored if it is
not needed.
2023-11-20 12:06:21 +01:00
Stefan Strigler 84ff7b0b38 feat(emqx_bridge): action_info with dynamic lookup
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
Thales Macedo Garitezi eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Kjell Winblad 86c126ffcd feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
2023-11-17 16:10:08 -03:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Andrew Mayorov d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default
feat(router): switch to v2 routing store by default
2023-11-16 15:56:26 +07:00
Andrew Mayorov 893e90b372
fix(kafka): use safe publish in consumer
Routing with v2 schema is actually more strict with respect to input to
`emqx_router` module routines. This causes Kafka consumer bridge to
crash when it tries to publish a message to a topic that looks like a
topic filter.
2023-11-15 22:37:00 +07:00
Ivan Dyachkov 7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Andrew Mayorov c300eb41a7
feat(kafka): accept wrapped secrets as passwords 2023-11-14 16:05:25 +07:00
Ivan Dyachkov a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00