yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,027 Commits 393 Branches 669 Tags 74 MiB
abcb3166f6
Commit Graph

52 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
abcb3166f6 fix: use binary string for raw default values 2023-10-24 14:46:01 +02:00
Thales Macedo Garitezi
eebfb44f72 fix(resource): create simple_async_internal_buffer query mode for bridges with internal buffering 
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi
79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers 
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi
34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in connecting state 
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
 2023-10-06 11:43:29 -03:00
Zaiming (Stone) Shi
bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Ilya Averyanov
14983ec14a chore(hooks): validate hookpoints and document hook callbacks 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-26 14:27:42 +03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool
9e75ff88af
chore: apply suggestions from code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-20 22:15:39 +08:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Zaiming (Stone) Shi
18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-30 14:48:10 -03:00
Paulo Zulato
0b86f04bae fix(kafka): fix template processing for header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-28 17:28:05 -03:00
Paulo Zulato
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling 
fix(kafka): fix result handling when sending message with invalid header
 2023-08-25 14:57:09 -03:00
Paulo Zulato
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id 
Fix Kafka/AEH ClientId uniqueness
 2023-08-25 10:14:44 -03:00
Paulo Zulato
535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi
0381ac0410 fix(kafka_producer): use correct timestamp template field 
Fixes https://emqx.atlassian.net/browse/EMQX-10847
 2023-08-24 14:20:42 -03:00
Paulo Zulato
60e6217496 fix(kafka): fix result handling when sending message with invalid header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi
ffca581229 feat(kafka): add option to configure health check interval 
Fixes https://emqx.atlassian.net/browse/EMQX-10781
 2023-08-16 11:08:06 -03:00
Thales Macedo Garitezi
b6decf9592
Merge pull request #11402 from thalesmg/dynamic-kconsu-mqtt-topics-20230807 
feat(kafka_consumer): add mqtt topic placeholder support
 2023-08-07 15:10:29 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms 
Fixes https://emqx.atlassian.net/browse/EMQX-10653
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
9900a32850 feat(kafka_consumer): add mqtt topic placeholder support 
Fixes https://emqx.atlassian.net/browse/EMQX-10678
 2023-08-07 11:42:34 -03:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10338
 2023-07-27 09:22:39 -03:00
Thales Macedo Garitezi
6cd503865b fix(machine_boot): ensure emqx_bridge starts after its companion apps 
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
 2023-07-20 13:11:44 -03:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
Paulo Zulato
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers 
feat(kafka): add support for kafka headers and ext headers
 2023-06-22 12:04:32 -03:00
Zaiming (Stone) Shi
75eb18bc81 chore: bump app vsns 2023-06-22 11:09:09 +02:00
Paulo Zulato
f47cc2a458 feat(kafka): add support for kafka headers and ext headers 
Fixes https://emqx.atlassian.net/browse/EMQX-9175
 2023-06-21 18:27:39 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi
25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 13:54:57 -03:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Serge Tupchii
48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option 
Closes: EMQX-8725
 2023-06-09 22:10:17 +03:00
Paulo Zulato
720a84d48b fix(kafka): fix timeout field for some kafka fields 
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
 2023-06-09 13:38:04 -03:00
Andrew Mayorov
b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder 
refactor: tear `emqx_plugin_libs` application apart
 2023-06-09 17:02:14 +02:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app 
Also make user that existing code calls it directly.
 2023-06-09 14:44:36 +03:00
Kjell Winblad
1c7834e056 fix: fixes due to comments from @zmstone 2023-06-08 16:47:02 +02:00
Kjell Winblad
d524f8c805 refactor: rename config parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad
6f2271e9f0 test: add Kafka producer bridge test case for query_mode parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad
ed9e29e769 refactor: refacor query_mode detection code 
This commit refactor the query_mode resource detection code according to
a suggestion from @zmstone. This commit should not contain any
functional change except for a change of the Kafka producer bridge
config.
 2023-06-08 16:26:55 +02:00
Kjell Winblad
47fa17b3c1 feat: add sync/async option to the Kafka producer bridge 
This commit makes it possible to configure if a Kafka bridge should work
in query mode sync or async by setting the resource_opts.query_mode
configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-8631
 2023-06-08 13:16:06 +02:00
Thales Macedo Garitezi
46393343e2 chore: use timeout_duration types for timer fields 
Fixes https://emqx.atlassian.net/browse/EMQX-10020
 2023-06-05 11:46:38 -03:00
Thales Macedo Garitezi
9c3f838e14
Merge pull request #10841 from thalesmg/kafka-validate-key-v50 
feat({kafka,pulsar}_producer): add validation for empty message key when strategy = key_dispatch
 2023-05-30 09:37:15 -03:00
Thales Macedo Garitezi
9b7e473cf6 feat(kafka_producer): add validation for empty message key when strategy = key_dispatch 
Fixes https://emqx.atlassian.net/browse/EMQX-9979
 2023-05-26 12:02:58 -03:00
Thales Macedo Garitezi
0ca3f51503 fix(kafka): improve shutdown and health check logs during shutdown 2023-05-25 16:38:09 -03:00
Thales Macedo Garitezi
5df7314255 feat(kafka): ensure allocated resources are removed on failures 2023-05-25 16:38:09 -03:00
Zaiming (Stone) Shi
aaf3bf27c4 fix(kafka): ensure brod_gssapi sasl_auth are include in release 2023-05-07 09:43:58 +02:00
Andrew Mayorov
90cf1ade74
chore: bump application versions 
* emqx_connector 0.1.22
* emqx_bridge_gcp_pubsub 0.1.1
* emqx_bridge_kafka 0.1.2
* emqx_bridge_pulsar 0.1.1
* emqx_ee_connector 0.1.12
 2023-05-02 17:29:22 +03:00
Andrew Mayorov
4575167607
feat(resource): drop manager_id() type 2023-05-02 17:29:20 +03:00
Thales Macedo Garitezi
120d3e70ea chore: bump app vsns 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
4bcfbea056 refactor(kafka_consumer): follow up refactoring requested from previous pull request 
Follow up from https://github.com/emqx/emqx/pull/10273
 2023-04-24 09:28:36 -03:00