yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,128 Commits 393 Branches 669 Tags 74 MiB
5f17a8f2ce
Commit Graph

66 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
5f17a8f2ce fix(kafka): use client config for topic existence check 
Prior to this fix, Kafka producer config was used as client config
 2023-10-27 11:53:42 +02:00
Thales Macedo Garitezi
04a832a80a refactor(kafka): fix typo and take connector type from input 2023-10-27 11:53:42 +02:00
Kjell Winblad
90a23d98fe fix: channel handling when resource not connected 2023-10-27 11:53:42 +02:00
Zaiming (Stone) Shi
e2aca352b6 fix(kafka): Fix server name indication config for Kafka 
Also impoved azur event hub's doc for SNI
 2023-10-27 11:53:41 +02:00
Stefan Strigler
2228a0d477 feat: port azure event hub to bridge_v2 2023-10-27 11:53:41 +02:00
Kjell Winblad
e72b952138 fix: problems reported by dialyzer 2023-10-27 11:53:41 +02:00
Kjell Winblad
2cd1c88f7f fix: fixup after rebasing on release-23 2023-10-27 11:53:41 +02:00
Stefan Strigler
6568759a7e fix: add bridge_v2_probe 2023-10-27 11:53:41 +02:00
Stefan Strigler
38f5c7dcf4 feat: HTTP API for bridge_v2 2023-10-27 11:53:41 +02:00
Stefan Strigler
8567ccafc1 fix: don't assume resource-id starts with "connector:" 2023-10-27 11:53:40 +02:00
Stefan Strigler
4b4eb19b0b fix: add schema desc for connector 2023-10-27 11:53:40 +02:00
Stefan Strigler
229bc0ee34 fix: schema fixes for connector and kafka as connector 2023-10-27 11:53:40 +02:00
Kjell Winblad
c0df85ac09 feat: Bridge V2 compatiblilty layer progress and local topic 
* Most Bridge V1 HTTP API calls are now compatible with Bridge V2
* Local topics works for Bridge V2 now
* A lot of work on trying to get the old Kafka producer test suite
  to work after the refactorings
 2023-10-27 11:53:40 +02:00
Kjell Winblad
f7984be946 feat: add connector schema scaffold and break out Kafka conector 
This commit is the beginning of an effort to split bridges into a
connector part and a bridge part.

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
 2023-10-27 11:53:40 +02:00
Zaiming (Stone) Shi
abcb3166f6 fix: use binary string for raw default values 2023-10-24 14:46:01 +02:00
Thales Macedo Garitezi
eebfb44f72 fix(resource): create simple_async_internal_buffer query mode for bridges with internal buffering 
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi
79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers 
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi
34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in connecting state 
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
 2023-10-06 11:43:29 -03:00
Zaiming (Stone) Shi
bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Ilya Averyanov
14983ec14a chore(hooks): validate hookpoints and document hook callbacks 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-26 14:27:42 +03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool
9e75ff88af
chore: apply suggestions from code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-20 22:15:39 +08:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Zaiming (Stone) Shi
18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-30 14:48:10 -03:00
Paulo Zulato
0b86f04bae fix(kafka): fix template processing for header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-28 17:28:05 -03:00
Paulo Zulato
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling 
fix(kafka): fix result handling when sending message with invalid header
 2023-08-25 14:57:09 -03:00
Paulo Zulato
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id 
Fix Kafka/AEH ClientId uniqueness
 2023-08-25 10:14:44 -03:00
Paulo Zulato
535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi
0381ac0410 fix(kafka_producer): use correct timestamp template field 
Fixes https://emqx.atlassian.net/browse/EMQX-10847
 2023-08-24 14:20:42 -03:00
Paulo Zulato
60e6217496 fix(kafka): fix result handling when sending message with invalid header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi
ffca581229 feat(kafka): add option to configure health check interval 
Fixes https://emqx.atlassian.net/browse/EMQX-10781
 2023-08-16 11:08:06 -03:00
Thales Macedo Garitezi
b6decf9592
Merge pull request #11402 from thalesmg/dynamic-kconsu-mqtt-topics-20230807 
feat(kafka_consumer): add mqtt topic placeholder support
 2023-08-07 15:10:29 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms 
Fixes https://emqx.atlassian.net/browse/EMQX-10653
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
9900a32850 feat(kafka_consumer): add mqtt topic placeholder support 
Fixes https://emqx.atlassian.net/browse/EMQX-10678
 2023-08-07 11:42:34 -03:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10338
 2023-07-27 09:22:39 -03:00
Thales Macedo Garitezi
6cd503865b fix(machine_boot): ensure emqx_bridge starts after its companion apps 
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
 2023-07-20 13:11:44 -03:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
Paulo Zulato
f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers 
feat(kafka): add support for kafka headers and ext headers
 2023-06-22 12:04:32 -03:00
Zaiming (Stone) Shi
75eb18bc81 chore: bump app vsns 2023-06-22 11:09:09 +02:00
Paulo Zulato
f47cc2a458 feat(kafka): add support for kafka headers and ext headers 
Fixes https://emqx.atlassian.net/browse/EMQX-9175
 2023-06-21 18:27:39 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists 
Fixes https://emqx.atlassian.net/browse/EMQX-9026
 2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi
25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1) 
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
 2023-06-13 13:54:57 -03:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Serge Tupchii
48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option 
Closes: EMQX-8725
 2023-06-09 22:10:17 +03:00
Paulo Zulato
720a84d48b fix(kafka): fix timeout field for some kafka fields 
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
 2023-06-09 13:38:04 -03:00
Andrew Mayorov
b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder 
refactor: tear `emqx_plugin_libs` application apart
 2023-06-09 17:02:14 +02:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app 
Also make user that existing code calls it directly.
 2023-06-09 14:44:36 +03:00
Kjell Winblad
1c7834e056 fix: fixes due to comments from @zmstone 2023-06-08 16:47:02 +02:00