yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c8af5a702c
emqx/apps/emqx_bridge_azure_event_hub
History
Thales Macedo Garitezi 016ae0524f fix(aeh_producer): remove timestamp template field 
Fixes https://emqx.atlassian.net/browse/EMQX-10847

Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.

Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type

Example message consumed from AEH:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test0",
  "ts" => 1692879703006,
  "ts_type" => "append",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Note the ts_type above is append.

Example message from a Kafka broker whose ts type is create:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test-topic-three-partitions",
  "ts" => 1692881883668,
  "ts_type" => "create",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
2023-08-24 14:20:42 -03:00
..
src fix(aeh_producer): remove timestamp template field 2023-08-24 14:20:42 -03:00
test fix(aeh_producer): remove timestamp template field 2023-08-24 14:20:42 -03:00
BSL.txt feat: add Azure Event Hub Producer bridge 2023-07-27 09:22:39 -03:00
docker-ct feat: add Azure Event Hub Producer bridge 2023-07-27 09:22:39 -03:00
README.md feat: add Azure Event Hub Producer bridge 2023-07-27 09:22:39 -03:00
rebar.config feat: add Azure Event Hub Producer bridge 2023-07-27 09:22:39 -03:00

README.md

Azure Event Hub Data Integration Bridge

This application houses the Azure Event Hub (AEH) Producer data integration bridge for EMQX Enterprise Edition. It provides the means to connect to Azure Event Hub Producer and publish messages to it via the Kafka protocol.

Currently, our Kafka Producer library (wolff) has its own replayq buffering implementation, so this bridge does not require buffer workers from emqx_resource. It implements the connection management and interaction without need for a separate connector app, since it's not used by authentication and authorization applications.

Documentation links

For more information about Kafka interface for AEH, please see the official docs.

Configurations

Please see Ingest Data into Azure Event Hub for more detailed info.

Contributing

Please see our contributing.md.

License

EMQ Business Source License 1.1, refer to LICENSE.