9dc3a169b3
feat: split bridges into a connector part and a bridge part
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.
Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
cf2075d7d8
chore: remove mention of `is_buffer_supported` from typespec
2023-10-10 09:49:18 -03:00
d6781efee2
fix(resource): change how buffer workers are started
2023-10-09 15:02:25 -03:00
902b1d6ec5
fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
...
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
eebfb44f72
fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
...
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
79cf0a2ced
fix(kafka_producer): correctly handle metrics for connector that have internal buffers
...
Fixes https://emqx.atlassian.net/browse/EMQX-11086
There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).
We use simple_sync_query to call the Kafka Producer bridge. If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
34186fcc74
fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
...
Fixes https://emqx.atlassian.net/browse/EMQX-11085
Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
c2d750aa09
fix(resource): redact query args in exception log
2023-09-29 09:20:42 +02:00
dca8fdb17f
fix(resource): respect the start_timeout
2023-09-28 16:36:41 +08:00
dafd7c6085
chore: bump apps versions
2023-09-21 10:58:42 +02:00
123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
...
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
dd687d9582
fix: dialyzer warning
2023-09-20 22:41:26 +08:00
2f1fa2e961
chore: unified slog message formatting to improve logging consistency
2023-09-20 18:13:00 +08:00
c26a18e949
fix: always return ok when remove local resource
2023-09-20 18:02:42 +08:00
dfcede8794
fix: increment matched counter when bridge is unhealthy
...
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-30 10:52:53 -03:00
cc3ba18734
fix: increment dropped message counter when bridge is unhealthy
...
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
42877e282d
fix: flatten error message on resource validator
...
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
ebecbd1545
fix(bridge): make dryrun health check timeout more malleable
...
Fixes https://emqx.atlassian.net/browse/EMQX-10773
- Makes the timeout for probing a bridge more malleable to account for differences between
each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
slowness/throttling.
2023-08-17 09:21:19 -03:00
cbfca8c043
chore: merge master into release-51
2023-07-27 15:19:57 +02:00
3b1e436d3f
refactor: use `emqx_pool:async_submit` to avoid excessive spawning
2023-07-20 10:17:43 -03:00
eb41b77de4
fix(rule_metrics): notify rule metrics of late replies and expired requests
...
Fixes https://emqx.atlassian.net/browse/EMQX-10600
2023-07-19 11:39:28 -03:00
01b143c5ad
fix(resource): don't destruct error tuple
...
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
2023-07-13 16:12:33 -03:00
b5cc8fb3c3
fix: start_after_created's default value
2023-07-07 16:39:26 +08:00
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
...
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
321fd53132
fix: use ReplyTo in QUERY for async
2023-06-29 16:09:45 +02:00
40dd34a704
fix: use reply_to instead of async_reply_fun
2023-06-29 16:09:45 +02:00
1363108678
fix: fix simple_sync_query
2023-06-29 16:09:45 +02:00
2274a192cc
fix(emqx_resource): call async reply fun in simple_aysnc_query
2023-06-29 16:09:45 +02:00
ae636a52d7
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-29 16:09:45 +02:00
30e0b4be54
test(gcp_pubsub_consumer): add more tests and improve bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
c4fc0e767e
feat: allow specifying more helpful messages for unhealthy targets
2023-06-27 17:13:43 -03:00
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
...
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
c58a98954b
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-22 11:05:51 +02:00
62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
...
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
...
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00
86d787eced
chore: bump hocon to 0.39.10
...
Which comes with a fix for slightly more user-friendly validation error
messages.
2023-06-21 21:25:43 +02:00
18f0510353
fix(mongo): return health check failure reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
decfd6df2b
feat(buffer_worker): log expired message count
...
Fixes https://emqx.atlassian.net/browse/EMQX-10165
```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
2023-06-21 14:38:51 -03:00
7cf8a6c892
chore: bump app vsns
2023-06-21 16:36:51 +02:00
1d791d7a8c
fix(resource): validate maximum worker pool size
...
Fixes https://emqx.atlassian.net/browse/EMQX-10297
2023-06-20 14:26:42 -03:00
13746c2cdf
fix(resource): check status when (re)starting a resource
...
Fixes https://emqx.atlassian.net/browse/EMQX-10290
2023-06-19 18:01:02 -03:00
7f850f7499
fix(resource): fix `query_mode/0` type and usage
2023-06-19 15:59:00 -03:00
9454af9a8b
feat(postgresql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb
...
fix(mongodb): hide batch_size for mongodb resource
2023-06-11 21:26:12 +02:00
ddef751527
fix(mongodb): hide batch_size for mongodb resource
...
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
2023-06-11 11:08:58 +02:00
4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631
...
feat: add sync/async option to the Kafka producer bridge
2023-06-09 12:51:23 +02:00
2671e8ecf9
fix: dialyzer type problem
2023-06-09 11:00:05 +02:00
86a7b2d69a
fix(resource): improve log security when resource creation fails
2023-06-09 11:43:42 +08:00
cb3a5fdbd4
style: only callback modules should do dynamic calls
2023-06-08 16:27:04 +02:00
Kjell Winblad