Commit Graph

533 Commits

Author SHA1 Message Date
JianBo He c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
Zaiming (Stone) Shi 3261a12140 fix(emqx_resource): do not allow leading _ or - as resource name 2023-11-22 10:58:54 +01:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Thales Macedo Garitezi d6e9bbb95c fix(connector): validate connector name before converting ssl certs
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
2023-11-14 09:29:59 -03:00
Ivan Dyachkov a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Thales Macedo Garitezi b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi 7dcdbc9e51 fix(resource): take error from action/connector before attempting query
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
Thales Macedo Garitezi 2b8cf50a1d chore: rename `bridges_v2` -> `actions` in the public facing APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi 600747b7e5 fix(bridge): do not allow dot in bridge name
also validate name at the API entry
2023-11-03 20:44:57 +01:00
Kjell Winblad 357b664c8d fix(bridge_v2): more fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:54 +01:00
Kjell Winblad 96d6c6db49 test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change 2023-11-01 15:27:53 +01:00
Kjell Winblad edb1d37e67 chore(bridge_v2): make fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:53 +01:00
Kjell Winblad 95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
Thales Macedo Garitezi cf2075d7d8 chore: remove mention of `is_buffer_supported` from typespec 2023-10-10 09:49:18 -03:00
Thales Macedo Garitezi d6781efee2 fix(resource): change how buffer workers are started 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 902b1d6ec5 fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi eebfb44f72 fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
Zaiming (Stone) Shi c2d750aa09 fix(resource): redact query args in exception log 2023-09-29 09:20:42 +02:00
firest dca8fdb17f fix(resource): respect the start_timeout 2023-09-28 16:36:41 +08:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
zhongwencool c26a18e949 fix: always return ok when remove local resource 2023-09-20 18:02:42 +08:00
Paulo Zulato dfcede8794 fix: increment matched counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-30 10:52:53 -03:00
Paulo Zulato cc3ba18734 fix: increment dropped message counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
Paulo Zulato 42877e282d fix: flatten error message on resource validator
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
Thales Macedo Garitezi ebecbd1545 fix(bridge): make dryrun health check timeout more malleable
Fixes https://emqx.atlassian.net/browse/EMQX-10773

- Makes the timeout for probing a bridge more malleable to account for differences between
  each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
  slowness/throttling.
2023-08-17 09:21:19 -03:00
Ivan Dyachkov cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi 3b1e436d3f refactor: use `emqx_pool:async_submit` to avoid excessive spawning 2023-07-20 10:17:43 -03:00
Thales Macedo Garitezi eb41b77de4 fix(rule_metrics): notify rule metrics of late replies and expired requests
Fixes https://emqx.atlassian.net/browse/EMQX-10600
2023-07-19 11:39:28 -03:00
Thales Macedo Garitezi 01b143c5ad fix(resource): don't destruct error tuple
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
2023-07-13 16:12:33 -03:00
zhongwencool b5cc8fb3c3 fix: start_after_created's default value 2023-07-07 16:39:26 +08:00
Stefan Strigler 07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
Stefan Strigler 321fd53132 fix: use ReplyTo in QUERY for async 2023-06-29 16:09:45 +02:00
Stefan Strigler 40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler 1363108678 fix: fix simple_sync_query 2023-06-29 16:09:45 +02:00
Stefan Strigler 2274a192cc fix(emqx_resource): call async reply fun in simple_aysnc_query 2023-06-29 16:09:45 +02:00
Stefan Strigler ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi 30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
Thales Macedo Garitezi c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi 2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
Thales Macedo Garitezi 7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
Paulo Zulato 62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
Andrew Mayorov 62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00