yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,386 Commits 393 Branches 684 Tags 74 MiB
Commit Graph

628 Commits

Author SHA1 Message Date
Kjell Winblad ef705c2285 feat: add apply rule API, clientid/ruleid tracing for rule and connector 
This commit adds:

* Support for forwarding the rule id and client id to the connector so
  that events such as template rendered successfully can be traced.
* HTTP API for for applying/activating a rule with the given context
 2024-04-06 17:20:47 +02:00
Thales Macedo Garitezi 60cad74286 feat(resource): non-blocking channel health checks 
Fixes https://emqx.atlassian.net/browse/EMQX-12015

Continuation of https://github.com/emqx/emqx/pull/12812
 2024-04-04 16:13:30 -03:00
Thales Macedo Garitezi bade09b56e feat(resource manager): perform non-blocking resource health checks 
Fixes https://emqx.atlassian.net/browse/EMQX-12015

This introduces only _resource_ non-blocking health checks.  _Channel_ non-blocking health
checks may be introduced later.
 2024-04-01 14:46:15 -03:00
JimMoen 9143d5994d
feat: refactor MS SQL Server bridge to connector and action 2024-03-04 17:13:21 +08:00
zhongwencool 54c542c795 chore: rename _probe_ to t_probe_ 2024-02-27 17:57:15 +08:00
zhongwencool c67f2130f2 fix: check connector and bridge_v2 with the right schema 2024-02-27 17:57:10 +08:00
Thales Macedo Garitezi 15f919e60f
Merge pull request #12564 from thalesmg/bw-support-batch-list-resp-m-20240221 
feat(resource): allow `on_batch_query{,_async}` to return a list of individual results
 2024-02-23 09:37:42 -03:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Thales Macedo Garitezi d003f77021 feat(resource): allow `on_batch_query{,_async}` to return a list of individual results 
Fixes https://emqx.atlassian.net/browse/EMQX-11892

This allows callers of batching resources to receive results specific to their requests,
rather than a broad success or failure for the whole batch.
 2024-02-22 16:26:02 -03:00
Zaiming (Stone) Shi 415d27a4c9
Merge pull request #12544 from zmstone/0220-delete-non-prod-code 
0220 delete non prod code in emqx_resource.erl
 2024-02-21 14:08:08 +01:00
Thales Macedo Garitezi 4118fcfdc6
Merge pull request #12486 from thalesmg/gcp-consumer-source-m-20240205 
feat: convert `gcp_pubsub_consumer` to connector/source
 2024-02-21 10:06:26 -03:00
Zaiming (Stone) Shi b14e51d7c4 docs: delete useless comments 2024-02-21 13:33:29 +01:00
Zaiming (Stone) Shi c8f9e12de1 refactor: deprecate emqx_resource_proto_v1 with v2 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 06e0a7f2ed refactor: delete emqx_resource:remove api 
this api is only used in tests,
changed to call the _local flavor instead
 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 8541d2efc1 refactor: delete emqx_resource:recreate 
this api is only used in tests, changed to call the _local flavor
instead
 2024-02-21 10:53:22 +01:00
Zaiming (Stone) Shi 1878fded5a refactor: delete emqx_resource:create/4 and /5 
the functions are only used in tests,
now the tests are rewritten to call create_local/4 and /5 instead
 2024-02-21 10:53:22 +01:00
Thales Macedo Garitezi d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
Thales Macedo Garitezi 9a32895a1a feat: convert `gcp_pubsub_consumer` to connector/source 
Fixes https://emqx.atlassian.net/browse/EMQX-11471
 2024-02-19 14:42:35 -03:00
zhongwencool 42a4bc3382
Merge pull request #12435 from zhongwencool/dont-add-disable-resource-to-connector 
fix: don't add disable bridge to connector's channel
 2024-02-01 10:39:47 +08:00
zhongwencool f9d16340e0 fix: don't add disable bridge to connector's channel 2024-02-01 09:12:43 +08:00
zhongwencool e9c8446d57 feat: upgrade erlfmt to support maybe syntax 2024-01-31 20:06:49 +08:00
Zaiming (Stone) Shi 55fdc83aa5 Merge remote-tracking branch 'origin/release-55' into 0130-syn-release-55-to-master 2024-01-30 12:06:43 +01:00
Thales Macedo Garitezi 6410f5a717 fix(resource_metrics): avoid detaching handler on crashes 
Fixes https://emqx.atlassian.net/browse/EMQX-11821
 2024-01-26 13:05:41 -03:00
Kjell Winblad 5bf37cbe0a
Merge pull request #12376 from kjellwinblad/kjell/kinesis 
feat: refactor kinesis bridge to connector and action
 2024-01-25 14:58:40 +01:00
Kjell Winblad fa842736d2 feat: refactor kinesis bridge to connector and action 
Fixes:
https://emqx.atlassian.net/browse/EMQX-11461
 2024-01-24 18:31:03 +01:00
firest 51e8c955ff fix(connector): make the connector id unique when doing probe testing 2024-01-23 13:03:12 +08:00
firest d221f34ce8 chore: bump version && update changes 2024-01-06 13:34:07 +08:00
firest d16458ccd0 feat(iotdb): improve the IoTDB bridge to v2 style 2024-01-05 16:54:31 +08:00
Thales Macedo Garitezi 432ddc5a3b fix(bridge_v1_schema): undo changing v1 schema to avoid confusion 2023-12-15 12:55:34 -03:00
Thales Macedo Garitezi a7fe5da8d6 fix(connector_api): avoid calling resource process to get channels 
Fixes https://emqx.atlassian.net/browse/EMQX-11586
 2023-12-14 10:22:46 -03:00
Thales Macedo Garitezi b4a5c141ad fix(actions): use action query mode instead of connector's query mode 2023-12-11 14:10:51 -03:00
Thales Macedo Garitezi 2495f59c91 fix(actions): increment rule statistics even if channel is not installed 
Fixes new bug posted after https://emqx.atlassian.net/browse/EMQX-11494 was already fixed.

Also reduces the usage of error throwing for flow control a bit.
 2023-12-11 14:01:30 -03:00
Zaiming (Stone) Shi 423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Thales Macedo Garitezi c1c7d1246f
Merge pull request #12084 from thalesmg/refactor-resource-status-state-m-20231201 
refactor(resource_manager): use macros and better differentiate status from state
 2023-12-04 17:55:27 -03:00
Thales Macedo Garitezi d594b38ceb
Merge pull request #12083 from thalesmg/fix-connector-bws-m-20231201 
fix(connector): don't start buffer workers for the connector itself
 2023-12-04 16:14:15 -03:00
Thales Macedo Garitezi 10b1a8bbda feat(connector): add some `resource_opts` schema facilities 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi dc5e3b939c refactor(resource_manager): use macros and better differentiate status from state 
Internally in `emqx_resource_manager`, there seems to be many points where the
`gen_statem` states are conflated with resource status, since their names coincide.  While
that works for now, introducing a new `gen_statem` state, an internal state, shouldn't
necessarily imply a new, externally facing resource status.

Here we also introduce the usage of some macros to avoid the pitfalls of making a typo in
a state/status name.
 2023-12-01 18:23:05 -03:00
Thales Macedo Garitezi 29ae45c39d fix(connector): don't start buffer workers for the connector itself 
Fixes https://emqx.atlassian.net/browse/EMQX-11448
 2023-12-01 17:20:38 -03:00
Thales Macedo Garitezi cb60880bab
Merge pull request #12067 from thalesmg/fix-kafka-check-client-m-20231130 
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
 2023-12-01 09:55:12 -03:00
Thales Macedo Garitezi 0388e1c1c4 fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity 
Fixes https://emqx.atlassian.net/browse/EMQX-11494
 2023-11-30 17:51:32 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Ivan Dyachkov ec10c51073 Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53 2023-11-30 19:51:12 +01:00
Zaiming (Stone) Shi 14644988e0 chore: change triple-quotes to single-quotes 2023-11-29 16:15:18 +01:00
JianBo He c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Zaiming (Stone) Shi 3261a12140 fix(emqx_resource): do not allow leading _ or - as resource name 2023-11-22 10:58:54 +01:00
Thales Macedo Garitezi 9e1796ec4f feat(gcp_pubsub_producer): migrate GCP PubSub producer to actions 
Fixes https://emqx.atlassian.net/browse/EMQX-11157
 2023-11-21 14:22:42 -03:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect 
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
 2023-11-16 14:50:23 -03:00
Thales Macedo Garitezi d6e9bbb95c fix(connector): validate connector name before converting ssl certs 
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
 2023-11-14 09:29:59 -03:00
Ivan Dyachkov a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Thales Macedo Garitezi b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106 
fix(resource): take error from action/connector before attempting query
 2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi 7dcdbc9e51 fix(resource): take error from action/connector before attempting query 
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
 2023-11-07 10:04:04 -03:00
Thales Macedo Garitezi 2b8cf50a1d chore: rename `bridges_v2` -> `actions` in the public facing APIs 
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
 2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi 600747b7e5 fix(bridge): do not allow dot in bridge name 
also validate name at the API entry
 2023-11-03 20:44:57 +01:00
Kjell Winblad 357b664c8d fix(bridge_v2): more fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:54 +01:00
Kjell Winblad 96d6c6db49 test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change 2023-11-01 15:27:53 +01:00
Kjell Winblad edb1d37e67 chore(bridge_v2): make fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:53 +01:00
Kjell Winblad 95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting 
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
 2023-11-01 15:27:53 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
 2023-10-30 14:48:47 +01:00
Thales Macedo Garitezi cf2075d7d8 chore: remove mention of `is_buffer_supported` from typespec 2023-10-10 09:49:18 -03:00
Thales Macedo Garitezi d6781efee2 fix(resource): change how buffer workers are started 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 902b1d6ec5 fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar 
Since it has internal buffering, it necessitates the same fix as Kafka producer.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi eebfb44f72 fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering 
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers 
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state 
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
 2023-10-06 11:43:29 -03:00
Zaiming (Stone) Shi c2d750aa09 fix(resource): redact query args in exception log 2023-09-29 09:20:42 +02:00
firest dca8fdb17f fix(resource): respect the start_timeout 2023-09-28 16:36:41 +08:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource 
fix: always return ok when remove local resource
 2023-09-21 09:21:45 +08:00
zhongwencool dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
zhongwencool c26a18e949 fix: always return ok when remove local resource 2023-09-20 18:02:42 +08:00
Paulo Zulato dfcede8794 fix: increment matched counter when bridge is unhealthy 
Fixes https://emqx.atlassian.net/browse/EMQX-10767
 2023-08-30 10:52:53 -03:00
Paulo Zulato cc3ba18734 fix: increment dropped message counter when bridge is unhealthy 
Fixes https://emqx.atlassian.net/browse/EMQX-10767
 2023-08-28 19:47:11 -03:00
Paulo Zulato 42877e282d fix: flatten error message on resource validator 
Fixes https://emqx.atlassian.net/browse/EMQX-10864
 2023-08-25 13:53:52 -03:00
Thales Macedo Garitezi ebecbd1545 fix(bridge): make dryrun health check timeout more malleable 
Fixes https://emqx.atlassian.net/browse/EMQX-10773

- Makes the timeout for probing a bridge more malleable to account for differences between
  each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
  slowness/throttling.
 2023-08-17 09:21:19 -03:00
Ivan Dyachkov cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi 3b1e436d3f refactor: use `emqx_pool:async_submit` to avoid excessive spawning 2023-07-20 10:17:43 -03:00
Thales Macedo Garitezi eb41b77de4 fix(rule_metrics): notify rule metrics of late replies and expired requests 
Fixes https://emqx.atlassian.net/browse/EMQX-10600
 2023-07-19 11:39:28 -03:00
Thales Macedo Garitezi 01b143c5ad fix(resource): don't destruct error tuple 
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
 2023-07-13 16:12:33 -03:00
zhongwencool b5cc8fb3c3 fix: start_after_created's default value 2023-07-07 16:39:26 +08:00
Stefan Strigler 07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics 
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
 2023-06-30 20:07:23 +02:00
Stefan Strigler 321fd53132 fix: use ReplyTo in QUERY for async 2023-06-29 16:09:45 +02:00
Stefan Strigler 40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler 1363108678 fix: fix simple_sync_query 2023-06-29 16:09:45 +02:00
Stefan Strigler 2274a192cc fix(emqx_resource): call async reply fun in simple_aysnc_query 2023-06-29 16:09:45 +02:00
Stefan Strigler ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi 30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10309
 2023-06-28 14:08:40 -03:00
Thales Macedo Garitezi c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi 2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master 
fix(mongo): return health check failure reason
 2023-06-22 09:30:34 -03:00
Thales Macedo Garitezi 7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer 
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
 2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
Paulo Zulato 62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable 
Data bridge target unavailable
 2023-06-21 18:19:23 -03:00
Andrew Mayorov 62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon 
chore: bump hocon to 0.39.10
 2023-06-21 23:13:35 +02:00
Andrew Mayorov 86d787eced
chore: bump hocon to 0.39.10 
Which comes with a fix for slightly more user-friendly validation error
messages.
 2023-06-21 21:25:43 +02:00
Thales Macedo Garitezi 18f0510353 fix(mongo): return health check failure reason 
Fixes https://emqx.atlassian.net/browse/EMQX-10335
 2023-06-21 15:09:37 -03:00
Thales Macedo Garitezi decfd6df2b feat(buffer_worker): log expired message count 
Fixes https://emqx.atlassian.net/browse/EMQX-10165

```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
 2023-06-21 14:38:51 -03:00
Zaiming (Stone) Shi 7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Thales Macedo Garitezi 1d791d7a8c fix(resource): validate maximum worker pool size 
Fixes https://emqx.atlassian.net/browse/EMQX-10297
 2023-06-20 14:26:42 -03:00