Commit Graph

149 Commits

Author SHA1 Message Date
Thales Macedo Garitezi fd2940cd77 feat(pulsar): ensure allocated resources are removed on failures (v5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9937
2023-05-24 12:29:00 -03:00
Kjell Winblad 70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00
Andrew Mayorov 4575167607
feat(resource): drop `manager_id()` type 2023-05-02 17:29:20 +03:00
Andrew Mayorov aaef95b1da
feat(resman): stop adding uniqueness to manager ids
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
2023-05-02 17:28:26 +03:00
Andrew Mayorov 9c9f39d0f7
feat(resman): also move out metrics collection for debugging
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
2023-04-12 16:14:42 +03:00
Andrew Mayorov e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
zhongwencool d63680cf25
Merge pull request #10307 from emqx/release-50
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Thales Macedo Garitezi 5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
Thales Macedo Garitezi 632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
Thales Macedo Garitezi f8d5d53908 feat(buffer_worker): decouple query mode from underlying connector call mode
Fixes https://emqx.atlassian.net/browse/EMQX-9129

Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.

Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:

1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
Andrew Mayorov a9bc8a4464
refactor(resman): rename `ets_lookup` → `lookup_cached`
That way we hide the impementation details + the interface becomes
cleaner and more obvious.
2023-03-15 19:17:30 +03:00
Andrew Mayorov cad6492c99
perf(bridge-api): ask bridge listings in parallel
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
Andrew Mayorov 686bf8255b
fix(bridge): reply `emqx_resource:get_instance/1` from cache
The resource manager may be busy at times, so this change ensures that
getting resource instance state will not block. Currently, no users of
`emqx_resource:get_instance/1` do seem to be relying on state being
"as-actual-as-possible" guarantee it was providing.
2023-03-13 14:35:08 +03:00
Zaiming (Stone) Shi 9864587389 fix: send to buffer-supported connector even when disconnected 2023-02-02 12:04:17 +01:00
Zaiming (Stone) Shi 5fdf7fd24c fix(kafka): use async callback to bump success counters
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
Thales Macedo Garitezi 47f796dd12 refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
Thales Macedo Garitezi 087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Erik Timan b9d012e072 refactor(emqx_resource): ingress bridge counter
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Shawn b45f3de8db refactor(resource): rename metrics batched,queued -> batching,queuing 2022-09-02 12:41:14 +08:00
Shawn 73e19d84ee feat: use the new metrics to bridge APIs 2022-08-30 23:47:58 +08:00
Shawn a896aa8b27 fix: incorrect replayq dir for the emqx_resource 2022-08-25 16:06:18 +08:00
Shawn 86577365e4 fix: use gen_statem:cast/3 for async query 2022-08-23 22:41:45 +08:00
Shawn de3a325953 fix: revert the changes in connector mysql 2022-08-16 09:06:13 +08:00
Xinyu Liu 2898966439
Merge branch 'dev/ee5.0' into resource_opts 2022-08-15 21:43:22 +08:00
Shawn 19d85d485b refactor(resource): add resource_opts level into config structure 2022-08-15 21:40:10 +08:00
JimMoen 68946f1f6c feat: influxdb support `async`/`batch_async` query 2022-08-15 14:02:17 +08:00
JimMoen 594d071c05 feat(influxdb): add async callback 2022-08-12 18:26:47 +08:00
Shawn 0cdf4b47f1 feat: add more resource creation opts 2022-08-12 13:47:45 +08:00
JimMoen 3a76a50382 fix: syntax error and compile error 2022-08-11 20:58:43 +08:00
Shawn 2872f0b668 fix(bridges): support create resources with options 2022-08-11 19:11:44 +08:00
JimMoen 0f6c371760 feat(influxdb): influxdb connector add `on_batch_query/3` callback 2022-08-11 18:12:41 +08:00
Shawn 6203a01320 feat: add inflight window to emqx_resource 2022-08-11 08:36:35 +08:00
Shawn 82550a585a fix: add test cases for query async 2022-08-10 00:45:34 +08:00
Shawn 145ff66a9a fix: issues found by dialyzer and elvis 2022-08-10 00:45:26 +08:00
Shawn 35fe70b887 feat: support aysnc callback to connector modules 2022-08-10 00:34:35 +08:00
Shawn 0377d3cf61 fix: update existing testcases for new emqx_resource 2022-08-10 00:34:35 +08:00
Shawn 2fb42e4d37 refactor: create emqx_resource_worker_sup for resource workers 2022-08-10 00:34:35 +08:00
Shawn 12904d797f feat(resource): first commit for batching/async/caching mechanism 2022-08-10 00:34:35 +08:00
Shawn d6ef2f7502 refactor: graceful recreate resources 2022-06-17 05:29:18 +08:00
Shawn cc25f92273 feat: add start_after_created option to resource:create/4 2022-06-16 23:34:52 +08:00
Zaiming (Stone) Shi 2065be569e fix(emqx_cluster_rpc): fail fast on stale state
Due to:

* Cluster RPC MFA is not idempotent!
* There is a lack of rollback for callback's side-effects

For instance, when two nodes try to add a cluster-singleton
concurrently, one of them will have to wait for the table lock
then try to catch-up, then try to apply MFA.
The catch-up will have the singleton created, but the initiated
initiated multicall apply will fail causing the commit to rollback,
but not to 'undo' the singleton creation.
Later, the retries will fail indefinitely.
2022-06-12 20:18:48 +02:00
Shawn 88ca25c60c fix(resource): fast return when starting a unavailable resource 2022-06-01 08:24:53 +08:00
Shawn d37a66e9b8 fix(test): update test cases for emqx_resource:health_check/1 2022-05-31 10:14:37 +08:00
Shawn 1054c364ad refactor(resource): improve health check and alarm it if resource down 2022-05-31 01:40:40 +08:00
JimMoen a5ddc5390f refactor(resource): add resource recreate fun with empty opts 2022-05-12 14:19:56 +08:00
Chris 0b3e30e813 feat: isolate resource manager processes 2022-05-09 13:24:34 +02:00
Zaiming (Stone) Shi 4e65322667 refactor: move emqx_plugin_libs_metrics to emqx app
because it can not depend on other apps
2022-04-29 12:41:36 +08:00
DDDHuang 132b37813c refactor: code format emqx_connector emqx_resource 2022-04-28 15:32:47 +08:00
DDDHuang 2a2308bbf8 refactor: resource check & connector status 2022-04-28 15:32:35 +08:00
Zaiming (Stone) Shi 02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
Zaiming (Stone) Shi f42a5b90df Revert "feat: isolate resource manager processes"
This reverts commit 40cca58d4f.
2022-04-26 16:13:38 +02:00
Chris 40cca58d4f feat: isolate resource manager processes 2022-04-26 13:28:29 +02:00
Shawn 278e9145b0 fix: go to different resource instance when health check 2022-04-19 23:00:34 +08:00
EMQ-YangM 8f06a9ec62 feat: impl resource reset_metrics 2022-04-11 10:25:48 +08:00
EMQ-YangM 9a2d70f98e fix(emqx_resource): remove extra space 2022-03-25 18:26:18 +08:00
EMQ-YangM 6b662d87ba fix(emqx_resource): fix dialyzer warning 2022-03-25 18:15:23 +08:00
zhongwencool 3414e0b601 feat(plugin): http api 2022-03-11 15:55:02 +08:00
EMQ-YangM 583624fb8d fix(emqx_authn): fix test suite 2022-03-09 13:53:05 +08:00
EMQ-YangM f29877bb6a fix(emqx_resource): remove create_opts async_create 2022-03-08 14:09:39 +08:00
EMQ-YangM 376c9ee261 refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting' 2022-02-25 15:02:41 +08:00
EMQ-YangM df57daaabb refactor(emqx_resource): improve grouping strategy for emqx_resource_instance 2022-02-11 18:36:55 +08:00
Chris c2772fdd09 refactor: remove unused/old functions 2022-02-09 10:14:13 +01:00
Zaiming (Stone) Shi f7703c906e refactor: move plain_check impl to emqx_hocon 2022-01-29 10:16:55 +01:00
Zaiming (Stone) Shi d6f7ffec3c
Merge pull request #6881 from zmstone/refactor-upgrade-hocon-0.23.0
refactor: upgrade to hocon 0.23.0
2022-01-27 15:48:23 +01:00
Kian-Meng Ang fc92e4c8bf docs: fix typos 2022-01-27 12:29:38 +01:00
Zaiming (Stone) Shi cf1f19258e refactor(emqx_resource): catch only hocon throw exceptions 2022-01-27 00:23:08 +01:00
Zaiming (Stone) Shi 252d7e85d9 refactor: call new hocon apis 2022-01-27 00:23:08 +01:00
Yang Miao b528862c67
Merge branch 'master' into health_check_timeout 2022-01-24 14:48:55 +08:00
k32 542c9a2b68 refactor(emqx_resource): Decorate RPCs 2022-01-19 22:00:34 +01:00
EMQ-YangM fd7e0c800a feat(emqx_resource_health_check): add timeout to single health_check 2022-01-19 16:45:56 +08:00
Shawn 67a60e1153 refactor(rule): add more metrics for rule and bridges 2022-01-07 09:34:54 +08:00
Shawn 78aa0abd3d fix(metrics): update the calls to emqx_plugin_libs_metrcis module 2022-01-07 09:34:48 +08:00
Zaiming (Stone) Shi 63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
Shawn 2277b75b2f refactor(resource): improve the process starting/stopping resource instances 2022-01-02 17:14:02 +08:00
Shawn 657ecef67b fix(resource): don't crash on resource stopped 2021-12-31 20:57:34 +08:00
Shawn a879ec0f3a feat(resource): add option 'force_create' to emqx_resource:create/4 2021-12-20 10:26:27 +08:00
Shawn 11e8e0db69 fix(bridge): stop http failed due to econnrefused 2021-12-18 15:33:25 +08:00
Shawn 29ad6d215e feat(resource): add metrics to emqx_resource 2021-11-23 10:04:31 +08:00
Ilya Averyanov 071c2c99e8 refactor(authn resources): add `emqx_resource` and `emqx_authn` tests 2021-11-22 21:08:04 +03:00
Shawn 9c93ea0338 feat(connector): add API for /connectors_test 2021-11-22 17:35:33 +08:00
zhanghongtong 2307bdd868 fix(emqx_resource): fix InstId type error 2021-10-25 13:43:28 +08:00
zhouzb ddfc010fdb fix(resource): fix undefined function 2021-09-16 16:53:43 +08:00
Shawn 304c5613ac refactor(resource): rename the emqx_resource:update/4 to recreate/4 2021-09-15 17:46:42 +08:00
zhongwencool c1c24af002 fix: dialyzer warning 2021-08-27 10:09:26 +08:00
zhongwencool 73238ed81f feat: emqx_resource support cluster_call 2021-08-26 17:23:39 +08:00
Shawn 8978464269
change resource,connectors,data_bridges as normal apps (#5034) 2021-06-19 16:27:21 +08:00
turtleDeng 0515ef6e45
feat(connector): add more connector (#4985) 2021-06-16 11:28:26 +08:00
Shawn 1de2f5cb2f fix(emqx_data_bridge): update bridge failed 2021-06-08 21:46:52 +08:00
Shawn 65dc2f2be8 feat(emqx_resource): remove the schema sugar from parse transformed code 2021-06-08 21:46:52 +08:00
Shawn 3c1c457697 fix(emqx_resource): call to undefined function hocon:richmap_to_map/1 2021-06-07 21:54:17 +08:00
Shawn 4914b003ac feat(emqx_resource): update the unused APIs 2021-06-07 17:20:34 +08:00
Shawn 2ff92d2880 feat(emqx_data_bridge): add HTTP APIs for data bridge 2021-06-07 16:12:06 +08:00
Shawn f1552f4f4f feat(emqx_data_bridge): create emqx_data_bridge 2021-06-04 23:47:16 +08:00
Shawn 3da62e59d6 fix(emqx_connector): the HTTP API for emqx_connector_mysql 2021-06-03 01:24:30 +08:00
Shawn 5d52ce044d feat(emqx_resource): read and save configs from and to file 2021-06-02 01:58:32 +08:00
Shawn e7ffa07a1a feat(emqx_connector): load connectors from emqx_connector.conf 2021-05-31 22:49:42 +08:00
Shawn 2a31c43e0d feat(emqx_resource): add behaviour emqx_resource 2021-05-28 21:43:54 +08:00