Thales Macedo Garitezi
fd2940cd77
feat(pulsar): ensure allocated resources are removed on failures (v5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9937
2023-05-24 12:29:00 -03:00
Kjell Winblad
70cf1533db
feat: add RabbitMQ bridge
2023-05-09 14:32:26 +02:00
Andrew Mayorov
4575167607
feat(resource): drop `manager_id()` type
2023-05-02 17:29:20 +03:00
Andrew Mayorov
aaef95b1da
feat(resman): stop adding uniqueness to manager ids
...
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
2023-05-02 17:28:26 +03:00
Andrew Mayorov
9c9f39d0f7
feat(resman): also move out metrics collection for debugging
...
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
2023-04-12 16:14:42 +03:00
Andrew Mayorov
e70deae1c3
feat(resource): ask for metrics only when needed
2023-04-11 12:00:19 +03:00
zhongwencool
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Thales Macedo Garitezi
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
Thales Macedo Garitezi
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
Thales Macedo Garitezi
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
Andrew Mayorov
a9bc8a4464
refactor(resman): rename `ets_lookup` → `lookup_cached`
...
That way we hide the impementation details + the interface becomes
cleaner and more obvious.
2023-03-15 19:17:30 +03:00
Andrew Mayorov
cad6492c99
perf(bridge-api): ask bridge listings in parallel
...
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
Andrew Mayorov
686bf8255b
fix(bridge): reply `emqx_resource:get_instance/1` from cache
...
The resource manager may be busy at times, so this change ensures that
getting resource instance state will not block. Currently, no users of
`emqx_resource:get_instance/1` do seem to be relying on state being
"as-actual-as-possible" guarantee it was providing.
2023-03-13 14:35:08 +03:00
Zaiming (Stone) Shi
9864587389
fix: send to buffer-supported connector even when disconnected
2023-02-02 12:04:17 +01:00
Zaiming (Stone) Shi
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
Thales Macedo Garitezi
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
Thales Macedo Garitezi
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
Erik Timan
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
Zaiming (Stone) Shi
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
Shawn
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
Shawn
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
Shawn
a896aa8b27
fix: incorrect replayq dir for the emqx_resource
2022-08-25 16:06:18 +08:00
Shawn
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
Shawn
de3a325953
fix: revert the changes in connector mysql
2022-08-16 09:06:13 +08:00
Xinyu Liu
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
Shawn
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
JimMoen
68946f1f6c
feat: influxdb support `async`/`batch_async` query
2022-08-15 14:02:17 +08:00
JimMoen
594d071c05
feat(influxdb): add async callback
2022-08-12 18:26:47 +08:00
Shawn
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
JimMoen
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
Shawn
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
JimMoen
0f6c371760
feat(influxdb): influxdb connector add `on_batch_query/3` callback
2022-08-11 18:12:41 +08:00
Shawn
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
Shawn
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
Shawn
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
Shawn
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
Shawn
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
Shawn
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
Shawn
12904d797f
feat(resource): first commit for batching/async/caching mechanism
2022-08-10 00:34:35 +08:00
Shawn
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
Shawn
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
Zaiming (Stone) Shi
2065be569e
fix(emqx_cluster_rpc): fail fast on stale state
...
Due to:
* Cluster RPC MFA is not idempotent!
* There is a lack of rollback for callback's side-effects
For instance, when two nodes try to add a cluster-singleton
concurrently, one of them will have to wait for the table lock
then try to catch-up, then try to apply MFA.
The catch-up will have the singleton created, but the initiated
initiated multicall apply will fail causing the commit to rollback,
but not to 'undo' the singleton creation.
Later, the retries will fail indefinitely.
2022-06-12 20:18:48 +02:00
Shawn
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
Shawn
d37a66e9b8
fix(test): update test cases for emqx_resource:health_check/1
2022-05-31 10:14:37 +08:00
Shawn
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
JimMoen
a5ddc5390f
refactor(resource): add resource recreate fun with empty opts
2022-05-12 14:19:56 +08:00
Chris
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
Zaiming (Stone) Shi
4e65322667
refactor: move emqx_plugin_libs_metrics to emqx app
...
because it can not depend on other apps
2022-04-29 12:41:36 +08:00
DDDHuang
132b37813c
refactor: code format emqx_connector emqx_resource
2022-04-28 15:32:47 +08:00
DDDHuang
2a2308bbf8
refactor: resource check & connector status
2022-04-28 15:32:35 +08:00
Zaiming (Stone) Shi
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
Zaiming (Stone) Shi
f42a5b90df
Revert "feat: isolate resource manager processes"
...
This reverts commit 40cca58d4f
.
2022-04-26 16:13:38 +02:00
Chris
40cca58d4f
feat: isolate resource manager processes
2022-04-26 13:28:29 +02:00
Shawn
278e9145b0
fix: go to different resource instance when health check
2022-04-19 23:00:34 +08:00
EMQ-YangM
8f06a9ec62
feat: impl resource reset_metrics
2022-04-11 10:25:48 +08:00
EMQ-YangM
9a2d70f98e
fix(emqx_resource): remove extra space
2022-03-25 18:26:18 +08:00
EMQ-YangM
6b662d87ba
fix(emqx_resource): fix dialyzer warning
2022-03-25 18:15:23 +08:00
zhongwencool
3414e0b601
feat(plugin): http api
2022-03-11 15:55:02 +08:00
EMQ-YangM
583624fb8d
fix(emqx_authn): fix test suite
2022-03-09 13:53:05 +08:00
EMQ-YangM
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
EMQ-YangM
376c9ee261
refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting'
2022-02-25 15:02:41 +08:00
EMQ-YangM
df57daaabb
refactor(emqx_resource): improve grouping strategy for emqx_resource_instance
2022-02-11 18:36:55 +08:00
Chris
c2772fdd09
refactor: remove unused/old functions
2022-02-09 10:14:13 +01:00
Zaiming (Stone) Shi
f7703c906e
refactor: move plain_check impl to emqx_hocon
2022-01-29 10:16:55 +01:00
Zaiming (Stone) Shi
d6f7ffec3c
Merge pull request #6881 from zmstone/refactor-upgrade-hocon-0.23.0
...
refactor: upgrade to hocon 0.23.0
2022-01-27 15:48:23 +01:00
Kian-Meng Ang
fc92e4c8bf
docs: fix typos
2022-01-27 12:29:38 +01:00
Zaiming (Stone) Shi
cf1f19258e
refactor(emqx_resource): catch only hocon throw exceptions
2022-01-27 00:23:08 +01:00
Zaiming (Stone) Shi
252d7e85d9
refactor: call new hocon apis
2022-01-27 00:23:08 +01:00
Yang Miao
b528862c67
Merge branch 'master' into health_check_timeout
2022-01-24 14:48:55 +08:00
k32
542c9a2b68
refactor(emqx_resource): Decorate RPCs
2022-01-19 22:00:34 +01:00
EMQ-YangM
fd7e0c800a
feat(emqx_resource_health_check): add timeout to single health_check
2022-01-19 16:45:56 +08:00
Shawn
67a60e1153
refactor(rule): add more metrics for rule and bridges
2022-01-07 09:34:54 +08:00
Shawn
78aa0abd3d
fix(metrics): update the calls to emqx_plugin_libs_metrcis module
2022-01-07 09:34:48 +08:00
Zaiming (Stone) Shi
63167cea70
chore: update copyright
2022-01-05 20:55:00 +01:00
Shawn
2277b75b2f
refactor(resource): improve the process starting/stopping resource instances
2022-01-02 17:14:02 +08:00
Shawn
657ecef67b
fix(resource): don't crash on resource stopped
2021-12-31 20:57:34 +08:00
Shawn
a879ec0f3a
feat(resource): add option 'force_create' to emqx_resource:create/4
2021-12-20 10:26:27 +08:00
Shawn
11e8e0db69
fix(bridge): stop http failed due to econnrefused
2021-12-18 15:33:25 +08:00
Shawn
29ad6d215e
feat(resource): add metrics to emqx_resource
2021-11-23 10:04:31 +08:00
Ilya Averyanov
071c2c99e8
refactor(authn resources): add `emqx_resource` and `emqx_authn` tests
2021-11-22 21:08:04 +03:00
Shawn
9c93ea0338
feat(connector): add API for /connectors_test
2021-11-22 17:35:33 +08:00
zhanghongtong
2307bdd868
fix(emqx_resource): fix InstId type error
2021-10-25 13:43:28 +08:00
zhouzb
ddfc010fdb
fix(resource): fix undefined function
2021-09-16 16:53:43 +08:00
Shawn
304c5613ac
refactor(resource): rename the emqx_resource:update/4 to recreate/4
2021-09-15 17:46:42 +08:00
zhongwencool
c1c24af002
fix: dialyzer warning
2021-08-27 10:09:26 +08:00
zhongwencool
73238ed81f
feat: emqx_resource support cluster_call
2021-08-26 17:23:39 +08:00
Shawn
8978464269
change resource,connectors,data_bridges as normal apps ( #5034 )
2021-06-19 16:27:21 +08:00
turtleDeng
0515ef6e45
feat(connector): add more connector ( #4985 )
2021-06-16 11:28:26 +08:00
Shawn
1de2f5cb2f
fix(emqx_data_bridge): update bridge failed
2021-06-08 21:46:52 +08:00
Shawn
65dc2f2be8
feat(emqx_resource): remove the schema sugar from parse transformed code
2021-06-08 21:46:52 +08:00
Shawn
3c1c457697
fix(emqx_resource): call to undefined function hocon:richmap_to_map/1
2021-06-07 21:54:17 +08:00
Shawn
4914b003ac
feat(emqx_resource): update the unused APIs
2021-06-07 17:20:34 +08:00
Shawn
2ff92d2880
feat(emqx_data_bridge): add HTTP APIs for data bridge
2021-06-07 16:12:06 +08:00
Shawn
f1552f4f4f
feat(emqx_data_bridge): create emqx_data_bridge
2021-06-04 23:47:16 +08:00
Shawn
3da62e59d6
fix(emqx_connector): the HTTP API for emqx_connector_mysql
2021-06-03 01:24:30 +08:00
Shawn
5d52ce044d
feat(emqx_resource): read and save configs from and to file
2021-06-02 01:58:32 +08:00
Shawn
e7ffa07a1a
feat(emqx_connector): load connectors from emqx_connector.conf
2021-05-31 22:49:42 +08:00
Shawn
2a31c43e0d
feat(emqx_resource): add behaviour emqx_resource
2021-05-28 21:43:54 +08:00