fd2940cd77
feat(pulsar): ensure allocated resources are removed on failures (v5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9937
2023-05-24 12:29:00 -03:00
70cf1533db
feat: add RabbitMQ bridge
2023-05-09 14:32:26 +02:00
4575167607
feat(resource): drop `manager_id()` type
2023-05-02 17:29:20 +03:00
aaef95b1da
feat(resman): stop adding uniqueness to manager ids
...
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
2023-05-02 17:28:26 +03:00
9c9f39d0f7
feat(resman): also move out metrics collection for debugging
...
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
2023-04-12 16:14:42 +03:00
e70deae1c3
feat(resource): ask for metrics only when needed
2023-04-11 12:00:19 +03:00
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
a9bc8a4464
refactor(resman): rename `ets_lookup` → `lookup_cached`
...
That way we hide the impementation details + the interface becomes
cleaner and more obvious.
2023-03-15 19:17:30 +03:00
cad6492c99
perf(bridge-api): ask bridge listings in parallel
...
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
686bf8255b
fix(bridge): reply `emqx_resource:get_instance/1` from cache
...
The resource manager may be busy at times, so this change ensures that
getting resource instance state will not block. Currently, no users of
`emqx_resource:get_instance/1` do seem to be relying on state being
"as-actual-as-possible" guarantee it was providing.
2023-03-13 14:35:08 +03:00
9864587389
fix: send to buffer-supported connector even when disconnected
2023-02-02 12:04:17 +01:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
a896aa8b27
fix: incorrect replayq dir for the emqx_resource
2022-08-25 16:06:18 +08:00
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
de3a325953
fix: revert the changes in connector mysql
2022-08-16 09:06:13 +08:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
68946f1f6c
feat: influxdb support `async`/`batch_async` query
2022-08-15 14:02:17 +08:00
594d071c05
feat(influxdb): add async callback
2022-08-12 18:26:47 +08:00
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
0f6c371760
feat(influxdb): influxdb connector add `on_batch_query/3` callback
2022-08-11 18:12:41 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
12904d797f
feat(resource): first commit for batching/async/caching mechanism
2022-08-10 00:34:35 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
2065be569e
fix(emqx_cluster_rpc): fail fast on stale state
...
Due to:
* Cluster RPC MFA is not idempotent!
* There is a lack of rollback for callback's side-effects
For instance, when two nodes try to add a cluster-singleton
concurrently, one of them will have to wait for the table lock
then try to catch-up, then try to apply MFA.
The catch-up will have the singleton created, but the initiated
initiated multicall apply will fail causing the commit to rollback,
but not to 'undo' the singleton creation.
Later, the retries will fail indefinitely.
2022-06-12 20:18:48 +02:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
d37a66e9b8
fix(test): update test cases for emqx_resource:health_check/1
2022-05-31 10:14:37 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
a5ddc5390f
refactor(resource): add resource recreate fun with empty opts
2022-05-12 14:19:56 +08:00
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
4e65322667
refactor: move emqx_plugin_libs_metrics to emqx app
...
because it can not depend on other apps
2022-04-29 12:41:36 +08:00
132b37813c
refactor: code format emqx_connector emqx_resource
2022-04-28 15:32:47 +08:00
2a2308bbf8
refactor: resource check & connector status
2022-04-28 15:32:35 +08:00
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
f42a5b90df
Revert "feat: isolate resource manager processes"
...
This reverts commit 40cca58d4f
2022-04-26 16:13:38 +02:00
40cca58d4f
feat: isolate resource manager processes
2022-04-26 13:28:29 +02:00
278e9145b0
fix: go to different resource instance when health check
2022-04-19 23:00:34 +08:00
8f06a9ec62
feat: impl resource reset_metrics
2022-04-11 10:25:48 +08:00
9a2d70f98e
fix(emqx_resource): remove extra space
2022-03-25 18:26:18 +08:00
6b662d87ba
fix(emqx_resource): fix dialyzer warning
2022-03-25 18:15:23 +08:00
3414e0b601
feat(plugin): http api
2022-03-11 15:55:02 +08:00
583624fb8d
fix(emqx_authn): fix test suite
2022-03-09 13:53:05 +08:00
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
376c9ee261
refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting'
2022-02-25 15:02:41 +08:00
df57daaabb
refactor(emqx_resource): improve grouping strategy for emqx_resource_instance
2022-02-11 18:36:55 +08:00
c2772fdd09
refactor: remove unused/old functions
2022-02-09 10:14:13 +01:00
f7703c906e
refactor: move plain_check impl to emqx_hocon
2022-01-29 10:16:55 +01:00
d6f7ffec3c
Merge pull request #6881 from zmstone/refactor-upgrade-hocon-0.23.0
...
refactor: upgrade to hocon 0.23.0
2022-01-27 15:48:23 +01:00
fc92e4c8bf
docs: fix typos
2022-01-27 12:29:38 +01:00
cf1f19258e
refactor(emqx_resource): catch only hocon throw exceptions
2022-01-27 00:23:08 +01:00
252d7e85d9
refactor: call new hocon apis
2022-01-27 00:23:08 +01:00
b528862c67
Merge branch 'master' into health_check_timeout
2022-01-24 14:48:55 +08:00
542c9a2b68
refactor(emqx_resource): Decorate RPCs
2022-01-19 22:00:34 +01:00
fd7e0c800a
feat(emqx_resource_health_check): add timeout to single health_check
2022-01-19 16:45:56 +08:00
67a60e1153
refactor(rule): add more metrics for rule and bridges
2022-01-07 09:34:54 +08:00
78aa0abd3d
fix(metrics): update the calls to emqx_plugin_libs_metrcis module
2022-01-07 09:34:48 +08:00
63167cea70
chore: update copyright
2022-01-05 20:55:00 +01:00
2277b75b2f
refactor(resource): improve the process starting/stopping resource instances
2022-01-02 17:14:02 +08:00
657ecef67b
fix(resource): don't crash on resource stopped
2021-12-31 20:57:34 +08:00
a879ec0f3a
feat(resource): add option 'force_create' to emqx_resource:create/4
2021-12-20 10:26:27 +08:00
11e8e0db69
fix(bridge): stop http failed due to econnrefused
2021-12-18 15:33:25 +08:00
29ad6d215e
feat(resource): add metrics to emqx_resource
2021-11-23 10:04:31 +08:00
071c2c99e8
refactor(authn resources): add `emqx_resource` and `emqx_authn` tests
2021-11-22 21:08:04 +03:00
9c93ea0338
feat(connector): add API for /connectors_test
2021-11-22 17:35:33 +08:00
2307bdd868
fix(emqx_resource): fix InstId type error
2021-10-25 13:43:28 +08:00
ddfc010fdb
fix(resource): fix undefined function
2021-09-16 16:53:43 +08:00
304c5613ac
refactor(resource): rename the emqx_resource:update/4 to recreate/4
2021-09-15 17:46:42 +08:00
c1c24af002
fix: dialyzer warning
2021-08-27 10:09:26 +08:00
73238ed81f
feat: emqx_resource support cluster_call
2021-08-26 17:23:39 +08:00
8978464269
change resource,connectors,data_bridges as normal apps ( #5034 )
2021-06-19 16:27:21 +08:00
0515ef6e45
feat(connector): add more connector ( #4985 )
2021-06-16 11:28:26 +08:00
1de2f5cb2f
fix(emqx_data_bridge): update bridge failed
2021-06-08 21:46:52 +08:00
65dc2f2be8
feat(emqx_resource): remove the schema sugar from parse transformed code
2021-06-08 21:46:52 +08:00
3c1c457697
fix(emqx_resource): call to undefined function hocon:richmap_to_map/1
2021-06-07 21:54:17 +08:00
4914b003ac
feat(emqx_resource): update the unused APIs
2021-06-07 17:20:34 +08:00
2ff92d2880
feat(emqx_data_bridge): add HTTP APIs for data bridge
2021-06-07 16:12:06 +08:00
f1552f4f4f
feat(emqx_data_bridge): create emqx_data_bridge
2021-06-04 23:47:16 +08:00
3da62e59d6
fix(emqx_connector): the HTTP API for emqx_connector_mysql
2021-06-03 01:24:30 +08:00
5d52ce044d
feat(emqx_resource): read and save configs from and to file
2021-06-02 01:58:32 +08:00
e7ffa07a1a
feat(emqx_connector): load connectors from emqx_connector.conf
2021-05-31 22:49:42 +08:00
2a31c43e0d
feat(emqx_resource): add behaviour emqx_resource
2021-05-28 21:43:54 +08:00
Thales Macedo Garitezi