8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
85a8eff90b
fix(emqx_resource_manager): do not start when disabled
2023-01-11 08:33:48 +01:00
70eb5ffb58
refactor: remove unused function
2023-01-05 10:16:01 -03:00
56437228dc
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections.
2023-01-05 10:16:01 -03:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
bf3983e7c4
feat(buffer_worker): use offload mode for `replayq`
...
To avoid confusion for the users as to what persistence guarantees we
offer when buffering bridges/resources, we will always enable offload
mode for `replayq`. With this, when the buffer size is above the max
segment size, it'll flush the queue to disk, but on recovery after a
restart it'll clean the existing segments rather than resuming from
them.
2023-01-05 10:11:59 -03:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
305ed68916
chore: bump app vsns
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
f93c22045d
fix: non-empty field should not be undefined
2022-12-24 11:41:45 +01:00
479e191dcf
refactor: refine worker pool config and doc
...
worker pool is a buffer pool
the description hinted connection pool which is wrong.
2022-12-20 09:02:51 +01:00
f611cbab45
chore: cap replayq seg size under total size
2022-12-19 23:16:05 +01:00
8a0ca38a77
fix: drop no longer supported dialyzer option
2022-12-16 13:45:05 +03:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
34e9056779
refactor: fix typo in variable name
...
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
62eeb4b8e8
feat(resource): reset metrics when stopping a resource
2022-10-18 09:32:35 -03:00
2d01726b22
fix: account calls when resource is not connected as matched
2022-10-13 15:32:04 -03:00
1b2b629cdd
feat: emit telemetry events for all resource worker metrics
2022-10-13 15:32:04 -03:00
f0ff32c031
test: fix tests after counter changes
2022-10-11 17:45:48 -03:00
357e5919ce
chore: add copyright disclaimer
2022-10-11 09:51:16 -03:00
57270fb8fc
feat: add support for counters and gauges to the Kafka Bridge
...
This commit adds support for counters and gauges to the Kafka Brige.
The Kafka bridge uses [Wolff](https://github.com/kafka4beam/wolff ) for
the Kafka connection. Wolff does its own batching and does not use the
batching functionality in `emqx_resource_worker` that is used by other
bridge types. Therefore, the counter events have to be generated by
Wolff. We have added
[telemetry](https://github.com/beam-telemetry/telemetry ) events to Wolff
that we hook into to change counters and gauges for the Kafka bridge. The
counter called `matched` does not depend on specific functionality of
any bridge type so the updates of this counter is moved higher up in the
call chain then previously so that it also gets updated for Kafka
bridges.
2022-10-10 14:40:57 -03:00
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
...
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
b325633390
refactor(resource): resume from queue/inflight-window with async-sending and batching
2022-09-21 22:58:47 +08:00
9aa7e826cb
refactor(resource): fast resume resource worker if inflight msgs are ACKed
2022-09-17 00:34:30 +08:00
8307f04c2e
refactor(resource): save inflight size into the ETS table
2022-09-16 16:52:08 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
4e211c12d3
fix(mqtt_bridge): return value of sending messages was discarded
2022-09-15 08:57:01 +08:00
1c03c236f5
fix(mqtt_bridge): handle send_to_remote in idle state
2022-09-14 15:19:30 +08:00
f41adb0997
refactor: change some default values of resource_opts
2022-09-14 15:18:07 +08:00
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
b9ae4ea276
refactor: rename some metrics for emqx_resource
2022-09-13 14:04:25 +08:00
2b33ca6d49
fix: no error log print if insert bool values into mysql
2022-09-07 16:00:09 +08:00
26234d38b9
fix: mark the async msg 'queuing' not 'sent.inflight' on recoverable_error
2022-09-02 18:41:43 +08:00
83f21b4c65
refactor(resource): remove metrics 'sent.exception'
2022-09-02 12:46:53 +08:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
33c9c7d497
fix: incorrect message order when batch is enabled
2022-09-01 14:51:13 +08:00
0ef0b68de4
refactor: change '{recoverable_error,Reason}' to '{error,{recoverable_error,Reason}}'
2022-08-31 18:25:00 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
9e50866cd0
fix: rename queue_max_bytes -> max_queue_bytes
2022-08-30 17:18:54 +08:00
c4106c0d77
fix: resume the resource worker on health check success
2022-08-30 12:28:43 +08:00
6fde37791c
refactor: new metrics for resources
2022-08-30 10:14:10 +08:00
1625b8eaeb
fix(mysql_bridge): export the query_mode option to the APIs
2022-08-26 17:11:24 +08:00
6b0ccfbc43
refactor: rename the error return resource_down -> recoverable_error
2022-08-26 17:11:12 +08:00
a896aa8b27
fix: incorrect replayq dir for the emqx_resource
2022-08-25 16:06:18 +08:00
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
f0c2b53868
fix(bpapi): make bpapi static_checks happy
2022-08-22 10:51:44 +08:00
62ecf6f545
fix(resource): keep `auto_retry` in `disconnected` state
...
Automatic retries should be maintained even in `disconnected` state without any state transition.
2022-08-22 02:52:06 +08:00
7c4ea38c06
fix(resource): make some resource opts internal
...
Resource options `start_after_created` and `start_timeout` are internal opts.
Not provided to users anymore.
2022-08-22 02:22:57 +08:00
06363e63d9
fix(influxdb): connector use a fallbacke `pool_size` for influxdb client
2022-08-19 15:54:19 +08:00
9e35032d78
fix: make resume_interval defaults to health_check_interval
2022-08-16 10:09:02 +08:00
de3a325953
fix: revert the changes in connector mysql
2022-08-16 09:06:13 +08:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
d1de262f31
fix: inc 'actions.failed' if bridge query failed
2022-08-15 17:21:14 +08:00
665ef4142d
fix: unify the health check interval
2022-08-15 17:21:14 +08:00
68946f1f6c
feat: influxdb support `async`/`batch_async` query
2022-08-15 14:02:17 +08:00
b01ae8ece6
chore: refine influxdb bridge/connector i18n
2022-08-15 14:00:14 +08:00
594d071c05
feat(influxdb): add async callback
2022-08-12 18:26:47 +08:00
fa5e8f1422
chore: refine i18n label
2022-08-12 16:39:03 +08:00
3678673124
fix: schema default value using raw type before convert
2022-08-12 16:38:46 +08:00
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
c3c4ed02b4
fix: bump emqx_dashboard to 5.0.4
2022-08-12 00:24:58 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
0f6c371760
feat(influxdb): influxdb connector add `on_batch_query/3` callback
2022-08-11 18:12:41 +08:00
22a4ca311c
feat(resource): resource batch/async/queue config schema
2022-08-11 16:59:18 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
efd6c56dd9
fix: test cases for batch query sync
2022-08-10 00:45:34 +08:00
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
f1419d52f1
fix(resource): remove resource at the end of each test
2022-08-10 00:34:35 +08:00
a2afdeeb48
feat: add test cases for batching query
2022-08-10 00:34:35 +08:00
75adba0781
fix: increase resource metrics using the resource id
2022-08-10 00:34:35 +08:00
d3950b9534
fix(resource): make option 'queue_enabled' disabled by default
2022-08-10 00:34:35 +08:00
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
0087b7c960
fix: remove the extra file replay.erl
2022-08-10 00:34:35 +08:00
d8d8d674e4
feat(resource): start emqx_resource_worker in pools
2022-08-10 00:34:35 +08:00
12904d797f
feat(resource): first commit for batching/async/caching mechanism
2022-08-10 00:34:35 +08:00
98b36c4681
fix: hstream db connector , TODO: start apps
2022-07-27 11:38:45 +08:00
a78a389206
chore: using standard log format
2022-07-01 12:06:35 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
2065be569e
fix(emqx_cluster_rpc): fail fast on stale state
...
Due to:
* Cluster RPC MFA is not idempotent!
* There is a lack of rollback for callback's side-effects
For instance, when two nodes try to add a cluster-singleton
concurrently, one of them will have to wait for the table lock
then try to catch-up, then try to apply MFA.
The catch-up will have the singleton created, but the initiated
initiated multicall apply will fail causing the commit to rollback,
but not to 'undo' the singleton creation.
Later, the retries will fail indefinitely.
2022-06-12 20:18:48 +02:00
b7f27157e5
fix: also alarm resource down when start resource failed
2022-06-01 15:41:55 +08:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
9f69e3cad6
fix(resource): discard dry_run resource down alarm
2022-06-01 08:24:53 +08:00
d37a66e9b8
fix(test): update test cases for emqx_resource:health_check/1
2022-05-31 10:14:37 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
574a40b327
fix: wait for test_resource stop
2022-05-16 17:00:42 +08:00
b5addf7e05
fix: log all ignore events
2022-05-16 15:08:03 +08:00
bbbfea1b5b
fix: ignore all other events
2022-05-16 15:08:03 +08:00
1a1c82932a
fix: when connecting health check failed, update status.
2022-05-16 10:47:20 +08:00
93799e3ac6
refactor: delete now unused emqx_resource modules
2022-05-16 09:54:26 +08:00
c4fd31ae25
Merge pull request #7916 from emqx/EMQX-4204-auto-timer-based-retry-when-in-disconnected-state
...
feat: add auto_retry for disconnected state in resource manager
2022-05-16 09:34:08 +08:00
3f59650e4b
Merge pull request #7944 from EMQ-YangM/fix_bridge_status
...
fix: restart resource should not clear metrics
2022-05-16 09:16:12 +08:00
Stefan Strigler