734e6b9c96
chore: fix flaky test cases, log labels and review comments
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-01-13 11:05:02 +01:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
62eeb4b8e8
feat(resource): reset metrics when stopping a resource
2022-10-18 09:32:35 -03:00
f0ff32c031
test: fix tests after counter changes
2022-10-11 17:45:48 -03:00
9aa7e826cb
refactor(resource): fast resume resource worker if inflight msgs are ACKed
2022-09-17 00:34:30 +08:00
8307f04c2e
refactor(resource): save inflight size into the ETS table
2022-09-16 16:52:08 +08:00
b9ae4ea276
refactor: rename some metrics for emqx_resource
2022-09-13 14:04:25 +08:00
26234d38b9
fix: mark the async msg 'queuing' not 'sent.inflight' on recoverable_error
2022-09-02 18:41:43 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
6b0ccfbc43
refactor: rename the error return resource_down -> recoverable_error
2022-08-26 17:11:12 +08:00
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
22a4ca311c
feat(resource): resource batch/async/queue config schema
2022-08-11 16:59:18 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
efd6c56dd9
fix: test cases for batch query sync
2022-08-10 00:45:34 +08:00
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
f1419d52f1
fix(resource): remove resource at the end of each test
2022-08-10 00:34:35 +08:00
a2afdeeb48
feat: add test cases for batching query
2022-08-10 00:34:35 +08:00
d3950b9534
fix(resource): make option 'queue_enabled' disabled by default
2022-08-10 00:34:35 +08:00
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
d37a66e9b8
fix(test): update test cases for emqx_resource:health_check/1
2022-05-31 10:14:37 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
574a40b327
fix: wait for test_resource stop
2022-05-16 17:00:42 +08:00
6574c33797
feat: add auto_retry for disconnected state in resource manager
2022-05-13 11:19:39 +02:00
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
132b37813c
refactor: code format emqx_connector emqx_resource
2022-04-28 15:32:47 +08:00
2a2308bbf8
refactor: resource check & connector status
2022-04-28 15:32:35 +08:00
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
f42a5b90df
Revert "feat: isolate resource manager processes"
...
This reverts commit 40cca58d4f
2022-04-26 16:13:38 +02:00
40cca58d4f
feat: isolate resource manager processes
2022-04-26 13:28:29 +02:00
e5f04f3bf7
chore(emqx_authn_jwt): wrap JWKS connector into emqx_resourse
2022-04-18 15:47:33 +03:00
8f06a9ec62
feat: impl resource reset_metrics
2022-04-11 10:25:48 +08:00
db0e9e3358
fix(emqx_resource_instance): fix dialyzer warning
2022-03-08 14:09:39 +08:00
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
47a4fa5732
Merge pull request #7140 from EMQ-YangM/tmp_change_status
...
refactor(emqx_resource): change the status of emqx_resource to 'conne…
2022-02-28 11:13:47 +08:00
376c9ee261
refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting'
2022-02-25 15:02:41 +08:00
db584f79d6
feat: upgrade hocon to 0.25.0 to replace nullable with required.
2022-02-24 22:39:03 +08:00
48942f9c93
refactor(emqx_resource): move unused macro to test
2022-02-14 17:40:39 +08:00
df57daaabb
refactor(emqx_resource): improve grouping strategy for emqx_resource_instance
2022-02-11 18:36:55 +08:00
8cfbdc2730
test(emqx_resource): improve emqx_resource test coverage to 80%
2022-01-25 17:59:29 +08:00
d312f315ac
test(emqx_resource_health_check): add more test to
...
health_check_timeout_checker
2022-01-25 15:07:54 +08:00
cb9f14f658
feat(emqx_resource_health_check): add timeout params to health_check_timeout_checker
2022-01-25 14:54:40 +08:00
127384a9ae
test(emqx_resource_SUITE): add more test
2022-01-25 14:39:35 +08:00
b528862c67
Merge branch 'master' into health_check_timeout
2022-01-24 14:48:55 +08:00
Kjell Winblad