d464e2aad5
refactor: rename test resource prefix
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
03342923b9
fix(bridge): use the same dry run prefix
...
Kafka Producer and Consumer bridges rely on this prefix for detecting
a dry run and avoid leaking atoms. At some point, this prefix was
changed, effectively disabling the check in Kafka Producer.
2023-03-16 13:43:01 -03:00
e9d3fc511f
chore(buffer_worker): change default `batch_time` to 0 and improve docs
2023-03-06 15:31:28 -03:00
fb61c2b266
perf: avoid getting metrics (gen_server:call) for each resource lookup
2023-02-10 19:40:37 +01:00
c0d478bd41
fix(buffer_worker): type spec
2023-02-02 14:11:12 +01:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
44a6e5ed15
chore(resources): add missing parameters to emqx_resource schema
2023-01-18 14:33:45 +02:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
479e191dcf
refactor: refine worker pool config and doc
...
worker pool is a buffer pool
the description hinted connection pool which is wrong.
2022-12-20 09:02:51 +01:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
f41adb0997
refactor: change some default values of resource_opts
2022-09-14 15:18:07 +08:00
2b33ca6d49
fix: no error log print if insert bool values into mysql
2022-09-07 16:00:09 +08:00
0ef0b68de4
refactor: change '{recoverable_error,Reason}' to '{error,{recoverable_error,Reason}}'
2022-08-31 18:25:00 +08:00
9e50866cd0
fix: rename queue_max_bytes -> max_queue_bytes
2022-08-30 17:18:54 +08:00
6b0ccfbc43
refactor: rename the error return resource_down -> recoverable_error
2022-08-26 17:11:12 +08:00
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
f0c2b53868
fix(bpapi): make bpapi static_checks happy
2022-08-22 10:51:44 +08:00
7c4ea38c06
fix(resource): make some resource opts internal
...
Resource options `start_after_created` and `start_timeout` are internal opts.
Not provided to users anymore.
2022-08-22 02:22:57 +08:00
06363e63d9
fix(influxdb): connector use a fallbacke `pool_size` for influxdb client
2022-08-19 15:54:19 +08:00
9e35032d78
fix: make resume_interval defaults to health_check_interval
2022-08-16 10:09:02 +08:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
d1de262f31
fix: inc 'actions.failed' if bridge query failed
2022-08-15 17:21:14 +08:00
3678673124
fix: schema default value using raw type before convert
2022-08-12 16:38:46 +08:00
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
22a4ca311c
feat(resource): resource batch/async/queue config schema
2022-08-11 16:59:18 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
12904d797f
feat(resource): first commit for batching/async/caching mechanism
2022-08-10 00:34:35 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
b7f27157e5
fix: also alarm resource down when start resource failed
2022-06-01 15:41:55 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
4e65322667
refactor: move emqx_plugin_libs_metrics to emqx app
...
because it can not depend on other apps
2022-04-29 12:41:36 +08:00
2a2308bbf8
refactor: resource check & connector status
2022-04-28 15:32:35 +08:00
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
583624fb8d
fix(emqx_authn): fix test suite
2022-03-09 13:53:05 +08:00
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
376c9ee261
refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting'
2022-02-25 15:02:41 +08:00
252d7e85d9
refactor: call new hocon apis
2022-01-27 00:23:08 +01:00
63167cea70
chore: update copyright
2022-01-05 20:55:00 +01:00
e1ab331a30
refactor(resource): support async create mode
2022-01-02 20:20:39 +08:00
Thales Macedo Garitezi