yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,069 Commits 393 Branches 669 Tags 74 MiB
8fde169abb
Commit Graph

244 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov
44a6e5ed15 chore(resources): add missing parameters to emqx_resource schema 2023-01-18 14:33:45 +02:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Ivan Dyachkov
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames 
fix: ensure no colon in filenames
 2023-01-18 09:36:16 +01:00
Zaiming (Stone) Shi
faf5916ed6 test: relax recoverable/unrecoverable error check 
for now, treat all other errors unrecoverable
 2023-01-18 07:52:28 +01:00
Thales Macedo Garitezi
5c2ac0ac81 chore: don't cancel inflight items upon worker death; retry them 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
4ed7bff33f chore: fix dialyzer warnings 2023-01-17 16:49:16 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
b82009bc29 refactor: use monotonic times as refs and store initial times when creating ets 
with this, we may measure latencies in the future.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
3ba65c4377 feat: poke the buffer workers when inflight is no longer full 
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
b5aaef084c refactor: enter running state directly 
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd0e2a74ba refactor: rename inflight_name field to inflight_tid 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
731ac6567a fix(buffer_worker): don't retry all kinds of inflight requests 
Some requests should not be retried during the blocked state.  For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5425f3d88e refactor: rm unused fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5dd24a64c3 refactor(buffer_worker): check if inflight is full before flushing 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
344eeebe63 fix: always ack async replies 
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd95a95409 refactor: remove redundant BlockWorker arg, change boolean to ack/nack 
`BlockWorker` was always false (ack).  Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/)
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
30a227bd38 refactor: rename resume state timeout to unblock 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
7401d6f0ce refactor: rename ack fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
196bf1c5ba feat: mass collect calls from mailbox also when blocked 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
d4724d6ce9 refactor: remove redundant function 
`retry_queue` does basically what the running state does, now that we
refactored the buffer workers to always use the queue.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
d6a9d0aa48 fix: set queuing to 0 after buffer worker termination 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
81fc561ed5 fix(buffer_worker): check for overflow after enqueuing new requests 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4cb83d0c9a fix: fix some expressions after refactoring 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
fecdbac9a8 refactor: rename a few functions 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
cdd8de11b0 chore: fix a typo in function name 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
618b97870b refactor: call local function queue_count everywhere 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
249c4c1c79 refactor: use 'bufs' for resource worker replayq dir 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
af6807e863 refactor: cancel flush timer sooner 
Avoids the cancellation being delayed.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
477c55d8ef fix: sanitizy replayq dir filepath 
Colons (`:`) are not allowed in Windows.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4c04a01370 refactor(buffer_worker): remove ?Q_ITEM wrapping and use lightweight size estimate 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
ff23d25e8b chore(replayq): update replayq -> 0.3.6 and use clean_start for buffer workers 
So we can truly avoid resuming work after a node restart.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
c383558467 fix(buffer): fix replayq usages in buffer workers (5.0) 
https://emqx.atlassian.net/browse/EMQX-8700

Fixes a few errors in the usage of `replayq` queues.

- Close `replayq` when `emqx_resource_worker` terminates.
- Do not keep old references to `replayq` after any `pop`s.
- Clear `replayq`'s data directories when removing a resource.
 2023-01-17 16:48:48 -03:00
Stefan Strigler
e54f2f83b3 test: use same default timeout as elsewhere 2023-01-17 15:29:19 +01:00
Ivan Dyachkov
676f017ec0 fix: ensure no colon in filenames 2023-01-16 21:27:01 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Stefan Strigler
1690a6dcfc
Merge branch 'master' into dev/api-refactor 2023-01-13 15:34:13 +01:00
Erik Timan
61e98900be chore: bump app vsn of emqx_resource 2023-01-13 15:13:35 +01:00
Kjell Winblad
734e6b9c96 chore: fix flaky test cases, log labels and review comments 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-01-13 11:05:02 +01:00
Kjell Winblad
8c482e03d1 fix: remove atom leaks 
Both emqx_resource_managers and emqx_resource_workers leaked atoms as they
created an unique atoms to use as registered names. This is fixed by
removing the need to register the names.

Fixes: https://emqx.atlassian.net/browse/EMQX-8583
 2023-01-11 17:03:28 +01:00
Stefan Strigler
8ad8288195 feat: report error in create_dry_run 2023-01-11 14:22:37 +01:00
Zaiming (Stone) Shi
85a8eff90b fix(emqx_resource_manager): do not start when disabled 2023-01-11 08:33:48 +01:00
Thales Macedo Garitezi
70eb5ffb58 refactor: remove unused function 2023-01-05 10:16:01 -03:00
Thales Macedo Garitezi
fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00