Commit Graph

15210 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 3891aeb5fc ci(ct/run.sh): best-effort file ownership restore 2023-01-18 10:52:29 +01:00
Erik Timan 42182279b7 fix(emqx_management): ensure trace file dir is deleted on zip exception 2023-01-18 10:20:41 +01:00
Zhongwen Deng fb84d5b817 chore: make spellcheck happy 2023-01-18 17:06:46 +08:00
Ivan Dyachkov 4dd6e06680
Merge pull request #9773 from id/ci-revert-to-use-upstream-setup-beam
ci: back to use upstream setup-beam action
2023-01-18 09:36:59 +01:00
Ivan Dyachkov 430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames
fix: ensure no colon in filenames
2023-01-18 09:36:16 +01:00
Zhongwen Deng 0d852d9122 docs: improve the dashboard's document 2023-01-18 16:28:35 +08:00
Andrew Mayorov 3c13dd38f6
feat: make `create_generation` safer against bad input 2023-01-18 11:11:49 +03:00
Andrew Mayorov 418ecbcbbc
feat: make iteration fully generation-aware 2023-01-18 11:11:49 +03:00
Andrew Mayorov e42f009d3f
chore: simplify iteration-related typespecs 2023-01-18 11:11:49 +03:00
Andrew Mayorov 4e36456843
feat: allow to create new storage generations 2023-01-18 11:11:46 +03:00
Andrew Mayorov a0f97ede67
Merge pull request #9740 from feat/rocksdb-replay-queue/iterator-refresh
feat: enable periodic iterator refresh
2023-01-18 12:10:51 +04:00
Zaiming (Stone) Shi faf5916ed6 test: relax recoverable/unrecoverable error check
for now, treat all other errors unrecoverable
2023-01-18 07:52:28 +01:00
zhongwencool a7d6daf632
Merge pull request #9794 from Rory-Z/fix/fix-helm-env-error
fix(helm): fix error environment variables
2023-01-18 10:36:34 +08:00
zhongwencool bc9d97ea53
Merge pull request #9791 from zhongwencool/crash-dump-doc
chore: more detail about crash dump config
2023-01-18 09:56:21 +08:00
Rory Z b8a92b7371 fix(helm): fix error environment variables 2023-01-18 09:56:21 +08:00
Thales Macedo Garitezi 5c2ac0ac81 chore: don't cancel inflight items upon worker death; retry them 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi 087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Stefan Strigler f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub
EMQX 8754 test function return 500 of data integration google pubsub
2023-01-17 22:49:28 +01:00
lafirest dea0c8230e
Merge pull request #9787 from lafirest/fix/webhook_bridge_cfg_upgrade
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
2023-01-18 04:47:08 +08:00
Thales Macedo Garitezi 4ed7bff33f chore: fix dialyzer warnings 2023-01-17 16:49:16 -03:00
Thales Macedo Garitezi fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi b82009bc29 refactor: use monotonic times as refs and store initial times when creating ets
with this, we may measure latencies in the future.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 3ba65c4377 feat: poke the buffer workers when inflight is no longer full
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi b5aaef084c refactor: enter running state directly
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi bd0e2a74ba refactor: rename inflight_name field to inflight_tid 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 731ac6567a fix(buffer_worker): don't retry all kinds of inflight requests
Some requests should not be retried during the blocked state.  For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 5425f3d88e refactor: rm unused fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 5dd24a64c3 refactor(buffer_worker): check if inflight is full before flushing 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 344eeebe63 fix: always ack async replies
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi bd95a95409 refactor: remove redundant `BlockWorker` arg, change boolean to ack/nack
`BlockWorker` was always false (ack).  Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/)
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 478fcc6ffd test: fix flaky test 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 30a227bd38 refactor: rename `resume` state timeout to `unblock` 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 7401d6f0ce refactor: rename ack fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 196bf1c5ba feat: mass collect calls from mailbox also when blocked 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi d4724d6ce9 refactor: remove redundant function
`retry_queue` does basically what the running state does, now that we
refactored the buffer workers to always use the queue.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi d6a9d0aa48 fix: set queuing to 0 after buffer worker termination 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 81fc561ed5 fix(buffer_worker): check for overflow after enqueuing new requests 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 4cb83d0c9a fix: fix some expressions after refactoring 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi fecdbac9a8 refactor: rename a few functions 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi cdd8de11b0 chore: fix a typo in function name 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi 618b97870b refactor: call local function queue_count everywhere 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi 249c4c1c79 refactor: use 'bufs' for resource worker replayq dir 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi af6807e863 refactor: cancel flush timer sooner
Avoids the cancellation being delayed.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 477c55d8ef fix: sanitizy replayq dir filepath
Colons (`:`) are not allowed in Windows.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 4c04a01370 refactor(buffer_worker): remove `?Q_ITEM` wrapping and use lightweight size estimate 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 32a9e60313 feat(buffer_worker): also use the inflight table for sync requests
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi ff23d25e8b chore(replayq): update replayq -> 0.3.6 and use `clean_start` for buffer workers
So we can truly avoid resuming work after a node restart.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi c383558467 fix(buffer): fix `replayq` usages in buffer workers (5.0)
https://emqx.atlassian.net/browse/EMQX-8700

Fixes a few errors in the usage of `replayq` queues.

- Close `replayq` when `emqx_resource_worker` terminates.
- Do not keep old references to `replayq` after any `pop`s.
- Clear `replayq`'s data directories when removing a resource.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 4c25be8a2c chore(influxdb): remove deprecated value from example 2023-01-17 16:48:48 -03:00