yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,147 Commits 393 Branches 669 Tags 74 MiB
b3ad9e97d2
Commit Graph

306 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback 
feat(mqtt-bridge): avoid middleman process
 2023-01-31 19:12:18 +01:00
Andrew Mayorov
c76311c9c3
fix(buffer): count inflight batches properly 2023-01-31 18:30:42 +03:00
Zaiming (Stone) Shi
b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback 
0127 refactor buffer worker no need to keep request for reply callback
 2023-01-31 08:44:01 +01:00
Stefan Strigler
27881064dc fix: increase dropped.queue_full by number of messages 2023-01-30 11:37:35 +01:00
Zaiming (Stone) Shi
d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
fc38ea9571 refactor(buffer_worker): do not keep request body in reply context 
the request body can be potentially very large
the reply context is sent to the async call handler and kept
in its memory until the async reply is received from bridge
target service.

this commit tries to minimize the size of the reply context
by replacing the request body with `[]`.
 2023-01-27 17:12:55 +01:00
Zaiming (Stone) Shi
578271ea3d refactor: use lists:map instead of lc for safty 2023-01-27 15:15:46 +01:00
Zaiming (Stone) Shi
f793807bc1 refactor(buffer_worker): rename function 
batch_reply_after_query to handle_async_batch_reply
 2023-01-27 15:04:28 +01:00
Zaiming (Stone) Shi
262c3a2869 refactor(buffer_worker): rename function 
from reply_after_query to handle_async_reply
 2023-01-27 15:03:18 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi
514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak 
0127 fix influxdb bridge atom leak
 2023-01-27 14:30:20 +01:00
Zaiming (Stone) Shi
d53106145f fix: stop resource when resource manager terminates 2023-01-27 12:39:05 +01:00
Stefan Strigler
2d62de5188 test: fix expected result from timeout error 2023-01-27 11:43:48 +01:00
Stefan Strigler
a180bd9aa5 fix: catch error, not exit 2023-01-27 11:40:06 +01:00
Stefan Strigler
b7e3f9d5a6 fix: try-case-of rather than try-of 
try-of catches only what happens within but not after
 2023-01-27 11:40:06 +01:00
Zaiming (Stone) Shi
db2f631a8a refactor(buffer_worker): simplify caller reply 2023-01-27 11:33:45 +01:00
Zaiming (Stone) Shi
d4fab92b72 refactor(buffer_worker): no need to keep request for REPLY macro 2023-01-27 10:41:30 +01:00
Zaiming (Stone) Shi
1f799dfd59 fix: reply with {error, buffer_overflow} when discarded 2023-01-26 17:15:36 +01:00
Zaiming (Stone) Shi
ed28789164 refactor(buffer_worker): no need to return after collect into buf queue 2023-01-26 14:50:40 +01:00
Zaiming (Stone) Shi
25b4821adc refactor: move the the per-message overflow log from error to info level 2023-01-26 14:48:43 +01:00
Zaiming (Stone) Shi
bb26632c8a fix(buffer_worker): fix a wrong assertion 
the assertion is to ensure queue items are not binary
but should not assert the queue itself
 2023-01-26 14:33:16 +01:00
Erik Timan
805d08e823 fix: reduce log level from error to warning in several places 
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
 2023-01-25 14:49:50 +01:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Erik Timan
9d20431257 fix(emqx_resource): fix crash while flushing queue 
We used next_event for flushing the queue in emqx_resource, but this
leads to a crash. We now call flush_worker/1 instead.
 2023-01-24 14:13:35 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi
1c3e055b13
Merge pull request #9822 from JimMoen/fix-schema-typo 
chore: i18n typo fix
 2023-01-20 11:11:18 +01:00
JimMoen
16f45a60fd
chore: i18n typo fix 2023-01-20 11:50:01 +08:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov
44a6e5ed15 chore(resources): add missing parameters to emqx_resource schema 2023-01-18 14:33:45 +02:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Ivan Dyachkov
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames 
fix: ensure no colon in filenames
 2023-01-18 09:36:16 +01:00
Zaiming (Stone) Shi
faf5916ed6 test: relax recoverable/unrecoverable error check 
for now, treat all other errors unrecoverable
 2023-01-18 07:52:28 +01:00
Thales Macedo Garitezi
5c2ac0ac81 chore: don't cancel inflight items upon worker death; retry them 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
4ed7bff33f chore: fix dialyzer warnings 2023-01-17 16:49:16 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
b82009bc29 refactor: use monotonic times as refs and store initial times when creating ets 
with this, we may measure latencies in the future.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
3ba65c4377 feat: poke the buffer workers when inflight is no longer full 
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
b5aaef084c refactor: enter running state directly 
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd0e2a74ba refactor: rename inflight_name field to inflight_tid 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
731ac6567a fix(buffer_worker): don't retry all kinds of inflight requests 
Some requests should not be retried during the blocked state.  For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5425f3d88e refactor: rm unused fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5dd24a64c3 refactor(buffer_worker): check if inflight is full before flushing 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
344eeebe63 fix: always ack async replies 
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd95a95409 refactor: remove redundant BlockWorker arg, change boolean to ack/nack 
`BlockWorker` was always false (ack).  Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/)
 2023-01-17 16:48:48 -03:00