b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback
...
0127 refactor buffer worker no need to keep request for reply callback
2023-01-31 08:44:01 +01:00
27881064dc
fix: increase dropped.queue_full by number of messages
2023-01-30 11:37:35 +01:00
d47941601d
refactor(buffer_worker): rename trace points
2023-01-28 11:52:11 +01:00
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
...
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
fc38ea9571
refactor(buffer_worker): do not keep request body in reply context
...
the request body can be potentially very large
the reply context is sent to the async call handler and kept
in its memory until the async reply is received from bridge
target service.
this commit tries to minimize the size of the reply context
by replacing the request body with `[]`.
2023-01-27 17:12:55 +01:00
578271ea3d
refactor: use lists:map instead of lc for safty
2023-01-27 15:15:46 +01:00
f793807bc1
refactor(buffer_worker): rename function
...
batch_reply_after_query to handle_async_batch_reply
2023-01-27 15:04:28 +01:00
262c3a2869
refactor(buffer_worker): rename function
...
from reply_after_query to handle_async_reply
2023-01-27 15:03:18 +01:00
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker
...
EMQX 8774 failure to handle timeout error in resource worker
2023-01-27 14:36:44 +01:00
514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak
...
0127 fix influxdb bridge atom leak
2023-01-27 14:30:20 +01:00
d53106145f
fix: stop resource when resource manager terminates
2023-01-27 12:39:05 +01:00
2d62de5188
test: fix expected result from timeout error
2023-01-27 11:43:48 +01:00
a180bd9aa5
fix: catch error, not exit
2023-01-27 11:40:06 +01:00
b7e3f9d5a6
fix: try-case-of rather than try-of
...
try-of catches only what happens within but not after
2023-01-27 11:40:06 +01:00
db2f631a8a
refactor(buffer_worker): simplify caller reply
2023-01-27 11:33:45 +01:00
d4fab92b72
refactor(buffer_worker): no need to keep request for REPLY macro
2023-01-27 10:41:30 +01:00
1f799dfd59
fix: reply with {error, buffer_overflow} when discarded
2023-01-26 17:15:36 +01:00
ed28789164
refactor(buffer_worker): no need to return after collect into buf queue
2023-01-26 14:50:40 +01:00
25b4821adc
refactor: move the the per-message overflow log from error to info level
2023-01-26 14:48:43 +01:00
bb26632c8a
fix(buffer_worker): fix a wrong assertion
...
the assertion is to ensure queue items are not binary
but should not assert the queue itself
2023-01-26 14:33:16 +01:00
805d08e823
fix: reduce log level from error to warning in several places
...
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
2023-01-25 14:49:50 +01:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
9d20431257
fix(emqx_resource): fix crash while flushing queue
...
We used next_event for flushing the queue in emqx_resource, but this
leads to a crash. We now call flush_worker/1 instead.
2023-01-24 14:13:35 +01:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
1c3e055b13
Merge pull request #9822 from JimMoen/fix-schema-typo
...
chore: i18n typo fix
2023-01-20 11:11:18 +01:00
16f45a60fd
chore: i18n typo fix
2023-01-20 11:50:01 +08:00
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
44a6e5ed15
chore(resources): add missing parameters to emqx_resource schema
2023-01-18 14:33:45 +02:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames
...
fix: ensure no colon in filenames
2023-01-18 09:36:16 +01:00
faf5916ed6
test: relax recoverable/unrecoverable error check
...
for now, treat all other errors unrecoverable
2023-01-18 07:52:28 +01:00
5c2ac0ac81
chore: don't cancel inflight items upon worker death; retry them
2023-01-17 19:50:30 -03:00
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
4ed7bff33f
chore: fix dialyzer warnings
2023-01-17 16:49:16 -03:00
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
b82009bc29
refactor: use monotonic times as refs and store initial times when creating ets
...
with this, we may measure latencies in the future.
2023-01-17 16:48:48 -03:00
3ba65c4377
feat: poke the buffer workers when inflight is no longer full
...
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
2023-01-17 16:48:48 -03:00
b5aaef084c
refactor: enter running state directly
...
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
2023-01-17 16:48:48 -03:00
bd0e2a74ba
refactor: rename inflight_name field to inflight_tid
2023-01-17 16:48:48 -03:00
006b4bda97
feat(buffer_worker): monitor async workers and cancel their inflight requests upon death
2023-01-17 16:48:48 -03:00
731ac6567a
fix(buffer_worker): don't retry all kinds of inflight requests
...
Some requests should not be retried during the blocked state. For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
2023-01-17 16:48:48 -03:00
5425f3d88e
refactor: rm unused fn
2023-01-17 16:48:48 -03:00
5dd24a64c3
refactor(buffer_worker): check if inflight is full before flushing
2023-01-17 16:48:48 -03:00
344eeebe63
fix: always ack async replies
...
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
2023-01-17 16:48:48 -03:00
bd95a95409
refactor: remove redundant `BlockWorker` arg, change boolean to ack/nack
...
`BlockWorker` was always false (ack). Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/ )
2023-01-17 16:48:48 -03:00
30a227bd38
refactor: rename `resume` state timeout to `unblock`
2023-01-17 16:48:48 -03:00
7401d6f0ce
refactor: rename ack fn
2023-01-17 16:48:48 -03:00
196bf1c5ba
feat: mass collect calls from mailbox also when blocked
2023-01-17 16:48:48 -03:00
d4724d6ce9
refactor: remove redundant function
...
`retry_queue` does basically what the running state does, now that we
refactored the buffer workers to always use the queue.
2023-01-17 16:48:48 -03:00
d6a9d0aa48
fix: set queuing to 0 after buffer worker termination
2023-01-17 16:48:48 -03:00
81fc561ed5
fix(buffer_worker): check for overflow after enqueuing new requests
2023-01-17 16:48:48 -03:00
4cb83d0c9a
fix: fix some expressions after refactoring
2023-01-17 16:48:48 -03:00
fecdbac9a8
refactor: rename a few functions
2023-01-17 16:48:48 -03:00
cdd8de11b0
chore: fix a typo in function name
2023-01-17 16:48:48 -03:00
618b97870b
refactor: call local function queue_count everywhere
2023-01-17 16:48:48 -03:00
249c4c1c79
refactor: use 'bufs' for resource worker replayq dir
2023-01-17 16:48:48 -03:00
af6807e863
refactor: cancel flush timer sooner
...
Avoids the cancellation being delayed.
2023-01-17 16:48:48 -03:00
477c55d8ef
fix: sanitizy replayq dir filepath
...
Colons (`:`) are not allowed in Windows.
2023-01-17 16:48:48 -03:00
4c04a01370
refactor(buffer_worker): remove `?Q_ITEM` wrapping and use lightweight size estimate
2023-01-17 16:48:48 -03:00
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
ff23d25e8b
chore(replayq): update replayq -> 0.3.6 and use `clean_start` for buffer workers
...
So we can truly avoid resuming work after a node restart.
2023-01-17 16:48:48 -03:00
c383558467
fix(buffer): fix `replayq` usages in buffer workers (5.0)
...
https://emqx.atlassian.net/browse/EMQX-8700
Fixes a few errors in the usage of `replayq` queues.
- Close `replayq` when `emqx_resource_worker` terminates.
- Do not keep old references to `replayq` after any `pop`s.
- Clear `replayq`'s data directories when removing a resource.
2023-01-17 16:48:48 -03:00
e54f2f83b3
test: use same default timeout as elsewhere
2023-01-17 15:29:19 +01:00
676f017ec0
fix: ensure no colon in filenames
2023-01-16 21:27:01 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
61e98900be
chore: bump app vsn of emqx_resource
2023-01-13 15:13:35 +01:00
1ac03ab208
Merge pull request #9730 from kjellwinblad/kjell/fix/resource_atom_leak/EMQX-8583
...
fix: remove atom leaks
2023-01-13 14:38:28 +01:00
734e6b9c96
chore: fix flaky test cases, log labels and review comments
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-01-13 11:05:02 +01:00
b5d3e9d8b8
fix: remove time unit from duration fields description
2023-01-12 14:18:55 +01:00
8c482e03d1
fix: remove atom leaks
...
Both emqx_resource_managers and emqx_resource_workers leaked atoms as they
created an unique atoms to use as registered names. This is fixed by
removing the need to register the names.
Fixes: https://emqx.atlassian.net/browse/EMQX-8583
2023-01-11 17:03:28 +01:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
85a8eff90b
fix(emqx_resource_manager): do not start when disabled
2023-01-11 08:33:48 +01:00
70eb5ffb58
refactor: remove unused function
2023-01-05 10:16:01 -03:00
56437228dc
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections.
2023-01-05 10:16:01 -03:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
bf3983e7c4
feat(buffer_worker): use offload mode for `replayq`
...
To avoid confusion for the users as to what persistence guarantees we
offer when buffering bridges/resources, we will always enable offload
mode for `replayq`. With this, when the buffer size is above the max
segment size, it'll flush the queue to disk, but on recovery after a
restart it'll clean the existing segments rather than resuming from
them.
2023-01-05 10:11:59 -03:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
305ed68916
chore: bump app vsns
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
f93c22045d
fix: non-empty field should not be undefined
2022-12-24 11:41:45 +01:00
479e191dcf
refactor: refine worker pool config and doc
...
worker pool is a buffer pool
the description hinted connection pool which is wrong.
2022-12-20 09:02:51 +01:00
f611cbab45
chore: cap replayq seg size under total size
2022-12-19 23:16:05 +01:00
8a0ca38a77
fix: drop no longer supported dialyzer option
2022-12-16 13:45:05 +03:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
34e9056779
refactor: fix typo in variable name
...
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
62eeb4b8e8
feat(resource): reset metrics when stopping a resource
2022-10-18 09:32:35 -03:00
2d01726b22
fix: account calls when resource is not connected as matched
2022-10-13 15:32:04 -03:00
1b2b629cdd
feat: emit telemetry events for all resource worker metrics
2022-10-13 15:32:04 -03:00
f0ff32c031
test: fix tests after counter changes
2022-10-11 17:45:48 -03:00
357e5919ce
chore: add copyright disclaimer
2022-10-11 09:51:16 -03:00
57270fb8fc
feat: add support for counters and gauges to the Kafka Bridge
...
This commit adds support for counters and gauges to the Kafka Brige.
The Kafka bridge uses [Wolff](https://github.com/kafka4beam/wolff ) for
the Kafka connection. Wolff does its own batching and does not use the
batching functionality in `emqx_resource_worker` that is used by other
bridge types. Therefore, the counter events have to be generated by
Wolff. We have added
[telemetry](https://github.com/beam-telemetry/telemetry ) events to Wolff
that we hook into to change counters and gauges for the Kafka bridge. The
counter called `matched` does not depend on specific functionality of
any bridge type so the updates of this counter is moved higher up in the
call chain then previously so that it also gets updated for Kafka
bridges.
2022-10-10 14:40:57 -03:00
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
...
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
b325633390
refactor(resource): resume from queue/inflight-window with async-sending and batching
2022-09-21 22:58:47 +08:00
9aa7e826cb
refactor(resource): fast resume resource worker if inflight msgs are ACKed
2022-09-17 00:34:30 +08:00
8307f04c2e
refactor(resource): save inflight size into the ETS table
2022-09-16 16:52:08 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
4e211c12d3
fix(mqtt_bridge): return value of sending messages was discarded
2022-09-15 08:57:01 +08:00
1c03c236f5
fix(mqtt_bridge): handle send_to_remote in idle state
2022-09-14 15:19:30 +08:00
f41adb0997
refactor: change some default values of resource_opts
2022-09-14 15:18:07 +08:00
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
b9ae4ea276
refactor: rename some metrics for emqx_resource
2022-09-13 14:04:25 +08:00
2b33ca6d49
fix: no error log print if insert bool values into mysql
2022-09-07 16:00:09 +08:00
26234d38b9
fix: mark the async msg 'queuing' not 'sent.inflight' on recoverable_error
2022-09-02 18:41:43 +08:00
83f21b4c65
refactor(resource): remove metrics 'sent.exception'
2022-09-02 12:46:53 +08:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
33c9c7d497
fix: incorrect message order when batch is enabled
2022-09-01 14:51:13 +08:00
0ef0b68de4
refactor: change '{recoverable_error,Reason}' to '{error,{recoverable_error,Reason}}'
2022-08-31 18:25:00 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
9e50866cd0
fix: rename queue_max_bytes -> max_queue_bytes
2022-08-30 17:18:54 +08:00
c4106c0d77
fix: resume the resource worker on health check success
2022-08-30 12:28:43 +08:00
6fde37791c
refactor: new metrics for resources
2022-08-30 10:14:10 +08:00
1625b8eaeb
fix(mysql_bridge): export the query_mode option to the APIs
2022-08-26 17:11:24 +08:00
6b0ccfbc43
refactor: rename the error return resource_down -> recoverable_error
2022-08-26 17:11:12 +08:00
a896aa8b27
fix: incorrect replayq dir for the emqx_resource
2022-08-25 16:06:18 +08:00
86577365e4
fix: use gen_statem:cast/3 for async query
2022-08-23 22:41:45 +08:00
f0c2b53868
fix(bpapi): make bpapi static_checks happy
2022-08-22 10:51:44 +08:00
62ecf6f545
fix(resource): keep `auto_retry` in `disconnected` state
...
Automatic retries should be maintained even in `disconnected` state without any state transition.
2022-08-22 02:52:06 +08:00
7c4ea38c06
fix(resource): make some resource opts internal
...
Resource options `start_after_created` and `start_timeout` are internal opts.
Not provided to users anymore.
2022-08-22 02:22:57 +08:00
06363e63d9
fix(influxdb): connector use a fallbacke `pool_size` for influxdb client
2022-08-19 15:54:19 +08:00
9e35032d78
fix: make resume_interval defaults to health_check_interval
2022-08-16 10:09:02 +08:00
de3a325953
fix: revert the changes in connector mysql
2022-08-16 09:06:13 +08:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
d1de262f31
fix: inc 'actions.failed' if bridge query failed
2022-08-15 17:21:14 +08:00
665ef4142d
fix: unify the health check interval
2022-08-15 17:21:14 +08:00
68946f1f6c
feat: influxdb support `async`/`batch_async` query
2022-08-15 14:02:17 +08:00
b01ae8ece6
chore: refine influxdb bridge/connector i18n
2022-08-15 14:00:14 +08:00
594d071c05
feat(influxdb): add async callback
2022-08-12 18:26:47 +08:00
fa5e8f1422
chore: refine i18n label
2022-08-12 16:39:03 +08:00
3678673124
fix: schema default value using raw type before convert
2022-08-12 16:38:46 +08:00
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
c3c4ed02b4
fix: bump emqx_dashboard to 5.0.4
2022-08-12 00:24:58 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
0f6c371760
feat(influxdb): influxdb connector add `on_batch_query/3` callback
2022-08-11 18:12:41 +08:00
22a4ca311c
feat(resource): resource batch/async/queue config schema
2022-08-11 16:59:18 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
82550a585a
fix: add test cases for query async
2022-08-10 00:45:34 +08:00
efd6c56dd9
fix: test cases for batch query sync
2022-08-10 00:45:34 +08:00
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
35fe70b887
feat: support aysnc callback to connector modules
2022-08-10 00:34:35 +08:00
f1419d52f1
fix(resource): remove resource at the end of each test
2022-08-10 00:34:35 +08:00
a2afdeeb48
feat: add test cases for batching query
2022-08-10 00:34:35 +08:00
75adba0781
fix: increase resource metrics using the resource id
2022-08-10 00:34:35 +08:00
d3950b9534
fix(resource): make option 'queue_enabled' disabled by default
2022-08-10 00:34:35 +08:00
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
2fb42e4d37
refactor: create emqx_resource_worker_sup for resource workers
2022-08-10 00:34:35 +08:00
0087b7c960
fix: remove the extra file replay.erl
2022-08-10 00:34:35 +08:00
d8d8d674e4
feat(resource): start emqx_resource_worker in pools
2022-08-10 00:34:35 +08:00
12904d797f
feat(resource): first commit for batching/async/caching mechanism
2022-08-10 00:34:35 +08:00
98b36c4681
fix: hstream db connector , TODO: start apps
2022-07-27 11:38:45 +08:00
a78a389206
chore: using standard log format
2022-07-01 12:06:35 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
cc25f92273
feat: add start_after_created option to resource:create/4
2022-06-16 23:34:52 +08:00
2065be569e
fix(emqx_cluster_rpc): fail fast on stale state
...
Due to:
* Cluster RPC MFA is not idempotent!
* There is a lack of rollback for callback's side-effects
For instance, when two nodes try to add a cluster-singleton
concurrently, one of them will have to wait for the table lock
then try to catch-up, then try to apply MFA.
The catch-up will have the singleton created, but the initiated
initiated multicall apply will fail causing the commit to rollback,
but not to 'undo' the singleton creation.
Later, the retries will fail indefinitely.
2022-06-12 20:18:48 +02:00
b7f27157e5
fix: also alarm resource down when start resource failed
2022-06-01 15:41:55 +08:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
9f69e3cad6
fix(resource): discard dry_run resource down alarm
2022-06-01 08:24:53 +08:00
d37a66e9b8
fix(test): update test cases for emqx_resource:health_check/1
2022-05-31 10:14:37 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
574a40b327
fix: wait for test_resource stop
2022-05-16 17:00:42 +08:00
b5addf7e05
fix: log all ignore events
2022-05-16 15:08:03 +08:00
bbbfea1b5b
fix: ignore all other events
2022-05-16 15:08:03 +08:00
1a1c82932a
fix: when connecting health check failed, update status.
2022-05-16 10:47:20 +08:00
93799e3ac6
refactor: delete now unused emqx_resource modules
2022-05-16 09:54:26 +08:00
c4fd31ae25
Merge pull request #7916 from emqx/EMQX-4204-auto-timer-based-retry-when-in-disconnected-state
...
feat: add auto_retry for disconnected state in resource manager
2022-05-16 09:34:08 +08:00
3f59650e4b
Merge pull request #7944 from EMQ-YangM/fix_bridge_status
...
fix: restart resource should not clear metrics
2022-05-16 09:16:12 +08:00
c355c40ea8
refactor: call emqx_alarm:ensure_deactivated everywhere
2022-05-13 16:02:55 +02:00
6574c33797
feat: add auto_retry for disconnected state in resource manager
2022-05-13 11:19:39 +02:00
d5c416736b
fix: restart resource should not clear metrics
2022-05-13 16:05:41 +08:00
a5ddc5390f
refactor(resource): add resource recreate fun with empty opts
2022-05-12 14:19:56 +08:00
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
c52b464b3c
fix: check process alive before health check
2022-04-29 17:34:26 +08:00
1bf33f75cc
fix: set resource status disconnected
2022-04-29 17:05:12 +08:00
4e65322667
refactor: move emqx_plugin_libs_metrics to emqx app
...
because it can not depend on other apps
2022-04-29 12:41:36 +08:00
132b37813c
refactor: code format emqx_connector emqx_resource
2022-04-28 15:32:47 +08:00
667da90e52
refactor: resource instance do_create_dry_run
2022-04-28 15:32:41 +08:00
2a2308bbf8
refactor: resource check & connector status
2022-04-28 15:32:35 +08:00
a50980c496
fix: disconnected status in auto_reconnect = false
2022-04-28 09:47:36 +08:00
02c3f87b31
style: reformat all remaining apps
2022-04-27 15:51:18 +02:00
94e24c2621
refactor: move ssl file handling from resources to bridges
2022-04-27 11:59:15 +08:00
f42a5b90df
Revert "feat: isolate resource manager processes"
...
This reverts commit 40cca58d4f
2022-04-26 16:13:38 +02:00
40cca58d4f
feat: isolate resource manager processes
2022-04-26 13:28:29 +02:00
19630e9a99
feat: save ssl cert files for data bridges
2022-04-21 09:00:06 +08:00
3ce969fd79
refactor: always recreate resources no matter it is connected or not
2022-04-20 11:43:05 +08:00
278e9145b0
fix: go to different resource instance when health check
2022-04-19 23:00:34 +08:00
e5f04f3bf7
chore(emqx_authn_jwt): wrap JWKS connector into emqx_resourse
2022-04-18 15:47:33 +03:00
8f06a9ec62
feat: impl resource reset_metrics
2022-04-11 10:25:48 +08:00
9a2d70f98e
fix(emqx_resource): remove extra space
2022-03-25 18:26:18 +08:00
6b662d87ba
fix(emqx_resource): fix dialyzer warning
2022-03-25 18:15:23 +08:00
bb12378806
fix(emqx_resource_instance): improve the pattern match of the function call_health_check
2022-03-25 17:35:55 +08:00
3414e0b601
feat(plugin): http api
2022-03-11 15:55:02 +08:00
1d023b541f
refactor(connnector): rename waiting_connect_complete -> wait_for_resource_ready
...
Rename the option to wait_for_resource_ready and defaults to 5s.
2022-03-10 10:46:57 +08:00
583624fb8d
fix(emqx_authn): fix test suite
2022-03-09 13:53:05 +08:00
db0e9e3358
fix(emqx_resource_instance): fix dialyzer warning
2022-03-08 14:09:39 +08:00
f29877bb6a
fix(emqx_resource): remove create_opts async_create
2022-03-08 14:09:39 +08:00
bf57bf717c
feat(emqx_resource_instance): add wait_connected Opts
2022-03-08 14:09:14 +08:00
47a4fa5732
Merge pull request #7140 from EMQ-YangM/tmp_change_status
...
refactor(emqx_resource): change the status of emqx_resource to 'conne…
2022-02-28 11:13:47 +08:00
376c9ee261
refactor(emqx_resource): change the status of emqx_resource to 'connected/connecting/disconnecting'
2022-02-25 15:02:41 +08:00
db584f79d6
feat: upgrade hocon to 0.25.0 to replace nullable with required.
2022-02-24 22:39:03 +08:00
48942f9c93
refactor(emqx_resource): move unused macro to test
2022-02-14 17:40:39 +08:00
df57daaabb
refactor(emqx_resource): improve grouping strategy for emqx_resource_instance
2022-02-11 18:36:55 +08:00
c2772fdd09
refactor: remove unused/old functions
2022-02-09 10:14:13 +01:00
3b3f3fed76
refactor(emqx_resource_instance): add function create_default_checker
2022-02-07 17:49:51 +08:00
8976e25b7a
fix(emqx_resource_instance): create health_check process when create resources synchronously
2022-02-07 17:49:51 +08:00
f7703c906e
refactor: move plain_check impl to emqx_hocon
2022-01-29 10:16:55 +01:00
d6f7ffec3c
Merge pull request #6881 from zmstone/refactor-upgrade-hocon-0.23.0
...
refactor: upgrade to hocon 0.23.0
2022-01-27 15:48:23 +01:00
fc92e4c8bf
docs: fix typos
2022-01-27 12:29:38 +01:00
cf1f19258e
refactor(emqx_resource): catch only hocon throw exceptions
2022-01-27 00:23:08 +01:00
252d7e85d9
refactor: call new hocon apis
2022-01-27 00:23:08 +01:00
8cfbdc2730
test(emqx_resource): improve emqx_resource test coverage to 80%
2022-01-25 17:59:29 +08:00
d312f315ac
test(emqx_resource_health_check): add more test to
...
health_check_timeout_checker
2022-01-25 15:07:54 +08:00
cb9f14f658
feat(emqx_resource_health_check): add timeout params to health_check_timeout_checker
2022-01-25 14:54:40 +08:00
127384a9ae
test(emqx_resource_SUITE): add more test
2022-01-25 14:39:35 +08:00
11f76db67d
fix(emqx_resource_validator): remove unused code
2022-01-24 16:11:32 +08:00
40b73fb199
fix(emqx_resource_api): remove unused module
2022-01-24 16:11:32 +08:00
b528862c67
Merge branch 'master' into health_check_timeout
2022-01-24 14:48:55 +08:00
c870a2c78c
test(emqx_resource_health_check): add async_create to create_local
2022-01-24 14:24:31 +08:00
8506bed489
fix(emqx_resource_health_check): link heath_check, health_check_timeout_checker
2022-01-24 10:18:18 +08:00
acc4ad0542
fix(emqx_resource): fix resource leakage
2022-01-21 22:50:30 +03:00
542c9a2b68
refactor(emqx_resource): Decorate RPCs
2022-01-19 22:00:34 +01:00
fd7e0c800a
feat(emqx_resource_health_check): add timeout to single health_check
2022-01-19 16:45:56 +08:00
82e3565344
chore: major mode file variable for Emacs
2022-01-13 15:53:47 +08:00
67a60e1153
refactor(rule): add more metrics for rule and bridges
2022-01-07 09:34:54 +08:00
78aa0abd3d
fix(metrics): update the calls to emqx_plugin_libs_metrcis module
2022-01-07 09:34:48 +08:00
ab5ad22b1d
fix(metrics): update the calls to emqx_plugin_libs_metrcis module
2022-01-07 09:34:41 +08:00
63167cea70
chore: update copyright
2022-01-05 20:55:00 +01:00
7b478817bd
test: test authz against 'single' mogodb
2022-01-04 00:54:09 +01:00
2898fa76e1
Merge remote-tracking branch 'origin/release-5.0-beta.3' into merge-5.0-beta.3-to-master
2022-01-03 11:39:06 +01:00
11736dc1d7
fix(bridge): check health immediately after updated
2022-01-02 22:45:32 +08:00
a64b29ff76
fix(resource): re-create the helth checker if already exists
2022-01-02 20:36:30 +08:00
e1ab331a30
refactor(resource): support async create mode
2022-01-02 20:20:39 +08:00
2277b75b2f
refactor(resource): improve the process starting/stopping resource instances
2022-01-02 17:14:02 +08:00
95340b7baa
Merge pull request #6564 from EMQ-YangM/beta3
...
feat(emqx_resource): add health_ckeck process
2022-01-02 12:46:22 +08:00
efec4564f0
fix(resource): update test cases on resource not_found
2021-12-31 22:25:45 +08:00
657ecef67b
fix(resource): don't crash on resource stopped
2021-12-31 20:57:34 +08:00
c7693246fe
feat(emqx_resource): add health_ckeck process, it will periodically perform health checks, and print error logs and generate alarms when the checks fail.
2021-12-30 21:53:51 -08:00
ea2d4674df
fix(resource): metrics were cleared after updating the resource
2021-12-31 00:03:03 +08:00
9d733c2ec5
fix(resource): typos on restart a resource
2021-12-30 23:48:41 +08:00
d11cf6ad64
fix(bridges): store connector name and bridge name to config files
2021-12-30 23:48:41 +08:00
11c169501e
chore(mix): fix app dependencies and reboot apps (emqx_conf)
...
While declaring `emqx_conf` as an application dependency of
`emqx_resource` worked for releases, it messed up the startup
relationship during tests. Since only removing `emqx_conf` from the
`applications` key in `emqx_resource` breaks the list of apps that
need to be rebooted on config changes (since `emqx_conf` is not on any
apps dependencies list, it was not being added to the final
topologically sorted list), we now always add it as a vertex to ensure
its presence there. This (apparently) makes the rebar3 release, the
mix release and test runs behave normally.
2021-12-23 13:48:28 -03:00
34d6dbca61
chore(mix): simplify build by using rebar3 for umbrella apps
...
By treating the apps in the umbrella as dependencies to be managed and
built by rebar3, we can simplify the maintenance of the release, at
the cost of increased build times: using Mix as before, it could track
changed files better than using rebar. But the complexity and
possibility of discrepancies make it using rebar much more compelling.
2021-12-22 09:40:01 -03:00
917575de5a
chore(mix): minimal elixir mix release build
...
This commit enables a minimal working build of EMQX release using
Mix. However, to properly start the release, several configuration
steps are still missing. A `mix_release.sh` script does a few hacks
to get the release built with Mix to start properly, by first assuming
that `make emqx` has been run prior to the release, ran once to
generate the `app.*.config` files, and then it copies that and some
other files to the expected places.
Also, `emqx_telemetry` hangs the start procedure because it thinks
it's in an official release and tries to make a request. We disable
it temporarily via config just to get a working build for now.
2021-12-21 13:51:22 -03:00
a879ec0f3a
feat(resource): add option 'force_create' to emqx_resource:create/4
2021-12-20 10:26:27 +08:00
a44e18e869
fix(bridge): filter out some extra fields from the request body
2021-12-18 19:19:58 +08:00
11e8e0db69
fix(bridge): stop http failed due to econnrefused
2021-12-18 15:33:25 +08:00
46838a08cc
fix(resource): update testcases for after_query functions
2021-11-23 10:41:45 +08:00
9aa63358e6
fix(resource): type spec for resource_data()
2021-11-23 10:04:31 +08:00
29ad6d215e
feat(resource): add metrics to emqx_resource
2021-11-23 10:04:31 +08:00
071c2c99e8
refactor(authn resources): add `emqx_resource` and `emqx_authn` tests
2021-11-22 21:08:04 +03:00
9c93ea0338
feat(connector): add API for /connectors_test
2021-11-22 17:35:33 +08:00
56e2a9741f
style: ensure newline at EOF for all files
2021-11-04 14:40:14 +01:00
2307bdd868
fix(emqx_resource): fix InstId type error
2021-10-25 13:43:28 +08:00
3524942766
fix: change all ~s in io format to ~ts
...
we should use unicode to replace latin1
2021-10-12 15:09:24 +08:00
ddfc010fdb
fix(resource): fix undefined function
2021-09-16 16:53:43 +08:00
deac54c847
chore(for editor): Add comments for Emacs major mode.
2021-09-16 15:43:30 +08:00
304c5613ac
refactor(resource): rename the emqx_resource:update/4 to recreate/4
2021-09-15 17:46:42 +08:00
d46241fe2f
feat(bridges): avoid clientid competition between bridges on different nodes
2021-09-10 14:21:17 +08:00
bfb2df37ce
refactor(bridge): rename emqx_data_bridge to emqx_bridge
2021-09-10 11:43:03 +08:00
c1c24af002
fix: dialyzer warning
2021-08-27 10:09:26 +08:00
73238ed81f
feat: emqx_resource support cluster_call
2021-08-26 17:23:39 +08:00
e6232665a3
refactor(emqx): ensure app shutdown and reboot order
...
make use of application dependency to ensure start order.
also the same dependency top-sorted for shutdown and reboot order
2021-08-04 08:40:03 +02:00
b10fb4e95d
refactor: unify all io:format calls to use the same macro
2021-08-01 09:36:59 +02:00
d0e9d4c4d5
fix(emqx_resource): raise error when failed to pick worker
2021-07-29 12:34:16 +02:00
178bafbabf
chore: rename required function to not_empty
...
Signed-off-by: zhanghongtong <rory-z@outlook.com>
2021-07-09 14:18:41 +08:00
c10d154dab
chore(connector): update schema file
...
Signed-off-by: zhanghongtong <rory-z@outlook.com>
2021-07-09 14:18:41 +08:00
704af9f3b1
The config handler phase2 ( #5052 )
...
* refator(config_handler): handle and validate the updates to raw_configs
* fix(hocon): update hocon to 0.8.0
* refactor(config_handler): check and apply envs only in top-level handler
* refactor(config_handler): update config from top level to bottom level
* refactor(emqx_data_bridge): move configs to emqx.conf
* fix(emqx_schema): remove the extra config path
* fix(config_handler): load the emqx.conf when starting emqx_config_handler
* fix(data_bridge): API not working
* feat(config_handler): save updated configs to emqx_override.conf
* fix(config_handler): cannot find the emqx.conf and emqx_override.conf
* fix(emqx_config): cannot find the correct path for etc dir
* fix(test): load load emqx_schema foreign refereced apps
* refactor(emqx_plugin): do not generate configs before load plugins
All configs (including the configs for plugins) now should go into
the `emqx.conf`.
* fix(tests): update the test cases for plugins
* fix(tests): don't include schema from apps when testing
* fix(tests): use emqx-ct-helper branch hocon
2021-06-25 11:47:18 +08:00
8978464269
change resource,connectors,data_bridges as normal apps ( #5034 )
2021-06-19 16:27:21 +08:00
54dedc8343
Revert "chore: mv emqx_connector to emqx_data_bridge"
...
This reverts commit d640e2ccfa
2021-06-17 19:11:10 +08:00
d640e2ccfa
chore: mv emqx_connector to emqx_data_bridge
2021-06-17 16:42:41 +08:00
0515ef6e45
feat(connector): add more connector ( #4985 )
2021-06-16 11:28:26 +08:00
444c721f93
fix(config): change handler name to emqx_data_bridge_config_handler
2021-06-16 11:27:34 +08:00
b11afc6997
fix(config): start the top level config handler at emqx bootup
2021-06-16 11:27:34 +08:00
9a78f812e1
fix(emqx_resource): fix schema schema definition
2021-06-15 12:35:08 +02:00
1de2f5cb2f
fix(emqx_data_bridge): update bridge failed
2021-06-08 21:46:52 +08:00
fdd0233103
feat(emqx_resource): update the demo.md
2021-06-08 21:46:52 +08:00
65dc2f2be8
feat(emqx_resource): remove the schema sugar from parse transformed code
2021-06-08 21:46:52 +08:00
bdf6374414
fix(emqx_resource): disable the debug print for parse-transformed code
2021-06-07 22:11:05 +08:00
149a54cef4
fix(emqx_resource): dialyzer issues
2021-06-07 22:08:21 +08:00
3c1c457697
fix(emqx_resource): call to undefined function hocon:richmap_to_map/1
2021-06-07 21:54:17 +08:00
16c0a5b80a
fix(emqx_resource): merge conflict
2021-06-07 21:25:19 +08:00
a37b3957e3
Merge branch 'master' into emqx_connector
2021-06-07 21:12:00 +08:00
4914b003ac
feat(emqx_resource): update the unused APIs
2021-06-07 17:20:34 +08:00
2ff92d2880
feat(emqx_data_bridge): add HTTP APIs for data bridge
2021-06-07 16:12:06 +08:00
4f451ee862
feat(emqx_resource): delete API generation from parse transfrom
2021-06-05 15:30:21 +08:00
f1552f4f4f
feat(emqx_data_bridge): create emqx_data_bridge
2021-06-04 23:47:16 +08:00
9617b65d52
fix(dialyzer): incorrect function spec
2021-06-03 13:35:57 +08:00
03519d7e61
fix(emqx_resource): HTTP APIs for emqx_resource not working
2021-06-03 13:35:57 +08:00
3da62e59d6
fix(emqx_connector): the HTTP API for emqx_connector_mysql
2021-06-03 01:24:30 +08:00
49c5edce2e
fix(emqx_connector): start emqx_connector_mysql failed
2021-06-02 17:28:47 +08:00
5d52ce044d
feat(emqx_resource): read and save configs from and to file
2021-06-02 01:58:32 +08:00
9cb2facfe1
chore(emqx_resource): unify dependency hocon to root rebar.config
2021-06-01 21:05:04 +09:00
e7ffa07a1a
feat(emqx_connector): load connectors from emqx_connector.conf
2021-05-31 22:49:42 +08:00
Zaiming (Stone) Shi