yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,613 Commits 393 Branches 669 Tags 74 MiB
24f476e35f
Commit Graph

104 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
24f476e35f test: add README to influxdb test script 2023-02-28 19:38:43 +01:00
Andrew Mayorov
9cbe64a132
fix(test): make strings json-friendly in kafka testsuite 2023-02-24 15:05:20 +03:00
Ivan Dyachkov
c869eff6e8 test(kafka): disable overload protection in 2 more places 2023-02-20 21:38:14 +01:00
Ivan Dyachkov
9d30a52910 test: disable replayq memory overload protection 2023-02-20 19:29:26 +01:00
firest
0420e9acb5 test(bridges): add test cases for TDEngine 2023-02-14 22:04:29 +08:00
Stefan Strigler
c7535fce56 test: fix test expecting 200 instead of 204 2023-02-08 09:14:35 +01:00
Zhongwen Deng
1c9035d24c test: remove async from redis ct 2023-02-02 17:37:18 +08:00
Ilya Averyanov
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover 
fix: fix redis cluster resource recovery
 2023-02-01 10:38:52 +02:00
Ilya Averyanov
fce1e74c3d fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00
Zaiming (Stone) Shi
d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Andrew Mayorov
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching 
feat(redis): disable batching in redis_cluster bridges
 2023-01-27 15:27:57 +04:00
Stefan Strigler
7d18128ba9 test: async write can return noproc 2023-01-27 11:43:51 +01:00
Stefan Strigler
2d62de5188 test: fix expected result from timeout error 2023-01-27 11:43:48 +01:00
Zaiming (Stone) Shi
f6b3b930b0 chore: improve a error log 2023-01-26 14:21:27 +01:00
Andrew Mayorov
26fcaecad7
fix(redis): disable batching in redis_cluster bridges 
Through configuration subsystem.
 2023-01-25 17:28:11 +03:00
Andrew Mayorov
903a77b471
test(redis): ensure batch query hit different cluster shards 
This will inevitably fail: it's not generally possible to update
different keys through the same cluster connection, one or more
update will fail with `MOVED` status. This testcase should serve
as a regression test later.
 2023-01-25 15:33:05 +03:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov
f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test 
chore(ee bridge): fix Redis bridge test flakyness
 2023-01-18 20:56:13 +02:00
Zaiming (Stone) Shi
2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy 
0118 fix key dispatch kafka produce strategy
 2023-01-18 13:52:49 +01:00
Ilya Averyanov
f6fbbf3ee3 chore(bridges): reduce Redis bridge flakyness 2023-01-18 14:34:11 +02:00
Zaiming (Stone) Shi
8f275a66d0 test: add coverage for key_dispatch partition strategy 2023-01-18 11:47:37 +01:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Stefan Strigler
f37b3e4bc4 test: test against bridges_probe API 2023-01-17 15:29:19 +01:00
Zaiming (Stone) Shi
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs 
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
 2023-01-16 13:37:29 +01:00
Zaiming (Stone) Shi
47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Zaiming (Stone) Shi
91c5a89985 test: wait for redis connected state 
the case is sometimes flaky because the health check sometimes
return connecting
 2023-01-14 18:33:55 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Erik Timan
7a17fb7308 test(emqx_ee_bridge): fix bridge enable/disable in kafka producer suite 2023-01-13 14:40:54 +01:00
Thales Macedo Garitezi
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50 
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
 2023-01-13 10:23:49 -03:00
Thales Macedo Garitezi
ed2be4d086 feat(mongo): expose buffer worker opts to the bridge frontend (5.0) 2023-01-12 15:23:41 -03:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Thales Macedo Garitezi
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50 
feat(mongodb): add `payload_template` field for bridge (e5.0)
 2023-01-11 11:14:46 -03:00
Zaiming (Stone) Shi
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection 
0101 authz schema union member selection
 2023-01-09 22:17:51 +01:00
Thales Macedo Garitezi
b9f258b737 feat(mongodb): add payload_template field for bridge (e5.0) 
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
 2023-01-09 13:38:11 -03:00
lafirest
20f49b903f
Merge pull request #9702 from lafirest/feat/timescale_matrix 
feat(bridges): add timescale && matrix bridges
 2023-01-10 00:06:17 +08:00
firest
062f14bd65 test(bridges): add timescale && matrix test cases 2023-01-09 23:04:36 +08:00
Zaiming (Stone) Shi
c3635f537a ci: wait for redis in emqx_ee_bridge_redis_SUITE 2023-01-09 14:39:56 +01:00
Zaiming (Stone) Shi
0697c692ed refactor: mongo_type and redis_type are not mandatory 
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:

[
    {mongo_type = rs,
     ...
    }
    {another backend config}
]

however the brdige types are maps like, for example:

mongodb_rs.$name {
    mongo_type = rs
    ...
}

in which case, the mongo_type is not required.

in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.

However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
ffb09f0c4d test: rename a test option name to avoid clashing with prod config name 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
e52f9d5920 refactor: use union member type selector for authz sources 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip 
test: do not use tc_user_skip for test cases
 2023-01-09 09:02:06 +01:00
firest
ea405fe55d test(bridges): add test case for the PostgreSQL backend 2023-01-06 11:56:28 +08:00