yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,020 Commits 393 Branches 684 Tags 74 MiB
f6fbbf3ee3
Commit Graph

278 Commits

Author SHA1 Message Date
Ilya Averyanov
f6fbbf3ee3 chore(bridges): reduce Redis bridge flakyness 2023-01-18 14:34:11 +02:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4c25be8a2c chore(influxdb): remove deprecated value from example 2023-01-17 16:48:48 -03:00
Stefan Strigler
f37b3e4bc4 test: test against bridges_probe API 2023-01-17 15:29:19 +01:00
Stefan Strigler
5fc31dcd9e fix: use atom keys in example to make sure they exist 2023-01-17 15:29:19 +01:00
Shawn
1c7fe91a91 fix: typos in emqx_ee_bridge_mongodb.conf 2023-01-17 16:33:54 +08:00
JimMoen
40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zaiming (Stone) Shi
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs 
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
 2023-01-16 13:37:29 +01:00
Zaiming (Stone) Shi
47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Zaiming (Stone) Shi
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions 
0112 docs fix some descriptions
 2023-01-16 09:56:56 +01:00
Zaiming (Stone) Shi
91c5a89985 test: wait for redis connected state 
the case is sometimes flaky because the health check sometimes
return connecting
 2023-01-14 18:33:55 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Erik Timan
7a17fb7308 test(emqx_ee_bridge): fix bridge enable/disable in kafka producer suite 2023-01-13 14:40:54 +01:00
Thales Macedo Garitezi
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50 
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
 2023-01-13 10:23:49 -03:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Zaiming (Stone) Shi
522e091dd2 docs: chane cache to buffer for Kafka bridge description 2023-01-13 07:46:23 +01:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Thales Macedo Garitezi
ed2be4d086 feat(mongo): expose buffer worker opts to the bridge frontend (5.0) 2023-01-12 15:23:41 -03:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Xinyu Liu
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field 
refactor: remove the auto_reconnect field
 2023-01-12 11:11:00 +08:00
Thales Macedo Garitezi
2d5208fc29
Merge pull request #9485 from thalesmg/refactor-doc-gen-50-mkIII 
feat(docs): add tags to schemas
 2023-01-11 16:36:38 -03:00
Thales Macedo Garitezi
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50 
feat(mongodb): add `payload_template` field for bridge (e5.0)
 2023-01-11 11:14:46 -03:00
Shawn
3e9c4f444f refactor: remove the auto_reconnect field 2023-01-11 21:47:06 +08:00
Thales Macedo Garitezi
a9d0143d5f chore(license): change key type to binary() 
So that the `raw_default` field in the schema JSON renders properly.
 2023-01-11 09:10:04 -03:00
Thales Macedo Garitezi
48e1ba4832 feat(docs): add tags to schemas 
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
 2023-01-11 09:10:03 -03:00
Thales Macedo Garitezi
f3cc722172
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-10 09:06:12 -03:00
Zaiming (Stone) Shi
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection 
0101 authz schema union member selection
 2023-01-09 22:17:51 +01:00
Thales Macedo Garitezi
b9f258b737 feat(mongodb): add payload_template field for bridge (e5.0) 
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
 2023-01-09 13:38:11 -03:00
lafirest
20f49b903f
Merge pull request #9702 from lafirest/feat/timescale_matrix 
feat(bridges): add timescale && matrix bridges
 2023-01-10 00:06:17 +08:00
firest
062f14bd65 test(bridges): add timescale && matrix test cases 2023-01-09 23:04:36 +08:00
firest
e08d6dbc9b feat(bridges): add timescale && matrix bridges 2023-01-09 23:04:29 +08:00
Zaiming (Stone) Shi
c3635f537a ci: wait for redis in emqx_ee_bridge_redis_SUITE 2023-01-09 14:39:56 +01:00
Zaiming (Stone) Shi
0697c692ed refactor: mongo_type and redis_type are not mandatory 
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:

[
    {mongo_type = rs,
     ...
    }
    {another backend config}
]

however the brdige types are maps like, for example:

mongodb_rs.$name {
    mongo_type = rs
    ...
}

in which case, the mongo_type is not required.

in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.

However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
ffb09f0c4d test: rename a test option name to avoid clashing with prod config name 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
e52f9d5920 refactor: use union member type selector for authz sources 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi
e498010f23 chore: update default license (100 conns) 2023-01-09 09:11:09 +01:00
Zaiming (Stone) Shi
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip 
test: do not use tc_user_skip for test cases
 2023-01-09 09:02:06 +01:00
firest
ea405fe55d test(bridges): add test case for the PostgreSQL backend 2023-01-06 11:56:28 +08:00
firest
f7b50c56da feat(bridges): integrate PostgreSQL into bridges 
support both simple and batch query
 2023-01-06 11:47:22 +08:00
Zaiming (Stone) Shi
5f12cdff6c ci: should not skip tests in github actions 2023-01-05 22:47:08 +01:00
Zaiming (Stone) Shi
fc12a8c4c8 test: do not use tc_user_skip for test cases 2023-01-05 22:47:06 +01:00
Thales Macedo Garitezi
af31ed4264 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-05 10:16:00 -03:00
Thales Macedo Garitezi
fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00
Thales Macedo Garitezi
0fd8880d0a fix(kafka_producer): avoid multiplication of metrics when bridge is recreated 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi
8e59319bfe fix(kafka_producer): fix message loss when kafka connection is down 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi
5bd9f110d6 test: attempt to reduce flakiness 2023-01-05 10:11:59 -03:00