yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,086 Commits 393 Branches 684 Tags 74 MiB
bafc430184
Commit Graph

297 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
bafc430184
Merge pull request #9837 from zmstone/0123-fix-resource-worker-leak-influxdb 
0123 fix resource worker leak influxdb
 2023-01-26 09:14:38 +01:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Zaiming (Stone) Shi
feb0238911 fix(influxdb): ensure client is stopped before returning error 2023-01-24 20:35:02 +01:00
Erik Timan
7603ab490b fix(emqx_ee_connector): fix passing of influxdb ssl opts 2023-01-24 14:30:40 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Kjell Winblad
8c52264c41 fix: atom leak when doing Kafka bridge dry-run 
A new atom was created every time one did a dry run of a Kafka bridge
(that is, clicking the Test button in the settings dialog for the
bridge).

After this fix, we will only create a new atom when a bridge with a new
name is created. This should be acceptable as bridges with new names are
created relatively rarely and it seems to be useful to have an unique
atom for every Wolff producer.

Fixes: https://emqx.atlassian.net/browse/EMQX-8739
 2023-01-20 11:50:20 +01:00
Thales Macedo Garitezi
a5424959c6 refactor: avoid operator precedence bugs 2023-01-18 16:15:34 -03:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov
f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test 
chore(ee bridge): fix Redis bridge test flakyness
 2023-01-18 20:56:13 +02:00
Zaiming (Stone) Shi
caf50d1b35
Merge pull request #9792 from savonarola/redis-bridge-examples 
chore(bridges): add more helpful descriptions
 2023-01-18 18:01:35 +01:00
Ilya Averyanov
f9c3115cd4 chore(bridges): add more helpful descriptions 
Add more helpful descriptions for `command_template` option
of Redis bridge.
 2023-01-18 18:17:08 +02:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter 
fix(schema): add password converter to ensure its binary() type
 2023-01-18 15:09:05 +01:00
Zaiming (Stone) Shi
2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy 
0118 fix key dispatch kafka produce strategy
 2023-01-18 13:52:49 +01:00
Ilya Averyanov
f6fbbf3ee3 chore(bridges): reduce Redis bridge flakyness 2023-01-18 14:34:11 +02:00
Zaiming (Stone) Shi
8f275a66d0 test: add coverage for key_dispatch partition strategy 2023-01-18 11:47:37 +01:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Zaiming (Stone) Shi
3872c4451f fix(kafka): olp fix was accidentally deleted in 0fd8880d0a 2023-01-18 10:52:29 +01:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4c25be8a2c chore(influxdb): remove deprecated value from example 2023-01-17 16:48:48 -03:00
Stefan Strigler
f37b3e4bc4 test: test against bridges_probe API 2023-01-17 15:29:19 +01:00
Stefan Strigler
5fc31dcd9e fix: use atom keys in example to make sure they exist 2023-01-17 15:29:19 +01:00
Zaiming (Stone) Shi
4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
Shawn
1c7fe91a91 fix: typos in emqx_ee_bridge_mongodb.conf 2023-01-17 16:33:54 +08:00
JimMoen
40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zaiming (Stone) Shi
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs 
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
 2023-01-16 13:37:29 +01:00
Zaiming (Stone) Shi
47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Zaiming (Stone) Shi
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions 
0112 docs fix some descriptions
 2023-01-16 09:56:56 +01:00
Zaiming (Stone) Shi
91c5a89985 test: wait for redis connected state 
the case is sometimes flaky because the health check sometimes
return connecting
 2023-01-14 18:33:55 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Erik Timan
7a17fb7308 test(emqx_ee_bridge): fix bridge enable/disable in kafka producer suite 2023-01-13 14:40:54 +01:00
Thales Macedo Garitezi
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50 
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
 2023-01-13 10:23:49 -03:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Zaiming (Stone) Shi
522e091dd2 docs: chane cache to buffer for Kafka bridge description 2023-01-13 07:46:23 +01:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Thales Macedo Garitezi
ed2be4d086 feat(mongo): expose buffer worker opts to the bridge frontend (5.0) 2023-01-12 15:23:41 -03:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Xinyu Liu
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field 
refactor: remove the auto_reconnect field
 2023-01-12 11:11:00 +08:00
Thales Macedo Garitezi
2d5208fc29
Merge pull request #9485 from thalesmg/refactor-doc-gen-50-mkIII 
feat(docs): add tags to schemas
 2023-01-11 16:36:38 -03:00
Thales Macedo Garitezi
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50 
feat(mongodb): add `payload_template` field for bridge (e5.0)
 2023-01-11 11:14:46 -03:00
Shawn
3e9c4f444f refactor: remove the auto_reconnect field 2023-01-11 21:47:06 +08:00
Thales Macedo Garitezi
a9d0143d5f chore(license): change key type to binary() 
So that the `raw_default` field in the schema JSON renders properly.
 2023-01-11 09:10:04 -03:00
Thales Macedo Garitezi
48e1ba4832 feat(docs): add tags to schemas 
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
 2023-01-11 09:10:03 -03:00
Thales Macedo Garitezi
f3cc722172
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-10 09:06:12 -03:00
Zaiming (Stone) Shi
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection 
0101 authz schema union member selection
 2023-01-09 22:17:51 +01:00