yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,613 Commits 393 Branches 669 Tags 74 MiB
24f476e35f
Commit Graph

121 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
ada68e12e6 fix(kafka): change default value for memory_overload_protection to false 2023-02-22 13:46:51 +01:00
Zaiming (Stone) Shi
9316690c29 fix(schema): binary string for default values 
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
 2023-02-21 09:09:51 +01:00
Zaiming (Stone) Shi
53ecfb98d0 chore: fix stale code comments 2023-02-21 09:09:51 +01:00
Erik Timan
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts 
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
 2023-02-20 10:05:47 +01:00
Erik Timan
caacc08ff9 chore: bump app VSN 2023-02-17 15:28:27 +01:00
Erik Timan
843ed464d5 fix: remove all mentions of UDP support for InfluxDB EE bridge 
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
 2023-02-17 10:47:57 +01:00
firest
81adb02402 chore: bump apps version 2023-02-14 22:04:29 +08:00
firest
530b21268a feat(bridges): add TDengine 2023-02-14 22:04:23 +08:00
zhongwencool
ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async 
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
 2023-02-02 21:08:01 +08:00
Zaiming (Stone) Shi
0eb554a62e fix(kafka): check Kafka partition leader connectivity 2023-02-02 12:04:17 +01:00
Zhongwen Deng
e776751860 fix: delete enable_batch from redis bridge 2023-02-02 17:37:18 +08:00
Zhongwen Deng
22c3f50020 fix: add query_mode_sync_only for mysql pgsql redis mongodb bridge 2023-02-02 17:37:18 +08:00
Zhongwen Deng
dcc6bd9c21 fix: mongodb bridge'd default connect_opt should be async, not sync 2023-02-02 17:37:18 +08:00
Zaiming (Stone) Shi
d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Andrew Mayorov
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching 
feat(redis): disable batching in redis_cluster bridges
 2023-01-27 15:27:57 +04:00
Andrew Mayorov
26fcaecad7
fix(redis): disable batching in redis_cluster bridges 
Through configuration subsystem.
 2023-01-25 17:28:11 +03:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Kjell Winblad
8c52264c41 fix: atom leak when doing Kafka bridge dry-run 
A new atom was created every time one did a dry run of a Kafka bridge
(that is, clicking the Test button in the settings dialog for the
bridge).

After this fix, we will only create a new atom when a bridge with a new
name is created. This should be acceptable as bridges with new names are
created relatively rarely and it seems to be useful to have an unique
atom for every Wolff producer.

Fixes: https://emqx.atlassian.net/browse/EMQX-8739
 2023-01-20 11:50:20 +01:00
Thales Macedo Garitezi
a5424959c6 refactor: avoid operator precedence bugs 2023-01-18 16:15:34 -03:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter 
fix(schema): add password converter to ensure its binary() type
 2023-01-18 15:09:05 +01:00
Zaiming (Stone) Shi
2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy 
0118 fix key dispatch kafka produce strategy
 2023-01-18 13:52:49 +01:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Zaiming (Stone) Shi
3872c4451f fix(kafka): olp fix was accidentally deleted in 0fd8880d0a 2023-01-18 10:52:29 +01:00
Thales Macedo Garitezi
4c25be8a2c chore(influxdb): remove deprecated value from example 2023-01-17 16:48:48 -03:00
Stefan Strigler
5fc31dcd9e fix: use atom keys in example to make sure they exist 2023-01-17 15:29:19 +01:00
Zaiming (Stone) Shi
4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
Zaiming (Stone) Shi
47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Thales Macedo Garitezi
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50 
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
 2023-01-13 10:23:49 -03:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Thales Macedo Garitezi
ed2be4d086 feat(mongo): expose buffer worker opts to the bridge frontend (5.0) 2023-01-12 15:23:41 -03:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Xinyu Liu
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field 
refactor: remove the auto_reconnect field
 2023-01-12 11:11:00 +08:00
Thales Macedo Garitezi
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50 
feat(mongodb): add `payload_template` field for bridge (e5.0)
 2023-01-11 11:14:46 -03:00
Shawn
3e9c4f444f refactor: remove the auto_reconnect field 2023-01-11 21:47:06 +08:00
Zaiming (Stone) Shi
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection 
0101 authz schema union member selection
 2023-01-09 22:17:51 +01:00
Thales Macedo Garitezi
b9f258b737 feat(mongodb): add payload_template field for bridge (e5.0) 
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
 2023-01-09 13:38:11 -03:00
firest
e08d6dbc9b feat(bridges): add timescale && matrix bridges 2023-01-09 23:04:29 +08:00
Zaiming (Stone) Shi
e52f9d5920 refactor: use union member type selector for authz sources 2023-01-09 14:26:16 +01:00
firest
f7b50c56da feat(bridges): integrate PostgreSQL into bridges 
support both simple and batch query
 2023-01-06 11:47:22 +08:00
Thales Macedo Garitezi
af31ed4264 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-05 10:16:00 -03:00
Thales Macedo Garitezi
fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00
Thales Macedo Garitezi
0fd8880d0a fix(kafka_producer): avoid multiplication of metrics when bridge is recreated 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi
8e59319bfe fix(kafka_producer): fix message loss when kafka connection is down 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi
4feaf5916f fix(mongodb): add type and name fields to the get/post API docs 
https://emqx.atlassian.net/browse/EMQX-8643

Adds thoses fields to MongoDB API docs so the frontend may use them to
render properly.
 2023-01-03 11:46:58 -03:00
Thales Macedo Garitezi
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50 
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
 2023-01-02 10:56:47 -03:00
Thales Macedo Garitezi
69ac6b9e0d fix(kafka): fix handling of dropped.queue_full event from wolff 
https://emqx.atlassian.net/browse/EMQX-8530

cd20a37e65/src/wolff_producer.erl (L772-L773)

Wolff emits 2 events related to dropped messages when replayq reports
overflow.  Since in EMQX's side we also bump `dropped` when
`dropped_queue_full` happens, that was leading to wrong metrics.
 2023-01-02 09:23:38 -03:00