5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
871ee90b3e
refactor(kafka_bridge): move kafka bridge into its own app
...
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
677b76afce
refactor: replace hidden => true with importance => hidden
2023-04-03 13:13:43 +02:00
9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
...
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
...
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
9bbca07943
Merge pull request #10298 from emqx/release-50
...
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
ee5b9d7e0a
fix(tdengine): remove the redundant table name in SQL template
2023-03-30 10:34:10 +08:00
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
624c168dde
fix: hide nodelay Kafka client socket option
...
also fix a few descriptions
2023-03-29 11:42:17 +02:00
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
...
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
1c8333030c
fix(kafka_producer): add back `is_buffer_supported` callback
...
Fixes https://emqx.atlassian.net/browse/EMQX-9366
This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
d07987288a
chore: add some example annotations for config importance level
2023-03-28 14:29:24 +02:00
69fc1123ee
refactor: change enum constructors and improve docs
2023-03-27 17:30:17 -03:00
5cf09209cd
feat: tie `offset_reset_policy` and `begin_offset` together
...
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:
1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.
2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
7085a2d6d7
chore: hidden the password field
...
ref: https://github.com/emqx/emqx/issues/10222
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 09:47:51 +08:00
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
b37f186142
Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master
2023-03-24 21:47:15 +01:00
5bdc71af6f
docs(kafka_consumer): make specific labels for key/value encoding modes
...
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
9b63bdc1e0
chore: apply review suggestions
...
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
ac41c7e653
chore: format codes
2023-03-23 11:53:26 +08:00
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
208813330a
fix: remove metrics from rocketmq example
2023-03-22 10:59:20 -03:00
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
...
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
65c2da7ef5
Merge remote-tracking branch 'ce/master' into cassa
2023-03-22 09:30:50 +08:00
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
27b8445337
fix: add inflight window setting to the clickhouse bridge
...
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 17:14:03 +01:00
3a46681dde
feat: handle escaped characters in InfluxDB data bridge write_syntax
...
Closes: EMQX-7834
2023-03-20 16:42:23 +02:00
d1689f6957
chore: correct api examples
...
follow https://github.com/emqx/emqx/pull/10114
2023-03-20 17:03:05 +08:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
12942b676d
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-20 09:50:27 +08:00
75c1592e7c
chore: make static_checks happy
2023-03-17 11:34:52 +08:00
c0a216a740
feat(bridge): support cassandra bridge
2023-03-17 11:34:48 +08:00
0587e0d3f3
refactor(kafka_consumer): change the generated consumer group id
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
fc5dfa108a
fix(kafka_consumer): rename `force_utf8` to `none`
...
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
2023-03-16 13:43:01 -03:00
37a35b22df
docs(kafka): improve kafka consumer/producer API examples
2023-03-16 13:43:01 -03:00
53979b6261
feat(kafka_consumer): support multiple topic mappings, payload
...
templates and key/value encodings
Added after feedback from the product team.
2023-03-16 13:43:01 -03:00
f31f15e44e
chore(kafka_producer): make schema changes more backwards compatible
...
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
c182a4053e
fix(kafka_consumer): avoid leaking atoms in bridge probe API
2023-03-16 13:43:01 -03:00
27dfd98f46
fix(kafka): fix api methods schemas and bridge type enum
2023-03-16 13:43:01 -03:00
fecc08bb8f
refactor(kafka_consumer): apply review suggestions
2023-03-16 13:43:01 -03:00
2a3aef92bb
refactor(kafka_consumer): redact error info
2023-03-16 13:43:01 -03:00
e1fdd041b3
feat(kafka_consumer): add `offset_commit_interval_seconds` kafka parameter
2023-03-16 13:43:01 -03:00
65c15b3fae
refactor(kafka_consumer): move subscriber startup logic to separate fn
2023-03-16 13:43:01 -03:00
969fa03ecc
feat: implement kafka consumer
2023-03-16 13:43:01 -03:00
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
b8b1a78025
feat(bridges): integrate the DynamoDB into data bridges
2023-03-08 11:13:38 +08:00
18ab7ed197
chore: bump app vsns
2023-03-06 15:31:28 -03:00
9825998207
chore(gcp_pubsub): just deprecate `request_timeout` instead of removing
2023-03-06 10:12:38 -03:00
9b087a21f5
fix(gcp_pubsub): remove conflicting `request_timeout` option
...
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
2023-03-06 10:12:38 -03:00
c24a84e370
docs: fix spelling
2023-03-03 12:06:18 +01:00
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
ada68e12e6
fix(kafka): change default value for memory_overload_protection to false
2023-02-22 13:46:51 +01:00
9316690c29
fix(schema): binary string for default values
...
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
2023-02-21 09:09:51 +01:00
53ecfb98d0
chore: fix stale code comments
2023-02-21 09:09:51 +01:00
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
...
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
caacc08ff9
chore: bump app VSN
2023-02-17 15:28:27 +01:00
843ed464d5
fix: remove all mentions of UDP support for InfluxDB EE bridge
...
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
81adb02402
chore: bump apps version
2023-02-14 22:04:29 +08:00
530b21268a
feat(bridges): add TDengine
2023-02-14 22:04:23 +08:00
ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async
...
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
2023-02-02 21:08:01 +08:00
0eb554a62e
fix(kafka): check Kafka partition leader connectivity
2023-02-02 12:04:17 +01:00
e776751860
fix: delete enable_batch from redis bridge
2023-02-02 17:37:18 +08:00
22c3f50020
fix: add query_mode_sync_only for mysql pgsql redis mongodb bridge
2023-02-02 17:37:18 +08:00
dcc6bd9c21
fix: mongodb bridge'd default connect_opt should be async, not sync
2023-02-02 17:37:18 +08:00
d47941601d
refactor(buffer_worker): rename trace points
2023-01-28 11:52:11 +01:00
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching
...
feat(redis): disable batching in redis_cluster bridges
2023-01-27 15:27:57 +04:00
26fcaecad7
fix(redis): disable batching in `redis_cluster` bridges
...
Through configuration subsystem.
2023-01-25 17:28:11 +03:00
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
8c52264c41
fix: atom leak when doing Kafka bridge dry-run
...
A new atom was created every time one did a dry run of a Kafka bridge
(that is, clicking the Test button in the settings dialog for the
bridge).
After this fix, we will only create a new atom when a bridge with a new
name is created. This should be acceptable as bridges with new names are
created relatively rarely and it seems to be useful to have an unique
atom for every Wolff producer.
Fixes: https://emqx.atlassian.net/browse/EMQX-8739
2023-01-20 11:50:20 +01:00
a5424959c6
refactor: avoid operator precedence bugs
2023-01-18 16:15:34 -03:00
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy
...
0118 fix key dispatch kafka produce strategy
2023-01-18 13:52:49 +01:00
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
3872c4451f
fix(kafka): olp fix was accidentally deleted in 0fd8880d0a
2023-01-18 10:52:29 +01:00
4c25be8a2c
chore(influxdb): remove deprecated value from example
2023-01-17 16:48:48 -03:00
5fc31dcd9e
fix: use atom keys in example to make sure they exist
2023-01-17 15:29:19 +01:00
4a7e74f5d6
fix(schema): add password converter to ensure its binary() type
2023-01-17 11:12:17 +01:00
47414e0d53
docs: improve kafka key and value field description
2023-01-16 11:32:09 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50
...
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
2023-01-13 10:23:49 -03:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
ed2be4d086
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
2023-01-12 15:23:41 -03:00
1711823487
refactor: remove default value of timestamp field of influxdb
2023-01-12 22:25:44 +08:00
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field
...
refactor: remove the auto_reconnect field
2023-01-12 11:11:00 +08:00
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50
...
feat(mongodb): add `payload_template` field for bridge (e5.0)
2023-01-11 11:14:46 -03:00
3e9c4f444f
refactor: remove the auto_reconnect field
2023-01-11 21:47:06 +08:00
JimMoen