437096985b
test: fix SUITE case `t_create_disconnected/1`
2023-04-14 10:02:47 +08:00
27fdf644aa
fix(sqlserver): let a single query be a length 1 batch
...
Need to be fixed:
`emqx_plugin_libs_rule:proc_tmpl/2` won't add single quotes for string
2023-04-14 10:02:46 +08:00
a379d909bf
test: use type VARCHAR to use utf8 encoding in sqlserver
2023-04-14 10:02:46 +08:00
c366267b0f
test: MS SQL Server data bridge
2023-04-14 10:02:46 +08:00
5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
8e62afecdd
chore: suite module rename
2023-04-14 09:57:00 +08:00
8ceeafb0de
chore: fix file license in lib-ee
2023-04-14 09:56:59 +08:00
9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
...
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
601564fa0a
Merge pull request #10324 from kjellwinblad/kjell/fix/EMQX-9374
...
fix: check Clickhouse connection after creation
2023-04-13 16:31:35 +02:00
0cec52d5e4
fix: check Clickhouse connection after creation
...
When pressing reconnect on a badly configured Clickhouse bridge in the
dashboard, no error message was shown. This commit fixes this issue by
testing the connection after creation and returning an error tuple if
the connection is not working.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
2023-04-13 10:09:35 +02:00
a4d9234b24
test(dynamo): remove the flaky test case
2023-04-13 12:02:22 +08:00
871ee90b3e
refactor(kafka_bridge): move kafka bridge into its own app
...
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
...
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
d16b5c40d6
Merge pull request #10356 from HJianBo/async-batch-cassa
...
feat: support async and batch callback for cassandra connector
2023-04-12 20:59:36 +08:00
9560fdc5a2
chore: typo fixes
2023-04-12 14:16:40 +08:00
914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50
...
feat: implement schema registry for 5.0 (avro)
2023-04-11 16:46:27 -03:00
97b94b8882
feat: implement schema registry for 5.0 (avro)
...
Part of https://emqx.atlassian.net/browse/EMQX-9251
This ports part of the Schema Registry app from 4.x to 5.0. Here,
only support for Avro is added. Subsequent PRs will follow to add
support for other formats.
2023-04-11 14:50:45 -03:00
c50326360f
chore: update ecql version
2023-04-10 18:03:58 +08:00
30bdffe318
feat: support async and batch callback for cassandra connector
2023-04-10 15:08:10 +08:00
0d86ef8d3a
fix(ft-s3): use AWS4 signed urls for S3 export URIs
...
This will ensure that the S3 export URIs are valid in all AWS regions.
2023-04-07 22:15:01 +03:00
6fbc7dc822
fix(ft): remove s3 roots from ee schema
2023-04-07 17:36:47 +03:00
d21daae7ae
fix(ft): fix dynamic function calls
2023-04-07 17:36:47 +03:00
eae2478678
fix(ft): add descriptions for missing ft schema fields
2023-04-07 17:36:47 +03:00
2f1970adbc
fix(ft): make ee schema aware of ee fields
2023-04-07 17:36:47 +03:00
18aa39f892
fix(ft): add emqx_ft_schema to ee schemas
2023-04-07 17:36:47 +03:00
c7865e5eae
fix(ft-s3): fix dependency dependencies
2023-04-07 17:36:45 +03:00
71965d90e4
fix(ft-s3): synchronize erlcloud version across apps
2023-04-07 17:26:53 +03:00
33100ecca6
feat: implement schema registry for 5.0 (avro)
...
Part of https://emqx.atlassian.net/browse/EMQX-9251
This ports part of the Schema Registry app from 4.x to 5.0. Here,
only support for Avro is added. Subsequent PRs will follow to add
support for other formats.
2023-04-06 16:28:45 -03:00
4c24b08244
fix(rule_action): fix metrics for bridges returning `async_return`
...
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50
...
docs(kafka_producer): update ZH translation of field (rv5.0)
2023-04-06 16:13:41 +02:00
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
0b6fd7fe14
fix(buffer_worker): check request timeout and health check interval
...
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`
Fixes https://emqx.atlassian.net/browse/EMQX-9099
Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`. This had the problem that,
with default values, `health_check_interval = request_timeout`. This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.
Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
ed31b735a8
Merge pull request #10220 from lafirest/bump/hackney
...
chore: bump hackney version
2023-04-04 10:35:09 +08:00
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
...
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
8b5a717a1f
test(peer): increase init and startup timeout for peer nodes
...
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
ec1871ffde
test(janitor): catch each callback invocation
2023-04-03 10:20:19 -03:00
51eec32d6b
docs: improve descriptions
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-03 09:11:14 -03:00
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1
...
Doc: Refined Chinese UI
2023-04-03 13:31:20 +02:00
677b76afce
refactor: replace hidden => true with importance => hidden
2023-04-03 13:13:43 +02:00
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
c8dca74b18
chore: bump hackney version
2023-04-03 18:41:49 +08:00
9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
...
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
...
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
...
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label
...
docs: make clickhouse config label for Server URL more concise
2023-03-31 16:17:26 +02:00
945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb
...
test: fix flaky influxdb test
2023-03-31 15:36:12 +02:00
9bbca07943
Merge pull request #10298 from emqx/release-50
...
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
aabbbf5deb
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-31 09:07:00 -03:00
6f6d12f930
docs: better Chinese labels
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-03-31 09:39:04 +02:00
e808fef1e4
feat: (MongoDB bridge) use ${var} syntax for MongoDB collection
...
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
2023-03-30 17:49:56 +02:00
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
81a104690d
test: fix flaky influxdb test
2023-03-30 16:19:22 +02:00
8c4d211b54
fix: on_get_status clickhouse just reporting connecting
...
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-30 15:10:03 +02:00
91a784134c
docs: update Chinese version of batch value separator label
2023-03-30 14:55:58 +02:00
aefa0e5ffb
docs: make clickhouse config label for Server URL more concise
2023-03-30 14:55:22 +02:00
dfc9141da6
docs: Refined Chinese UI
...
Also updated the English hints according to the refined Chinese according to Guowei's comments.
Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
2023-03-30 14:50:24 +02:00
b3c6ae99d2
Merge pull request #10269 from thalesmg/tdengine-required-password-rv50
...
fix(tdengine): make password a required field (rv5.0)
2023-03-30 09:25:38 -03:00
07e1f2e440
docs(kafka_producer): update ZH translation of field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9411
2023-03-30 09:18:16 -03:00
808bf1d232
Merge pull request #10270 from kjellwinblad/kjell/clickhouse/on_get_status
...
fix: on_get_status clickhouse just reporting connecting
2023-03-30 10:57:48 +02:00
ee5b9d7e0a
fix(tdengine): remove the redundant table name in SQL template
2023-03-30 10:34:10 +08:00
94726eb7f1
docs: refine zh translations
2023-03-29 15:07:19 +02:00
b2b9732edb
fix(tdengine): make password a required field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9395
```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
2023-03-29 10:07:02 -03:00
c693851740
fix: on_get_status clickhouse just reporting connecting
...
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-29 14:26:08 +02:00
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
624c168dde
fix: hide nodelay Kafka client socket option
...
also fix a few descriptions
2023-03-29 11:42:17 +02:00
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
64faccf50b
test: fix flaky kafka consumer test
2023-03-28 14:50:55 -03:00
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
...
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
1c8333030c
fix(kafka_producer): add back `is_buffer_supported` callback
...
Fixes https://emqx.atlassian.net/browse/EMQX-9366
This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
d07987288a
chore: add some example annotations for config importance level
2023-03-28 14:29:24 +02:00
69fc1123ee
refactor: change enum constructors and improve docs
2023-03-27 17:30:17 -03:00
5cf09209cd
feat: tie `offset_reset_policy` and `begin_offset` together
...
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:
1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.
2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
da5e6f3d0a
test: test with only one Kafka partition for bad config recover test
2023-03-27 17:38:34 +02:00
0d36b179c0
docs: fix kafka offset reset policy config description
2023-03-27 11:22:01 -03:00
7a6494cda6
Merge pull request #10242 from zmstone/0326-fix-logging-avoid-using-pid-as-log-data-field
...
fix: avoid uisng 'pid' as log data field
2023-03-27 16:13:50 +02:00
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker
...
fix: for connection used ecpool, let worker do health check fun
2023-03-27 09:51:52 +08:00
7085a2d6d7
chore: hidden the password field
...
ref: https://github.com/emqx/emqx/issues/10222
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 09:47:51 +08:00
19e136bb66
fix: avoid uisng 'pid' as log data field
...
Since it clashes with log metad data field
2023-03-26 10:33:18 +02:00
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
b37f186142
Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master
2023-03-24 21:47:15 +01:00
ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50
...
feat(buffer_worker): decouple query mode from underlying connector call mode
2023-03-24 13:49:00 -03:00
c92af4f763
docs: fix quote
2023-03-24 09:48:35 -03:00
a239f0523a
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:23:06 -03:00
ca820dfefc
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:21:08 -03:00
5bdc71af6f
docs(kafka_consumer): make specific labels for key/value encoding modes
...
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
d0f0a93888
fix: ecpool worker exec dynamo health check fun
2023-03-24 11:15:52 +08:00
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
28544e62df
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:27:09 +08:00
1cb975a811
Merge pull request #10148 from Kinplemelon/kinple/kafka-nodelay-label
...
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:24:24 +08:00
1a466fda0b
chore: use binary type InstanceId as PoolName
2023-03-23 16:06:51 +08:00
9b63bdc1e0
chore: apply review suggestions
...
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
ac41c7e653
chore: format codes
2023-03-23 11:53:26 +08:00
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout
...
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
2023-03-22 20:21:04 -03:00
1ca6a51425
Merge pull request #10198 from thalesmg/fix-flaky-kconsumer-test-v50
...
test: attempt to fix flaky kafka consumer test
2023-03-22 17:19:50 -03:00
127a075b66
test(dynamo): attempt to fix dynamo tests
...
Those tests in the `flaky` test are really flaky and require lots of
CI retries.
Apparently, the flakiness comes from race conditions from restarting
bridges with the same name too fast between test cases. Previously,
all test cases were sharing the same bridge name (the module name).
2023-03-22 14:34:37 -03:00
61cb03b45a
fix(buffer_worker): change the default `resume_interval` value and expose it as hidden config
...
Also removes the previously added alarm for request timeout.
There are situations where having a short request timeout and a long
health check interval make sense, so we don't want to alarm the user
for those situations. Instead, we automatically attempt to set a
reasonable `resume_interval` value.
2023-03-22 11:47:36 -03:00
208813330a
fix: remove metrics from rocketmq example
2023-03-22 10:59:20 -03:00
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
...
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
4ad3579966
test(bridges): add test suite for RocketMQ
2023-03-22 10:36:58 +08:00
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
65c2da7ef5
Merge remote-tracking branch 'ce/master' into cassa
2023-03-22 09:30:50 +08:00
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
7e6f52e8fe
test: attempt to fix flaky kafka consumer test
...
It might need some time for the metrics to be set.
2023-03-21 17:45:58 -03:00
27b8445337
fix: add inflight window setting to the clickhouse bridge
...
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 17:14:03 +01:00
539ec2f774
chore(bridge): cover username/password auth for cassandra bridges
2023-03-21 13:55:53 +08:00
ae5c24445b
chore: make shellcheck happy
2023-03-21 11:18:15 +08:00
3a46681dde
feat: handle escaped characters in InfluxDB data bridge write_syntax
...
Closes: EMQX-7834
2023-03-20 16:42:23 +02:00
d1689f6957
chore: correct api examples
...
follow https://github.com/emqx/emqx/pull/10114
2023-03-20 17:03:05 +08:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
12942b676d
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-20 09:50:27 +08:00
c2c9abd568
chore: add running ct
2023-03-20 09:47:58 +08:00
d8e6e2a1be
test: refine CASSANDRA_HOST name
2023-03-17 18:52:52 +08:00
678cc937c0
test(bridge): cover ssl testing for cassandra bridge
2023-03-17 18:25:05 +08:00
5f0828a2ea
ci: add certs for cassandra tls
2023-03-17 16:39:10 +08:00
a20c39c83a
test: correct CASSANDRA_HOST
2023-03-17 13:55:26 +08:00
75c1592e7c
chore: make static_checks happy
2023-03-17 11:34:52 +08:00
c0a216a740
feat(bridge): support cassandra bridge
2023-03-17 11:34:48 +08:00
966276127e
test: trying to make tests more stable
2023-03-16 13:43:01 -03:00
0587e0d3f3
refactor(kafka_consumer): change the generated consumer group id
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
41b8d47696
test(kafka_consumer): add more clusterized tests
2023-03-16 13:43:01 -03:00
19f9c35662
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
fc5dfa108a
fix(kafka_consumer): rename `force_utf8` to `none`
...
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
2023-03-16 13:43:01 -03:00
37a35b22df
docs(kafka): improve kafka consumer/producer API examples
2023-03-16 13:43:01 -03:00
53979b6261
feat(kafka_consumer): support multiple topic mappings, payload
...
templates and key/value encodings
Added after feedback from the product team.
2023-03-16 13:43:01 -03:00
f31f15e44e
chore(kafka_producer): make schema changes more backwards compatible
...
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
c182a4053e
fix(kafka_consumer): avoid leaking atoms in bridge probe API
2023-03-16 13:43:01 -03:00
27dfd98f46
fix(kafka): fix api methods schemas and bridge type enum
2023-03-16 13:43:01 -03:00
fecc08bb8f
refactor(kafka_consumer): apply review suggestions
2023-03-16 13:43:01 -03:00
2a3aef92bb
refactor(kafka_consumer): redact error info
2023-03-16 13:43:01 -03:00
9037314aea
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
1b78f22d92
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
e1fdd041b3
feat(kafka_consumer): add `offset_commit_interval_seconds` kafka parameter
2023-03-16 13:43:01 -03:00
1d5fe14a30
test: remove sleeps
2023-03-16 13:43:01 -03:00
65c15b3fae
refactor(kafka_consumer): move subscriber startup logic to separate fn
2023-03-16 13:43:01 -03:00
969fa03ecc
feat: implement kafka consumer
2023-03-16 13:43:01 -03:00
5ab5236ad3
test: fix flaky test
2023-03-16 13:42:38 -03:00
651f72eabc
docs: correct the chinese label for the `socket_nodelay` field
2023-03-16 11:39:29 +08:00
a530ccbe3d
Merge pull request #10095 from fix/EEC-782/mysql-prepstmt-exhaustion
...
fix(mysql): be explicit that batch queries are parameterless
2023-03-14 10:21:45 +03:00
e3595f2e79
chore(mria): Bump version to 0.4.0
2023-03-11 00:37:25 +01:00
f7c0d29478
test(tde): add testcase for a nasty string in SQL query
...
Similar to what we have in mysql and pgqsl testsuites.
2023-03-10 18:43:30 +03:00
0a7f6c7d03
fix(mysql): ensure proper escaping in batch inserts
...
Also hexencode non-utf8 binaries. This is essentially an heuristic.
We don't know column types in runtime, and there's no simple way
to find them out. Since we're already doing full binary scan during
escaping it should be cheap to bail out on non-utf8 strings and
hexencode them instead.
Also introduce separate function to highlight that this escaping
is MySQL-specific.
2023-03-10 18:42:04 +03:00
fc37d9b3cd
fix(mysql): be explicit that batch queries are parameterless
...
So that mysql client won't attempt to prepare them automatically, thus
trashing the server's prepared statements table and making interaction
overall heavier.
2023-03-10 18:42:04 +03:00
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
97e71c54d4
fix: use default template if timestamp is empty (undefined) in InfluxDB bridge
...
Closes EMQX-8926
2023-03-08 11:58:23 +01:00
984dd3446d
test(bridges): add test suite for DynamoDB
2023-03-08 11:13:51 +08:00
bfc2920a55
feat(bridges): introduce DynamoDB definition files from v4
2023-03-08 11:13:51 +08:00
b8b1a78025
feat(bridges): integrate the DynamoDB into data bridges
2023-03-08 11:13:38 +08:00
18ab7ed197
chore: bump app vsns
2023-03-06 15:31:28 -03:00
e9d3fc511f
chore(buffer_worker): change default `batch_time` to 0 and improve docs
2023-03-06 15:31:28 -03:00
9825998207
chore(gcp_pubsub): just deprecate `request_timeout` instead of removing
2023-03-06 10:12:38 -03:00
9b087a21f5
fix(gcp_pubsub): remove conflicting `request_timeout` option
...
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
2023-03-06 10:12:38 -03:00
c01f62a1c1
chore: bump apps versions
2023-03-03 13:58:32 +01:00
c24a84e370
docs: fix spelling
2023-03-03 12:06:18 +01:00
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
083330ad80
Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master
2023-03-01 08:53:03 +01:00
41cb7263ca
docs(influxdb): add a node to the write_syntax config
2023-02-28 21:53:32 +01:00
24f476e35f
test: add README to influxdb test script
2023-02-28 19:38:43 +01:00
6ce5029d79
Merge pull request #9881 from olcai/log-influxdb-is-alive-reason
...
fix(emqx_ee_connector): log reason for failure when starting influxdb connector
2023-02-28 09:49:08 +00:00
79bf77c2f1
Merge pull request #10020 from zmstone/0222-fix-bridge-async-mode-counters
...
fix(bridge): fix dropped counter and inflight gauge
2023-02-24 19:51:24 +01:00
9cbe64a132
fix(test): make strings json-friendly in kafka testsuite
2023-02-24 15:05:20 +03:00
44c0aae16b
fix(emqx_ee_connector): improve log message on influxdb start failures
2023-02-24 11:31:57 +01:00
da42c91fb2
test(emqx_ee_bridge): check influxdb:is_alive/2 return
2023-02-24 09:03:34 +01:00
8314c07147
chore: update influxdb-client-erl dep
2023-02-24 09:03:34 +01:00
603532dacd
fix(emqx_ee_connector): log reason for failure when starting influxdb connector
2023-02-24 09:03:34 +01:00
ada68e12e6
fix(kafka): change default value for memory_overload_protection to false
2023-02-22 13:46:51 +01:00
9316690c29
fix(schema): binary string for default values
...
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
2023-02-21 09:09:51 +01:00
53ecfb98d0
chore: fix stale code comments
2023-02-21 09:09:51 +01:00
c869eff6e8
test(kafka): disable overload protection in 2 more places
2023-02-20 21:38:14 +01:00
9d30a52910
test: disable replayq memory overload protection
2023-02-20 19:29:26 +01:00
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
...
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
ca28be4387
test(emqx_ee_connector): review fixes of influxdb tests
2023-02-17 15:28:49 +01:00
caacc08ff9
chore: bump app VSN
2023-02-17 15:28:27 +01:00
843ed464d5
fix: remove all mentions of UDP support for InfluxDB EE bridge
...
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
d9554c36ac
chore: Bump umbrealla app versions
2023-02-17 09:50:50 +01:00
794bedef9f
fix(emqx_ctl): Start CLI before ekka
2023-02-17 09:50:50 +01:00
81adb02402
chore: bump apps version
2023-02-14 22:04:29 +08:00
0420e9acb5
test(bridges): add test cases for TDEngine
2023-02-14 22:04:29 +08:00
530b21268a
feat(bridges): add TDengine
2023-02-14 22:04:23 +08:00
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
4eb4430fe2
chore: bump VSN
2023-02-09 17:22:19 +01:00
bc6c653c9f
test(emqx_ee_connector): rework and improve influxdb tests
2023-02-09 17:18:47 +01:00
cfd0e9ebdd
refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts
...
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
2023-02-09 17:18:47 +01:00
cf77dcf25e
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
...
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
2023-02-09 17:18:47 +01:00
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
c7535fce56
test: fix test expecting 200 instead of 204
2023-02-08 09:14:35 +01:00
6362476508
feat: disable telemetry for ee by default
2023-02-03 09:29:39 +01:00
feca4cc0a5
Merge pull request #9892 from zmstone/0202-docs-cosmetic
...
0202 docs cosmetic
2023-02-02 15:43:58 +01:00
ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async
...
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
2023-02-02 21:08:01 +08:00
d5c482b0b0
docs: remove timer unit from description
...
the user input has time unit. e.g. "5s" for 5 seconds etc.
2023-02-02 13:49:20 +01:00
7ad80d012c
chore(kafka): upgrade to woff 1.7.5
...
comparing to 1.7.4, a counter bump bug was fixed
but it does not affect EMQX as the bridge does not make use
of the counter which had issue.
2023-02-02 12:23:15 +01:00
0eb554a62e
fix(kafka): check Kafka partition leader connectivity
2023-02-02 12:04:17 +01:00
1c9035d24c
test: remove async from redis ct
2023-02-02 17:37:18 +08:00
JimMoen