cd1197925f
test: fix unit tests after type names changed
2023-04-18 09:48:28 +02:00
9fc8a498f8
chore: bump apps versions
2023-04-17 09:09:08 +02:00
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
...
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
f0c13e0134
fix: stale ref to emqx_map_lib
2023-04-14 18:43:53 +02:00
e073bc90bc
refactor(buffer_worker): rename `s/queue/buffer/g`
2023-04-14 11:37:19 -03:00
edd1bc579f
fix: stale ref to emqx_json after rebase
2023-04-14 16:32:42 +02:00
062ce5f819
refactor: rename emqx_map_lib to emqx_utils_maps
2023-04-14 13:41:34 +02:00
9c11bfce80
refactor: rename emqx_misc to emqx_utils
2023-04-14 13:41:27 +02:00
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
9ce0dbae7b
test: fix batch_size parameter
2023-04-14 13:11:45 +08:00
2484a79c7a
test: create bridge with invalid password
2023-04-14 12:52:08 +08:00
437096985b
test: fix SUITE case `t_create_disconnected/1`
2023-04-14 10:02:47 +08:00
a379d909bf
test: use type VARCHAR to use utf8 encoding in sqlserver
2023-04-14 10:02:46 +08:00
c366267b0f
test: MS SQL Server data bridge
2023-04-14 10:02:46 +08:00
5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
...
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
a4d9234b24
test(dynamo): remove the flaky test case
2023-04-13 12:02:22 +08:00
871ee90b3e
refactor(kafka_bridge): move kafka bridge into its own app
...
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
...
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
9560fdc5a2
chore: typo fixes
2023-04-12 14:16:40 +08:00
c50326360f
chore: update ecql version
2023-04-10 18:03:58 +08:00
30bdffe318
feat: support async and batch callback for cassandra connector
2023-04-10 15:08:10 +08:00
4c24b08244
fix(rule_action): fix metrics for bridges returning `async_return`
...
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50
...
docs(kafka_producer): update ZH translation of field (rv5.0)
2023-04-06 16:13:41 +02:00
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
0b6fd7fe14
fix(buffer_worker): check request timeout and health check interval
...
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`
Fixes https://emqx.atlassian.net/browse/EMQX-9099
Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`. This had the problem that,
with default values, `health_check_interval = request_timeout`. This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.
Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
...
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
8b5a717a1f
test(peer): increase init and startup timeout for peer nodes
...
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
ec1871ffde
test(janitor): catch each callback invocation
2023-04-03 10:20:19 -03:00
51eec32d6b
docs: improve descriptions
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-03 09:11:14 -03:00
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1
...
Doc: Refined Chinese UI
2023-04-03 13:31:20 +02:00
677b76afce
refactor: replace hidden => true with importance => hidden
2023-04-03 13:13:43 +02:00
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
...
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
...
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
...
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb
...
test: fix flaky influxdb test
2023-03-31 15:36:12 +02:00
9bbca07943
Merge pull request #10298 from emqx/release-50
...
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
aabbbf5deb
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-31 09:07:00 -03:00
6f6d12f930
docs: better Chinese labels
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-03-31 09:39:04 +02:00
e808fef1e4
feat: (MongoDB bridge) use ${var} syntax for MongoDB collection
...
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
2023-03-30 17:49:56 +02:00
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
81a104690d
test: fix flaky influxdb test
2023-03-30 16:19:22 +02:00
91a784134c
docs: update Chinese version of batch value separator label
2023-03-30 14:55:58 +02:00
dfc9141da6
docs: Refined Chinese UI
...
Also updated the English hints according to the refined Chinese according to Guowei's comments.
Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
2023-03-30 14:50:24 +02:00
07e1f2e440
docs(kafka_producer): update ZH translation of field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9411
2023-03-30 09:18:16 -03:00
ee5b9d7e0a
fix(tdengine): remove the redundant table name in SQL template
2023-03-30 10:34:10 +08:00
94726eb7f1
docs: refine zh translations
2023-03-29 15:07:19 +02:00
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
624c168dde
fix: hide nodelay Kafka client socket option
...
also fix a few descriptions
2023-03-29 11:42:17 +02:00
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
64faccf50b
test: fix flaky kafka consumer test
2023-03-28 14:50:55 -03:00
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
...
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
1c8333030c
fix(kafka_producer): add back `is_buffer_supported` callback
...
Fixes https://emqx.atlassian.net/browse/EMQX-9366
This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
d07987288a
chore: add some example annotations for config importance level
2023-03-28 14:29:24 +02:00
69fc1123ee
refactor: change enum constructors and improve docs
2023-03-27 17:30:17 -03:00
5cf09209cd
feat: tie `offset_reset_policy` and `begin_offset` together
...
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:
1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.
2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
da5e6f3d0a
test: test with only one Kafka partition for bad config recover test
2023-03-27 17:38:34 +02:00
0d36b179c0
docs: fix kafka offset reset policy config description
2023-03-27 11:22:01 -03:00
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
7085a2d6d7
chore: hidden the password field
...
ref: https://github.com/emqx/emqx/issues/10222
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 09:47:51 +08:00
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
b37f186142
Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master
2023-03-24 21:47:15 +01:00
ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50
...
feat(buffer_worker): decouple query mode from underlying connector call mode
2023-03-24 13:49:00 -03:00
c92af4f763
docs: fix quote
2023-03-24 09:48:35 -03:00
a239f0523a
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:23:06 -03:00
ca820dfefc
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:21:08 -03:00
5bdc71af6f
docs(kafka_consumer): make specific labels for key/value encoding modes
...
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
28544e62df
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:27:09 +08:00
1cb975a811
Merge pull request #10148 from Kinplemelon/kinple/kafka-nodelay-label
...
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:24:24 +08:00
9b63bdc1e0
chore: apply review suggestions
...
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
ac41c7e653
chore: format codes
2023-03-23 11:53:26 +08:00
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout
...
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
2023-03-22 20:21:04 -03:00
1ca6a51425
Merge pull request #10198 from thalesmg/fix-flaky-kconsumer-test-v50
...
test: attempt to fix flaky kafka consumer test
2023-03-22 17:19:50 -03:00
127a075b66
test(dynamo): attempt to fix dynamo tests
...
Those tests in the `flaky` test are really flaky and require lots of
CI retries.
Apparently, the flakiness comes from race conditions from restarting
bridges with the same name too fast between test cases. Previously,
all test cases were sharing the same bridge name (the module name).
2023-03-22 14:34:37 -03:00
61cb03b45a
fix(buffer_worker): change the default `resume_interval` value and expose it as hidden config
...
Also removes the previously added alarm for request timeout.
There are situations where having a short request timeout and a long
health check interval make sense, so we don't want to alarm the user
for those situations. Instead, we automatically attempt to set a
reasonable `resume_interval` value.
2023-03-22 11:47:36 -03:00
208813330a
fix: remove metrics from rocketmq example
2023-03-22 10:59:20 -03:00
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
...
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
4ad3579966
test(bridges): add test suite for RocketMQ
2023-03-22 10:36:58 +08:00
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
65c2da7ef5
Merge remote-tracking branch 'ce/master' into cassa
2023-03-22 09:30:50 +08:00
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
7e6f52e8fe
test: attempt to fix flaky kafka consumer test
...
It might need some time for the metrics to be set.
2023-03-21 17:45:58 -03:00
27b8445337
fix: add inflight window setting to the clickhouse bridge
...
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 17:14:03 +01:00
539ec2f774
chore(bridge): cover username/password auth for cassandra bridges
2023-03-21 13:55:53 +08:00
ae5c24445b
chore: make shellcheck happy
2023-03-21 11:18:15 +08:00
3a46681dde
feat: handle escaped characters in InfluxDB data bridge write_syntax
...
Closes: EMQX-7834
2023-03-20 16:42:23 +02:00
d1689f6957
chore: correct api examples
...
follow https://github.com/emqx/emqx/pull/10114
2023-03-20 17:03:05 +08:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
12942b676d
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-20 09:50:27 +08:00
678cc937c0
test(bridge): cover ssl testing for cassandra bridge
2023-03-17 18:25:05 +08:00
5f0828a2ea
ci: add certs for cassandra tls
2023-03-17 16:39:10 +08:00
75c1592e7c
chore: make static_checks happy
2023-03-17 11:34:52 +08:00
Zaiming (Stone) Shi