yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,563 Commits 393 Branches 684 Tags 74 MiB
2b0106e047
Commit Graph

540 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
33100ecca6 feat: implement schema registry for 5.0 (avro) 
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
 2023-04-06 16:28:45 -03:00
Thales Macedo Garitezi
4c24b08244 fix(rule_action): fix metrics for bridges returning async_return 
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
 2023-04-06 16:00:01 -03:00
Zaiming (Stone) Shi
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50 
docs(kafka_producer): update ZH translation of field (rv5.0)
 2023-04-06 16:13:41 +02:00
Thales Macedo Garitezi
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50 
feat(bridges): enable async query mode for all bridges with buffer workers
 2023-04-04 17:10:46 -03:00
Thales Macedo Garitezi
0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval 
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
 2023-04-04 08:58:36 -03:00
firest
11b3264251 fix: redact the password to ****** in API examples 
fix #10222
 2023-04-04 16:02:45 +08:00
lafirest
ed31b735a8
Merge pull request #10220 from lafirest/bump/hackney 
chore: bump hackney version
 2023-04-04 10:35:09 +08:00
Thales Macedo Garitezi
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50 
test(peer): increase init and startup timeout for peer nodes
 2023-04-03 15:46:13 -03:00
Thales Macedo Garitezi
f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers 
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
 2023-04-03 14:49:51 -03:00
Thales Macedo Garitezi
8b5a717a1f test(peer): increase init and startup timeout for peer nodes 
Attempt to stabilize tests that use cluster nodes.
 2023-04-03 13:20:22 -03:00
Thales Macedo Garitezi
ec1871ffde test(janitor): catch each callback invocation 2023-04-03 10:20:19 -03:00
Thales Macedo Garitezi
51eec32d6b
docs: improve descriptions 
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
 2023-04-03 09:11:14 -03:00
Kjell Winblad
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1 
Doc: Refined Chinese UI
 2023-04-03 13:31:20 +02:00
Zaiming (Stone) Shi
677b76afce refactor: replace hidden => true with importance => hidden 2023-04-03 13:13:43 +02:00
Zaiming (Stone) Shi
36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
firest
c8dca74b18 chore: bump hackney version 2023-04-03 18:41:49 +08:00
lafirest
9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine 
fix(tdengine): remove the redundant table name in SQL template
 2023-04-03 18:13:41 +08:00
zhongwencool
d63680cf25
Merge pull request #10307 from emqx/release-50 
Sync release-50 back to master
 2023-04-02 11:36:41 +08:00
Thales Macedo Garitezi
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50 
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
 2023-03-31 16:25:26 -03:00
Kjell Winblad
58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246 
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
 2023-03-31 17:18:27 +02:00
Thales Macedo Garitezi
5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-9422
 2023-03-31 11:33:15 -03:00
Kjell Winblad
fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label 
docs: make clickhouse config label for Server URL more concise
 2023-03-31 16:17:26 +02:00
Zaiming (Stone) Shi
945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb 
test: fix flaky influxdb test
 2023-03-31 15:36:12 +02:00
Zaiming (Stone) Shi
9bbca07943
Merge pull request #10298 from emqx/release-50 
Merge release-50 back to master
 2023-03-31 15:35:51 +02:00
Thales Macedo Garitezi
aabbbf5deb
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-31 09:07:00 -03:00
Kjell Winblad
6f6d12f930
docs: better Chinese labels 
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
 2023-03-31 09:39:04 +02:00
Kjell Winblad
e808fef1e4 feat: (MongoDB bridge) use ${var} syntax for MongoDB collection 
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
 2023-03-30 17:49:56 +02:00
Thales Macedo Garitezi
632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
 2023-03-30 11:51:36 -03:00
Zaiming (Stone) Shi
81a104690d test: fix flaky influxdb test 2023-03-30 16:19:22 +02:00
Kjell Winblad
8c4d211b54 fix: on_get_status clickhouse just reporting connecting 
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
 2023-03-30 15:10:03 +02:00
Kjell Winblad
91a784134c docs: update Chinese version of batch value separator label 2023-03-30 14:55:58 +02:00
Kjell Winblad
aefa0e5ffb docs: make clickhouse config label for Server URL more concise 2023-03-30 14:55:22 +02:00
Meggielqk
dfc9141da6 docs: Refined Chinese UI 
Also updated the English hints according to the refined Chinese according to Guowei's comments.

Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
 2023-03-30 14:50:24 +02:00
Thales Macedo Garitezi
b3c6ae99d2
Merge pull request #10269 from thalesmg/tdengine-required-password-rv50 
fix(tdengine): make password a required field (rv5.0)
 2023-03-30 09:25:38 -03:00
Thales Macedo Garitezi
07e1f2e440 docs(kafka_producer): update ZH translation of field (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9411
 2023-03-30 09:18:16 -03:00
Kjell Winblad
808bf1d232
Merge pull request #10270 from kjellwinblad/kjell/clickhouse/on_get_status 
fix: on_get_status clickhouse just reporting connecting
 2023-03-30 10:57:48 +02:00
firest
ee5b9d7e0a fix(tdengine): remove the redundant table name in SQL template 2023-03-30 10:34:10 +08:00
Zaiming (Stone) Shi
94726eb7f1 docs: refine zh translations 2023-03-29 15:07:19 +02:00
Thales Macedo Garitezi
b2b9732edb fix(tdengine): make password a required field (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9395

```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
 2023-03-29 10:07:02 -03:00
Kjell Winblad
c693851740 fix: on_get_status clickhouse just reporting connecting 
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
 2023-03-29 14:26:08 +02:00
Zaiming (Stone) Shi
80eb9d7542
Merge pull request #10252 from emqx/release-50 
0327 merge release-50 to master
 2023-03-29 12:33:17 +02:00
Zaiming (Stone) Shi
624c168dde fix: hide nodelay Kafka client socket option 
also fix a few descriptions
 2023-03-29 11:42:17 +02:00
Kjell Winblad
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell 
fix: add inflight window setting to the clickhouse bridge
 2023-03-29 09:38:24 +02:00
Thales Macedo Garitezi
64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50 
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
 2023-03-28 11:37:46 -03:00
Thales Macedo Garitezi
1c8333030c fix(kafka_producer): add back is_buffer_supported callback 
Fixes https://emqx.atlassian.net/browse/EMQX-9366

This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
 2023-03-28 09:33:18 -03:00
Zaiming (Stone) Shi
d07987288a chore: add some example annotations for config importance level 2023-03-28 14:29:24 +02:00
Thales Macedo Garitezi
69fc1123ee refactor: change enum constructors and improve docs 2023-03-27 17:30:17 -03:00
Thales Macedo Garitezi
5cf09209cd feat: tie offset_reset_policy and begin_offset together 
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:

1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.

2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
 2023-03-27 14:20:31 -03:00
Zaiming (Stone) Shi
da5e6f3d0a test: test with only one Kafka partition for bad config recover test 2023-03-27 17:38:34 +02:00
Thales Macedo Garitezi
0d36b179c0 docs: fix kafka offset reset policy config description 2023-03-27 11:22:01 -03:00
Zaiming (Stone) Shi
7a6494cda6
Merge pull request #10242 from zmstone/0326-fix-logging-avoid-using-pid-as-log-data-field 
fix: avoid uisng 'pid' as log data field
 2023-03-27 16:13:50 +02:00
JianBo He
bfa5922209
Merge pull request #10140 from HJianBo/cassa 
feat: support cassandra data bridge
 2023-03-27 10:23:02 +08:00
JimMoen
798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker 
fix: for connection used ecpool, let worker do health check fun
 2023-03-27 09:51:52 +08:00
JianBo He
7085a2d6d7
chore: hidden the password field 
ref: https://github.com/emqx/emqx/issues/10222

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-03-27 09:47:51 +08:00
Zaiming (Stone) Shi
19e136bb66 fix: avoid uisng 'pid' as log data field 
Since it clashes with log metad data field
 2023-03-26 10:33:18 +02:00
Zaiming (Stone) Shi
dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
Zaiming (Stone) Shi
b37f186142 Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master 2023-03-24 21:47:15 +01:00
Thales Macedo Garitezi
ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50 
feat(buffer_worker): decouple query mode from underlying connector call mode
 2023-03-24 13:49:00 -03:00
Thales Macedo Garitezi
c92af4f763 docs: fix quote 2023-03-24 09:48:35 -03:00
Thales Macedo Garitezi
a239f0523a
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-24 09:23:06 -03:00
Thales Macedo Garitezi
ca820dfefc
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-24 09:21:08 -03:00
Thales Macedo Garitezi
5bdc71af6f docs(kafka_consumer): make specific labels for key/value encoding modes 
https://emqx.atlassian.net/browse/EMQX-9312
 2023-03-24 09:08:27 -03:00
JimMoen
d0f0a93888
fix: ecpool worker exec dynamo health check fun 2023-03-24 11:15:52 +08:00
Thales Macedo Garitezi
f8d5d53908 feat(buffer_worker): decouple query mode from underlying connector call mode 
Fixes https://emqx.atlassian.net/browse/EMQX-9129

Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.

Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:

1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
 2023-03-23 13:40:31 -03:00
Kinplemelon
28544e62df docs: correct the chinese label for the socket_nodelay field 2023-03-23 16:27:09 +08:00
Kinplemelon
1cb975a811
Merge pull request #10148 from Kinplemelon/kinple/kafka-nodelay-label 
docs: correct the chinese label for the `socket_nodelay` field
 2023-03-23 16:24:24 +08:00
JianBo He
1a466fda0b chore: use binary type InstanceId as PoolName 2023-03-23 16:06:51 +08:00
JianBo He
9b63bdc1e0 chore: apply review suggestions 
- Rename sql to cql
- Add tests for `bridges_probe` API
 2023-03-23 15:27:34 +08:00
JianBo He
ac41c7e653 chore: format codes 2023-03-23 11:53:26 +08:00
JianBo He
8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
Thales Macedo Garitezi
ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout 
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
 2023-03-22 20:21:04 -03:00
Thales Macedo Garitezi
1ca6a51425
Merge pull request #10198 from thalesmg/fix-flaky-kconsumer-test-v50 
test: attempt to fix flaky kafka consumer test
 2023-03-22 17:19:50 -03:00
Thales Macedo Garitezi
127a075b66 test(dynamo): attempt to fix dynamo tests 
Those tests in the `flaky` test are really flaky and require lots of
CI retries.

Apparently, the flakiness comes from race conditions from restarting
bridges with the same name too fast between test cases.  Previously,
all test cases were sharing the same bridge name (the module name).
 2023-03-22 14:34:37 -03:00
Thales Macedo Garitezi
61cb03b45a fix(buffer_worker): change the default resume_interval value and expose it as hidden config 
Also removes the previously added alarm for request timeout.

There are situations where having a short request timeout and a long
health check interval make sense, so we don't want to alarm the user
for those situations.  Instead, we automatically attempt to set a
reasonable `resume_interval` value.
 2023-03-22 11:47:36 -03:00
Thales Macedo Garitezi
208813330a fix: remove metrics from rocketmq example 2023-03-22 10:59:20 -03:00
lafirest
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq 
feat(bridges): integrate RocketMQ into data bridges
 2023-03-22 20:43:22 +08:00
firest
4ad3579966 test(bridges): add test suite for RocketMQ 2023-03-22 10:36:58 +08:00
firest
9853d00cad feat(bridges): integrate RocketMQ into data bridges 2023-03-22 10:36:52 +08:00
JianBo He
65c2da7ef5 Merge remote-tracking branch 'ce/master' into cassa 2023-03-22 09:30:50 +08:00
Zaiming (Stone) Shi
e6091db351 Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master 2023-03-21 22:03:31 +01:00
Thales Macedo Garitezi
7e6f52e8fe test: attempt to fix flaky kafka consumer test 
It might need some time for the metrics to be set.
 2023-03-21 17:45:58 -03:00
Kjell Winblad
27b8445337 fix: add inflight window setting to the clickhouse bridge 
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.

Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-21 17:14:03 +01:00
JianBo He
539ec2f774 chore(bridge): cover username/password auth for cassandra bridges 2023-03-21 13:55:53 +08:00
JianBo He
ae5c24445b chore: make shellcheck happy 2023-03-21 11:18:15 +08:00
Serge Tupchii
3a46681dde feat: handle escaped characters in InfluxDB data bridge write_syntax 
Closes: EMQX-7834
 2023-03-20 16:42:23 +02:00
JianBo He
d1689f6957 chore: correct api examples 
follow https://github.com/emqx/emqx/pull/10114
 2023-03-20 17:03:05 +08:00
Erik Timan
2d75c7d6d9 fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints 
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
 2023-03-20 09:43:11 +01:00
JianBo He
12942b676d Merge remote-tracking branch 'upstream/master' into cassa 2023-03-20 09:50:27 +08:00
JianBo He
c2c9abd568 chore: add running ct 2023-03-20 09:47:58 +08:00
JianBo He
d8e6e2a1be test: refine CASSANDRA_HOST name 2023-03-17 18:52:52 +08:00
JianBo He
678cc937c0 test(bridge): cover ssl testing for cassandra bridge 2023-03-17 18:25:05 +08:00
JianBo He
5f0828a2ea ci: add certs for cassandra tls 2023-03-17 16:39:10 +08:00
JianBo He
a20c39c83a test: correct CASSANDRA_HOST 2023-03-17 13:55:26 +08:00
JianBo He
75c1592e7c chore: make static_checks happy 2023-03-17 11:34:52 +08:00
JianBo He
c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Thales Macedo Garitezi
966276127e test: trying to make tests more stable 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
0587e0d3f3 refactor(kafka_consumer): change the generated consumer group id 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
41b8d47696 test(kafka_consumer): add more clusterized tests 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
19f9c35662 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00