yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,064 Commits 393 Branches 684 Tags 74 MiB
cf71f9148e
Commit Graph

425 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50 
docs(kafka_producer): update ZH translation of field (rv5.0)
 2023-04-06 16:13:41 +02:00
Thales Macedo Garitezi
0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval 
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
 2023-04-04 08:58:36 -03:00
Thales Macedo Garitezi
51eec32d6b
docs: improve descriptions 
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
 2023-04-03 09:11:14 -03:00
Kjell Winblad
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1 
Doc: Refined Chinese UI
 2023-04-03 13:31:20 +02:00
Thales Macedo Garitezi
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50 
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
 2023-03-31 16:25:26 -03:00
Thales Macedo Garitezi
5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-9422
 2023-03-31 11:33:15 -03:00
Kjell Winblad
fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label 
docs: make clickhouse config label for Server URL more concise
 2023-03-31 16:17:26 +02:00
Thales Macedo Garitezi
aabbbf5deb
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-31 09:07:00 -03:00
Kjell Winblad
6f6d12f930
docs: better Chinese labels 
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
 2023-03-31 09:39:04 +02:00
Thales Macedo Garitezi
632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
 2023-03-30 11:51:36 -03:00
Kjell Winblad
8c4d211b54 fix: on_get_status clickhouse just reporting connecting 
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
 2023-03-30 15:10:03 +02:00
Kjell Winblad
91a784134c docs: update Chinese version of batch value separator label 2023-03-30 14:55:58 +02:00
Kjell Winblad
aefa0e5ffb docs: make clickhouse config label for Server URL more concise 2023-03-30 14:55:22 +02:00
Meggielqk
dfc9141da6 docs: Refined Chinese UI 
Also updated the English hints according to the refined Chinese according to Guowei's comments.

Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
 2023-03-30 14:50:24 +02:00
Thales Macedo Garitezi
b3c6ae99d2
Merge pull request #10269 from thalesmg/tdengine-required-password-rv50 
fix(tdengine): make password a required field (rv5.0)
 2023-03-30 09:25:38 -03:00
Thales Macedo Garitezi
07e1f2e440 docs(kafka_producer): update ZH translation of field (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9411
 2023-03-30 09:18:16 -03:00
Zaiming (Stone) Shi
94726eb7f1 docs: refine zh translations 2023-03-29 15:07:19 +02:00
Thales Macedo Garitezi
b2b9732edb fix(tdengine): make password a required field (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9395

```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
 2023-03-29 10:07:02 -03:00
Zaiming (Stone) Shi
624c168dde fix: hide nodelay Kafka client socket option 
also fix a few descriptions
 2023-03-29 11:42:17 +02:00
Thales Macedo Garitezi
64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50 
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
 2023-03-28 11:37:46 -03:00
Thales Macedo Garitezi
1c8333030c fix(kafka_producer): add back is_buffer_supported callback 
Fixes https://emqx.atlassian.net/browse/EMQX-9366

This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
 2023-03-28 09:33:18 -03:00
Thales Macedo Garitezi
69fc1123ee refactor: change enum constructors and improve docs 2023-03-27 17:30:17 -03:00
Thales Macedo Garitezi
5cf09209cd feat: tie offset_reset_policy and begin_offset together 
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:

1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.

2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
 2023-03-27 14:20:31 -03:00
Zaiming (Stone) Shi
da5e6f3d0a test: test with only one Kafka partition for bad config recover test 2023-03-27 17:38:34 +02:00
Thales Macedo Garitezi
0d36b179c0 docs: fix kafka offset reset policy config description 2023-03-27 11:22:01 -03:00
Thales Macedo Garitezi
c92af4f763 docs: fix quote 2023-03-24 09:48:35 -03:00
Thales Macedo Garitezi
a239f0523a
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-24 09:23:06 -03:00
Thales Macedo Garitezi
ca820dfefc
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-24 09:21:08 -03:00
Thales Macedo Garitezi
5bdc71af6f docs(kafka_consumer): make specific labels for key/value encoding modes 
https://emqx.atlassian.net/browse/EMQX-9312
 2023-03-24 09:08:27 -03:00
Kinplemelon
28544e62df docs: correct the chinese label for the socket_nodelay field 2023-03-23 16:27:09 +08:00
Erik Timan
2d75c7d6d9 fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints 
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
 2023-03-20 09:43:11 +01:00
Thales Macedo Garitezi
966276127e test: trying to make tests more stable 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
0587e0d3f3 refactor(kafka_consumer): change the generated consumer group id 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
41b8d47696 test(kafka_consumer): add more clusterized tests 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
19f9c35662 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
fc5dfa108a fix(kafka_consumer): rename force_utf8 to none 
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
37a35b22df docs(kafka): improve kafka consumer/producer API examples 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
53979b6261 feat(kafka_consumer): support multiple topic mappings, payload 
templates and key/value encodings

Added after feedback from the product team.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
f31f15e44e chore(kafka_producer): make schema changes more backwards compatible 
This will still require fixes to the frontend.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
c182a4053e fix(kafka_consumer): avoid leaking atoms in bridge probe API 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
27dfd98f46 fix(kafka): fix api methods schemas and bridge type enum 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
fecc08bb8f refactor(kafka_consumer): apply review suggestions 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
2a3aef92bb refactor(kafka_consumer): redact error info 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
9037314aea docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
1b78f22d92 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
e1fdd041b3 feat(kafka_consumer): add offset_commit_interval_seconds kafka parameter 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
1d5fe14a30 test: remove sleeps 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
65c15b3fae refactor(kafka_consumer): move subscriber startup logic to separate fn 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
969fa03ecc feat: implement kafka consumer 2023-03-16 13:43:01 -03:00