4c24b08244
fix(rule_action): fix metrics for bridges returning `async_return`
...
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50
...
docs(kafka_producer): update ZH translation of field (rv5.0)
2023-04-06 16:13:41 +02:00
0b6fd7fe14
fix(buffer_worker): check request timeout and health check interval
...
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`
Fixes https://emqx.atlassian.net/browse/EMQX-9099
Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`. This had the problem that,
with default values, `health_check_interval = request_timeout`. This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.
Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
51eec32d6b
docs: improve descriptions
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-03 09:11:14 -03:00
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1
...
Doc: Refined Chinese UI
2023-04-03 13:31:20 +02:00
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
...
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
5011486b18
fix(kafka_consumer): return better error messages when probing kafka consumer bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label
...
docs: make clickhouse config label for Server URL more concise
2023-03-31 16:17:26 +02:00
aabbbf5deb
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-31 09:07:00 -03:00
6f6d12f930
docs: better Chinese labels
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-03-31 09:39:04 +02:00
632bffd451
fix: return friendly message when kafka producer fails to start (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9392
The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn). However, such information is printed to the logs.
This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
8c4d211b54
fix: on_get_status clickhouse just reporting connecting
...
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-30 15:10:03 +02:00
91a784134c
docs: update Chinese version of batch value separator label
2023-03-30 14:55:58 +02:00
aefa0e5ffb
docs: make clickhouse config label for Server URL more concise
2023-03-30 14:55:22 +02:00
dfc9141da6
docs: Refined Chinese UI
...
Also updated the English hints according to the refined Chinese according to Guowei's comments.
Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
2023-03-30 14:50:24 +02:00
b3c6ae99d2
Merge pull request #10269 from thalesmg/tdengine-required-password-rv50
...
fix(tdengine): make password a required field (rv5.0)
2023-03-30 09:25:38 -03:00
07e1f2e440
docs(kafka_producer): update ZH translation of field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9411
2023-03-30 09:18:16 -03:00
94726eb7f1
docs: refine zh translations
2023-03-29 15:07:19 +02:00
b2b9732edb
fix(tdengine): make password a required field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9395
```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
2023-03-29 10:07:02 -03:00
624c168dde
fix: hide nodelay Kafka client socket option
...
also fix a few descriptions
2023-03-29 11:42:17 +02:00
64faccf50b
test: fix flaky kafka consumer test
2023-03-28 14:50:55 -03:00
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
...
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
1c8333030c
fix(kafka_producer): add back `is_buffer_supported` callback
...
Fixes https://emqx.atlassian.net/browse/EMQX-9366
This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
69fc1123ee
refactor: change enum constructors and improve docs
2023-03-27 17:30:17 -03:00
5cf09209cd
feat: tie `offset_reset_policy` and `begin_offset` together
...
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:
1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.
2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
da5e6f3d0a
test: test with only one Kafka partition for bad config recover test
2023-03-27 17:38:34 +02:00
0d36b179c0
docs: fix kafka offset reset policy config description
2023-03-27 11:22:01 -03:00
c92af4f763
docs: fix quote
2023-03-24 09:48:35 -03:00
a239f0523a
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:23:06 -03:00
ca820dfefc
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:21:08 -03:00
5bdc71af6f
docs(kafka_consumer): make specific labels for key/value encoding modes
...
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
28544e62df
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:27:09 +08:00
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
966276127e
test: trying to make tests more stable
2023-03-16 13:43:01 -03:00
0587e0d3f3
refactor(kafka_consumer): change the generated consumer group id
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
41b8d47696
test(kafka_consumer): add more clusterized tests
2023-03-16 13:43:01 -03:00
19f9c35662
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
fc5dfa108a
fix(kafka_consumer): rename `force_utf8` to `none`
...
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
2023-03-16 13:43:01 -03:00
37a35b22df
docs(kafka): improve kafka consumer/producer API examples
2023-03-16 13:43:01 -03:00
53979b6261
feat(kafka_consumer): support multiple topic mappings, payload
...
templates and key/value encodings
Added after feedback from the product team.
2023-03-16 13:43:01 -03:00
f31f15e44e
chore(kafka_producer): make schema changes more backwards compatible
...
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
c182a4053e
fix(kafka_consumer): avoid leaking atoms in bridge probe API
2023-03-16 13:43:01 -03:00
27dfd98f46
fix(kafka): fix api methods schemas and bridge type enum
2023-03-16 13:43:01 -03:00
fecc08bb8f
refactor(kafka_consumer): apply review suggestions
2023-03-16 13:43:01 -03:00
2a3aef92bb
refactor(kafka_consumer): redact error info
2023-03-16 13:43:01 -03:00
9037314aea
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
1b78f22d92
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
e1fdd041b3
feat(kafka_consumer): add `offset_commit_interval_seconds` kafka parameter
2023-03-16 13:43:01 -03:00
1d5fe14a30
test: remove sleeps
2023-03-16 13:43:01 -03:00
65c15b3fae
refactor(kafka_consumer): move subscriber startup logic to separate fn
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi