Thales Macedo Garitezi
07e1f2e440
docs(kafka_producer): update ZH translation of field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9411
2023-03-30 09:18:16 -03:00
firest
ee5b9d7e0a
fix(tdengine): remove the redundant table name in SQL template
2023-03-30 10:34:10 +08:00
Zaiming (Stone) Shi
94726eb7f1
docs: refine zh translations
2023-03-29 15:07:19 +02:00
Zaiming (Stone) Shi
80eb9d7542
Merge pull request #10252 from emqx/release-50
...
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
Zaiming (Stone) Shi
624c168dde
fix: hide nodelay Kafka client socket option
...
also fix a few descriptions
2023-03-29 11:42:17 +02:00
Kjell Winblad
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
...
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
Thales Macedo Garitezi
64faccf50b
test: fix flaky kafka consumer test
2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi
1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
...
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
Thales Macedo Garitezi
1c8333030c
fix(kafka_producer): add back `is_buffer_supported` callback
...
Fixes https://emqx.atlassian.net/browse/EMQX-9366
This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
Zaiming (Stone) Shi
d07987288a
chore: add some example annotations for config importance level
2023-03-28 14:29:24 +02:00
Thales Macedo Garitezi
69fc1123ee
refactor: change enum constructors and improve docs
2023-03-27 17:30:17 -03:00
Thales Macedo Garitezi
5cf09209cd
feat: tie `offset_reset_policy` and `begin_offset` together
...
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:
1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.
2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
Zaiming (Stone) Shi
da5e6f3d0a
test: test with only one Kafka partition for bad config recover test
2023-03-27 17:38:34 +02:00
Thales Macedo Garitezi
0d36b179c0
docs: fix kafka offset reset policy config description
2023-03-27 11:22:01 -03:00
JianBo He
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
JianBo He
7085a2d6d7
chore: hidden the password field
...
ref: https://github.com/emqx/emqx/issues/10222
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 09:47:51 +08:00
Zaiming (Stone) Shi
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
Zaiming (Stone) Shi
b37f186142
Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master
2023-03-24 21:47:15 +01:00
Thales Macedo Garitezi
ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50
...
feat(buffer_worker): decouple query mode from underlying connector call mode
2023-03-24 13:49:00 -03:00
Thales Macedo Garitezi
c92af4f763
docs: fix quote
2023-03-24 09:48:35 -03:00
Thales Macedo Garitezi
a239f0523a
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:23:06 -03:00
Thales Macedo Garitezi
ca820dfefc
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:21:08 -03:00
Thales Macedo Garitezi
5bdc71af6f
docs(kafka_consumer): make specific labels for key/value encoding modes
...
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
Thales Macedo Garitezi
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
Kinplemelon
28544e62df
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:27:09 +08:00
Kinplemelon
1cb975a811
Merge pull request #10148 from Kinplemelon/kinple/kafka-nodelay-label
...
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:24:24 +08:00
JianBo He
9b63bdc1e0
chore: apply review suggestions
...
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
JianBo He
ac41c7e653
chore: format codes
2023-03-23 11:53:26 +08:00
JianBo He
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
Thales Macedo Garitezi
ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout
...
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
2023-03-22 20:21:04 -03:00
Thales Macedo Garitezi
1ca6a51425
Merge pull request #10198 from thalesmg/fix-flaky-kconsumer-test-v50
...
test: attempt to fix flaky kafka consumer test
2023-03-22 17:19:50 -03:00
Thales Macedo Garitezi
127a075b66
test(dynamo): attempt to fix dynamo tests
...
Those tests in the `flaky` test are really flaky and require lots of
CI retries.
Apparently, the flakiness comes from race conditions from restarting
bridges with the same name too fast between test cases. Previously,
all test cases were sharing the same bridge name (the module name).
2023-03-22 14:34:37 -03:00
Thales Macedo Garitezi
61cb03b45a
fix(buffer_worker): change the default `resume_interval` value and expose it as hidden config
...
Also removes the previously added alarm for request timeout.
There are situations where having a short request timeout and a long
health check interval make sense, so we don't want to alarm the user
for those situations. Instead, we automatically attempt to set a
reasonable `resume_interval` value.
2023-03-22 11:47:36 -03:00
Thales Macedo Garitezi
208813330a
fix: remove metrics from rocketmq example
2023-03-22 10:59:20 -03:00
lafirest
84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
...
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
firest
4ad3579966
test(bridges): add test suite for RocketMQ
2023-03-22 10:36:58 +08:00
firest
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
JianBo He
65c2da7ef5
Merge remote-tracking branch 'ce/master' into cassa
2023-03-22 09:30:50 +08:00
Zaiming (Stone) Shi
e6091db351
Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master
2023-03-21 22:03:31 +01:00
Thales Macedo Garitezi
7e6f52e8fe
test: attempt to fix flaky kafka consumer test
...
It might need some time for the metrics to be set.
2023-03-21 17:45:58 -03:00
Kjell Winblad
27b8445337
fix: add inflight window setting to the clickhouse bridge
...
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 17:14:03 +01:00
JianBo He
539ec2f774
chore(bridge): cover username/password auth for cassandra bridges
2023-03-21 13:55:53 +08:00
JianBo He
ae5c24445b
chore: make shellcheck happy
2023-03-21 11:18:15 +08:00
Serge Tupchii
3a46681dde
feat: handle escaped characters in InfluxDB data bridge write_syntax
...
Closes: EMQX-7834
2023-03-20 16:42:23 +02:00
JianBo He
d1689f6957
chore: correct api examples
...
follow https://github.com/emqx/emqx/pull/10114
2023-03-20 17:03:05 +08:00
Erik Timan
2d75c7d6d9
fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
...
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
JianBo He
12942b676d
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-20 09:50:27 +08:00
JianBo He
678cc937c0
test(bridge): cover ssl testing for cassandra bridge
2023-03-17 18:25:05 +08:00
JianBo He
5f0828a2ea
ci: add certs for cassandra tls
2023-03-17 16:39:10 +08:00
JianBo He
75c1592e7c
chore: make static_checks happy
2023-03-17 11:34:52 +08:00
JianBo He
c0a216a740
feat(bridge): support cassandra bridge
2023-03-17 11:34:48 +08:00
Thales Macedo Garitezi
966276127e
test: trying to make tests more stable
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
0587e0d3f3
refactor(kafka_consumer): change the generated consumer group id
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
41b8d47696
test(kafka_consumer): add more clusterized tests
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
19f9c35662
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
fc5dfa108a
fix(kafka_consumer): rename `force_utf8` to `none`
...
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
37a35b22df
docs(kafka): improve kafka consumer/producer API examples
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
53979b6261
feat(kafka_consumer): support multiple topic mappings, payload
...
templates and key/value encodings
Added after feedback from the product team.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
f31f15e44e
chore(kafka_producer): make schema changes more backwards compatible
...
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
c182a4053e
fix(kafka_consumer): avoid leaking atoms in bridge probe API
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
27dfd98f46
fix(kafka): fix api methods schemas and bridge type enum
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
fecc08bb8f
refactor(kafka_consumer): apply review suggestions
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
2a3aef92bb
refactor(kafka_consumer): redact error info
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
9037314aea
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
1b78f22d92
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
e1fdd041b3
feat(kafka_consumer): add `offset_commit_interval_seconds` kafka parameter
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
1d5fe14a30
test: remove sleeps
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
65c15b3fae
refactor(kafka_consumer): move subscriber startup logic to separate fn
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
969fa03ecc
feat: implement kafka consumer
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi
5ab5236ad3
test: fix flaky test
2023-03-16 13:42:38 -03:00
Kinplemelon
651f72eabc
docs: correct the chinese label for the `socket_nodelay` field
2023-03-16 11:39:29 +08:00
Andrew Mayorov
f7c0d29478
test(tde): add testcase for a nasty string in SQL query
...
Similar to what we have in mysql and pgqsl testsuites.
2023-03-10 18:43:30 +03:00
Andrew Mayorov
0a7f6c7d03
fix(mysql): ensure proper escaping in batch inserts
...
Also hexencode non-utf8 binaries. This is essentially an heuristic.
We don't know column types in runtime, and there's no simple way
to find them out. Since we're already doing full binary scan during
escaping it should be cheap to bail out on non-utf8 strings and
hexencode them instead.
Also introduce separate function to highlight that this escaping
is MySQL-specific.
2023-03-10 18:42:04 +03:00
Andrew Mayorov
fc37d9b3cd
fix(mysql): be explicit that batch queries are parameterless
...
So that mysql client won't attempt to prepare them automatically, thus
trashing the server's prepared statements table and making interaction
overall heavier.
2023-03-10 18:42:04 +03:00
Andrew Mayorov
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
Serge Tupchii
97e71c54d4
fix: use default template if timestamp is empty (undefined) in InfluxDB bridge
...
Closes EMQX-8926
2023-03-08 11:58:23 +01:00
firest
984dd3446d
test(bridges): add test suite for DynamoDB
2023-03-08 11:13:51 +08:00
firest
bfc2920a55
feat(bridges): introduce DynamoDB definition files from v4
2023-03-08 11:13:51 +08:00
firest
b8b1a78025
feat(bridges): integrate the DynamoDB into data bridges
2023-03-08 11:13:38 +08:00
Thales Macedo Garitezi
18ab7ed197
chore: bump app vsns
2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi
e9d3fc511f
chore(buffer_worker): change default `batch_time` to 0 and improve docs
2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi
9825998207
chore(gcp_pubsub): just deprecate `request_timeout` instead of removing
2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi
9b087a21f5
fix(gcp_pubsub): remove conflicting `request_timeout` option
...
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
2023-03-06 10:12:38 -03:00
Kjell Winblad
c24a84e370
docs: fix spelling
2023-03-03 12:06:18 +01:00
Kjell Winblad
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi
083330ad80
Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master
2023-03-01 08:53:03 +01:00
Zaiming (Stone) Shi
41cb7263ca
docs(influxdb): add a node to the write_syntax config
2023-02-28 21:53:32 +01:00
Zaiming (Stone) Shi
24f476e35f
test: add README to influxdb test script
2023-02-28 19:38:43 +01:00
Ivan Dyachkov
6ce5029d79
Merge pull request #9881 from olcai/log-influxdb-is-alive-reason
...
fix(emqx_ee_connector): log reason for failure when starting influxdb connector
2023-02-28 09:49:08 +00:00
Zaiming (Stone) Shi
79bf77c2f1
Merge pull request #10020 from zmstone/0222-fix-bridge-async-mode-counters
...
fix(bridge): fix dropped counter and inflight gauge
2023-02-24 19:51:24 +01:00
Andrew Mayorov
9cbe64a132
fix(test): make strings json-friendly in kafka testsuite
2023-02-24 15:05:20 +03:00
Erik Timan
da42c91fb2
test(emqx_ee_bridge): check influxdb:is_alive/2 return
2023-02-24 09:03:34 +01:00
Zaiming (Stone) Shi
ada68e12e6
fix(kafka): change default value for memory_overload_protection to false
2023-02-22 13:46:51 +01:00
Zaiming (Stone) Shi
9316690c29
fix(schema): binary string for default values
...
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
2023-02-21 09:09:51 +01:00
Zaiming (Stone) Shi
53ecfb98d0
chore: fix stale code comments
2023-02-21 09:09:51 +01:00
Ivan Dyachkov
c869eff6e8
test(kafka): disable overload protection in 2 more places
2023-02-20 21:38:14 +01:00
Ivan Dyachkov
9d30a52910
test: disable replayq memory overload protection
2023-02-20 19:29:26 +01:00
Erik Timan
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
...
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
Erik Timan
caacc08ff9
chore: bump app VSN
2023-02-17 15:28:27 +01:00
Erik Timan
843ed464d5
fix: remove all mentions of UDP support for InfluxDB EE bridge
...
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
firest
81adb02402
chore: bump apps version
2023-02-14 22:04:29 +08:00
firest
0420e9acb5
test(bridges): add test cases for TDEngine
2023-02-14 22:04:29 +08:00
firest
530b21268a
feat(bridges): add TDengine
2023-02-14 22:04:23 +08:00
Zaiming (Stone) Shi
42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour
...
EMQX 8861 improve bridge restart button behaviour
2023-02-09 18:00:48 +01:00
Zaiming (Stone) Shi
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
Stefan Strigler
c7535fce56
test: fix test expecting 200 instead of 204
2023-02-08 09:14:35 +01:00
Zaiming (Stone) Shi
feca4cc0a5
Merge pull request #9892 from zmstone/0202-docs-cosmetic
...
0202 docs cosmetic
2023-02-02 15:43:58 +01:00
zhongwencool
ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async
...
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
2023-02-02 21:08:01 +08:00
Zaiming (Stone) Shi
d5c482b0b0
docs: remove timer unit from description
...
the user input has time unit. e.g. "5s" for 5 seconds etc.
2023-02-02 13:49:20 +01:00
Zaiming (Stone) Shi
7ad80d012c
chore(kafka): upgrade to woff 1.7.5
...
comparing to 1.7.4, a counter bump bug was fixed
but it does not affect EMQX as the bridge does not make use
of the counter which had issue.
2023-02-02 12:23:15 +01:00
Zaiming (Stone) Shi
0eb554a62e
fix(kafka): check Kafka partition leader connectivity
2023-02-02 12:04:17 +01:00
Zhongwen Deng
1c9035d24c
test: remove async from redis ct
2023-02-02 17:37:18 +08:00
Zhongwen Deng
e776751860
fix: delete enable_batch from redis bridge
2023-02-02 17:37:18 +08:00
Zhongwen Deng
22c3f50020
fix: add query_mode_sync_only for mysql pgsql redis mongodb bridge
2023-02-02 17:37:18 +08:00
Zhongwen Deng
dcc6bd9c21
fix: mongodb bridge'd default connect_opt should be async, not sync
2023-02-02 17:37:18 +08:00
Zhongwen Deng
1a90c1654c
chore: bad typo
2023-02-02 11:43:04 +08:00
Ilya Averyanov
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover
...
fix: fix redis cluster resource recovery
2023-02-01 10:38:52 +02:00
Ilya Averyanov
fce1e74c3d
fix(connector): fix redis cluster resource recovery
2023-01-31 16:55:05 +02:00
Zaiming (Stone) Shi
d47941601d
refactor(buffer_worker): rename trace points
2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker
...
EMQX 8774 failure to handle timeout error in resource worker
2023-01-27 14:36:44 +01:00
Andrew Mayorov
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching
...
feat(redis): disable batching in redis_cluster bridges
2023-01-27 15:27:57 +04:00
Stefan Strigler
7d18128ba9
test: async write can return noproc
2023-01-27 11:43:51 +01:00
Stefan Strigler
2d62de5188
test: fix expected result from timeout error
2023-01-27 11:43:48 +01:00
Zaiming (Stone) Shi
f6b3b930b0
chore: improve a error log
2023-01-26 14:21:27 +01:00
Andrew Mayorov
26fcaecad7
fix(redis): disable batching in `redis_cluster` bridges
...
Through configuration subsystem.
2023-01-25 17:28:11 +03:00
Andrew Mayorov
903a77b471
test(redis): ensure batch query hit different cluster shards
...
This will inevitably fail: it's not generally possible to update
different keys through the same cluster connection, one or more
update will fail with `MOVED` status. This testcase should serve
as a regression test later.
2023-01-25 15:33:05 +03:00
Zaiming (Stone) Shi
5fdf7fd24c
fix(kafka): use async callback to bump success counters
...
some telemetry events from wolff are discarded:
* dropped:
this is double counted in wolff,
we now only subscribe to the dropped_queue_full event
* retried_failed:
it has different meanings in wolff,
in wolff, it means it's the 2nd (or onward) produce attempt
in EMQX, it means it's eventually failed after some retries
* retried_success
since we are going to handle the success counters in callbac
this having this reported from wolff will only make things
harder to understand
* failed
wolff never fails (unelss drop which is a different counter)
2023-01-24 21:12:36 +01:00
Erik Timan
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
Kjell Winblad
8c52264c41
fix: atom leak when doing Kafka bridge dry-run
...
A new atom was created every time one did a dry run of a Kafka bridge
(that is, clicking the Test button in the settings dialog for the
bridge).
After this fix, we will only create a new atom when a bridge with a new
name is created. This should be acceptable as bridges with new names are
created relatively rarely and it seems to be useful to have an unique
atom for every Wolff producer.
Fixes: https://emqx.atlassian.net/browse/EMQX-8739
2023-01-20 11:50:20 +01:00
Thales Macedo Garitezi
a5424959c6
refactor: avoid operator precedence bugs
2023-01-18 16:15:34 -03:00
Thales Macedo Garitezi
47f796dd12
refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker`
...
To make it more clear that it's purpose is serve as a buffering layer.
2023-01-18 16:15:34 -03:00
Ilya Averyanov
f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test
...
chore(ee bridge): fix Redis bridge test flakyness
2023-01-18 20:56:13 +02:00
Zaiming (Stone) Shi
caf50d1b35
Merge pull request #9792 from savonarola/redis-bridge-examples
...
chore(bridges): add more helpful descriptions
2023-01-18 18:01:35 +01:00
Ilya Averyanov
f9c3115cd4
chore(bridges): add more helpful descriptions
...
Add more helpful descriptions for `command_template` option
of Redis bridge.
2023-01-18 18:17:08 +02:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
Zaiming (Stone) Shi
2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy
...
0118 fix key dispatch kafka produce strategy
2023-01-18 13:52:49 +01:00
Ilya Averyanov
f6fbbf3ee3
chore(bridges): reduce Redis bridge flakyness
2023-01-18 14:34:11 +02:00
Zaiming (Stone) Shi
8f275a66d0
test: add coverage for key_dispatch partition strategy
2023-01-18 11:47:37 +01:00
Zaiming (Stone) Shi
d4f3b4c8c2
Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50
2023-01-18 11:39:47 +01:00
Zaiming (Stone) Shi
3872c4451f
fix(kafka): olp fix was accidentally deleted in 0fd8880d0a
2023-01-18 10:52:29 +01:00
Thales Macedo Garitezi
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
fa01deb3eb
chore: retry as much as possible, don't reply to caller too soon
2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
32a9e60313
feat(buffer_worker): also use the inflight table for sync requests
...
Related: https://emqx.atlassian.net/browse/EMQX-8692
This should also correctly account for `retried.*` metrics for sync
requests.
Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.
Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4c25be8a2c
chore(influxdb): remove deprecated value from example
2023-01-17 16:48:48 -03:00
Stefan Strigler
f37b3e4bc4
test: test against `bridges_probe` API
2023-01-17 15:29:19 +01:00
Stefan Strigler
5fc31dcd9e
fix: use atom keys in example to make sure they exist
2023-01-17 15:29:19 +01:00