8aa7c014e7
perf(buffer_worker): avoid calling `ets:info/2`
...
(Almost?) fixes https://emqx.atlassian.net/browse/EMQX-9637
During the course of performance tests comparing the performance of
e5.0.3 and e4.4.16 regarding the webhook bridge in sync mode, we
observed that the throughput in e5.0.3 (sync) was much lower than in
e4.4.16: ~ 9 k msgs / s vs. ~ 50 k msgs / s, respectively.
Analyzing `observer_cli` output, we noticed that a lot of the time
both buffer workers and ehttpc processes was spent in `ets:info/2`.
That function was called to check the size of the inflight table when
updating metrics and checking if the inflight table was full. Other
uses of `ets:info/2` were contained inside the arguments to some
`?tp/2` macro usages (https://github.com/kafka4beam/snabbkaffe/pull/60 ).
By using a specific record to track the size of the table, we managed
to improve the bridge performance to ~ 45 k msgs / s in sync mode.
2023-05-02 17:05:32 -03:00
57cc854a4a
test(bridge): fix bridge map type filed converters
...
now the converters on map type fields only work at the wrapping map
level but not the values
2023-04-30 10:45:11 +02:00
b58d3e8f94
Merge pull request #10529 from zmstone/0426-ensure-buffer-worker-monitors-cassandra-conn-pid
...
0426 ensure buffer worker monitors cassandra conn pid
2023-04-27 22:54:30 +02:00
c83d630c97
fix(cassandra): ensure async calls return connection pid
...
so the buffer worker can monitor it and perform retries
if the connection restarted
2023-04-26 14:33:37 +02:00
9eccfa5cdf
fix(dynamo): fix test case errors
2023-04-26 17:03:01 +08:00
2b0106e047
Merge pull request #10485 from JimMoen/fix-mssql-bridge-default
...
fix: use default health check timeout for sqlserver
2023-04-23 18:20:10 +08:00
4e7472090b
fix: refine default sql and driver name for mssql bridge
2023-04-23 15:47:18 +08:00
d826b0921d
fix(dynamo): separate the implementation of connector and client of Dynamo bridge
2023-04-23 10:57:36 +08:00
d865998a63
fix(rocketmq): fix test cases
2023-04-21 11:02:14 +08:00
e89f4d4565
fix(dynamo): fix terminology erros
...
- Changed `username` to `aws_access_key_id`
- Changed `password` to `aws_secret_access_key`
2023-04-19 16:26:42 +08:00
bc353b0a06
fix(dynamo): change `database` to `table` in the schema of the DynamoDB bridge
...
there is no term like `database` in DynamoDB, the correct concept should be `table`
2023-04-19 15:56:29 +08:00
9fc8a498f8
chore: bump apps versions
2023-04-17 09:09:08 +02:00
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
...
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
f0c13e0134
fix: stale ref to emqx_map_lib
2023-04-14 18:43:53 +02:00
e073bc90bc
refactor(buffer_worker): rename `s/queue/buffer/g`
2023-04-14 11:37:19 -03:00
edd1bc579f
fix: stale ref to emqx_json after rebase
2023-04-14 16:32:42 +02:00
062ce5f819
refactor: rename emqx_map_lib to emqx_utils_maps
2023-04-14 13:41:34 +02:00
9c11bfce80
refactor: rename emqx_misc to emqx_utils
2023-04-14 13:41:27 +02:00
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
9ce0dbae7b
test: fix batch_size parameter
2023-04-14 13:11:45 +08:00
2484a79c7a
test: create bridge with invalid password
2023-04-14 12:52:08 +08:00
437096985b
test: fix SUITE case `t_create_disconnected/1`
2023-04-14 10:02:47 +08:00
a379d909bf
test: use type VARCHAR to use utf8 encoding in sqlserver
2023-04-14 10:02:46 +08:00
c366267b0f
test: MS SQL Server data bridge
2023-04-14 10:02:46 +08:00
5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
...
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
a4d9234b24
test(dynamo): remove the flaky test case
2023-04-13 12:02:22 +08:00
871ee90b3e
refactor(kafka_bridge): move kafka bridge into its own app
...
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
...
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
9560fdc5a2
chore: typo fixes
2023-04-12 14:16:40 +08:00
c50326360f
chore: update ecql version
2023-04-10 18:03:58 +08:00
30bdffe318
feat: support async and batch callback for cassandra connector
2023-04-10 15:08:10 +08:00
4c24b08244
fix(rule_action): fix metrics for bridges returning `async_return`
...
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50
...
docs(kafka_producer): update ZH translation of field (rv5.0)
2023-04-06 16:13:41 +02:00
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
0b6fd7fe14
fix(buffer_worker): check request timeout and health check interval
...
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`
Fixes https://emqx.atlassian.net/browse/EMQX-9099
Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`. This had the problem that,
with default values, `health_check_interval = request_timeout`. This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.
Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
...
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
8b5a717a1f
test(peer): increase init and startup timeout for peer nodes
...
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
ec1871ffde
test(janitor): catch each callback invocation
2023-04-03 10:20:19 -03:00
51eec32d6b
docs: improve descriptions
...
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-03 09:11:14 -03:00
7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1
...
Doc: Refined Chinese UI
2023-04-03 13:31:20 +02:00
677b76afce
refactor: replace hidden => true with importance => hidden
2023-04-03 13:13:43 +02:00
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
...
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
...
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
...
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
Thales Macedo Garitezi