Commit Graph

668 Commits

Author SHA1 Message Date
firest 0b46acda87 test(opents): add test cases for OpenTSDB 2023-04-23 11:03:20 +08:00
firest d826b0921d fix(dynamo): separate the implementation of connector and client of Dynamo bridge 2023-04-23 10:57:36 +08:00
firest 5074825075 feat(opents): OpenTSDB integration 2023-04-23 09:56:24 +08:00
Ilya Averyanov ebb75b275e feat(ft): update app versions 2023-04-21 17:52:13 +03:00
Ilya Averyanov 0211bcf030 Merge branch 'master' into file-transfer
* master: (279 commits)
  chore: shorten ct/run.sh script
  chore: rename cassandra_impl to cassandra_connector
  chore: fix mix.exs checking
  refactor(cassandra): move cassandra bridge into its own app
  chore: apply review suggestions
  chore: update changes/ce/fix-10449.en.md
  test: add a test for authn {}
  chore: add changlog for authn_http validation
  fix: always check authn_http's header and ssl_option
  chore: apply suggestions from code review
  fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
  fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager
  perf(emqx_alarm): use dirty Mnesia operations to activate an alarm
  ci: simplify find-apps.sh for ee apps
  perf(emqx_resource): don't reactivate alarms on reoccurring errors
  ci: check if Elixir files are formatted in pre-commit hook
  fix(dynamo): fix field name errors
  chore: remove *_collector for prometheus api's example
  chore: make plugins config to low level
  chore: re-split dynamo i18n file
  ...
2023-04-21 17:37:17 +03:00
JianBo He b270623c46 chore: rename cassandra_impl to cassandra_connector 2023-04-21 16:49:25 +08:00
firest 7704995279 fix(rocketmq): expose the driver parameter `sync_timeout` into the RocketMQ bridge configuration 2023-04-21 15:10:25 +08:00
JianBo He 5cc28a7b45 chore: fix mix.exs checking 2023-04-21 14:46:09 +08:00
JianBo He bdce32e713 refactor(cassandra): move cassandra bridge into its own app 2023-04-21 14:45:38 +08:00
JianBo He 6532cfb337
Merge pull request #10452 from thalesmg/refactor-gcp-pubsub-bridge-v50
refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
2023-04-21 14:42:00 +08:00
lafirest 48402d0476
Merge pull request #10461 from lafirest/fix/dynamo_db_svrs
fix(rocketmq): allow setting multiple addresses in RocketMQ bridge
2023-04-21 14:37:37 +08:00
firest f602900a53 fix(rocketmq): fix that the status check of RocketMQ bridge may not accurate 2023-04-21 13:35:29 +08:00
firest d865998a63 fix(rocketmq): fix test cases 2023-04-21 11:02:14 +08:00
firest 21473e7ca5 fix(dynamo): fix field name errors 2023-04-20 18:16:04 +08:00
firest 6e12abff39 fix(rocketmq): allow setting multiple addresses in RocketMQ bridge 2023-04-20 17:58:12 +08:00
Zaiming (Stone) Shi 415830a0a2 Merge remote-tracking branch 'origin/release-50' into 0420-sync-release-50-to-masteer 2023-04-20 11:03:39 +02:00
Kjell Winblad 7d3367467a
Merge pull request #10408 from kjellwinblad/kjell/rule_engine/add_missing_mongo_date_functions/EMQX-9244
feat: add mongo_date functions to the rule engine
2023-04-20 09:46:45 +02:00
Thales Macedo Garitezi a9bd91fcff refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9536

Note: since GCP PubSub is not shared by any authn/authz backend,
there's no need to separate its connector into another app.
2023-04-19 13:24:32 -03:00
firest e89f4d4565 fix(dynamo): fix terminology erros
- Changed `username` to `aws_access_key_id`
- Changed `password` to `aws_secret_access_key`
2023-04-19 16:26:42 +08:00
firest bc353b0a06 fix(dynamo): change `database` to `table` in the schema of the DynamoDB bridge
there is no term like `database` in DynamoDB, the correct concept should be `table`
2023-04-19 15:56:29 +08:00
Zaiming (Stone) Shi 9efbb71ae8
Merge pull request #10441 from zmstone/0418-fix-flakyness
test: pick random port number for gcp pubsub mock server
2023-04-18 19:11:30 +02:00
Ivan Dyachkov dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
JianBo He ad4b7d59eb
Merge pull request #10440 from HJianBo/fix-cassa-flaky-test
chore: fix flaky test in emqx_ee_bridge_cassa_SUITE
2023-04-18 22:44:54 +08:00
Andrew Mayorov 75813683be
Merge pull request #10371 from ft/EMQX-9257/resource-pool-switch
feat(respool): switch to `emqx_resource_pool`
2023-04-18 16:58:17 +03:00
JianBo He 87c5170696 chore: fix flaky test 2023-04-18 21:33:41 +08:00
Zaiming (Stone) Shi 97b8f00508 test: pick random port number for gcp pubsub mock server 2023-04-18 15:07:57 +02:00
Andrew Mayorov a08c000222
test(mongo): do not rely on internals to clean database 2023-04-18 12:51:17 +03:00
Andrew Mayorov 21e19a33ce
feat(respool): switch to `emqx_resource_pool`
Which was previously known as `emqx_plugin_libs_pool`. This is part
of the effort to get rid of `emqx_plugin_libs` application.
2023-04-18 12:51:14 +03:00
Kjell Winblad 8a3fccb330 style: fix variable name style 2023-04-18 10:32:37 +02:00
Kjell Winblad 29584ca721 feat: add mongo_date functions to the rule engine
This commit adds mong_date built-in functions to the rule engine
SQL-like language. Corresponding functions already existed in EMQX 4.4
and this commit makes sure that EMQX 5.X also has these functions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9244
2023-04-18 10:01:35 +02:00
Zaiming (Stone) Shi cd1197925f test: fix unit tests after type names changed 2023-04-18 09:48:28 +02:00
Thales Macedo Garitezi 9d15247dd5 feat(schema_registry): add support for protobuf schemas
Fixes https://emqx.atlassian.net/browse/EMQX-9470
2023-04-17 13:04:54 -03:00
Ivan Dyachkov 9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Stefan Strigler 7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules
New emqx_utils app to collect utility modules
2023-04-14 20:33:11 +02:00
Stefan Strigler f0c13e0134 fix: stale ref to emqx_map_lib 2023-04-14 18:43:53 +02:00
Thales Macedo Garitezi e073bc90bc refactor(buffer_worker): rename `s/queue/buffer/g` 2023-04-14 11:37:19 -03:00
Stefan Strigler edd1bc579f fix: stale ref to emqx_json after rebase 2023-04-14 16:32:42 +02:00
Stefan Strigler 4f2f0ebb6a fix: bad ref to emqx_misc after rebase 2023-04-14 14:58:19 +02:00
Stefan Strigler 062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler 6e8665365b refactor: rename emqx_tables to emqx_utils_ets 2023-04-14 13:41:33 +02:00
Stefan Strigler d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler 9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen 9ce0dbae7b
test: fix batch_size parameter 2023-04-14 13:11:45 +08:00
JimMoen 2484a79c7a
test: create bridge with invalid password 2023-04-14 12:52:08 +08:00
JimMoen 8cec62bd87
fix: async reply arg list 2023-04-14 12:43:36 +08:00
JimMoen 65c4234829
refactor: rename instance_id, which is also PoolName 2023-04-14 10:02:48 +08:00
JimMoen 2d1b94f7f7
fix: file mode 755 magic number 2023-04-14 10:02:48 +08:00
JimMoen 675f9894ab
fix: fix single quotes not applied for single query
- alwasy use `emqx_plugin_libs_rule:proc_batch_sql/3` to add single quotes for string
2023-04-14 10:02:47 +08:00
JimMoen f8c0aa49a2
style: make elvis and shellcheck happy 2023-04-14 10:02:47 +08:00
JimMoen 437096985b
test: fix SUITE case `t_create_disconnected/1` 2023-04-14 10:02:47 +08:00
JimMoen 27fdf644aa
fix(sqlserver): let a single query be a length 1 batch
Need to be fixed:
`emqx_plugin_libs_rule:proc_tmpl/2` won't add single quotes for string
2023-04-14 10:02:46 +08:00
JimMoen a379d909bf
test: use type VARCHAR to use utf8 encoding in sqlserver 2023-04-14 10:02:46 +08:00
JimMoen c366267b0f
test: MS SQL Server data bridge 2023-04-14 10:02:46 +08:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
JimMoen 8e62afecdd
chore: suite module rename 2023-04-14 09:57:00 +08:00
JimMoen 8ceeafb0de
chore: fix file license in lib-ee 2023-04-14 09:56:59 +08:00
Thales Macedo Garitezi 9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
Kjell Winblad 601564fa0a
Merge pull request #10324 from kjellwinblad/kjell/fix/EMQX-9374
fix: check Clickhouse connection after creation
2023-04-13 16:31:35 +02:00
Kjell Winblad 0cec52d5e4 fix: check Clickhouse connection after creation
When pressing reconnect on a badly configured Clickhouse bridge in the
dashboard, no error message was shown. This commit fixes this issue by
testing the connection after creation and returning an error tuple if
the connection is not working.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
2023-04-13 10:09:35 +02:00
firest a4d9234b24 test(dynamo): remove the flaky test case 2023-04-13 12:02:22 +08:00
Thales Macedo Garitezi 871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
JianBo He d16b5c40d6
Merge pull request #10356 from HJianBo/async-batch-cassa
feat: support async and batch callback for cassandra connector
2023-04-12 20:59:36 +08:00
JianBo He 9560fdc5a2 chore: typo fixes 2023-04-12 14:16:40 +08:00
Thales Macedo Garitezi 914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50
feat: implement schema registry for 5.0 (avro)
2023-04-11 16:46:27 -03:00
Thales Macedo Garitezi 97b94b8882 feat: implement schema registry for 5.0 (avro)
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
2023-04-11 14:50:45 -03:00
JianBo He c50326360f chore: update ecql version 2023-04-10 18:03:58 +08:00
JianBo He 30bdffe318 feat: support async and batch callback for cassandra connector 2023-04-10 15:08:10 +08:00
Andrew Mayorov 0d86ef8d3a
fix(ft-s3): use AWS4 signed urls for S3 export URIs
This will ensure that the S3 export URIs are valid in all AWS regions.
2023-04-07 22:15:01 +03:00
Ilya Averyanov 6fbc7dc822 fix(ft): remove s3 roots from ee schema 2023-04-07 17:36:47 +03:00
Ilya Averyanov d21daae7ae fix(ft): fix dynamic function calls 2023-04-07 17:36:47 +03:00
Ilya Averyanov eae2478678 fix(ft): add descriptions for missing ft schema fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov 2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov 18aa39f892 fix(ft): add emqx_ft_schema to ee schemas 2023-04-07 17:36:47 +03:00
Ilya Averyanov c7865e5eae fix(ft-s3): fix dependency dependencies 2023-04-07 17:36:45 +03:00
Ilya Averyanov 71965d90e4 fix(ft-s3): synchronize erlcloud version across apps 2023-04-07 17:26:53 +03:00
Thales Macedo Garitezi 33100ecca6 feat: implement schema registry for 5.0 (avro)
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
2023-04-06 16:28:45 -03:00
Thales Macedo Garitezi 4c24b08244 fix(rule_action): fix metrics for bridges returning `async_return`
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
Zaiming (Stone) Shi cf71f9148e
Merge pull request #10287 from thalesmg/fix-kprod-zh-desc-mqtt-topic-rv50
docs(kafka_producer): update ZH translation of field (rv5.0)
2023-04-06 16:13:41 +02:00
Thales Macedo Garitezi 5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
Thales Macedo Garitezi 0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
firest 11b3264251 fix: redact the password to `******` in API examples
fix #10222
2023-04-04 16:02:45 +08:00
lafirest ed31b735a8
Merge pull request #10220 from lafirest/bump/hackney
chore: bump hackney version
2023-04-04 10:35:09 +08:00
Thales Macedo Garitezi a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Thales Macedo Garitezi 8b5a717a1f test(peer): increase init and startup timeout for peer nodes
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
Thales Macedo Garitezi ec1871ffde test(janitor): catch each callback invocation 2023-04-03 10:20:19 -03:00
Thales Macedo Garitezi 51eec32d6b
docs: improve descriptions
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-03 09:11:14 -03:00
Kjell Winblad 7c60a1e276
Merge pull request #10255 from emqx/Meggielqk-patch-1
Doc: Refined Chinese UI
2023-04-03 13:31:20 +02:00
Zaiming (Stone) Shi 677b76afce refactor: replace hidden => true with importance => hidden 2023-04-03 13:13:43 +02:00
Zaiming (Stone) Shi 36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
firest c8dca74b18 chore: bump hackney version 2023-04-03 18:41:49 +08:00
lafirest 9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
zhongwencool d63680cf25
Merge pull request #10307 from emqx/release-50
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Thales Macedo Garitezi 246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
Kjell Winblad 58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
Thales Macedo Garitezi 5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
Kjell Winblad fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label
docs: make clickhouse config label for Server URL more concise
2023-03-31 16:17:26 +02:00
Zaiming (Stone) Shi 945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb
test: fix flaky influxdb test
2023-03-31 15:36:12 +02:00
Zaiming (Stone) Shi 9bbca07943
Merge pull request #10298 from emqx/release-50
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
Thales Macedo Garitezi aabbbf5deb
docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-31 09:07:00 -03:00
Kjell Winblad 6f6d12f930
docs: better Chinese labels
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-03-31 09:39:04 +02:00
Kjell Winblad e808fef1e4 feat: (MongoDB bridge) use ${var} syntax for MongoDB collection
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
2023-03-30 17:49:56 +02:00
Thales Macedo Garitezi 632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
Zaiming (Stone) Shi 81a104690d test: fix flaky influxdb test 2023-03-30 16:19:22 +02:00
Kjell Winblad 8c4d211b54 fix: on_get_status clickhouse just reporting connecting
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-30 15:10:03 +02:00
Kjell Winblad 91a784134c docs: update Chinese version of batch value separator label 2023-03-30 14:55:58 +02:00
Kjell Winblad aefa0e5ffb docs: make clickhouse config label for Server URL more concise 2023-03-30 14:55:22 +02:00
Meggielqk dfc9141da6 docs: Refined Chinese UI
Also updated the English hints according to the refined Chinese according to Guowei's comments.

Co-authored-by: Kjell Winblad <kjellwinblad@gmail.com>
2023-03-30 14:50:24 +02:00
Thales Macedo Garitezi b3c6ae99d2
Merge pull request #10269 from thalesmg/tdengine-required-password-rv50
fix(tdengine): make password a required field (rv5.0)
2023-03-30 09:25:38 -03:00
Thales Macedo Garitezi 07e1f2e440 docs(kafka_producer): update ZH translation of field (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9411
2023-03-30 09:18:16 -03:00
Kjell Winblad 808bf1d232
Merge pull request #10270 from kjellwinblad/kjell/clickhouse/on_get_status
fix: on_get_status clickhouse just reporting connecting
2023-03-30 10:57:48 +02:00
firest ee5b9d7e0a fix(tdengine): remove the redundant table name in SQL template 2023-03-30 10:34:10 +08:00
Zaiming (Stone) Shi 94726eb7f1 docs: refine zh translations 2023-03-29 15:07:19 +02:00
Thales Macedo Garitezi b2b9732edb fix(tdengine): make password a required field (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9395

```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
2023-03-29 10:07:02 -03:00
Kjell Winblad c693851740 fix: on_get_status clickhouse just reporting connecting
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-29 14:26:08 +02:00
Zaiming (Stone) Shi 80eb9d7542
Merge pull request #10252 from emqx/release-50
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
Zaiming (Stone) Shi 624c168dde fix: hide nodelay Kafka client socket option
also fix a few descriptions
2023-03-29 11:42:17 +02:00
Kjell Winblad 8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
Thales Macedo Garitezi 64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi 1a7ca7235e
Merge pull request #10249 from thalesmg/fix-kafka-offset-doc-rv50
feat(kafka_consumer): tie `offset_reset_policy` and `begin_offset` together (rv5.0)
2023-03-28 11:37:46 -03:00
Thales Macedo Garitezi 1c8333030c fix(kafka_producer): add back `is_buffer_supported` callback
Fixes https://emqx.atlassian.net/browse/EMQX-9366

This callback was accidentally removed while adding another feature,
which made the buffer workers to be used for this bridge while they
shouldn’t be.
2023-03-28 09:33:18 -03:00
Zaiming (Stone) Shi d07987288a chore: add some example annotations for config importance level 2023-03-28 14:29:24 +02:00
Thales Macedo Garitezi 69fc1123ee refactor: change enum constructors and improve docs 2023-03-27 17:30:17 -03:00
Thales Macedo Garitezi 5cf09209cd feat: tie `offset_reset_policy` and `begin_offset` together
To make the configuration more intuitive and avoid exposing more
parameters to the user, we should:

1) Remove reset_by_subscriber as an enum constructor for
`offset_reset_policy`, as that might make the consumer hang
indefinitely without manual action.

2) Set the `begin_offset` `brod_consumer` parameter to `earliest` or
`latest` depending on the value of `offset_reset_policy`, as that’s
probably the user’s intention.
2023-03-27 14:20:31 -03:00
Zaiming (Stone) Shi da5e6f3d0a test: test with only one Kafka partition for bad config recover test 2023-03-27 17:38:34 +02:00
Thales Macedo Garitezi 0d36b179c0 docs: fix kafka offset reset policy config description 2023-03-27 11:22:01 -03:00
Zaiming (Stone) Shi 7a6494cda6
Merge pull request #10242 from zmstone/0326-fix-logging-avoid-using-pid-as-log-data-field
fix: avoid uisng 'pid' as log data field
2023-03-27 16:13:50 +02:00
JianBo He bfa5922209
Merge pull request #10140 from HJianBo/cassa
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
JimMoen 798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker
fix: for connection used ecpool, let worker do health check fun
2023-03-27 09:51:52 +08:00
JianBo He 7085a2d6d7
chore: hidden the password field
ref: https://github.com/emqx/emqx/issues/10222

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 09:47:51 +08:00
Zaiming (Stone) Shi 19e136bb66 fix: avoid uisng 'pid' as log data field
Since it clashes with log metad data field
2023-03-26 10:33:18 +02:00
Zaiming (Stone) Shi dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
Zaiming (Stone) Shi b37f186142 Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master 2023-03-24 21:47:15 +01:00
Thales Macedo Garitezi ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50
feat(buffer_worker): decouple query mode from underlying connector call mode
2023-03-24 13:49:00 -03:00
Thales Macedo Garitezi c92af4f763 docs: fix quote 2023-03-24 09:48:35 -03:00
Thales Macedo Garitezi a239f0523a
docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:23:06 -03:00
Thales Macedo Garitezi ca820dfefc
docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-24 09:21:08 -03:00
Thales Macedo Garitezi 5bdc71af6f docs(kafka_consumer): make specific labels for key/value encoding modes
https://emqx.atlassian.net/browse/EMQX-9312
2023-03-24 09:08:27 -03:00
JimMoen d0f0a93888
fix: ecpool worker exec dynamo health check fun 2023-03-24 11:15:52 +08:00
Thales Macedo Garitezi f8d5d53908 feat(buffer_worker): decouple query mode from underlying connector call mode
Fixes https://emqx.atlassian.net/browse/EMQX-9129

Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.

Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:

1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
Kinplemelon 28544e62df docs: correct the chinese label for the `socket_nodelay` field 2023-03-23 16:27:09 +08:00
Kinplemelon 1cb975a811
Merge pull request #10148 from Kinplemelon/kinple/kafka-nodelay-label
docs: correct the chinese label for the `socket_nodelay` field
2023-03-23 16:24:24 +08:00
JianBo He 1a466fda0b chore: use binary type InstanceId as PoolName 2023-03-23 16:06:51 +08:00
JianBo He 9b63bdc1e0 chore: apply review suggestions
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
JianBo He ac41c7e653 chore: format codes 2023-03-23 11:53:26 +08:00
JianBo He 8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
Thales Macedo Garitezi ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
2023-03-22 20:21:04 -03:00