Commit Graph

222 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 47f826f99a chore: fix typo 2023-06-09 09:31:32 +08:00
Thales Macedo Garitezi 33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
2023-06-02 16:20:36 -03:00
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Paulo Zulato 3a6ad3b049 fix(oracle): require either sid or service name
Fixes https://emqx.atlassian.net/browse/EMQX-9980
2023-05-31 17:18:45 -03:00
Thales Macedo Garitezi 04bd39861d chore: bump app vsns 2023-05-30 11:08:20 -03:00
Thales Macedo Garitezi 9c3f838e14
Merge pull request #10841 from thalesmg/kafka-validate-key-v50
feat({kafka,pulsar}_producer): add validation for empty message key when strategy = key_dispatch
2023-05-30 09:37:15 -03:00
Thales Macedo Garitezi 3edbad9f56 feat(pulsar_producer): add validation for empty message key when strategy = key_dispatch 2023-05-29 10:04:19 -03:00
Zaiming (Stone) Shi 36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi 732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
Thales Macedo Garitezi 5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi f83f9a2d79 chore: bump app vsns 2023-05-17 14:56:11 -03:00
Kjell Winblad bce43978a2 refactor: the clickhouse bridge to use the new bridge app structure
Fixes:
https://emqx.atlassian.net/browse/EMQX-9538
2023-05-17 10:12:43 +02:00
firest 6497bcae46 refactor(influxdb): move influxdb bridge into its own app 2023-05-12 11:48:49 +08:00
JimMoen fbd516d740
Merge pull request #10622 from JimMoen/refactor-dynamo-bridge
refactor(dynamo): move dynamo bridge into its own app
2023-05-11 11:00:50 +08:00
JimMoen 8914e006c2
refactor(dynamo): move dynamo bridge into its own app 2023-05-10 18:49:21 +08:00
firest 066ed5c6ec refactor(pgsql): move pgsql && matrix && timescale bridges into their own app 2023-05-10 16:51:33 +08:00
firest 0f979b3d24 refactor(tdengine): move tdengine bridge into its own app 2023-05-10 14:12:57 +08:00
Kjell Winblad 70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00
firest a119100a0e refactor(rocketmq): move rocketmq bridge into its own app 2023-05-09 15:57:56 +08:00
Stefan Strigler d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
JimMoen d5dce771bd
refactor(sqlserver): move sqlserver bridge into its own app 2023-05-06 10:22:05 +08:00
firest 518b6e1211 fix(tdengine): redact the password for the TDEngine example API 2023-05-05 13:51:38 +08:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi 57cc854a4a test(bridge): fix bridge map type filed converters
now the converters on map type fields only work at the wrapping map
level but not the values
2023-04-30 10:45:11 +02:00
Paulo Zulato dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
zhongwencool 9d893b49eb
Merge branch 'master' into sync-release-50-to-master 2023-04-26 10:54:46 +08:00
Thales Macedo Garitezi ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-8398
2023-04-24 10:28:26 -03:00
JimMoen 4e7472090b
fix: refine default sql and driver name for mssql bridge 2023-04-23 15:47:18 +08:00
firest 5ad5d7ee8d fix(opents): adjust code structure 2023-04-23 14:28:14 +08:00
firest 5074825075 feat(opents): OpenTSDB integration 2023-04-23 09:56:24 +08:00
JianBo He b270623c46 chore: rename cassandra_impl to cassandra_connector 2023-04-21 16:49:25 +08:00
JianBo He bdce32e713 refactor(cassandra): move cassandra bridge into its own app 2023-04-21 14:45:38 +08:00
JianBo He 6532cfb337
Merge pull request #10452 from thalesmg/refactor-gcp-pubsub-bridge-v50
refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
2023-04-21 14:42:00 +08:00
Zaiming (Stone) Shi 415830a0a2 Merge remote-tracking branch 'origin/release-50' into 0420-sync-release-50-to-masteer 2023-04-20 11:03:39 +02:00
Thales Macedo Garitezi a9bd91fcff refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9536

Note: since GCP PubSub is not shared by any authn/authz backend,
there's no need to separate its connector into another app.
2023-04-19 13:24:32 -03:00
firest e89f4d4565 fix(dynamo): fix terminology erros
- Changed `username` to `aws_access_key_id`
- Changed `password` to `aws_secret_access_key`
2023-04-19 16:26:42 +08:00
firest bc353b0a06 fix(dynamo): change `database` to `table` in the schema of the DynamoDB bridge
there is no term like `database` in DynamoDB, the correct concept should be `table`
2023-04-19 15:56:29 +08:00
Ivan Dyachkov 9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Thales Macedo Garitezi e073bc90bc refactor(buffer_worker): rename `s/queue/buffer/g` 2023-04-14 11:37:19 -03:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
Thales Macedo Garitezi 871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Thales Macedo Garitezi 5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
firest 11b3264251 fix: redact the password to `******` in API examples
fix #10222
2023-04-04 16:02:45 +08:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Zaiming (Stone) Shi 677b76afce refactor: replace hidden => true with importance => hidden 2023-04-03 13:13:43 +02:00
lafirest 9c7c49e9bf
Merge pull request #10201 from lafirest/fix/tdengine
fix(tdengine): remove the redundant table name in SQL template
2023-04-03 18:13:41 +08:00
zhongwencool d63680cf25
Merge pull request #10307 from emqx/release-50
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Thales Macedo Garitezi 246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
Thales Macedo Garitezi 5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00