Commit Graph

134 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
firest 6081ce8b00 feat: refactored some bridges to avoid leaking resources during crashes at creation 2023-05-31 16:48:13 +08:00
Andrew Mayorov 26819a647c
fix(mqttbridge): clarify schema descriptions + log messages
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-05-30 22:21:22 +03:00
Andrew Mayorov 95cc9b9b72
fix(mqttbridge): ensure elixir release assembles successfully 2023-05-30 17:56:35 +03:00
Andrew Mayorov 1c2719236c
chore(mqttbridge): add README 2023-05-30 17:56:35 +03:00
Andrew Mayorov 7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Shawn 037b75a276 refactor(bridges): remote the old emqx_bridge_mqtt app 2021-09-10 11:43:03 +08:00
Shawn 298cf24f00 fix(bridges): mqtt bridge cannot forward msgs using payload template 2021-09-10 11:43:03 +08:00
Shawn eb8822ce41 feat(bridges): start one mqtt bridge worker for each in/out channel 2021-09-10 11:43:03 +08:00
Shawn 4f82debbe7 feat(bridge): new configs for mqtt bridge 2021-09-10 11:43:03 +08:00
Shawn 39bb1b8d9d fix(bridges): start mqtt bridge failed 2021-09-10 11:43:03 +08:00
Shawn 9ed90ba7a9 feat(bridges): add connector for mqtt bridge 2021-09-10 11:43:03 +08:00
Zaiming Shi ec13463f4a refactor(schema): prepare for hocon schema doc generation 2021-09-03 11:02:31 +02:00
Zaiming Shi 5165fd6b30 refactor(schema): implement new hocon_schema callbacks 2021-08-30 09:56:03 +08:00
Turtle e4f5e9332e feat: support array bridge_mqtt conf 2021-08-25 13:34:00 +08:00
Shawn f3efc89192
refactor(config): replace all ':' with '=' in the *.conf (#5531) 2021-08-19 19:57:42 +08:00
Shawn e8e95d39ef
refactor(config): move emqx_config:get/get_raw to emqx:get_config/get_raw_config (#5517) 2021-08-18 14:52:57 +08:00
Zaiming Shi e6232665a3 refactor(emqx): ensure app shutdown and reboot order
make use of application dependency to ensure start order.
also the same dependency top-sorted for shutdown and reboot order
2021-08-04 08:40:03 +02:00
Zaiming Shi 5d59ac1f02 refactor(logging): delete log headers from all modules 2021-08-01 09:36:59 +02:00
Turtle 9e97de04c9 chore(plugins): delete emqx_plugins module attributes 2021-07-30 14:32:01 +08:00
Turtle 918a26e921 feat(conf): merge all conf to emqx.conf 2021-07-02 17:01:16 +08:00
turtleDeng d4f726419f
feat(bridge-mqtt): Update the configuration file to hocon (#5142) 2021-07-01 10:51:59 +08:00
Turtle faad90c9d4 chore(plugins): rm emqx-web-hook and mv webhook action to emqx_rule_actions 2021-06-28 17:07:46 +08:00
Zaiming (Stone) Shi 913420588d
Merge branch 'dev/v5.0' into resolve-conflict-to-5.0 2021-05-31 09:17:21 +02:00
Turtle c02a9f0101 fix(appup): update appup 2021-05-27 13:55:07 +08:00
Zaiming Shi d904a1048b Merge remote-tracking branch 'origin/master' into merge-master-to-dev/v5.0 2021-04-30 15:09:31 +02:00
Zaiming Shi 6354e75626 chore: update copyrights 2021-04-23 20:43:00 +02:00
Zaiming Shi 2ffa71abde test(emqx_bridge_mqtt): fix race condition 2021-04-20 23:17:39 +02:00
Zaiming (Stone) Shi 24fdd7aef5
Merge pull request #4526 from zmstone/fix-mqtt-bridge-retry
fix(emqx_bridge_mqtt): fix retry_inflight
2021-04-13 12:04:14 +02:00
z8674558 b0c3c713b5 Merge branch 'master' of github.com:emqx/emqx into merge-master-to-v5 2021-04-13 00:09:17 +09:00
wwhai 765d84ef48
test(mgmt): add data import tests (#4481) 2021-04-12 09:06:22 +08:00
Zaiming Shi d66f67d411 fix(emqx_bridge_mqtt): fix retry_inflight
The Inflight list should not be used to update State.inflight
2021-04-10 14:16:08 +02:00
Zaiming Shi e4242814f6 fix(emqx_bridge_worker_SUITE): ensure snabbkaffe start stop 2021-04-09 10:18:25 +02:00
Zaiming Shi ae9278a409 fix(bridge_mqtt): push to max-inflight limit
the old pop_and_send implementation may leave inflight queue starving
2021-04-09 10:18:25 +02:00
Zaiming Shi ae688e2c90 fix(bridge_mqtt): fix inflight reference booking
Prior to this change, the inflight batches are referenced
by the last packet ID for non-QoS-0 messages, other packet
IDs sent back from downstream causes an error log:
"Can't be found from the inflight"

Even worse, the batch is appended back to the queue for retry.
2021-04-09 10:18:25 +02:00
z8674558 971e6ca90e Merge branch 'master' of github.com:emqx/emqx into merge-master-to-v5 2021-04-07 20:29:08 +09:00
Zaiming Shi 71559dd001 refactor(ssl): Move data dir for ssl certs upload down to lib module 2021-03-22 16:41:01 +01:00
Zaiming Shi 8a41a4ee00 fix(ssl): Fix dir name for uploaded certificates
The dir is "rules" not "rule" in the original code
ref: f1eba1c648/src/emqx_rule_actions_utils.erl
2021-03-22 16:41:01 +01:00
z8674558 efd4ef8091 Merge branch 'dev/v5.0' of private:emqx/emqx into fix-merge-conflict-master-to-5.0 2021-03-04 22:12:09 +09:00
z8674558 119bbc4881 chore(emqx_bridge_msg): add empty props 2021-03-03 20:06:31 +09:00
JianBo He 2fb5dbd546 revert(bridge-mqtt): remove the mqtt_sub resource
We have several reasons to remove this feature:

1. The design does not make sense. A rule engine resource should
not have an impact on the system's messages directly after it is created.
This mqtt_sub actually conflicts with any design concept of the rules engine.

2. The implementation is incorrect. mqtt_sub uses a client pool to establish
a subscription relationship to an MQTT Broker. This causes a message to be
sent repeatedly to EMQ X. Unless a shared subscription is used,
or a Pool Size of 1 is configured.

3. The emqx-bridge-mqtt supports all the features of mqtt_sub.

This feature introduced by https://github.com/emqx/emqx-bridge-mqtt/pull/78.
And it released to v4.2.0 (NOT WORK), v4.2.1-v4.2.7 (FIXED)
2021-03-02 18:56:21 +08:00
z8674558 2e199126dc fix(emqx_bridge_mqtt_actions): string -> file 2021-03-02 15:00:42 +09:00
z8674558 829a39eade fix(emqx_bridge_mqtt_actions): string -> password 2021-03-02 15:00:42 +09:00
Zaiming Shi 5edef84332
Merge branch 'dev/v5.0' into fix-merge-conflict-master-to-5.0 2021-03-01 19:58:16 +01:00
Shawn 990b7bd494 fix(bridge_mqtt): typos 2021-03-01 21:00:17 +08:00
Zaiming Shi 675603fd73 fix(bridge-mqtt): no ssl-option at all (empty list) when ssl is disabled 2021-03-01 21:00:17 +08:00
Zaiming Shi 700fa71754 refactor(tls): abstract lib for tls options parsing 2021-03-01 21:00:17 +08:00
Zaiming Shi da6f1104dc chore(bridge-mqtt): sync enterprise code 2021-03-01 21:00:17 +08:00
Zaiming Shi 3863b7f116 fix(config): Quote config string values 2021-02-12 11:16:45 +01:00
Zaiming Shi d11487268e Merge remote-tracking branch 'origin/dev/v4.3.0' into resolve-conflict-v4.3.0-to-v5.0 2021-02-12 10:36:16 +01:00
Zaiming Shi f000b6583c fix(tls): Ensure tls config integrity
For default tsl version and ciphers, we try to use otp release number
to determin if we want to use tlsv1.3

For default configs, we try to porivde both tlsv1.3 and
ciphers in config (even for commented out configs)
2021-02-09 22:56:28 +01:00
Zaiming Shi 4756eb7e5f
Merge branch 'dev/v5.0' into v4.3.0-to-v5.0-resolve-tls-config-conflicts 2021-02-09 09:37:19 +01:00
Zaiming Shi 7c94a02bdd
Merge pull request #4134 from jovdipp/tlsv1.3-as-default
Tlsv1.3 as default
2021-02-05 21:33:03 +01:00
Jóvan S. Dippenaar 5e15e3add8 fix(default-config-tls-ciphers): TLS1.3 exclusive ciphers prepended to TLS1.1+TLS1.2 ciphers 2021-02-05 16:21:28 +13:00
Jóvan Dippenaar c3c28f4b72 feat(tls.13): tls1.3 ciphers set in .conf's 2021-01-31 18:38:34 +13:00
z8674558 e8729fb00e Merge branch 'dev/v5.0' of private:emqx/emqx into merge430 2021-01-29 17:11:12 +09:00
JianBo He 456e3e5faa chore(bridge-mqtt): fix styles 2021-01-28 16:24:42 +08:00
JianBo He d98d26942c fix(bridge-mqtt): add max_inflight_size option back
For compatibility with versions below 4.2, we still
use the `max_inflight_size` option, but generate it
as the `max_inflight`

Previous PR: https://github.com/emqx/emqx/pull/3938
2021-01-28 16:24:42 +08:00
Zaiming Shi 937390fe0f
Merge pull request #4072 from emqx/dev/v4.3.0
Auto-pull-request-on-2021-01-25
2021-01-25 22:05:03 +01:00
JianBo He 67e56658a7 fix(bridge-mqtt): fix bad time unit 2021-01-25 15:35:04 +08:00
JianBo He 3de8fea435 fix(bridge-mqtt): fix bad received metrics counting 2021-01-25 15:35:04 +08:00
Yudai Kiyofuji 455f210b4c
feat(cuttlefish): use hocon as a parsing function (#4008)
* chore(conf): add quotation for hocon

* chore(conf): fix paths incompatible with hocon

* chore(conf): use hocon as parsing function

* chore(docker): add quotation to some env variables for hocon
2021-01-19 19:52:17 +09:00
Yudai Kiyofuji 56920b6ca6
fix(bridge_mqtt): max_inflight_size -> max_inflight; this enables emqtt.erl to use the value (#3938)
* fix(bridge_mqtt): max_inflight_size -> max_inflight; this enables emqtt.erl to use the value

* chore(bridge_mqtt): fix elvis

* fix(worker): max_inflight_batches -> max_inflight
2020-12-28 13:43:13 +08:00
Zaiming Shi d60212787b chore(build): remove emqx from app's applications list 2020-12-12 23:02:26 +01:00
Shawn 10dbb5bc96
Fix testcases warns (#3906) 2020-12-11 18:54:48 +08:00
Zaiming Shi 7bdd093ef5 Merge remote-tracking branch 'origin/dev/v4.3.0' into umbrella-fix-build 2020-12-11 08:55:57 +01:00
Zaiming Shi 3a91ff8a7b chore(proj): resync apps 2020-12-11 08:19:26 +01:00
Ayodele 56d7b3deb1 Umbrella fix build (#8) 2020-12-10 11:25:25 +01:00
Ayodele 0cb9cbce71 Umbrella fix build (#6)
* fix(emqx_bridge_rpc): Dialyzer warnings

* fix(emqx_coap_app): Dialyzer warnings

* fix(emqx_coap_mqtt_adapter): Dialyzer warnings

* fix(emqx_lwm2m_json): Dialyzer warnings

* fix(emqx_lwm2m_message): Dialyzer warnings

* fix(emqx_mgmt): Dialyzer warnings

* fix(emqx_mgmt_cli): Dialyzer warnings

* fix(emqx_mgmt): Dialyzer warnings

* fix(emqx_bridge_rpc): Dialyzer warnings

* fix(emqx_mgmt): Dialyzer warnings

* fix(emqx_exproto_gcli): Dialyzer warnings
2020-12-10 11:22:38 +01:00
Ayodele 13b67c0d19 Umbrella fix build (#5)
* fix(emqx_auth_jwt): Dialyzer warnings

* fix(emqx_auth_mnesia_cli): Dialyzer warnings

* fix(emqx_bridge_mqtt_cli): Dialyzer warnings

* fix(emqx_bridge_mqtt_cli): Dialyzer warnings

* fix(emqx_auth_redis_sup): Dialyzer warnings

* fix(emqx_bridge_mqtt): Dialyzer warnings

* fix(emqx_auth_pgsql_cli): Dialyzer warnings

* fix(emqx_bridge_mqtt_cli): Dialyzer warnings

* fix(emqx_auth_mnesia_cli): Dialyzer warnings
2020-12-10 11:22:38 +01:00
Zaiming Shi aac3d79b7d fix(build): pre-compile parse_transform modules 2020-12-09 00:28:39 +01:00
Zaiming Shi 80e1d6cc9d chore(apps): delete LICENSE files for apps 2020-12-08 22:28:46 +01:00
Zaiming Shi 7fdbfba06a Merge branch 'umbrella-for-430-auto-sync' into umbrella-for-430 2020-12-08 22:09:59 +01:00
Zaiming Shi 2fda0a1565 chore(apps): Sync again all apps 2020-12-07 21:30:21 +01:00
Zaiming Shi 036eda27f6 chore(rebar): Delete profiles from apps' rebar.config
otherwise rebar3 crashes
2020-12-05 13:36:56 +01:00
Zaiming Shi db792ec577 chore(apps): Sync again with 4.3.0 2020-12-05 03:02:35 +01:00
Zaiming Shi 8a12018863 Add more cleanups 2020-12-05 02:45:01 +01:00
Zaiming Shi 0fb5fb31a5 refactor(proj) sync 4.3.0 plugins with tracked files 2020-12-05 02:43:04 +01:00
Zaiming Shi 686c006d6e refactor(proj): Add apps 2020-12-04 22:11:53 +01:00
Zaiming Shi 73d02beace refactor(apps): Merge emqx_libs back to emqx
And move emqx app back to root level

After realising the challenges of developing plugins in standalone
rebar projects. it is perhaps more smooth to keep using emqx as
a rebar dependency
2020-12-04 19:58:58 +01:00
Zaiming Shi 2cbd2bc800 chore(app.src): use semver 4.3.0 for apps version base 2020-12-03 19:59:41 +01:00
Zaiming Shi 5bb0cccb53 refactor(build): Clean up app version management
Starting from 5.0.0, all apps in apps dir should follow semver
strictly. Do not use the umbrella project's release tags.

The universal tags made sense when we have applications scattered
in different repos. Now since this is an umbrella project, it is
enough to have a single package level version number corelated with
git tag.
2020-12-01 23:22:33 +01:00
terry-xiaoyu e586119752 fix(project): point the hrl files to emqx_lib 2020-10-31 12:34:52 +08:00
terry-xiaoyu 0cd1c57a54 Migrate plugins from tag 4.2.2 2020-10-31 12:10:19 +08:00