Commit Graph

42 Commits

Author SHA1 Message Date
Ivan Dyachkov
606d829256 Merge remote-tracking branch 'upstream/release-57' into 20240724-sync-release-57 2024-07-24 10:28:00 +02:00
zhongwencool
e7d07ea17c feat: add resource_type to emqx_resource behaviour 2024-07-23 18:24:28 +08:00
Kjell Winblad
b994e0f1c0 fix: make RabbitMQ error log messages easier to understand
Fixes:
https://emqx.atlassian.net/browse/EMQX-12462
2024-07-04 16:43:53 +02:00
zmstone
5d1608f34b chore: minimize oss/platform diff 2024-06-24 15:30:03 +02:00
Kjell Winblad
ef9884cf47 refactor(rule trace): templates rendered trace to increase code reuse
* The code for passing the trace context to a sub process has been
  improved to increase code reuse. This code is used when the action
  templates are rendered in a sub process.
* A macro has also been added for the error term that is thrown when the
  action shall be stopped after the templates has been rendered. This is
  also done to reduce code duplication and to reduce the risk of
  introducing bugs due to typos.
* Fix incorrect type spec

Thanks to @zmstone for suggesting these improvements in comments to a PR
(https://github.com/emqx/emqx/pull/12916).
2024-04-25 13:15:36 +02:00
Kjell Winblad
d6ceeb3b30 feat: add stop after render and after render trace to rabbitmq action 2024-04-25 11:51:46 +02:00
zmstone
51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
Ivan Dyachkov
db9efb9317 chore: bump apps versions 2024-03-28 10:19:09 +01:00
Kjell Winblad
4bfc51c052 refactor(emqx_connector_info): improve documentation and type specs 2024-03-21 16:08:31 +01:00
Kjell Winblad
a3b75eb39d refactor(remaining connectors): to use emqx_connector_info
This commit refactors the sqlserver, iotdb, elasticsearch, opents,
greptimedb, pulsar, tdengine, rabbitmq and s3 connectors to use the
`emqx_connector_info` behavior.
2024-03-21 09:47:57 +01:00
zhongwencool
7f1b4cef27 feat: pulsar bridge v2 2024-02-29 16:23:37 +08:00
zhongwencool
51a6ada015 fix: rabbitmq action failed to increase dropped metrics 2024-02-23 08:35:00 +08:00
zhongwencool
ed23ad906a fix: rabbitmq source metrics not working 2024-02-21 21:10:23 +08:00
zhongwencool
a0f8e4f328 test: rafator rabbitmq test SUITE, delete redundance code 2024-02-08 12:38:18 +08:00
zhongwencool
90ba2977fe feat: don't publish mqtt message in rabbitmq's source 2024-02-08 12:38:18 +08:00
zhongwencool
e3cb49db8e chore: add copyright header 2024-02-06 07:57:13 +08:00
zhongwencool
688701eedb chore: add desc for rabbitmq source 2024-02-06 07:57:13 +08:00
zhongwencool
b444c82a42 feat: rabbitmq bridge v2 integration 2024-02-06 07:57:12 +08:00
Ivan Dyachkov
9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Ivan Dyachkov
7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Andrew Mayorov
34aeeab041
feat(rabbitmq): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Zaiming (Stone) Shi
e653c6b4e0 chore: bump app versions 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi
d603de10e6 refactor(schema): add namespace/0 to all schema modules 2023-11-10 13:41:51 +01:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Thales Macedo Garitezi
399f849f7b feat(rabbitmq_bridge): add TLS support
Fixes https://emqx.atlassian.net/browse/EMQX-10605
2023-07-27 15:52:46 -03:00
Thales Macedo Garitezi
6cd503865b fix(machine_boot): ensure emqx_bridge starts after its companion apps
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
2023-07-20 13:11:44 -03:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
Andrew Mayorov
b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder
refactor: tear `emqx_plugin_libs` application apart
2023-06-09 17:02:14 +02:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
Kjell Winblad
ed9e29e769 refactor: refacor query_mode detection code
This commit refactor the query_mode resource detection code according to
a suggestion from @zmstone. This commit should not contain any
functional change except for a change of the Kafka producer bridge
config.
2023-06-08 16:26:55 +02:00
JianBo He
0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple
feat: refactored some bridges to avoid leaking resources part.2
2023-06-08 09:37:31 +08:00
Thales Macedo Garitezi
46393343e2 chore: use timeout_duration types for timer fields
Fixes https://emqx.atlassian.net/browse/EMQX-10020
2023-06-05 11:46:38 -03:00
JianBo He
dbc0cdce67 chore: dont allocate resource for simple connectiors 2023-06-05 10:59:00 +08:00
JianBo He
e717ddafd7 chore: update changes 2023-06-05 10:19:46 +08:00
JianBo He
3739230435 feat: refactor connectors on_stop function to avoid resources leaking
Supplement to https://github.com/emqx/emqx/pull/10895
2023-06-05 10:19:46 +08:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
2023-06-02 16:20:36 -03:00
Thales Macedo Garitezi
0072f4a598 fix(ecpool,bridge): treat {error, ecpool_empty} as a retriable error
Example from flaky CI test:

```
/emqx/apps/emqx_resource/src/emqx_resource_buffer_worker.erl:716 2023-06-01T19:30:37.119862+00:00 [buffer_worker_flush_ack] #{batch_or_query => [{query,undefined,{send_message,#{payload => <<"-576460752303420927">>,timestamp => 1668602148000,topic => <<"t_write_failure">>}},false,-576460614954536809}],queue_count => 0,result => {async_return,{error,{unrecoverable_error,ecpool_empty}}},'~meta' => #{gl => <0.3883.5>,node => 'test@127.0.0.1',pid => <0.4724.5>}}.
```
2023-06-01 17:13:06 -03:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Kjell Winblad
a4fcc0dcb3 fix: password log leak in RabbitMQ bridge
This fixes a vulnerability in the RabbitMQ bridge, which could
potentially expose passwords to log files. This was accomplished by
initializing the encryption library specifically designed for RabbitMQ's
passwords. Consequently, passwords are no longer stored in unencrypted
format. As a result, they will no longer be visible as plain text in log
messages, thereby enhancing the system's security.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9976
2023-05-30 14:36:02 +02:00
Kjell Winblad
ea2fea77d7 fix: RabbitMQ field not marked as required
This commit makes sure that the RabbitMQ password filed is marked as
required. This ensures that the user provides a password and that the
bridge does not throw a function clause exception if the password filed
is not set.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9974
2023-05-23 15:50:18 +02:00
Kjell Winblad
41f5eff014 fix: issues with the RabbitMQ config 2023-05-10 10:53:40 +02:00
Kjell Winblad
70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00