Commit Graph

44 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 4094032649 fix(bridge v2 api): don't attempt to start disabled connector when starting action/source
Fixes https://emqx.atlassian.net/browse/EMQX-12435
2024-05-22 13:33:55 -03:00
ieQu1 bac5100635
Merge pull request #12874 from ieQu1/dev/EMQX-12030-subscriptions-api
API for durable subscriptions
2024-04-19 17:11:53 +02:00
ieQu1 d12966db5b
test: Avoid dumping raw snabbkaffe traces to the console 2024-04-19 01:20:29 +02:00
zmstone 51c8173174 feat(bridge): add is_template flag to bridge config fields 2024-04-17 13:10:55 +02:00
Kjell Winblad c3d7d68cfc refactor(kinesis,pgsql,timescale,matrix connectors): emqx_connector_info
This commit refactors the kinesis, pgsql, timescale and matrix
connectors to use the `emqx_connector_info` behavior.
2024-03-19 15:57:23 +01:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Thales Macedo Garitezi 64367d834b test(bridge_v2): actually check schema 2024-02-08 13:39:31 -03:00
Zaiming (Stone) Shi 82403167c2 chore: update BSL license change date 2024-01-29 16:47:31 +01:00
Thales Macedo Garitezi 14b99737e9 fix(mqtt_bridge): add missing fields to POST api spec; fix test 2024-01-16 10:42:52 -03:00
JimMoen 5e100f52b8
style: erlfmt all `rebar.config` files and `bin/nodetool` 2023-12-29 09:08:03 +08:00
Thales Macedo Garitezi a5d848515b fix: project sub-fields in action/connector `resource_opts` for some bridges
Fixes https://emqx.atlassian.net/browse/EMQX-11589
2023-12-14 12:09:32 -03:00
Kjell Winblad 0c80c74554 fix: remove unnecessary prepare_statement from pgsql action 2023-12-08 09:59:19 +01:00
Thales Macedo Garitezi fdeedc360e fix(actions_api): add missing fields to `POST` API schema 2023-12-06 11:13:09 -03:00
Thales Macedo Garitezi e03b8fd80e chore(postgres): prettify logged errors
Fixes https://emqx.atlassian.net/browse/EMQX-11490
2023-12-05 15:55:18 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Kjell Winblad c6c1d886f0 fix: make pgsql action schema properly 2023-11-24 21:19:26 +01:00
Kjell Winblad 66945dcc5c fix: address more comments from @thalesmg 2023-11-24 20:31:29 +01:00
Kjell Winblad 30e248061f
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-24 20:07:37 +01:00
Kjell Winblad f070d80b1a fix: swagger examples for PostgreSQL, Matrix and Timescale 2023-11-24 18:50:52 +01:00
Kjell Winblad d03674a505 fix: duplicate key in example 2023-11-24 17:59:56 +01:00
Kjell Winblad 3dca83c854 fix: all missing descriptions 2023-11-24 17:33:39 +01:00
Kjell Winblad f7296d549f fix: elvis problem 2023-11-24 17:33:39 +01:00
Kjell Winblad 64c015cf6f fix: unify the schema modules in emqx_bridge_pgsql 2023-11-24 17:33:39 +01:00
Kjell Winblad b8f510d956 fix: add env variables about action info 2023-11-24 17:33:39 +01:00
Kjell Winblad d5b62eead0 feat: split pgsql, matrix and timescale into connector action
This commit splits the bridges pgsql, matrix and timescale into
connector and action.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
2023-11-24 17:33:35 +01:00
Andrew Mayorov 4385b2f020
feat(pgsql): accept wrapped secrets as passwords
That are coming from `emqx_schema_secret`. Also adapt pgsql-related
connectors.
2023-11-14 16:05:23 +07:00
Andrew Mayorov 35902dc72d
feat(tpl): switch basic connectors to `emqx_connector_template`
Also avoid `filename:join/2` in HTTP connector since it's both OS specific
and an overkill.
2023-11-02 17:11:09 +07:00
Ilya Averyanov 4f14e8df22 chore(source dir): split out postgresql connector 2023-10-13 16:58:57 +03:00
Thales Macedo Garitezi d5847f33a8 test: attempt to stabilize flaky tests 2023-08-07 13:08:34 -03:00
Ivan Dyachkov 5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master 2023-07-27 16:58:09 +02:00
Thales Macedo Garitezi 2ac8ba2add test: fix flaky test 2023-07-25 17:43:33 -03:00
Thales Macedo Garitezi 7a16ff4f04 fix(postgres_bridge): fix table existence check and handle sync_required
Fixes https://emqx.atlassian.net/browse/EMQX-10629

During health checking, we check whether tables in the SQL statement
exist.  Such check was done by asking the backend to parse the
statement using a named prepared statements.  Concurrent health checks
could then result in the error:

```erlang
{error,{error,error,<<"42P05">>,duplicate_prepared_statement,<<"prepared statement \"get_status\" already exists">>,[{file,<<"prepare.c">>},{line,<<"451">>},{routine,<<"StorePreparedStatement">>},{severity,<<"ERROR">>}]}}
```

This could lead to an inconsistent state in the driver process, which
would crash later when a message from the backend (`READY_FOR_QUERY`, "idle"):

```
  2023-07-24T13:05:58.892043+00:00 [error] Generic server <0.2134.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,...
```

Added calls to `epgsql:sync/1` for functions that could return
`{error, sync_required}`.

Also, redundant calls to `parse2` were removed to reduce the number of requests.
2023-07-25 09:15:14 -03:00
Thales Macedo Garitezi 6cd503865b fix(machine_boot): ensure `emqx_bridge` starts after its companion apps
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
2023-07-20 13:11:44 -03:00
JimMoen b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
Paulo Zulato 9454af9a8b feat(postgresql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
Thales Macedo Garitezi 99796224d8 refactor(resource): rename `request_timeout` -> `request_ttl`
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Thales Macedo Garitezi 7f88521836 test(pgsql): reduce flakiness
Depending on timing, `t_write_timeout` was getting stuck while
checking the resource health, and the previous request timeout options
were making a response to never be sent if that process took too long.
2023-05-24 15:41:25 -03:00
JianBo He 383fec9dfc docs: fix invalid links 2023-05-15 11:18:09 +08:00
firest f6a2f752ff chore: update changes && bump app versions 2023-05-10 17:23:26 +08:00
firest 066ed5c6ec refactor(pgsql): move pgsql && matrix && timescale bridges into their own app 2023-05-10 16:51:33 +08:00
JianBo He c53ccfea61 chore: fix typos 2023-04-18 13:48:55 +08:00
JianBo He ec8d8b805f chore: add ee bridge apps 2023-04-18 09:36:41 +08:00
JianBo He 7eacbffae9 chore: create dirs and README template for all ee bridges 2023-04-18 09:36:41 +08:00