Commit Graph

690 Commits

Author SHA1 Message Date
Stefan Strigler
229bc0ee34 fix: schema fixes for connector and kafka as connector 2023-10-27 11:53:40 +02:00
Stefan Strigler
64faf32842 style: cleanup examples 2023-10-27 11:53:40 +02:00
Stefan Strigler
7641c22455 fix: add missing parts 2023-10-27 11:53:40 +02:00
Stefan Strigler
ed8aa46602 refactor: copy bridge api code over to emqx_connector 2023-10-27 11:53:40 +02:00
Stefan Strigler
d05f2010b3 fix: fail remove connector with active channels 2023-10-27 11:53:40 +02:00
Stefan Strigler
cc864f4821 feat: add update/3 2023-10-27 11:53:40 +02:00
Stefan Strigler
9195838c17 test: add disable_enable tests 2023-10-27 11:53:40 +02:00
Stefan Strigler
009b9c3d39 style: remove unused commented fn 2023-10-27 11:53:40 +02:00
Kjell Winblad
c0df85ac09 feat: Bridge V2 compatiblilty layer progress and local topic
* Most Bridge V1 HTTP API calls are now compatible with Bridge V2
* Local topics works for Bridge V2 now
* A lot of work on trying to get the old Kafka producer test suite
  to work after the refactorings
2023-10-27 11:53:40 +02:00
Stefan Strigler
16d7f4d3e6 style: cleanup emqx_connector_resource 2023-10-27 11:53:40 +02:00
Stefan Strigler
99904cab27 fix: don't break root for non enterprise 2023-10-27 11:53:40 +02:00
Stefan Strigler
bc6e1da2fb test: add suite for emqx_connector 2023-10-27 11:53:40 +02:00
Stefan Strigler
8f109da62b style: cleanup emqx_connector 2023-10-27 11:53:40 +02:00
Kjell Winblad
f7984be946 feat: add connector schema scaffold and break out Kafka conector
This commit is the beginning of an effort to split bridges into a
connector part and a bridge part.

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-27 11:53:40 +02:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool
2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
Ivan Dyachkov
24230a64df chore: bump app versions 2023-09-04 11:39:21 +02:00
Serge Tupchii
ed9afe3345 refactor(emqx_connector): use hocon_schema:override/2 to make pgsql 'username' field required 2023-09-01 19:56:07 +03:00
firest
0b066fa20c fix(ldap): fix dependency problem 2023-08-14 16:59:37 +08:00
firest
efff585b82 feat(ldap-authn): add test suite for the LDAP authenticator 2023-08-03 18:56:41 +08:00
Ivan Dyachkov
63adeabf72 chore: bump app versions 2023-07-27 15:29:03 +02:00
Thales Macedo Garitezi
7a16ff4f04 fix(postgres_bridge): fix table existence check and handle sync_required
Fixes https://emqx.atlassian.net/browse/EMQX-10629

During health checking, we check whether tables in the SQL statement
exist.  Such check was done by asking the backend to parse the
statement using a named prepared statements.  Concurrent health checks
could then result in the error:

```erlang
{error,{error,error,<<"42P05">>,duplicate_prepared_statement,<<"prepared statement \"get_status\" already exists">>,[{file,<<"prepare.c">>},{line,<<"451">>},{routine,<<"StorePreparedStatement">>},{severity,<<"ERROR">>}]}}
```

This could lead to an inconsistent state in the driver process, which
would crash later when a message from the backend (`READY_FOR_QUERY`, "idle"):

```
  2023-07-24T13:05:58.892043+00:00 [error] Generic server <0.2134.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,...
```

Added calls to `epgsql:sync/1` for functions that could return
`{error, sync_required}`.

Also, redundant calls to `parse2` were removed to reduce the number of requests.
2023-07-25 09:15:14 -03:00
Kjell Winblad
f28510b3ad refactor: HTTP connector into emqx_bridge_http app 2023-07-12 14:46:43 +02:00
firest
d01eee7fe4 chore: update changes && app version 2023-07-07 18:22:54 +08:00
firest
8cee75139d fix(bridge): ensure the username of pgsql must exists 2023-07-07 18:06:28 +08:00
Thales Macedo Garitezi
ca435975de fix(webhook): treat http status code 429 as recoverable 2023-06-30 09:46:03 -03:00
Thales Macedo Garitezi
59b109eb5c fix(webhook): treat 404 and other error replies as errors in async requests
Fixes https://emqx.atlassian.net/browse/EMQX-10405

The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
Kjell Winblad
85a130f68b refactor: move MySQL open source connector to its own app 2023-06-27 16:15:18 +02:00
Kjell Winblad
dd6aa96c99 refactor: move Redis open source connector to its own app 2023-06-27 10:24:40 +02:00
Kjell Winblad
b78a75bb5c refactor: the MongoDB connector into its own application 2023-06-26 14:32:59 +02:00
Thales Macedo Garitezi
2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
Thales Macedo Garitezi
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi
c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
Thales Macedo Garitezi
18f0510353 fix(mongo): return health check failure reason
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
Paulo Zulato
8430ec673c feat(kafka): check whether target topic exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
Zaiming (Stone) Shi
7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Thales Macedo Garitezi
dabefdb01b fix(gcp_pubsub): redact request when errors occur 2023-06-19 15:59:00 -03:00
Paulo Zulato
c9a2ddf98c feat(mysql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
Paulo Zulato
9454af9a8b feat(postgresql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
Andrew Mayorov
e6fb0203b4
refactor(pluglib): move connectivity checks to emqx_connector_lib 2023-06-09 14:44:37 +03:00
Andrew Mayorov
8919a6ef93
refactor(pluglib): provide SQL related utils in emqx_utils_sql 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
Zaiming (Stone) Shi
641166a67a
Merge pull request #10971 from zmstone/0607-merge-master-to-release-51
0607 merge master to release 51
2023-06-08 09:22:21 +02:00
JianBo He
0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple
feat: refactored some bridges to avoid leaking resources part.2
2023-06-08 09:37:31 +08:00
Zaiming (Stone) Shi
ccd2589ff2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-07 21:43:29 +02:00
Thales Macedo Garitezi
260fae296b feat(gcp_pubsub): generate jwt tokens on demand without workers (5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-9603

Rather than relying on a JWT worker to produce and refresh tokens, we
could just produce then on demand when pushing the messages to GCP
PubSub.  That can generate a bit of extra work (as multiple processes
might realize it’s time to refresh the JWT and do so), but that
shouldn’t be much.  In return, we avoid any possibility of not having
a fresh JWT when pushing messages.
2023-06-06 13:19:24 -03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
2023-06-05 22:41:07 +03:00
Thales Macedo Garitezi
46393343e2 chore: use timeout_duration types for timer fields
Fixes https://emqx.atlassian.net/browse/EMQX-10020
2023-06-05 11:46:38 -03:00