7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
...
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
c58a98954b
Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master
2023-06-22 11:05:51 +02:00
18f0510353
fix(mongo): return health check failure reason
...
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
8430ec673c
feat(kafka): check whether target topic exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
7cf8a6c892
chore: bump app vsns
2023-06-21 16:36:51 +02:00
dabefdb01b
fix(gcp_pubsub): redact request when errors occur
2023-06-19 15:59:00 -03:00
c9a2ddf98c
feat(mysql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
9454af9a8b
feat(postgresql): check whether target table exists
...
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
0d6d441f4c
test(emqx_connector): start/stop test for webhook bridge
2023-06-14 09:56:50 +02:00
b2a5065641
fix(emqx_connector): report errors in on_start handler
2023-06-13 16:57:08 +02:00
e6fb0203b4
refactor(pluglib): move connectivity checks to `emqx_connector_lib`
2023-06-09 14:44:37 +03:00
8919a6ef93
refactor(pluglib): provide SQL related utils in `emqx_utils_sql`
2023-06-09 14:44:37 +03:00
d6c1ee183f
refactor(pluglib): move `emqx_placeholder` to utils app
...
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
641166a67a
Merge pull request #10971 from zmstone/0607-merge-master-to-release-51
...
0607 merge master to release 51
2023-06-08 09:22:21 +02:00
0b95bc1c1b
Merge pull request #10907 from HJianBo/refactor-on-stop-simple
...
feat: refactored some bridges to avoid leaking resources part.2
2023-06-08 09:37:31 +08:00
ccd2589ff2
Merge remote-tracking branch 'origin/master' into release-51
2023-06-07 21:43:29 +02:00
260fae296b
feat(gcp_pubsub): generate jwt tokens on demand without workers (5.1)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9603
Rather than relying on a JWT worker to produce and refresh tokens, we
could just produce then on demand when pushing the messages to GCP
PubSub. That can generate a bit of extra work (as multiple processes
might realize it’s time to refresh the JWT and do so), but that
shouldn’t be much. In return, we avoid any possibility of not having
a fresh JWT when pushing messages.
2023-06-06 13:19:24 -03:00
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys
...
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
2023-06-05 22:41:07 +03:00
46393343e2
chore: use `timeout_duration` types for timer fields
...
Fixes https://emqx.atlassian.net/browse/EMQX-10020
2023-06-05 11:46:38 -03:00
dbc0cdce67
chore: dont allocate resource for simple connectiors
2023-06-05 10:59:00 +08:00
0072f4a598
fix(ecpool,bridge): treat `{error, ecpool_empty}` as a retriable error
...
Example from flaky CI test:
```
/emqx/apps/emqx_resource/src/emqx_resource_buffer_worker.erl:716 2023-06-01T19:30:37.119862+00:00 [buffer_worker_flush_ack] #{batch_or_query => [{query,undefined,{send_message,#{payload => <<"-576460752303420927">>,timestamp => 1668602148000,topic => <<"t_write_failure">>}},false,-576460614954536809}],queue_count => 0,result => {async_return,{error,{unrecoverable_error,ecpool_empty}}},'~meta' => #{gl => <0.3883.5>,node => 'test@127.0.0.1',pid => <0.4724.5>}}.
```
2023-06-01 17:13:06 -03:00
cb31032e6d
chore: make dialyzer && test happy
2023-06-01 11:15:51 +08:00
6081ce8b00
feat: refactored some bridges to avoid leaking resources during crashes at creation
2023-05-31 16:48:13 +08:00
7e7b50c5ba
refactor(mqttbridge): move into separate application
2023-05-30 17:56:34 +03:00
ebd612b194
chore: bump applications versions
...
* emqx_connector 0.1.25
* emqx_rule_engine 5.0.19
* emqx_ee_bridge 0.1.15
2023-05-30 17:21:45 +03:00
eed9358abd
chore: bump `ecpool` to 0.5.4
...
With fixed typings and empty pool handling.
2023-05-30 17:21:45 +03:00
a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools
...
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
2023-05-30 17:21:44 +03:00
c7528e9b35
feat(mqttconn): add `pool_size` config parameter
...
That currently tunes the number of MQTT clients employed both for
subscriptions (if shared subscription is used) and for publishing to
a remote broker.
2023-05-30 17:21:44 +03:00
6e97dffdb8
feat(mqttconn): deprecate `mode` config parameter
...
It adds no value: the only mode was `cluster_shareload` and we just as
well can decide to "share" the load across cluster just by looking if
the remote topic is shared subcription filter or not.
2023-05-30 17:21:44 +03:00
4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription
...
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
2023-05-30 14:50:05 +03:00
81e78516aa
feat(mqttconn): employ ecpool instead of a single worker
2023-05-30 14:50:05 +03:00
6967f621d8
fix(mqttconn): unify error interpretation in sync/async modes
...
Also move this logic to the mqtt connector itself, in order to avoid
dealing with extra callback layer.
2023-05-30 14:50:05 +03:00
67d703f8c5
refactor(mqttconn): simplify mqtt connector
...
Inline `emqx_connector_mqtt_msg` module code into
`emqx_connector_mqtt_worker` module, since it's not really used
anywhere else and does not provide any reusable abstractions.
2023-05-30 14:50:05 +03:00
bd956d00b6
feat(mqttconn): stop using gproc in hot path
...
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
2023-05-30 14:50:05 +03:00
4da0d83faf
chore(mqttconn): remove dead code
2023-05-30 14:50:04 +03:00
36e268c933
chore: bump app versions
2023-05-26 16:05:37 +02:00
cc5b4d3748
Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test
2023-05-26 15:58:16 +02:00
194faefc8a
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-10001
Recently, we unified request_timeout in a single field located at the
webhook connector schema. However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
2023-05-25 13:19:27 -03:00
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50
2023-05-24 19:34:12 +08:00
256cb73e43
Merge pull request #10792 from thalesmg/fix-webhook-bridge-health-check-clause-r50
...
fix(webhook): fix empty ehttpc worker list clause on health check (r5.0)
2023-05-24 08:46:59 +02:00
25437adf02
fix(webhook): fix empty ehttpc worker list clause on health check (r5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9970
2023-05-23 14:06:40 -03:00
ea2fea77d7
fix: RabbitMQ field not marked as required
...
This commit makes sure that the RabbitMQ password filed is marked as
required. This ensures that the user provides a password and that the
bridge does not throw a function clause exception if the password filed
is not set.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9974
2023-05-23 15:50:18 +02:00
732a7be187
Merge remote-tracking branch 'origin/release-50'
2023-05-22 17:46:54 +02:00
be90c63c78
chore(mqtt-connector): refine logging level
...
connect failure should be at warning level but not error,
the connecting state is visiable from dashbaord
also the resource manager logs connection failures in general
at warning level
2023-05-19 18:55:16 +02:00
5bbcf4b712
fix(mqtt-connector): faster shutdown
2023-05-19 18:55:16 +02:00
f83f9a2d79
chore: bump app vsns
2023-05-17 14:56:11 -03:00
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50
...
perf(webhook): add async retries and evaluate reply callback in fresh process
2023-05-17 11:06:48 -03:00
bc7d0d5fd6
perf(webhook): evaluate replies in fresh process rather than in ehttpc
...
This surprisingly simple change yields a big performance improvement
in throughput.
While the previous commit achieves ~ 55 k messages / s
in throughput under some test conditions (100 k concurrent publishers
publishing 1 QoS 1 message per second), the simple change in this
commit improves it further to ~ 63 k messages / s.
Benchmarks indicated that the evaluating one reply function is
consistently quite fast (~ 20 µs), which makes this performance gain
counterintuitive. Perhaps, although each call is cheap, `ehttpc`
calls several of these in a row when there are several sent requests,
and those costs might add up in latency.
2023-05-17 09:20:50 -03:00
a7b41e1cdf
perf(webhook): add retry attempts for async
...
This is a performance improvement for webhook bridge.
Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well. Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.
This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
2023-05-17 09:20:50 -03:00
01cf7fc703
feat(webhook): use clientid for hash-loadbalancing ehttpc workers
2023-05-17 09:20:50 -03:00
7566c25199
refactor: delete unused connector field 'base_url'
2023-05-11 09:04:55 +02:00
83257617cd
Merge pull request #10630 from zhongwencool/sync-release-50-to-master
...
Sync release 50 to master
2023-05-08 14:02:27 +02:00
9e324e8523
style: fix typo
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-05-08 09:56:43 +02:00
b639353cf4
fix(emqx_connector_http): wrap and unwrap auth headers
...
This so we don't have sensitive information in a process' state
2023-05-08 09:56:43 +02:00
4f396a36a9
Merge remote-tracking branch 'upstream/master' into release-50
2023-05-08 14:58:03 +08:00
a4bc15cfa1
Merge remote-tracking branch 'origin/master' into 0502-merge-release-50-back-to-master
2023-05-02 18:59:31 +02:00
f6da18dc1b
fix(webhook): consider `{shutdown, closed}` return as recoverable
2023-05-02 13:12:03 -03:00
d5f5f35787
Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master
2023-05-02 18:08:04 +02:00
90cf1ade74
chore: bump application versions
...
* emqx_connector 0.1.22
* emqx_bridge_gcp_pubsub 0.1.1
* emqx_bridge_kafka 0.1.2
* emqx_bridge_pulsar 0.1.1
* emqx_ee_connector 0.1.12
2023-05-02 17:29:22 +03:00
aaef95b1da
feat(resman): stop adding uniqueness to manager ids
...
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
2023-05-02 17:28:26 +03:00
a19621e533
fix(webhook): treat `{shutdown, normal}` and `{closed, _}` async reply as retriable
...
Apparently, the async reply returned by ehttpc can be `{shutdown,
normal}` or `{closed, "The connection was lost."}`, in which case the
request should be retried.
```
Apr 28 17:40:41 emqx-0.int.thales bash[48880]: 17:40:41.803 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:shutdown, :normal}]
Apr 28 18:36:37 emqx-0.int.thales bash[53368]: 18:36:37.605 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:closed, 'The connection was lost.'}]
```
2023-04-28 18:07:32 -03:00
377b143325
refactor: split `parse_server` into smaller functions, improve return type to use map
2023-04-24 14:17:29 -03:00
fdf9b2a383
chore: apply review suggestions
2023-04-21 12:19:38 +08:00
8c4b32580e
Merge pull request #10420 from savonarola/0417-authn-authz-fix-path-quoting
...
Fix path joining and quoting in authn/authz
2023-04-20 18:03:04 +03:00
9efbb71ae8
Merge pull request #10441 from zmstone/0418-fix-flakyness
...
test: pick random port number for gcp pubsub mock server
2023-04-18 19:11:30 +02:00
dc78ecb41c
chore: merge upstream/master
2023-04-18 17:33:32 +02:00
97b8f00508
test: pick random port number for gcp pubsub mock server
2023-04-18 15:07:57 +02:00
21e19a33ce
feat(respool): switch to `emqx_resource_pool`
...
Which was previously known as `emqx_plugin_libs_pool`. This is part
of the effort to get rid of `emqx_plugin_libs` application.
2023-04-18 12:51:14 +03:00
88ca94b417
fix(auth): fix uri path handling
...
Fix uri path handling `emqx_connector_http`,
HTTP authentication and authorization backends.
2023-04-17 23:57:56 +03:00
9fc8a498f8
chore: bump apps versions
2023-04-17 09:09:08 +02:00
062ce5f819
refactor: rename emqx_map_lib to emqx_utils_maps
2023-04-14 13:41:34 +02:00
9c11bfce80
refactor: rename emqx_misc to emqx_utils
2023-04-14 13:41:27 +02:00
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
8ceeafb0de
chore: fix file license in lib-ee
2023-04-14 09:56:59 +08:00
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
806ee716ee
fix(redis): remove `required` meta from schema
...
Since there's a default value, this field shouldn't be marked as
required.
2023-03-29 16:43:52 -03:00
13a35cc045
fix: schema default value so that it shows up correctly in UI
...
This complements PR https://github.com/emqx/emqx/pull/10124 .
The default values for duration_ms() fields needs to be formatted
as a binary string with unit to show up correctly in the dashboard
UI.
2023-03-15 11:04:16 +01:00
a530ccbe3d
Merge pull request #10095 from fix/EEC-782/mysql-prepstmt-exhaustion
...
fix(mysql): be explicit that batch queries are parameterless
2023-03-14 10:21:45 +03:00
1cf01197bb
fix: increase heartbeat time to avoid extreme MongoDB logging
...
Our MongoDB driver creates a new temporary connection, for every active
connection, to just do a single heartbeat test. There is configurable
delay between every heartbeat test. When the user has an EMQX cluster
with a MongoDB bridge (to a MongoDB replica set), there will be a lot of
connections. Furthermore, as MongoDB creates a log entry every time a
new connection is created, the log will be flooded with info about new
connection. One user have reported more than 1MB of log data in a 10
minute period.
This commit tries to fix this by increasing the default delay between
heartbeats. A better fix would be to change the MongoDB driver so that
it does not create a new connection just to do a heartbeat check, but
this is more complicated so we leave this to the future. We might also
swap out the current MongoDB driver to something better.
Fixes:
https://github.com/emqx/emqx/issues/9851
2023-03-13 14:58:53 +01:00
0a7f6c7d03
fix(mysql): ensure proper escaping in batch inserts
...
Also hexencode non-utf8 binaries. This is essentially an heuristic.
We don't know column types in runtime, and there's no simple way
to find them out. Since we're already doing full binary scan during
escaping it should be cheap to bail out on non-utf8 strings and
hexencode them instead.
Also introduce separate function to highlight that this escaping
is MySQL-specific.
2023-03-10 18:42:04 +03:00
fc37d9b3cd
fix(mysql): be explicit that batch queries are parameterless
...
So that mysql client won't attempt to prepare them automatically, thus
trashing the server's prepared statements table and making interaction
overall heavier.
2023-03-10 18:42:04 +03:00
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
ca947e3e70
fix: lost messages when HTTP connection times out
...
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.
Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
c01f62a1c1
chore: bump apps versions
2023-03-03 13:58:32 +01:00
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
7d762593f6
fix(connector): redact the http body in error logs for security reasons
2023-02-27 11:41:34 +08:00
a1ba7bc004
Merge pull request #9976 from zmstone/0215-docs-improve-bridge_mode-description
...
docs: improve bridge_mode description
2023-02-21 20:55:20 +01:00
9316690c29
fix(schema): binary string for default values
...
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
2023-02-21 09:09:51 +01:00
dbda504f2c
fix(pgsql): fix ssl option for pgsql connector to match previous behavior (5.0)
...
Fixes https://github.com/emqx/emqx/issues/9907
At v5.0.14, we changed the `ssl` option for the Postgres connector
from `true` to `required`, but there was another transformation in
`conn_opts/2` that led to an incorrect configuration. This change
ended up preventing users from connecting to Postgres with their
previous configurations after upgrading EMQX.
2023-02-15 14:06:33 -03:00
1cf88ea0b4
docs: improve bridge_mode description
2023-02-15 15:24:10 +01:00
81adb02402
chore: bump apps version
2023-02-14 22:04:29 +08:00
530b21268a
feat(bridges): add TDengine
2023-02-14 22:04:23 +08:00
72fe10f18b
chore: fix spell check errors
2023-02-10 19:41:14 +01:00
fe450ca2d9
Merge pull request #9938 from keynslug/feat/mqtt-bridge-async-errors
...
feat(mqtt-bridge): report recoverable errors of async queries
2023-02-10 20:40:58 +04:00
cbb2885499
fix(mqtt-bridge): allow to configure `clean_start` for ingresses
2023-02-10 19:15:48 +03:00
7002fe2ef4
fix(mqtt-bridge): disallow QoS 2 on ingress bridges
2023-02-10 17:17:59 +03:00
8bd7c0eb3f
feat(mqtt-bridge): report recoverable errors of async queries
...
This should help to avoid delivery failures of messages which could
be safely retried, in the event of intermittent connectivity loss
for example.
It should now be safe since 73d5592b
2023-02-10 12:26:59 +03:00
3e85cb3804
Merge pull request #9946 from zmstone/0209-fix-add-back-deprecate-field
...
fix(config): add back reconnect_interval field as deprecated
2023-02-10 09:19:12 +01:00
5328955ff4
fix(config): add back reconnect_interval field as deprecated
2023-02-09 17:30:09 +01:00
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
1c748070c1
fix(emqx_connector_mqtt_worker): publish_async can only return ok # Please enter the commit message for your changes. Lines starting
2023-02-02 14:11:12 +01:00
2b308af354
Merge pull request #9889 from keynslug/fix/EMQX-8857/force-clean-start-mqtt-bridge
...
fix(mqtt-bridge): stop respecting `clean_start` config parameter
2023-02-02 12:55:58 +01:00
0912f13c1f
fix(mqtt-bridge): stop respecting `clean_start` config parameter
...
We are ignoring the user configuration because there's currently no
reliable way to ensure proper session recovery according to the MQTT
spec.
2023-02-02 12:55:10 +03:00
2cf193e2fd
Merge pull request #9839 from kjellwinblad/kjell/fix/Authorization_header_log_leak_webhook/EMQX-8791
...
fix: Authorization header leak in log entries for webhook
2023-02-02 10:38:42 +01:00
ca5c192f4b
Merge pull request #9882 from fwup/fix/no-mqtt-bridge-middleman
...
refactor(mqtt-worker): avoid unnecessary abstraction
2023-02-02 13:11:31 +04:00
32d18ec594
Merge pull request #9878 from zmstone/0201-upgrade-ehttpc-0.4.6
...
fix(ehttpc): fix crash when webhook bridge uses `hash` pool type.
2023-02-01 15:00:28 +01:00
f0395be383
refactor(mqtt-worker): avoid unnecessary abstraction
...
So the code is easier to follow.
2023-02-01 14:46:26 +03:00
9f6b6cedc6
fix(webhook): pick worker according to pool type
2023-02-01 09:53:25 +01:00
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover
...
fix: fix redis cluster resource recovery
2023-02-01 10:38:52 +02:00
f4381d90ca
refactor: http cliet should not need to retry HTTP requests for bridge
2023-02-01 09:30:30 +01:00
6ca816f7a4
docs: remove impl details from user doc
2023-02-01 09:30:10 +01:00
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback
...
feat(mqtt-bridge): avoid middleman process
2023-01-31 19:12:18 +01:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
fce1e74c3d
fix(connector): fix redis cluster resource recovery
2023-01-31 16:55:05 +02:00
4d146c521b
fix(mqtt-bridge): ensure proper feedback on async forwards
...
So that buffer worker would notice a connection loss in time, and
recycle inflight messages subsequently.
2023-01-31 11:09:39 +03:00
35c429ef1d
refactor: drop a couple of unused macros / includes
2023-01-31 11:09:39 +03:00
0c4134c423
test: add unit test case for redact function in http connector
2023-01-30 11:51:28 +01:00
72f39b9b72
fix(docs): correct Redis conf field description
2023-01-27 17:39:16 +02:00
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching
...
feat(redis): disable batching in redis_cluster bridges
2023-01-27 15:27:57 +04:00
71f996b9d5
refactor(mqtt-bridge): unwrap single statem actions
...
So that the code would be easier to follow and harder to break.
Also drop a couple of unused macrodefs.
2023-01-26 21:13:18 +03:00
c95f979413
fix(mqtt_bridge): use correct gen_statem reply action
2023-01-26 21:12:05 +03:00
86cfbfb43c
fix: Authorization header leak in log entries for webhook
...
There might be another possibility for leakage. If the resource mangager
for the webhook resource crashes, OTP might log the spec for the
resource manager which contains the Config and thus the Authorization
header. This is probably an issue for other resources as well and should
be fixed in another commit. The following issue has been created for
that:
https://emqx.atlassian.net/browse/EMQX-8794
Fixes:
https://emqx.atlassian.net/browse/EMQX-8791
2023-01-26 12:44:16 +01:00
2ee00b75a7
fix(redis): unwrap pipeline queries against redis cluster
...
This is an additional safety measure in addition to the disabled
batching on the bridge level.
2023-01-25 17:28:11 +03:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
16f45a60fd
chore: i18n typo fix
2023-01-20 11:50:01 +08:00
d755b43c77
fix(jwt_worker): handle exceptions when decoding jwk from pem
...
Returns a more controlled error if users attempt to use the Service
Account JSON from the GCP PubSub example from swagger, which is
redacted.
2023-01-19 09:24:45 -03:00
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
006b4bda97
feat(buffer_worker): monitor async workers and cancel their inflight requests upon death
2023-01-17 16:48:48 -03:00
4a7e74f5d6
fix(schema): add password converter to ensure its binary() type
2023-01-17 11:12:17 +01:00
44f635300d
chore: update mysql-otp to 1.7.2
2023-01-17 13:36:26 +08:00
54cea3b3f2
chore: update epgsql to 4.7.0.1 with full OTP-25 compat
2023-01-16 15:35:51 +03:00
9a53410efb
fix: don't add empty list element to options proplist
2023-01-16 12:00:03 +01:00
edfcea7f43
fix: add log for timeout
2023-01-16 12:00:03 +01:00
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions
...
0112 docs fix some descriptions
2023-01-16 09:56:56 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
59ac0b1424
fix(mqtt-bridge): transmit raw msg payload with empty template
2023-01-13 17:23:04 +08:00
c7ac55520d
docs: update pool_size doc
2023-01-13 07:46:23 +01:00
2a81fa1522
fix: remove redundant log message
2023-01-12 14:29:11 +01:00
04f46f5227
feat: make http connector report errors
2023-01-12 14:26:36 +01:00
77f043dedf
fix: don't require username and password
...
not enforced by schema so we shouldn't either
2023-01-12 14:26:36 +01:00
82e80e5610
fix: remove password from info logs
...
Information log messages for several connectors contained sensitive
password information. This commit fixes this by hiding the sensitive
information from the log messages.
Fixes: https://emqx.atlassian.net/browse/EMQX-8718
2023-01-12 09:28:01 +01:00
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field
...
refactor: remove the auto_reconnect field
2023-01-12 11:11:00 +08:00
3e9c4f444f
refactor: remove the auto_reconnect field
2023-01-11 21:47:06 +08:00
fde8752452
Merge pull request #9710 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master_fix
...
chore: upgrade mongodb-erlang to v3.0.19
2023-01-11 12:41:11 +01:00
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
...
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
724015af47
fix: remove the default value of webhook body field
2023-01-10 09:45:25 +08:00
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection
...
0101 authz schema union member selection
2023-01-09 22:17:51 +01:00
0bd5bec982
chore: upgrade mongodb-erlang to v3.0.19
...
This upgrades mongodb-erlang from v3.0.18 to v3.0.19. This fixes a bug
that occurred when one issued a command specified as a map*. This fix
should not affect present EMQX as this functionality is currently not
used by EMQX. However, I think it is good to do the upgrade anyway in
case we will use the fixed functionality in the future. There is no need
to update the changelog.
* https://github.com/emqx/mongodb-erlang/pull/36
2023-01-09 18:14:11 +01:00
f1ebfe40a5
Merge pull request #9697 from lafirest/fix/ldap_i18n
...
chore: add i18n file for ldap connector
2023-01-09 22:54:09 +08:00
c3635f537a
ci: wait for redis in emqx_ee_bridge_redis_SUITE
2023-01-09 14:39:56 +01:00
0697c692ed
refactor: mongo_type and redis_type are not mandatory
...
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:
[
{mongo_type = rs,
...
}
{another backend config}
]
however the brdige types are maps like, for example:
mongodb_rs.$name {
mongo_type = rs
...
}
in which case, the mongo_type is not required.
in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.
However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
2023-01-09 14:26:16 +01:00
e52f9d5920
refactor: use union member type selector for authz sources
2023-01-09 14:26:16 +01:00
ba8c304723
Merge pull request #9691 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master
...
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-09 14:15:48 +01:00
0db31f30a1
Merge pull request #9675 from keynslug/chore/otp-25-compat-deps
...
chore: bump deps versions with full OTP-25 compat
2023-01-09 14:54:14 +04:00
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip
...
test: do not use tc_user_skip for test cases
2023-01-09 09:02:06 +01:00
1336a57b3b
chore: bump to emqx/eredis_cluster 0.7.5
...
* Full OTP-25 compat.
* Password censoring in status and crash reports.
* No more crashes on query timeout.
2023-01-08 13:13:31 +03:00
d2456f9fbe
chore: add i18n file for ldap connector
2023-01-06 17:28:06 +08:00
f7b50c56da
feat(bridges): integrate PostgreSQL into bridges
...
support both simple and batch query
2023-01-06 11:47:22 +08:00
7a4e7a6de9
test: wait for redis
2023-01-05 22:47:08 +01:00
5f12cdff6c
ci: should not skip tests in github actions
2023-01-05 22:47:08 +01:00
5266722e51
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-05 16:40:45 +01:00
cf6996cbea
test(flaky): fix flaky jwt worker test
2023-01-05 11:19:43 -03:00
00b91d8a60
Merge pull request #9664 from thalesmg/fix-mongo-duration-field-descs-v50
...
docs(mongodb_connector): fix duration field descriptions
2023-01-04 10:03:38 -03:00
f767db4d8f
Merge pull request #9660 from olcai/refactor-ingress-bridge-counter
...
refactor(emqx_resource): ingress bridge counter
2023-01-04 10:43:07 +01:00
c3cfbae3c2
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections
2023-01-03 11:38:09 -03:00
7bbdc20993
docs(mongodb_connector): fix duration field descriptions
...
https://emqx.atlassian.net/browse/EMQX-8642
Adds different descriptions for the different fields of type duration
for MongoDB.
Based off: https://www.mongodb.com/docs/manual/reference/connection-string/
2023-01-03 10:01:56 -03:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
bc37556963
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
2023-01-02 21:46:24 +08:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
231d6da4dc
Merge pull request #9644 from id/fix-kafka-bridge-connectivity-detection
...
fix(kafka): detect connectivity in on_get_status
2022-12-30 21:54:56 +01:00
0ce1ca89b7
refactor: use string type for server and servers
2022-12-30 14:20:23 +01:00
f410201dc3
chore: fix flaky test case in emqx_connector_jwt_worker_SUITE
2022-12-30 11:45:13 +01:00
446a4c74d0
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
...
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652
Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.
Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
81b226a801
Merge pull request #9608 from zmstone/1123-pin-hocon-0.33.0
...
1123 pin hocon 0.33.0
2022-12-26 19:22:21 +08:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
fa7433628c
fix(logs): set password field sensitive for connectors
2022-12-23 12:46:03 +01:00
6292d7ec68
refactor: remove dead code
2022-12-23 12:07:36 +01:00
13942f5c49
refactor: rename error return in mysql connector
2022-12-22 10:29:12 +01:00
7df24000a0
test: add more EE mysql bridge test cases
2022-12-22 10:27:38 +01:00
23ac426608
fix(emqx_connector): check for key among prepared statements on query
...
An infinite loop was triggered in the mysql connector when a query
used a prepared statement key that was not among the defined prepared
statements on start. We now check that the key is defined among the
prepared statements before recursing. It seems that this bug was never
triggered in any production code flow and simply found while writing
tests.
An error return spell fix is also included as well as a FIXME comment
regarding running mysql:prepare and not distinguishing between
transient failures and syntax errors. Syntax errors should not be
retried.
2022-12-22 10:27:38 +01:00
516147ad00
Merge pull request #9581 from zmstone/1219-fix-mqtt-bridge-config
...
1219 fix mqtt bridge config
2022-12-20 14:47:02 +01:00
da51433dc3
refactor: add default value for eggress remote 'qos' and 'retain'
...
otherwise when updating from dashboard, there is no way to set
'false' for 'retain' because it's a checkbox,
it's either 'true' or 'undefined'
2022-12-20 00:00:31 +01:00
29f394aa70
fix(shared): handle unsubscribe for sticky strategy
...
prior to this change, the message is dispatched to a shared subscriber
even after unsubscribed
2022-12-19 19:12:38 +01:00
ed2d5aa48a
fix: handle status in `format_status/1` callback
...
Which is expected argument type for this callback. Also try to
make sure that random maps won't pass through this callback unnoticed.
2022-12-16 13:45:06 +03:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
42c58e2a91
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 15:29:13 +01:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
34e9056779
refactor: fix typo in variable name
...
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
40809b2ad0
Merge remote-tracking branch 'origin/dev/ee5.0' into release-50
2022-12-09 11:45:52 +01:00
6692b0c895
feat(bridge): add Redis bridge
2022-12-06 23:15:42 +03:00
1c2fc4b6c3
Merge remote-tracking branch 'origin/release-50' into 1206-chore-merge-ee50-to-release-50
2022-12-06 16:35:56 +01:00
485c4ddcab
Merge pull request #9393 from olcai/add-mysql-bridge-test
...
test: add basic mysql bridge test (ee5.0)
2022-12-02 09:43:22 +01:00
eb017ab034
test: add test case to cover mqtt bridge config upgrade
2022-12-01 16:50:10 +01:00
5461311edc
test: more EE mysql bridge tests
2022-12-01 10:27:51 +01:00
96bff1d32e
test: improve basic tests for EE mysql bridge
2022-12-01 10:27:51 +01:00
9a6901987f
feat: add mqtt bridge config upgrade converter
2022-11-30 21:01:59 +01:00
983e904858
fix(connector): fix ssl clear
2022-11-30 17:03:41 +01:00
81586a4c8d
chore: bump connector version
2022-11-30 23:14:54 +08:00
51ace31f30
fix(connector): fix previous incorrectly fixes
...
There was an error in the try_clear_certs2/3 clause that the argument isn't the ssl field but including it.
2022-11-30 22:14:44 +08:00
b398617614
chore: bump app versions
2022-11-28 21:12:43 +01:00
7ee53e5319
Merge tag 'v5.0.11' into dev/ee5.0
2022-11-28 21:02:21 +01:00
fc2793c300
fix: fix upside down function clause
2022-11-24 10:37:52 +08:00
67cef65fdb
refactor(emqx_connector_ssl): simplify implementation
2022-11-23 23:49:43 +01:00
6dc5078a96
fix: fix obsolete SSL files aren't deleted after the bridge configuration update
2022-11-23 23:49:43 +01:00
00c57de4c3
feat: do not drop MQTTv5 properties in rule/bridge
2022-11-22 20:09:59 +01:00
09455edae8
Merge tag 'v5.0.10' into dev/ee5.0
2022-11-16 16:20:30 +01:00
d0687c7aaa
fix(connector): keep connector name type as binary don't convert it to atom
...
After investigation, it was confirmed that there was no need to convert the connector name to atom
2022-11-01 15:36:31 +08:00
c157392452
docs: fix self-closing html tag, change </br> to <br/>
2022-10-27 13:57:18 +02:00
a314950be9
docs: fix zh punctuations
2022-10-26 16:19:38 +02:00
834910b6f6
chore: update app.src
2022-09-27 17:24:44 +08:00
39ad172377
chore: replace ip_port() wuth host_port() for all connectors
2022-09-27 15:19:42 +08:00
00e4b4da5a
fix(schema): support hostname.domain:port for mqtt bridge
2022-09-27 09:02:49 +02:00
dfe14be8b2
chore: bump app vsns
2022-09-26 18:23:28 +08:00
4135910b42
chore: merge master into dev/ee5.0
2022-09-26 09:52:33 +08:00
a660004fa4
fix(bridge): mqtt bridge client info `username` & `password`
...
for non configured `username` and `password`, default values `<<>>` should be
used. emqtt needs the values.
2022-09-16 15:09:51 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
4e211c12d3
fix(mqtt_bridge): return value of sending messages was discarded
2022-09-15 08:57:01 +08:00
1c03c236f5
fix(mqtt_bridge): handle send_to_remote in idle state
2022-09-14 15:19:30 +08:00
f018b8ab49
fix(bridge/authz/exhook): no need to drop invalid certs
2022-09-14 14:08:55 +08:00
b5bf5d3adb
fix: mqtt bridge payload default value
2022-09-13 18:58:01 +08:00
81ec4c4403
Merge pull request #8906 from zmstone/merge-master-to-ee50-a
...
Merge master to ee50
2022-09-07 11:19:41 +02:00
2b33ca6d49
fix: no error log print if insert bool values into mysql
2022-09-07 16:00:09 +08:00
befc4acced
Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a
2022-09-06 20:31:38 +02:00
f785da075b
ci: only start required docker for integration tests
2022-09-06 19:25:53 +02:00
23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge
...
feat: add mongodb bridge (e5.0)
2022-09-03 12:27:01 +08:00
7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources
...
Some improvements and bug fixes to emqx_resources
2022-09-03 12:16:49 +08:00
d533e35df0
chore: update app vsn
2022-09-02 22:26:21 +08:00
26234d38b9
fix: mark the async msg 'queuing' not 'sent.inflight' on recoverable_error
2022-09-02 18:41:43 +08:00
16618131ad
fix(bridge): mqtt source/sink bridge use empty payload template
2022-09-02 14:44:38 +08:00
2693fc263d
fix(bridge): show file paths for SSL certs in dashboard for bridge
2022-09-02 14:12:03 +08:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
74659bb864
fix(bridge): rm mqtt bridge default `username` and `password`
2022-09-02 10:43:32 +08:00
275171d217
feat: handle lists of servers in mongo servers config
2022-09-01 14:47:14 -03:00
3d4afd65df
feat: add mongodb bridge (e5.0)
2022-09-01 14:47:14 -03:00
8a5a9ca2bf
Merge pull request #8854 from zmstone/0831-chore-merge-main-to-dev-ee50
...
0831 chore merge main to dev ee50
2022-09-01 11:35:23 +01:00
a5ac5b6f3a
Merge branch 'dev/ee5.0' into refactor-mqtt-bridge
2022-09-01 09:30:37 +08:00
5a3fb4a011
Merge remote-tracking branch 'origin/master' into dev/ee5.0
2022-08-31 17:11:15 +02:00
0ef0b68de4
refactor: change '{recoverable_error,Reason}' to '{error,{recoverable_error,Reason}}'
2022-08-31 18:25:00 +08:00
ca52b8eb29
fix: start connector-mqtt failed when username/password not provided
2022-08-31 09:18:10 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
65dfa63324
fix: update the counters for data bridges
2022-08-30 12:28:01 +08:00
Thales Macedo Garitezi