9f6b6cedc6
fix(webhook): pick worker according to pool type
2023-02-01 09:53:25 +01:00
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover
...
fix: fix redis cluster resource recovery
2023-02-01 10:38:52 +02:00
f4381d90ca
refactor: http cliet should not need to retry HTTP requests for bridge
2023-02-01 09:30:30 +01:00
6ca816f7a4
docs: remove impl details from user doc
2023-02-01 09:30:10 +01:00
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback
...
feat(mqtt-bridge): avoid middleman process
2023-01-31 19:12:18 +01:00
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter
2023-01-31 18:30:42 +03:00
d0c10b59aa
feat(mqtt-bridge): avoid middleman process
...
Instead, supervise `emqtt` client process directly.
2023-01-31 17:59:03 +03:00
fce1e74c3d
fix(connector): fix redis cluster resource recovery
2023-01-31 16:55:05 +02:00
4d146c521b
fix(mqtt-bridge): ensure proper feedback on async forwards
...
So that buffer worker would notice a connection loss in time, and
recycle inflight messages subsequently.
2023-01-31 11:09:39 +03:00
35c429ef1d
refactor: drop a couple of unused macros / includes
2023-01-31 11:09:39 +03:00
0c4134c423
test: add unit test case for redact function in http connector
2023-01-30 11:51:28 +01:00
72f39b9b72
fix(docs): correct Redis conf field description
2023-01-27 17:39:16 +02:00
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching
...
feat(redis): disable batching in redis_cluster bridges
2023-01-27 15:27:57 +04:00
71f996b9d5
refactor(mqtt-bridge): unwrap single statem actions
...
So that the code would be easier to follow and harder to break.
Also drop a couple of unused macrodefs.
2023-01-26 21:13:18 +03:00
c95f979413
fix(mqtt_bridge): use correct gen_statem reply action
2023-01-26 21:12:05 +03:00
86cfbfb43c
fix: Authorization header leak in log entries for webhook
...
There might be another possibility for leakage. If the resource mangager
for the webhook resource crashes, OTP might log the spec for the
resource manager which contains the Config and thus the Authorization
header. This is probably an issue for other resources as well and should
be fixed in another commit. The following issue has been created for
that:
https://emqx.atlassian.net/browse/EMQX-8794
Fixes:
https://emqx.atlassian.net/browse/EMQX-8791
2023-01-26 12:44:16 +01:00
2ee00b75a7
fix(redis): unwrap pipeline queries against redis cluster
...
This is an additional safety measure in addition to the disabled
batching on the bridge level.
2023-01-25 17:28:11 +03:00
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50
...
feat(buffer_worker): add expiration time to requests
2023-01-24 13:54:04 +01:00
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
16f45a60fd
chore: i18n typo fix
2023-01-20 11:50:01 +08:00
d755b43c77
fix(jwt_worker): handle exceptions when decoding jwk from pem
...
Returns a more controlled error if users attempt to use the Service
Account JSON from the GCP PubSub example from swagger, which is
redacted.
2023-01-19 09:24:45 -03:00
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
006b4bda97
feat(buffer_worker): monitor async workers and cancel their inflight requests upon death
2023-01-17 16:48:48 -03:00
4a7e74f5d6
fix(schema): add password converter to ensure its binary() type
2023-01-17 11:12:17 +01:00
44f635300d
chore: update mysql-otp to 1.7.2
2023-01-17 13:36:26 +08:00
54cea3b3f2
chore: update epgsql to 4.7.0.1 with full OTP-25 compat
2023-01-16 15:35:51 +03:00
9a53410efb
fix: don't add empty list element to options proplist
2023-01-16 12:00:03 +01:00
edfcea7f43
fix: add log for timeout
2023-01-16 12:00:03 +01:00
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions
...
0112 docs fix some descriptions
2023-01-16 09:56:56 +01:00
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge
...
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
2023-01-16 09:57:20 +08:00
1690a6dcfc
Merge branch 'master' into dev/api-refactor
2023-01-13 15:34:13 +01:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
59ac0b1424
fix(mqtt-bridge): transmit raw msg payload with empty template
2023-01-13 17:23:04 +08:00
c7ac55520d
docs: update pool_size doc
2023-01-13 07:46:23 +01:00
2a81fa1522
fix: remove redundant log message
2023-01-12 14:29:11 +01:00
04f46f5227
feat: make http connector report errors
2023-01-12 14:26:36 +01:00
77f043dedf
fix: don't require username and password
...
not enforced by schema so we shouldn't either
2023-01-12 14:26:36 +01:00
82e80e5610
fix: remove password from info logs
...
Information log messages for several connectors contained sensitive
password information. This commit fixes this by hiding the sensitive
information from the log messages.
Fixes: https://emqx.atlassian.net/browse/EMQX-8718
2023-01-12 09:28:01 +01:00
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field
...
refactor: remove the auto_reconnect field
2023-01-12 11:11:00 +08:00
3e9c4f444f
refactor: remove the auto_reconnect field
2023-01-11 21:47:06 +08:00
fde8752452
Merge pull request #9710 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master_fix
...
chore: upgrade mongodb-erlang to v3.0.19
2023-01-11 12:41:11 +01:00
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
...
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
724015af47
fix: remove the default value of webhook body field
2023-01-10 09:45:25 +08:00
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection
...
0101 authz schema union member selection
2023-01-09 22:17:51 +01:00
0bd5bec982
chore: upgrade mongodb-erlang to v3.0.19
...
This upgrades mongodb-erlang from v3.0.18 to v3.0.19. This fixes a bug
that occurred when one issued a command specified as a map*. This fix
should not affect present EMQX as this functionality is currently not
used by EMQX. However, I think it is good to do the upgrade anyway in
case we will use the fixed functionality in the future. There is no need
to update the changelog.
* https://github.com/emqx/mongodb-erlang/pull/36
2023-01-09 18:14:11 +01:00
f1ebfe40a5
Merge pull request #9697 from lafirest/fix/ldap_i18n
...
chore: add i18n file for ldap connector
2023-01-09 22:54:09 +08:00
c3635f537a
ci: wait for redis in emqx_ee_bridge_redis_SUITE
2023-01-09 14:39:56 +01:00
0697c692ed
refactor: mongo_type and redis_type are not mandatory
...
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:
[
{mongo_type = rs,
...
}
{another backend config}
]
however the brdige types are maps like, for example:
mongodb_rs.$name {
mongo_type = rs
...
}
in which case, the mongo_type is not required.
in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.
However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
2023-01-09 14:26:16 +01:00
e52f9d5920
refactor: use union member type selector for authz sources
2023-01-09 14:26:16 +01:00
ba8c304723
Merge pull request #9691 from kjellwinblad/kjell/mongodb_upgrade_in_emqx_5_master
...
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-09 14:15:48 +01:00
0db31f30a1
Merge pull request #9675 from keynslug/chore/otp-25-compat-deps
...
chore: bump deps versions with full OTP-25 compat
2023-01-09 14:54:14 +04:00
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip
...
test: do not use tc_user_skip for test cases
2023-01-09 09:02:06 +01:00
1336a57b3b
chore: bump to emqx/eredis_cluster 0.7.5
...
* Full OTP-25 compat.
* Password censoring in status and crash reports.
* No more crashes on query timeout.
2023-01-08 13:13:31 +03:00
d2456f9fbe
chore: add i18n file for ldap connector
2023-01-06 17:28:06 +08:00
f7b50c56da
feat(bridges): integrate PostgreSQL into bridges
...
support both simple and batch query
2023-01-06 11:47:22 +08:00
7a4e7a6de9
test: wait for redis
2023-01-05 22:47:08 +01:00
5f12cdff6c
ci: should not skip tests in github actions
2023-01-05 22:47:08 +01:00
5266722e51
feat: upgrade MongoDB library to support MongoDB 5.1+
2023-01-05 16:40:45 +01:00
cf6996cbea
test(flaky): fix flaky jwt worker test
2023-01-05 11:19:43 -03:00
00b91d8a60
Merge pull request #9664 from thalesmg/fix-mongo-duration-field-descs-v50
...
docs(mongodb_connector): fix duration field descriptions
2023-01-04 10:03:38 -03:00
f767db4d8f
Merge pull request #9660 from olcai/refactor-ingress-bridge-counter
...
refactor(emqx_resource): ingress bridge counter
2023-01-04 10:43:07 +01:00
c3cfbae3c2
docs: improve descriptions
...
Thanks to @qzhuyan for the corrections
2023-01-03 11:38:09 -03:00
7bbdc20993
docs(mongodb_connector): fix duration field descriptions
...
https://emqx.atlassian.net/browse/EMQX-8642
Adds different descriptions for the different fields of type duration
for MongoDB.
Based off: https://www.mongodb.com/docs/manual/reference/connection-string/
2023-01-03 10:01:56 -03:00
b9d012e072
refactor(emqx_resource): ingress bridge counter
...
Unify code paths for resource metrics by removing
emqx_resource:inc_received/1 and adding
emqx_resource_metrics:received_inc/1 & friends.
2023-01-02 15:11:52 +01:00
bc37556963
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
2023-01-02 21:46:24 +08:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
231d6da4dc
Merge pull request #9644 from id/fix-kafka-bridge-connectivity-detection
...
fix(kafka): detect connectivity in on_get_status
2022-12-30 21:54:56 +01:00
0ce1ca89b7
refactor: use string type for server and servers
2022-12-30 14:20:23 +01:00
f410201dc3
chore: fix flaky test case in emqx_connector_jwt_worker_SUITE
2022-12-30 11:45:13 +01:00
446a4c74d0
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
...
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652
Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.
Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
81b226a801
Merge pull request #9608 from zmstone/1123-pin-hocon-0.33.0
...
1123 pin hocon 0.33.0
2022-12-26 19:22:21 +08:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
fa7433628c
fix(logs): set password field sensitive for connectors
2022-12-23 12:46:03 +01:00
6292d7ec68
refactor: remove dead code
2022-12-23 12:07:36 +01:00
13942f5c49
refactor: rename error return in mysql connector
2022-12-22 10:29:12 +01:00
7df24000a0
test: add more EE mysql bridge test cases
2022-12-22 10:27:38 +01:00
23ac426608
fix(emqx_connector): check for key among prepared statements on query
...
An infinite loop was triggered in the mysql connector when a query
used a prepared statement key that was not among the defined prepared
statements on start. We now check that the key is defined among the
prepared statements before recursing. It seems that this bug was never
triggered in any production code flow and simply found while writing
tests.
An error return spell fix is also included as well as a FIXME comment
regarding running mysql:prepare and not distinguishing between
transient failures and syntax errors. Syntax errors should not be
retried.
2022-12-22 10:27:38 +01:00
516147ad00
Merge pull request #9581 from zmstone/1219-fix-mqtt-bridge-config
...
1219 fix mqtt bridge config
2022-12-20 14:47:02 +01:00
da51433dc3
refactor: add default value for eggress remote 'qos' and 'retain'
...
otherwise when updating from dashboard, there is no way to set
'false' for 'retain' because it's a checkbox,
it's either 'true' or 'undefined'
2022-12-20 00:00:31 +01:00
29f394aa70
fix(shared): handle unsubscribe for sticky strategy
...
prior to this change, the message is dispatched to a shared subscriber
even after unsubscribed
2022-12-19 19:12:38 +01:00
ed2d5aa48a
fix: handle status in `format_status/1` callback
...
Which is expected argument type for this callback. Also try to
make sure that random maps won't pass through this callback unnoticed.
2022-12-16 13:45:06 +03:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
42c58e2a91
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 15:29:13 +01:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
34e9056779
refactor: fix typo in variable name
...
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
40809b2ad0
Merge remote-tracking branch 'origin/dev/ee5.0' into release-50
2022-12-09 11:45:52 +01:00
6692b0c895
feat(bridge): add Redis bridge
2022-12-06 23:15:42 +03:00
1c2fc4b6c3
Merge remote-tracking branch 'origin/release-50' into 1206-chore-merge-ee50-to-release-50
2022-12-06 16:35:56 +01:00
485c4ddcab
Merge pull request #9393 from olcai/add-mysql-bridge-test
...
test: add basic mysql bridge test (ee5.0)
2022-12-02 09:43:22 +01:00
eb017ab034
test: add test case to cover mqtt bridge config upgrade
2022-12-01 16:50:10 +01:00
5461311edc
test: more EE mysql bridge tests
2022-12-01 10:27:51 +01:00
96bff1d32e
test: improve basic tests for EE mysql bridge
2022-12-01 10:27:51 +01:00
9a6901987f
feat: add mqtt bridge config upgrade converter
2022-11-30 21:01:59 +01:00
983e904858
fix(connector): fix ssl clear
2022-11-30 17:03:41 +01:00
81586a4c8d
chore: bump connector version
2022-11-30 23:14:54 +08:00
51ace31f30
fix(connector): fix previous incorrectly fixes
...
There was an error in the try_clear_certs2/3 clause that the argument isn't the ssl field but including it.
2022-11-30 22:14:44 +08:00
b398617614
chore: bump app versions
2022-11-28 21:12:43 +01:00
7ee53e5319
Merge tag 'v5.0.11' into dev/ee5.0
2022-11-28 21:02:21 +01:00
fc2793c300
fix: fix upside down function clause
2022-11-24 10:37:52 +08:00
67cef65fdb
refactor(emqx_connector_ssl): simplify implementation
2022-11-23 23:49:43 +01:00
6dc5078a96
fix: fix obsolete SSL files aren't deleted after the bridge configuration update
2022-11-23 23:49:43 +01:00
00c57de4c3
feat: do not drop MQTTv5 properties in rule/bridge
2022-11-22 20:09:59 +01:00
09455edae8
Merge tag 'v5.0.10' into dev/ee5.0
2022-11-16 16:20:30 +01:00
d0687c7aaa
fix(connector): keep connector name type as binary don't convert it to atom
...
After investigation, it was confirmed that there was no need to convert the connector name to atom
2022-11-01 15:36:31 +08:00
c157392452
docs: fix self-closing html tag, change </br> to <br/>
2022-10-27 13:57:18 +02:00
a314950be9
docs: fix zh punctuations
2022-10-26 16:19:38 +02:00
834910b6f6
chore: update app.src
2022-09-27 17:24:44 +08:00
39ad172377
chore: replace ip_port() wuth host_port() for all connectors
2022-09-27 15:19:42 +08:00
00e4b4da5a
fix(schema): support hostname.domain:port for mqtt bridge
2022-09-27 09:02:49 +02:00
dfe14be8b2
chore: bump app vsns
2022-09-26 18:23:28 +08:00
4135910b42
chore: merge master into dev/ee5.0
2022-09-26 09:52:33 +08:00
a660004fa4
fix(bridge): mqtt bridge client info `username` & `password`
...
for non configured `username` and `password`, default values `<<>>` should be
used. emqtt needs the values.
2022-09-16 15:09:51 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
4e211c12d3
fix(mqtt_bridge): return value of sending messages was discarded
2022-09-15 08:57:01 +08:00
1c03c236f5
fix(mqtt_bridge): handle send_to_remote in idle state
2022-09-14 15:19:30 +08:00
f018b8ab49
fix(bridge/authz/exhook): no need to drop invalid certs
2022-09-14 14:08:55 +08:00
b5bf5d3adb
fix: mqtt bridge payload default value
2022-09-13 18:58:01 +08:00
81ec4c4403
Merge pull request #8906 from zmstone/merge-master-to-ee50-a
...
Merge master to ee50
2022-09-07 11:19:41 +02:00
2b33ca6d49
fix: no error log print if insert bool values into mysql
2022-09-07 16:00:09 +08:00
befc4acced
Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a
2022-09-06 20:31:38 +02:00
f785da075b
ci: only start required docker for integration tests
2022-09-06 19:25:53 +02:00
23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge
...
feat: add mongodb bridge (e5.0)
2022-09-03 12:27:01 +08:00
7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources
...
Some improvements and bug fixes to emqx_resources
2022-09-03 12:16:49 +08:00
d533e35df0
chore: update app vsn
2022-09-02 22:26:21 +08:00
26234d38b9
fix: mark the async msg 'queuing' not 'sent.inflight' on recoverable_error
2022-09-02 18:41:43 +08:00
16618131ad
fix(bridge): mqtt source/sink bridge use empty payload template
2022-09-02 14:44:38 +08:00
2693fc263d
fix(bridge): show file paths for SSL certs in dashboard for bridge
2022-09-02 14:12:03 +08:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
74659bb864
fix(bridge): rm mqtt bridge default `username` and `password`
2022-09-02 10:43:32 +08:00
275171d217
feat: handle lists of servers in mongo servers config
2022-09-01 14:47:14 -03:00
3d4afd65df
feat: add mongodb bridge (e5.0)
2022-09-01 14:47:14 -03:00
8a5a9ca2bf
Merge pull request #8854 from zmstone/0831-chore-merge-main-to-dev-ee50
...
0831 chore merge main to dev ee50
2022-09-01 11:35:23 +01:00
a5ac5b6f3a
Merge branch 'dev/ee5.0' into refactor-mqtt-bridge
2022-09-01 09:30:37 +08:00
5a3fb4a011
Merge remote-tracking branch 'origin/master' into dev/ee5.0
2022-08-31 17:11:15 +02:00
0ef0b68de4
refactor: change '{recoverable_error,Reason}' to '{error,{recoverable_error,Reason}}'
2022-08-31 18:25:00 +08:00
ca52b8eb29
fix: start connector-mqtt failed when username/password not provided
2022-08-31 09:18:10 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
65dfa63324
fix: update the counters for data bridges
2022-08-30 12:28:01 +08:00
262e68f7d2
fix: return error when receive HTTP code other than 2xx
2022-08-30 10:16:02 +08:00
b5ad5233a1
fix(mqtt-bridge): username and password defaults to undefined
2022-08-30 10:14:56 +08:00
a6eff81163
chore: update emqtt to 1.7.0-rc.1
2022-08-30 09:50:36 +08:00
0aa10702db
feat(bridge): support async mode resource options
2022-08-30 09:34:56 +08:00
ca7ad9cc15
chore: refactor mqtt connector
2022-08-30 09:34:56 +08:00
2eb078ff3b
chore: bump changed app versions
2022-08-26 14:32:28 +02:00
1625b8eaeb
fix(mysql_bridge): export the query_mode option to the APIs
2022-08-26 17:11:24 +08:00
4ac6f12252
refactor: configs and APIs for mysql bridge
2022-08-24 10:08:03 +08:00
45352206a3
fix(auth): remove emqx_connector from testcases of authz/authn
2022-08-22 20:20:45 +08:00
55c18c0b5f
fix(bridges): update the test cases for new config structure
2022-08-22 18:24:59 +08:00
Zaiming (Stone) Shi