emqx/apps/emqx_connector
Kjell Winblad 1cf01197bb fix: increase heartbeat time to avoid extreme MongoDB logging
Our MongoDB driver creates a new temporary connection, for every active
connection, to just do a single heartbeat test. There is configurable
delay between every heartbeat test. When the user has an EMQX cluster
with a MongoDB bridge (to a MongoDB replica set), there will be a lot of
connections. Furthermore, as MongoDB creates a log entry every time a
new connection is created, the log will be flooded with info about new
connection. One user have reported more than 1MB of log data in a 10
minute period.

This commit tries to fix this by increasing the default delay between
heartbeats. A better fix would be to change the MongoDB driver so that
it does not create a new connection just to do a heartbeat check, but
this is more complicated so we leave this to the future. We might also
swap out the current MongoDB driver to something better.

Fixes:
https://github.com/emqx/emqx/issues/9851
2023-03-13 14:58:53 +01:00
..
etc feat: generate a minimized emqx.conf 2022-05-31 19:20:27 +08:00
i18n fix: increase heartbeat time to avoid extreme MongoDB logging 2023-03-13 14:58:53 +01:00
include feat: add clickhouse database bridge 2023-03-02 12:22:11 +01:00
src fix: increase heartbeat time to avoid extreme MongoDB logging 2023-03-13 14:58:53 +01:00
test Merge pull request #9842 from savonarola/fix-redis-cluster-recover 2023-02-01 10:38:52 +02:00
.gitignore Revert "chore: mv emqx_connector to emqx_data_bridge" 2021-06-17 19:11:10 +08:00
docker-ct fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00
README.md refactor: remove the auto_reconnect field 2023-01-11 21:47:06 +08:00
rebar.config fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00

emqx_connector

This application is a collection of connectors.

A connector is a callback module of emqx_resource that maintains the data related to external resources. Put all resource related callback modules in a single application is good as we can put some util functions/modules here for reusing purpose.

For example, a MySQL connector is an emqx resource that maintains all the MySQL connection related parameters (configs) and the TCP connections to the MySQL server.

An MySQL connector can be used as following:

(emqx@127.0.0.1)5> emqx_resource:list_instances_verbose().
[#{config =>
       #{cacertfile => [],certfile => [],
         database => "mqtt",keyfile => [],password => "public",
         pool_size => 1,
         server => {{127,0,0,1},3306},
         ssl => false,user => "root",verify => false},
   id => <<"mysql-abc">>,mod => emqx_connector_mysql,
   state => #{poolname => 'mysql-abc'},
   status => connected}]
(emqx@127.0.0.1)6> emqx_resource:query(<<"mysql-abc">>, {sql, <<"SELECT count(1)">>}).
{ok,[<<"count(1)">>],[[1]]}