Commit Graph

73 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
24f476e35f test: add README to influxdb test script 2023-02-28 19:38:43 +01:00
Erik Timan
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
Erik Timan
843ed464d5 fix: remove all mentions of UDP support for InfluxDB EE bridge
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
firest
81adb02402 chore: bump apps version 2023-02-14 22:04:29 +08:00
firest
530b21268a feat(bridges): add TDengine 2023-02-14 22:04:23 +08:00
Erik Timan
4eb4430fe2 chore: bump VSN 2023-02-09 17:22:19 +01:00
Erik Timan
cfd0e9ebdd refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
2023-02-09 17:18:47 +01:00
Erik Timan
cf77dcf25e test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
2023-02-09 17:18:47 +01:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
30a8a436b4 fix(influxdb_bridge): no need to create atom for pool name 2023-01-27 12:39:05 +01:00
Erik Timan
1f235ffee9 refactor(emqx_ee_connector): redo readact key function 2023-01-26 10:30:13 +01:00
Erik Timan
805d08e823 fix: reduce log level from error to warning in several places
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
2023-01-25 14:49:50 +01:00
Erik Timan
8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi
feb0238911 fix(influxdb): ensure client is stopped before returning error 2023-01-24 20:35:02 +01:00
Erik Timan
7603ab490b fix(emqx_ee_connector): fix passing of influxdb ssl opts 2023-01-24 14:30:40 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
JimMoen
40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Thales Macedo Garitezi
b9f258b737 feat(mongodb): add payload_template field for bridge (e5.0)
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
2023-01-09 13:38:11 -03:00
Zaiming (Stone) Shi
dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi
0ce1ca89b7 refactor: use string type for server and servers 2022-12-30 14:20:23 +01:00
Thales Macedo Garitezi
446a4c74d0 fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652

Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.

Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
Thales Macedo Garitezi
c0b208dd9e fix(influxdb): check if fields are empty before sending
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461

Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.

```
  14:03:35.563 [error] [InfluxDB] Encode [
    %{
      fields: %{},
      measurement: "t/topic",
      tags: %{},
      timestamp: 1670864615563
    }
  ] failed: :error :missing_field [
    {:influxdb_line, :encode_fields, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 60
     ]},
    {:influxdb_line, :encode_, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 44
     ]},
    {:influxdb_line, :"-encode_/1-fun-0-", 2,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 38
     ]},
    {:influxdb, :write, 2,
     [file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
    {:emqx_ee_connector_influxdb, :do_query, 3,
     [
       file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
       line: 322
     ]},
    {:emqx_resource_worker, :apply_query_fun, 6,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 454
     ]},
    {:emqx_resource_worker, :query_or_acc, 3,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 306
     ]},
    {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]
  2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

  14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
  2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
  []

  14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
  iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```

Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.

Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi
9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi
42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi
2932aceaf9 refactor: rename and move emqx_ee_bridge_gcp_pubsub_resource to emqx_ee_connector_gcp_pubsub 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi
34e9056779 refactor: fix typo in variable name
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
Ilya Averyanov
6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi
fa90638de2 chore: add description to apps 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi
83fe35b148 Merge remote-tracking branch 'origin/dev/ee5.0' into 1116-sync-v5.0.10-to-ee50 2022-11-18 19:23:31 +01:00
Zaiming (Stone) Shi
c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Thales Macedo Garitezi
04588148b7 test(influxdb): increase influxdb bridge/connector coverage (ee5.0) 2022-11-07 15:15:49 -03:00
Shawn
dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Zaiming (Stone) Shi
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
JimMoen
03495d8d36 fix: influxdb server string support scheme prefix 2022-09-20 14:46:59 +08:00
Zaiming (Stone) Shi
0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Shawn
e0a6a61d73 fix: return error on start hstreamdb crash 2022-08-30 10:19:40 +08:00
Shawn
6b0ccfbc43 refactor: rename the error return resource_down -> recoverable_error 2022-08-26 17:11:12 +08:00
JimMoen
ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
Shawn
aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
JimMoen
2648362c62 fix(bridge): password for bridge/db format as password for dashboard 2022-08-19 15:54:19 +08:00
JimMoen
06363e63d9 fix(influxdb): connector use a fallbacke pool_size for influxdb client 2022-08-19 15:54:19 +08:00
JimMoen
d0e923590e fix: write influxdb line with undefined value 2022-08-15 14:02:17 +08:00