Commit Graph

44 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
c0b208dd9e fix(influxdb): check if fields are empty before sending
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461

Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.

```
  14:03:35.563 [error] [InfluxDB] Encode [
    %{
      fields: %{},
      measurement: "t/topic",
      tags: %{},
      timestamp: 1670864615563
    }
  ] failed: :error :missing_field [
    {:influxdb_line, :encode_fields, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 60
     ]},
    {:influxdb_line, :encode_, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 44
     ]},
    {:influxdb_line, :"-encode_/1-fun-0-", 2,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 38
     ]},
    {:influxdb, :write, 2,
     [file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
    {:emqx_ee_connector_influxdb, :do_query, 3,
     [
       file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
       line: 322
     ]},
    {:emqx_resource_worker, :apply_query_fun, 6,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 454
     ]},
    {:emqx_resource_worker, :query_or_acc, 3,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 306
     ]},
    {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]
  2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

  14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
  2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
  []

  14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
  iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```

Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.

Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi
9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi
42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi
2932aceaf9 refactor: rename and move emqx_ee_bridge_gcp_pubsub_resource to emqx_ee_connector_gcp_pubsub 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi
34e9056779 refactor: fix typo in variable name
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
Ilya Averyanov
6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi
fa90638de2 chore: add description to apps 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi
83fe35b148 Merge remote-tracking branch 'origin/dev/ee5.0' into 1116-sync-v5.0.10-to-ee50 2022-11-18 19:23:31 +01:00
Zaiming (Stone) Shi
c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Thales Macedo Garitezi
04588148b7 test(influxdb): increase influxdb bridge/connector coverage (ee5.0) 2022-11-07 15:15:49 -03:00
Shawn
dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Zaiming (Stone) Shi
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
JimMoen
03495d8d36 fix: influxdb server string support scheme prefix 2022-09-20 14:46:59 +08:00
Zaiming (Stone) Shi
0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Shawn
e0a6a61d73 fix: return error on start hstreamdb crash 2022-08-30 10:19:40 +08:00
Shawn
6b0ccfbc43 refactor: rename the error return resource_down -> recoverable_error 2022-08-26 17:11:12 +08:00
JimMoen
ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
Shawn
aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
JimMoen
2648362c62 fix(bridge): password for bridge/db format as password for dashboard 2022-08-19 15:54:19 +08:00
JimMoen
06363e63d9 fix(influxdb): connector use a fallbacke pool_size for influxdb client 2022-08-19 15:54:19 +08:00
JimMoen
d0e923590e fix: write influxdb line with undefined value 2022-08-15 14:02:17 +08:00
JimMoen
68946f1f6c feat: influxdb support async/batch_async query 2022-08-15 14:02:17 +08:00
JimMoen
594d071c05 feat(influxdb): add async callback 2022-08-12 18:26:47 +08:00
JimMoen
db3e4f0b90 fix: flatten Points list 2022-08-11 20:58:56 +08:00
JimMoen
0f6c371760 feat(influxdb): influxdb connector add on_batch_query/3 callback 2022-08-11 18:12:41 +08:00
Shawn
145ff66a9a fix: issues found by dialyzer and elvis 2022-08-10 00:45:26 +08:00
JimMoen
d51ca2672a fix: spellcheck and update checker repo version 2022-08-09 11:50:05 +08:00
JimMoen
7305c3c04c fix(influxdb): parsed boolean type match 2022-08-08 18:22:46 +08:00
JimMoen
35e347aec8 feat: refine influxdb bridge conf
Consistent influxdb line protocol config to raw syntax format.
See also
[InfluxDB 2.3 Line Protocol](https://docs.influxdata.com/influxdb/v2.3/reference/syntax/line-protocol/) and
[InfluxDB 1.8 Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/)
2022-08-08 18:06:49 +08:00
JimMoen
06f246a896 fix: prevent unexcepted square brackets in influxdb line protocol 2022-08-08 10:45:18 +08:00
JimMoen
33a604dcaa fix: refine influxdb connector api type name 2022-08-07 19:37:44 +08:00
JimMoen
2e336fbc01 fix: bridge and connector namespace contains DB name
Contains Detailed DB name for identifier hocon schema refs.
2022-08-07 19:37:44 +08:00
JimMoen
4d0516c6e9 chore: use HStreamDB for module name 2022-08-07 19:37:44 +08:00
DDDHuang
c1542e7a77 fix: influxdb connector bad encode message 2022-08-01 10:28:15 +08:00
DDDHuang
55b9684541 fix: influxdb bridge api available 2022-08-01 10:28:07 +08:00
DDDHuang
fa54bf5612 fix: influxdb connector api available 2022-08-01 10:27:58 +08:00
DDDHuang
4c7ca2217c fix: add influxdb udp api_v1 api_v2 connector 2022-08-01 10:27:42 +08:00
DDDHuang
01e9e8a0eb feat: add ee influxdb connector & bridge 2022-08-01 10:27:35 +08:00
DDDHuang
a4992ef1b5 fix: hstreamdb connector conf & api 2022-07-28 16:47:30 +08:00
DDDHuang
be6de4aad0 fix: ee bridge & connector remove app mod 2022-07-27 11:38:45 +08:00
DDDHuang
fa4bc921ac fix: hstream db connector & bridge, TODO: SUITE 2022-07-27 11:38:45 +08:00
DDDHuang
98b36c4681 fix: hstream db connector , TODO: start apps 2022-07-27 11:38:45 +08:00
DDDHuang
f00a7417bb feat: hstream bridge & connector 2022-07-27 11:38:45 +08:00
DDDHuang
e4eba157c3 feat: add emqx enterprise app connector & bridge 2022-07-25 09:36:45 +08:00