yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
16,280 Commits 393 Branches 684 Tags 74 MiB
Commit Graph

206 Commits

Author SHA1 Message Date
JianBo He 8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
firest 4ad3579966 test(bridges): add test suite for RocketMQ 2023-03-22 10:36:58 +08:00
firest 9853d00cad feat(bridges): integrate RocketMQ into data bridges 2023-03-22 10:36:52 +08:00
JianBo He 539ec2f774 chore(bridge): cover username/password auth for cassandra bridges 2023-03-21 13:55:53 +08:00
JianBo He c2c9abd568 chore: add running ct 2023-03-20 09:47:58 +08:00
JianBo He d8e6e2a1be test: refine CASSANDRA_HOST name 2023-03-17 18:52:52 +08:00
JianBo He 678cc937c0 test(bridge): cover ssl testing for cassandra bridge 2023-03-17 18:25:05 +08:00
JianBo He a20c39c83a test: correct CASSANDRA_HOST 2023-03-17 13:55:26 +08:00
JianBo He 75c1592e7c chore: make static_checks happy 2023-03-17 11:34:52 +08:00
JianBo He c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Andrew Mayorov a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
Serge Tupchii 97e71c54d4 fix: use default template if timestamp is empty (undefined) in InfluxDB bridge 
Closes EMQX-8926
 2023-03-08 11:58:23 +01:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Thales Macedo Garitezi 9b087a21f5 fix(gcp_pubsub): remove conflicting `request_timeout` option 
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
 2023-03-06 10:12:38 -03:00
Ivan Dyachkov c01f62a1c1 chore: bump apps versions 2023-03-03 13:58:32 +01:00
Kjell Winblad 67acdf0888 feat: add clickhouse database bridge 
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
 2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi 083330ad80 Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master 2023-03-01 08:53:03 +01:00
Zaiming (Stone) Shi 24f476e35f test: add README to influxdb test script 2023-02-28 19:38:43 +01:00
Erik Timan 44c0aae16b fix(emqx_ee_connector): improve log message on influxdb start failures 2023-02-24 11:31:57 +01:00
Erik Timan da42c91fb2 test(emqx_ee_bridge): check influxdb:is_alive/2 return 2023-02-24 09:03:34 +01:00
Erik Timan 8314c07147 chore: update influxdb-client-erl dep 2023-02-24 09:03:34 +01:00
Erik Timan 603532dacd fix(emqx_ee_connector): log reason for failure when starting influxdb connector 2023-02-24 09:03:34 +01:00
Erik Timan 291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts 
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
 2023-02-20 10:05:47 +01:00
Erik Timan ca28be4387 test(emqx_ee_connector): review fixes of influxdb tests 2023-02-17 15:28:49 +01:00
Erik Timan 843ed464d5 fix: remove all mentions of UDP support for InfluxDB EE bridge 
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
 2023-02-17 10:47:57 +01:00
firest 81adb02402 chore: bump apps version 2023-02-14 22:04:29 +08:00
firest 530b21268a feat(bridges): add TDengine 2023-02-14 22:04:23 +08:00
Erik Timan 4eb4430fe2 chore: bump VSN 2023-02-09 17:22:19 +01:00
Erik Timan bc6c653c9f test(emqx_ee_connector): rework and improve influxdb tests 2023-02-09 17:18:47 +01:00
Erik Timan cfd0e9ebdd refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts 
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
 2023-02-09 17:18:47 +01:00
Erik Timan cf77dcf25e test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts 
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
 2023-02-09 17:18:47 +01:00
Zaiming (Stone) Shi 157c919ba1 ci: add i18n style check script 2023-02-09 11:41:52 +01:00
Zaiming (Stone) Shi 7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi 30a8a436b4 fix(influxdb_bridge): no need to create atom for pool name 2023-01-27 12:39:05 +01:00
Erik Timan 1f235ffee9 refactor(emqx_ee_connector): redo readact key function 2023-01-26 10:30:13 +01:00
Erik Timan 805d08e823 fix: reduce log level from error to warning in several places 
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
 2023-01-25 14:49:50 +01:00
Erik Timan 8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi feb0238911 fix(influxdb): ensure client is stopped before returning error 2023-01-24 20:35:02 +01:00
Erik Timan 7603ab490b fix(emqx_ee_connector): fix passing of influxdb ssl opts 2023-01-24 14:30:40 +01:00
Erik Timan 28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi 6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi 7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter 
fix(schema): add password converter to ensure its binary() type
 2023-01-18 15:09:05 +01:00
Thales Macedo Garitezi 087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi 006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi 4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
JimMoen 40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zhongwen Deng f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Shawn 1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Thales Macedo Garitezi b9f258b737 feat(mongodb): add `payload_template` field for bridge (e5.0) 
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
 2023-01-09 13:38:11 -03:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 0ce1ca89b7 refactor: use string type for server and servers 2022-12-30 14:20:23 +01:00
Thales Macedo Garitezi a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50 
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
 2022-12-30 09:33:41 -03:00
Thales Macedo Garitezi 446a4c74d0 fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0) 
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652

Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.

Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
 2022-12-29 16:30:36 -03:00
Zaiming (Stone) Shi 5ae4bd6722 chore: bump influxdb client lib version from 1.1.4 to 1.1.5 
fixed a compile warning
 2022-12-29 10:29:36 +01:00
Thales Macedo Garitezi c0b208dd9e fix(influxdb): check if fields are empty before sending 
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461

Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.

```
  14:03:35.563 [error] [InfluxDB] Encode [
    %{
      fields: %{},
      measurement: "t/topic",
      tags: %{},
      timestamp: 1670864615563
    }
  ] failed: :error :missing_field [
    {:influxdb_line, :encode_fields, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 60
     ]},
    {:influxdb_line, :encode_, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 44
     ]},
    {:influxdb_line, :"-encode_/1-fun-0-", 2,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 38
     ]},
    {:influxdb, :write, 2,
     [file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
    {:emqx_ee_connector_influxdb, :do_query, 3,
     [
       file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
       line: 322
     ]},
    {:emqx_resource_worker, :apply_query_fun, 6,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 454
     ]},
    {:emqx_resource_worker, :query_or_acc, 3,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 306
     ]},
    {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]
  2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

  14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
  2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
  []

  14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
  iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```

Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.

Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
 2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi 9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi 42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi 2932aceaf9 refactor: rename and move `emqx_ee_bridge_gcp_pubsub_resource` to `emqx_ee_connector_gcp_pubsub` 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi 34e9056779 refactor: fix typo in variable name 
Might confuse people to think it's related to `replayq`.
 2022-12-12 17:17:51 -03:00
Ilya Averyanov 6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi fa90638de2 chore: add description to apps 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi 83fe35b148 Merge remote-tracking branch 'origin/dev/ee5.0' into 1116-sync-v5.0.10-to-ee50 2022-11-18 19:23:31 +01:00
Zaiming (Stone) Shi c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Thales Macedo Garitezi 04588148b7 test(influxdb): increase influxdb bridge/connector coverage (ee5.0) 2022-11-07 15:15:49 -03:00
Shawn dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Zaiming (Stone) Shi f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector 
feat: Add Kafka connector
 2022-09-24 22:57:50 +02:00
JimMoen 03495d8d36 fix: influxdb server string support scheme prefix 2022-09-20 14:46:59 +08:00
Zaiming (Stone) Shi 0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Thales Macedo Garitezi 3d4afd65df feat: add mongodb bridge (e5.0) 2022-09-01 14:47:14 -03:00
Shawn e0a6a61d73 fix: return error on start hstreamdb crash 2022-08-30 10:19:40 +08:00
Shawn 6b0ccfbc43 refactor: rename the error return resource_down -> recoverable_error 2022-08-26 17:11:12 +08:00
JimMoen ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
Shawn aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
JimMoen 2648362c62 fix(bridge): password for bridge/db format as `password` for dashboard 2022-08-19 15:54:19 +08:00
JimMoen 06363e63d9 fix(influxdb): connector use a fallbacke `pool_size` for influxdb client 2022-08-19 15:54:19 +08:00
JimMoen d0e923590e fix: write influxdb line with undefined value 2022-08-15 14:02:17 +08:00
JimMoen 68946f1f6c feat: influxdb support `async`/`batch_async` query 2022-08-15 14:02:17 +08:00
JimMoen b01ae8ece6 chore: refine influxdb bridge/connector i18n 2022-08-15 14:00:14 +08:00
JimMoen 594d071c05 feat(influxdb): add async callback 2022-08-12 18:26:47 +08:00
JimMoen 441d8c9d57 chore: bump `influxdb-client-erl` vsn 2022-08-12 18:26:42 +08:00
JimMoen db3e4f0b90 fix: flatten Points list 2022-08-11 20:58:56 +08:00
JimMoen 0f6c371760 feat(influxdb): influxdb connector add `on_batch_query/3` callback 2022-08-11 18:12:41 +08:00
Shawn 145ff66a9a fix: issues found by dialyzer and elvis 2022-08-10 00:45:26 +08:00
JimMoen d51ca2672a fix: spellcheck and update checker repo version 2022-08-09 11:50:05 +08:00
JimMoen 7305c3c04c fix(influxdb): parsed boolean type match 2022-08-08 18:22:46 +08:00
JimMoen 35e347aec8 feat: refine influxdb bridge conf 
Consistent influxdb line protocol config to raw syntax format.
See also
[InfluxDB 2.3 Line Protocol](https://docs.influxdata.com/influxdb/v2.3/reference/syntax/line-protocol/) and
[InfluxDB 1.8 Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/)
 2022-08-08 18:06:49 +08:00
JimMoen 06f246a896 fix: prevent unexcepted square brackets in influxdb line protocol 2022-08-08 10:45:18 +08:00
JimMoen 33a604dcaa fix: refine influxdb connector api type name 2022-08-07 19:37:44 +08:00
JimMoen 2e336fbc01 fix: bridge and connector namespace contains DB name 
Contains Detailed DB name for identifier hocon schema refs.
 2022-08-07 19:37:44 +08:00
JimMoen 4d0516c6e9 chore: use HStreamDB for module name 2022-08-07 19:37:44 +08:00
DDDHuang 4663e354bf fix: update influxdb sdk version to 1.1.3. Adapted master deps version 2022-08-01 10:34:57 +08:00
DDDHuang c1542e7a77 fix: influxdb connector bad encode message 2022-08-01 10:28:15 +08:00
DDDHuang 55b9684541 fix: influxdb bridge api available 2022-08-01 10:28:07 +08:00
DDDHuang fa54bf5612 fix: influxdb connector api available 2022-08-01 10:27:58 +08:00
DDDHuang 4c7ca2217c fix: add influxdb udp api_v1 api_v2 connector 2022-08-01 10:27:42 +08:00
DDDHuang 01e9e8a0eb feat: add ee influxdb connector & bridge 2022-08-01 10:27:35 +08:00
DDDHuang b7c245c5b0 fix: hstreamdb zh docs 2022-07-28 16:50:34 +08:00
DDDHuang a4992ef1b5 fix: hstreamdb connector conf & api 2022-07-28 16:47:30 +08:00
DDDHuang 9ae7c62656 fix: exs deps & bad suites 2022-07-27 16:24:13 +08:00
DDDHuang be6de4aad0 fix: ee bridge & connector remove app mod 2022-07-27 11:38:45 +08:00
DDDHuang fa4bc921ac fix: hstream db connector & bridge, TODO: SUITE 2022-07-27 11:38:45 +08:00
DDDHuang 98b36c4681 fix: hstream db connector , TODO: start apps 2022-07-27 11:38:45 +08:00
DDDHuang f00a7417bb feat: hstream bridge & connector 2022-07-27 11:38:45 +08:00
DDDHuang e4eba157c3 feat: add emqx enterprise app connector & bridge 2022-07-25 09:36:45 +08:00