Commit Graph

198 Commits

Author SHA1 Message Date
Stefan Strigler f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen 8cec62bd87
fix: async reply arg list 2023-04-14 12:43:36 +08:00
JimMoen 65c4234829
refactor: rename instance_id, which is also PoolName 2023-04-14 10:02:48 +08:00
JimMoen 2d1b94f7f7
fix: file mode 755 magic number 2023-04-14 10:02:48 +08:00
JimMoen 675f9894ab
fix: fix single quotes not applied for single query
- alwasy use `emqx_plugin_libs_rule:proc_batch_sql/3` to add single quotes for string
2023-04-14 10:02:47 +08:00
JimMoen f8c0aa49a2
style: make elvis and shellcheck happy 2023-04-14 10:02:47 +08:00
JimMoen 27fdf644aa
fix(sqlserver): let a single query be a length 1 batch
Need to be fixed:
`emqx_plugin_libs_rule:proc_tmpl/2` won't add single quotes for string
2023-04-14 10:02:46 +08:00
JimMoen c366267b0f
test: MS SQL Server data bridge 2023-04-14 10:02:46 +08:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
JimMoen 8e62afecdd
chore: suite module rename 2023-04-14 09:57:00 +08:00
JimMoen 8ceeafb0de
chore: fix file license in lib-ee 2023-04-14 09:56:59 +08:00
Thales Macedo Garitezi 9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
Kjell Winblad 601564fa0a
Merge pull request #10324 from kjellwinblad/kjell/fix/EMQX-9374
fix: check Clickhouse connection after creation
2023-04-13 16:31:35 +02:00
Kjell Winblad 0cec52d5e4 fix: check Clickhouse connection after creation
When pressing reconnect on a badly configured Clickhouse bridge in the
dashboard, no error message was shown. This commit fixes this issue by
testing the connection after creation and returning an error tuple if
the connection is not working.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
2023-04-13 10:09:35 +02:00
Thales Macedo Garitezi 871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
JianBo He 9560fdc5a2 chore: typo fixes 2023-04-12 14:16:40 +08:00
JianBo He 30bdffe318 feat: support async and batch callback for cassandra connector 2023-04-10 15:08:10 +08:00
Thales Macedo Garitezi 5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
lafirest ed31b735a8
Merge pull request #10220 from lafirest/bump/hackney
chore: bump hackney version
2023-04-04 10:35:09 +08:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Zaiming (Stone) Shi 36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
firest c8dca74b18 chore: bump hackney version 2023-04-03 18:41:49 +08:00
zhongwencool d63680cf25
Merge pull request #10307 from emqx/release-50
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Kjell Winblad 58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
Kjell Winblad fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label
docs: make clickhouse config label for Server URL more concise
2023-03-31 16:17:26 +02:00
Zaiming (Stone) Shi 945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb
test: fix flaky influxdb test
2023-03-31 15:36:12 +02:00
Zaiming (Stone) Shi 9bbca07943
Merge pull request #10298 from emqx/release-50
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
Kjell Winblad e808fef1e4 feat: (MongoDB bridge) use ${var} syntax for MongoDB collection
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
2023-03-30 17:49:56 +02:00
Zaiming (Stone) Shi 81a104690d test: fix flaky influxdb test 2023-03-30 16:19:22 +02:00
Kjell Winblad 8c4d211b54 fix: on_get_status clickhouse just reporting connecting
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-30 15:10:03 +02:00
Kjell Winblad aefa0e5ffb docs: make clickhouse config label for Server URL more concise 2023-03-30 14:55:22 +02:00
Thales Macedo Garitezi b2b9732edb fix(tdengine): make password a required field (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9395

```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
2023-03-29 10:07:02 -03:00
Kjell Winblad c693851740 fix: on_get_status clickhouse just reporting connecting
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-29 14:26:08 +02:00
JianBo He bfa5922209
Merge pull request #10140 from HJianBo/cassa
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
JimMoen 798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker
fix: for connection used ecpool, let worker do health check fun
2023-03-27 09:51:52 +08:00
Zaiming (Stone) Shi dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
JimMoen d0f0a93888
fix: ecpool worker exec dynamo health check fun 2023-03-24 11:15:52 +08:00
Thales Macedo Garitezi f8d5d53908 feat(buffer_worker): decouple query mode from underlying connector call mode
Fixes https://emqx.atlassian.net/browse/EMQX-9129

Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.

Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:

1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
JianBo He 1a466fda0b chore: use binary type InstanceId as PoolName 2023-03-23 16:06:51 +08:00
JianBo He 9b63bdc1e0 chore: apply review suggestions
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
JianBo He 8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
firest 4ad3579966 test(bridges): add test suite for RocketMQ 2023-03-22 10:36:58 +08:00
firest 9853d00cad feat(bridges): integrate RocketMQ into data bridges 2023-03-22 10:36:52 +08:00
JianBo He 539ec2f774 chore(bridge): cover username/password auth for cassandra bridges 2023-03-21 13:55:53 +08:00
JianBo He c2c9abd568 chore: add running ct 2023-03-20 09:47:58 +08:00
JianBo He d8e6e2a1be test: refine CASSANDRA_HOST name 2023-03-17 18:52:52 +08:00
JianBo He 678cc937c0 test(bridge): cover ssl testing for cassandra bridge 2023-03-17 18:25:05 +08:00
JianBo He a20c39c83a test: correct CASSANDRA_HOST 2023-03-17 13:55:26 +08:00
JianBo He 75c1592e7c chore: make static_checks happy 2023-03-17 11:34:52 +08:00
JianBo He c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Andrew Mayorov a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
Serge Tupchii 97e71c54d4 fix: use default template if timestamp is empty (undefined) in InfluxDB bridge
Closes EMQX-8926
2023-03-08 11:58:23 +01:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Thales Macedo Garitezi 9b087a21f5 fix(gcp_pubsub): remove conflicting `request_timeout` option
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
2023-03-06 10:12:38 -03:00
Ivan Dyachkov c01f62a1c1 chore: bump apps versions 2023-03-03 13:58:32 +01:00
Kjell Winblad 67acdf0888 feat: add clickhouse database bridge
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi 083330ad80 Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master 2023-03-01 08:53:03 +01:00
Zaiming (Stone) Shi 24f476e35f test: add README to influxdb test script 2023-02-28 19:38:43 +01:00
Erik Timan 44c0aae16b fix(emqx_ee_connector): improve log message on influxdb start failures 2023-02-24 11:31:57 +01:00
Erik Timan da42c91fb2 test(emqx_ee_bridge): check influxdb:is_alive/2 return 2023-02-24 09:03:34 +01:00
Erik Timan 8314c07147 chore: update influxdb-client-erl dep 2023-02-24 09:03:34 +01:00
Erik Timan 603532dacd fix(emqx_ee_connector): log reason for failure when starting influxdb connector 2023-02-24 09:03:34 +01:00
Erik Timan 291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
Erik Timan ca28be4387 test(emqx_ee_connector): review fixes of influxdb tests 2023-02-17 15:28:49 +01:00
Erik Timan 843ed464d5 fix: remove all mentions of UDP support for InfluxDB EE bridge
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
firest 81adb02402 chore: bump apps version 2023-02-14 22:04:29 +08:00
firest 530b21268a feat(bridges): add TDengine 2023-02-14 22:04:23 +08:00
Erik Timan 4eb4430fe2 chore: bump VSN 2023-02-09 17:22:19 +01:00
Erik Timan bc6c653c9f test(emqx_ee_connector): rework and improve influxdb tests 2023-02-09 17:18:47 +01:00
Erik Timan cfd0e9ebdd refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
2023-02-09 17:18:47 +01:00
Erik Timan cf77dcf25e test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
2023-02-09 17:18:47 +01:00
Zaiming (Stone) Shi 157c919ba1 ci: add i18n style check script 2023-02-09 11:41:52 +01:00
Zaiming (Stone) Shi 7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi 30a8a436b4 fix(influxdb_bridge): no need to create atom for pool name 2023-01-27 12:39:05 +01:00
Erik Timan 1f235ffee9 refactor(emqx_ee_connector): redo readact key function 2023-01-26 10:30:13 +01:00
Erik Timan 805d08e823 fix: reduce log level from error to warning in several places
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
2023-01-25 14:49:50 +01:00
Erik Timan 8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi feb0238911 fix(influxdb): ensure client is stopped before returning error 2023-01-24 20:35:02 +01:00
Erik Timan 7603ab490b fix(emqx_ee_connector): fix passing of influxdb ssl opts 2023-01-24 14:30:40 +01:00
Erik Timan 28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi 6fa6c679bb feat(buffer_worker): add expiration time to requests
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi 7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
Thales Macedo Garitezi 087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi 006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi 4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
JimMoen 40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zhongwen Deng f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Shawn 1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Thales Macedo Garitezi b9f258b737 feat(mongodb): add `payload_template` field for bridge (e5.0)
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
2023-01-09 13:38:11 -03:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 0ce1ca89b7 refactor: use string type for server and servers 2022-12-30 14:20:23 +01:00
Thales Macedo Garitezi a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
2022-12-30 09:33:41 -03:00
Thales Macedo Garitezi 446a4c74d0 fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652

Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.

Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
Zaiming (Stone) Shi 5ae4bd6722 chore: bump influxdb client lib version from 1.1.4 to 1.1.5
fixed a compile warning
2022-12-29 10:29:36 +01:00
Thales Macedo Garitezi c0b208dd9e fix(influxdb): check if fields are empty before sending
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461

Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.

```
  14:03:35.563 [error] [InfluxDB] Encode [
    %{
      fields: %{},
      measurement: "t/topic",
      tags: %{},
      timestamp: 1670864615563
    }
  ] failed: :error :missing_field [
    {:influxdb_line, :encode_fields, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 60
     ]},
    {:influxdb_line, :encode_, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 44
     ]},
    {:influxdb_line, :"-encode_/1-fun-0-", 2,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 38
     ]},
    {:influxdb, :write, 2,
     [file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
    {:emqx_ee_connector_influxdb, :do_query, 3,
     [
       file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
       line: 322
     ]},
    {:emqx_resource_worker, :apply_query_fun, 6,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 454
     ]},
    {:emqx_resource_worker, :query_or_acc, 3,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 306
     ]},
    {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]
  2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

  14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
  2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
  []

  14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
  iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```

Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.

Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi 9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00