Stefan Strigler
f8e9e54393
refactor: move emqx_json to emqx_utils_json
2023-04-14 13:31:27 +02:00
JimMoen
8cec62bd87
fix: async reply arg list
2023-04-14 12:43:36 +08:00
JimMoen
65c4234829
refactor: rename instance_id, which is also PoolName
2023-04-14 10:02:48 +08:00
JimMoen
2d1b94f7f7
fix: file mode 755 magic number
2023-04-14 10:02:48 +08:00
JimMoen
675f9894ab
fix: fix single quotes not applied for single query
...
- alwasy use `emqx_plugin_libs_rule:proc_batch_sql/3` to add single quotes for string
2023-04-14 10:02:47 +08:00
JimMoen
f8c0aa49a2
style: make elvis and shellcheck happy
2023-04-14 10:02:47 +08:00
JimMoen
27fdf644aa
fix(sqlserver): let a single query be a length 1 batch
...
Need to be fixed:
`emqx_plugin_libs_rule:proc_tmpl/2` won't add single quotes for string
2023-04-14 10:02:46 +08:00
JimMoen
c366267b0f
test: MS SQL Server data bridge
2023-04-14 10:02:46 +08:00
JimMoen
5841969877
feat: implement Microsoft SQL Server bridge (e5.0)
2023-04-14 10:02:45 +08:00
JimMoen
8e62afecdd
chore: suite module rename
2023-04-14 09:57:00 +08:00
JimMoen
8ceeafb0de
chore: fix file license in lib-ee
2023-04-14 09:56:59 +08:00
Thales Macedo Garitezi
9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
...
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
Kjell Winblad
601564fa0a
Merge pull request #10324 from kjellwinblad/kjell/fix/EMQX-9374
...
fix: check Clickhouse connection after creation
2023-04-13 16:31:35 +02:00
Kjell Winblad
0cec52d5e4
fix: check Clickhouse connection after creation
...
When pressing reconnect on a badly configured Clickhouse bridge in the
dashboard, no error message was shown. This commit fixes this issue by
testing the connection after creation and returning an error tuple if
the connection is not working.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
2023-04-13 10:09:35 +02:00
Thales Macedo Garitezi
871ee90b3e
refactor(kafka_bridge): move kafka bridge into its own app
...
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov
f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
...
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
Ivan Dyachkov
bdffa925db
chore: merge upstream/master release-50
2023-04-12 15:30:20 +02:00
JianBo He
9560fdc5a2
chore: typo fixes
2023-04-12 14:16:40 +08:00
JianBo He
30bdffe318
feat: support async and batch callback for cassandra connector
2023-04-10 15:08:10 +08:00
Thales Macedo Garitezi
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
lafirest
ed31b735a8
Merge pull request #10220 from lafirest/bump/hackney
...
chore: bump hackney version
2023-04-04 10:35:09 +08:00
Thales Macedo Garitezi
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Zaiming (Stone) Shi
36000abf51
refactor: relocate i18n files for apps/emqx
2023-04-03 13:12:24 +02:00
firest
c8dca74b18
chore: bump hackney version
2023-04-03 18:41:49 +08:00
zhongwencool
d63680cf25
Merge pull request #10307 from emqx/release-50
...
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Kjell Winblad
58898ea11d
Merge pull request #10294 from kjellwinblad/kjell/feat/collection_var_syntax_mongodb/EMQX-9246
...
feat: (MongoDB bridge) use ${var} syntax in MongoDB collection field
2023-03-31 17:18:27 +02:00
Kjell Winblad
fe400c7f8d
Merge pull request #10272 from kjellwinblad/kjell/better_label
...
docs: make clickhouse config label for Server URL more concise
2023-03-31 16:17:26 +02:00
Zaiming (Stone) Shi
945c26d027
Merge pull request #10291 from zmstone/0330-fix-flaky-test-influxdb
...
test: fix flaky influxdb test
2023-03-31 15:36:12 +02:00
Zaiming (Stone) Shi
9bbca07943
Merge pull request #10298 from emqx/release-50
...
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
Kjell Winblad
e808fef1e4
feat: (MongoDB bridge) use ${var} syntax for MongoDB collection
...
This commit makes it possible to use the ${var} syntax to refer to
variables in the payload of the message in the collection field.
This makes it possible to select which collection to insert into
dynamically.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9246
2023-03-30 17:49:56 +02:00
Zaiming (Stone) Shi
81a104690d
test: fix flaky influxdb test
2023-03-30 16:19:22 +02:00
Kjell Winblad
8c4d211b54
fix: on_get_status clickhouse just reporting connecting
...
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-30 15:10:03 +02:00
Kjell Winblad
aefa0e5ffb
docs: make clickhouse config label for Server URL more concise
2023-03-30 14:55:22 +02:00
Thales Macedo Garitezi
b2b9732edb
fix(tdengine): make password a required field (rv5.0)
...
Fixes https://emqx.atlassian.net/browse/EMQX-9395
```
2023-03-29T09:59:02.655495+00:00 [warning] msg: start_resource_failed, mfa: emqx_resource_manager:start_resource/2, line: 524, id: <<"bridge:tdengine:jimmoen-test-trigger">>, reason: {error,function_clause,[{emqx_ee_connector_tdengine,on_start,[<<"bridge:tdengine:jimmoen-test-trigger:1076">>,#{database => <<"db">>,enable => true,pool_size => 8,resource_opts => #{auto_restart_interval => 60000,batch_size => 1,batch_time => 0,health_check_interval => 15000,max_queue_bytes => 104857600,query_mode => sync,request_timeout => 15000,start_after_created => true,start_timeout => 5000,worker_pool_size => 16},server => "127.0.0.1",sql => <<"insert into mqtt.t_mqtt_msg(ts, msgid, mqtt_topic, qos, payload, arrived) values (${ts}, ${id}, ${topic}, ${qos}, ${payload}, ${timestamp})">>,username => <<"root">>}],[{file,"emqx_ee_connector_tdengine.erl"},{line,76}]},{emqx_resource,call_start,3,[{file,"emqx_resource.erl"},{line,359}]},{emqx_resource_manager,start_resource,2,[{file,"emqx_resource_manager.erl"},{line,513}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1205}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
```
2023-03-29 10:07:02 -03:00
Kjell Winblad
c693851740
fix: on_get_status clickhouse just reporting connecting
...
The on_get_status callback for clickhouse just returned `connecting`
without error information when the status check was unsuccessful. This is
fixed by letting the callback return error information similarly to how
the HTTP connector does it.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9374
https://emqx.atlassian.net/browse/EMQX-9278
2023-03-29 14:26:08 +02:00
JianBo He
bfa5922209
Merge pull request #10140 from HJianBo/cassa
...
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
JimMoen
798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker
...
fix: for connection used ecpool, let worker do health check fun
2023-03-27 09:51:52 +08:00
Zaiming (Stone) Shi
dd996ad1dc
chore: bump app vsns
2023-03-24 21:47:15 +01:00
JimMoen
d0f0a93888
fix: ecpool worker exec dynamo health check fun
2023-03-24 11:15:52 +08:00
Thales Macedo Garitezi
f8d5d53908
feat(buffer_worker): decouple query mode from underlying connector call mode
...
Fixes https://emqx.atlassian.net/browse/EMQX-9129
Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.
Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:
1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
JianBo He
1a466fda0b
chore: use binary type InstanceId as PoolName
2023-03-23 16:06:51 +08:00
JianBo He
9b63bdc1e0
chore: apply review suggestions
...
- Rename sql to cql
- Add tests for `bridges_probe` API
2023-03-23 15:27:34 +08:00
JianBo He
8cbbc9f271
Merge remote-tracking branch 'upstream/master' into cassa
2023-03-23 11:53:17 +08:00
firest
4ad3579966
test(bridges): add test suite for RocketMQ
2023-03-22 10:36:58 +08:00
firest
9853d00cad
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 10:36:52 +08:00
JianBo He
539ec2f774
chore(bridge): cover username/password auth for cassandra bridges
2023-03-21 13:55:53 +08:00
JianBo He
c2c9abd568
chore: add running ct
2023-03-20 09:47:58 +08:00
JianBo He
d8e6e2a1be
test: refine CASSANDRA_HOST name
2023-03-17 18:52:52 +08:00
JianBo He
678cc937c0
test(bridge): cover ssl testing for cassandra bridge
2023-03-17 18:25:05 +08:00
JianBo He
a20c39c83a
test: correct CASSANDRA_HOST
2023-03-17 13:55:26 +08:00
JianBo He
75c1592e7c
chore: make static_checks happy
2023-03-17 11:34:52 +08:00
JianBo He
c0a216a740
feat(bridge): support cassandra bridge
2023-03-17 11:34:48 +08:00
Andrew Mayorov
a86d06f043
chore: bump app versions following last merge-back
2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi
fe27604010
Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master
2023-03-08 16:46:45 +01:00
Serge Tupchii
97e71c54d4
fix: use default template if timestamp is empty (undefined) in InfluxDB bridge
...
Closes EMQX-8926
2023-03-08 11:58:23 +01:00
firest
b8b1a78025
feat(bridges): integrate the DynamoDB into data bridges
2023-03-08 11:13:38 +08:00
Thales Macedo Garitezi
9b087a21f5
fix(gcp_pubsub): remove conflicting `request_timeout` option
...
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
2023-03-06 10:12:38 -03:00
Ivan Dyachkov
c01f62a1c1
chore: bump apps versions
2023-03-03 13:58:32 +01:00
Kjell Winblad
67acdf0888
feat: add clickhouse database bridge
...
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi
083330ad80
Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master
2023-03-01 08:53:03 +01:00
Zaiming (Stone) Shi
24f476e35f
test: add README to influxdb test script
2023-02-28 19:38:43 +01:00
Erik Timan
44c0aae16b
fix(emqx_ee_connector): improve log message on influxdb start failures
2023-02-24 11:31:57 +01:00
Erik Timan
da42c91fb2
test(emqx_ee_bridge): check influxdb:is_alive/2 return
2023-02-24 09:03:34 +01:00
Erik Timan
8314c07147
chore: update influxdb-client-erl dep
2023-02-24 09:03:34 +01:00
Erik Timan
603532dacd
fix(emqx_ee_connector): log reason for failure when starting influxdb connector
2023-02-24 09:03:34 +01:00
Erik Timan
291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts
...
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
2023-02-20 10:05:47 +01:00
Erik Timan
ca28be4387
test(emqx_ee_connector): review fixes of influxdb tests
2023-02-17 15:28:49 +01:00
Erik Timan
843ed464d5
fix: remove all mentions of UDP support for InfluxDB EE bridge
...
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
2023-02-17 10:47:57 +01:00
firest
81adb02402
chore: bump apps version
2023-02-14 22:04:29 +08:00
firest
530b21268a
feat(bridges): add TDengine
2023-02-14 22:04:23 +08:00
Erik Timan
4eb4430fe2
chore: bump VSN
2023-02-09 17:22:19 +01:00
Erik Timan
bc6c653c9f
test(emqx_ee_connector): rework and improve influxdb tests
2023-02-09 17:18:47 +01:00
Erik Timan
cfd0e9ebdd
refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts
...
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
2023-02-09 17:18:47 +01:00
Erik Timan
cf77dcf25e
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
...
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
2023-02-09 17:18:47 +01:00
Zaiming (Stone) Shi
157c919ba1
ci: add i18n style check script
2023-02-09 11:41:52 +01:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens
...
fix: redact influxdb tokens in logs and reduce log level
2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
30a8a436b4
fix(influxdb_bridge): no need to create atom for pool name
2023-01-27 12:39:05 +01:00
Erik Timan
1f235ffee9
refactor(emqx_ee_connector): redo readact key function
2023-01-26 10:30:13 +01:00
Erik Timan
805d08e823
fix: reduce log level from error to warning in several places
...
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
2023-01-25 14:49:50 +01:00
Erik Timan
8836494542
fix: redact influxdb tokens in a few logs
2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi
feb0238911
fix(influxdb): ensure client is stopped before returning error
2023-01-24 20:35:02 +01:00
Erik Timan
7603ab490b
fix(emqx_ee_connector): fix passing of influxdb ssl opts
2023-01-24 14:30:40 +01:00
Erik Timan
28718edbfd
chore: bump application VSNs
2023-01-24 14:12:34 +01:00
Thales Macedo Garitezi
ca4a262b75
refactor: re-organize dealing with unrecoverable errors
2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb
feat(buffer_worker): add expiration time to requests
...
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.
Also introduces two new counters:
- `dropped.expired` :: happens when a request expires before being
sent downstream
- `late_reply` :: when a response is receive from downstream, but the
caller is no longer for a reply because the request has expired, and
the caller might even have retried it.
2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter
...
fix(schema): add password converter to ensure its binary() type
2023-01-18 15:09:05 +01:00
Thales Macedo Garitezi
087b667263
fix(buffer_worker): allow signalling unrecoverable errors
2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
006b4bda97
feat(buffer_worker): monitor async workers and cancel their inflight requests upon death
2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
4a7e74f5d6
fix(schema): add password converter to ensure its binary() type
2023-01-17 11:12:17 +01:00
JimMoen
40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api
2023-01-17 09:29:22 +08:00
Zhongwen Deng
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
Shawn
1711823487
refactor: remove default value of timestamp field of influxdb
2023-01-12 22:25:44 +08:00
Thales Macedo Garitezi
b9f258b737
feat(mongodb): add `payload_template` field for bridge (e5.0)
...
https://emqx.atlassian.net/browse/EMQX-8705
Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
2023-01-09 13:38:11 -03:00
Zaiming (Stone) Shi
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi
0ce1ca89b7
refactor: use string type for server and servers
2022-12-30 14:20:23 +01:00
Thales Macedo Garitezi
a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50
...
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
2022-12-30 09:33:41 -03:00
Thales Macedo Garitezi
446a4c74d0
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
...
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652
Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.
Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
Zaiming (Stone) Shi
5ae4bd6722
chore: bump influxdb client lib version from 1.1.4 to 1.1.5
...
fixed a compile warning
2022-12-29 10:29:36 +01:00
Thales Macedo Garitezi
c0b208dd9e
fix(influxdb): check if fields are empty before sending
...
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461
Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.
```
14:03:35.563 [error] [InfluxDB] Encode [
%{
fields: %{},
measurement: "t/topic",
tags: %{},
timestamp: 1670864615563
}
] failed: :error :missing_field [
{:influxdb_line, :encode_fields, 1,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 60
]},
{:influxdb_line, :encode_, 1,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 44
]},
{:influxdb_line, :"-encode_/1-fun-0-", 2,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 38
]},
{:influxdb, :write, 2,
[file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
{:emqx_ee_connector_influxdb, :do_query, 3,
[
file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
line: 322
]},
{:emqx_resource_worker, :apply_query_fun, 6,
[
file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
line: 454
]},
{:emqx_resource_worker, :query_or_acc, 3,
[
file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
line: 306
]},
{:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
]
2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]
14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
[]
14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```
Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.
Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00