yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,153 Commits 393 Branches 684 Tags 74 MiB
Commit Graph

703 Commits

Author SHA1 Message Date
Erik Timan 2d75c7d6d9 fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints 
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
 2023-03-20 09:43:11 +01:00
JianBo He 12942b676d Merge remote-tracking branch 'upstream/master' into cassa 2023-03-20 09:50:27 +08:00
JianBo He c2c9abd568 chore: add running ct 2023-03-20 09:47:58 +08:00
JianBo He d8e6e2a1be test: refine CASSANDRA_HOST name 2023-03-17 18:52:52 +08:00
JianBo He 678cc937c0 test(bridge): cover ssl testing for cassandra bridge 2023-03-17 18:25:05 +08:00
JianBo He 5f0828a2ea ci: add certs for cassandra tls 2023-03-17 16:39:10 +08:00
JianBo He a20c39c83a test: correct CASSANDRA_HOST 2023-03-17 13:55:26 +08:00
JianBo He 75c1592e7c chore: make static_checks happy 2023-03-17 11:34:52 +08:00
JianBo He c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Thales Macedo Garitezi 966276127e test: trying to make tests more stable 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 0587e0d3f3 refactor(kafka_consumer): change the generated consumer group id 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 41b8d47696 test(kafka_consumer): add more clusterized tests 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 19f9c35662 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi fc5dfa108a fix(kafka_consumer): rename `force_utf8` to `none` 
We actually enforce the key/value to be a valid UTF-8 string when
using `emqx_json:encode`, if we do encode using that, which is
template-dependent.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 37a35b22df docs(kafka): improve kafka consumer/producer API examples 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 53979b6261 feat(kafka_consumer): support multiple topic mappings, payload 
templates and key/value encodings

Added after feedback from the product team.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi f31f15e44e chore(kafka_producer): make schema changes more backwards compatible 
This will still require fixes to the frontend.
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi c182a4053e fix(kafka_consumer): avoid leaking atoms in bridge probe API 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 27dfd98f46 fix(kafka): fix api methods schemas and bridge type enum 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi fecc08bb8f refactor(kafka_consumer): apply review suggestions 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 2a3aef92bb refactor(kafka_consumer): redact error info 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 9037314aea docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 1b78f22d92 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi e1fdd041b3 feat(kafka_consumer): add `offset_commit_interval_seconds` kafka parameter 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 1d5fe14a30 test: remove sleeps 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 65c15b3fae refactor(kafka_consumer): move subscriber startup logic to separate fn 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 969fa03ecc feat: implement kafka consumer 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 5ab5236ad3 test: fix flaky test 2023-03-16 13:42:38 -03:00
Kinplemelon 651f72eabc docs: correct the chinese label for the `socket_nodelay` field 2023-03-16 11:39:29 +08:00
Andrew Mayorov a530ccbe3d
Merge pull request #10095 from fix/EEC-782/mysql-prepstmt-exhaustion 
fix(mysql): be explicit that batch queries are parameterless
 2023-03-14 10:21:45 +03:00
ieQu1 e3595f2e79 chore(mria): Bump version to 0.4.0 2023-03-11 00:37:25 +01:00
Andrew Mayorov f7c0d29478
test(tde): add testcase for a nasty string in SQL query 
Similar to what we have in mysql and pgqsl testsuites.
 2023-03-10 18:43:30 +03:00
Andrew Mayorov 0a7f6c7d03
fix(mysql): ensure proper escaping in batch inserts 
Also hexencode non-utf8 binaries. This is essentially an heuristic.
We don't know column types in runtime, and there's no simple way
to find them out. Since we're already doing full binary scan during
escaping it should be cheap to bail out on non-utf8 strings and
hexencode them instead.

Also introduce separate function to highlight that this escaping
is MySQL-specific.
 2023-03-10 18:42:04 +03:00
Andrew Mayorov fc37d9b3cd
fix(mysql): be explicit that batch queries are parameterless 
So that mysql client won't attempt to prepare them automatically, thus
trashing the server's prepared statements table and making interaction
overall heavier.
 2023-03-10 18:42:04 +03:00
Andrew Mayorov a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
Serge Tupchii 97e71c54d4 fix: use default template if timestamp is empty (undefined) in InfluxDB bridge 
Closes EMQX-8926
 2023-03-08 11:58:23 +01:00
firest 984dd3446d test(bridges): add test suite for DynamoDB 2023-03-08 11:13:51 +08:00
firest bfc2920a55 feat(bridges): introduce DynamoDB definition files from v4 2023-03-08 11:13:51 +08:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Thales Macedo Garitezi 18ab7ed197 chore: bump app vsns 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi e9d3fc511f chore(buffer_worker): change default `batch_time` to 0 and improve docs 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi 9825998207 chore(gcp_pubsub): just deprecate `request_timeout` instead of removing 2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi 9b087a21f5 fix(gcp_pubsub): remove conflicting `request_timeout` option 
Since the buffer worker schema already contains that configuration,
having it two places can lead to quite confusing behavior.
 2023-03-06 10:12:38 -03:00
Ivan Dyachkov c01f62a1c1 chore: bump apps versions 2023-03-03 13:58:32 +01:00
Kjell Winblad c24a84e370 docs: fix spelling 2023-03-03 12:06:18 +01:00
Kjell Winblad 67acdf0888 feat: add clickhouse database bridge 
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
 2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi 083330ad80 Merge remote-tracking branch 'origin/master' into 0301-merge-release-50-to-master 2023-03-01 08:53:03 +01:00
Zaiming (Stone) Shi 41cb7263ca docs(influxdb): add a node to the write_syntax config 2023-02-28 21:53:32 +01:00
Zaiming (Stone) Shi 24f476e35f test: add README to influxdb test script 2023-02-28 19:38:43 +01:00
Ivan Dyachkov 6ce5029d79
Merge pull request #9881 from olcai/log-influxdb-is-alive-reason 
fix(emqx_ee_connector): log reason for failure when starting influxdb connector
 2023-02-28 09:49:08 +00:00
Zaiming (Stone) Shi 79bf77c2f1
Merge pull request #10020 from zmstone/0222-fix-bridge-async-mode-counters 
fix(bridge): fix dropped counter and inflight gauge
 2023-02-24 19:51:24 +01:00
Andrew Mayorov 9cbe64a132
fix(test): make strings json-friendly in kafka testsuite 2023-02-24 15:05:20 +03:00
Erik Timan 44c0aae16b fix(emqx_ee_connector): improve log message on influxdb start failures 2023-02-24 11:31:57 +01:00
Erik Timan da42c91fb2 test(emqx_ee_bridge): check influxdb:is_alive/2 return 2023-02-24 09:03:34 +01:00
Erik Timan 8314c07147 chore: update influxdb-client-erl dep 2023-02-24 09:03:34 +01:00
Erik Timan 603532dacd fix(emqx_ee_connector): log reason for failure when starting influxdb connector 2023-02-24 09:03:34 +01:00
Zaiming (Stone) Shi ada68e12e6 fix(kafka): change default value for memory_overload_protection to false 2023-02-22 13:46:51 +01:00
Zaiming (Stone) Shi 9316690c29 fix(schema): binary string for default values 
A lot of the string value fields had default value defined in
schema as list-string rather than binary-string.
This caused the generated schema dump (in JSON format)
to have raw_default field as an integer array.
 2023-02-21 09:09:51 +01:00
Zaiming (Stone) Shi 53ecfb98d0 chore: fix stale code comments 2023-02-21 09:09:51 +01:00
Ivan Dyachkov c869eff6e8 test(kafka): disable overload protection in 2 more places 2023-02-20 21:38:14 +01:00
Ivan Dyachkov 9d30a52910 test: disable replayq memory overload protection 2023-02-20 19:29:26 +01:00
Erik Timan 291755f81f
Merge pull request #9895 from olcai/add-test-for-influxdb-ssl-opts 
test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts
 2023-02-20 10:05:47 +01:00
Erik Timan ca28be4387 test(emqx_ee_connector): review fixes of influxdb tests 2023-02-17 15:28:49 +01:00
Erik Timan caacc08ff9 chore: bump app VSN 2023-02-17 15:28:27 +01:00
Erik Timan 843ed464d5 fix: remove all mentions of UDP support for InfluxDB EE bridge 
The InfluxDB EE bridge doesn't support the UDP protocol, but it is
defined in the schema. This commit removes all such traces in the
schema.
 2023-02-17 10:47:57 +01:00
ieQu1 d9554c36ac chore: Bump umbrealla app versions 2023-02-17 09:50:50 +01:00
ieQu1 794bedef9f fix(emqx_ctl): Start CLI before ekka 2023-02-17 09:50:50 +01:00
firest 81adb02402 chore: bump apps version 2023-02-14 22:04:29 +08:00
firest 0420e9acb5 test(bridges): add test cases for TDEngine 2023-02-14 22:04:29 +08:00
firest 530b21268a feat(bridges): add TDengine 2023-02-14 22:04:23 +08:00
Zaiming (Stone) Shi 42dfaf3ef2
Merge pull request #9910 from sstrigler/EMQX-8861-improve-bridge-restart-button-behaviour 
EMQX 8861 improve bridge restart button behaviour
 2023-02-09 18:00:48 +01:00
Erik Timan 4eb4430fe2 chore: bump VSN 2023-02-09 17:22:19 +01:00
Erik Timan bc6c653c9f test(emqx_ee_connector): rework and improve influxdb tests 2023-02-09 17:18:47 +01:00
Erik Timan cfd0e9ebdd refactor(emqx_ee_connector): use emqx_tls_lib for influx ssl opts 
We used to simply pass on SSL options given to the influxdb EE
connector, but we now pass them to emqx_tls_lib instead. This ensures a
proper handling of SSL options and also allow us to use meck to inject
custom options in tests.
 2023-02-09 17:18:47 +01:00
Erik Timan cf77dcf25e test(emqx_ee_connector): add basic tests for influxdb incl. SSL opts 
This adds a test suite for the emqx_ee_connector_influxdb. We add it so
that SSL transport options are properly tested.
 2023-02-09 17:18:47 +01:00
Zaiming (Stone) Shi 157c919ba1 ci: add i18n style check script 2023-02-09 11:41:52 +01:00
Stefan Strigler c7535fce56 test: fix test expecting 200 instead of 204 2023-02-08 09:14:35 +01:00
Zaiming (Stone) Shi 6362476508 feat: disable telemetry for ee by default 2023-02-03 09:29:39 +01:00
Zaiming (Stone) Shi feca4cc0a5
Merge pull request #9892 from zmstone/0202-docs-cosmetic 
0202 docs cosmetic
 2023-02-02 15:43:58 +01:00
zhongwencool ee852d8204
Merge pull request #9886 from zhongwencool/mongo-connection-default-async 
fix: remove async mode from mongodb/redis/mysql/pgsql bridge
 2023-02-02 21:08:01 +08:00
Zaiming (Stone) Shi d5c482b0b0 docs: remove timer unit from description 
the user input has time unit. e.g. "5s" for 5 seconds etc.
 2023-02-02 13:49:20 +01:00
Zaiming (Stone) Shi 7ad80d012c chore(kafka): upgrade to woff 1.7.5 
comparing to 1.7.4, a counter bump bug was fixed
but it does not affect EMQX as the bridge does not make use
of the counter which had issue.
 2023-02-02 12:23:15 +01:00
Zaiming (Stone) Shi 0eb554a62e fix(kafka): check Kafka partition leader connectivity 2023-02-02 12:04:17 +01:00
Zhongwen Deng 1c9035d24c test: remove async from redis ct 2023-02-02 17:37:18 +08:00
Zhongwen Deng e776751860 fix: delete enable_batch from redis bridge 2023-02-02 17:37:18 +08:00
Zhongwen Deng 22c3f50020 fix: add query_mode_sync_only for mysql pgsql redis mongodb bridge 2023-02-02 17:37:18 +08:00
Zhongwen Deng dcc6bd9c21 fix: mongodb bridge'd default connect_opt should be async, not sync 2023-02-02 17:37:18 +08:00
Zhongwen Deng 1a90c1654c chore: bad typo 2023-02-02 11:43:04 +08:00
Ilya Averyanov 9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover 
fix: fix redis cluster resource recovery
 2023-02-01 10:38:52 +02:00
Ilya Averyanov fce1e74c3d fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00
Zaiming (Stone) Shi b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback 
0127 refactor buffer worker no need to keep request for reply callback
 2023-01-31 08:44:01 +01:00
Zaiming (Stone) Shi d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi 7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi 52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi 514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak 
0127 fix influxdb bridge atom leak
 2023-01-27 14:30:20 +01:00
Zaiming (Stone) Shi 30a8a436b4 fix(influxdb_bridge): no need to create atom for pool name 2023-01-27 12:39:05 +01:00
Andrew Mayorov d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching 
feat(redis): disable batching in redis_cluster bridges
 2023-01-27 15:27:57 +04:00
Stefan Strigler 7d18128ba9 test: async write can return noproc 2023-01-27 11:43:51 +01:00
Stefan Strigler 2d62de5188 test: fix expected result from timeout error 2023-01-27 11:43:48 +01:00
Zaiming (Stone) Shi f6b3b930b0 chore: improve a error log 2023-01-26 14:21:27 +01:00
Erik Timan 1f235ffee9 refactor(emqx_ee_connector): redo readact key function 2023-01-26 10:30:13 +01:00
Zaiming (Stone) Shi bafc430184
Merge pull request #9837 from zmstone/0123-fix-resource-worker-leak-influxdb 
0123 fix resource worker leak influxdb
 2023-01-26 09:14:38 +01:00
Andrew Mayorov 26fcaecad7
fix(redis): disable batching in `redis_cluster` bridges 
Through configuration subsystem.
 2023-01-25 17:28:11 +03:00
Erik Timan 805d08e823 fix: reduce log level from error to warning in several places 
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
 2023-01-25 14:49:50 +01:00
Erik Timan 8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Andrew Mayorov 903a77b471
test(redis): ensure batch query hit different cluster shards 
This will inevitably fail: it's not generally possible to update
different keys through the same cluster connection, one or more
update will fail with `MOVED` status. This testcase should serve
as a regression test later.
 2023-01-25 15:33:05 +03:00
Zaiming (Stone) Shi 5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Zaiming (Stone) Shi feb0238911 fix(influxdb): ensure client is stopped before returning error 2023-01-24 20:35:02 +01:00
Erik Timan 7603ab490b fix(emqx_ee_connector): fix passing of influxdb ssl opts 2023-01-24 14:30:40 +01:00
Erik Timan 28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi 8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Thales Macedo Garitezi ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi 6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Kjell Winblad 8c52264c41 fix: atom leak when doing Kafka bridge dry-run 
A new atom was created every time one did a dry run of a Kafka bridge
(that is, clicking the Test button in the settings dialog for the
bridge).

After this fix, we will only create a new atom when a bridge with a new
name is created. This should be acceptable as bridges with new names are
created relatively rarely and it seems to be useful to have an unique
atom for every Wolff producer.

Fixes: https://emqx.atlassian.net/browse/EMQX-8739
 2023-01-20 11:50:20 +01:00
Thales Macedo Garitezi a5424959c6 refactor: avoid operator precedence bugs 2023-01-18 16:15:34 -03:00
Thales Macedo Garitezi 47f796dd12 refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker` 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test 
chore(ee bridge): fix Redis bridge test flakyness
 2023-01-18 20:56:13 +02:00
Zaiming (Stone) Shi caf50d1b35
Merge pull request #9792 from savonarola/redis-bridge-examples 
chore(bridges): add more helpful descriptions
 2023-01-18 18:01:35 +01:00
Ilya Averyanov f9c3115cd4 chore(bridges): add more helpful descriptions 
Add more helpful descriptions for `command_template` option
of Redis bridge.
 2023-01-18 18:17:08 +02:00
Zaiming (Stone) Shi 7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter 
fix(schema): add password converter to ensure its binary() type
 2023-01-18 15:09:05 +01:00
Zaiming (Stone) Shi 2d01e604a5
Merge pull request #9799 from zmstone/0118-fix-key_dispatch-kafka-produce-strategy 
0118 fix key dispatch kafka produce strategy
 2023-01-18 13:52:49 +01:00
Ilya Averyanov f6fbbf3ee3 chore(bridges): reduce Redis bridge flakyness 2023-01-18 14:34:11 +02:00
Zaiming (Stone) Shi 8f275a66d0 test: add coverage for key_dispatch partition strategy 2023-01-18 11:47:37 +01:00
Zaiming (Stone) Shi d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Zaiming (Stone) Shi 3872c4451f fix(kafka): olp fix was accidentally deleted in 0fd8880d0a 2023-01-18 10:52:29 +01:00
Thales Macedo Garitezi 087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi 006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi 4c25be8a2c chore(influxdb): remove deprecated value from example 2023-01-17 16:48:48 -03:00
Stefan Strigler f37b3e4bc4 test: test against `bridges_probe` API 2023-01-17 15:29:19 +01:00
Stefan Strigler 5fc31dcd9e fix: use atom keys in example to make sure they exist 2023-01-17 15:29:19 +01:00
Zaiming (Stone) Shi 4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
Shawn 1c7fe91a91 fix: typos in emqx_ee_bridge_mongodb.conf 2023-01-17 16:33:54 +08:00
JimMoen 40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api 2023-01-17 09:29:22 +08:00
Zaiming (Stone) Shi a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs 
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
 2023-01-16 13:37:29 +01:00
Zaiming (Stone) Shi 47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Zaiming (Stone) Shi 376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions 
0112 docs fix some descriptions
 2023-01-16 09:56:56 +01:00
Zaiming (Stone) Shi 91c5a89985 test: wait for redis connected state 
the case is sometimes flaky because the health check sometimes
return connecting
 2023-01-14 18:33:55 +01:00
Stefan Strigler e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Erik Timan 7a17fb7308 test(emqx_ee_bridge): fix bridge enable/disable in kafka producer suite 2023-01-13 14:40:54 +01:00
Thales Macedo Garitezi f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50 
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
 2023-01-13 10:23:49 -03:00
Zhongwen Deng f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
Zaiming (Stone) Shi 522e091dd2 docs: chane cache to buffer for Kafka bridge description 2023-01-13 07:46:23 +01:00
Erik Timan 860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Thales Macedo Garitezi ed2be4d086 feat(mongo): expose buffer worker opts to the bridge frontend (5.0) 2023-01-12 15:23:41 -03:00
Shawn 1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
Xinyu Liu bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field 
refactor: remove the auto_reconnect field
 2023-01-12 11:11:00 +08:00
Thales Macedo Garitezi 2d5208fc29
Merge pull request #9485 from thalesmg/refactor-doc-gen-50-mkIII 
feat(docs): add tags to schemas
 2023-01-11 16:36:38 -03:00
Thales Macedo Garitezi 9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50 
feat(mongodb): add `payload_template` field for bridge (e5.0)
 2023-01-11 11:14:46 -03:00
Shawn 3e9c4f444f refactor: remove the auto_reconnect field 2023-01-11 21:47:06 +08:00
Thales Macedo Garitezi a9d0143d5f chore(license): change `key` type to `binary()` 
So that the `raw_default` field in the schema JSON renders properly.
 2023-01-11 09:10:04 -03:00
Thales Macedo Garitezi 48e1ba4832 feat(docs): add tags to schemas 
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
 2023-01-11 09:10:03 -03:00
Thales Macedo Garitezi f3cc722172
docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-10 09:06:12 -03:00
Zaiming (Stone) Shi 67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection 
0101 authz schema union member selection
 2023-01-09 22:17:51 +01:00
Thales Macedo Garitezi b9f258b737 feat(mongodb): add `payload_template` field for bridge (e5.0) 
https://emqx.atlassian.net/browse/EMQX-8705

Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
 2023-01-09 13:38:11 -03:00
lafirest 20f49b903f
Merge pull request #9702 from lafirest/feat/timescale_matrix 
feat(bridges): add timescale && matrix bridges
 2023-01-10 00:06:17 +08:00
firest 062f14bd65 test(bridges): add timescale && matrix test cases 2023-01-09 23:04:36 +08:00
firest e08d6dbc9b feat(bridges): add timescale && matrix bridges 2023-01-09 23:04:29 +08:00
Zaiming (Stone) Shi c3635f537a ci: wait for redis in emqx_ee_bridge_redis_SUITE 2023-01-09 14:39:56 +01:00
Zaiming (Stone) Shi 0697c692ed refactor: mongo_type and redis_type are not mandatory 
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:

[
    {mongo_type = rs,
     ...
    }
    {another backend config}
]

however the brdige types are maps like, for example:

mongodb_rs.$name {
    mongo_type = rs
    ...
}

in which case, the mongo_type is not required.

in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.

However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi ffb09f0c4d test: rename a test option name to avoid clashing with prod config name 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi e52f9d5920 refactor: use union member type selector for authz sources 2023-01-09 14:26:16 +01:00
Zaiming (Stone) Shi e498010f23 chore: update default license (100 conns) 2023-01-09 09:11:09 +01:00
Zaiming (Stone) Shi 3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip 
test: do not use tc_user_skip for test cases
 2023-01-09 09:02:06 +01:00
firest ea405fe55d test(bridges): add test case for the PostgreSQL backend 2023-01-06 11:56:28 +08:00
firest f7b50c56da feat(bridges): integrate PostgreSQL into bridges 
support both simple and batch query
 2023-01-06 11:47:22 +08:00
Zaiming (Stone) Shi 5f12cdff6c ci: should not skip tests in github actions 2023-01-05 22:47:08 +01:00
Zaiming (Stone) Shi fc12a8c4c8 test: do not use tc_user_skip for test cases 2023-01-05 22:47:06 +01:00
Thales Macedo Garitezi af31ed4264 docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-05 10:16:00 -03:00
Thales Macedo Garitezi fd360ac6c0 feat(buffer_worker): refactor buffer/resource workers to always use queue 
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.

Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled.  The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`.  The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
 2023-01-05 10:15:09 -03:00
Thales Macedo Garitezi 0fd8880d0a fix(kafka_producer): avoid multiplication of metrics when bridge is recreated 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi 8e59319bfe fix(kafka_producer): fix message loss when kafka connection is down 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi 5bd9f110d6 test: attempt to reduce flakiness 2023-01-05 10:11:59 -03:00
Thales Macedo Garitezi 3304025a9a
Merge pull request #9677 from thalesmg/test-other-gcp-flaky-v50 
test: attempt to fix flaky test
 2023-01-04 13:30:53 -03:00
Thales Macedo Garitezi 5df485df17 test: attempt to fix flaky test 2023-01-04 10:38:21 -03:00
Thales Macedo Garitezi 4feaf5916f fix(mongodb): add `type` and `name` fields to the `get`/`post` API docs 
https://emqx.atlassian.net/browse/EMQX-8643

Adds thoses fields to MongoDB API docs so the frontend may use them to
render properly.
 2023-01-03 11:46:58 -03:00
Thales Macedo Garitezi d6c8a106da test(gcp_pubsub): fix flaky test 2023-01-03 08:52:37 -03:00
lafirest 7985cd3536
Merge pull request #9638 from lafirest/fix/mysql_dup 
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
 2023-01-02 23:15:44 +08:00
Thales Macedo Garitezi 7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50 
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
 2023-01-02 10:56:47 -03:00
firest c77717b1f1 test(mysql): fix test case error 2023-01-02 21:55:22 +08:00
Thales Macedo Garitezi 69ac6b9e0d fix(kafka): fix handling of `dropped.queue_full` event from wolff 
https://emqx.atlassian.net/browse/EMQX-8530

cd20a37e65/src/wolff_producer.erl (L772-L773)

Wolff emits 2 events related to dropped messages when replayq reports
overflow.  Since in EMQX's side we also bump `dropped` when
`dropped_queue_full` happens, that was leading to wrong metrics.
 2023-01-02 09:23:38 -03:00
Zaiming (Stone) Shi fc6d9d0890 chore: issue new Enterprise evaluation license 2023-01-02 11:50:57 +01:00
Zaiming (Stone) Shi b5d2fc0ede chore: update BSL license year 2023-01-02 11:23:14 +01:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 231d6da4dc
Merge pull request #9644 from id/fix-kafka-bridge-connectivity-detection 
fix(kafka): detect connectivity in on_get_status
 2022-12-30 21:54:56 +01:00
Thales Macedo Garitezi 61246c43c4 fix(kakfa_producer): prevent multiple producers from multiplying each other's metrics 2022-12-30 16:51:24 -03:00
Thales Macedo Garitezi 8b060a75f1 refactor(metrics): use absolute gauge values rather than deltas 
https://emqx.atlassian.net/browse/EMQX-8548

Currently, we face several issues trying to keep resource metrics
reasonable.  For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.

Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID.  This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
 2022-12-30 16:51:24 -03:00
Zaiming (Stone) Shi 0ce1ca89b7 refactor: use string type for server and servers 2022-12-30 14:20:23 +01:00
Thales Macedo Garitezi a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50 
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
 2022-12-30 09:33:41 -03:00
Ivan Dyachkov 75770f2842 fix(kafka): detect connectivity in on_get_status 2022-12-30 10:18:52 +01:00
Thales Macedo Garitezi 24bae2641e chore: upgrade wolff -> 1.7.4 2022-12-29 16:58:00 -03:00
Thales Macedo Garitezi 446a4c74d0 fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0) 
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652

Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.

Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
 2022-12-29 16:30:36 -03:00
Zaiming (Stone) Shi 728cb85156
Merge pull request #9629 from zmstone/1128-fix-flaky-test-listener-restart-eaddrinuse 
test: add a 2 seconds sleep between listener sotp and restart
 2022-12-29 16:04:19 +01:00
Zaiming (Stone) Shi 5ae4bd6722 chore: bump influxdb client lib version from 1.1.4 to 1.1.5 
fixed a compile warning
 2022-12-29 10:29:36 +01:00
Zaiming (Stone) Shi a5e3f6f05b fix(kafka): Memory OLP is only applicable in linux systems 2022-12-28 15:31:53 +01:00
Thales Macedo Garitezi ce43e6b3d6 chore: upgrade kafka_protocol, wolff, brod 2022-12-27 10:30:14 -03:00
Thales Macedo Garitezi 4819794401 test(refactor): decrease test teardown noise 2022-12-27 10:30:14 -03:00
Thales Macedo Garitezi a78ecc4bb6
Merge pull request #9579 from thalesmg/bugfix-kafka-producer-stop-client-ee50 
fix(kafka_producer): cleanup client after failing to start producers
 2022-12-26 08:59:36 -03:00
Zaiming (Stone) Shi 59970272ff chore: pin hocon 0.33.0 2022-12-23 12:07:36 +01:00
Erik Timan 13942f5c49 refactor: rename error return in mysql connector 2022-12-22 10:29:12 +01:00
Erik Timan aab914d65a test: refactor EE mysql bridge test case 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2022-12-22 10:27:38 +01:00
Erik Timan 7df24000a0 test: add more EE mysql bridge test cases 2022-12-22 10:27:38 +01:00
Thales Macedo Garitezi 7242ce426b fix(kafka_producer): cleanup client after failing to start producers 
https://emqx.atlassian.net/browse/EMQX-8547

If a Kafka Producer bridge is given bad configuration (e.g.: bad authn
credentials), the Wolff client process is started successfully, as it
does not attempt to connect, but when the producer process is
attempted to be started, it fails (only then the client tries to
connect to Kafka).  At this point, an error was thrown, but the
supervised client process remained running.

If the configuration was later fixed and the bridge updated, which
prompted its removal and recreation, the Wolff client would report to
be "already started", so it would never pick up the new (fixed)
configuration, and the producers would perpetually fail to start until
the node would be restarted.

We simply ensure the client is stopped before throwing the error,
unrolling the start-up procedure.
 2022-12-19 17:48:40 -03:00
Thales Macedo Garitezi c0b208dd9e fix(influxdb): check if fields are empty before sending 
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461

Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.

```
  14:03:35.563 [error] [InfluxDB] Encode [
    %{
      fields: %{},
      measurement: "t/topic",
      tags: %{},
      timestamp: 1670864615563
    }
  ] failed: :error :missing_field [
    {:influxdb_line, :encode_fields, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 60
     ]},
    {:influxdb_line, :encode_, 1,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 44
     ]},
    {:influxdb_line, :"-encode_/1-fun-0-", 2,
     [
       file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
       line: 38
     ]},
    {:influxdb, :write, 2,
     [file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
    {:emqx_ee_connector_influxdb, :do_query, 3,
     [
       file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
       line: 322
     ]},
    {:emqx_resource_worker, :apply_query_fun, 6,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 454
     ]},
    {:emqx_resource_worker, :query_or_acc, 3,
     [
       file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
       line: 306
     ]},
    {:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
    {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
  ]
  2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]

  14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
  2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
  []

  14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
  iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```

Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.

Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
 2022-12-16 09:52:34 -03:00
Zaiming (Stone) Shi 9e3da5b661 chore: bump app versions 2022-12-14 20:07:41 +01:00
Zaiming (Stone) Shi 56066a03b5 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 20:04:20 +01:00
Zaiming (Stone) Shi 34f7fff97d chore: pin hocon 0.32.0 2022-12-14 15:53:42 +01:00
Zaiming (Stone) Shi 42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi 2932aceaf9 refactor: rename and move `emqx_ee_bridge_gcp_pubsub_resource` to `emqx_ee_connector_gcp_pubsub` 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi 464d0a5057 refactor(test): use a linked janitor for test teardown 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi b66d2904be docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi b9bc82f87a feat(gcp_pubsub): add `local_topic` config 
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi ac048dbafa docs(gcp_pubsub): describe where to get credentials 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi a095867358 test(refactor): add ids to namespace `on_exit` callbacks 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi 697b3ecf2d docs: improve descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi 1cd91a24e9 feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0) 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi 82be9d878d test(flaky): avoid inter-suite flakiness 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi c69022f3c4 test(ci): clean config after test 
Trying to fix this error for profile `emqx`:

```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.9682.1>
    registered_name: []
    exception exit: {bad_return,
                        {{emqx_conf_app,start,[normal,[]]},
                         {emqx_conf_schema,
                             [#{kind => validation_error,path => "bridges",
                                reason => unknown_fields,
                                unknown => <<"influxdb_api_v1">>,
                                unmatched => <<"mqtt,webhook">>}]}}}
      in function  application_master:init/4 (application_master.erl, line 142)
    ancestors: [<0.9681.1>]
    message_queue_len: 1
    messages: [{'EXIT',<0.9683.1>,normal}]
    links: [<0.9681.1>,<0.44.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 29
    reductions: 195
  neighbours:

=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
    application: emqx_conf
    exited: {bad_return,
                {{emqx_conf_app,start,[normal,[]]},
                 {emqx_conf_schema,
                     [#{kind => validation_error,path => "bridges",
                        reason => unknown_fields,
                        unknown => <<"influxdb_api_v1">>,
                        unmatched => <<"mqtt,webhook">>}]}}}
    type: temporary

%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
     {emqx_conf,
         {bad_return,
             {{emqx_conf_app,start,[normal,[]]},
              {emqx_conf_schema,
                  [#{kind => validation_error,path => "bridges",
                     reason => unknown_fields,
                     unknown => <<"influxdb_api_v1">>,
                     unmatched => <<"mqtt,webhook">>}]}}}}},
 [{emqx_common_test_helpers,start_app,4,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
       {line,227}]},
  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
  {emqx_plugins_SUITE,init_per_suite,1,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
       {line,34}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
    {failed,
        {emqx_plugins_SUITE,init_per_suite,
            {'EXIT',
                {{failed_to_start_app,emqx_conf,
                     {emqx_conf,
                         {bad_return,
                             {{emqx_conf_app,start,[normal,[]]},
                              {emqx_conf_schema,
                                  [#{kind => validation_error,
                                     path => "bridges",
                                     reason => unknown_fields,
                                     unknown => <<"influxdb_api_v1">>,
                                     unmatched => <<"mqtt,webhook">>}]}}}}},
                 [{emqx_common_test_helpers,start_app,4,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                       {line,227}]},
                  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
                  {emqx_plugins_SUITE,init_per_suite,1,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
                       {line,34}]},
                  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
                  {test_server,run_test_case_eval1,6,
                      [{file,"test_server.erl"},{line,1380}]},
                  {test_server,run_test_case_eval,9,
                      [{file,"test_server.erl"},{line,1224}]}]}}}}}
```
 2022-12-12 17:18:13 -03:00
Thales Macedo Garitezi 34e9056779 refactor: fix typo in variable name 
Might confuse people to think it's related to `replayq`.
 2022-12-12 17:17:51 -03:00
Zaiming (Stone) Shi 40809b2ad0 Merge remote-tracking branch 'origin/dev/ee5.0' into release-50 2022-12-09 11:45:52 +01:00
Zaiming (Stone) Shi dc14cd450d test: render config for emqx_conf only for ee bridge tests 2022-12-07 18:27:06 +01:00
Ilya Averyanov 6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Erik Timan 485c4ddcab
Merge pull request #9393 from olcai/add-mysql-bridge-test 
test: add basic mysql bridge test (ee5.0)
 2022-12-02 09:43:22 +01:00
Zaiming (Stone) Shi 9ea22d062d refactor: make all bridges optional (required = false) 2022-12-01 16:50:10 +01:00
Zaiming (Stone) Shi 7adb539ae1 chore: pin hocon 0.31.2 2022-12-01 16:50:10 +01:00
Erik Timan 499a32ce36 test: remove unnecessary async tests in EE mysql bridge 
The async test cases is not needed since the mysql connector uses the
always_sync callback mode.
 2022-12-01 16:38:03 +01:00
Erik Timan 5461311edc test: more EE mysql bridge tests 2022-12-01 10:27:51 +01:00
Erik Timan eb62192838 test: expand EE mysql bridge test with toxiproxy 2022-12-01 10:27:51 +01:00
Erik Timan 96bff1d32e test: improve basic tests for EE mysql bridge 2022-12-01 10:27:51 +01:00
Erik Timan 3e679ceb57 test: add basic tests for EE mysql bridge 2022-12-01 10:27:51 +01:00
Zaiming (Stone) Shi fa90638de2 chore: add description to apps 2022-11-30 17:03:41 +01:00
Zaiming (Stone) Shi 9382399728 chore: add an empty file emqx_license.conf to make ci happy 2022-11-30 12:52:26 +01:00
Zaiming (Stone) Shi 83fe35b148 Merge remote-tracking branch 'origin/dev/ee5.0' into 1116-sync-v5.0.10-to-ee50 2022-11-18 19:23:31 +01:00
Zaiming (Stone) Shi 7305923d12 fix: bridge name parser should not leak atom 2022-11-16 18:25:42 +01:00
Zaiming (Stone) Shi c940b901f5 chore: fix app versions 2022-11-16 16:26:43 +01:00
Thales Macedo Garitezi 0f9cc0d93f test(refactor): stop snabbkaffe on every test 2022-11-11 10:03:30 -03:00
Thales Macedo Garitezi 04588148b7 test(influxdb): increase influxdb bridge/connector coverage (ee5.0) 2022-11-07 15:15:49 -03:00
Thales Macedo Garitezi ee4c723fcb refactor: simplify wolff telemetry handler id 2022-10-17 16:26:04 -03:00
Thales Macedo Garitezi 1b2b629cdd feat: emit telemetry events for all resource worker metrics 2022-10-13 15:32:04 -03:00
Thales Macedo Garitezi 1ad3b5df17 fix: uninstall telemetry handler on resource stop, use unique id 2022-10-13 10:29:53 -03:00
Thales Macedo Garitezi 4475289ce4 feat: use upstream newly tagged 1.7.0 wolff 2022-10-11 09:47:40 -03:00
Thales Macedo Garitezi 98500313eb fix(kafka): some fixes for kafka producer 
- MQTT topic should be a binary
- use correct gauge functions from `wolff_metrics`.
- don't double increment success counter for kafka action
- adds a few more metrics assertions
 2022-10-10 17:11:29 -03:00
Kjell Winblad 57270fb8fc feat: add support for counters and gauges to the Kafka Bridge 
This commit adds support for counters and gauges to the Kafka Brige.
The Kafka bridge uses [Wolff](https://github.com/kafka4beam/wolff) for
the  Kafka connection. Wolff does its own batching and does not use the
batching functionality in `emqx_resource_worker` that is used by other
bridge types. Therefore, the counter events have to be generated by
Wolff. We have added
[telemetry](https://github.com/beam-telemetry/telemetry) events to Wolff
that we hook into to change counters and gauges for the Kafka bridge. The
counter called `matched` does not depend on specific functionality of
any bridge type so the updates of this counter is moved higher up in the
call chain then previously so that it also gets updated for Kafka
bridges.
 2022-10-10 14:40:57 -03:00
Shawn dfe14be8b2 chore: bump app vsns 2022-09-26 18:23:28 +08:00
Shawn 4135910b42 chore: merge master into dev/ee5.0 2022-09-26 09:52:33 +08:00
Zaiming (Stone) Shi f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector 
feat: Add Kafka connector
 2022-09-24 22:57:50 +02:00
Kjell Winblad a3c88b40a0 test: changes to make Kafka container run in GitHub action 2022-09-23 14:33:41 +02:00
Kjell Winblad 8e514680d8 test: fix bad binary pattern 2022-09-23 13:54:11 +02:00
Zaiming (Stone) Shi 7b601bf970 chore: delete bad parse_bridge function clause 2022-09-23 11:53:02 +02:00
Kjell Winblad adc67b165b test: test cases for Kafka bridge REST API 2022-09-23 10:09:07 +02:00
JimMoen 6788881642 fix: hide influxdb udp config 2022-09-20 14:46:59 +08:00
JimMoen 03495d8d36 fix: influxdb server string support scheme prefix 2022-09-20 14:46:59 +08:00
Kjell Winblad ac2922fc4c test: Kafka bridge cases for all combinations of SASL and SSL 2022-09-16 16:44:12 +02:00
JimMoen 0336b935a0
Merge pull request #8901 from JimMoen/fix-bridge-api 
- for connectors using ecpool, parse connect error `Reason` when start pool failed.
- Refine InfluxDB bridge desc.
 2022-09-16 10:22:06 +08:00
Kjell Winblad be7a8c11a8 test: make bridge name unique in tests 2022-09-15 16:21:32 +02:00
Kjell Winblad 4dc26eeba7 fix: use different instance id in Kafka auth test 2022-09-15 07:33:07 +02:00
JimMoen 54a9c8d201 chore: refine influxdb bridge description 2022-09-15 09:36:52 +08:00
Thales Macedo Garitezi deff429579 chore: unify default license references in fn 2022-09-14 17:26:48 -03:00
Thales Macedo Garitezi 741eb4ff51
docs: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2022-09-14 17:08:45 -03:00
Thales Macedo Garitezi 21f89fd83e chore(license): delete default config and add info to i18n 2022-09-14 17:00:21 -03:00
Kjell Winblad 5820b028cb feat: add test case for Kerberos Kafka authentication 2022-09-14 17:03:37 +02:00
Thales Macedo Garitezi 8cefdb00b8 chore(license): add default key as default value in schema 2022-09-14 09:45:01 -03:00
Thales Macedo Garitezi d4416dc581 chore: drop support for license files (ee5.0) 
Related issue: EMQX-7247

EMQX ee5.0 is to drop support for both the old license format (x509)
and for files containing the new format.
 2022-09-14 09:44:59 -03:00
Shawn f41adb0997 refactor: change some default values of resource_opts 2022-09-14 15:18:07 +08:00
Zaiming (Stone) Shi e45c99bf79 fix: kafka bridge schema 2022-09-13 20:02:48 +02:00
Kjell Winblad f0e03086a6 test: add test cases for Kafka SASL auth mechanisms plain and scram 2022-09-13 19:46:56 +02:00
Zaiming (Stone) Shi 0c1595be02 feat: Add Kafka connector 2022-09-13 19:46:56 +02:00
Zaiming (Stone) Shi befc4acced Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a 2022-09-06 20:31:38 +02:00
Xinyu Liu 23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge 
feat: add mongodb bridge (e5.0)
 2022-09-03 12:27:01 +08:00
Xinyu Liu 7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources 
Some improvements and bug fixes to emqx_resources
 2022-09-03 12:16:49 +08:00
Thales Macedo Garitezi f1048babd8 test: refactor to use hocon and schema 2022-09-02 12:04:37 -03:00
Shawn b45f3de8db refactor(resource): rename metrics batched,queued -> batching,queuing 2022-09-02 12:41:14 +08:00
Thales Macedo Garitezi 3d4afd65df feat: add mongodb bridge (e5.0) 2022-09-01 14:47:14 -03:00
Zaiming (Stone) Shi 5a3fb4a011 Merge remote-tracking branch 'origin/master' into dev/ee5.0 2022-08-31 17:11:15 +02:00
Shawn ba1f5eecd3 fix: update the swagger for new resource metrics 2022-08-31 11:14:36 +08:00
Shawn e0a6a61d73 fix: return error on start hstreamdb crash 2022-08-30 10:19:40 +08:00
Zaiming (Stone) Shi 2eb078ff3b chore: bump changed app versions 2022-08-26 14:32:28 +02:00
Shawn 6b0ccfbc43 refactor: rename the error return resource_down -> recoverable_error 2022-08-26 17:11:12 +08:00
Shawn a896aa8b27 fix: incorrect replayq dir for the emqx_resource 2022-08-25 16:06:18 +08:00
Shawn 9327c0f51b fix(mysql_bridge): export the query_mode option to the APIs 2022-08-25 11:38:50 +08:00
JimMoen a481e8e2ee fix(influxdb): refine influxdb bridge example 2022-08-24 14:44:53 +08:00
JimMoen eb21a37145
Merge pull request #8785 from JimMoen/feat-influxdb-config 
Remove influxdb connector config layer. See #8773
 2022-08-24 10:56:26 +08:00
Shawn 4ac6f12252 refactor: configs and APIs for mysql bridge 2022-08-24 10:08:03 +08:00
JimMoen ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
Shawn aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
lafirest 4ad04b646f
Merge pull request #8759 from lafirest/feat/mysql_batch_query 
feat(bridge): add `on_batch_query` on emqx_connector_mysql
 2022-08-22 17:22:39 +08:00
firest 2471580c47 fix(bridge): make spellcheck happy 2022-08-22 15:58:11 +08:00
JimMoen 2648362c62 fix(bridge): password for bridge/db format as `password` for dashboard 2022-08-19 15:54:19 +08:00
JimMoen 06363e63d9 fix(influxdb): connector use a fallbacke `pool_size` for influxdb client 2022-08-19 15:54:19 +08:00
firest dce47aac17 fix(bridge): don't export internal fields 2022-08-19 15:31:37 +08:00
firest 8d8afd1688 feat(bridge): add `on_batch_query` on emqx_connector_mysql 2022-08-19 13:16:22 +08:00
Shawn de3a325953 fix: revert the changes in connector mysql 2022-08-16 09:06:13 +08:00
Thales Macedo Garitezi 7a31897f8e docs: add manual newlines 2022-08-15 16:55:53 -03:00
Thales Macedo Garitezi 34c1c61343
docs: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2022-08-15 16:37:50 -03:00
Thales Macedo Garitezi 900c6ec43e docs: add missing descriptions for license schema fields 2022-08-15 14:20:39 -03:00
Xinyu Liu 2898966439
Merge branch 'dev/ee5.0' into resource_opts 2022-08-15 21:43:22 +08:00
Shawn 19d85d485b refactor(resource): add resource_opts level into config structure 2022-08-15 21:40:10 +08:00
JimMoen d0e923590e fix: write influxdb line with undefined value 2022-08-15 14:02:17 +08:00
JimMoen 68946f1f6c feat: influxdb support `async`/`batch_async` query 2022-08-15 14:02:17 +08:00
JimMoen b01ae8ece6 chore: refine influxdb bridge/connector i18n 2022-08-15 14:00:14 +08:00
Shawn b3162fe5ff fix: conflicts between master and ee5.0 2022-08-14 22:24:09 +08:00
JimMoen 594d071c05 feat(influxdb): add async callback 2022-08-12 18:26:47 +08:00
JimMoen 441d8c9d57 chore: bump `influxdb-client-erl` vsn 2022-08-12 18:26:42 +08:00
JimMoen fa5e8f1422 chore: refine i18n label 2022-08-12 16:39:03 +08:00
lafirest 404e79b7d7
Merge pull request #8687 from lafirest/fix/bridge_mysql 
fix(bridge): replace prepare_statement by sql_template
 2022-08-12 10:12:34 +08:00
JimMoen 0c1285cdab fix: bridge creation opts refs 2022-08-12 10:07:54 +08:00
JimMoen db3e4f0b90 fix: flatten Points list 2022-08-11 20:58:56 +08:00
JimMoen 3a76a50382 fix: syntax error and compile error 2022-08-11 20:58:43 +08:00
Shawn 2872f0b668 fix(bridges): support create resources with options 2022-08-11 19:11:44 +08:00
JimMoen 0f6c371760 feat(influxdb): influxdb connector add `on_batch_query/3` callback 2022-08-11 18:12:41 +08:00
firest dc80499399 fix(bridge): add sql_template field format 2022-08-11 15:58:48 +08:00
firest 7581082fcb fix(bridge): replace prepare_statement by sql_template 2022-08-11 13:22:16 +08:00
JimMoen 223b84017e fix(influxdb): api schema `write_syntax` using raw type `string()` 2022-08-11 11:36:08 +08:00
JimMoen 0090a3ee93 fix(influxdb): fix illegal `wirte_syntax` example 2022-08-11 11:26:53 +08:00
Shawn 145ff66a9a fix: issues found by dialyzer and elvis 2022-08-10 00:45:26 +08:00
JimMoen 9d17bf31f8 fix: InfluxDB api v1 not support uint type 2022-08-09 14:28:54 +08:00
JimMoen d51ca2672a fix: spellcheck and update checker repo version 2022-08-09 11:50:05 +08:00
JimMoen 7305c3c04c fix(influxdb): parsed boolean type match 2022-08-08 18:22:46 +08:00
JimMoen 35e347aec8 feat: refine influxdb bridge conf 
Consistent influxdb line protocol config to raw syntax format.
See also
[InfluxDB 2.3 Line Protocol](https://docs.influxdata.com/influxdb/v2.3/reference/syntax/line-protocol/) and
[InfluxDB 1.8 Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/)
 2022-08-08 18:06:49 +08:00
JimMoen 06f246a896 fix: prevent unexcepted square brackets in influxdb line protocol 2022-08-08 10:45:18 +08:00
JimMoen 33a604dcaa fix: refine influxdb connector api type name 2022-08-07 19:37:44 +08:00
JimMoen 2e336fbc01 fix: bridge and connector namespace contains DB name 
Contains Detailed DB name for identifier hocon schema refs.
 2022-08-07 19:37:44 +08:00
JimMoen 4d0516c6e9 chore: use HStreamDB for module name 2022-08-07 19:37:44 +08:00
Thales Macedo Garitezi e76682d368 test: apply review comments 2022-08-04 15:27:45 -03:00
Thales Macedo Garitezi 68592373e3 test: add a couple more tests 2022-08-04 14:01:27 -03:00
Thales Macedo Garitezi 7e6b94fc12 test: avoid stopping cover and using meck 2022-08-04 10:23:49 -03:00
firest aa6077bdea fix(bridge): remove emqx_ee_connector_mysql 2022-08-04 14:52:16 +08:00
firest 88fd7e14dc feat(bridge): add mysql sink 2022-08-03 15:56:31 +08:00
Thales Macedo Garitezi 69ec76f586 chore(license): treat license file API as an upload (5.0) 
Improving the HTTP API for license after PR feedback.
 2022-08-02 17:25:33 -03:00
Thales Macedo Garitezi 889829e47d docs: use better translation 2022-08-02 09:31:59 -03:00
Thales Macedo Garitezi 1a66e53c49 chore(license): change api responses after review 2022-08-02 09:31:59 -03:00
Thales Macedo Garitezi b19e8fb3cd feat(license): add HTTP API for license 2022-08-02 09:31:59 -03:00
JimMoen 10b19acd35 chore: module and fun call rename 2022-08-02 16:59:56 +08:00
JimMoen d778a71d3b chore: unifined lib-ee name style 2022-08-02 16:47:09 +08:00
DDDHuang 4663e354bf fix: update influxdb sdk version to 1.1.3. Adapted master deps version 2022-08-01 10:34:57 +08:00
DDDHuang c1542e7a77 fix: influxdb connector bad encode message 2022-08-01 10:28:15 +08:00
DDDHuang 55b9684541 fix: influxdb bridge api available 2022-08-01 10:28:07 +08:00
DDDHuang fa54bf5612 fix: influxdb connector api available 2022-08-01 10:27:58 +08:00
DDDHuang 4c7ca2217c fix: add influxdb udp api_v1 api_v2 connector 2022-08-01 10:27:42 +08:00
DDDHuang 01e9e8a0eb feat: add ee influxdb connector & bridge 2022-08-01 10:27:35 +08:00
DDDHuang 3fe76446bc Merge branch 'master' into dev/ee5.0 2022-08-01 10:14:21 +08:00
Thales Macedo Garitezi 7c34011c12 chore: bump app vsns 2022-07-28 17:45:44 -03:00
Thales Macedo Garitezi f8a1bd0715 feat(license): copy license file to cluster when updating it 2022-07-28 17:29:59 -03:00
Thales Macedo Garitezi 80d35feb33 fix(license): change schema to allow updating license to different type 
Before this change, if a license in `emqx.conf` was of type
`key`/`file` and then changed to the opposite type, such change would
be saved to `{cluster,local}-overrides.conf`.  When the node restarts
after that, the configs are merged naively and becomes invalid, as it
contains both the `key` and `file` keys, and crashes.
 2022-07-28 17:29:59 -03:00
DDDHuang b7c245c5b0 fix: hstreamdb zh docs 2022-07-28 16:50:34 +08:00
DDDHuang a4992ef1b5 fix: hstreamdb connector conf & api 2022-07-28 16:47:30 +08:00
DDDHuang 9ae7c62656 fix: exs deps & bad suites 2022-07-27 16:24:13 +08:00
DDDHuang be6de4aad0 fix: ee bridge & connector remove app mod 2022-07-27 11:38:45 +08:00