40948f3c0b
fix(influxdb): authentication is not required for influxdb v1 api
2023-01-17 09:29:22 +08:00
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs
...
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
2023-01-16 13:37:29 +01:00
47414e0d53
docs: improve kafka key and value field description
2023-01-16 11:32:09 +01:00
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions
...
0112 docs fix some descriptions
2023-01-16 09:56:56 +01:00
91c5a89985
test: wait for redis connected state
...
the case is sometimes flaky because the health check sometimes
return connecting
2023-01-14 18:33:55 +01:00
e08c1d2229
Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor
2023-01-13 15:49:52 +01:00
7a17fb7308
test(emqx_ee_bridge): fix bridge enable/disable in kafka producer suite
2023-01-13 14:40:54 +01:00
f25bd288ad
Merge pull request #9742 from thalesmg/expose-resource-opts-mongo-v50
...
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
2023-01-13 10:23:49 -03:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
522e091dd2
docs: chane cache to buffer for Kafka bridge description
2023-01-13 07:46:23 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
ed2be4d086
feat(mongo): expose buffer worker opts to the bridge frontend (5.0)
2023-01-12 15:23:41 -03:00
1711823487
refactor: remove default value of timestamp field of influxdb
2023-01-12 22:25:44 +08:00
bae811e8b4
Merge pull request #9725 from terry-xiaoyu/remove_the_auto_reconnect_field
...
refactor: remove the auto_reconnect field
2023-01-12 11:11:00 +08:00
2d5208fc29
Merge pull request #9485 from thalesmg/refactor-doc-gen-50-mkIII
...
feat(docs): add tags to schemas
2023-01-11 16:36:38 -03:00
9d99d180f9
Merge pull request #9709 from thalesmg/mongodb-bridge-payload-template-v50
...
feat(mongodb): add `payload_template` field for bridge (e5.0)
2023-01-11 11:14:46 -03:00
3e9c4f444f
refactor: remove the auto_reconnect field
2023-01-11 21:47:06 +08:00
a9d0143d5f
chore(license): change `key` type to `binary()`
...
So that the `raw_default` field in the schema JSON renders properly.
2023-01-11 09:10:04 -03:00
48e1ba4832
feat(docs): add tags to schemas
...
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
2023-01-11 09:10:03 -03:00
f3cc722172
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-01-10 09:06:12 -03:00
67f2159a27
Merge pull request #9653 from zmstone/0101-authz-schema-union-member-selection
...
0101 authz schema union member selection
2023-01-09 22:17:51 +01:00
b9f258b737
feat(mongodb): add `payload_template` field for bridge (e5.0)
...
https://emqx.atlassian.net/browse/EMQX-8705
Adds a `payload_template` fields that allows users to customize the
payload to publish to MongoDB.
2023-01-09 13:38:11 -03:00
20f49b903f
Merge pull request #9702 from lafirest/feat/timescale_matrix
...
feat(bridges): add timescale && matrix bridges
2023-01-10 00:06:17 +08:00
062f14bd65
test(bridges): add timescale && matrix test cases
2023-01-09 23:04:36 +08:00
e08d6dbc9b
feat(bridges): add timescale && matrix bridges
2023-01-09 23:04:29 +08:00
c3635f537a
ci: wait for redis in emqx_ee_bridge_redis_SUITE
2023-01-09 14:39:56 +01:00
0697c692ed
refactor: mongo_type and redis_type are not mandatory
...
the connector schemas are shared between authn, auth and bridges,
the difference is that authn and authz configs are unions like:
[
{mongo_type = rs,
...
}
{another backend config}
]
however the brdige types are maps like, for example:
mongodb_rs.$name {
mongo_type = rs
...
}
in which case, the mongo_type is not required.
in order to keep the schema static as much as possible,
this field is chanegd to 'required => false' with a default value.
However, for authn and authz, the union selector will still raise
exception if the there is no type provided.
2023-01-09 14:26:16 +01:00
ffb09f0c4d
test: rename a test option name to avoid clashing with prod config name
2023-01-09 14:26:16 +01:00
e52f9d5920
refactor: use union member type selector for authz sources
2023-01-09 14:26:16 +01:00
e498010f23
chore: update default license (100 conns)
2023-01-09 09:11:09 +01:00
3859878985
Merge pull request #9684 from zmstone/0105-do-not-use-testcase-skip
...
test: do not use tc_user_skip for test cases
2023-01-09 09:02:06 +01:00
ea405fe55d
test(bridges): add test case for the PostgreSQL backend
2023-01-06 11:56:28 +08:00
f7b50c56da
feat(bridges): integrate PostgreSQL into bridges
...
support both simple and batch query
2023-01-06 11:47:22 +08:00
5f12cdff6c
ci: should not skip tests in github actions
2023-01-05 22:47:08 +01:00
fc12a8c4c8
test: do not use tc_user_skip for test cases
2023-01-05 22:47:06 +01:00
af31ed4264
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-01-05 10:16:00 -03:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
0fd8880d0a
fix(kafka_producer): avoid multiplication of metrics when bridge is recreated
2023-01-05 10:11:59 -03:00
8e59319bfe
fix(kafka_producer): fix message loss when kafka connection is down
2023-01-05 10:11:59 -03:00
5bd9f110d6
test: attempt to reduce flakiness
2023-01-05 10:11:59 -03:00
3304025a9a
Merge pull request #9677 from thalesmg/test-other-gcp-flaky-v50
...
test: attempt to fix flaky test
2023-01-04 13:30:53 -03:00
5df485df17
test: attempt to fix flaky test
2023-01-04 10:38:21 -03:00
4feaf5916f
fix(mongodb): add `type` and `name` fields to the `get`/`post` API docs
...
https://emqx.atlassian.net/browse/EMQX-8643
Adds thoses fields to MongoDB API docs so the frontend may use them to
render properly.
2023-01-03 11:46:58 -03:00
d6c8a106da
test(gcp_pubsub): fix flaky test
2023-01-03 08:52:37 -03:00
7985cd3536
Merge pull request #9638 from lafirest/fix/mysql_dup
...
fix(mysql): fix the problem of data loss and bad match when mysql is disconnected
2023-01-02 23:15:44 +08:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
c77717b1f1
test(mysql): fix test case error
2023-01-02 21:55:22 +08:00
69ac6b9e0d
fix(kafka): fix handling of `dropped.queue_full` event from wolff
...
https://emqx.atlassian.net/browse/EMQX-8530
cd20a37e65/src/wolff_producer.erl (L772-L773)
2023-01-02 09:23:38 -03:00
fc6d9d0890
chore: issue new Enterprise evaluation license
2023-01-02 11:50:57 +01:00
b5d2fc0ede
chore: update BSL license year
2023-01-02 11:23:14 +01:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
231d6da4dc
Merge pull request #9644 from id/fix-kafka-bridge-connectivity-detection
...
fix(kafka): detect connectivity in on_get_status
2022-12-30 21:54:56 +01:00
61246c43c4
fix(kakfa_producer): prevent multiple producers from multiplying each other's metrics
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
0ce1ca89b7
refactor: use string type for server and servers
2022-12-30 14:20:23 +01:00
a612eacf3a
Merge pull request #9641 from thalesmg/fix-gcp-pubsub-jwt-refresh-v50
...
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
2022-12-30 09:33:41 -03:00
75770f2842
fix(kafka): detect connectivity in on_get_status
2022-12-30 10:18:52 +01:00
24bae2641e
chore: upgrade wolff -> 1.7.4
2022-12-29 16:58:00 -03:00
446a4c74d0
fix(gcp_pubsub): fix potential jwt accumulation and lack of refresh (v5.0)
...
https://emqx.atlassian.net/browse/EMQX-8653
Related:
- https://emqx.atlassian.net/browse/EEC-737
- https://emqx.atlassian.net/browse/EMQX-8652
Since the rule resource testing mechanism creates a new resource to
test the configuration, a new JWT associated with an unique temporary
resource was being created and left in the JWT table, leaking it.
Also, a wrong case clause when setting the new refresh timer for the
JWT worker was preventing it from refreshing from the 2nd refresh
onward.
2022-12-29 16:30:36 -03:00
728cb85156
Merge pull request #9629 from zmstone/1128-fix-flaky-test-listener-restart-eaddrinuse
...
test: add a 2 seconds sleep between listener sotp and restart
2022-12-29 16:04:19 +01:00
5ae4bd6722
chore: bump influxdb client lib version from 1.1.4 to 1.1.5
...
fixed a compile warning
2022-12-29 10:29:36 +01:00
a5e3f6f05b
fix(kafka): Memory OLP is only applicable in linux systems
2022-12-28 15:31:53 +01:00
ce43e6b3d6
chore: upgrade kafka_protocol, wolff, brod
2022-12-27 10:30:14 -03:00
4819794401
test(refactor): decrease test teardown noise
2022-12-27 10:30:14 -03:00
a78ecc4bb6
Merge pull request #9579 from thalesmg/bugfix-kafka-producer-stop-client-ee50
...
fix(kafka_producer): cleanup client after failing to start producers
2022-12-26 08:59:36 -03:00
59970272ff
chore: pin hocon 0.33.0
2022-12-23 12:07:36 +01:00
13942f5c49
refactor: rename error return in mysql connector
2022-12-22 10:29:12 +01:00
aab914d65a
test: refactor EE mysql bridge test case
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2022-12-22 10:27:38 +01:00
7df24000a0
test: add more EE mysql bridge test cases
2022-12-22 10:27:38 +01:00
7242ce426b
fix(kafka_producer): cleanup client after failing to start producers
...
https://emqx.atlassian.net/browse/EMQX-8547
If a Kafka Producer bridge is given bad configuration (e.g.: bad authn
credentials), the Wolff client process is started successfully, as it
does not attempt to connect, but when the producer process is
attempted to be started, it fails (only then the client tries to
connect to Kafka). At this point, an error was thrown, but the
supervised client process remained running.
If the configuration was later fixed and the bridge updated, which
prompted its removal and recreation, the Wolff client would report to
be "already started", so it would never pick up the new (fixed)
configuration, and the producers would perpetually fail to start until
the node would be restarted.
We simply ensure the client is stopped before throwing the error,
unrolling the start-up procedure.
2022-12-19 17:48:40 -03:00
c0b208dd9e
fix(influxdb): check if fields are empty before sending
...
Related Issue: https://emqx.atlassian.net/browse/EMQX-8461
Currently, the InfluxDB client raises an error if an empty `fields`
map is passed to it for pushing data.
```
14:03:35.563 [error] [InfluxDB] Encode [
%{
fields: %{},
measurement: "t/topic",
tags: %{},
timestamp: 1670864615563
}
] failed: :error :missing_field [
{:influxdb_line, :encode_fields, 1,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 60
]},
{:influxdb_line, :encode_, 1,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 44
]},
{:influxdb_line, :"-encode_/1-fun-0-", 2,
[
file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl',
line: 38
]},
{:influxdb, :write, 2,
[file: '/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl', line: 79]},
{:emqx_ee_connector_influxdb, :do_query, 3,
[
file: '/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl',
line: 322
]},
{:emqx_resource_worker, :apply_query_fun, 6,
[
file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
line: 454
]},
{:emqx_resource_worker, :query_or_acc, 3,
[
file: '/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl',
line: 306
]},
{:gen_statem, :loop_state_callback, 11, [file: 'gen_statem.erl', line: 1203]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}
]
2022-12-12T14:03:35.563607-03:00 [error] [InfluxDB] Encode [#{fields => #{},measurement => <<"t/topic">>,tags => #{},timestamp => 1670864615563}] failed: error missing_field [{influxdb_line,encode_fields,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,60}]},{influxdb_line,encode_,1,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,44}]},{influxdb_line,'-encode_/1-fun-0-',2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb_line.erl"},{line,38}]},{influxdb,write,2,[{file,"/home/thales/dev/emqx/emqx/deps/influxdb/src/influxdb.erl"},{line,79}]},{emqx_ee_connector_influxdb,do_query,3,[{file,"/home/thales/dev/emqx/emqx/lib-ee/emqx_ee_connector/src/emqx_ee_connector_influxdb.erl"},{line,322}]},{emqx_resource_worker,apply_query_fun,6,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,454}]},{emqx_resource_worker,query_or_acc,3,[{file,"/home/thales/dev/emqx/emqx/apps/emqx_resource/src/emqx_resource_worker.erl"},{line,306}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1203}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]
14:03:35.565 [error] [connector: "bridge:influxdb_api_v2:aaa", msg: 'influxdb write point failed', reason: :missing_field]
2022-12-12T14:03:35.565345-03:00 [error] connector: <<"bridge:influxdb_api_v2:aaa">>, line: 335, mfa: emqx_ee_connector_influxdb:do_query/3, msg: influxdb write point failed, reason: missing_field
[]
14:03:35.565 [error] [id: "bridge:influxdb_api_v2:aaa", msg: :send_error, reason: :missing_field]
iex(emqx@127.0.0.1)2> 2022-12-12T14:03:35.565913-03:00 [error] id: <<"bridge:influxdb_api_v2:aaa">>, line: 396, mfa: emqx_resource_worker:handle_query_result/4, msg: send_error, reason: missing_field
```
Instead of raising, we check if the interpolation resulted in an empty
map due to lack of context and just fail the push more gracefully.
Related to this, the original issue _appears_ to be related to a
frontend issue (to be confirmed and fixed separately), where the
it is not encoding the field types:
https://emqx.atlassian.net/browse/EMQX-8461?focusedCommentId=24805
2022-12-16 09:52:34 -03:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
56066a03b5
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 20:04:20 +01:00
34f7fff97d
chore: pin hocon 0.32.0
2022-12-14 15:53:42 +01:00
42c58e2a91
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 15:29:13 +01:00
2932aceaf9
refactor: rename and move `emqx_ee_bridge_gcp_pubsub_resource` to `emqx_ee_connector_gcp_pubsub`
2022-12-12 17:18:19 -03:00
464d0a5057
refactor(test): use a linked janitor for test teardown
2022-12-12 17:18:19 -03:00
b66d2904be
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2022-12-12 17:18:19 -03:00
b9bc82f87a
feat(gcp_pubsub): add `local_topic` config
...
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
2022-12-12 17:18:19 -03:00
ac048dbafa
docs(gcp_pubsub): describe where to get credentials
2022-12-12 17:18:19 -03:00
a095867358
test(refactor): add ids to namespace `on_exit` callbacks
2022-12-12 17:18:19 -03:00
697b3ecf2d
docs: improve descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2022-12-12 17:18:19 -03:00
1cd91a24e9
feat(gcp_pubsub): implement GCP PubSub bridge (ee5.0)
2022-12-12 17:18:19 -03:00
82be9d878d
test(flaky): avoid inter-suite flakiness
2022-12-12 17:18:19 -03:00
c69022f3c4
test(ci): clean config after test
...
Trying to fix this error for profile `emqx`:
```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
crasher:
initial call: application_master:init/4
pid: <0.9682.1>
registered_name: []
exception exit: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
in function application_master:init/4 (application_master.erl, line 142)
ancestors: [<0.9681.1>]
message_queue_len: 1
messages: [{'EXIT',<0.9683.1>,normal}]
links: [<0.9681.1>,<0.44.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 29
reductions: 195
neighbours:
=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
application: emqx_conf
exited: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
type: temporary
%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}
%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
{failed,
{emqx_plugins_SUITE,init_per_suite,
{'EXIT',
{{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,
path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1224}]}]}}}}}
```
2022-12-12 17:18:13 -03:00
34e9056779
refactor: fix typo in variable name
...
Might confuse people to think it's related to `replayq`.
2022-12-12 17:17:51 -03:00
40809b2ad0
Merge remote-tracking branch 'origin/dev/ee5.0' into release-50
2022-12-09 11:45:52 +01:00
dc14cd450d
test: render config for emqx_conf only for ee bridge tests
2022-12-07 18:27:06 +01:00
6692b0c895
feat(bridge): add Redis bridge
2022-12-06 23:15:42 +03:00
485c4ddcab
Merge pull request #9393 from olcai/add-mysql-bridge-test
...
test: add basic mysql bridge test (ee5.0)
2022-12-02 09:43:22 +01:00
9ea22d062d
refactor: make all bridges optional (required = false)
2022-12-01 16:50:10 +01:00
7adb539ae1
chore: pin hocon 0.31.2
2022-12-01 16:50:10 +01:00
499a32ce36
test: remove unnecessary async tests in EE mysql bridge
...
The async test cases is not needed since the mysql connector uses the
always_sync callback mode.
2022-12-01 16:38:03 +01:00
5461311edc
test: more EE mysql bridge tests
2022-12-01 10:27:51 +01:00
eb62192838
test: expand EE mysql bridge test with toxiproxy
2022-12-01 10:27:51 +01:00
96bff1d32e
test: improve basic tests for EE mysql bridge
2022-12-01 10:27:51 +01:00
3e679ceb57
test: add basic tests for EE mysql bridge
2022-12-01 10:27:51 +01:00
fa90638de2
chore: add description to apps
2022-11-30 17:03:41 +01:00
9382399728
chore: add an empty file emqx_license.conf to make ci happy
2022-11-30 12:52:26 +01:00
83fe35b148
Merge remote-tracking branch 'origin/dev/ee5.0' into 1116-sync-v5.0.10-to-ee50
2022-11-18 19:23:31 +01:00
7305923d12
fix: bridge name parser should not leak atom
2022-11-16 18:25:42 +01:00
c940b901f5
chore: fix app versions
2022-11-16 16:26:43 +01:00
0f9cc0d93f
test(refactor): stop snabbkaffe on every test
2022-11-11 10:03:30 -03:00
04588148b7
test(influxdb): increase influxdb bridge/connector coverage (ee5.0)
2022-11-07 15:15:49 -03:00
ee4c723fcb
refactor: simplify wolff telemetry handler id
2022-10-17 16:26:04 -03:00
1b2b629cdd
feat: emit telemetry events for all resource worker metrics
2022-10-13 15:32:04 -03:00
1ad3b5df17
fix: uninstall telemetry handler on resource stop, use unique id
2022-10-13 10:29:53 -03:00
4475289ce4
feat: use upstream newly tagged 1.7.0 wolff
2022-10-11 09:47:40 -03:00
98500313eb
fix(kafka): some fixes for kafka producer
...
- MQTT topic should be a binary
- use correct gauge functions from `wolff_metrics`.
- don't double increment success counter for kafka action
- adds a few more metrics assertions
2022-10-10 17:11:29 -03:00
57270fb8fc
feat: add support for counters and gauges to the Kafka Bridge
...
This commit adds support for counters and gauges to the Kafka Brige.
The Kafka bridge uses [Wolff](https://github.com/kafka4beam/wolff ) for
the Kafka connection. Wolff does its own batching and does not use the
batching functionality in `emqx_resource_worker` that is used by other
bridge types. Therefore, the counter events have to be generated by
Wolff. We have added
[telemetry](https://github.com/beam-telemetry/telemetry ) events to Wolff
that we hook into to change counters and gauges for the Kafka bridge. The
counter called `matched` does not depend on specific functionality of
any bridge type so the updates of this counter is moved higher up in the
call chain then previously so that it also gets updated for Kafka
bridges.
2022-10-10 14:40:57 -03:00
dfe14be8b2
chore: bump app vsns
2022-09-26 18:23:28 +08:00
4135910b42
chore: merge master into dev/ee5.0
2022-09-26 09:52:33 +08:00
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
...
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
a3c88b40a0
test: changes to make Kafka container run in GitHub action
2022-09-23 14:33:41 +02:00
8e514680d8
test: fix bad binary pattern
2022-09-23 13:54:11 +02:00
7b601bf970
chore: delete bad parse_bridge function clause
2022-09-23 11:53:02 +02:00
adc67b165b
test: test cases for Kafka bridge REST API
2022-09-23 10:09:07 +02:00
6788881642
fix: hide influxdb udp config
2022-09-20 14:46:59 +08:00
03495d8d36
fix: influxdb server string support scheme prefix
2022-09-20 14:46:59 +08:00
ac2922fc4c
test: Kafka bridge cases for all combinations of SASL and SSL
2022-09-16 16:44:12 +02:00
0336b935a0
Merge pull request #8901 from JimMoen/fix-bridge-api
...
- for connectors using ecpool, parse connect error `Reason` when start pool failed.
- Refine InfluxDB bridge desc.
2022-09-16 10:22:06 +08:00
be7a8c11a8
test: make bridge name unique in tests
2022-09-15 16:21:32 +02:00
4dc26eeba7
fix: use different instance id in Kafka auth test
2022-09-15 07:33:07 +02:00
54a9c8d201
chore: refine influxdb bridge description
2022-09-15 09:36:52 +08:00
deff429579
chore: unify default license references in fn
2022-09-14 17:26:48 -03:00
741eb4ff51
docs: apply suggestions from code review
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2022-09-14 17:08:45 -03:00
21f89fd83e
chore(license): delete default config and add info to i18n
2022-09-14 17:00:21 -03:00
5820b028cb
feat: add test case for Kerberos Kafka authentication
2022-09-14 17:03:37 +02:00
8cefdb00b8
chore(license): add default key as default value in schema
2022-09-14 09:45:01 -03:00
d4416dc581
chore: drop support for license files (ee5.0)
...
Related issue: EMQX-7247
EMQX ee5.0 is to drop support for both the old license format (x509)
and for files containing the new format.
2022-09-14 09:44:59 -03:00
f41adb0997
refactor: change some default values of resource_opts
2022-09-14 15:18:07 +08:00
e45c99bf79
fix: kafka bridge schema
2022-09-13 20:02:48 +02:00
f0e03086a6
test: add test cases for Kafka SASL auth mechanisms plain and scram
2022-09-13 19:46:56 +02:00
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
befc4acced
Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a
2022-09-06 20:31:38 +02:00
23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge
...
feat: add mongodb bridge (e5.0)
2022-09-03 12:27:01 +08:00
7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources
...
Some improvements and bug fixes to emqx_resources
2022-09-03 12:16:49 +08:00
f1048babd8
test: refactor to use hocon and schema
2022-09-02 12:04:37 -03:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
3d4afd65df
feat: add mongodb bridge (e5.0)
2022-09-01 14:47:14 -03:00
5a3fb4a011
Merge remote-tracking branch 'origin/master' into dev/ee5.0
2022-08-31 17:11:15 +02:00
ba1f5eecd3
fix: update the swagger for new resource metrics
2022-08-31 11:14:36 +08:00
e0a6a61d73
fix: return error on start hstreamdb crash
2022-08-30 10:19:40 +08:00
2eb078ff3b
chore: bump changed app versions
2022-08-26 14:32:28 +02:00
6b0ccfbc43
refactor: rename the error return resource_down -> recoverable_error
2022-08-26 17:11:12 +08:00
a896aa8b27
fix: incorrect replayq dir for the emqx_resource
2022-08-25 16:06:18 +08:00
9327c0f51b
fix(mysql_bridge): export the query_mode option to the APIs
2022-08-25 11:38:50 +08:00
a481e8e2ee
fix(influxdb): refine influxdb bridge example
2022-08-24 14:44:53 +08:00
eb21a37145
Merge pull request #8785 from JimMoen/feat-influxdb-config
...
Remove influxdb connector config layer. See #8773
2022-08-24 10:56:26 +08:00
4ac6f12252
refactor: configs and APIs for mysql bridge
2022-08-24 10:08:03 +08:00
ca6533395c
feat: influxdb bridge structure fits new style
2022-08-24 09:59:22 +08:00
aea8c77b49
refactor: new config structure for mqtt bridge
2022-08-22 18:24:59 +08:00
4ad04b646f
Merge pull request #8759 from lafirest/feat/mysql_batch_query
...
feat(bridge): add `on_batch_query` on emqx_connector_mysql
2022-08-22 17:22:39 +08:00
2471580c47
fix(bridge): make spellcheck happy
2022-08-22 15:58:11 +08:00
2648362c62
fix(bridge): password for bridge/db format as `password` for dashboard
2022-08-19 15:54:19 +08:00
06363e63d9
fix(influxdb): connector use a fallbacke `pool_size` for influxdb client
2022-08-19 15:54:19 +08:00
dce47aac17
fix(bridge): don't export internal fields
2022-08-19 15:31:37 +08:00
8d8afd1688
feat(bridge): add `on_batch_query` on emqx_connector_mysql
2022-08-19 13:16:22 +08:00
de3a325953
fix: revert the changes in connector mysql
2022-08-16 09:06:13 +08:00
7a31897f8e
docs: add manual newlines
2022-08-15 16:55:53 -03:00
34c1c61343
docs: apply suggestions from code review
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2022-08-15 16:37:50 -03:00
900c6ec43e
docs: add missing descriptions for license schema fields
2022-08-15 14:20:39 -03:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
d0e923590e
fix: write influxdb line with undefined value
2022-08-15 14:02:17 +08:00
68946f1f6c
feat: influxdb support `async`/`batch_async` query
2022-08-15 14:02:17 +08:00
b01ae8ece6
chore: refine influxdb bridge/connector i18n
2022-08-15 14:00:14 +08:00
b3162fe5ff
fix: conflicts between master and ee5.0
2022-08-14 22:24:09 +08:00
594d071c05
feat(influxdb): add async callback
2022-08-12 18:26:47 +08:00
441d8c9d57
chore: bump `influxdb-client-erl` vsn
2022-08-12 18:26:42 +08:00
fa5e8f1422
chore: refine i18n label
2022-08-12 16:39:03 +08:00
404e79b7d7
Merge pull request #8687 from lafirest/fix/bridge_mysql
...
fix(bridge): replace prepare_statement by sql_template
2022-08-12 10:12:34 +08:00
0c1285cdab
fix: bridge creation opts refs
2022-08-12 10:07:54 +08:00
db3e4f0b90
fix: flatten Points list
2022-08-11 20:58:56 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
0f6c371760
feat(influxdb): influxdb connector add `on_batch_query/3` callback
2022-08-11 18:12:41 +08:00
dc80499399
fix(bridge): add sql_template field format
2022-08-11 15:58:48 +08:00
7581082fcb
fix(bridge): replace prepare_statement by sql_template
2022-08-11 13:22:16 +08:00
223b84017e
fix(influxdb): api schema `write_syntax` using raw type `string()`
2022-08-11 11:36:08 +08:00
0090a3ee93
fix(influxdb): fix illegal `wirte_syntax` example
2022-08-11 11:26:53 +08:00
145ff66a9a
fix: issues found by dialyzer and elvis
2022-08-10 00:45:26 +08:00
9d17bf31f8
fix: InfluxDB api v1 not support uint type
2022-08-09 14:28:54 +08:00
d51ca2672a
fix: spellcheck and update checker repo version
2022-08-09 11:50:05 +08:00
7305c3c04c
fix(influxdb): parsed boolean type match
2022-08-08 18:22:46 +08:00
35e347aec8
feat: refine influxdb bridge conf
...
Consistent influxdb line protocol config to raw syntax format.
See also
[InfluxDB 2.3 Line Protocol](https://docs.influxdata.com/influxdb/v2.3/reference/syntax/line-protocol/ ) and
[InfluxDB 1.8 Line Protocol](https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/ )
2022-08-08 18:06:49 +08:00
06f246a896
fix: prevent unexcepted square brackets in influxdb line protocol
2022-08-08 10:45:18 +08:00
33a604dcaa
fix: refine influxdb connector api type name
2022-08-07 19:37:44 +08:00
2e336fbc01
fix: bridge and connector namespace contains DB name
...
Contains Detailed DB name for identifier hocon schema refs.
2022-08-07 19:37:44 +08:00
4d0516c6e9
chore: use HStreamDB for module name
2022-08-07 19:37:44 +08:00
e76682d368
test: apply review comments
2022-08-04 15:27:45 -03:00
68592373e3
test: add a couple more tests
2022-08-04 14:01:27 -03:00
7e6b94fc12
test: avoid stopping cover and using meck
2022-08-04 10:23:49 -03:00
aa6077bdea
fix(bridge): remove emqx_ee_connector_mysql
2022-08-04 14:52:16 +08:00
88fd7e14dc
feat(bridge): add mysql sink
2022-08-03 15:56:31 +08:00
69ec76f586
chore(license): treat license file API as an upload (5.0)
...
Improving the HTTP API for license after PR feedback.
2022-08-02 17:25:33 -03:00
889829e47d
docs: use better translation
2022-08-02 09:31:59 -03:00
1a66e53c49
chore(license): change api responses after review
2022-08-02 09:31:59 -03:00
b19e8fb3cd
feat(license): add HTTP API for license
2022-08-02 09:31:59 -03:00
10b19acd35
chore: module and fun call rename
2022-08-02 16:59:56 +08:00
d778a71d3b
chore: unifined lib-ee name style
2022-08-02 16:47:09 +08:00
4663e354bf
fix: update influxdb sdk version to 1.1.3. Adapted master deps version
2022-08-01 10:34:57 +08:00
c1542e7a77
fix: influxdb connector bad encode message
2022-08-01 10:28:15 +08:00
55b9684541
fix: influxdb bridge api available
2022-08-01 10:28:07 +08:00
fa54bf5612
fix: influxdb connector api available
2022-08-01 10:27:58 +08:00
4c7ca2217c
fix: add influxdb udp api_v1 api_v2 connector
2022-08-01 10:27:42 +08:00
01e9e8a0eb
feat: add ee influxdb connector & bridge
2022-08-01 10:27:35 +08:00
3fe76446bc
Merge branch 'master' into dev/ee5.0
2022-08-01 10:14:21 +08:00
7c34011c12
chore: bump app vsns
2022-07-28 17:45:44 -03:00
f8a1bd0715
feat(license): copy license file to cluster when updating it
2022-07-28 17:29:59 -03:00
80d35feb33
fix(license): change schema to allow updating license to different type
...
Before this change, if a license in `emqx.conf` was of type
`key`/`file` and then changed to the opposite type, such change would
be saved to `{cluster,local}-overrides.conf`. When the node restarts
after that, the configs are merged naively and becomes invalid, as it
contains both the `key` and `file` keys, and crashes.
2022-07-28 17:29:59 -03:00
b7c245c5b0
fix: hstreamdb zh docs
2022-07-28 16:50:34 +08:00
a4992ef1b5
fix: hstreamdb connector conf & api
2022-07-28 16:47:30 +08:00
9ae7c62656
fix: exs deps & bad suites
2022-07-27 16:24:13 +08:00
be6de4aad0
fix: ee bridge & connector remove app mod
2022-07-27 11:38:45 +08:00
fa4bc921ac
fix: hstream db connector & bridge, TODO: SUITE
2022-07-27 11:38:45 +08:00
98b36c4681
fix: hstream db connector , TODO: start apps
2022-07-27 11:38:45 +08:00
f00a7417bb
feat: hstream bridge & connector
2022-07-27 11:38:45 +08:00
e4eba157c3
feat: add emqx enterprise app connector & bridge
2022-07-25 09:36:45 +08:00
92019e4d4c
chore: don't copy i18n.conf
2022-07-01 09:33:35 +08:00
48f685a0dd
fix: remove the APIs for emqx_hooks:put/2 and emqx_hooks:add/2
2022-06-16 00:43:50 +08:00
c355c40ea8
refactor: call emqx_alarm:ensure_deactivated everywhere
2022-05-13 16:02:55 +02:00
b4b9853cfc
Merge pull request #7732 from terry-xiaoyu/rule_bridges_api_docs2
...
emqx_schema i18n docs
2022-04-24 14:05:48 +08:00
a77bb994e1
fix: add timeout for schema json gen test
2022-04-24 12:05:07 +08:00
c32fc33c1a
style: cosmetic fix for code style
2022-04-23 09:48:58 +02:00
1c74c83f5e
fix(license): print warning at start
2022-04-22 17:44:18 +02:00
123e456287
feat: change default trial license
...
* Changed number of connections limit from 10 to 1000
* Added a 'deployment' field, so we know which is which when
a customer has multiple licenses
* Default expiration is 5 years
2022-04-21 08:48:37 +02:00
164754b182
test: dashboard ct failed with lack i18n_lang
2022-04-15 12:04:01 +08:00
d12229700a
feat(license): port 4.4 parser to 5.0
2022-04-11 09:25:06 -03:00
43aa2ea481
docs(schema): Add documentation for license application
2022-04-05 10:32:24 +02:00
4e3fd9febd
style(license): reformat emqx_license application
2022-03-31 17:41:43 +08:00
93cb86d3d2
chore: make Elvis happy
2022-03-25 20:06:11 +01:00
1aa8299261
style: erlfmt lib-ee/emqx_enterprise_conf
2022-03-25 16:54:23 +01:00
793add0219
style: use a macro for 'catch'
2022-03-25 08:21:50 +01:00
7923f017be
style(elvis): avoid using if
2022-03-25 07:25:02 +01:00
4f396cceb8
style(emqx_license): erlfmt refomrat apps/emqx_license
2022-03-25 07:13:08 +01:00
4e64010a3b
build: more structured build info
2022-03-02 21:22:03 +01:00
d029a73b99
refactor: make release edition static info at compile time
2022-02-28 16:37:47 +01:00
4a336b6b4e
test: add eunit test for schema dump
2022-02-28 10:47:10 +01:00
2a6d11c7f1
Merge pull request #7038 from zmstone/chore-rename-emqx-no-space-source-code
...
chore: EMQ X -> EMQX in source code
2022-02-17 21:13:43 +01:00
c88504f918
Merge pull request #7023 from zhongwencool/license-alarm-support
...
License alarm support
2022-02-17 22:09:51 +08:00
001b209811
fix(license): don't do not lose watermark fields.
2022-02-17 15:49:41 +08:00
7853eb75df
fix(license): fix crash update license.watermark.
2022-02-17 15:05:22 +08:00
64db085163
chore: EMQ X -> EMQX in source code
2022-02-16 22:08:12 +01:00
0528262705
chore(app.src): EMQ X -> EMQX in .app.src files
2022-02-16 17:51:45 +01:00
ea1bf58a2d
Merge pull request #7033 from zmstone/feat-emqx-license-allow-comma-in-numbers
...
feat(emqx_license): allow comma in integer numbers in license body
2022-02-16 17:31:34 +01:00
fa9300e875
feat(emqx_license): allow comma in integer numbers in license body
2022-02-16 15:37:24 +01:00
d900418ce2
fix(license): fix maps:get/2 crash
2022-02-16 18:15:47 +08:00
98b2cd683b
fix: dialyzer warning
2022-02-16 17:41:51 +08:00
34fe5082c4
fix(schema): schema global validations not working
2022-02-16 14:20:15 +08:00
ea558b2bc5
feat(license): license expriy early alarm.
2022-02-16 12:31:25 +08:00
2e94880bb6
docs: More EMQ X -> EMQX rename in descriptive strings
2022-02-15 22:39:11 +01:00
667d66eb1e
docs(README): EMQ X -> EMQX
2022-02-15 16:19:26 +01:00
bc67ede384
chore: update to use official pub key for 5.0 license
2022-02-15 14:22:14 +01:00
7494c47919
test: change to use ec keys for testing
2022-02-15 13:51:56 +01:00
c8c7ad896e
refactor(emqx_license): test with pem format input
...
no need to perform der encode/decode for pub key
2022-02-15 10:39:30 +01:00
0f09e7bbd2
test(emqx_license): no need to meck public key for tests
2022-02-15 00:39:10 +01:00
b7aac96282
Merge pull request #6975 from savonarola/emqx-license-early-check
...
feature(license): check license before node start
2022-02-14 15:02:48 +03:00
64c59b5469
feat(license): check license before node start
2022-02-11 11:49:29 +03:00
fc8eeb8819
fix(emqx_license): raise throw exception instead of error
2022-02-10 12:40:46 +01:00
cd3565d357
refactor(emqx_license): simplify parser errors
...
prior to this change, badmatch exception may cause the license key
to be dumped in logs
2022-02-10 12:40:46 +01:00
4cee12614b
fix: mark license key sensitive
...
so it's not logged to console when environment variable is used
2022-02-08 17:28:40 +01:00
ae9c8098ba
test: ensure coverage for the `no_license` case
2022-02-08 17:28:40 +01:00
c936706a1f
refactor(emqx_license): log a different msg when license is not loaded
2022-02-08 17:28:40 +01:00
319d08678e
chore: update license year
2022-02-08 17:28:40 +01:00
17599432d1
feat(license): add license application
2022-02-07 19:05:47 +03:00
7f49934216
chore: add BSL.txt
2021-12-29 13:35:49 +01:00
6edd862dd5
refactor: no more EMQX_ENTERPRISE compile flag
...
The compile flag was introduced in EQM X 4.3 series
where CE and EE code was diverged large enough which made
non-practicle to determin edition at runtime.
such approach made testing quite challenging as we'll have to
build with different compile flags inorder to run per-edition
test cases
In this commit, we try to retrieve edition info from EMQX's
description text, (put to PT for fast access) at runtime
so we can test ALL editions from a super-set edition (EE).
2021-11-20 20:41:06 +01:00
JimMoen