c3133fb6a2
fix(emqx_bridge): small fixes from review
2023-01-13 14:54:54 +01:00
8dd52e5a6e
chore: add translations to schemas
2023-01-13 14:19:23 +01:00
f1c58c34ed
test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE
2023-01-13 14:19:23 +01:00
f15b29b1ef
chore: upgrade app version
2023-01-13 18:22:29 +08:00
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge
2023-01-13 18:12:53 +08:00
42f42de4d9
feat(emqx_bridge): add separate endpoint for enable/disable of bridge
...
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
2023-01-12 19:39:43 +01:00
860e21d40f
feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics
...
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
2023-01-12 19:39:41 +01:00
c2fd1a4482
feat(emqx_bridge): shorten operation endpoint URLs
...
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
2023-01-12 19:36:22 +01:00
c5f557e315
fix: disable basic auth for HTTP API
2023-01-12 21:35:49 +08:00
8ad8288195
feat: report error in create_dry_run
2023-01-11 14:22:37 +01:00
48e1ba4832
feat(docs): add tags to schemas
...
This'll allow us to split the generated `schema.json` file into
subsections for better documentation navigation.
2023-01-11 09:10:03 -03:00
6f0e228a9b
Merge pull request #9717 from zmstone/0110-do-not-start-resource-after-recreate-if-disabled
...
fix(emqx_resource_manager): do not start when disabled
2023-01-11 16:12:53 +08:00
85a8eff90b
fix(emqx_resource_manager): do not start when disabled
2023-01-11 08:33:48 +01:00
ee89de816c
fix: upgrade bridge config converter
2023-01-10 20:42:30 +01:00
0f70786d13
Merge pull request #9705 from terry-xiaoyu/remove-default-value-of-webhook-body
...
fix: remove the default value of webhook body field
2023-01-10 18:23:50 +08:00
724015af47
fix: remove the default value of webhook body field
2023-01-10 09:45:25 +08:00
692c1f0d1b
Merge pull request #9678 from kjellwinblad/kjell/fix/jira/EMQX-8648
...
fix: no feedback when deleting non-existing configuration
2023-01-09 13:52:40 +01:00
c85a988a43
fix: split 'server' param into host and port for mqtt bridge
2023-01-09 13:27:42 +01:00
c42c99f94f
fix: set a timeout for tcp connect
2023-01-09 10:01:35 +01:00
96ca0d9f49
feat(emqx_bridge): add /bridges_probe API endpoint
2023-01-09 10:01:33 +01:00
13b2f45405
fix: function_clause when sending messages to bridges
2023-01-06 09:24:38 +08:00
b59c4c34c5
fix(Bridge REST API): no feedback when deleting bridge
...
This fixes https://emqx.atlassian.net/browse/EMQX-8648 . The issue
described in `EMQX-8648` is that when deleting a non-existing bridge the
server gives a success response. See below:
```
curl --head -u admin:public2 -X 'DELETE' 'http://localhost:18083/api/v5/bridges/webhook:i_do_not_exist '
HTTP/1.1 204 No Content
date: Tue, 03 Jan 2023 16:59:01 GMT
server: Cowboy
```
After the fix, deleting a non existing bridge will give the following
response:
```
HTTP/1.1 404 Not Found
content-length: 49
content-type: application/json
date: Thu, 05 Jan 2023 12:40:35 GMT
server: Cowboy
```
Closes: EMQX-8648
2023-01-05 16:19:46 +01:00
fd360ac6c0
feat(buffer_worker): refactor buffer/resource workers to always use queue
...
This makes the buffer/resource workers always use `replayq` for
queuing, along with collecting multiple requests in a single call.
This is done to avoid long message queues for the buffer workers and
rely on `replayq`'s capabilities of offloading to disk and detecting
overflow.
Also, this deprecates the `enable_batch` and `enable_queue` resource
creation options, as: i) queuing is now always enables; ii) batch_size
> 1 <=> batch_enabled. The corresponding metric
`dropped.queue_not_enabled` is dropped, along with `batching`. The
batching is too ephemeral, especially considering a default batch time
of 20 ms, and is not shown in the dashboard, so it was removed.
2023-01-05 10:15:09 -03:00
b43be50a98
test: add the `redacted` test case for bridges api
2023-01-03 22:16:34 +08:00
b016695cb1
fix(bridges): obfuscate the password in bridges API responses
2023-01-03 22:09:29 +08:00
7e02eac3bc
Merge pull request #9619 from thalesmg/refactor-gauges-v50
...
refactor(metrics): use absolute gauge values rather than deltas (v5.0)
2023-01-02 10:56:47 -03:00
dbc10c2eed
chore: update copyright year 2023
2023-01-02 09:22:27 +01:00
305ed68916
chore: bump app vsns
2022-12-30 16:51:24 -03:00
8b060a75f1
refactor(metrics): use absolute gauge values rather than deltas
...
https://emqx.atlassian.net/browse/EMQX-8548
Currently, we face several issues trying to keep resource metrics
reasonable. For example, when a resource is re-created and has its
metrics reset, but then its durable queue resumes its previous work
and leads to strange (often negative) metrics.
Instead using `counters` that are shared by more than one worker to
manage gauges, we introduce an ETS table whose key is not only scoped
by the Resource ID as before, but also by the worker ID. This way,
when a worker starts/terminates, they should set their own gauges to
their values (often 0 or `replayq:count` when resuming off a queue).
With this scoping and initialization procedure, we'll hopefully avoid
hitting those strange metrics scenarios and have better control over
the gauges.
2022-12-30 16:51:24 -03:00
0b43ae621d
ci: dump docker-compose log if failed to run ct
2022-12-29 09:23:11 +01:00
35dc75b7ed
feat(mqtt): add option to customize clientid prefix for egress bridges
...
https://emqx.atlassian.net/browse/EMQX-8445
Currently the bridge client’s client ID is prefixed with the resource
ID.
Sometimes it’s useful for users to have control of this prefix,
e.g. prefix based ACL rules in the target broker.
2022-12-23 09:50:26 -03:00
c085ffa0fe
refactor: default mqtt bridgge buffer pool size down to 4
2022-12-19 23:59:46 +01:00
9e3da5b661
chore: bump app versions
2022-12-14 20:07:41 +01:00
56066a03b5
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 20:04:20 +01:00
2d7099e3ae
refactor(emqx_bridge_resource): rename a variable
2022-12-14 15:53:42 +01:00
42c58e2a91
Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams
2022-12-14 15:29:13 +01:00
b9bc82f87a
feat(gcp_pubsub): add `local_topic` config
...
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
2022-12-12 17:18:19 -03:00
c69022f3c4
test(ci): clean config after test
...
Trying to fix this error for profile `emqx`:
```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
crasher:
initial call: application_master:init/4
pid: <0.9682.1>
registered_name: []
exception exit: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
in function application_master:init/4 (application_master.erl, line 142)
ancestors: [<0.9681.1>]
message_queue_len: 1
messages: [{'EXIT',<0.9683.1>,normal}]
links: [<0.9681.1>,<0.44.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 610
stack_size: 29
reductions: 195
neighbours:
=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
application: emqx_conf
exited: {bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}
type: temporary
%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}
%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
{failed,
{emqx_plugins_SUITE,init_per_suite,
{'EXIT',
{{failed_to_start_app,emqx_conf,
{emqx_conf,
{bad_return,
{{emqx_conf_app,start,[normal,[]]},
{emqx_conf_schema,
[#{kind => validation_error,
path => "bridges",
reason => unknown_fields,
unknown => <<"influxdb_api_v1">>,
unmatched => <<"mqtt,webhook">>}]}}}}},
[{emqx_common_test_helpers,start_app,4,
[{file,
"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
{line,227}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
{emqx_plugins_SUITE,init_per_suite,1,
[{file,
"/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
{line,34}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1380}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1224}]}]}}}}}
```
2022-12-12 17:18:13 -03:00
6b6bb09d4a
fix: ingress only bridge causes egress bridge traffic stop
2022-12-09 15:36:20 +01:00
40809b2ad0
Merge remote-tracking branch 'origin/dev/ee5.0' into release-50
2022-12-09 11:45:52 +01:00
6692b0c895
feat(bridge): add Redis bridge
2022-12-06 23:15:42 +03:00
31098d6c67
test: fix bridge api tests, metrics is now a separate api
2022-12-06 19:03:32 +01:00
eb017ab034
test: add test case to cover mqtt bridge config upgrade
2022-12-01 16:50:10 +01:00
9ea22d062d
refactor: make all bridges optional (required = false)
2022-12-01 16:50:10 +01:00
9a6901987f
feat: add mqtt bridge config upgrade converter
2022-11-30 21:01:59 +01:00
983e904858
fix(connector): fix ssl clear
2022-11-30 17:03:41 +01:00
b398617614
chore: bump app versions
2022-11-28 21:12:43 +01:00
7ee53e5319
Merge tag 'v5.0.11' into dev/ee5.0
2022-11-28 21:02:21 +01:00
6dc5078a96
fix: fix obsolete SSL files aren't deleted after the bridge configuration update
2022-11-23 23:49:43 +01:00
1719414c04
chore: update app versions
2022-11-23 11:42:49 +08:00
19405114e2
fix: fix newly found unsafe `binary_to_atom`
2022-11-23 11:07:20 +08:00
e5ced07665
fix: add no_return function spec to make dialyzer happy
2022-11-16 19:07:51 +01:00
7305923d12
fix: bridge name parser should not leak atom
2022-11-16 18:25:42 +01:00
c940b901f5
chore: fix app versions
2022-11-16 16:26:43 +01:00
09455edae8
Merge tag 'v5.0.10' into dev/ee5.0
2022-11-16 16:20:30 +01:00
6c52d5de1f
fix(bridge): Ensure that the node name is known
2022-11-01 15:36:10 +08:00
73b4ac9f65
fix(bridge): keep bridge name type as binary don't convert it to atom
...
After investigation, it was confirmed that there was no need to convert the bridge name to atom
2022-11-01 11:24:24 +08:00
c157392452
docs: fix self-closing html tag, change </br> to <br/>
2022-10-27 13:57:18 +02:00
2d01726b22
fix: account calls when resource is not connected as matched
2022-10-13 15:32:04 -03:00
f0ff32c031
test: fix tests after counter changes
2022-10-11 17:45:48 -03:00
dfe14be8b2
chore: bump app vsns
2022-09-26 18:23:28 +08:00
4135910b42
chore: merge master into dev/ee5.0
2022-09-26 09:52:33 +08:00
f6ac4c3a76
Merge pull request #8798 from zmstone/0815-feat-add-kafka-connector
...
feat: Add Kafka connector
2022-09-24 22:57:50 +02:00
d24441b204
fix(bridges): some issues found in code review
2022-09-23 18:32:39 +08:00
7b601bf970
chore: delete bad parse_bridge function clause
2022-09-23 11:53:02 +02:00
adc67b165b
test: test cases for Kafka bridge REST API
2022-09-23 10:09:07 +02:00
6788881642
fix: hide influxdb udp config
2022-09-20 14:46:59 +08:00
cc327629c3
Merge pull request #8992 from terry-xiaoyu/fast_resume_from_inflight_full
...
Fast resume from inflight full
2022-09-18 17:49:04 +08:00
9aa7e826cb
refactor(resource): fast resume resource worker if inflight msgs are ACKed
2022-09-17 00:34:30 +08:00
0336b935a0
Merge pull request #8901 from JimMoen/fix-bridge-api
...
- for connectors using ecpool, parse connect error `Reason` when start pool failed.
- Refine InfluxDB bridge desc.
2022-09-16 10:22:06 +08:00
d5d3972ff5
chore: add test cases for MQTT Bridge reconnecting
2022-09-15 10:19:33 +08:00
0390a5e547
fix(bridge): mysql bridge error case
2022-09-15 09:36:52 +08:00
f41adb0997
refactor: change some default values of resource_opts
2022-09-14 15:18:07 +08:00
f018b8ab49
fix(bridge/authz/exhook): no need to drop invalid certs
2022-09-14 14:08:55 +08:00
0c1595be02
feat: Add Kafka connector
2022-09-13 19:46:56 +02:00
b9ae4ea276
refactor: rename some metrics for emqx_resource
2022-09-13 14:04:25 +08:00
81ec4c4403
Merge pull request #8906 from zmstone/merge-master-to-ee50-a
...
Merge master to ee50
2022-09-07 11:19:41 +02:00
befc4acced
Merge remote-tracking branch 'origin/master' into merge-master-to-ee50-a
2022-09-06 20:31:38 +02:00
60a90858f8
feat: check dependent actions before removing the bridges
2022-09-06 15:34:23 +08:00
23b6ff399d
Merge pull request #8844 from thalesmg/mongodb-bridge
...
feat: add mongodb bridge (e5.0)
2022-09-03 12:27:01 +08:00
7fed4faac3
Merge pull request #8859 from terry-xiaoyu/refactor_counters_for_resources
...
Some improvements and bug fixes to emqx_resources
2022-09-03 12:16:49 +08:00
d533e35df0
chore: update app vsn
2022-09-02 22:26:21 +08:00
2693fc263d
fix(bridge): show file paths for SSL certs in dashboard for bridge
2022-09-02 14:12:03 +08:00
83f21b4c65
refactor(resource): remove metrics 'sent.exception'
2022-09-02 12:46:53 +08:00
b45f3de8db
refactor(resource): rename metrics batched,queued -> batching,queuing
2022-09-02 12:41:14 +08:00
3d4afd65df
feat: add mongodb bridge (e5.0)
2022-09-01 14:47:14 -03:00
8a5a9ca2bf
Merge pull request #8854 from zmstone/0831-chore-merge-main-to-dev-ee50
...
0831 chore merge main to dev ee50
2022-09-01 11:35:23 +01:00
c1afb34a86
test: fix failed tests
2022-09-01 11:34:31 +08:00
a5ac5b6f3a
Merge branch 'dev/ee5.0' into refactor-mqtt-bridge
2022-09-01 09:30:37 +08:00
dac178cbaf
chore: ensure version bumps
2022-08-31 17:23:47 +02:00
ba1f5eecd3
fix: update the swagger for new resource metrics
2022-08-31 11:14:36 +08:00
73e19d84ee
feat: use the new metrics to bridge APIs
2022-08-30 23:47:58 +08:00
65dfa63324
fix: update the counters for data bridges
2022-08-30 12:28:01 +08:00
a6eff81163
chore: update emqtt to 1.7.0-rc.1
2022-08-30 09:50:36 +08:00
0aa10702db
feat(bridge): support async mode resource options
2022-08-30 09:34:56 +08:00
ca6533395c
feat: influxdb bridge structure fits new style
2022-08-24 09:59:22 +08:00
43c964c87e
fix(docs): ee bridge api docs generation
...
Use try catch. Because function was unexported before called.
2022-08-24 09:52:38 +08:00
45352206a3
fix(auth): remove emqx_connector from testcases of authz/authn
2022-08-22 20:20:45 +08:00
55c18c0b5f
fix(bridges): update the test cases for new config structure
2022-08-22 18:24:59 +08:00
aea8c77b49
refactor: new config structure for mqtt bridge
2022-08-22 18:24:59 +08:00
9e35032d78
fix: make resume_interval defaults to health_check_interval
2022-08-16 10:09:02 +08:00
2898966439
Merge branch 'dev/ee5.0' into resource_opts
2022-08-15 21:43:22 +08:00
19d85d485b
refactor(resource): add resource_opts level into config structure
2022-08-15 21:40:10 +08:00
b3162fe5ff
fix: conflicts between master and ee5.0
2022-08-14 22:24:09 +08:00
3cb79e811f
fix: remove the extra fields max_retries
2022-08-12 18:05:03 +08:00
0cdf4b47f1
feat: add more resource creation opts
2022-08-12 13:47:45 +08:00
a05cc20db7
Merge pull request #8693 from JimMoen/feat-influxdb-batch-query
...
Feat influxdb batch query
2022-08-12 00:22:23 +08:00
83746daad5
fix: update bridge config badmap error
2022-08-11 23:15:53 +08:00
458dab53c5
fix: undefined_functions dialyzer warning
2022-08-11 22:55:22 +08:00
ed796acb95
fix: fetch resource options after reboot
2022-08-11 22:11:28 +08:00
88388b0c54
feat: http connector support async sending
2022-08-11 21:44:22 +08:00
3a76a50382
fix: syntax error and compile error
2022-08-11 20:58:43 +08:00
2872f0b668
fix(bridges): support create resources with options
2022-08-11 19:11:44 +08:00
6203a01320
feat: add inflight window to emqx_resource
2022-08-11 08:36:35 +08:00
0377d3cf61
fix: update existing testcases for new emqx_resource
2022-08-10 00:34:35 +08:00
2e336fbc01
fix: bridge and connector namespace contains DB name
...
Contains Detailed DB name for identifier hocon schema refs.
2022-08-07 19:37:44 +08:00
55b9684541
fix: influxdb bridge api available
2022-08-01 10:28:07 +08:00
3fe76446bc
Merge branch 'master' into dev/ee5.0
2022-08-01 10:14:21 +08:00
a4992ef1b5
fix: hstreamdb connector conf & api
2022-07-28 16:47:30 +08:00
23eeb21b13
fix(webhook): the 'max_retries' opt means request retry times
2022-07-27 17:06:14 +08:00
9ae7c62656
fix: exs deps & bad suites
2022-07-27 16:24:13 +08:00
be6de4aad0
fix: ee bridge & connector remove app mod
2022-07-27 11:38:45 +08:00
fa4bc921ac
fix: hstream db connector & bridge, TODO: SUITE
2022-07-27 11:38:45 +08:00
98b36c4681
fix: hstream db connector , TODO: start apps
2022-07-27 11:38:45 +08:00
f00a7417bb
feat: hstream bridge & connector
2022-07-27 11:38:45 +08:00
6685a3c5a8
fix: remove the 'headers' field from the rule events
2022-06-17 15:57:10 +08:00
d6ef2f7502
refactor: graceful recreate resources
2022-06-17 05:29:18 +08:00
3b00b16abe
Merge pull request #8221 from terry-xiaoyu/fix_delayed_module_disbled_after_emqx_stop
...
feat: fix the hook priorities
2022-06-16 09:42:42 +08:00
738a551550
refactor: best-effort json for hocon validation errors
2022-06-15 15:01:15 +02:00
39b1b20506
feat: fix the hook priorities
2022-06-15 19:03:40 +08:00
dd099137f0
fix: catch errors when load bridges
2022-06-13 22:31:58 +08:00
9a0fdc4b11
feat: support configure connector configs directly in bridges
2022-06-10 23:06:20 +08:00
69fba6958b
refactor: remove the emqx_bridge_monitor module
2022-06-01 15:31:48 +08:00
88ca25c60c
fix(resource): fast return when starting a unavailable resource
2022-06-01 08:24:53 +08:00
efc0ca2b62
Merge branch 'master' into changes-enable_pipelining-type
2022-05-31 20:49:10 +08:00
f0ddac63cb
chore: change the enable_pipelining default value
2022-05-31 19:32:53 +08:00
d61b44ef97
fix: ct failed again
2022-05-31 19:20:27 +08:00
8aa60cc0a5
feat: generate a minimized emqx.conf
2022-05-31 19:20:27 +08:00
1054c364ad
refactor(resource): improve health check and alarm it if resource down
2022-05-31 01:40:40 +08:00
a157539710
refactor(rule): rename http_bridge -> webhook
2022-05-25 18:56:12 +08:00
71de9616d3
refactor(rule): rename outputs -> actions
2022-05-25 17:02:37 +08:00
d89d692d3e
refactor: move some APIs to new module emqx_bridge_resource
2022-05-18 15:12:34 +08:00
8b0adf69d2
Merge pull request #7926 from DDDHuang/rule_api_errormsg
...
fix: better error message for rule engine
2022-05-12 19:07:47 +08:00
2de69c97ba
fix: ruleengine & connector & bridge api, better error message
2022-05-12 16:51:36 +08:00
47ed9cbc76
fix(telemetry): count bridges referenced in `from` rule clauses
2022-05-11 15:15:17 -03:00
841acb7828
Merge pull request #7783 from emqx/EMQX-4199-introduce-a-new-emqx-resource-manager-module
...
feat: isolate resource manager processes
2022-05-09 18:33:01 +02:00
0b3e30e813
feat: isolate resource manager processes
2022-05-09 13:24:34 +02:00
9a033af2e8
fix: restart bridges via HTTP APIs to wrong node
2022-05-05 18:47:19 +08:00
1ce8a8c886
fix: testing connector removes ssl cert files that are in use
2022-05-05 14:55:40 +08:00
c7241f2abc
Merge pull request #7813 from DDDHuang/bridge_api_bad_format
...
fix: bridge api response bad node name format
2022-04-29 09:12:22 +08:00
Erik Timan