yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,131 Commits 393 Branches 669 Tags 74 MiB
f8d330c2f3
Commit Graph

612 Commits

Author SHA1 Message Date
JimMoen
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
Thales Macedo Garitezi
29e706c83d refactor: move catch to dry run fn 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
4b8cea4498 fix: make error handling more flexible 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
963d63b899 refactor: rename temporary name 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms 
Fixes https://emqx.atlassian.net/browse/EMQX-10653
 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi
a3be1529c6 fix(schema): ensure enterprise module is correctly loaded 
Fixes https://emqx.atlassian.net/browse/EMQX-10654

Despite loading the application in `nodetool`, we need to invoke `:module_info()` to force
the module to be actually loaded, especially when it's called in `call_hocon` to generate
the initial configurations.  Otherwise, it'll fail to list all the bridge schemas.
 2023-07-28 16:27:43 -03:00
Ivan Dyachkov
5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master 2023-07-27 16:58:09 +02:00
Ivan Dyachkov
cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi
5abe4bed88 feat: add Azure Event Hub Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10338
 2023-07-27 09:22:39 -03:00
JianBo He
511d1b6ca1 chore: hide the hstreamdb http api 2023-07-24 20:11:44 +08:00
firest
8439ce0e84 chore: update app version && reboot_lists 2023-07-24 19:08:41 +08:00
Dennis Zhuang
c9550cc2e5 refactor: rename bridge greptimedb_grpc_v1 to greptimedb 2023-07-24 17:58:48 +08:00
Dennis Zhuang
6f7fbcf693 fix: compile error 2023-07-24 17:58:48 +08:00
Dennis Zhuang
975795a6e0 feat: add ci test 2023-07-24 17:58:48 +08:00
Dennis Zhuang
89bce99870 test: greptimedb data brige 2023-07-24 17:58:48 +08:00
Dennis Zhuang
c5078980f3 feat: adds the greptimedb bridge to emqx modules 2023-07-24 17:58:48 +08:00
JimMoen
613dc1646c
chore: hidden hstreamdb bridge schema 2023-07-24 17:22:29 +08:00
Thales Macedo Garitezi
41b780f7f2
Merge pull request #11280 from thalesmg/unhide-gcp-consumer-master-20230717 
chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
 2023-07-19 10:26:42 -03:00
Ivan Dyachkov
b7f0f7b32d chore: merge master into release-51 2023-07-19 09:28:05 +02:00
Paulo Zulato
484519dcf9 feat(kinesis): implement Amazon Kinesis Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10474
Fixes https://emqx.atlassian.net/browse/EMQX-10475
 2023-07-17 11:51:42 -03:00
Thales Macedo Garitezi
05c3e023a9 chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0 
Fixes https://emqx.atlassian.net/browse/EMQX-10506
 2023-07-17 11:24:21 -03:00
Thales Macedo Garitezi
ab5fd1e5c3
Merge pull request #11260 from thalesmg/fix-rule-maps-nested-put-r51 
fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
 2023-07-14 15:02:24 -03:00
Thales Macedo Garitezi
01b143c5ad fix(resource): don't destruct error tuple 
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
 2023-07-13 16:12:33 -03:00
Thales Macedo Garitezi
924ed06760
Merge pull request #11256 from thalesmg/bridge-api-ecpool-400-errors-20230712-r51 
fix(bridge_api): return 400 for all ecpool start errors
 2023-07-13 10:05:07 -03:00
Thales Macedo Garitezi
bbff1d6ae9 fix(bridge_api): return 400 for all ecpool start errors 
Fixes https://emqx.atlassian.net/browse/EMQX-10460
(https://emqx.atlassian.net/browse/EMQX-10460?focusedCommentId=30126)
 2023-07-12 15:32:34 -03:00
Thales Macedo Garitezi
bffef386c1 fix(rule_maps): avoid losing data when using emqx_rule_maps:nested_put 
Fixes https://emqx.atlassian.net/browse/EMQX-10541
 2023-07-12 14:28:18 -03:00
Kjell Winblad
6ce47614d2 refactor: HTTP bridge into the emqx_bridge_http app 
Fixes:
https://emqx.atlassian.net/browse/EMQX-9568
 2023-07-12 14:47:02 +02:00
Kjell Winblad
f28510b3ad refactor: HTTP connector into emqx_bridge_http app 2023-07-12 14:46:43 +02:00
JimMoen
b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
firest
a1d2b2ca51 chore: bump app versions 2023-07-07 11:17:54 +08:00
Thales Macedo Garitezi
353cc065db
Merge pull request #11175 from thalesmg/fix-mysql-start-error 
fix(mysql_bridge): make nxdomain a 400 API error
 2023-07-06 08:26:12 -03:00
Andrew Mayorov
d568d43fe4
test(bridge): switch emqx_bridge_api_SUITE to use new tooling 2023-07-04 13:20:03 +02:00
Andrew Mayorov
f76f3b77d7
test: add tooling to make testruns more isolated and predictable 
Also showcase their usage in some of the existing testsuites.
 2023-07-04 12:53:39 +02:00
Thales Macedo Garitezi
c216dfd96b fix(mysql_bridge): make nxdomain a 400 API error 
Fixes https://emqx.atlassian.net/browse/EMQX-10460
 2023-06-30 16:10:09 -03:00
Stefan Strigler
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics 
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
 2023-06-30 20:07:23 +02:00
Thales Macedo Garitezi
6fe6aa7997
Merge pull request #11162 from thalesmg/treat-404-as-failure-master 
fix(webhook): treat 404 and other error replies as errors in async requests
 2023-06-30 13:13:18 -03:00
Thales Macedo Garitezi
ca435975de fix(webhook): treat http status code 429 as recoverable 2023-06-30 09:46:03 -03:00
Thales Macedo Garitezi
59b109eb5c fix(webhook): treat 404 and other error replies as errors in async requests 
Fixes https://emqx.atlassian.net/browse/EMQX-10405

The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
 2023-06-29 15:45:23 -03:00
Stefan Strigler
d71d70c889 fix: allow to pass in Opts rather than just a ReplyTo 2023-06-29 16:09:45 +02:00
Stefan Strigler
40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler
837acd4234 test(emqx_rule_engine): test rule metrics 2023-06-29 16:09:45 +02:00
Stefan Strigler
ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi
30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10309
 2023-06-28 14:08:40 -03:00
Thales Macedo Garitezi
c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi
f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer 
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
 2023-06-22 09:17:45 -03:00
zhongwencool
093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
Andrew Mayorov
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view 
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
 2023-06-17 00:54:24 +02:00
zhongwencool
e42cc58694
Merge pull request #11050 from emqx/release-51 
chore: sync release-51 back to master
 2023-06-15 09:45:31 +08:00
Andrew Mayorov
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes 
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
 2023-06-15 00:50:15 +03:00
zhongwencool
9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup 
feat: implement configuration and user data export/import
 2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
Kjell Winblad
273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609 
fix: friendly error message when creating bridges with too long names
 2023-06-08 15:46:42 +02:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process 
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys 
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
 2023-06-05 22:41:07 +03:00
Kjell Winblad
375661c6a1 test: add test case for error when bridge name is too long 2023-06-05 17:02:08 +02:00
Andrew Mayorov
5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests 
chore(bridge): drop deprecated `mode` setting from examples / tests
 2023-06-02 22:46:31 +03:00
Zaiming (Stone) Shi
aa46c0a484
Merge pull request #10928 from zmstone/0602-sync-release-51-to-master 
0602 sync release 51 to master
 2023-06-02 21:45:21 +02:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50 
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
 2023-06-02 16:20:36 -03:00
Andrew Mayorov
09290f8d61
chore(bridge): drop deprecated mode setting from examples / tests 2023-06-02 20:31:31 +03:00
Zaiming (Stone) Shi
28c564d15b Merge remote-tracking branch 'origin/master' into release-51 2023-06-02 16:25:48 +02:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl 
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
 2023-06-01 13:01:53 -03:00
某文
c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval 
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
 2023-06-01 11:20:06 -03:00
Stefan Strigler
e011b5532f
Merge pull request #10881 from sstrigler/EMQX-9355-5-0-x-test-coverage-80 
IoTDB test coverage
 2023-06-01 15:48:56 +02:00
Kjell Winblad
75ff76a16b fix: friendly error message when creating bridges with too long names 
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
 2023-06-01 15:30:06 +02:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id 
fix(ruleeng): parse bridge id on create / update
 2023-06-01 10:21:26 +03:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update 
Instead of parsing it every time in the `send_message/2` hot path.
 2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test 
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
 2023-05-31 10:19:55 -03:00
Andrew Mayorov
7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Andrew Mayorov
a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools 
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
 2023-05-30 17:21:44 +03:00
Stefan Strigler
767f7c57e7 test: check device_id is used from topic or config 2023-05-30 15:55:21 +02:00
Stefan Strigler
938d62a666 test: handle bad payload 2023-05-30 15:55:21 +02:00
Stefan Strigler
64d582770d test: add tracepoints 2023-05-30 15:55:21 +02:00
Andrew Mayorov
4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription 
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
81e78516aa
feat(mqttconn): employ ecpool instead of a single worker 2023-05-30 14:50:05 +03:00
Andrew Mayorov
bd956d00b6
feat(mqttconn): stop using gproc in hot path 
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
 2023-05-30 14:50:05 +03:00
Zaiming (Stone) Shi
91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master 
0530 merge release 50 to master
 2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi
9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
Thales Macedo Garitezi
67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50 
feat(kafka): ensure allocated resources are removed on failures
 2023-05-29 16:49:29 -03:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi
cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi
772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50 
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
 2023-05-26 08:39:00 +02:00
Zaiming (Stone) Shi
407cab220b
Merge pull request #10760 from paulozulato/fix-failure-joining 
fix: avoid error 500 when node is re-joining cluster
 2023-05-26 03:08:17 +02:00
Thales Macedo Garitezi
e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Paulo Zulato
ea86f4442b fix: avoid error 500 when node is re-joining cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-9899
 2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi
194faefc8a fix(webhook): keep resource_opts.request_timeout for webhook bridge (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
 2023-05-25 13:19:27 -03:00
JianBo He
6b5ff24952 test: assert the block_until result 2023-05-25 18:12:38 +08:00
JianBo He
de7f1c8aec test: add tests for auto_restart_interval 2023-05-25 17:15:19 +08:00
zhongwencool
14a6b36899
Merge pull request #10803 from JimMoen/merge-release-50 
Merge release-50 into master
 2023-05-24 23:08:46 +08:00
JimMoen
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad
4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0 
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
 2023-05-24 13:09:47 +02:00
某文
7da0860d6c chore: upgrade minirest to 1.3.10 for more clear error msg 2023-05-24 10:44:52 +08:00
Thales Macedo Garitezi
44ed6a19ee fix(bridge): pass resource option request_timeout = infinity along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00