yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,861 Commits 393 Branches 684 Tags 74 MiB
bdf24d0ec8
Commit Graph

571 Commits

Author SHA1 Message Date
Stefan Strigler
ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
Thales Macedo Garitezi
30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10309
 2023-06-28 14:08:40 -03:00
Thales Macedo Garitezi
c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Thales Macedo Garitezi
f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer 
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
 2023-06-22 09:17:45 -03:00
zhongwencool
093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool
07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
Andrew Mayorov
13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view 
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
 2023-06-17 00:54:24 +02:00
zhongwencool
e42cc58694
Merge pull request #11050 from emqx/release-51 
chore: sync release-51 back to master
 2023-06-15 09:45:31 +08:00
Andrew Mayorov
acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes 
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
 2023-06-15 00:50:15 +03:00
zhongwencool
9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
Stefan Strigler
0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
Stefan Strigler
b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
SergeTupchiy
e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup 
feat: implement configuration and user data export/import
 2023-06-09 14:18:39 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
firest
86a7b2d69a fix(resource): improve log security when resource creation fails 2023-06-09 11:43:42 +08:00
Kjell Winblad
273dedf7a6
Merge pull request #10911 from kjellwinblad/kjell/bridge/atom_length_too_long/EMQX-9609 
fix: friendly error message when creating bridges with too long names
 2023-06-08 15:46:42 +02:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process 
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
95f706bb9e
fix(ssl): avoid explicit deletion of managed certs / keys 
This logic was incorrect because it didn't take into account
certfiles / keyfiles "refcounts".
 2023-06-05 22:41:07 +03:00
Kjell Winblad
375661c6a1 test: add test case for error when bridge name is too long 2023-06-05 17:02:08 +02:00
Andrew Mayorov
5b56a35c0a
Merge pull request #10932 from chore/drop-mqtt-mode-tests 
chore(bridge): drop deprecated `mode` setting from examples / tests
 2023-06-02 22:46:31 +03:00
Zaiming (Stone) Shi
aa46c0a484
Merge pull request #10928 from zmstone/0602-sync-release-51-to-master 
0602 sync release 51 to master
 2023-06-02 21:45:21 +02:00
Thales Macedo Garitezi
33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50 
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
 2023-06-02 16:20:36 -03:00
Andrew Mayorov
09290f8d61
chore(bridge): drop deprecated mode setting from examples / tests 2023-06-02 20:31:31 +03:00
Zaiming (Stone) Shi
28c564d15b Merge remote-tracking branch 'origin/master' into release-51 2023-06-02 16:25:48 +02:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl 
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
 2023-06-01 13:01:53 -03:00
某文
c27d844244 feat: improve authz/bridge/rule_engine/schema_registry config update 2023-06-01 23:20:56 +08:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval 
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
 2023-06-01 11:20:06 -03:00
Stefan Strigler
e011b5532f
Merge pull request #10881 from sstrigler/EMQX-9355-5-0-x-test-coverage-80 
IoTDB test coverage
 2023-06-01 15:48:56 +02:00
Kjell Winblad
75ff76a16b fix: friendly error message when creating bridges with too long names 
This commit makes the error message and log entry that appear when one
tries to create a bridge with a name the exceeds 255 bytes (the max
length for atoms) more friendly and easier to understand.

An even better fix would be to not store bridge names as atoms but this
probably requires a more substantial change.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9609
 2023-06-01 15:30:06 +02:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id 
fix(ruleeng): parse bridge id on create / update
 2023-06-01 10:21:26 +03:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update 
Instead of parsing it every time in the `send_message/2` hot path.
 2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test 
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
 2023-05-31 10:19:55 -03:00
Andrew Mayorov
7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Andrew Mayorov
a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools 
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
 2023-05-30 17:21:44 +03:00
Stefan Strigler
767f7c57e7 test: check device_id is used from topic or config 2023-05-30 15:55:21 +02:00
Stefan Strigler
938d62a666 test: handle bad payload 2023-05-30 15:55:21 +02:00
Stefan Strigler
64d582770d test: add tracepoints 2023-05-30 15:55:21 +02:00
Andrew Mayorov
4e6269bedb
feat(mqttconn): subscribe each worker if shared subcription 
Also rename `subscriptions` -> `ingress` and `forwards` -> `egress` for
consistency with the config schema.
 2023-05-30 14:50:05 +03:00
Andrew Mayorov
81e78516aa
feat(mqttconn): employ ecpool instead of a single worker 2023-05-30 14:50:05 +03:00
Andrew Mayorov
bd956d00b6
feat(mqttconn): stop using gproc in hot path 
Also drop fiddling with `mountpoint` since this option seems not to be
used anywhere.
 2023-05-30 14:50:05 +03:00
Zaiming (Stone) Shi
91cdc69976
Merge pull request #10867 from zmstone/0530-merge-release-50-to-master 
0530 merge release 50 to master
 2023-05-30 09:54:57 +02:00
Zaiming (Stone) Shi
9529919046 chore: bump app versions 2023-05-30 08:08:29 +02:00
Thales Macedo Garitezi
67e182e0c9
Merge pull request #10813 from thalesmg/refactor-kafka-on-stop-v50 
feat(kafka): ensure allocated resources are removed on failures
 2023-05-29 16:49:29 -03:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Zaiming (Stone) Shi
cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi
772a4575ca
Merge pull request #10821 from thalesmg/fix-webhook-bridge-req-timeout-r50 
fix(webhook): keep `resource_opts.request_timeout` for webhook bridge (r5.0)
 2023-05-26 08:39:00 +02:00
Zaiming (Stone) Shi
407cab220b
Merge pull request #10760 from paulozulato/fix-failure-joining 
fix: avoid error 500 when node is re-joining cluster
 2023-05-26 03:08:17 +02:00
Thales Macedo Garitezi
e43517188f fix(bridge): remove bridge if exception happens during creation 2023-05-25 16:38:09 -03:00
Paulo Zulato
ea86f4442b fix: avoid error 500 when node is re-joining cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-9899
 2023-05-25 13:32:28 -03:00
Thales Macedo Garitezi
194faefc8a fix(webhook): keep resource_opts.request_timeout for webhook bridge (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-10001

Recently, we unified request_timeout in a single field located at the
webhook connector schema.  However, the correct fix would be to use
the resource_opts.request_timeout one, as that’s the only one that
allows infinity timeout.
 2023-05-25 13:19:27 -03:00
JianBo He
6b5ff24952 test: assert the block_until result 2023-05-25 18:12:38 +08:00
JianBo He
de7f1c8aec test: add tests for auto_restart_interval 2023-05-25 17:15:19 +08:00
zhongwencool
14a6b36899
Merge pull request #10803 from JimMoen/merge-release-50 
Merge release-50 into master
 2023-05-24 23:08:46 +08:00
JimMoen
28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Kjell Winblad
4f21bdb885 fix: REST API version 2 not available in IoTDB 0.13 and 1.0 
This commit makes sure that REST API version 1 is used when the
user has selected to use IoTDB 0.13 or 1.0

Fixes:
https://emqx.atlassian.net/browse/EMQX-9920
 2023-05-24 13:09:47 +02:00
某文
7da0860d6c chore: upgrade minirest to 1.3.10 for more clear error msg 2023-05-24 10:44:52 +08:00
Thales Macedo Garitezi
44ed6a19ee fix(bridge): pass resource option request_timeout = infinity along to buffer workers (r5.0) 2023-05-23 14:50:43 -03:00
Thales Macedo Garitezi
b5f63f4151 test: fix wrong test listeners option 2023-05-23 14:50:43 -03:00
某文
1deb3c50c8 test: bridge test init dashboard listener failed 2023-05-23 10:28:01 +08:00
某文
67a30ee976 test: bridge test init dashboard listener failed 2023-05-23 10:16:43 +08:00
Zaiming (Stone) Shi
732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
Thales Macedo Garitezi
7d798c10e9 perf(buffer_worker): flush metrics periodically inside buffer worker process 
Fixes https://emqx.atlassian.net/browse/EMQX-9905

Since calling `telemetry` is costly in a hot path, we instead collect
metrics inside the buffer workers state and periodically flush them,
rather than immediately as events happen.
 2023-05-22 09:11:23 -03:00
William Yang
e824e1db39
Merge pull request #10698 from qzhuyan/perf/william/force-atom-conf-path 
perf(config): enforcing atom key  path in hotcode path
 2023-05-22 09:38:09 +02:00
Thales Macedo Garitezi
09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-9872
 2023-05-18 14:25:13 -03:00
Thales Macedo Garitezi
5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi
447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Thales Macedo Garitezi
b2afe4e90c
Merge pull request #10690 from thalesmg/perf-webhook-retry-async-reply-v50 
perf(webhook): add async retries and evaluate reply callback in fresh process
 2023-05-17 11:06:48 -03:00
Thales Macedo Garitezi
a7b41e1cdf perf(webhook): add retry attempts for async 
This is a performance improvement for webhook bridge.

Since this bridge is called using `async` callback mode, and `ehttpc`
frequently returns errors of the form `normal` and `{shutdown,
normal}` that are retried "for free" by `ehttpc`, we add this behavior
to async requests as well.  Other errors are retried too, but they are
not "free": 3 attempts are made at a maximum.

This is important because, when using buffer workers, we should avoid
making them enter the `blocked` state, since that halts all progress
and makes throughput plummet.
 2023-05-17 09:20:50 -03:00
William Yang
cdf42760fa chore: little change avoid atom leak 2023-05-17 11:01:03 +02:00
某文
2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
Zaiming (Stone) Shi
9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field 
0421 refactor delete unused schema field
 2023-05-16 22:22:03 +02:00
某文
a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
William Yang
1c746ed289 perf(config): enforcing atom key path in hotcode path 2023-05-16 09:57:04 +02:00
某文
7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
zhongwencool
a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zaiming (Stone) Shi
7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Zhongwen Deng
14952658d5 chore: convert rotation struct to roation_count 2023-05-09 11:24:51 +08:00
Stefan Strigler
d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zaiming (Stone) Shi
856129984b refactor: remove raw_with_default config load option 
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
 2023-05-04 09:48:28 +02:00
Zaiming (Stone) Shi
d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi
57cc854a4a test(bridge): fix bridge map type filed converters 
now the converters on map type fields only work at the wrapping map
level but not the values
 2023-04-30 10:45:11 +02:00
Paulo Zulato
dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi
ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-8398
 2023-04-24 10:28:26 -03:00
Serge Tupchii
45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid 
Fixes: EMQX-9310
 2023-04-20 17:21:43 +03:00
Zaiming (Stone) Shi
30fd9b10f7 refactor: change bridges config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Ivan Dyachkov
dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
某文
fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
Ivan Dyachkov
9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
Stefan Strigler
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules 
New emqx_utils app to collect utility modules
 2023-04-14 20:33:11 +02:00
Thales Macedo Garitezi
e073bc90bc refactor(buffer_worker): rename s/queue/buffer/g 2023-04-14 11:37:19 -03:00
Stefan Strigler
4f80690162 fix: byebye jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JimMoen
790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge 
feat: implement Microsoft SQL Server bridge (e5.0)
 2023-04-14 17:55:48 +08:00
Andrew Mayorov
5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics 
feat(resource): ask for metrics only when needed
 2023-04-14 12:28:52 +03:00