yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,418 Commits 393 Branches 684 Tags 74 MiB
7374e00a02
Commit Graph

7472 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
cb76e5a241 docs: add changelog for 10755 2023-05-19 20:41:26 +02:00
Zaiming (Stone) Shi
0d8ffc0d59 fix(resource-manager): ensure no false creation 
Update is implemented as remove + create.
If a dleete call is made while the create is in progress
the remove call is likely to timeout too.

This causes the follwing creation to falsely succeed,
because there is alreay a running child under the supervisor.

As a result, the resource is permanently removed after
resource_manager eventually handles the remove call.
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
f5e5c59763 refactor(resource-manager-sup): do not force kill resource manager 
the shutdown timeout is now set to infinity so it will never
force kill a resource manager, otherwise there will be
resource leaks
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
21de0f8274 fix(buffer-worker-sup): fast stop 
the timeout shutdown in child spec may
significantly slow down the deletion of a resource

this commit chagnes the shutdown to brutal kill

also, the pool worker removal code has been delete
because it's not necessary since the entier pool is
going to be force-delete later anyway
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
be90c63c78 chore(mqtt-connector): refine logging level 
connect failure should be at warning level but not error,
the connecting state is visiable from dashbaord
also the resource manager logs connection failures in general
at warning level
 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
5bbcf4b712 fix(mqtt-connector): faster shutdown 2023-05-19 18:55:16 +02:00
Zaiming (Stone) Shi
7eef86363a test: make erlfmt happy 2023-05-19 17:58:14 +02:00
Andrew Mayorov
9aaa0b6fe1
Merge pull request #9927 from emqx/file-transfer 
feat: implement file transfer over mqtt
 2023-05-19 18:35:13 +03:00
Thales Macedo Garitezi
4327e40740 fix(pulsar): redact error reason 
Fixes https://emqx.atlassian.net/browse/EMQX-9940

Error log after fix:

```
2023-05-19T13:09:26.304769+00:00 [error] msg: failed_to_start_pulsar_client, mfa: emqx_bridge_pulsar_impl_producer:on_start/2, line: 104, instance_id: <<"bridge:pulsar_producer:pprodu">>, pulsar_hosts: pulsar://pulsar:6652, reason: {#{{"pulsar",6652} => #{error => 'AuthenticationError',message => "Unable to authenticate",request_id => 18446744073709551615}},{child,undefined,'pulsar_producer:pprodu:emqx@127.0.0.1',{pulsar_client,start_link,['pulsar_producer:pprodu:emqx@127.0.0.1',["pulsar://pulsar:6652"],#{conn_opts => #{auth_data => <<"******">>,auth_method_name => <<"basic">>},ssl_opts => []}]},transient,false,5000,worker,[pulsar_client]}}
```
 2023-05-19 10:09:57 -03:00
lafirest
264404e180
Merge pull request #10511 from lafirest/fix/resource_safe_log 
fix(resource): make some logging of the resource manager more secure
 2023-05-19 20:42:21 +08:00
lafirest
1aa904d019
Merge pull request #10738 from lafirest/fix/tdengine_template 
fix(tdengine): add supports for the `super table` feature in the SQL template
 2023-05-19 20:41:50 +08:00
Thales Macedo Garitezi
8cfcc8fbaf
Merge pull request #10743 from thalesmg/fix-bridge-metric-aggregation-r50 
fix(bridge_api): don't crash when formatting empty/unknown bridge metrics
 2023-05-19 09:07:22 -03:00
Kjell Winblad
28dda39e39
Merge pull request #10724 from kjellwinblad/kjell/docs/missing_http_api_endpoint_descriptions/EMQX-9767 
docs: make sure there is a summary for all API endpoints
 2023-05-19 12:05:41 +02:00
firest
baeb96a6e4 chore: update changes 2023-05-19 15:36:18 +08:00
firest
0eea8438bf fix(resource): make some logging of the resource manager more secure 2023-05-19 15:28:19 +08:00
firest
fa799e95a4 fix: supports test the $events/delivery_dropped event by API 2023-05-19 14:20:26 +08:00
firest
5a08a7b9de fix(tdengine): minor improvement of code and changes 2023-05-19 11:14:31 +08:00
Paulo Zulato
9c15720fee
Merge pull request #10726 from paulozulato/fix-large-health-check-interval 
fix: validate range for some bridge options
 2023-05-18 19:25:24 -03:00
Ilya Averyanov
9c48b016a9 fix(evacuation): handle expire interval correctly 2023-05-18 22:34:54 +03:00
Thales Macedo Garitezi
09ea2e2224 fix(bridge_api): don't crash when formatting empty/unknown bridge metrics 
Fixes https://emqx.atlassian.net/browse/EMQX-9872
 2023-05-18 14:25:13 -03:00
Thales Macedo Garitezi
56634e5238
Merge pull request #10725 from thalesmg/bump-gproc-091-r50 
chore: bump gproc -> 0.9.0.1 (r5.0)
 2023-05-18 10:38:06 -03:00
Thales Macedo Garitezi
49424cc9a2
Merge pull request #10733 from thalesmg/fix-pulsar-sensitive-struct-r50 
fix(pulsar): mark whole auth struct as sensitive, update dashboard version and improve bridge type matching errors (r5.0)
 2023-05-18 10:15:10 -03:00
Paulo Zulato
5d289ade56 fix: validate range for some bridge options 
Fixes https://emqx.atlassian.net/browse/EMQX-9864

Setting a very large interval can cause `erlang:start_timer` to crash.
Also, setting auto_restart_interval or health_check_interval to "0s"
causes the state machine to be in loop as time 0 is handled separately:

| state_timeout() = timeout() | integer()
| (...)
| If Time is relative and 0 no timer is actually started, instead the the
| time-out event is enqueued to ensure that it gets processed before any
| not yet received external event.
from "https://www.erlang.org/doc/man/gen_statem.html#type-state_timeout"

Therefore, both fields are now validated against the range [1ms, 1h],
which doesn't cause above issues.
 2023-05-18 10:10:58 -03:00
firest
f1a3e5965e chore: update apps version && changes 2023-05-18 17:34:28 +08:00
firest
142125b9e4 test(tdengine): add test cases to cover the super table feature 2023-05-18 17:01:27 +08:00
firest
6ff77b221b fix(tdengine): add supports for the automatically create feature in the SQL template 2023-05-18 15:13:08 +08:00
Thales Macedo Garitezi
5d5c16a56d feat(bridges): use union member selector function for better error messages 2023-05-17 17:57:29 -03:00
Thales Macedo Garitezi
659cf64ad7 feat(pulsar): use an union member selector for better error messages 2023-05-17 17:56:53 -03:00
Thales Macedo Garitezi
10f6edd6ea
Merge pull request #10717 from thalesmg/fix-bw-flush-timer-full-infl-r50 
fix(buffer_worker): avoid setting flush timer when inflight is full
 2023-05-17 14:49:27 -03:00
Thales Macedo Garitezi
dcccc0910a fix(pulsar): mark whole auth struct as sensitive (r5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9900

I tried to patch hocon itself to filter the sensitive data, but the
way it's currently structured doesn't seem to keep that field
metadata.  So, for now, we can just mark the whole auth union as
sensitive.
 2023-05-17 11:50:22 -03:00
Thales Macedo Garitezi
060efd6964 chore: bump gproc -> 0.9.0.1 (r5.0) 
Includes this fix: https://github.com/uwiger/gproc/pull/193
 2023-05-17 11:05:44 -03:00
zhongwencool
ea8ac877b0
Merge pull request #10713 from zhongwencool/put-webhook-request-timeout-into-resource-opts 
feat: update wehbook's request_timeout into resource_opts
 2023-05-17 21:20:42 +08:00
Andrew Mayorov
c95ef71fb5
chore(ft): provide more details in README.md 2023-05-17 12:49:20 +03:00
Kjell Winblad
ab8b8ccaad docs: make sure there is a summary for all API endpoints 
Many HTTP API endpoints did not have a summary when looking at the API
docs at "http://emqx_host_name:18083/api-docs". This has been fixed by
making sure there is a summary corresponding to all endpoints.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9767
 2023-05-17 09:58:56 +02:00
Ilya Averyanov
c54d04450c
Merge pull request #10723 from savonarola/0516-improve-rebalance-logging 
Improve rebalance logging
 2023-05-17 09:21:10 +03:00
某文
2b99a9b988 feat: hide resource_opts's request_timeout 2023-05-17 13:41:11 +08:00
lafirest
b5da9eb9ad
Merge pull request #10728 from lafirest/fix/port_ruleengine_export_colums 
fix: cannot access columns exported by FOREACH in DO clause
 2023-05-17 11:46:17 +08:00
firest
1f7ede90a4 chore: update app version && changes 2023-05-17 11:01:31 +08:00
firest
18043150be fix: cannot access columns exported by FOREACH in DO clause 2023-05-17 10:55:47 +08:00
JianBo He
0dd3325d70
Merge pull request #10711 from lafirest/fix/fix_infinity_bucket 
fix(limiter): a bucket with an infinity rate shouldn't be added to limiter server
 2023-05-17 10:40:59 +08:00
Ilya Averyanov
b71955e368 fix(ft): bump application versions 2023-05-17 00:29:15 +03:00
Ilya Averyanov
1a8cf0e392 Merge branch 'release-50' into file-transfer 
* release-50:
  fix(pulsar): use a binary duration as default `health_check_interval`
  docs: add changelog entry
  docs: clarify description of bridge username and password
  chore: bump to v5.0.25
  fix(limiter): adjust type for compatibility
  fix(limiter): fix that update node-level limiter config will not working
  chore: upgrade dashboard to v1.2.4-1 for ce
  chore: upgarde rulesql to 0.1.6 to fix invaid utf8 input
  chore: add changelog for 10659
  fix: crash when sysmon.os.mem_check_interval = disabled
  chore: bump influxdb version && update changes
  refactor(influxdb): move influxdb bridge into its own app
  chore: add listener default changelog
  fix: ocsp cache SUITE failed
  fix: ensure atom key for emqx_config:get
  fix: only fill cerf_file default in server side
  fix: authn init is empty
  fix: bad listeners default ssl_options
 2023-05-17 00:27:55 +03:00
Thales Macedo Garitezi
85089a3210 fix(buffer_worker): correctly flush the buffer workers when inflight table room is made 
The previous commit uncovered another bug that was hidden by it:
`maybe_flush_after_async_reply` was sending a message to the wrong
PID.  It was sending a message to `self()` meaning to target a buffer
worker, but `self()` in that context is never the buffer worker, it's
the connector's worker.

This change also revealed a race condition where the buffer workers
could stop flushing messages.  So we piggy-backed on the atomic update
of the table size count to check if the buffer worker should be poked
to continue flushing.  This allows us to get rid of
`maybe_flush_after_async_reply` altogether.
 2023-05-16 17:15:42 -03:00
Ilya Averyanov
d4b60c5610 chore(rebalance): improve debug logging 2023-05-16 18:59:12 +03:00
Thales Macedo Garitezi
cebde87114 fix(pulsar): use a binary duration as default health_check_interval 
Fixes https://emqx.atlassian.net/browse/EMQX-9885

The frontend needs the default value to match the duration (binary)
type to display correctly.
 2023-05-16 11:29:29 -03:00
Thales Macedo Garitezi
657df05ad9 fix(buffer_worker): avoid setting flush timer when inflight is full 
Fixes https://emqx.atlassian.net/browse/EMQX-9902

When the buffer worker inflight window is full, we don’t need to set a
timer to flush the messages again because there’s no more room, and
one of the inflight windows will flush the buffer worker by calling
`flush_worker`.

Currently, we do set the timer on such situation, and this fact
combined with the default batch time of 0 yields a busy loop situation
where the CPU spins a lot while inflight messages do not return.
 2023-05-16 11:28:58 -03:00
Andrew Mayorov
6f8f21106b
fix(assert): use unpredictable binding names in macros 
Also translate macro vars to ALLCAPS for consistency.
 2023-05-16 16:46:05 +03:00
Andrew Mayorov
e3b822c1a0
chore: remove empty header 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
 2023-05-16 16:45:41 +03:00
Andrew Mayorov
a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
某文
a2aa6b4666 chore: make ci happy again 2023-05-16 21:29:52 +08:00
Andrew Mayorov
be3a0ce974
fix(maybe): correct apply/2 typespec 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
 2023-05-16 16:26:26 +03:00
firest
255f616d26 chore: bump emqx app version 2023-05-16 16:36:36 +08:00
某文
7d7c069257 feat: update wehbook's request_timeout into resource_opts 2023-05-16 15:32:43 +08:00
firest
50e7de9db2 fix(limiter): a bucket with an infinity rate shouldn't be added to limiter server 2023-05-16 13:30:41 +08:00
某文
5ef2a603a1 chore: bump to v5.0.25 2023-05-12 20:27:36 +08:00
firest
95a67f390f fix(limiter): adjust type for compatibility 2023-05-12 18:41:41 +08:00
firest
a38b270b03 fix(limiter): fix that update node-level limiter config will not working 2023-05-12 17:39:49 +08:00
Zhongwen Deng
90549abec7 fix: crash when sysmon.os.mem_check_interval = disabled 2023-05-12 14:42:23 +08:00
lafirest
33d1872e8f
Merge pull request #10679 from lafirest/refactor/influxdb_dir 
refactor(influxdb): move influxdb bridge into its own app
 2023-05-12 14:31:29 +08:00
zhongwencool
1aceed7b7c
Merge pull request #10672 from zhongwencool/fix-listener-default 
fix: bad listeners default ssl_options
 2023-05-12 14:28:23 +08:00
firest
7d46fe8390 chore: bump influxdb version && update changes 2023-05-12 11:48:56 +08:00
firest
6497bcae46 refactor(influxdb): move influxdb bridge into its own app 2023-05-12 11:48:49 +08:00
某文
91f97f6c29 fix: ocsp cache SUITE failed 2023-05-11 21:26:23 +08:00
Ilya Averyanov
5fdcfad60c fix(ft): synchronize erlcloud deps 2023-05-11 16:18:45 +05:00
Ilya Averyanov
5b5d7ceac5 Merge branch 'release-50' into file-transfer 
* release-50:
  fix(limiter): fix an error when setting `max_conn_rate` in a listener
  chore: bump erlcloud dependencies vsns
  chore: rename dynamo template files
  refactor(dynamo): move dynamo bridge into its own app
  chore: update changes && bump app versions
  fix: issues with the RabbitMQ config
  refactor(pgsql): move pgsql && matrix && timescale bridges into their own app
  fix: the iotdb password field so it has the password format
  chore: update changes
  refactor(tdengine): move tdengine bridge into its own app
  feat: deprecate listeners's authn http api
 2023-05-11 16:15:38 +05:00
某文
6056b0e7a8 fix: ensure atom key for emqx_config:get 2023-05-11 17:40:03 +08:00
Zhongwen Deng
3d41449fde fix: only fill cerf_file default in server side 2023-05-11 17:33:18 +08:00
Zhongwen Deng
78c2c53daa fix: authn init is empty 2023-05-11 16:26:47 +08:00
Zhongwen Deng
d9f9e951ec fix: bad listeners default ssl_options 2023-05-11 16:25:25 +08:00
lafirest
d3a7d6d9d8
Merge pull request #10668 from lafirest/fix/max_conn_rate 
fix(limiter): fix an error when setting `max_conn_rate` in a listener
 2023-05-11 15:49:44 +08:00
firest
b7126257a5 fix(limiter): fix an error when setting max_conn_rate in a listener 2023-05-11 13:52:13 +08:00
JimMoen
fbd516d740
Merge pull request #10622 from JimMoen/refactor-dynamo-bridge 
refactor(dynamo): move dynamo bridge into its own app
 2023-05-11 11:00:50 +08:00
lafirest
c8758190b5
Merge pull request #10662 from lafirest/refactor/pgsql_dir 
refactor(pgsql): move pgsql && matrix && timescale bridges into their own app
 2023-05-11 10:05:51 +08:00
Andrew Mayorov
6b688d6646
fix(ft): anticipate repeated kickoffs + fix testcase 2023-05-10 17:35:53 +03:00
Kjell Winblad
296c870896
Merge pull request #10664 from kjellwinblad/kjell/fix/rabbitmq_and_iotdb_issues 
Fix small RabbitMQ and IotDB issues
 2023-05-10 14:48:20 +02:00
JimMoen
e32ab10d01
chore: rename dynamo template files 2023-05-10 18:49:21 +08:00
JimMoen
8914e006c2
refactor(dynamo): move dynamo bridge into its own app 2023-05-10 18:49:21 +08:00
firest
f6a2f752ff chore: update changes && bump app versions 2023-05-10 17:23:26 +08:00
Kjell Winblad
41f5eff014 fix: issues with the RabbitMQ config 2023-05-10 10:53:40 +02:00
firest
066ed5c6ec refactor(pgsql): move pgsql && matrix && timescale bridges into their own app 2023-05-10 16:51:33 +08:00
lafirest
594ceaf9e6
Merge pull request #10650 from lafirest/refactor/tdengine_dir 
refactor(tdengine): move tdengine bridge into its own app
 2023-05-10 16:42:35 +08:00
Kjell Winblad
08af90daa9 fix: the iotdb password field so it has the password format 2023-05-10 10:38:40 +02:00
zhongwencool
48db98d75b
Merge pull request #10657 from zhongwencool/deprecated-listeners-authn 
feat: deprecate listeners authn
 2023-05-10 15:35:49 +08:00
Ilya Averyanov
8d9b785bd7 Merge branch 'release-50' into file-transfer 
* release-50:
  chore(rebalance): fix app metadata
  chore(rebalance): move apps from lib-ee, add READMEs
  docs: refine zh tr
  docs: delete APL header from ee file
  docs: delete zh changelog
  chore(rebalance): review fixes
  chore(rebalance): rebase and review fixes
  feat(rebalance): port apps from 4.x
 2023-05-10 11:55:23 +05:00
Ilya Averyanov
42f5433aaf
Merge pull request #10655 from savonarola/0510-idempotent-fin 
Make FT fin command idempotent
 2023-05-10 11:39:43 +05:00
firest
0f979b3d24 refactor(tdengine): move tdengine bridge into its own app 2023-05-10 14:12:57 +08:00
Zhongwen Deng
83e7b30a80 feat: deprecate listeners's authn http api 2023-05-10 09:50:53 +08:00
Ilya Averyanov
244188982b fix(ft): add missing translations 2023-05-09 23:32:17 +05:00
Ilya Averyanov
45875bfee5 fix(ft): update s3 exporter tests to the new config 2023-05-09 23:32:17 +05:00
Ilya Averyanov
9381c895bb fix(ft): fix dialyzer issues 2023-05-09 23:32:16 +05:00
Andrew Mayorov
079b8e9476 fix(ft): silence warnings when some root is not yet mkdired 2023-05-09 23:32:16 +05:00
Andrew Mayorov
5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends 
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
 2023-05-09 23:32:16 +05:00
Ilya Averyanov
8ac881a140 chore(ft): handle multiple/concurrent fins more gracefully 2023-05-09 23:31:39 +05:00
Ilya Averyanov
7fa166f034 Merge branch 'release-50' into file-transfer 
* release-50: (73 commits)
  feat: add RabbitMQ bridge
  docs: improve rule engine labels and descriptions
  chore: bump version && update changes
  refactor(rocketmq): move rocketmq bridge into its own app
  test: dashboard_listener_test crash
  chore: bump chart versions
  chore: bump ee version to e5.0.4-alpha.1
  test: fix inter-suite flakiness
  build: compatibility to make 4.4+
  feat: add IotDB bridge
  ci: ensure git safe dir in build_packages
  ci: ensure git safe dir
  test: check_oom's max_mailbox_size
  feat: rename max_message_queue_len to max_mailbox_size
  fix(buffer_worker): fix inflight count when updating inflight item
  chore: prepare for v5.0.25-rc.1 release
  docs: add change log entry
  fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
  chore: `MQTT X` -> `MQTTX`
  chore: make sure brod_gssapi app is included in relese package
  ...
 2023-05-09 23:30:22 +05:00
Ilya Averyanov
61deda3ea6 chore(rebalance): fix app metadata 2023-05-09 21:05:20 +05:00
Ilya Averyanov
8d83dc12e7 chore(rebalance): move apps from lib-ee, add READMEs 2023-05-09 20:51:24 +05:00
Ilya Averyanov
e683d28973 chore(rebalance): rebase and review fixes 2023-05-09 20:51:24 +05:00
Ilya Averyanov
609f7bd8fd feat(rebalance): port apps from 4.x 2023-05-09 20:51:22 +05:00
Kjell Winblad
70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00
lafirest
5ebb96fcc8
Merge pull request #10648 from lafirest/refactor/rocketmq_dir 
refactor(rocketmq): move rocketmq bridge into its own app
 2023-05-09 17:55:10 +08:00