From 8ce21225160f1abb9385f46a4e508c77f5a775c9 Mon Sep 17 00:00:00 2001 From: "Zaiming (Stone) Shi" Date: Fri, 5 May 2023 12:56:34 +0200 Subject: [PATCH 1/6] chore: prepare to cut e5.0.3-rc.1 --- apps/emqx/include/emqx_release.hrl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/emqx/include/emqx_release.hrl b/apps/emqx/include/emqx_release.hrl index 21eb85dfb..f6dcdd917 100644 --- a/apps/emqx/include/emqx_release.hrl +++ b/apps/emqx/include/emqx_release.hrl @@ -35,7 +35,7 @@ -define(EMQX_RELEASE_CE, "5.0.22"). %% Enterprise edition --define(EMQX_RELEASE_EE, "5.0.3-alpha.5"). +-define(EMQX_RELEASE_EE, "5.0.3-rc.1"). %% the HTTP API version -define(EMQX_API_VERSION, "5.0"). From cd5a375aade86036d1cc39590a277ba0a0ce9434 Mon Sep 17 00:00:00 2001 From: Ivan Dyachkov Date: Fri, 5 May 2023 16:05:00 +0200 Subject: [PATCH 2/6] chore: e5.0.3 changelog --- changes/e5.0.3.en.md | 187 +++++++++++++++++++++++++++++++++++++++++++ changes/e5.0.3.zh.md | 60 ++++++++++++++ 2 files changed, 247 insertions(+) create mode 100644 changes/e5.0.3.en.md create mode 100644 changes/e5.0.3.zh.md diff --git a/changes/e5.0.3.en.md b/changes/e5.0.3.en.md new file mode 100644 index 000000000..ecef80a31 --- /dev/null +++ b/changes/e5.0.3.en.md @@ -0,0 +1,187 @@ +# e5.0.3 + +## Enhancements + +- [#10077](https://github.com/emqx/emqx/pull/10077) Add support for QUIC TLS password protected certificate file. + + +- [#10128](https://github.com/emqx/emqx/pull/10128) Add support for OCSP stapling for SSL MQTT listeners. + +- [#10156](https://github.com/emqx/emqx/pull/10156) Change the priority of the configuration: + 1. If it is a new installation of EMQX, the priority of configuration is `ENV > emqx.conf > HTTP API`. + 2. If EMQX is upgraded from an old version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then the configuration priority remains the same as before. That is, `HTTP API > ENV > emqx.conf`. + + Deprecated data/configs/local-override.conf. + + Stabilizing the HTTP API for hot updates. + +- [#10164](https://github.com/emqx/emqx/pull/10164) Add CRL check support for TLS MQTT listeners. + +- [#10206](https://github.com/emqx/emqx/pull/10206) Decouple the query mode from the underlying call mode for buffer + workers. + + Prior to this change, setting the query mode of a resource + such as a bridge to `sync` would force the buffer to call the + underlying connector in a synchronous way, even if it supports async + calls. + +- [#10207](https://github.com/emqx/emqx/pull/10207) Use 'label' from i18n file as 'summary' in OpenAPI spec. + +- [#10210](https://github.com/emqx/emqx/pull/10210) Unregister Mnesia post commit hook when Mria is being stopped. + This fixes hook failures occasionally occurring on stopping/restarting Mria. + + [Mria PR](https://github.com/emqx/mria/pull/133) + +- [#10224](https://github.com/emqx/emqx/pull/10224) Add the option to customize `clusterIP` in Helm chart, so that a user may set it to a fixed IP. + +- [#10263](https://github.com/emqx/emqx/pull/10263) Add command 'eval-ex' for Elixir expression evaluation. + +- [#10278](https://github.com/emqx/emqx/pull/10278) Refactor the directory structure of all gateways. + +- [#10306](https://github.com/emqx/emqx/pull/10306) Add support for `async` query mode for most bridges. + + Before this change, some bridges (Cassandra, MongoDB, MySQL, Postgres, Redis, RocketMQ, TDengine) were only allowed to be created with a `sync` query mode. + +- [#10318](https://github.com/emqx/emqx/pull/10318) Now, the rule engine language's FROM clause supports both strings enclosed in double quotes (") and single quotes ('). + +- [#10336](https://github.com/emqx/emqx/pull/10336) Add `/rule_engine` API endpoint to manage configuration of rule engine. + +- [#10354](https://github.com/emqx/emqx/pull/10354) More specific error messages when configure with bad max_heap_size value. + Log current value and the max value when the `message_queue_too_long` error is thrown. + +- [#10358](https://github.com/emqx/emqx/pull/10358) Hide `flapping_detect/conn_congestion/stats` configuration. + Deprecate `flapping_detect.enable`. + +- [#10359](https://github.com/emqx/emqx/pull/10359) Metrics now are not implicitly collected in places where API handlers don't make any use of them. Instead, a separate backplane RPC gathers cluster-wide metrics. + +- [#10373](https://github.com/emqx/emqx/pull/10373) Deprecate the trace.payload_encode configuration. + Add payload_encode=[text,hidden,hex] option when creating a trace via HTTP API. + +- [#10381](https://github.com/emqx/emqx/pull/10381) Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. + +- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(rule_engine/bridge/authz/authn) from configuration files and documentation. + +- [#10391](https://github.com/emqx/emqx/pull/10391) Hide a large number of advanced options to simplify the configuration file. + + That includes `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, + `flapping_detect`, `conn_congestion`, `stats,auto_subscribe`, `broker_perf`, + `shared_subscription_group`, `slow_subs`, `ssl_options.user_lookup_fun` and some advance items + in `node` and `dashboard` section, [#10358](https://github.com/emqx/emqx/pull/10358), + [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385). + +- [#10404](https://github.com/emqx/emqx/pull/10404) Change the default queue mode for buffer workers to `memory_only`. + Before this change, the default queue mode was `volatile_offload`. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations. + +- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate `Cassandra` into `bridges` as a new backend. At the current stage: + - Only support Cassandra version 3.x, not yet 4.x. + +- [#10143](https://github.com/emqx/emqx/pull/10143) Add `RocketMQ` data integration bridge. + +- [#10165](https://github.com/emqx/emqx/pull/10165) Support escaped special characters in InfluxDB data bridge write_syntax. + This update allows to use escaped special characters in string elements in accordance with InfluxDB line protocol. + +- [#10294](https://github.com/emqx/emqx/pull/10294) When configuring a MongoDB bridge, you can now use the ${var} syntax to reference fields in the message payload within the collection field. This enables you to select the collection to insert data into dynamically. + +- [#10337](https://github.com/emqx/emqx/pull/10337) Add schema registry feature. + + With schema registry, one can encode and decode special serialization formats in payloads when transforming messages in Rule Engine. Currently, only [Apache Avro](https://avro.apache.org/) is supported. + +- [#10363](https://github.com/emqx/emqx/pull/10363) Implement Microsoft SQL Server bridge. + +- [#10573](https://github.com/emqx/emqx/pull/10573) Improved performance of Webhook bridge when using synchronous query mode. + This also should improve the performance of other bridges when they are configured with no batching. + +## Bug Fixes + +- [#10145](https://github.com/emqx/emqx/pull/10145) Fix `bridges` API to report error conditions for a failing bridge as + `status_reason`. Also when creating an alarm for a failing resource we include + this error condition with the alarm's message. + +- [#10172](https://github.com/emqx/emqx/pull/10172) Fix the incorrect default ACL rule, which was: + ``` + {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. + ``` + + However, it should use `{re, "^dashboard$"}` to perform a regular expression match: + ``` + {allow, {username, {re,"^dashboard$"}}, subscribe, ["$SYS/#"]}. + ``` + +- [#10174](https://github.com/emqx/emqx/pull/10174) Upgrade library `esockd` from 5.9.4 to 5.9.6. + Fix an unnecessary error level logging when a connection is closed before proxy protocol header is sent by the proxy. + +- [#10195](https://github.com/emqx/emqx/pull/10195) Add labels to API schemas where description contains HTML and breaks formatting of generated documentation otherwise. + +- [#10196](https://github.com/emqx/emqx/pull/10196) Use lower-case for schema summaries and descritptions to be used in menu of generated online documentation. + +- [#10209](https://github.com/emqx/emqx/pull/10209) Fix bug where a last will testament (LWT) message could be published + when kicking out a banned client. + +- [#10211](https://github.com/emqx/emqx/pull/10211) Hide `broker.broker_perf` config and API documents. + The two configs `route_lock_type` and `trie_compaction` are rarely used and requires a full cluster restart to take effect. They are not suitable for being exposed to users. + Detailed changes can be found here: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions + +- [#10225](https://github.com/emqx/emqx/pull/10225) Allow installing a plugin if its name matches the beginning of another (already installed) plugin name. + For example: if plugin "emqx_plugin_template_a" is installed, it must not block installing plugin "emqx_plugin_template". + +- [#10226](https://github.com/emqx/emqx/pull/10226) Don't crash on validation error in `/bridges` API, return `400` instead. + +- [#10242](https://github.com/emqx/emqx/pull/10242) Fixed a log data field name clash. + Piror to this fix, some debug logs may report a wrong Erlang PID which may affect troubleshooting session takeover issues. + +- [#10257](https://github.com/emqx/emqx/pull/10257) Fixed the issue where `auto_observe` was not working in LwM2M Gateway. + + Before the fix, OBSERVE requests were sent without a token, causing failures + that LwM2M clients could not handle. + + After the fix, LwM2M Gateway can correctly observe the resource list carried by + client, furthermore, unknown resources will be ignored and printing the following + warning log: + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) Enhance logging behaviour during boot failure. + When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated. + +- [#10297](https://github.com/emqx/emqx/pull/10297) Keeps `eval` command backward compatible with v4 by evaluating only Erlang expressions, even on Elixir node. For Elixir expressions, use `eval-ex` command. + +- [#10300](https://github.com/emqx/emqx/pull/10300) Fixed an issue where a build made with Elixir could not receive uploaded plugins until the `plugins` folder was created manually to receive the uploaded files. + +- [#10315](https://github.com/emqx/emqx/pull/10315) Fix crash checking `limit` and `page` parameters in `/mqtt/delayed/messages` API call. + +- [#10317](https://github.com/emqx/emqx/pull/10317) Do not expose listener level authentications before extensive verification. + +- [#10323](https://github.com/emqx/emqx/pull/10323) For security reasons, the value of the `password` field in the API examples is replaced with `******`. + + +- [#10410](https://github.com/emqx/emqx/pull/10410) Fix config check failed when gateways are configured in emqx.conf. + This issue was first introduced in v5.0.22 via [#10278](https://github.com/emqx/emqx/pull/10278), the boot-time config check was missing. + +- [#10449](https://github.com/emqx/emqx/pull/10449) Validate the ssl_options and header configurations when creating authentication http (`authn_http`). + Prior to this, incorrect `ssl` configuration could result in successful creation but the entire authn being unusable. + +- [#10455](https://github.com/emqx/emqx/pull/10455) Fixed an issue that could cause (otherwise harmless) noise in the logs. + + During some particularly slow synchronous calls to bridges, some late replies could be sent to connections processes that were no longer expecting a reply, and then emit an error log like: + + ``` + 2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, mfa: emqx_channel:handle_info/2, line: 1278, peername: 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, info: {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, ...">>}} + ``` + + Those logs are harmless, but they could flood and worry the users without need. + +- [#10548](https://github.com/emqx/emqx/pull/10548) Fixed a race condition in the HTTP driver that would result in an error rather than a retry of the request. + Related fix in the driver: https://github.com/emqx/ehttpc/pull/45 + +- [#10201](https://github.com/emqx/emqx/pull/10201) In TDengine, removed the redundant database name from the SQL template. + +- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse has got a fix that makes the error message better when users click the test button in the settings dialog. + +- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured Clickhouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed + +- [#10438](https://github.com/emqx/emqx/pull/10438) Fix some configuration item terminology errors in the DynamoDB data bridge: + + - Changed `database` to `table` + - Changed `username` to `aws_access_key_id` + - Changed `password` to `aws_secret_access_key` diff --git a/changes/e5.0.3.zh.md b/changes/e5.0.3.zh.md new file mode 100644 index 000000000..12139700f --- /dev/null +++ b/changes/e5.0.3.zh.md @@ -0,0 +1,60 @@ +# e5.0.3 + +## 增强 + +- [#10077](https://github.com/emqx/emqx/pull/10077) 增加对 QUIC TLS 密码保护证书文件的支持。 + +- [#10278](https://github.com/emqx/emqx/pull/10278) 重构所有网关的源码目录结构。 + +- [#10318](https://github.com/emqx/emqx/pull/10318) 现在，规则引擎语言的 FROM 子句支持使用双引号（"）和单引号（'）括起来的字符串。 + +- [#10140](https://github.com/emqx/emqx/pull/10140) 支持 Cassandra 数据桥接。在当前阶段： + - 仅支持 Cassandra 3.x 版本，暂不支持 4.x。 + +- [#10143](https://github.com/emqx/emqx/pull/10143) 为数据桥接增加 `RocketMQ` 支持。 + +- [#10294](https://github.com/emqx/emqx/pull/10294) 在配置 MongoDB 桥时，现在可以使用 ${var} 语法来引用消息负载中的字段，以便动态选择要插入的集合。 + + + +## 修复 + +- [#10172](https://github.com/emqx/emqx/pull/10172) 修复错误的默认 ACL 规则，之前是： + ``` + {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. + ``` + 但执行正则表达式的匹配应该使用 `{re, "^dashboard$"}`： + ``` + {allow, {username, {re, "^dashboard$"}}, subscribe, ["$SYS/#"]}. + ``` + +- [#10174](https://github.com/emqx/emqx/pull/10174) 依赖库 `esockd` 从 5.9.4 升级到 5.9.6。 + 修复了一个不必要的错误日志。如果连接在 proxy protocol 包头还没有发送前就关闭了， 则不打印错误日志。 + +- [#10211](https://github.com/emqx/emqx/pull/10211) 隐藏 `broker.broker_perf` 配置项，不再在 配置和 API 的文档中展示。 + `route_lock_type` 和 `trie_compaction` 这两个配置项很少使用，且需要全集群重启才能生效，不适合暴露给用户。 + 详细对比： https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions + +- [#10242](https://github.com/emqx/emqx/pull/10242) 修复log数据字段名称冲突。 + 在这个修复之前，一些调试日志可能会报告一个错误的Erlang PID，这可能会影响会话接管问题的故障调查。 + +- [#10257](https://github.com/emqx/emqx/pull/10257) 修复 LwM2M 网关 `auto_observe` 不工作的问题。 + + 在修复之前，下发的 OBSERVE 请求没有 Token 从而导致 LwM2M 客户端无法处理的失败。 + + 修复后，能正确监听 LwM2M 携带的资源列表、和会忽略未知的资源，并打印以下日志： + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) 优化启动失败的错误日志。 + 如果 EMQX 因为损坏的配置文件无法启动时，不会再打印过多的错误日志，也不再生成 crash.dump 文件。 + +- [#10317](https://github.com/emqx/emqx/pull/10317) 在大量验证完成前不暴露监听器级的认证功能。 + +- [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因，将 API 示例中 `password` 字段的值，统一更换为 `******`。 + + +- [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 桥接的 SQL 模板中，删除了多余的数据库表名。 + +- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse 已经修复了一个问题，当用户在设置对话框中点击测试按钮时，错误信息会更清晰。 From 644a39520ac14b14fc3a8f24bba7b922c42c8269 Mon Sep 17 00:00:00 2001 From: Ivan Dyachkov Date: Fri, 5 May 2023 16:10:10 +0200 Subject: [PATCH 3/6] chore: update e5.0.3.en.md changelog --- changes/e5.0.3.en.md | 143 ++++++++++++++++--------------------------- 1 file changed, 53 insertions(+), 90 deletions(-) diff --git a/changes/e5.0.3.en.md b/changes/e5.0.3.en.md index ecef80a31..1d23384ac 100644 --- a/changes/e5.0.3.en.md +++ b/changes/e5.0.3.en.md @@ -2,64 +2,54 @@ ## Enhancements -- [#10077](https://github.com/emqx/emqx/pull/10077) Add support for QUIC TLS password protected certificate file. - - - [#10128](https://github.com/emqx/emqx/pull/10128) Add support for OCSP stapling for SSL MQTT listeners. -- [#10156](https://github.com/emqx/emqx/pull/10156) Change the priority of the configuration: - 1. If it is a new installation of EMQX, the priority of configuration is `ENV > emqx.conf > HTTP API`. - 2. If EMQX is upgraded from an old version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then the configuration priority remains the same as before. That is, `HTTP API > ENV > emqx.conf`. +- [#10156](https://github.com/emqx/emqx/pull/10156) Change the configuration overlay order: - Deprecated data/configs/local-override.conf. + If it is a new installation of EMQX, emqx.conf + Environment variables overlays on top of API Updated Configs (cluster.hocon) - Stabilizing the HTTP API for hot updates. + If EMQX is upgraded from an older version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then it’s the same as before, that is cluster-override.conf overlays on top of emqx.conf + Environment variables. + + Please note that data/configs/cluster-override.conf is considered deprecated. After upgrade, you are encouraged to update emqx.conf to delete configs which are overridden by cluster-override.conf and move the configs in cluster-override.conf to cluster.hocon. + After upgrade, EMQX will continue to read local-override.conf (if it exists) as before, but you are encouraged to merge the configs to emqx.conf. - [#10164](https://github.com/emqx/emqx/pull/10164) Add CRL check support for TLS MQTT listeners. -- [#10206](https://github.com/emqx/emqx/pull/10206) Decouple the query mode from the underlying call mode for buffer - workers. +- [#10207](https://github.com/emqx/emqx/pull/10207) Improve OpenAPI (swagger) document readability. Prior to this change, there were a few `summary` docs which are lengthy and lack of translation, now it makes use of the more concise `label` field from schema i18n database instead. - Prior to this change, setting the query mode of a resource - such as a bridge to `sync` would force the buffer to call the - underlying connector in a synchronous way, even if it supports async - calls. - -- [#10207](https://github.com/emqx/emqx/pull/10207) Use 'label' from i18n file as 'summary' in OpenAPI spec. - -- [#10210](https://github.com/emqx/emqx/pull/10210) Unregister Mnesia post commit hook when Mria is being stopped. - This fixes hook failures occasionally occurring on stopping/restarting Mria. - - [Mria PR](https://github.com/emqx/mria/pull/133) +- [#10210](https://github.com/emqx/emqx/pull/10210) Eliminated a few harmless error level logs. + Prior to this change, there might be some Mnesia callback (hook) failures occasionally occurring when stopping/restarting Mria. + Now the callbacks (hooks) are unregistered prior to stop. See also [Mria PR](https://github.com/emqx/mria/pull/133). - [#10224](https://github.com/emqx/emqx/pull/10224) Add the option to customize `clusterIP` in Helm chart, so that a user may set it to a fixed IP. -- [#10263](https://github.com/emqx/emqx/pull/10263) Add command 'eval-ex' for Elixir expression evaluation. +- [#10263](https://github.com/emqx/emqx/pull/10263) Add command `eval-ex` for Elixir expression evaluation. - [#10278](https://github.com/emqx/emqx/pull/10278) Refactor the directory structure of all gateways. -- [#10306](https://github.com/emqx/emqx/pull/10306) Add support for `async` query mode for most bridges. +- [#10206](https://github.com/emqx/emqx/pull/10206) Support async query mode for all data bridges. - Before this change, some bridges (Cassandra, MongoDB, MySQL, Postgres, Redis, RocketMQ, TDengine) were only allowed to be created with a `sync` query mode. + Prior to this change, setting the query mode of a resource such as a bridge to sync would force the buffer to call the underlying connector in a synchronous way, even if it supports async calls. -- [#10318](https://github.com/emqx/emqx/pull/10318) Now, the rule engine language's FROM clause supports both strings enclosed in double quotes (") and single quotes ('). +- [#10306](https://github.com/emqx/emqx/pull/10306) Add support for async query mode for most bridges. + + This is a follow-up change after [#10206](https://github.com/emqx/emqx/pull/10206). Before this change, some bridges (Cassandra, MongoDB, MySQL, Postgres, Redis, RocketMQ, TDengine) were only allowed to be created with a sync query mode. Now async mode is also supported. + +- [#10318](https://github.com/emqx/emqx/pull/10318) Prior to this enhancement, only double quotes (") were allowed in rule engine SQL language's FROM clause. Now it also supports single quotes ('). - [#10336](https://github.com/emqx/emqx/pull/10336) Add `/rule_engine` API endpoint to manage configuration of rule engine. -- [#10354](https://github.com/emqx/emqx/pull/10354) More specific error messages when configure with bad max_heap_size value. - Log current value and the max value when the `message_queue_too_long` error is thrown. +- [#10354](https://github.com/emqx/emqx/pull/10354) More specific error messages when configure with `bad max_heap_size` value. Log current value and the max value when the `message_queue_too_long` error is thrown. -- [#10358](https://github.com/emqx/emqx/pull/10358) Hide `flapping_detect/conn_congestion/stats` configuration. - Deprecate `flapping_detect.enable`. +- [#10358](https://github.com/emqx/emqx/pull/10358) Hide `flapping_detect/conn_congestion/stats` configuration. Deprecate `flapping_detect.enable`. - [#10359](https://github.com/emqx/emqx/pull/10359) Metrics now are not implicitly collected in places where API handlers don't make any use of them. Instead, a separate backplane RPC gathers cluster-wide metrics. -- [#10373](https://github.com/emqx/emqx/pull/10373) Deprecate the trace.payload_encode configuration. - Add payload_encode=[text,hidden,hex] option when creating a trace via HTTP API. +- [#10373](https://github.com/emqx/emqx/pull/10373) Deprecate the `trace.payload_encode` configuration. Add `payload_encode=[text,hidden,hex]` option when creating a trace via HTTP API. - [#10381](https://github.com/emqx/emqx/pull/10381) Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. -- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(rule_engine/bridge/authz/authn) from configuration files and documentation. +- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(`rule_engine/bridge`) from configuration files and documentation. - [#10391](https://github.com/emqx/emqx/pull/10391) Hide a large number of advanced options to simplify the configuration file. @@ -69,98 +59,69 @@ in `node` and `dashboard` section, [#10358](https://github.com/emqx/emqx/pull/10358), [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385). -- [#10404](https://github.com/emqx/emqx/pull/10404) Change the default queue mode for buffer workers to `memory_only`. - Before this change, the default queue mode was `volatile_offload`. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations. +- [#10404](https://github.com/emqx/emqx/pull/10404) Change the default queue mode for buffer workers to `memory_only`. Before this change, the default queue mode was `volatile_offload`. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations. -- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate `Cassandra` into `bridges` as a new backend. At the current stage: - - Only support Cassandra version 3.x, not yet 4.x. +- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate Cassandra into bridges as a new backend. At the current stage: -- [#10143](https://github.com/emqx/emqx/pull/10143) Add `RocketMQ` data integration bridge. + Only support Cassandra version 3.x, not yet 4.x. -- [#10165](https://github.com/emqx/emqx/pull/10165) Support escaped special characters in InfluxDB data bridge write_syntax. - This update allows to use escaped special characters in string elements in accordance with InfluxDB line protocol. +- [#10143](https://github.com/emqx/emqx/pull/10143) Add RocketMQ data integration bridge. -- [#10294](https://github.com/emqx/emqx/pull/10294) When configuring a MongoDB bridge, you can now use the ${var} syntax to reference fields in the message payload within the collection field. This enables you to select the collection to insert data into dynamically. +- [#10165](https://github.com/emqx/emqx/pull/10165) Support escaped special characters in InfluxDB data bridge `write_syntax`. This update allows to use escaped special characters in string elements in accordance with InfluxDB line protocol. -- [#10337](https://github.com/emqx/emqx/pull/10337) Add schema registry feature. +- [#10211](https://github.com/emqx/emqx/pull/10211) Hide `broker.broker_perf` config and API documents. The two configs `route_lock_type` and `trie_compaction` are rarely used and requires a full cluster restart to take effect. They are not suitable for being exposed to users. Detailed changes can be found here: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions. - With schema registry, one can encode and decode special serialization formats in payloads when transforming messages in Rule Engine. Currently, only [Apache Avro](https://avro.apache.org/) is supported. +- [#10294](https://github.com/emqx/emqx/pull/10294) When configuring a MongoDB bridge, you can now use the `${field}` syntax to reference fields in the message. This enables you to select the collection to insert data into dynamically. - [#10363](https://github.com/emqx/emqx/pull/10363) Implement Microsoft SQL Server bridge. -- [#10573](https://github.com/emqx/emqx/pull/10573) Improved performance of Webhook bridge when using synchronous query mode. - This also should improve the performance of other bridges when they are configured with no batching. +- [#10573](https://github.com/emqx/emqx/pull/10573) Improved performance of Webhook bridge when using synchronous query mode. This also should improve the performance of other bridges when they are configured with no batching. ## Bug Fixes -- [#10145](https://github.com/emqx/emqx/pull/10145) Fix `bridges` API to report error conditions for a failing bridge as - `status_reason`. Also when creating an alarm for a failing resource we include - this error condition with the alarm's message. +- [#10145](https://github.com/emqx/emqx/pull/10145) Add field `status_reason` to `GET /bridges/:id` response in case this bridge is in status `disconnected` if internal health-check reports an error condition. Include this same error condition in message when creating an alarm for a failing bridge. -- [#10172](https://github.com/emqx/emqx/pull/10172) Fix the incorrect default ACL rule, which was: - ``` - {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. - ``` +- [#10172](https://github.com/emqx/emqx/pull/10172) Fix the incorrect regular expression in default ACL rule to allow specify username(dashboard) to subscribe `$SYS/#`. - However, it should use `{re, "^dashboard$"}` to perform a regular expression match: - ``` - {allow, {username, {re,"^dashboard$"}}, subscribe, ["$SYS/#"]}. - ``` +- [#10174](https://github.com/emqx/emqx/pull/10174) Upgrade library `esockd` from 5.9.4 to 5.9.6. Fix an unnecessary error level logging when a connection is closed before proxy protocol header is sent by the proxy. -- [#10174](https://github.com/emqx/emqx/pull/10174) Upgrade library `esockd` from 5.9.4 to 5.9.6. - Fix an unnecessary error level logging when a connection is closed before proxy protocol header is sent by the proxy. +- [#10195](https://github.com/emqx/emqx/pull/10195) Add labels to API schemas where description contains raw HTML, which would break formatting of generated documentation otherwise. -- [#10195](https://github.com/emqx/emqx/pull/10195) Add labels to API schemas where description contains HTML and breaks formatting of generated documentation otherwise. +- [#10196](https://github.com/emqx/emqx/pull/10196) Use lower-case for schema summaries and descriptions to be used in menu of generated online documentation. -- [#10196](https://github.com/emqx/emqx/pull/10196) Use lower-case for schema summaries and descritptions to be used in menu of generated online documentation. +- [#10209](https://github.com/emqx/emqx/pull/10209) Fix bug where a last will testament (LWT) message could be published when kicking out a banned client. -- [#10209](https://github.com/emqx/emqx/pull/10209) Fix bug where a last will testament (LWT) message could be published - when kicking out a banned client. +- [#10225](https://github.com/emqx/emqx/pull/10225) Allow installing a plugin if its name matches the beginning of another (already installed) plugin name. For example: if plugin `emqx_plugin_template_a` is installed, it must not block installing plugin `emqx_plugin_template`. -- [#10211](https://github.com/emqx/emqx/pull/10211) Hide `broker.broker_perf` config and API documents. - The two configs `route_lock_type` and `trie_compaction` are rarely used and requires a full cluster restart to take effect. They are not suitable for being exposed to users. - Detailed changes can be found here: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions +- [#10226](https://github.com/emqx/emqx/pull/10226) Handle validation error in `/bridges` API and return `400` instead of `500`. -- [#10225](https://github.com/emqx/emqx/pull/10225) Allow installing a plugin if its name matches the beginning of another (already installed) plugin name. - For example: if plugin "emqx_plugin_template_a" is installed, it must not block installing plugin "emqx_plugin_template". - -- [#10226](https://github.com/emqx/emqx/pull/10226) Don't crash on validation error in `/bridges` API, return `400` instead. - -- [#10242](https://github.com/emqx/emqx/pull/10242) Fixed a log data field name clash. - Piror to this fix, some debug logs may report a wrong Erlang PID which may affect troubleshooting session takeover issues. +- [#10242](https://github.com/emqx/emqx/pull/10242) Fixed a log data field name clash. Prior to this fix, some debug logs may report a wrong Erlang PID which may affect troubleshooting session takeover issues. - [#10257](https://github.com/emqx/emqx/pull/10257) Fixed the issue where `auto_observe` was not working in LwM2M Gateway. - Before the fix, OBSERVE requests were sent without a token, causing failures - that LwM2M clients could not handle. + Before the fix, `OBSERVE` requests were sent without a token, causing failures that LwM2M clients could not handle. - After the fix, LwM2M Gateway can correctly observe the resource list carried by - client, furthermore, unknown resources will be ignored and printing the following - warning log: - ``` - 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition - ``` + After the fix, LwM2M Gateway can correctly observe the resource list carried by client, furthermore, unknown resources will be ignored and printing the following warning log: -- [#10286](https://github.com/emqx/emqx/pull/10286) Enhance logging behaviour during boot failure. - When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated. +``` +2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition +``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) Enhance logging behaviour during boot failure. When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated. - [#10297](https://github.com/emqx/emqx/pull/10297) Keeps `eval` command backward compatible with v4 by evaluating only Erlang expressions, even on Elixir node. For Elixir expressions, use `eval-ex` command. -- [#10300](https://github.com/emqx/emqx/pull/10300) Fixed an issue where a build made with Elixir could not receive uploaded plugins until the `plugins` folder was created manually to receive the uploaded files. +- [#10300](https://github.com/emqx/emqx/pull/10300) Fixed issue with Elixir builds that prevented plugins from being configured via environment variables. - [#10315](https://github.com/emqx/emqx/pull/10315) Fix crash checking `limit` and `page` parameters in `/mqtt/delayed/messages` API call. - [#10317](https://github.com/emqx/emqx/pull/10317) Do not expose listener level authentications before extensive verification. -- [#10323](https://github.com/emqx/emqx/pull/10323) For security reasons, the value of the `password` field in the API examples is replaced with `******`. - +- [#10323](https://github.com/emqx/emqx/pull/10323) For security reasons, the value of the password field in the API examples is replaced with `******`. - [#10410](https://github.com/emqx/emqx/pull/10410) Fix config check failed when gateways are configured in emqx.conf. This issue was first introduced in v5.0.22 via [#10278](https://github.com/emqx/emqx/pull/10278), the boot-time config check was missing. -- [#10449](https://github.com/emqx/emqx/pull/10449) Validate the ssl_options and header configurations when creating authentication http (`authn_http`). - Prior to this, incorrect `ssl` configuration could result in successful creation but the entire authn being unusable. - - [#10455](https://github.com/emqx/emqx/pull/10455) Fixed an issue that could cause (otherwise harmless) noise in the logs. During some particularly slow synchronous calls to bridges, some late replies could be sent to connections processes that were no longer expecting a reply, and then emit an error log like: @@ -171,14 +132,16 @@ Those logs are harmless, but they could flood and worry the users without need. +- [#10449](https://github.com/emqx/emqx/pull/10449) Validate the `ssl_options` and `header` configurations when creating authentication http (`authn_http`). Prior to this, incorrect `ssl` configuration could result in successful creation but the entire authn being unusable. + - [#10548](https://github.com/emqx/emqx/pull/10548) Fixed a race condition in the HTTP driver that would result in an error rather than a retry of the request. Related fix in the driver: https://github.com/emqx/ehttpc/pull/45 -- [#10201](https://github.com/emqx/emqx/pull/10201) In TDengine, removed the redundant database name from the SQL template. +- [#10201](https://github.com/emqx/emqx/pull/10201) In TDengine data bridge, removed the redundant database name from the SQL template. -- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse has got a fix that makes the error message better when users click the test button in the settings dialog. +- [#10270](https://github.com/emqx/emqx/pull/10270) ClickHouse data bridge has got a fix that makes the error message better when users click the test button in the settings dialog. -- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured Clickhouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed +- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured ClickHouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed - [#10438](https://github.com/emqx/emqx/pull/10438) Fix some configuration item terminology errors in the DynamoDB data bridge: From 8ecd6364d1c5b5cd951faab00abad1b77a953a60 Mon Sep 17 00:00:00 2001 From: Ivan Dyachkov Date: Fri, 5 May 2023 16:42:44 +0200 Subject: [PATCH 4/6] chore: update e5.0.3.zh.md changelog and minor changes in e5.0.3.en.md --- changes/e5.0.3.en.md | 20 +++---- changes/e5.0.3.zh.md | 136 ++++++++++++++++++++++++++++++++----------- 2 files changed, 110 insertions(+), 46 deletions(-) diff --git a/changes/e5.0.3.en.md b/changes/e5.0.3.en.md index 1d23384ac..e83b677a1 100644 --- a/changes/e5.0.3.en.md +++ b/changes/e5.0.3.en.md @@ -6,12 +6,12 @@ - [#10156](https://github.com/emqx/emqx/pull/10156) Change the configuration overlay order: - If it is a new installation of EMQX, emqx.conf + Environment variables overlays on top of API Updated Configs (cluster.hocon) + If it is a new installation of EMQX, `emqx.conf` + Environment variables overlays on top of API Updated Configs (`cluster.hocon`) - If EMQX is upgraded from an older version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then it’s the same as before, that is cluster-override.conf overlays on top of emqx.conf + Environment variables. + If EMQX is upgraded from an older version (i.e., the `cluster-override.conf` file still exists in EMQX's `data` directory), then it’s the same as before, that is `cluster-override.conf` overlays on top of `emqx.conf` + Environment variables. - Please note that data/configs/cluster-override.conf is considered deprecated. After upgrade, you are encouraged to update emqx.conf to delete configs which are overridden by cluster-override.conf and move the configs in cluster-override.conf to cluster.hocon. - After upgrade, EMQX will continue to read local-override.conf (if it exists) as before, but you are encouraged to merge the configs to emqx.conf. + Please note that `data/configs/cluster-override.conf` is considered deprecated. After upgrade, you are encouraged to update `emqx.conf` to delete configs which are overridden by `cluster-override.conf` and move the configs in `cluster-override.conf` to `cluster.hocon`. + After upgrade, EMQX will continue to read `local-override.conf` (if it exists) as before, but you are encouraged to merge the configs to `emqx.conf`. - [#10164](https://github.com/emqx/emqx/pull/10164) Add CRL check support for TLS MQTT listeners. @@ -49,7 +49,7 @@ - [#10381](https://github.com/emqx/emqx/pull/10381) Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. -- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(`rule_engine/bridge`) from configuration files and documentation. +- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(`rule_engine/bridge/authz/authn`) from configuration files and documentation. - [#10391](https://github.com/emqx/emqx/pull/10391) Hide a large number of advanced options to simplify the configuration file. @@ -61,9 +61,7 @@ - [#10404](https://github.com/emqx/emqx/pull/10404) Change the default queue mode for buffer workers to `memory_only`. Before this change, the default queue mode was `volatile_offload`. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations. -- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate Cassandra into bridges as a new backend. At the current stage: - - Only support Cassandra version 3.x, not yet 4.x. +- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate Cassandra into bridges as a new backend. At the current stage only support Cassandra version 3.x, not yet 4.x. - [#10143](https://github.com/emqx/emqx/pull/10143) Add RocketMQ data integration bridge. @@ -103,9 +101,9 @@ After the fix, LwM2M Gateway can correctly observe the resource list carried by client, furthermore, unknown resources will be ignored and printing the following warning log: -``` -2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition -``` + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` - [#10286](https://github.com/emqx/emqx/pull/10286) Enhance logging behaviour during boot failure. When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated. diff --git a/changes/e5.0.3.zh.md b/changes/e5.0.3.zh.md index 12139700f..0ebae3e9c 100644 --- a/changes/e5.0.3.zh.md +++ b/changes/e5.0.3.zh.md @@ -1,60 +1,126 @@ # e5.0.3 -## 增强 +## 优化 -- [#10077](https://github.com/emqx/emqx/pull/10077) 增加对 QUIC TLS 密码保护证书文件的支持。 +- [#10278](https://github.com/emqx/emqx/pull/10278) SSL MQTT监听器增加对 OCSP Stapling 的支持。 -- [#10278](https://github.com/emqx/emqx/pull/10278) 重构所有网关的源码目录结构。 +- [#10156](https://github.com/emqx/emqx/pull/10156) 调整配置文件覆盖顺序机制 -- [#10318](https://github.com/emqx/emqx/pull/10318) 现在，规则引擎语言的 FROM 子句支持使用双引号（"）和单引号（'）括起来的字符串。 + 对于新安装的 EMQX，emqx.conf 和环境变量中的配置会覆盖 API 传入的配置（即 `cluster.hocon` 中的配置） -- [#10140](https://github.com/emqx/emqx/pull/10140) 支持 Cassandra 数据桥接。在当前阶段： - - 仅支持 Cassandra 3.x 版本，暂不支持 4.x。 + 对于从旧版本升级的 EMQX（即 `data` 文件夹中包含 `cluster-override.conf` 文件），保留之前的覆盖规则， 即 `cluster-override.conf` 中的配置会覆盖 `emqx.conf` 和环境变量的配置。 -- [#10143](https://github.com/emqx/emqx/pull/10143) 为数据桥接增加 `RocketMQ` 支持。 + 注意：`data/configs/cluster-override.conf` 已弃用。升级后，建议您在 `emqx.conf` 中重新配置之前被 `cluster-override.conf` 覆盖的配置项，并将 cluster-override.conf 中的配置迁移到 `cluster.hocon` 中。 -- [#10294](https://github.com/emqx/emqx/pull/10294) 在配置 MongoDB 桥时，现在可以使用 ${var} 语法来引用消息负载中的字段，以便动态选择要插入的集合。 +升级后，EMQX将像以前一样继续读取 `local-override.conf` (如果存在的话)，但建议您将配置合并到 `emqx.conf` 中。 +- [#10164](https://github.com/emqx/emqx/pull/10164) TLS MQTT 监听器增加对 CRL 检查的支持。 +- [#10207](https://github.com/emqx/emqx/pull/10207) 提高 OpenAPI (swagger) 文档的可读性。 在此更改之前，文档中有一些 `Summary` 字段冗长且缺乏翻译，现在使用了 i18n 数据库中更简洁的 `label` 字段。 + +- [#10210](https://github.com/emqx/emqx/pull/10210) 解决停止/重启 Mria 时 Mnesia callback 可能出现的问题。优化后，当 Mria 被停止前会取消 Mnesia callback 的注册。详情见 Mria PR + +- [#10224](https://github.com/emqx/emqx/pull/10224) 在 Helm 图表中增加自定义 `clusterIP` 选项，用户可以将其设置为固定 IP。 + +- [#10263](https://github.com/emqx/emqx/pull/10263) 添加用于评估 Elixir 表达式的命令 `eval-ex`。 + +- [#10278](https://github.com/emqx/emqx/pull/10278) 重构所有网关的目录结构。 + +- [#10206](https://github.com/emqx/emqx/pull/10206) 所有数据桥接支持异步查询模式。 + +优化前，如将某项资源（如数据桥接）的查询模式设为 sync（同步模式），缓存将以同步的模式调用底层连接器，即使它支持异步调用。 + +- [#10306](https://github.com/emqx/emqx/pull/10306) 大多数数据桥接支持 async 查询模式。 + 这是 [#10206](https://github.com/emqx/emqx/pull/10206) 的后续优化, 优化前，Cassandra、MongoDB、MySQL、Postgres、Redis、RocketMQ、TDengine 等数据桥接只支持同步查询模式。 + +- [#10318](https://github.com/emqx/emqx/pull/10318) 规则引擎中的 FROM 语句新增支持由引号（'）包裹的字符串。 + +- [#10336](https://github.com/emqx/emqx/pull/10336) 添加 API Endpoint `/rule_engine`，用以管理规则引擎的配置。 + +- [#10354](https://github.com/emqx/emqx/pull/10354) 优化 `max_heap_size` 配置错误时的报错信息。当发生 `message_queue_too_long` 报错时，会在日志文件中记录当前值和最大值。 + +- [#10358](https://github.com/emqx/emqx/pull/10358) 隐藏 `flapping_detect/conn_congestion/stats` 配置。弃用 `flapping_detect.enable` 配置项。 + +- [#10359](https://github.com/emqx/emqx/pull/10359) 通过独立的 RPC 收集针对集群级别的指标，不再隐式收集不被 API 调用的指标。 + +- [#10373](https://github.com/emqx/emqx/pull/10373) 废弃 `trace.payload_encode` 配置项。可以在通过 HTTP API 创建的日志追踪时使用 `trace.payload_encode = [text, hidden, hex]` 字段替代。 + +- [#10381](https://github.com/emqx/emqx/pull/10381) 隐藏 `auto_subscribe` 配置项，后续将只能通过 HTTP API 来修改自动订阅规则。 + +- [#10385](https://github.com/emqx/emqx/pull/10385) 从配置文件和文档中隐藏数据项（`rule_engine/bridge/authz/authn`）。 + +- [#10391](https://github.com/emqx/emqx/pull/10391) 简化配置文件并隐藏大量的配置项，包括 `rewrite`、`topic_metric`、`persistent_session_store`、`overload_protection`、`flapping_detect`、`conn_congestion`、`stats`、`auto_subscribe`、`broker_perf`、`shared_subscription_group`、`slow_subs`、`ssl_options.user_lookup_fun`、 `node` 和 `dashboard` 相关的部分高级配置项，[#10358](https://github.com/emqx/emqx/pull/10358), [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385)。 + +- [#10404](https://github.com/emqx/emqx/pull/10404) 将缓冲区工作线程的默认队列模式更改为 `memory_only`。在此优化前，默认队列模式为 `volatile_offload`，当消息速率很高，资源无法满足该需求时，缓冲区性能会由于频繁的磁盘操作而受影响。 + +- [#10140](https://github.com/emqx/emqx/pull/10140) 新增 Cassandra 数据桥接，目前仅支持 Cassandra 3.x 版本，暂不支持 4.x 版本。 + +- [#10143](https://github.com/emqx/emqx/pull/10143) 新增 RocketMQ 数据桥接。 + +- [#10165](https://github.com/emqx/emqx/pull/10165) InfluxDB 数据桥接中的 `write_syntax` 中支持转义特殊字符。优化后，用户可根据 InfluxDB Line Protocol 在字符串中使用经转义的特殊字符。 + +- [#10211](https://github.com/emqx/emqx/pull/10211) 隐藏 `broker.broker_perf` 配置和相关 API 文档。其中的两个配置项 `route_lock_type` 和 `trie_compaction` 很少使用，而且需要重新启动整个集群才能生效，不必要暴露给用户。更多信息可阅读：https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions 。 + +- [#10294](https://github.com/emqx/emqx/pull/10294) 配置 MongoDB 数据桥接时，现支持通过占位符 `${field}` 语法来引用消息中的字段，从而动态地选择要插入的数据集合。 + +- [#10363](https://github.com/emqx/emqx/pull/10363) 新增 Microsoft SQL Server 数桥接。 + +- [#10573](https://github.com/emqx/emqx/pull/10573) 提升了 WebHook 在同步请求模式下的性能表现，以及其他数据桥接在未配置批处理时的性能表现。 ## 修复 -- [#10172](https://github.com/emqx/emqx/pull/10172) 修复错误的默认 ACL 规则，之前是： - ``` - {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. - ``` - 但执行正则表达式的匹配应该使用 `{re, "^dashboard$"}`： - ``` - {allow, {username, {re, "^dashboard$"}}, subscribe, ["$SYS/#"]}. - ``` +- [#10145](https://github.com/emqx/emqx/pull/10145) 在针对 `GET /bridges/:id` 的API 调用响应中，如桥接的状态为断开，且内部健康检查返回错误，添加 `status_reason` 字段说明错误原因。在相应的告警信息中，同样增加 `status_reason` 字段说明断开原因。 -- [#10174](https://github.com/emqx/emqx/pull/10174) 依赖库 `esockd` 从 5.9.4 升级到 5.9.6。 - 修复了一个不必要的错误日志。如果连接在 proxy protocol 包头还没有发送前就关闭了， 则不打印错误日志。 +- [#10172](https://github.com/emqx/emqx/pull/10172) 修复默认ACL规则中不正确的正则表达式，从而允许 dashboard 用户名订阅 `$SYS/#` 主题 -- [#10211](https://github.com/emqx/emqx/pull/10211) 隐藏 `broker.broker_perf` 配置项，不再在 配置和 API 的文档中展示。 - `route_lock_type` 和 `trie_compaction` 这两个配置项很少使用，且需要全集群重启才能生效，不适合暴露给用户。 - 详细对比： https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions +- [#10174](https://github.com/emqx/emqx/pull/10174) 将库 `esockd` 从 5.9.4 升级至 5.9.6。如连接在代理发送代理协议头之前关闭，将不再产生的一条错误级别日志。 -- [#10242](https://github.com/emqx/emqx/pull/10242) 修复log数据字段名称冲突。 - 在这个修复之前，一些调试日志可能会报告一个错误的Erlang PID，这可能会影响会话接管问题的故障调查。 +- [#10195](https://github.com/emqx/emqx/pull/10195) 对包含 HTML 的 API Schema 添加标签，解决之前会破坏生成文档格式的问题。 -- [#10257](https://github.com/emqx/emqx/pull/10257) 修复 LwM2M 网关 `auto_observe` 不工作的问题。 +- [#10196](https://github.com/emqx/emqx/pull/10196) 针对用于生成在线文档菜单中的模式摘要和描述，使用小写字母。 - 在修复之前，下发的 OBSERVE 请求没有 Token 从而导致 LwM2M 客户端无法处理的失败。 +- [#10209](https://github.com/emqx/emqx/pull/10209) 修复在断开禁用客户端时，此类客户端仍可发布遗嘱消息的错误。 - 修复后，能正确监听 LwM2M 携带的资源列表、和会忽略未知的资源，并打印以下日志： - ``` - 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition - ``` +- [#10225](https://github.com/emqx/emqx/pull/10225) 对于名称与已安装的插件开头相同的插件，用户仍可继续安装。例如：如果插件 `emqx_plugin_template_a` 已安装，用户仍可安装名为 `emqx_plugin_template` 的插件。 -- [#10286](https://github.com/emqx/emqx/pull/10286) 优化启动失败的错误日志。 - 如果 EMQX 因为损坏的配置文件无法启动时，不会再打印过多的错误日志，也不再生成 crash.dump 文件。 +- [#10226](https://github.com/emqx/emqx/pull/10226) 在 `/bridges` API 验证错误时，返回 `400` 而非 `500`。 -- [#10317](https://github.com/emqx/emqx/pull/10317) 在大量验证完成前不暴露监听器级的认证功能。 +- [#10242](https://github.com/emqx/emqx/pull/10242) 修复日志中数据字段名称冲突。修复前，一些调试日志可能会报告错误的 Erlang PID，影响解决会话接管类问题。 -- [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因，将 API 示例中 `password` 字段的值，统一更换为 `******`。 +- [#10257](https://github.com/emqx/emqx/pull/10257) 修复 LwM2M 网关中 `auto_observe` 无法正常工作的问题。 + 修复前，在发送 `OBSERVE` 请求时没有发送 token，导致 LwM2M 网关无法处理客户端请求。 -- [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 桥接的 SQL 模板中，删除了多余的数据库表名。 + 修复后，LwM2M 网关可以正确观察到客户端携带的资源列表，此外，未知资源将被忽并打印以下警告日志： -- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse 已经修复了一个问题，当用户在设置对话框中点击测试按钮时，错误信息会更清晰。 + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) 优化 EMQX 启动失败时的日志记录行为。当 EMQX 由于配置文件破坏无法启动时，不会再产生过多的日志记录，也不会再产生崩溃转储文件。 + +- [#10297](https://github.com/emqx/emqx/pull/10297) 通过仅评估 Erlang 表达式实现 `eval` 命令与 v4 版本的向后兼容，该更新同样适用于 Elixir 节点。对于 Elixir 表达式，请使用 `eval-ex` 命令。 + +- [#10300](https://github.com/emqx/emqx/pull/10300) 针对通过 Elixir 构建的项目，修复无法通过环境变量配置插件的问题。 + +- [#10315](https://github.com/emqx/emqx/pull/10315) 修复在 `/mqtt/delayed/messages` API 调用中，在检查 `limit` 和 `page` 参数时的崩溃问题。 + +- [#10317](https://github.com/emqx/emqx/pull/10317) 在经过充分验证前，隐藏监听器级别的认证信息。 + +- [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因，API 示例中 `password` 字段的值被替换为 `******`。 + +- [#10449](https://github.com/emqx/emqx/pull/10449) 在通过 HTTP 服务（`authn_http`）创建身份验证时，将进行 `ssl_options` 和 `header` 配置验证。在此修复前，用户通过错误的 ssl 配置也可成功创建身份验证，但该验证整体不生效。 + +- [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 数据桥接中，移除 SQL 模板中冗余的数据库名称。 + +- [#10270](https://github.com/emqx/emqx/pull/10270) 在创建 ClickHouse 数据桥接时，优化用户点击测试按钮时的错误信息 + +- [#10324](https://github.com/emqx/emqx/pull/10324) 针对配置有误的 ClickHouse 数据桥接，当用户尝试通过 Dashboard 重连时，将收到报错。修复前，用户不会收到报错。 + +- [#10438](https://github.com/emqx/emqx/pull/10438) 修复 DynamoDB 数据桥接中的部分术语使用错误： + + 将 `database` 修改为 `table` + + 将 `username` 修改为 `aws_access_key_id` + + 将 `password` 修改为 `aws_secret_access_key` From e7800f5a717cc8d659d47f9043d869ad2e8eeb7f Mon Sep 17 00:00:00 2001 From: JianBo He Date: Sat, 6 May 2023 13:49:26 +0800 Subject: [PATCH 5/6] chore: apply suggestions from code review Co-authored-by: yanzhiemq <125347458+yanzhiemq@users.noreply.github.com> --- changes/e5.0.3.en.md | 6 ++---- changes/e5.0.3.zh.md | 22 +++++++++++++++++++--- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/changes/e5.0.3.en.md b/changes/e5.0.3.en.md index e83b677a1..f229bb0b8 100644 --- a/changes/e5.0.3.en.md +++ b/changes/e5.0.3.en.md @@ -49,8 +49,6 @@ - [#10381](https://github.com/emqx/emqx/pull/10381) Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. -- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(`rule_engine/bridge/authz/authn`) from configuration files and documentation. - - [#10391](https://github.com/emqx/emqx/pull/10391) Hide a large number of advanced options to simplify the configuration file. That includes `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, @@ -120,7 +118,7 @@ - [#10410](https://github.com/emqx/emqx/pull/10410) Fix config check failed when gateways are configured in emqx.conf. This issue was first introduced in v5.0.22 via [#10278](https://github.com/emqx/emqx/pull/10278), the boot-time config check was missing. -- [#10455](https://github.com/emqx/emqx/pull/10455) Fixed an issue that could cause (otherwise harmless) noise in the logs. +- [#10533](https://github.com/emqx/emqx/pull/10533) Fixed an issue that could cause (otherwise harmless) noise in the logs. During some particularly slow synchronous calls to bridges, some late replies could be sent to connections processes that were no longer expecting a reply, and then emit an error log like: @@ -139,7 +137,7 @@ - [#10270](https://github.com/emqx/emqx/pull/10270) ClickHouse data bridge has got a fix that makes the error message better when users click the test button in the settings dialog. -- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured ClickHouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed +- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured ClickHouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed. - [#10438](https://github.com/emqx/emqx/pull/10438) Fix some configuration item terminology errors in the DynamoDB data bridge: diff --git a/changes/e5.0.3.zh.md b/changes/e5.0.3.zh.md index 0ebae3e9c..838edf0a5 100644 --- a/changes/e5.0.3.zh.md +++ b/changes/e5.0.3.zh.md @@ -2,7 +2,7 @@ ## 优化 -- [#10278](https://github.com/emqx/emqx/pull/10278) SSL MQTT监听器增加对 OCSP Stapling 的支持。 +- [#10128](https://github.com/emqx/emqx/pull/10128) SSL MQTT监听器增加对 OCSP Stapling 的支持。 - [#10156](https://github.com/emqx/emqx/pull/10156) 调整配置文件覆盖顺序机制 @@ -47,8 +47,6 @@ - [#10381](https://github.com/emqx/emqx/pull/10381) 隐藏 `auto_subscribe` 配置项，后续将只能通过 HTTP API 来修改自动订阅规则。 -- [#10385](https://github.com/emqx/emqx/pull/10385) 从配置文件和文档中隐藏数据项（`rule_engine/bridge/authz/authn`）。 - - [#10391](https://github.com/emqx/emqx/pull/10391) 简化配置文件并隐藏大量的配置项，包括 `rewrite`、`topic_metric`、`persistent_session_store`、`overload_protection`、`flapping_detect`、`conn_congestion`、`stats`、`auto_subscribe`、`broker_perf`、`shared_subscription_group`、`slow_subs`、`ssl_options.user_lookup_fun`、 `node` 和 `dashboard` 相关的部分高级配置项，[#10358](https://github.com/emqx/emqx/pull/10358), [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385)。 - [#10404](https://github.com/emqx/emqx/pull/10404) 将缓冲区工作线程的默认队列模式更改为 `memory_only`。在此优化前，默认队列模式为 `volatile_offload`，当消息速率很高，资源无法满足该需求时，缓冲区性能会由于频繁的磁盘操作而受影响。 @@ -109,8 +107,26 @@ - [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因，API 示例中 `password` 字段的值被替换为 `******`。 +- [#10410](https://github.com/emqx/emqx/pull/10410) 针对由`emqx.conf` 配置的网关，修复配置检查失败问题。 + 此问题最早在 v5.0.22版本中由 [#10278](https://github.com/emqx/emqx/pull/10278)引入，启动时缺少配置检查。 + +- [#10533](https://github.com/emqx/emqx/pull/10533) 修复可能导致日志中的出现无害的噪音的问题。 + + 在对数据桥接进行同步调用时，一些迟到的回复可能会发送到不再期待回复的连接进程，导致产生错误日志，如： + + 2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, + mfa: emqx_channel:handle_info/2, line: 1278, peername: + 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, + info: + {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, + ...">>}} + + 这些日志是无害的，但它们可能会泛滥成灾，引起用户不必要的担心。 + - [#10449](https://github.com/emqx/emqx/pull/10449) 在通过 HTTP 服务（`authn_http`）创建身份验证时，将进行 `ssl_options` 和 `header` 配置验证。在此修复前，用户通过错误的 ssl 配置也可成功创建身份验证，但该验证整体不生效。 +- [#10548](https://github.com/emqx/emqx/pull/10548) 修复了HTTP驱动程序在竞争条件下会导致错误而不去重试的问题。 相关的驱动程序修复: [emqx/ehttpc#45](https://github.com/emqx/ehttpc/pull/45)` + - [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 数据桥接中，移除 SQL 模板中冗余的数据库名称。 - [#10270](https://github.com/emqx/emqx/pull/10270) 在创建 ClickHouse 数据桥接时，优化用户点击测试按钮时的错误信息 From 2fb4b5bcbc4eb63565e3aca13043924d199c6782 Mon Sep 17 00:00:00 2001 From: JianBo He Date: Sat, 6 May 2023 13:54:34 +0800 Subject: [PATCH 6/6] chore: fix bad indent --- changes/e5.0.3.zh.md | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/changes/e5.0.3.zh.md b/changes/e5.0.3.zh.md index 838edf0a5..71de3d779 100644 --- a/changes/e5.0.3.zh.md +++ b/changes/e5.0.3.zh.md @@ -107,25 +107,23 @@ - [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因，API 示例中 `password` 字段的值被替换为 `******`。 -- [#10410](https://github.com/emqx/emqx/pull/10410) 针对由`emqx.conf` 配置的网关，修复配置检查失败问题。 - 此问题最早在 v5.0.22版本中由 [#10278](https://github.com/emqx/emqx/pull/10278)引入，启动时缺少配置检查。 +- [#10410](https://github.com/emqx/emqx/pull/10410) 针对由`emqx.conf` 配置的网关，修复配置检查失败问题。 + 此问题最早在 v5.0.22版本中由 [#10278](https://github.com/emqx/emqx/pull/10278)引入，启动时缺少配置检查。 -- [#10533](https://github.com/emqx/emqx/pull/10533) 修复可能导致日志中的出现无害的噪音的问题。 +- [#10533](https://github.com/emqx/emqx/pull/10533) 修复可能导致日志中的出现无害的噪音的问题。 - 在对数据桥接进行同步调用时，一些迟到的回复可能会发送到不再期待回复的连接进程，导致产生错误日志，如： + 在对数据桥接进行同步调用时，一些迟到的回复可能会发送到不再期待回复的连接进程，导致产生错误日志，如： - 2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, - mfa: emqx_channel:handle_info/2, line: 1278, peername: - 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, - info: - {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, - ...">>}} + ``` + 2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, mfa: emqx_channel:handle_info/2, line: 1278, peername: 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, info: {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, ...">>}} + ``` - 这些日志是无害的，但它们可能会泛滥成灾，引起用户不必要的担心。 + 这些日志是无害的，但它们可能会泛滥成灾，引起用户不必要的担心。 - [#10449](https://github.com/emqx/emqx/pull/10449) 在通过 HTTP 服务（`authn_http`）创建身份验证时，将进行 `ssl_options` 和 `header` 配置验证。在此修复前，用户通过错误的 ssl 配置也可成功创建身份验证，但该验证整体不生效。 -- [#10548](https://github.com/emqx/emqx/pull/10548) 修复了HTTP驱动程序在竞争条件下会导致错误而不去重试的问题。 相关的驱动程序修复: [emqx/ehttpc#45](https://github.com/emqx/ehttpc/pull/45)` +- [#10548](https://github.com/emqx/emqx/pull/10548) 修复了 HTTP 驱动程序在竞争条件下会导致错误而不去重试的问题。 + 相关的驱动程序修复：[emqx/ehttpc#45](https://github.com/emqx/ehttpc/pull/45) - [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 数据桥接中，移除 SQL 模板中冗余的数据库名称。 @@ -135,8 +133,6 @@ - [#10438](https://github.com/emqx/emqx/pull/10438) 修复 DynamoDB 数据桥接中的部分术语使用错误： - 将 `database` 修改为 `table` - - 将 `username` 修改为 `aws_access_key_id` - - 将 `password` 修改为 `aws_secret_access_key` + - 将 `database` 修改为 `table` + - 将 `username` 修改为 `aws_access_key_id` + - 将 `password` 修改为 `aws_secret_access_key`