diff --git a/changes/e5.0.3.en.md b/changes/e5.0.3.en.md new file mode 100644 index 000000000..ecef80a31 --- /dev/null +++ b/changes/e5.0.3.en.md @@ -0,0 +1,187 @@ +# e5.0.3 + +## Enhancements + +- [#10077](https://github.com/emqx/emqx/pull/10077) Add support for QUIC TLS password protected certificate file. + + +- [#10128](https://github.com/emqx/emqx/pull/10128) Add support for OCSP stapling for SSL MQTT listeners. + +- [#10156](https://github.com/emqx/emqx/pull/10156) Change the priority of the configuration: + 1. If it is a new installation of EMQX, the priority of configuration is `ENV > emqx.conf > HTTP API`. + 2. If EMQX is upgraded from an old version (i.e., the cluster-override.conf file still exists in EMQX's data directory), then the configuration priority remains the same as before. That is, `HTTP API > ENV > emqx.conf`. + + Deprecated data/configs/local-override.conf. + + Stabilizing the HTTP API for hot updates. + +- [#10164](https://github.com/emqx/emqx/pull/10164) Add CRL check support for TLS MQTT listeners. + +- [#10206](https://github.com/emqx/emqx/pull/10206) Decouple the query mode from the underlying call mode for buffer + workers. + + Prior to this change, setting the query mode of a resource + such as a bridge to `sync` would force the buffer to call the + underlying connector in a synchronous way, even if it supports async + calls. + +- [#10207](https://github.com/emqx/emqx/pull/10207) Use 'label' from i18n file as 'summary' in OpenAPI spec. + +- [#10210](https://github.com/emqx/emqx/pull/10210) Unregister Mnesia post commit hook when Mria is being stopped. + This fixes hook failures occasionally occurring on stopping/restarting Mria. + + [Mria PR](https://github.com/emqx/mria/pull/133) + +- [#10224](https://github.com/emqx/emqx/pull/10224) Add the option to customize `clusterIP` in Helm chart, so that a user may set it to a fixed IP. + +- [#10263](https://github.com/emqx/emqx/pull/10263) Add command 'eval-ex' for Elixir expression evaluation. + +- [#10278](https://github.com/emqx/emqx/pull/10278) Refactor the directory structure of all gateways. + +- [#10306](https://github.com/emqx/emqx/pull/10306) Add support for `async` query mode for most bridges. + + Before this change, some bridges (Cassandra, MongoDB, MySQL, Postgres, Redis, RocketMQ, TDengine) were only allowed to be created with a `sync` query mode. + +- [#10318](https://github.com/emqx/emqx/pull/10318) Now, the rule engine language's FROM clause supports both strings enclosed in double quotes (") and single quotes ('). + +- [#10336](https://github.com/emqx/emqx/pull/10336) Add `/rule_engine` API endpoint to manage configuration of rule engine. + +- [#10354](https://github.com/emqx/emqx/pull/10354) More specific error messages when configure with bad max_heap_size value. + Log current value and the max value when the `message_queue_too_long` error is thrown. + +- [#10358](https://github.com/emqx/emqx/pull/10358) Hide `flapping_detect/conn_congestion/stats` configuration. + Deprecate `flapping_detect.enable`. + +- [#10359](https://github.com/emqx/emqx/pull/10359) Metrics now are not implicitly collected in places where API handlers don't make any use of them. Instead, a separate backplane RPC gathers cluster-wide metrics. + +- [#10373](https://github.com/emqx/emqx/pull/10373) Deprecate the trace.payload_encode configuration. + Add payload_encode=[text,hidden,hex] option when creating a trace via HTTP API. + +- [#10381](https://github.com/emqx/emqx/pull/10381) Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. + +- [#10385](https://github.com/emqx/emqx/pull/10385) Hide data items(rule_engine/bridge/authz/authn) from configuration files and documentation. + +- [#10391](https://github.com/emqx/emqx/pull/10391) Hide a large number of advanced options to simplify the configuration file. + + That includes `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, + `flapping_detect`, `conn_congestion`, `stats,auto_subscribe`, `broker_perf`, + `shared_subscription_group`, `slow_subs`, `ssl_options.user_lookup_fun` and some advance items + in `node` and `dashboard` section, [#10358](https://github.com/emqx/emqx/pull/10358), + [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385). + +- [#10404](https://github.com/emqx/emqx/pull/10404) Change the default queue mode for buffer workers to `memory_only`. + Before this change, the default queue mode was `volatile_offload`. When under high message rate pressure and when the resource is not keeping up with such rate, the buffer performance degraded a lot due to the constant disk operations. + +- [#10140](https://github.com/emqx/emqx/pull/10140) Integrate `Cassandra` into `bridges` as a new backend. At the current stage: + - Only support Cassandra version 3.x, not yet 4.x. + +- [#10143](https://github.com/emqx/emqx/pull/10143) Add `RocketMQ` data integration bridge. + +- [#10165](https://github.com/emqx/emqx/pull/10165) Support escaped special characters in InfluxDB data bridge write_syntax. + This update allows to use escaped special characters in string elements in accordance with InfluxDB line protocol. + +- [#10294](https://github.com/emqx/emqx/pull/10294) When configuring a MongoDB bridge, you can now use the ${var} syntax to reference fields in the message payload within the collection field. This enables you to select the collection to insert data into dynamically. + +- [#10337](https://github.com/emqx/emqx/pull/10337) Add schema registry feature. + + With schema registry, one can encode and decode special serialization formats in payloads when transforming messages in Rule Engine. Currently, only [Apache Avro](https://avro.apache.org/) is supported. + +- [#10363](https://github.com/emqx/emqx/pull/10363) Implement Microsoft SQL Server bridge. + +- [#10573](https://github.com/emqx/emqx/pull/10573) Improved performance of Webhook bridge when using synchronous query mode. + This also should improve the performance of other bridges when they are configured with no batching. + +## Bug Fixes + +- [#10145](https://github.com/emqx/emqx/pull/10145) Fix `bridges` API to report error conditions for a failing bridge as + `status_reason`. Also when creating an alarm for a failing resource we include + this error condition with the alarm's message. + +- [#10172](https://github.com/emqx/emqx/pull/10172) Fix the incorrect default ACL rule, which was: + ``` + {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. + ``` + + However, it should use `{re, "^dashboard$"}` to perform a regular expression match: + ``` + {allow, {username, {re,"^dashboard$"}}, subscribe, ["$SYS/#"]}. + ``` + +- [#10174](https://github.com/emqx/emqx/pull/10174) Upgrade library `esockd` from 5.9.4 to 5.9.6. + Fix an unnecessary error level logging when a connection is closed before proxy protocol header is sent by the proxy. + +- [#10195](https://github.com/emqx/emqx/pull/10195) Add labels to API schemas where description contains HTML and breaks formatting of generated documentation otherwise. + +- [#10196](https://github.com/emqx/emqx/pull/10196) Use lower-case for schema summaries and descritptions to be used in menu of generated online documentation. + +- [#10209](https://github.com/emqx/emqx/pull/10209) Fix bug where a last will testament (LWT) message could be published + when kicking out a banned client. + +- [#10211](https://github.com/emqx/emqx/pull/10211) Hide `broker.broker_perf` config and API documents. + The two configs `route_lock_type` and `trie_compaction` are rarely used and requires a full cluster restart to take effect. They are not suitable for being exposed to users. + Detailed changes can be found here: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions + +- [#10225](https://github.com/emqx/emqx/pull/10225) Allow installing a plugin if its name matches the beginning of another (already installed) plugin name. + For example: if plugin "emqx_plugin_template_a" is installed, it must not block installing plugin "emqx_plugin_template". + +- [#10226](https://github.com/emqx/emqx/pull/10226) Don't crash on validation error in `/bridges` API, return `400` instead. + +- [#10242](https://github.com/emqx/emqx/pull/10242) Fixed a log data field name clash. + Piror to this fix, some debug logs may report a wrong Erlang PID which may affect troubleshooting session takeover issues. + +- [#10257](https://github.com/emqx/emqx/pull/10257) Fixed the issue where `auto_observe` was not working in LwM2M Gateway. + + Before the fix, OBSERVE requests were sent without a token, causing failures + that LwM2M clients could not handle. + + After the fix, LwM2M Gateway can correctly observe the resource list carried by + client, furthermore, unknown resources will be ignored and printing the following + warning log: + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) Enhance logging behaviour during boot failure. + When EMQX fails to start due to corrupted configuration files, excessive logging is eliminated and no crash dump file is generated. + +- [#10297](https://github.com/emqx/emqx/pull/10297) Keeps `eval` command backward compatible with v4 by evaluating only Erlang expressions, even on Elixir node. For Elixir expressions, use `eval-ex` command. + +- [#10300](https://github.com/emqx/emqx/pull/10300) Fixed an issue where a build made with Elixir could not receive uploaded plugins until the `plugins` folder was created manually to receive the uploaded files. + +- [#10315](https://github.com/emqx/emqx/pull/10315) Fix crash checking `limit` and `page` parameters in `/mqtt/delayed/messages` API call. + +- [#10317](https://github.com/emqx/emqx/pull/10317) Do not expose listener level authentications before extensive verification. + +- [#10323](https://github.com/emqx/emqx/pull/10323) For security reasons, the value of the `password` field in the API examples is replaced with `******`. + + +- [#10410](https://github.com/emqx/emqx/pull/10410) Fix config check failed when gateways are configured in emqx.conf. + This issue was first introduced in v5.0.22 via [#10278](https://github.com/emqx/emqx/pull/10278), the boot-time config check was missing. + +- [#10449](https://github.com/emqx/emqx/pull/10449) Validate the ssl_options and header configurations when creating authentication http (`authn_http`). + Prior to this, incorrect `ssl` configuration could result in successful creation but the entire authn being unusable. + +- [#10455](https://github.com/emqx/emqx/pull/10455) Fixed an issue that could cause (otherwise harmless) noise in the logs. + + During some particularly slow synchronous calls to bridges, some late replies could be sent to connections processes that were no longer expecting a reply, and then emit an error log like: + + ``` + 2023-04-19T18:24:35.350233+00:00 [error] msg: unexpected_info, mfa: emqx_channel:handle_info/2, line: 1278, peername: 172.22.0.1:36384, clientid: caribdis_bench_sub_1137967633_4788, info: {#Ref<0.408802983.1941504010.189402>,{ok,200,[{<<"cache-control">>,<<"max-age=0, ...">>}} + ``` + + Those logs are harmless, but they could flood and worry the users without need. + +- [#10548](https://github.com/emqx/emqx/pull/10548) Fixed a race condition in the HTTP driver that would result in an error rather than a retry of the request. + Related fix in the driver: https://github.com/emqx/ehttpc/pull/45 + +- [#10201](https://github.com/emqx/emqx/pull/10201) In TDengine, removed the redundant database name from the SQL template. + +- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse has got a fix that makes the error message better when users click the test button in the settings dialog. + +- [#10324](https://github.com/emqx/emqx/pull/10324) Previously, when attempting to reconnect to a misconfigured Clickhouse bridge through the dashboard, users would not receive an error message. This issue is now resolved, and error messages will now be displayed + +- [#10438](https://github.com/emqx/emqx/pull/10438) Fix some configuration item terminology errors in the DynamoDB data bridge: + + - Changed `database` to `table` + - Changed `username` to `aws_access_key_id` + - Changed `password` to `aws_secret_access_key` diff --git a/changes/e5.0.3.zh.md b/changes/e5.0.3.zh.md new file mode 100644 index 000000000..12139700f --- /dev/null +++ b/changes/e5.0.3.zh.md @@ -0,0 +1,60 @@ +# e5.0.3 + +## 增强 + +- [#10077](https://github.com/emqx/emqx/pull/10077) 增加对 QUIC TLS 密码保护证书文件的支持。 + +- [#10278](https://github.com/emqx/emqx/pull/10278) 重构所有网关的源码目录结构。 + +- [#10318](https://github.com/emqx/emqx/pull/10318) 现在,规则引擎语言的 FROM 子句支持使用双引号(")和单引号(')括起来的字符串。 + +- [#10140](https://github.com/emqx/emqx/pull/10140) 支持 Cassandra 数据桥接。在当前阶段: + - 仅支持 Cassandra 3.x 版本,暂不支持 4.x。 + +- [#10143](https://github.com/emqx/emqx/pull/10143) 为数据桥接增加 `RocketMQ` 支持。 + +- [#10294](https://github.com/emqx/emqx/pull/10294) 在配置 MongoDB 桥时,现在可以使用 ${var} 语法来引用消息负载中的字段,以便动态选择要插入的集合。 + + + +## 修复 + +- [#10172](https://github.com/emqx/emqx/pull/10172) 修复错误的默认 ACL 规则,之前是: + ``` + {allow, {username, "^dashboard?"}, subscribe, ["$SYS/#"]}. + ``` + 但执行正则表达式的匹配应该使用 `{re, "^dashboard$"}`: + ``` + {allow, {username, {re, "^dashboard$"}}, subscribe, ["$SYS/#"]}. + ``` + +- [#10174](https://github.com/emqx/emqx/pull/10174) 依赖库 `esockd` 从 5.9.4 升级到 5.9.6。 + 修复了一个不必要的错误日志。如果连接在 proxy protocol 包头还没有发送前就关闭了, 则不打印错误日志。 + +- [#10211](https://github.com/emqx/emqx/pull/10211) 隐藏 `broker.broker_perf` 配置项,不再在 配置和 API 的文档中展示。 + `route_lock_type` 和 `trie_compaction` 这两个配置项很少使用,且需要全集群重启才能生效,不适合暴露给用户。 + 详细对比: https://gist.github.com/zmstone/01ad5754b9beaeaf3f5b86d14d49a0b7/revisions + +- [#10242](https://github.com/emqx/emqx/pull/10242) 修复log数据字段名称冲突。 + 在这个修复之前,一些调试日志可能会报告一个错误的Erlang PID,这可能会影响会话接管问题的故障调查。 + +- [#10257](https://github.com/emqx/emqx/pull/10257) 修复 LwM2M 网关 `auto_observe` 不工作的问题。 + + 在修复之前,下发的 OBSERVE 请求没有 Token 从而导致 LwM2M 客户端无法处理的失败。 + + 修复后,能正确监听 LwM2M 携带的资源列表、和会忽略未知的资源,并打印以下日志: + ``` + 2023-03-28T18:50:27.771123+08:00 [warning] msg: ignore_observer_resource, mfa: emqx_lwm2m_session:observe_object_list/3, line: 522, peername: 127.0.0.1:56830, clientid: testlwm2mclient, object_id: 31024, reason: no_xml_definition + ``` + +- [#10286](https://github.com/emqx/emqx/pull/10286) 优化启动失败的错误日志。 + 如果 EMQX 因为损坏的配置文件无法启动时,不会再打印过多的错误日志,也不再生成 crash.dump 文件。 + +- [#10317](https://github.com/emqx/emqx/pull/10317) 在大量验证完成前不暴露监听器级的认证功能。 + +- [#10323](https://github.com/emqx/emqx/pull/10323) 出于安全原因,将 API 示例中 `password` 字段的值,统一更换为 `******`。 + + +- [#10201](https://github.com/emqx/emqx/pull/10201) 在 TDengine 桥接的 SQL 模板中,删除了多余的数据库表名。 + +- [#10270](https://github.com/emqx/emqx/pull/10270) Clickhouse 已经修复了一个问题,当用户在设置对话框中点击测试按钮时,错误信息会更清晰。