16 KiB

Raw Permalink Blame History

e5.0.4

Enhancements

#10389 Unify the config formats for cluster.core_nodes and cluster.statics.seeds. Now they both support formats in array ["emqx1@127.0.0.1", "emqx2@127.0.0.1"] or semicolon-separated string "emqx1@127.0.0.1,emqx2@127.0.0.1".
#10392 A new function to convert a formatted date to an integer timestamp has been added: date_to_unix_ts/3
#10426 Optimize the configuration priority mechanism to fix the issue where the configuration changes made to etc/emqx.conf do not take effect after restarting EMQX.

More introduction about the new mechanism: Configure Override Rules
#10457 Deprecates the integration with StatsD.

There seemd to be no user using StatsD integration, so we have decided to hide this feature for now. We will either remove or revive it based on requirements in the future.
#10458 Set the level of plugin configuration options to low level, in most cases, users only need to manage plugins on the dashboard without the need for manual modification, so we lowered the level.
#10491 Rename etcd.ssl to etcd.ssl_options to keep all of SSL options consistent in the configuration file.
#10512 Improved the storage format of Unicode characters in data files, Now we can store Unicode characters normally. For example: "SELECT * FROM "t/1" WHERE clientid = "-测试专用-""
#10568 Add shutdown counter information to emqx ctl listeners command
#10571 Do not emit useless crash report when EMQX stops. Previously, when EMQX (and emqx_topic_metrics in particular) stopped and removed underlying tables, some messages were still being handled and crashed.
#10588 Increase the time precision of trace logs from second to microsecond. For example, change from 2023-05-02T08:43:50+00:00 to 2023-05-02T08:43:50.237945+00:00.
#10623 Renamed max_message_queue_len to max_mailbox_size in the force_shutdown configuration. Old name is kept as an alias, so this change is backward compatible.
#10713 We hide the request_timeout in resource_option of the webhook to keep it consistent with the http request_timeout of the webhook. From now on, when configuring a webhook through API or configuration files, it is no longer necessary to configure the request_timeout of the resource. Only configuring the http request_timeout is sufficient, and the request_timeout in the resource will automatically be consistent with the http request_timeout.
#10075 Add node rebalance/node evacuation functionality. See also: design doc
#10378 Implement Pulsar Producer Bridge, which supports publishing messages to Pulsar from MQTT topics.
#10408 The rule engine SQL-like language has got three more built-in functions for creating values of the MongoDB date type. These functions are useful for rules with MongoDB bridge actions only and not supported in other actions.
#10409 Add support for Protocol Buffers schemas in Schema Registry.
#10425 Implement OpenTSDB data bridge.
#10498 Implement Oracle Database Bridge, which supports publishing messages to Oracle Database from MQTT topics.
#10560 Add enterprise data bridge for Apache IoTDB.
#10648 Refactor the directory structure of the RocketMQ data bridge.
#10650 Refactor the directory structure of the TDEngine data bridge.
#10662 Refactor the directory structure of the PostgreSQL && Matrix && Timescale data bridges.
#10679 Refactor the directory structure of the InfluxDB data bridge.
#9927 Introduce support for the File Transfer over MQTT feature as described in EIP-0021, with support to publish transferred files either to the node-local file system or to the S3 API compatible remote object storage.
#10376 Simplify the configuration of the limiter feature and optimize some codes
- Rename message_in to messages
- Rename bytes_in to bytes
- Use burst instead of capacity
- Hide non-importance fields
- Optimize limiter instances in different rate settings
#10417 Improve get config performance by eliminating temporary references.
#10430 Simplify the configuration of the retainer feature.
- Mark flow_control as non-importance field.
#10487 Optimize the instance of limiter for whose rate is infinity to reduce memory and CPU usage.
#10490 Remove the default limit of connect rate which used to be 1000/s
#10511 Improve the security and privacy of some resource logs by masking sensitive information in the data.
#10525 Reduce resource usage per MQTT packet handling.
#10528 Reduce memory footprint in hot code path.
#10591 Improve the configuration of the limiter.
- Simplify the memory representation of the limiter configuration.
- Make sure the node-level limiter can really work when the listener's limiter configuration is omitted.
#10625 Simplify limiter configuration.
- Reduce the complexity of the limiter's configuration. e.g. now users can use limiter.messages_rate = 1000/s to quickly set the node-level limit for the message publish.
- Update the configs/limiter API to suit this refactor.

Bug Fixes

#10340 Fixed the issue that could lead to crash logs being printed when stopping EMQX via systemd.

2023-03-29T16:43:25.915761+08:00 [error] Generic server memsup terminating. Reason: {port_died,normal}. Last message: {'EXIT',<0.2117.0>,{port_died,normal}}. State: [{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",929959936},{"Total",3832242176}]}},{items,{"Worst Memory User",[{"Pid",<0.2031.0>},{"Memory",4720472}]}}].
2023-03-29T16:43:25.924764+08:00 [error] crasher: initial call: memsup:init/1, pid: <0.2116.0>, registered_name: memsup, exit: {{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [os_mon_sup,<0.2114.0>], message_queue_len: 0, messages: [], links: [<0.2115.0>], dictionary: [], trap_exit: true, status: running, heap_size: 4185, stack_size: 29, reductions: 187637; neighbours:
2023-03-29T16:43:25.924979+08:00 [error] Supervisor: {local,os_mon_sup}. Context: child_terminated. Reason: {port_died,normal}. Offender: id=memsup,pid=<0.2116.0>.

#10369 Fix error in /api/v5/monitor_current API endpoint that happens when some EMQX nodes are down.

Prior to this fix, sometimes the request returned HTTP code 500 and the following message:
```
{"code":"INTERNAL_ERROR","message":"error, badarg, [{erlang,'++',[{error,nodedown},[{node,'emqx@10.42.0.150'}]], ...
```
#10407 Improve 'emqx_alarm' performance by using Mnesia dirty operations and avoiding unnecessary calls from 'emqx_resource_manager' to reactivate alarms that have been already activated. Use new safe 'emqx_alarm' API to activate/deactivate alarms to ensure that emqx_resource_manager doesn't crash because of alarm timeouts. The crashes were possible when the following conditions co-occurred:
- a relatively high number of failing resources, e.g. bridges tried to activate alarms on re-occurring errors;
- the system experienced a very high load.
#10420 Fix HTTP path handling when composing the URL for the HTTP requests in authentication and authorization modules.
- Avoid unnecessary URL normalization since we cannot assume that external servers treat original and normalized URLs equally. This led to bugs like #10411.
- Fix the issue that path segments could be HTTP encoded twice.
#10422 Fixed a bug where external plugins could not be configured via environment variables in a lone-node cluster.
#10448 Fix a compatibility issue of limiter configuration introduced by v5.0.23 which broke the upgrade from previous versions if the capacity is infinity.

In v5.0.23 we have replaced capacity with burst. After this fix, a capacity = infinity config will be automatically converted to equivalent burst = 0.
#10462 Deprecate config broker.shared_dispatch_ack_enabled. This was designed to avoid dispatching messages to a shared-subscription session which has the client disconnected. However since v5.0.9, this feature is no longer useful because the shared-subscrption messages in a expired session will be redispatched to other sessions in the group. See also: https://github.com/emqx/emqx/pull/9104
#10463 Improve bridges API error handling. If Webhook bridge URL is not valid, bridges API will return '400' error instead of '500'.
#10484 Fix the issue that the priority of the configuration cannot be set during rolling upgrade. For example, when authorization is modified in v5.0.21 and then upgraded v5.0.23 through rolling upgrade, the authorization will be restored to the default.
#10495 Add the limiter API /configs/limiter which was deleted by mistake back.
#10500 Add several fixes, enhancements and features in Mria:
- protect mria:join/1,2 with a global lock to prevent conflicts between two nodes trying to join each other simultaneously Mria PR
- implement new function mria:sync_transaction/4,3,2, which blocks the caller until a transaction is imported to the local node (if the local node is a replicant, otherwise, it behaves exactly the same as mria:transaction/3,2) Mria PR
- optimize mria:running_nodes/0 Mria PR
- optimize mria:ro_transaction/2 when called on a replicant node Mria PR.
#10518 Add the following fixes and features in Mria:
- call mria_rlog:role/1 safely in mria_membership to ensure that mria_membership gen_server won't crash if RPC to another node fails Mria PR
- Add extra field to ?rlog_sync table to facilitate extending this functionality in future Mria PR.
#10556 Wrap potentially sensitive data in emqx_connector_http if Authorization headers are being passed at initialization.
#10659 Fix the issue where emqx cannot start when sysmon.os.mem_check_interval is disabled.
#10717 Fixed an issue where the buffering layer processes could use a lot of CPU when inflight window is full.
#10724 A summary has been added for all endpoints in the HTTP API documentation (accessible at "http://emqx_host_name:18083/api-docs").
#10726 Validate Health Check Interval and Auto Restart Interval against the range from 1ms to 1 hour.
#10728 Fixed an issue where the rule engine was unable to access variables exported by FOREACH in the DO clause.

Given a payload: {"date": "2023-05-06", "array": ["a"]}, as well as the following SQL statement:
```
FOREACH payload.date as date, payload.array as elem
DO date, elem
FROM "t/#"
```
Prior to the fix, the date variable exported by FOREACH could not be accessed in the DO clause of the above SQL, resulting in the following output for the SQL statement: [{"elem": "a","date": "undefined"}]. After the fix, the output of the SQL statement is: [{"elem": "a","date": "2023-05-06"}]
#10742 Check the correctness of the rules before saving the authorization file source. Previously, Saving wrong rules could lead to restart failure.
#10743 Fixes an issue where trying to get a bridge info or metrics could result in a crash when a node is joining a cluster.
#10746 Add missing support of the event $events/delivery_dropped into the rule engine test API rule_test.
#10747 Refactor date and time functions, format_date and date_to_unix_ts, in the rule engine to fix the implementation problem.
#10755 Fixed data bridge resource update race condition.

In the 'delete + create' process for EMQX resource updates, long bridge creation times could cause dashboard request timeouts. If a bridge resource update was initiated before completion of its creation, it led to an erroneous deletion from the runtime, despite being present in the config file.

This fix addresses the race condition in bridge resource updates, ensuring the accurate identification and addition of new resources, maintaining consistency between runtime and configuration file statuses.
#10760 Fix Internal Error 500 that occurred sometimes when bridge statistics page was updated while a node was (re)joining the cluster.
#10761 Fixing the issue where the default value of SSL certificate for Dashboard Listener was not correctly interpolated, which caused HTTPS to be inaccessible when verify_peer and cacertfile were using the default configuration.
#10801 Avoid duplicated percent decode the topic name in API /topics/{topic} and /topics.
#10817 Fix the error of not being able to configure auto_restart_interval as infinity
#10672 Fix the issue where the lack of a default value for ssl_options in listeners results in startup failure. For example, such command(EMQX_LISTENERS__WSS__DEFAULT__BIND='0.0.0.0:8089' ./bin/emqx console) would have caused a crash before.
#10738 Add support for the Supertable and Create Tables Automatically features of TDEngine to its data bridge. Before this fix, an insert with a supertable in the template will fail, like this: insert into ${clientid} using msg TAGS (${clientid}) values (${ts},${msg}).
#10741 Fix password leaking on stacktrace for Oracle Database.

16 KiB Raw Permalink Blame History

e5.0.4

Enhancements

Bug Fixes

16 KiB

Raw Permalink Blame History