yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
emqx/changes/e5.7.1.en.md

10 KiB
Raw Permalink Blame History

5.7.1

Release Date: 2024-06-26

Enhancements

  • #12983 Add new rule engine event $events/client_check_authn_complete for authentication completion event.

  • #13175 Added the disable_prepared_statements option for Postgres-based connectors.

    This option is to be used with endpoints that do not support the prepared statements session feature, such as PGBouncer and Supabase in Transaction mode.

  • #13180 Improved client message handling performance when EMQX is running on Erlang/OTP 26 and increased message throughput by 10% in fan-in mode.

  • #13191 Upgraded EMQX Docker images to run on Erlang/OTP 26.

    EMQX had been running on Erlang/OTP 26 since v5.5 except for docker images which were on Erlang/OTP 25. Now all releases are on Erlang/OTP 26. This upgrade fixed the following known issue:

    When an older version of EMQX joins a cluster with newer version nodes, the Schema Registry of the older version node may encounter an issue, emitting logs like the following:

    Error loading module '$schema_parser___CiYAWBja87PleCyKZ58h__SparkPlug_B_BUILT-IN':,
This BEAM file was compiled for a later version of the runtime system than the current (Erlang/OTP 25).

    This issue is fixed in the newer version. However, for older versions, a manual step is required. Execute the following command on one of the clustered nodes before the older version EMQX joins the cluster:

    emqx eval 'lists:foreach(fun(Key) -> mnesia:dirty_delete(emqx_ee_schema_registry_protobuf_cache_tab, Key) end, mnesia:dirty_all_keys(emqx_ee_schema_registry_protobuf_cache_tab)).'

    If the older version of EMQX is already in the cluster, execute the above command and restart the affected node.

  • #13242 Significantly increased the startup speed of EMQX Dashboard listener.

  • #13172 Added a rule function map_to_redis_hset_args to help preparing redis HSET (or HMSET) multi-fields values.

    For example, if payload.value is a map of multiple data fields, this rule SELECT map_to_redis_hset_args(payload.value) as hset_fields FROM "t/#" can prepare hset_fields for redis action to render the command template like HMSET name1 ${hset_fields}.

  • #13210 EMQX now validates that referenced schemas and message types exist in the Schema Registry when inserting or updating a Schema Validation.

  • #13211 Enhanced TLS listener to support more flexible TLS verifications.

    • partial_chain support: If the option partial_chain is set to true, connections with incomplete certificate chains are allowed. Check the Configuration Manual for more details.

    • Certificate Key Usage validation: Added support for required Extended Key Usage as defined in rfc5280. A new option (verify_peer_ext_key_usage) has been introduced to enforce specific key usages (such as "serverAuth") in peer certificates during the TLS handshake. This enhances security by ensuring certificates are used for their intended purposes, for example, "serverAuth,OID:1.3.6.1.5.5.7.3.2". Check the Configuration Manual for more details.

  • #13274 The RocketMQ connector now supports configuring SSL settings.

Bug Fixes

  • #13156 Resolved an issue where the Dashboard Monitoring pages would crash following the update to EMQX v5.7.0.

  • #13164 Fixed HTTP authorization request body encoding.

    Before this fix, the HTTP authorization request body encoding format was taken from the accept header. The fix is to respect the content-type header instead. Also added access templating variable for v4 compatibility. The access code of SUBSCRIBE action is 1 and PUBLISH action is 2.

  • #13238 Improved the logged error messages when an HTTP authorization request with an unsupported content-type header is returned.

  • #13258 Fixed an issue where the MQTT-SN gateway would not restart correctly due to incorrect startup order of gateway dependencies.

  • #13273 Fixed and improved handling of URIs in several configurations. The fix includes the following improvement details:

    • Authentication and authorization configurations: Corrected a previous error where valid pathless URIs such as https://example.com?q=x were mistakenly rejected. These URIs are now properly recognized as valid.
    • Connector configurations: Enhanced checks to ensure that URIs with potentially problematic components, such as user info or fragment parts, are no longer erroneously accepted.

  • #13276 Fixed an issue in the durable message storage mechanism where parts of the internal storage state were not correctly persisted during the setup of new storage generations. The concept of "generation" is used internally and is crucial for managing message expiration and cleanup. This could have manifested as messages being lost after a restart of EMQX.

  • #13291 Fixed an issue where durable storage sites that were down being reported as up.

  • #13290 Fixed an issue where the command $ bin/emqx ctl rules show rule_0hyd would produce no output when used to display rules with a data integration action attached.

  • #13293 Improved the restoration process from data backups by automating the re-indexing of imported retained messages. Previously, re-indexing required manual intervention using the emqx ctl retainer reindex start CLI command after importing a data backup file.

    This fix also extended the functionality to allow exporting retained messages to a backup file when the retainer.backend.storage_type is configured as ram. Previously, only setups with disc as the storage type supported exporting retained messages.

  • #13147 Improved the error messages for decoding failures in the rule engine protobuf decode functions by adding clear descriptions to indicate what went wrong when message decoding failed.

  • #13140 Fixed an issue that caused text traces for the republish action to crash and not display correctly.

  • #13148 Fixed an issue where a 500 HTTP status code could be returned by /connectors/:connector-id/start when there is a timeout waiting for the resource to be connected.

  • #13181 EMQX now forcefully shut down the connector process when attempting to stop a connector, if such operation times out. This fix also improved the clarity of error messages when disabling an action or source fails due to an unresponsive underlying connector.

  • #13216 Respect clientid_prefix config for MQTT bridges. Since EMQX v5.4.1, the MQTT client IDs are restricted to a maximum of 23 bytes. Previously, the system factored the clientid_prefix into the hash of the original, longer client ID, affecting the final shortened ID. The fix includes the following change details:

    • Without Prefix: The behavior remains unchanged. EMQX hashes the long client IDs (exceeding 23 bytes) to fit within the 23-byte limit.
    • With Prefix:
      • Prefix ≤ 19 bytes: The prefix is retained, and the remaining portion of the client ID is hashed into a 4-byte space, ensuring the total length does not exceed 23 bytes.
      • Prefix ≥ 20 bytes: EMQX will not attempt to shorten the client ID, fully preserving the configured prefix regardless of length.

  • #13189 Fixed an issue where the data integration with Microsoft SQL Server or MySQL could not use SQL templates with substring values in table name or column name.

  • #13070 Improved Kafka connector error logs to provide more diagnostic information by capturing specific error details, such as unreachable advertised listeners. To manage log verbosity, only the first occurrence of an error is logged, accompanied by the total count of similar errors.

  • #13093 Improved Kafka consumer group stability. Before this change, the Kafka consumer group sometimes needs to rebalance twice after the Kafka group coordinator restarted.

  • #13277 Refined the error handling for Kafka producers when encountering the message_too_large error. Previously, Kafka producers would repeatedly attempt to resend oversized message batches, hoping for a server-side adjustment in max.message.bytes.

    Now, oversized messages are automatically split into single-message batches for retry. If a message still exceeds size limits, it will be dropped to maintain data flow.

  • #13130 Improved the trace message formatting for Redis action batch requests. Spaces are now added between components of commands and semicolons are added between commands to make the trace message easier to read.

  • #13136 Improved the template-rendered traces for Oracle actions for better readability.

  • #13197 Fixed an issue in AWS S3 data integration that prevented automatic saving of TLS certificates and key files to the file system when they are supplied through the Dashboard UI or Connector API.

  • #13227 Fixed an issue in AWS S3 Sink running in aggregated mode. Before the fix, an invalid key template in the configuration was reported as an error during the Sink setup, but instead caused a storm of hard-to-recover crashes later.