From e3d6fa1f21a3997da02f2af78d0a67cac187d6c8 Mon Sep 17 00:00:00 2001 From: Zhongwen Deng Date: Tue, 18 Apr 2023 14:24:08 +0800 Subject: [PATCH 1/8] chore: update config's changelog and emqx_conf.template --- apps/emqx_conf/etc/emqx_conf.conf | 17 ++++++++++------- changes/ce/feat-10358.en.md | 2 -- changes/ce/feat-10381.en.md | 1 - changes/ce/feat-10385.en.md | 1 - changes/ce/feat-10391.en.md | 1 - changes/ce/feat-10426.en.md | 10 ++++++++++ rel/emqx_conf.template.en.md | 11 +++++------ rel/emqx_conf.template.zh.md | 6 +++--- 8 files changed, 28 insertions(+), 21 deletions(-) delete mode 100644 changes/ce/feat-10358.en.md delete mode 100644 changes/ce/feat-10381.en.md delete mode 100644 changes/ce/feat-10385.en.md delete mode 100644 changes/ce/feat-10391.en.md create mode 100644 changes/ce/feat-10426.en.md diff --git a/apps/emqx_conf/etc/emqx_conf.conf b/apps/emqx_conf/etc/emqx_conf.conf index 86147bf25..45733d158 100644 --- a/apps/emqx_conf/etc/emqx_conf.conf +++ b/apps/emqx_conf/etc/emqx_conf.conf @@ -1,11 +1,14 @@ ## NOTE: -## Configs in this file might be overridden by: -## 1. Environment variables which start with 'EMQX_' prefix -## 2. File $EMQX_NODE__DATA_DIR/configs/cluster-override.conf -## 3. File $EMQX_NODE__DATA_DIR/configs/local-override.conf -## -## The *-override.conf files are overwritten at runtime when changes -## are made from EMQX dashboard UI, management HTTP API, or CLI. +## The order of priority for configuration is: +## environment variables, 'etc/emqx.conf', and 'data/configs/cluster.hocon'. +## 1. Settings in environment variables starting with 'EMQX_' are given the highest priority. +## 2. Configuration settings in the 'etc/emqx.conf' file may be overwritten by environment variables. +## 3. Changes made through the EMQX dashboard UI, management HTTP API, or CLI +## overwrite the 'data/configs/cluster.hocon' file at runtime. +## 4. If you modify a configuration setting using the API, the change takes effect immediately. +## 5. However, if the same setting is configured differently in the 'etc/emqx.conf' file, +## 'etc/emqx.conf' takes priority after a reboot. + ## All configuration details can be found in emqx.conf.example node { diff --git a/changes/ce/feat-10358.en.md b/changes/ce/feat-10358.en.md deleted file mode 100644 index e6d05c84b..000000000 --- a/changes/ce/feat-10358.en.md +++ /dev/null @@ -1,2 +0,0 @@ -Hide `flapping_detect/conn_congestion/stats` configuration. -Deprecate `flapping_detect.enable`. diff --git a/changes/ce/feat-10381.en.md b/changes/ce/feat-10381.en.md deleted file mode 100644 index 3ea11188f..000000000 --- a/changes/ce/feat-10381.en.md +++ /dev/null @@ -1 +0,0 @@ -Hide the `auto_subscribe` configuration items so that they can be modified later only through the HTTP API. diff --git a/changes/ce/feat-10385.en.md b/changes/ce/feat-10385.en.md deleted file mode 100644 index 667e01890..000000000 --- a/changes/ce/feat-10385.en.md +++ /dev/null @@ -1 +0,0 @@ -Hide data items(rule_engine/bridge/authz/authn) from configuration files and documentation. diff --git a/changes/ce/feat-10391.en.md b/changes/ce/feat-10391.en.md deleted file mode 100644 index a64b01221..000000000 --- a/changes/ce/feat-10391.en.md +++ /dev/null @@ -1 +0,0 @@ -hide exhook/rewrite/topic_metric/persistent_session_store/overload_protection from the docs and configuration file. diff --git a/changes/ce/feat-10426.en.md b/changes/ce/feat-10426.en.md new file mode 100644 index 000000000..660d4baa2 --- /dev/null +++ b/changes/ce/feat-10426.en.md @@ -0,0 +1,10 @@ +1. Changes in configuration priority: + For new EMQX installations, configuration priority is `ENV > emqx.conf > HTTP API`. + For upgrades from an older version with cluster-override.conf, configuration priority remains the same: `HTTP API > ENV > emqx.conf`. +2. Deprecation of `data/configs/local-override.conf`. +3. Simplified configuration items, hidden some advanced items + The hidden configurations include: exhook,rewrite,topic_metric,persistent_session_store,overload_protection, + flapping_detect,conn_congestion,stats,auto_subscribe,broker_perf,rule_engine,bridge,shared_subscription_group,slow_subs + and some advance items in node/dashboard. +4. This hidden update doesn't change the functionality of the original configuration, + but it sets the stage for an improved presentation of configuration documentation in future versions. diff --git a/rel/emqx_conf.template.en.md b/rel/emqx_conf.template.en.md index 8740e4319..b63b50a14 100644 --- a/rel/emqx_conf.template.en.md +++ b/rel/emqx_conf.template.en.md @@ -9,18 +9,18 @@ From bottom up: 1. Immutable base: `emqx.conf` + `EMQX_` prefixed environment variables.
Changes in this layer require a full node restart to take effect. -1. Cluster overrides: `$EMQX_NODE__DATA_DIR/configs/cluster-override.conf` +2. Cluster overrides: `$EMQX_NODE__DATA_DIR/configs/cluster.hocon` When environment variable `$EMQX_NODE__DATA_DIR` is not set, config `node.data_dir` is used. -The `cluster-override.conf` file is overwritten at runtime when changes +The `cluster.hocon` file is overwritten at runtime when changes are made from dashboard UI, management HTTP API, or CLI. When clustered, after EMQX restarts, it copies the file from the node which has the greatest `uptime`. :::tip Tip Some of the configs (such as `node.name`) are boot-only configs and not overridable. -Config values from `*-override.conf` are **not** mapped to boot configs for +Config values from `cluster.hocon` are **not** mapped to boot configs for the config fields attributed with `mapping: path.to.boot.config.key` ::: @@ -148,7 +148,7 @@ export EMQX_LISTENERS__SSL__L1__AUTHENTICATION__SSL__CIPHERS='["TLS_AES_256_GCM_ However this also means a string value should be quoted if it happens to contain special characters such as `=` and `:`. -For example, a string value `"localhost:1883"` would be +For example, a string value `"localhost:1883"` would be parsed into object (struct): `{"localhost": 1883}`. To keep it as a string, one should quote the value like below: @@ -226,7 +226,7 @@ Arrays in EMQX config have two different representations Dot-separated paths with number in it are parsed to indexed-maps e.g. `authentication.1={...}` is parsed as `authentication={"1": {...}}` -This feature makes it easy to override array elment values. For example: +This feature makes it easy to override array element values. For example: ``` authentication=[{enable=true, backend="built_in_database", mechanism="password_based"}] @@ -322,4 +322,3 @@ ciphers = "PSK-AES128-CBC-SHA" ] ``` - diff --git a/rel/emqx_conf.template.zh.md b/rel/emqx_conf.template.zh.md index 9402760a2..916eed38d 100644 --- a/rel/emqx_conf.template.zh.md +++ b/rel/emqx_conf.template.zh.md @@ -7,18 +7,18 @@ EMQX的配置文件可分为二层，自底向上依次是： 1. 不可变的基础层 `emqx.conf` 加上 `EMQX_` 前缀的环境变量。
修改这一层的配置之后，需要重启节点来使之生效。 -1. 集群范围重载层：`$EMQX_NODE__DATA_DIR/configs/cluster-override.conf` +2. 集群范围重载层：`$EMQX_NODE__DATA_DIR/configs/cluster.hocon` 如果环境变量 `$EMQX_NODE__DATA_DIR` 没有设置，那么该目录会从 `emqx.conf` 的 `node.data_dir` 配置中读取。 -配置文件 `cluster-override.conf` 的内容会在运行时被EMQX重写。 +配置文件 `cluster.hocon` 的内容会在运行时被EMQX重写。 这些重写发生在 dashboard UI，管理HTTP API，或者CLI对集群配置进行修改时。 当EMQX运行在集群中时，一个EMQX节点重启之后，会从集群中其他节点复制该文件内容到本地。 :::tip Tip 有些配置项是不能被重载的（例如 `node.name`）。 配置项如果有 `mapping: path.to.boot.config.key` 这个属性， -则不能被添加到重载文件 `*-override.conf` 中。 +则不能被添加到重载文件 `cluster.hocon` 中。 ::: 更多的重载规则，请参考下文 [配置重载规则](#配置重载规则)。 From 22a1d05d7b64859249b182373c10a6c6976f66c1 Mon Sep 17 00:00:00 2001 From: Zhongwen Deng Date: Tue, 18 Apr 2023 14:38:17 +0800 Subject: [PATCH 2/8] feat: hide ssl_options.user_lookup_fun --- apps/emqx/src/emqx_schema.erl | 2 ++ changes/ce/feat-10426.en.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/apps/emqx/src/emqx_schema.erl b/apps/emqx/src/emqx_schema.erl index 8073e19b5..38566f3cb 100644 --- a/apps/emqx/src/emqx_schema.erl +++ b/apps/emqx/src/emqx_schema.erl @@ -2225,6 +2225,7 @@ common_ssl_opts_schema(Defaults) -> #{ default => AvailableVersions, desc => ?DESC(common_ssl_opts_schema_versions), + importance => ?IMPORTANCE_HIGH, validator => fun(Inputs) -> validate_tls_versions(AvailableVersions, Inputs) end } )}, @@ -2235,6 +2236,7 @@ common_ssl_opts_schema(Defaults) -> #{ default => <<"emqx_tls_psk:lookup">>, converter => fun ?MODULE:user_lookup_fun_tr/2, + importance => ?IMPORTANCE_HIDDEN, desc => ?DESC(common_ssl_opts_schema_user_lookup_fun) } )}, diff --git a/changes/ce/feat-10426.en.md b/changes/ce/feat-10426.en.md index 660d4baa2..b169dd19b 100644 --- a/changes/ce/feat-10426.en.md +++ b/changes/ce/feat-10426.en.md @@ -5,6 +5,6 @@ 3. Simplified configuration items, hidden some advanced items The hidden configurations include: exhook,rewrite,topic_metric,persistent_session_store,overload_protection, flapping_detect,conn_congestion,stats,auto_subscribe,broker_perf,rule_engine,bridge,shared_subscription_group,slow_subs - and some advance items in node/dashboard. + ssl_options.user_lookup_fun and some advance items in node/dashboard. 4. This hidden update doesn't change the functionality of the original configuration, but it sets the stage for an improved presentation of configuration documentation in future versions. From e3a84e0010eb3ea07e8b4e2f1b243d666c4d8e43 Mon Sep 17 00:00:00 2001 From: JianBo He Date: Tue, 18 Apr 2023 15:50:08 +0800 Subject: [PATCH 3/8] chore: improve the changes for configuration --- changes/ce/feat-10391.en.md | 7 +++++++ changes/ce/feat-10426.en.md | 14 ++++---------- 2 files changed, 11 insertions(+), 10 deletions(-) create mode 100644 changes/ce/feat-10391.en.md diff --git a/changes/ce/feat-10391.en.md b/changes/ce/feat-10391.en.md new file mode 100644 index 000000000..5c37436dd --- /dev/null +++ b/changes/ce/feat-10391.en.md @@ -0,0 +1,7 @@ +Hide a large number of advanced options to simplify the configuration file. + +That includes `exhook`, `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, +`flapping_detect`, `conn_congestion`, `stats,auto_subscribe`, `broker_perf`, `rule_engine`, `bridge`, +`shared_subscription_group`, `slow_subs`, `ssl_options.user_lookup_fun` and some advance items +in `node` and `dashboard` section, [#10358](https://github.com/emqx/emqx/pull/10358), +[#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385). diff --git a/changes/ce/feat-10426.en.md b/changes/ce/feat-10426.en.md index b169dd19b..8575347d6 100644 --- a/changes/ce/feat-10426.en.md +++ b/changes/ce/feat-10426.en.md @@ -1,10 +1,4 @@ -1. Changes in configuration priority: - For new EMQX installations, configuration priority is `ENV > emqx.conf > HTTP API`. - For upgrades from an older version with cluster-override.conf, configuration priority remains the same: `HTTP API > ENV > emqx.conf`. -2. Deprecation of `data/configs/local-override.conf`. -3. Simplified configuration items, hidden some advanced items - The hidden configurations include: exhook,rewrite,topic_metric,persistent_session_store,overload_protection, - flapping_detect,conn_congestion,stats,auto_subscribe,broker_perf,rule_engine,bridge,shared_subscription_group,slow_subs - ssl_options.user_lookup_fun and some advance items in node/dashboard. -4. This hidden update doesn't change the functionality of the original configuration, - but it sets the stage for an improved presentation of configuration documentation in future versions. +Optimize the configuration priority mechanism to fix the issue where the configuration +changes made to `etc/emqx.conf` do not take effect after restarting EMQX. + +More introduction about the new mechanism: [Configure Override Rules](https://www.emqx.io/docs/en/v5.0/configuration/configuration.html#configure-override-rules) From fbadfc06e421c749f08b33a0e09decda9e801801 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9F=90=E6=96=87?= Date: Tue, 18 Apr 2023 20:15:19 +0800 Subject: [PATCH 4/8] feat: change exhook, rule_engine, bridge to low importance level instead of hidden --- apps/emqx_bridge/src/schema/emqx_bridge_schema.erl | 2 +- apps/emqx_exhook/src/emqx_exhook_schema.erl | 2 +- apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl | 2 +- changes/ce/feat-10391.en.md | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/apps/emqx_bridge/src/schema/emqx_bridge_schema.erl b/apps/emqx_bridge/src/schema/emqx_bridge_schema.erl index e5def2d64..4b9b7e3fe 100644 --- a/apps/emqx_bridge/src/schema/emqx_bridge_schema.erl +++ b/apps/emqx_bridge/src/schema/emqx_bridge_schema.erl @@ -137,7 +137,7 @@ namespace() -> "bridge". tags() -> [<<"Bridge">>]. -roots() -> [{bridges, ?HOCON(?R_REF(bridges), #{importance => ?IMPORTANCE_HIDDEN})}]. +roots() -> [{bridges, ?HOCON(?R_REF(bridges), #{importance => ?IMPORTANCE_LOW})}]. fields(bridges) -> [ diff --git a/apps/emqx_exhook/src/emqx_exhook_schema.erl b/apps/emqx_exhook/src/emqx_exhook_schema.erl index 708e164fc..f6cc896f3 100644 --- a/apps/emqx_exhook/src/emqx_exhook_schema.erl +++ b/apps/emqx_exhook/src/emqx_exhook_schema.erl @@ -32,7 +32,7 @@ namespace() -> exhook. roots() -> - [{exhook, ?HOCON(?R_REF(exhook), #{importance => ?IMPORTANCE_HIDDEN})}]. + [{exhook, ?HOCON(?R_REF(exhook), #{importance => ?IMPORTANCE_LOW})}]. fields(exhook) -> [ diff --git a/apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl b/apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl index 242c86c71..bc8cae07a 100644 --- a/apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl +++ b/apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl @@ -38,7 +38,7 @@ namespace() -> rule_engine. tags() -> [<<"Rule Engine">>]. -roots() -> [{"rule_engine", ?HOCON(?R_REF("rule_engine"), #{importance => ?IMPORTANCE_HIDDEN})}]. +roots() -> [{"rule_engine", ?HOCON(?R_REF("rule_engine"), #{importance => ?IMPORTANCE_LOW})}]. fields("rule_engine") -> rule_engine_settings() ++ diff --git a/changes/ce/feat-10391.en.md b/changes/ce/feat-10391.en.md index 5c37436dd..f33757404 100644 --- a/changes/ce/feat-10391.en.md +++ b/changes/ce/feat-10391.en.md @@ -1,7 +1,7 @@ Hide a large number of advanced options to simplify the configuration file. -That includes `exhook`, `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, -`flapping_detect`, `conn_congestion`, `stats,auto_subscribe`, `broker_perf`, `rule_engine`, `bridge`, +That includes `rewrite`, `topic_metric`, `persistent_session_store`, `overload_protection`, +`flapping_detect`, `conn_congestion`, `stats,auto_subscribe`, `broker_perf`, `shared_subscription_group`, `slow_subs`, `ssl_options.user_lookup_fun` and some advance items in `node` and `dashboard` section, [#10358](https://github.com/emqx/emqx/pull/10358), [#10381](https://github.com/emqx/emqx/pull/10381), [#10385](https://github.com/emqx/emqx/pull/10385). From 444196922cb9523e6bf8eae915ed2b49c4298ea4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9F=90=E6=96=87?= Date: Tue, 18 Apr 2023 20:18:46 +0800 Subject: [PATCH 5/8] chore: update emqx.conf's note --- apps/emqx_conf/etc/emqx_conf.conf | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/apps/emqx_conf/etc/emqx_conf.conf b/apps/emqx_conf/etc/emqx_conf.conf index 45733d158..8ad84fd7c 100644 --- a/apps/emqx_conf/etc/emqx_conf.conf +++ b/apps/emqx_conf/etc/emqx_conf.conf @@ -1,13 +1,17 @@ ## NOTE: -## The order of priority for configuration is: -## environment variables, 'etc/emqx.conf', and 'data/configs/cluster.hocon'. -## 1. Settings in environment variables starting with 'EMQX_' are given the highest priority. -## 2. Configuration settings in the 'etc/emqx.conf' file may be overwritten by environment variables. -## 3. Changes made through the EMQX dashboard UI, management HTTP API, or CLI -## overwrite the 'data/configs/cluster.hocon' file at runtime. -## 4. If you modify a configuration setting using the API, the change takes effect immediately. -## 5. However, if the same setting is configured differently in the 'etc/emqx.conf' file, -## 'etc/emqx.conf' takes priority after a reboot. +## The EMQX configuration is prioritized (overlayed) in the following order: +## `cluster.hocon < emqx.conf < environment variables`. + +## Settings in environment variables that begin with 'EMQX_' have the highest priority +## and will override any settings in the `etc/emqx.conf` file. + +## Changes made through the EMQX dashboard UI, management HTTP API, or CLI +## will be written into the `data/configs/cluster.hocon` file at runtime and will take effect immediately. + +## However, if the same configuration items are set differently in the `etc/emqx.conf` file, +## the runtime updates will be overridden by the settings in `etc/emqx.conf` after the node restarts. + +## To avoid confusion, it is highly recommend NOT to have the same config keys in both `cluster.hocon` and `emqx.conf`. ## All configuration details can be found in emqx.conf.example From 9ca09383ba725887e2d00829fd6ae153a7a1550a Mon Sep 17 00:00:00 2001 From: zhongwencool Date: Tue, 18 Apr 2023 21:07:02 +0800 Subject: [PATCH 6/8] chore: remove desc from emqx.conf Co-authored-by: Zaiming (Stone) Shi --- apps/emqx_conf/etc/emqx_conf.conf | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/apps/emqx_conf/etc/emqx_conf.conf b/apps/emqx_conf/etc/emqx_conf.conf index 8ad84fd7c..dea04e9e4 100644 --- a/apps/emqx_conf/etc/emqx_conf.conf +++ b/apps/emqx_conf/etc/emqx_conf.conf @@ -2,18 +2,6 @@ ## The EMQX configuration is prioritized (overlayed) in the following order: ## `cluster.hocon < emqx.conf < environment variables`. -## Settings in environment variables that begin with 'EMQX_' have the highest priority -## and will override any settings in the `etc/emqx.conf` file. - -## Changes made through the EMQX dashboard UI, management HTTP API, or CLI -## will be written into the `data/configs/cluster.hocon` file at runtime and will take effect immediately. - -## However, if the same configuration items are set differently in the `etc/emqx.conf` file, -## the runtime updates will be overridden by the settings in `etc/emqx.conf` after the node restarts. - -## To avoid confusion, it is highly recommend NOT to have the same config keys in both `cluster.hocon` and `emqx.conf`. - -## All configuration details can be found in emqx.conf.example node { name = "emqx@127.0.0.1" From 59182ee0fce6e5be75c54508af645b44268dedbd Mon Sep 17 00:00:00 2001 From: zhongwencool Date: Tue, 18 Apr 2023 21:07:16 +0800 Subject: [PATCH 7/8] chore: update apps/emqx_conf/etc/emqx_conf.conf Co-authored-by: Zaiming (Stone) Shi --- apps/emqx_conf/etc/emqx_conf.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/apps/emqx_conf/etc/emqx_conf.conf b/apps/emqx_conf/etc/emqx_conf.conf index dea04e9e4..a54894dcd 100644 --- a/apps/emqx_conf/etc/emqx_conf.conf +++ b/apps/emqx_conf/etc/emqx_conf.conf @@ -1,6 +1,6 @@ ## NOTE: ## The EMQX configuration is prioritized (overlayed) in the following order: -## `cluster.hocon < emqx.conf < environment variables`. +## `data/configs/cluster.hocon < etc/emqx.conf < environment variables`. node { From 3cd9ca706795b2ccd5c3a14f0270a77a031450c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9F=90=E6=96=87?= Date: Tue, 18 Apr 2023 21:10:40 +0800 Subject: [PATCH 8/8] chore: change config seq --- rel/emqx_conf.template.en.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rel/emqx_conf.template.en.md b/rel/emqx_conf.template.en.md index b63b50a14..bded20da6 100644 --- a/rel/emqx_conf.template.en.md +++ b/rel/emqx_conf.template.en.md @@ -7,9 +7,10 @@ and a superset of JSON. EMQX configuration consists of two layers. From bottom up: -1. Immutable base: `emqx.conf` + `EMQX_` prefixed environment variables.
+1. Cluster configs: `$EMQX_NODE__DATA_DIR/configs/cluster.hocon` +2. `emqx.conf` + `EMQX_` prefixed environment variables.
Changes in this layer require a full node restart to take effect. -2. Cluster overrides: `$EMQX_NODE__DATA_DIR/configs/cluster.hocon` + When environment variable `$EMQX_NODE__DATA_DIR` is not set, config `node.data_dir` is used. @@ -144,8 +145,7 @@ For example, this environment variable sets an array value. ``` export EMQX_LISTENERS__SSL__L1__AUTHENTICATION__SSL__CIPHERS='["TLS_AES_256_GCM_SHA384"]' ``` - -However this also means a string value should be quoted if it happens to contain special +However, this also means a string value should be quoted if it happens to contain special characters such as `=` and `:`. For example, a string value `"localhost:1883"` would be