diff --git a/apps/emqx_conf/src/emqx_conf.erl b/apps/emqx_conf/src/emqx_conf.erl index c57d2c97b..eed6b0bb9 100644 --- a/apps/emqx_conf/src/emqx_conf.erl +++ b/apps/emqx_conf/src/emqx_conf.erl @@ -217,7 +217,7 @@ gen_doc(File, SchemaModule, I18nFile, Lang) -> Title = "# " ++ emqx_release:description() ++ " Configuration\n\n" ++ "", - BodyFile = filename:join([rel, "emqx_conf.template.en.md"]), + BodyFile = filename:join([rel, "emqx_conf.template." ++ Lang ++ ".md"]), {ok, Body} = file:read_file(BodyFile), Opts = #{title => Title, body => Body, desc_file => I18nFile, lang => Lang}, Doc = hocon_schema_md:gen(SchemaModule, Opts), diff --git a/rel/emqx_conf.template.en.md b/rel/emqx_conf.template.en.md index e79e79862..48fa01b61 100644 --- a/rel/emqx_conf.template.en.md +++ b/rel/emqx_conf.template.en.md @@ -15,11 +15,15 @@ From bottom up: When environment variable `$EMQX_NODE__DATA_DIR` is not set, config `node.data_dir` is used. -The `*-override.conf` files are overwritten at runtime when changes -are made from dashboard UI, management HTTP API, or CLI. +The `cluster-override.conf` file is overwritten at runtime when changes +are made from dashboard UI, management HTTP API, or CLI. When clustered, +after EMQX restarts, it copies the file from the node which has the greatest `uptime`. -**NOTE** Config values from `*-override.conf` are **not** mapped to boot configs for +:::tip Tip +Some of the configs (such as `node.name`) are boot-only configs and not overridable. +Config values from `*-override.conf` are **not** mapped to boot configs for the config fields attributed with `mapping: path.to.boot.config.key` +::: For detailed override rules, see [Config Overlay Rules](#config-overlay-rules). @@ -43,7 +47,7 @@ node.cookie = "mysecret" This flat format is almost backward compatible with EMQX's config file format in 4.x series (the so called 'cuttlefish' format). -It is 'almost' compatible because the often HOCON requires strings to be quoted, +It is not fully compatible because the often HOCON requires strings to be quoted, while cuttlefish treats all characters to the right of the `=` mark as the value. e.g. cuttlefish: `node.name = emqx@127.0.0.1`, HOCON: `node.name = "emqx@127.0.0.1"`. @@ -57,21 +61,33 @@ For more HOCON syntax, please refer to the [specification](https://github.com/li To make the HOCON objects type-safe, EMQX introduced a schema for it. The schema defines data types, and data fields' names and metadata for config value validation -and more. In fact, this config document itself is generated from schema metadata. +and more. + +::: tip Tip +The configuration document you are reading now is generated from schema metadata. +::: ### Complex Data Types There are 4 complex data types in EMQX's HOCON config: -1. Struct: Named using an unquoted string, followed by a predefined list of fields, - fields can not start with a number, and are only allowed to use - lowercase letters and underscores as word separator. +1. Struct: Named using an unquoted string, followed by a predefined list of fields. + Only lowercase letters and digits are allowed in struct and field names. + Alos, only underscore can be used as word separator. 1. Map: Map is like Struct, however the fields are not predefined. - 1-based index number can also be used as map keys for an alternative - representation of an Array. 1. Union: `MemberType1 | MemberType2 | ...` 1. Array: `[ElementType]` +::: tip Tip +If map filed name is a positive integer number, it is interpreted as an alternative representation of an `Array`. +For example: +``` +myarray.1 = 74 +myarray.2 = 75 +``` +will be interpreated as `myarray = [74, 75]`, which is handy when trying to override array elements. +::: + ### Primitive Data Types Complex types define data 'boxes' which may contain other complex data @@ -88,8 +104,10 @@ There are quite some different primitive types, to name a few: * `emqx_schema:duration()` # time duration, another format of integer() * ... -The primitive types are mostly self-describing, some are built-in, such -as `atom()`, some are defined in EMQX modules, such as `emqx_schema:duration()`. +::: tip Tip +The primitive types are mostly self-describing, so there is usually not a lot to document. +For types that are not so clear by their names, the field description is to be used to find the details. +::: ### Config Paths @@ -128,15 +146,19 @@ For example, this environment variable sets an array value. export EMQX_LISTENERS__SSL__L1__AUTHENTICATION__SSL__CIPHERS="[\"TLS_AES_256_GCM_SHA384\"]" ``` -Unknown environment variables are logged as a `warning` level log, for example: +::: tip Tip +Unknown root paths are silently discarded by EMQX, for example `EMQX_UNKNOWN_ROOT__FOOBAR` is +silently discarded because `unknown_root` is not a predefined root path. + +Unknown field names in environment variables are logged as a `warning` level log, for example: ``` [warning] unknown_env_vars: ["EMQX_AUTHENTICATION__ENABLED"] ``` because the field name is `enable`, not `enabled`. +::: -NOTE: Unknown root keys are however silently discarded. ### Config Overlay Rules @@ -193,14 +215,16 @@ Arrays in EMQX config have two different representations Dot-separated paths with number in it are parsed to indexed-maps e.g. `authentication.1={...}` is parsed as `authentication={"1": {...}}` -Indexed-map arrays can be used to override list arrays: +This feature makes it easy to override array elment values. For example: ``` authentication=[{enable=true, backend="built_in_database", mechanism="password_based"}] # we can disable this authentication provider with: authentication.1.enable=false ``` -However, list arrays do not get recursively merged into indexed-map arrays. + +::: warning Warning +List arrays is a full-array override, but not a recursive merge, into indexed-map arrays. e.g. ``` @@ -208,3 +232,4 @@ authentication=[{enable=true, backend="built_in_database", mechanism="password_b ## below value will replace the whole array, but not to override just one field. authentication=[{enable=true}] ``` +::: diff --git a/rel/emqx_conf.template.zh.md b/rel/emqx_conf.template.zh.md new file mode 100644 index 000000000..b4a56b28f --- /dev/null +++ b/rel/emqx_conf.template.zh.md @@ -0,0 +1,218 @@ +EMQX的配置文件格式是 [HOCON](https://github.com/emqx/hocon) . +HOCON(Human-Optimized Config Object Notation)是一个JSON的超集,非常适用于易于人类读写的配置数据存储。 + +## 分层结构 + +EMQX的配置文件可分为三层,自底向上依次是: + +1. 不可变的基础层 `emqx.conf` 加上 `EMQX_` 前缀的环境变量.
+ 修改这一层的配置之后,需要重启节点来使之生效。 +1. 集群范围重载层:`$EMQX_NODE__DATA_DIR/configs/cluster-override.conf` +1. 节点本地重载层:`$EMQX_NODE__DATA_DIR/configs/local-override.conf` + +如果环境变量 `$EMQX_NODE__DATA_DIR` 没有设置,那么该目录会从 emqx.conf 的 `node.data_dir`配置中读取。 + +配置文件 `cluster-override.conf` 的内容会在运行时被EMQX重写。 +这些重写发生在 dashboard UI,管理HTTP API,或者CLI对集群配置进行修改时。 +当EMQX运行在集群中时,一个EMQX节点重启之后,会从集群中其他节点复制该文件内容到本地。 + +:::tip Tip +有些配置项是不能被重载的(例如 `node.name`). +配置项如果有 `mapping: path.to.boot.config.key` 这个属性, +则不能被添加到重载文件中 `*-override.conf` 中。 +::: + +更多的重载规则,请参考下文 [配置重载规则](#配置重载规则). + +## 配置文件语法 + +在配置文件中,值可以被记为类似JSON的对象,例如 + +``` +node { + name = "emqx@127.0.0.1" + cookie = "mysecret" +} +``` + +另一种等价的表示方法是扁平的,例如 + +``` +node.name = "127.0.0.1" +node.cookie = "mysecret" +``` + +这种扁平格式几乎与EMQX的配置文件格式向后兼容 +在4.x系列中(所谓的'cuttlefish'格式)。 + +它并不是完全兼容,因为HOCON经常要求字符串两端加上引号。 +而cuttlefish把`=`符右边的所有字符都视为值。 + +例如,cuttlefish:`node.name = emqx@127.0.0.1`,HOCON:`node.name = "emqx@127.0.0.1"`。 + +没有特殊字符的字符串在HOCON中也可以不加引号。 +例如:`foo`,`foo_bar`和`foo_bar_1`。 + +关于更多的HOCON语法,请参考[规范](https://github.com/lightbend/config/blob/main/HOCON.md) + +## Schema + +为了使HOCON对象类型安全,EMQX为它引入了一个schema。 +该schema定义了数据类型,以及数据字段的名称和元数据,用于配置值的类型检查等等。 + +::: tip Tip +当前阅读到配置文件的文档本身就是由模式元数据生成的。 +::: + +### 复杂数据类型 + +EMQX的配置文件中,有4中复杂数据结构类型,它们分别是: + +1. Struct:结构体都是有类型名称的,结构体中可以有任意多个字段。 + 结构体和字段的名称由不带特殊字符的全小些字母组成,名称中可以带数字,但不得以数字开头,多个单词可用下划线分隔。 +1. Map: Map与Struct(结构体)类似,但是内部的字段不是预先定义好的. +1. Union: 联合 `MemberType1 | MemberType2 | ...`,可以理解为:“不是这个,就是那个” +1. Array: 数组 `[ElementType]` + +::: tip Tip +如果Map的字段名称是纯数字,它会被解释成一个数组。 +例如 +``` +myarray.1 = 74 +myarray.2 = 75 +``` +会被解析成 `myarray = [74, 75]`。这个用法在重载数组元素的值时候非常有用。 +::: + +### 原始数据类型 + +复杂类型定义了数据 "盒子",其中可能包含其他复杂数据或原始值。 +有很多不同的原始类型,仅举几个例子。 + +* 原子 `atom()` +* 布尔 `boolean()`. +* 字符串 `string()'。 +* 整形 `integer()'。 +* 浮点数 `float()'. +* 数值 `number()'。 +* 二进制编码的字符串 `binary()` # 是 `string()` 的另一种格式 +* 时间间隔 `emqx_schema:duration()` # 时间间隔,是 `integer()` 的另一种格式 +* ... + +::: tip Tip +原始类型的名称大多是自我描述的,所以不需要过多的注释。 +但是有一些不是那么直观的数据类型,则需要配合字段的描述文档进行理解 +::: + + +### 配置路径 + +如果我们把EMQX的配置值理解成一个类似目录树的结构,那么类似于文件系统中使用斜杠或反斜杠进行层级分割, +EMQX使用的配置路径的层级分割符是 `'.'` + +被`'.'`号分割的每一段,则是Struct(结构体)的字段,或Map的key. + +下面有几个例子: + +``` +node.name = "emqx.127.0.0.1" +zone.zone1.max_packet_size = "10M" +authentication.1.enable = true +``` + +### 环境变量重载 + +因为`'.'` 分隔符不能使用于环境变量,所以我们需要使用另一个分割符。EMQX选用的是双下划线`__`。 +为了与其他的环境变量有所区分,EMQX还增加了一个前缀 `EMQX_` 来用作环境变量命名空间。 + +例如 `node.name` 的重载变量名是 `EMQX_NODE__NAME`。 + +环境变量的值,是解析成HOCON值的。所以这也使得环境变量可以用来传递复杂数据类型的值。 + +例如,下面这个环境变量传入一个数组类型的值。 + +``` +export EMQX_LISTENERS__SSL__L1__AUTHENTICATION__SSL__CIPHERS="[\"TLS_AES_256_GCM_SHA384\"]" +``` + +::: tip Tip +未定义的根路径会被EMQX忽略,例如 `EMQX_UNKNOWN_ROOT__FOOBAR` 这个环境变量会被EMQX忽略, +因为 `UNKNOWN_ROOT` 不是预先定义好的根路径。 +对于已知的根路径,未知的字段名称将被记录为warning日志,比如下面这个例子。 + +``` +[warning] unknown_env_vars: ["EMQX_AUTHENTICATION__ENABLED"] +``` + +这是因为正确的字段名称是 `enable`,而不是 `enabled`. +::: + +### 配置重载规则 + +HOCON的值是分层覆盖的,普遍规则如下: + +- 在同一个文件中,后(在文件底部)定义的值,覆盖前(在文件顶部)到值。 +- 当按层级覆盖时,高层级的值覆盖低层级的值。 + +结下来的文档将解释更详细的规则。 + +#### 结构体 + +合并覆盖规则。在如下配置中,最后一行的 `debug` 值会覆盖覆盖原先`level`字段的 `error` 值 +但是`enable` 字段保持不变。 +``` +log { + console_handler{ + enable=true, + level=error + } +} + +## 控制台日志打印先定义为`error`级,后被覆写成`debug`级 + +log.console_handler.level=debug +``` + +#### Map + +Map与结构体类似,也是合并覆盖规则。 +如下例子中,`zone1` 的 `max_packet_size` 可以在文件后面覆写. + +``` +zone { + zone1 { + mqtt.max_packet_size = 1M + } +} + +## 报文大小限制最先被设置成1MB,后被覆写为10MB + +zone.zone1.mqtt.max_packet_size = 10M +``` + +#### 数组元素 + +如上面介绍过,EMQX配置中的数组有两种表达方式。 + +* 列表格式,例如: `[1, 2, 3]` +* 带下标的Map格式,例如: `{"1"=1, "2"=2, "3"=3}` + +点好(`'.'`)分隔到路径中的纯数字会被解析成数组下标。 +例如,`authentication.1={...}` 会被解析成 `authentication={"1": {...}}`,进而进一步解析成 `authentication=[{...}]` +有了这个特性,我们就可以轻松覆写数组某个元素的值,例如: + +``` +authentication=[{enable=true, backend="built_in_database", mechanism="password_based"}] +# 可以用下面的方式将第一个元素的 `enable` 字段覆写 +authentication.1.enable=false +``` + +::: warning Warning +使用列表格式是的数组将全量覆写原值,如下例: + +``` +authentication=[{enable=true, backend="built_in_database", mechanism="password_based"}] +## 下面这中方式会导致数组第一个元素的除了 `enable` 以外的其他字段全部丢失 +authentication=[{enable=true}] +``` +:::