Commit Graph

476 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 7a0a0bee98 refactor(schema): change file() type to string() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi af3dfc91b1 refactor(schema): change log_lvel to enum type 2023-11-10 08:45:20 +01:00
Thales Macedo Garitezi 371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
Ivan Dyachkov 0bf5deaed9 chore: merge master into release-54 2023-11-09 08:32:26 +01:00
Zaiming (Stone) Shi 1e3500ffd2
fix(emqx_conf_app): ensure log data fields
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-08 21:19:22 +01:00
Zaiming (Stone) Shi f9e9748cec fix(cluster-rpc): boot from local config if table loaded
When EMQX boots up, it tries to get latest config from peer (core type)
nodes, if none of the nodes are replying, the node will decide
to boot with local config (and replay the committed changes) if
the commit table is loaded from disk locally (an indication of the
data being latest), otherwise it will sleep for 1-2 seconds and
retry.

This lead to a race condition, e.g. in a two nodes cluster:

1. node1 boots up
2. node2 boots up and copy mnesia table from node1
3. node1 restart before node2 can sync cluster.hocon from it
4. node1 boots up and copy mnesia table from node2

Now that both node1 and node2 has the mnesia `load_node` pointing
to each other (i.e. not a local disk load).

Prior to this fix, the nodes would wait for each other in a dead loop.

This commit fixes the issue by allowing node to boot
with local config if it does not have a lagging.
2023-11-08 15:06:47 +01:00
Ivan Dyachkov 0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
Zaiming (Stone) Shi 9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Ivan Dyachkov b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
zhongwencool 0634ff61c0 fix: dialyzer warning 2023-10-30 09:57:30 +08:00
zhongwencool 995948f0e8 refactor: remove seq from audit record 2023-10-27 14:07:19 +08:00
William Yang f52b4a9e32 fix: gen_rpc tcp client port align with server port 2023-10-25 12:11:35 +02:00
William Yang 37dd44c3d7
Merge pull request #11813 from qzhuyan/fix/william/genrpc-ssl-port-align
Fix/william/genrpc ssl port align
2023-10-25 09:04:19 +02:00
William Yang 6346e0d28a fix(gen_rpc): ssl client port align with server port 2023-10-24 22:33:52 +02:00
Zaiming (Stone) Shi abcb3166f6 fix: use binary string for raw default values 2023-10-24 14:46:01 +02:00
Ilya Averyanov 8c6f6b56b8 chore(tests): report more friendly about unset PROFILE variable 2023-10-23 13:48:42 +03:00
William Yang dd5807b797 fix(gen_rpc): ensure client/server use same driver
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
zhongwencool 926c804314 feat: add /audit http api to filter audit log 2023-10-20 14:47:24 +08:00
Zaiming (Stone) Shi 55618dd28a
Merge pull request #11731 from zmstone/1008-expose-file-transfer-host-conf
fix(api-docs): add file-transfer config to hot-config scope
2023-10-17 11:23:16 +02:00
Zaiming (Stone) Shi 7c022c2c6a refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-10-14 10:13:31 +02:00
Zaiming (Stone) Shi 2626d793a7 test: try to resolve schema module from PROFILE in tests 2023-10-10 10:24:10 +02:00
Zaiming (Stone) Shi e7e696cd66 feat(rpc): add ipv6_only config 2023-10-10 10:05:14 +02:00
Zaiming (Stone) Shi 497e08448d feat(cluster): support ipv6 and tls on ipv6 for clustering
Made possible to configure inet6_tls for Erlang distribution
Also, added support to configure ipv6 listener for gen_rpc
2023-10-09 20:03:21 +02:00
Ilya Averyanov 5dff36474d chore(auth): get rid of hardcoded schema modules in auth 2023-10-05 13:41:50 +03:00
Ilya Averyanov 1eb75b43c4 chore(auth): split emqx_authn and emqx_authz apps 2023-10-05 13:41:50 +03:00
Zaiming (Stone) Shi bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Zaiming (Stone) Shi 1a13b2ac56 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-28 13:53:03 +02:00
ieQu1 0aa3ccdd65 feat(gen_rpc): Add schema for the TLS versions and cipher suites 2023-09-28 00:36:59 +02:00
Zaiming (Stone) Shi 4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
Thales Macedo Garitezi 5d212e1086 fix(audit): only support audit log on enterprise edition
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
Zaiming (Stone) Shi aea1e80290 feat: add 'format' as alias for log 'formatter' 2023-09-25 16:55:22 +02:00
JianBo He b970a34ee0 chore: redact some audit logs from CLI 2023-09-25 18:06:39 +08:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
Ivan Dyachkov 105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
William Yang ea73d674b9 chore: bump version to e5.2.1-alpha.2 2023-09-13 14:55:17 +02:00
Serge Tupchii e1456c1c04 feat(emqx_conf_schema): add node.default_bootstrap_batch_size config parameter 2023-09-13 10:11:14 +03:00
ieQu1 b41310af9c
fix: Add prevent_overlappiing_partitions to vm_args
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-12 20:01:56 +02:00
ieQu1 1a86c21e20 fix(distribution): Set prevent_overlapping_partitions to false 2023-09-12 17:41:51 +02:00
Andrew Mayorov b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
zhongwencool 053b7fb94a test: add more test for conf_cli 2023-08-28 11:14:36 +08:00
zhongwencool d8be248a3d fix: improve the suggest msg for update conf failed 2023-08-28 08:53:23 +08:00
Zaiming (Stone) Shi ae094e363c chore: fix a typo in log message 2023-08-24 12:25:04 +02:00
zhongwencool ae10415fc3 feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security 2023-08-23 09:03:22 +08:00
Ilya Averyanov 3943830eae chore(auth): remove unnecessary atom conversions 2023-08-16 20:50:35 +03:00
Ilya Averyanov 67e06b3171 chore(auth): make schema injection be more universal 2023-08-16 20:41:05 +03:00
Ilya Averyanov 8213aa42c9 chore(auth): move authn leftovers to the authn app 2023-08-16 18:52:14 +03:00
Ivan Dyachkov 801c93dd46 chore: v5.1.5-build.3 2023-08-10 12:56:28 +02:00
zhongwencool 3933227636 feat: support opentelemetry metrics 2023-08-09 20:23:40 +08:00
Serge Tupchii 466fe7e009 perf: add broker_pool_size, generic_pool_size and channel_cleanup_batch_size config options
Tuning these options can improve performance if cluster interconnect network latency is high.

Fixes: EMQX-10661
2023-08-03 19:44:00 +03:00
Serge Tupchii ed28c12a66 chore(emqx_conf): deprecate node.tlog_push_mode setting
It is not used since mria 0.2.10
2023-08-03 17:46:14 +03:00
JianBo He b9f0cd7ba4 chore: pretty the cli output style 2023-07-20 21:01:28 +08:00
zhongwencool 3cd223ff5b chore: add more detail msg for merging failed 2023-07-20 17:48:26 +08:00
zhongwencool 402f75592e chore: move rpc call from emqx_management_proto to emqx_conf_proto 2023-07-19 11:05:14 +08:00
zhongwencool 0a35b7875f
Merge pull request #11269 from zhongwencool/fix-process-limit-hidden
fix: hide node.process_limit completely
2023-07-14 22:32:41 +08:00
zhongwencool 11ee6abf02 fix: keep precent range is 0%~100% 2023-07-14 16:48:50 +08:00
zhongwencool 131511b10e fix: hide process_limit completely 2023-07-14 11:04:20 +08:00
zhongwencool 802a50601a chore: add comment for init_loader_done 2023-07-10 12:03:36 +08:00
zhongwencool c4ba558ee3 fix: unset config_loader in emqx's env when stop emqx app 2023-07-10 11:06:08 +08:00
zhongwencool f7513b900a fix: set load config done after update tnx_id 2023-07-07 12:25:34 +08:00
zhongwencool 9f57ba510e chore: add 11214 changelog 2023-07-07 12:25:34 +08:00
zhongwencool 221f6eba06 fix: bad tnx-id when rejoin cluster 2023-07-07 12:25:34 +08:00
Andrew Mayorov 8cc7022760
Merge pull request #11120 from ft/EMQX-10329/tooling
test: add tooling to make testruns more isolated and predictable
2023-07-04 14:36:39 +02:00
zhongwencool f2c5f4932c
Merge pull request #11148 from lafirest/fix/mcall_error_report
fix(emqx_conf): don't sync the MFA to a leaved node
2023-07-04 19:56:52 +08:00
Andrew Mayorov fe0b8cfbaf
test(conf): respect `init_config_load_done` in emqx_conf itself
This should make task of configuring various applications (emqx_conf
/ emqx) during testsuites setup much simpler.
2023-07-04 13:20:02 +02:00
firest afabdc3440 fix(emqx_conf): call clean callback before ekka stop 2023-07-04 18:04:33 +08:00
zhongwencool d509c47344 feat: add conf load api 2023-07-04 14:17:11 +08:00
zhongwencool bd83365d45 fix: update cluster.conf's k8s default api_server value 2023-06-30 09:55:36 +08:00
firest 71d34504bf fix(cluster_rpc): clean self-data by mria stop callback 2023-06-29 01:45:47 +00:00
firest 51ec8cb8fa fix(emqx_conf): don't sync the MFA to a leaved node 2023-06-29 01:45:47 +00:00
zhongwencool d556adf86c
Merge pull request #11146 from zhongwencool/hide-node-process-limit
feat: hide node.process_limit
2023-06-28 10:45:37 +08:00
Zaiming (Stone) Shi 03d05825db docs: hide core_nodes config doc
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
zhongwencool b4ae4260c0 feat: hide node.process_limit 2023-06-27 18:09:26 +08:00
Thales Macedo Garitezi f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Zaiming (Stone) Shi 5f47649d4d chore: downgrade log severity level of 'not_ready' nodes to info 2023-06-21 09:45:45 +02:00
Zaiming (Stone) Shi 045ed340dd fix(emqx_conf_app): fix release version detect during cluster conf sync 2023-06-21 09:28:46 +02:00
Zaiming (Stone) Shi b588b88a60 chore: change autoclean interval default value to 24h
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).

To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.

This PR also has the autoclean config removed from examples.
2023-06-20 15:43:37 +02:00
lafirest 8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example
chore: add examples of how to configure
2023-06-20 20:09:44 +08:00
Zaiming (Stone) Shi f95787d229 test: make possible to test core/replica locally 2023-06-20 12:42:00 +02:00
firest bb2bf059ac chore: modify the build to copy examples into the release 2023-06-20 14:53:14 +08:00
zhongwencool 36ee872361 chore: hide plugins from conf load cli 2023-06-20 10:34:09 +08:00
JimMoen c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal 2023-06-16 14:04:24 +08:00
zhongwencool 9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename 2023-06-15 20:51:52 +08:00
zhongwencool e6c2da8d19 feat: rename log.file.to to log.file.path 2023-06-15 18:24:21 +08:00
zhongwencool 0adbb0deec feat: try to save config when post_config_update failed not in init mfa status 2023-06-15 17:19:11 +08:00
zhongwencool 9a0856fd6b fix: authn/authz's keys should be binary 2023-06-14 18:00:36 +08:00
zhongwencool bd33dcb049 feat: remove zone from /clients api 2023-06-14 17:11:20 +08:00
zhongwencool 121cd9124a chore: improve show_keys output 2023-06-14 15:53:55 +08:00
zhongwencool 9b5c7065f2 feat: conf reload support --replace|merge 2023-06-14 09:12:39 +08:00
zhongwencool 095fb040c0 chore: format conf cli usage text 2023-06-14 09:12:39 +08:00
zhongwencool a8959900d0 feat: support --replace|merge with conf load 2023-06-14 09:12:39 +08:00
zhongwencool 505d677685 chore: move reload_local_etc_config from emqx_conf to emqx_conf_cli 2023-06-11 09:11:17 +08:00
zhongwencool cf9a207743 feat: check all config schema before loading 2023-06-11 09:11:17 +08:00
zhongwencool b829d8edc2 feat: support --auth-chains merge|replace mod 2023-06-11 09:11:17 +08:00
zhongwencool 994db58cbf fix: load authz's default sources 2023-06-11 09:11:17 +08:00
zhongwencool c819ac27f4 feat: add conf reload cli 2023-06-11 09:11:17 +08:00
Zaiming (Stone) Shi 97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi 38e767b808 fix(schema): map new duration types to dashboard's type name 2023-06-09 09:14:55 +02:00