Commit Graph

660 Commits

Author SHA1 Message Date
Tobias Lindahl ec429857e0 refactor(persistent_sessions): code format changes after review 2021-10-21 15:23:21 +02:00
Tobias Lindahl 655d31726a feat(persistent_session): add early removal of delivered message 2021-10-21 15:23:21 +02:00
Tobias Lindahl b9b8111edb test: add new proper types to #session{} after rebase 2021-10-21 15:23:21 +02:00
Tobias Lindahl fd71bc50ab fix(persistent_session): Make sure to discard expired sessions on reconnect 2021-10-21 15:23:21 +02:00
Tobias Lindahl f2d26f5e03 test(persistent_sessions): Remove bogus stopping of quicer 2021-10-21 15:23:21 +02:00
Tobias Lindahl 12639addb9 fix(persistent_session): Make sure the connection processs is dead 2021-10-21 15:23:21 +02:00
Tobias Lindahl ed0f435546 chore(persistent_sessions): correct copyright information in new files 2021-10-21 15:23:21 +02:00
Tobias Lindahl fdba441f4f fix(persistent_sessions): revert bogus table type change 2021-10-21 15:23:21 +02:00
Tobias Lindahl 7808ffe600 fix(persistent_sessions): protect (un)subscribe from calling session router 2021-10-21 15:23:21 +02:00
Tobias Lindahl 7b394267dd feat: persistent sessions stored in mnesia
* Persistent sessions can survive node/connection process down
* Internal SessionID is generated, external ClientID is as before
* Sessions are persisted to mnesia
* A session router is added in parallel to the ordinary router
* Messages that are subscribed to by a persistent session are
  persisted by publisher
* Information about persisted and delivered messages are stored in
  mnesia per session.
* A resume protocol similar to takeover is implemented for resuming
  from mnesia
* Can be configured (and enabled) by the top lever config
  "persistent_session_store"
2021-10-21 15:23:21 +02:00
Tobias Lindahl fc7b4c0009 refactor: make it possible to use different tries
In preparation for persistent sessions
2021-10-21 15:23:21 +02:00
Tobias Lindahl 24e870672c refactor: use set_session in preparation for persistent sessions 2021-10-21 15:23:21 +02:00
Ilya Averyanov e37ecc10a6 style(behaviour attrs): unify behaviour declarations 2021-10-21 15:10:59 +03:00
zhongwencool d784e63b9f
emqx_conf (#5939)
* feat(emqx_conf): move conf manager for emqx_machine to emqx_conf

* chore(emqx_conf): change emqx:get_config/2 to emqx_conf:get/2

* fix: common test failed

* fix: badmatch by typo wrong key

* fix(emqx_conf): get the wrong core nodes

* fix(emqx_conf): get core node's tnx_id not latest tnx_id

* fix: add ro_transation when copy conf file

* fix: delete debug info

* fix: change ekka_rlog to mria_rlog

* fix: remove cluster_rpc from emqx_machine.

* fix: don't call ekka:start/0 explicitly

* fix: ekka should be start in emqx_machine
2021-10-21 18:08:51 +08:00
k32 9fdd5e6a7e
fix(emqx_machine): Fix start/stop callbacks (#5969)
* fix(emqx_machine): Fix start/stop callbacks

* chore(ekka): Bump version to 0.11.1

* fix(router): Wait for the tables

* fix(emqx_cluster_rpc): Stop cluster RPC when joining a cluster

* fix(emqx_app): Fix a deadlock when joining the cluster

* fix(emqx_telemetry): Wait for mnesia tables

* test(ct_helper): Start ekka before emqx
2021-10-21 14:49:21 +08:00
Zaiming (Stone) Shi ed069cfecc
Merge pull request #5963 from zmstone/refactor-authn-schema
refactor(authn): check authenticator config with provider module
2021-10-21 08:07:48 +02:00
tigercl 45b484a26a
Merge pull request #5967 from tigercl/fix/authn-mysql
fix(authn): fix placeholders are not replaced correctly
2021-10-21 10:20:05 +08:00
zhouzb ef731c74eb chore(deps): unify version of hocon 2021-10-21 09:35:00 +08:00
William Yang 7b3537d822
Merge pull request #5966 from qzhuyan/dev/william/lc-0.1.2
feat(olp): bump to 0.1.2
2021-10-20 21:57:31 +02:00
Zaiming Shi ce5b456872 refactor(authn): make schema doc generation work 2021-10-20 17:42:06 +02:00
Zaiming Shi fa91e5f583 fix(authn): allow single authenticator instance from config 2021-10-20 15:06:08 +02:00
Zaiming Shi 019b9d17f6 refactor(emqx_config_handler): async remove 2021-10-20 14:12:46 +02:00
k32 8e58699ff5
Merge pull request #5937 from k32/mria
feat(mria): Replace ekka_mnesia with mria
2021-10-20 13:15:23 +02:00
Zaiming Shi 88d891a59a feat: support check_config callback for authenticator provider 2021-10-20 13:04:55 +02:00
Zaiming Shi 72c26931d7 fix: fill string fields' default value with binary 2021-10-20 11:52:30 +02:00
Zaiming Shi 1b9c082563 refactor(authn): check authenticator config with provider module
mainly two changes:

1. the schema is simplified at root level, per-authenticator checks
   are done after the type can be identified
2. the config handling part is split out from emqx_authentication
   module to emqx_authentication_config module
2021-10-20 10:13:11 +02:00
Zaiming Shi 0fcb3a1e93 docs: add more schema docs for authz 2021-10-20 09:17:21 +08:00
Zaiming Shi 0cf5aa5611 test: remove cuttlefish code 2021-10-20 09:17:21 +08:00
William Yang d3ddecb144 feat(olp): bump to 0.1.2
remove noop logging
2021-10-19 21:20:12 +02:00
Zaiming Shi 2e01eeb6b4 fix(emqx_schema): typo in nullable prop 2021-10-19 15:42:34 +02:00
William Yang c780a1a53c
Merge pull request #5957 from qzhuyan/dev/william/quic-0.0.9
feat(quic): bump quicer 0.0.9
2021-10-19 14:52:08 +02:00
William Yang 56d8e0f59f feat(quic): bump quicer 0.0.9 2021-10-19 13:54:43 +02:00
k32 b3a8753219 fix(mria): Replace mnesia calls with mria 2021-10-19 12:24:35 +02:00
tigercl bd53c488d1
Merge pull request #5950 from tigercl/fix/authn-version
fix(authn): fix the authentication instance cannot run after the update
2021-10-19 09:58:09 +08:00
zhouzb 31e2d9d2c0 fix(authn): fix the authentication instance cannot run after the update 2021-10-19 09:05:54 +08:00
k32 0d2d5f6bf9 fix(emqx_alarm): Wait for tables 2021-10-18 22:40:18 +02:00
k32 37a1c45af0 fix(dialyzer): Fix problems found by dialyzer 2021-10-18 22:04:05 +02:00
k32 07ea6e5689 fix(mria): Fix startup sequence 2021-10-18 22:04:05 +02:00
k32 fcdf10080d chore(ct): ekka_mnesia -> mria_mnesia 2021-10-18 22:04:05 +02:00
k32 d14b8e7da9 chore(mria): ekka_rlog -> mria_rlog 2021-10-18 22:04:05 +02:00
k32 326923850c chore(mria): Get rid of copy_mnesia callback 2021-10-18 22:04:05 +02:00
k32 e14a62d4d6 chore(mria): ekka_mnesia:start/stop -> mria:start/stop 2021-10-18 22:04:05 +02:00
k32 9965d6e028 chore(mria): ekka_mnesia:create_table -> mria:create_table 2021-10-18 22:04:05 +02:00
k32 ae2056da1b chore(mria): Fix clear_table and ro_transaction calls 2021-10-18 22:04:05 +02:00
k32 ff48322e0c chore(mria): ekka_mnesia:running_nodes -> mria:running_nodes 2021-10-18 22:04:05 +02:00
k32 b515a45a4f chore(mria): Rename module: ekka_mnesia -> mria 2021-10-18 22:04:05 +02:00
zhanghongtong b1bf21c73b fix(emqx schema): fix function clause by user_lookup_fun 2021-10-18 16:37:16 +08:00
zhongwencool c73205a589
Ct helpers migrate (#5943)
* chore: common test

* chore: delete emqx_ct_helpers deps
2021-10-18 13:45:52 +08:00
Zaiming Shi ad195d0eb0 chore(emqx_vm): delete unused function get_process_group_leader_info 2021-10-16 21:10:43 +02:00
Zaiming Shi 71a11834ad chore: pin rebar plugins and test deps 2021-10-15 16:46:49 +02:00
Zaiming Shi 68d91e770b test: remove ct_helpers from apps/emqx rebar.config 2021-10-15 16:11:48 +02:00
Tobias Lindahl aa19c2a908 chore: Remove emqx_ct_helpers as dependency 2021-10-15 10:40:57 +02:00
William Yang 30a0635ea5 fix: bump ekka to 10.0.9
fix ekka etcd cluster ssl options
2021-10-14 22:49:25 +02:00
William Yang dcca1d7544
Merge pull request #5867 from qzhuyan/dev/william/olp
feat(olp): first PR for overload protection
2021-10-14 10:00:01 +02:00
Shawn 9c7eef5295 feat(rules): update rule_engine configs from APIs 2021-10-14 15:09:37 +08:00
Shawn c6c314de1a fix(hocon): update hocon to 0.19.6 2021-10-14 15:09:37 +08:00
lafirest 8a2c5e2422 feat(rate_limiter): implement hierarchical token buckets 2021-10-14 10:20:05 +08:00
William Yang 23fc8afc50 feat(olp): rename olp mgmt API
on -> enable
off -> disable
2021-10-13 10:18:57 +02:00
William Yang 58033c083d chore(olp): update alarm text 2021-10-12 16:08:20 +02:00
lafirest 3524942766 fix: change all ~s in io format to ~ts
we should use unicode to replace latin1
2021-10-12 15:09:24 +08:00
Zaiming (Stone) Shi 04f2487108
Merge pull request #5908 from zmstone/refactor-emqx-more-slog
refactor(emqx): fix remaining legacy logging
2021-10-12 06:45:25 +02:00
JimMoen 19456d84a5 style: code format fix 2021-10-12 10:37:45 +08:00
William Yang b0c27c74fd feat(olp): bump lc to 0.1.1 2021-10-12 00:00:20 +02:00
Zaiming Shi eb43423552 refactor(emqx): fix remaining legacy logging 2021-10-11 21:51:39 +02:00
Zaiming Shi 7e9d7c4858 test(proper): move emqx_ct_proper_types to local repo
previously provided by emqx-ct-helpers
2021-10-11 15:28:12 +02:00
Spycsh d4657754eb chore(dialyzer): fix spec for record field message.extra 2021-10-11 14:10:22 +02:00
Zaiming Shi 71731c01f1 refactor(emqx): refine SLOG messages
unified logging for unexpected handle_cast handle_call and handle_info
2021-10-11 07:41:52 +02:00
Zaiming Shi 785793b345 refactor: add logging allow or not check for SLOG 2021-10-11 07:41:52 +02:00
Zaiming Shi 5274e0a726 refactor(emqx): add 'extra' field to message record 2021-10-11 02:07:33 +02:00
Spycsh fe5a169be1 chore: change to structured logging under apps/emqx/src 2021-10-09 14:02:22 +08:00
Shawn 63bf161522 fix(test): testcase t_validity_period failed occasionally 2021-10-09 09:17:09 +08:00
Zaiming (Stone) Shi aa0c58daa0
Merge pull request #5881 from zmstone/fix-flaky-tests-cm-session-race-condition
chore: fix flaky test cm_SUITE open session racecondition
2021-10-07 17:28:10 +02:00
William Yang 8d8969672b feat(olp): add desc in schema 2021-10-07 16:03:54 +02:00
Zaiming Shi a882a8c496 chore: fix flaky test cm_SUITE open session racecondition 2021-10-07 11:51:40 +02:00
Zaiming Shi 7ae4505636 test(emqx_broker): fix flaky tests 2021-10-07 11:31:22 +02:00
William Yang 77f8159ca1 docs(olp): emqx_olp module 2021-10-06 21:07:14 +02:00
William Yang 9ecb7821f9 feat(olp): use tagged deps libs 2021-10-06 10:38:39 +02:00
Zaiming Shi ce486e5540 fix(ssl): verify ciphers list against all available ciphers
Prior to this change the ciphers are only checked against
the list returned from from
`ssl:cipher_suites(all, 'tlsv1.2', openssl)`
which may cause some (weak) ciphers missing in certain
otp + openssl installation
2021-10-05 15:25:56 +02:00
Zaiming Shi b42a2f2bc2 fix(ssl): delete some weak cipher suites from the default list 2021-10-05 15:15:00 +02:00
William Yang afb4b5dbd7 test(olp): add CT suite emqx_olp_SUITE 2021-10-05 14:37:39 +02:00
William Yang bfe4346469 feat(olp): alarm without nodeid 2021-10-05 14:36:36 +02:00
William Yang 547484a2d1 chore(olp): add spec. 2021-10-05 09:47:21 +02:00
William Yang 6baf2dbd95 feat(olp): backoff new conn 2021-10-05 09:23:23 +02:00
William Yang 370edac031 feat(olp): add default config 2021-10-04 16:30:48 +02:00
William Yang 4dc63b26a8 feat(olp): metrics 2021-10-04 16:30:44 +02:00
William Yang 67267acb70 feat(olp): management API 2021-10-04 14:36:30 +02:00
William Yang eb895a9f80 feat(olp): quic 2021-10-04 12:31:14 +02:00
William Yang 9304e3c122 feat(olp): add config and backoff gc 2021-10-04 12:21:30 +02:00
William Yang 166f02edc4 feat(olp): don't hibernate conn proc when overloaded 2021-10-04 10:44:53 +02:00
William Yang e9710ade14 feat(olp): first commit for overload protection
- add lc app
- add alarm handler for lc runq alarm
- backoff when handling CONNECT message
- close new connswhen overload
2021-10-01 16:14:38 +02:00
zhongwencool 93c210887b chore: don't use emqx_machine.hrl in emqx 2021-09-30 16:04:30 +08:00
zhongwencool 44a6f04a45 chore: reorganize shards position 2021-09-30 16:04:30 +08:00
zhongwencool be123f613d
Fix: apps should boots after emqx_machine_sup's children spawn. (#5851)
* fix(cluster_call): apps should start after cluster_call init

* fix: undef function

* chore: reformat code

* fix: ekka must start before emqx
2021-09-30 13:56:18 +08:00
zhanghongtong b94dcb7bc7 chore(release): update emqx release version 2021-09-29 18:29:44 +08:00
tigercl 55ec5bab9e
Merge pull request #5773 from tigercl/feat/psk-file
feat(psk): support psk
2021-09-29 13:37:49 +08:00
Zaiming (Stone) Shi 83e7ab681b
Merge pull request #5820 from JimMoen/refactor-frame-parse-logging
Refactor frame parse logging with bug fix.
2021-09-28 19:31:39 +02:00
Zaiming Shi 225ebdaa87 fix(frame): simlify error excption context
We should not raise error with the exact context after logging it
because it will end up duplicated in the log
2021-09-28 18:15:32 +02:00
JimMoen c1ff30896a fix(frame): safely serializing and structured log. 2021-09-28 18:55:25 +08:00
JimMoen 9832eea250 fix(frame): safely parsings. 2021-09-28 14:11:18 +08:00
JimMoen 7d6180f10a fix(frame): variable byte integer could be larger than 4 bytes. 2021-09-28 14:11:18 +08:00
Zaiming Shi a158b0235d chore(tests): use assertError
the ?catch_error macro passes when there is no exception
2021-09-28 14:11:18 +08:00
Zaiming Shi fec3c6574e perf(emqx_frame): skip empty tail 2021-09-28 14:11:18 +08:00
Zaiming Shi 9d0ef5de6b refactor(emqx_frame): log frame parse error with more context 2021-09-28 14:11:18 +08:00
tigercl 79685a77ba
Merge branch 'master' into fix/authn2 2021-09-28 09:27:08 +08:00
Zaiming (Stone) Shi 3df21a9e14
Merge pull request #5822 from Spycsh/fix-windows-compile
chore: fix Windows compilation process
2021-09-27 23:30:59 +02:00
Zaiming Shi 697a11ded0 fix(tls): drop unsupported ciphers
we use a hard-coded list of pre-selected ciphers as config
default value. some of them may not be supported by the
underlying openssl lib.

now moved the pre-selected ciphers to emqx_tls_lib:selected_ciphers
which performs a filtering before return.
2021-09-27 20:51:48 +02:00
Shawn e2721c144c feat(bridge): support http bridge 2021-09-27 22:01:18 +08:00
zhouzb 74c9a38e9f fix(psk): fix bugs and add test case 2021-09-27 21:42:32 +08:00
Zaiming Shi 5417eb328a fix(schema): no ciphers validator for quic listener 2021-09-27 13:53:54 +02:00
Zaiming Shi 58ffc4651f fix(config): use default value for tls versions
default value for ssl.versions is dynamically resolved based
on otp version and underlying openssl installation
2021-09-27 13:53:54 +02:00
Zaiming Shi d376c0f9fc refactor(schema): call emqx_tls_lib for default tls versions 2021-09-27 13:53:54 +02:00
Zaiming Shi e7e8b8c77b fix(schema): check tlsv1.3 availability 2021-09-27 13:53:54 +02:00
zhouzb f9aa345787 chore(psk): improve logs for psk 2021-09-27 15:32:56 +08:00
zhouzb 2262bf508e chore(authn): improve code 2021-09-27 14:50:22 +08:00
zhongwencool 5280f83651
Merge pull request #5784 from zhongwencool/delayed-api-schema
feat(delayed_api): support hocon schema
2021-09-27 13:45:17 +08:00
Shawn 69f3cce75d feat(rules): hook on bridges events and query bridges from rules 2021-09-26 23:00:08 +08:00
Shawn bd081913b5 refactor(rules): remove emqx_rule_actions 2021-09-26 23:00:08 +08:00
lafirest acb5c693ba
Merge pull request #5767 from lafirest/fix/time_unit
fix(schema): fix some time unit in schema
2021-09-26 15:35:27 +08:00
JianBo He b78c914651 test(authn): cleanup the dirty configs 2021-09-26 15:19:55 +08:00
JianBo He f3c675b139 chore(deps): upgrade esockd to 5.8.3 2021-09-26 15:19:55 +08:00
lafirest 6e3ec6c9db fix(schema): fix authn/z's schema time unit 2021-09-26 14:41:46 +08:00
lafirest e31e175e47 fix(schema): fix some time unit in schema 2021-09-26 14:38:53 +08:00
zhouzb 096e85dc14 chore(authn): improve log and fix disabled authentication still working 2021-09-26 11:31:19 +08:00
zhouzb 6051ea92eb chore(authn): improve log 2021-09-24 17:24:04 +08:00
zhouzb e31840d943 fix(authn): add timeout option for mysql connector 2021-09-24 17:15:10 +08:00
zhouzb d391690505 chore(psk): fix bugs 2021-09-24 17:07:08 +08:00
zhouzb e9cd757432 chore(psk): delete old module and update configuration 2021-09-24 17:07:05 +08:00
zhouzb 7c61bc18cf feat(psk): support psk 2021-09-24 17:05:26 +08:00
zhongwencool 8c441673c2 feat(delayed_api): support hocon schema 2021-09-24 15:22:05 +08:00
JimMoen e737f18548 fix(mgmt_api): Friendly HTTP Status Code for Listeners. 2021-09-24 14:28:52 +08:00
Zaiming Shi 3027bc3a0c fix(schema): sll key and cert files are nullable 2021-09-23 16:57:16 +02:00
Zaiming Shi 4392357877 fix(schema): simplify ssl ciphers filed schema 2021-09-23 16:57:16 +02:00
Zaiming Shi 4f638b8242 fix(schema): upgrade to hocon 0.19.5
renamed no_conversion option to only_fill_defaults
2021-09-23 16:57:15 +02:00
Zaiming Shi 5bd67a49de chore(schema): reformat schema fields descriptions 2021-09-23 16:56:02 +02:00
Zaiming Shi 97e1cf65b7 refactor(schema): make a client ssl options schema
client and server ssl options share some common fields
this commit make an abstraction for the common fields
then export server_ssl_options_schema/2 and client_ssl_options_schema/1
for other schema modules to call
2021-09-23 16:56:02 +02:00
Zaiming Shi fbd5701989 fix(emqx_schema): make ssl config schema right 2021-09-23 16:56:02 +02:00
zhouzb 4860049268 chore(authn): insert final newline 2021-09-23 10:11:54 +08:00
zhouzb 79026d5900 chore(authn): add supervisor for authn and fix checking errors 2021-09-23 10:11:50 +08:00
tigercl 2b9870992c
Merge pull request #5763 from tigercl/feat/clear-certs
feat(clear certs): clear certs when deleting instance
2021-09-23 09:43:56 +08:00
zhongwencool 81a795ffc9
Merge pull request #5728 from zhongwencool/swagger-hocon-spec
chore(swagger): swagger support hocon schema
2021-09-22 17:42:07 +08:00
zhongwencool 085d8e8efa chore(schema): restore space 2021-09-22 16:41:00 +08:00
zhongwencool dab5fbf285 feat(swagger): swagger support hocon schema 2021-09-21 21:41:16 +08:00
Zaiming Shi 2edd3879b8 feat(config): save override config in hocon pretty-print format 2021-09-19 14:33:43 +02:00
zhouzb bb4e5fdb4f chore(authn): fix dialyzer 2021-09-18 09:14:03 +08:00
zhouzb 69755ad3fb fix(authn): fix bug in authn 2021-09-18 09:14:03 +08:00
zhouzb b89973ce7c fix(authn): fix type error 2021-09-18 09:14:00 +08:00
zhouzb 35a4a05f03 feat(clear certs): clear certs when deleting instance 2021-09-18 09:11:41 +08:00
JimMoen 6edc9f4221 fix(emqx_types): spec refs `emqx_types.erl` instead directly. 2021-09-17 19:28:22 +08:00
JimMoen 9c95557bfc fix(emqx_types): type spec refs does not exist file. 2021-09-17 19:28:22 +08:00
JianBo He cfabb8549e build(deps): upgrade emqx_http_lib to 0.4.1 2021-09-17 18:13:57 +08:00