Commit Graph

9656 Commits

Author SHA1 Message Date
William Yang f52b4a9e32 fix: gen_rpc tcp client port align with server port 2023-10-25 12:11:35 +02:00
William Yang 37dd44c3d7
Merge pull request #11813 from qzhuyan/fix/william/genrpc-ssl-port-align
Fix/william/genrpc ssl port align
2023-10-25 09:04:19 +02:00
William Yang 6346e0d28a fix(gen_rpc): ssl client port align with server port 2023-10-24 22:33:52 +02:00
Zaiming (Stone) Shi abcb3166f6 fix: use binary string for raw default values 2023-10-24 14:46:01 +02:00
Stefan Strigler 6e12569260
Merge pull request #11797 from sstrigler/EMQX-1057-fix-authz-api-donot-return-200
fix: return 404 if built_in_database not configured as auth source
2023-10-24 12:08:53 +02:00
Stefan Strigler ea9228108b fix: use minirest filter 2023-10-24 09:57:08 +02:00
JimMoen 99fab8dc5d
chore: missing change log for duplicated apikey fixing 2023-10-24 09:44:39 +08:00
Thales Macedo Garitezi d392c9989f
Merge pull request #11793 from thalesmg/test-rule-engine-foreach-m-20231019
test(rule_engine): add test to exemplify `foreach` with json array payload
2023-10-23 13:43:33 -03:00
Stefan Strigler 4e0e755b28 fix: return 404 if built_in_database not configured as auth source 2023-10-23 16:26:41 +02:00
Ilya Averyanov 8c6f6b56b8 chore(tests): report more friendly about unset PROFILE variable 2023-10-23 13:48:42 +03:00
Ilya Averyanov edde661da3 fix(authn): fix pbkdf2 option validation 2023-10-23 10:26:11 +03:00
William Yang a9b42af8b3
Merge pull request #11796 from qzhuyan/fix/william/gen_rpc-driver
fix(gen_rpc): ensure client/server use same driver
2023-10-23 08:24:17 +02:00
JimMoen e6576951ef
test: cleanup duplicated apikey with different name 2023-10-20 22:33:26 +08:00
JimMoen d467289bb2
fix: avoid duplicated apikey from data import 2023-10-20 21:13:04 +08:00
William Yang dd5807b797 fix(gen_rpc): ensure client/server use same driver
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
Thales Macedo Garitezi 8c4beec6f0 test(rule_engine): add test to exemplify `foreach` with json array payload
Relates to https://emqx.atlassian.net/browse/EMQX-11174
2023-10-19 18:19:42 -03:00
Ilya Averyanov 0b4600c293 feat(auth): improve redis command parsing and validation 2023-10-19 23:01:35 +03:00
Thales Macedo Garitezi eda82caf65
Merge pull request #11791 from thalesmg/fix-coap-heartbeat-m-20231018
fix(coap): increase received packet counter for keepalive
2023-10-19 13:12:41 -03:00
Ilya Averyanov 8d82c30b00
Merge pull request #11771 from savonarola/1015-validate-bcrypt-schema-in-api
feat(authn): allow authn providers to define a separate schama for API
2023-10-19 15:34:34 +03:00
Ilya Averyanov 8f9ac6c1f3
Merge pull request #11776 from savonarola/1017-auth-cleanup-2
chore(authn): remove dead code
2023-10-19 15:28:35 +03:00
Thales Macedo Garitezi 5b9866f630 fix(coap): increase received packet counter for keepalive
Fixes https://emqx.atlassian.net/browse/EMQX-11193
Fixes https://github.com/emqx/emqx/issues/11779
2023-10-18 16:35:33 -03:00
zhongwencool 2388d36b09 fix: allow viewers to change their own passwords 2023-10-18 18:40:02 +08:00
Ilya Averyanov e1c8317779 chore(authn): remove dead code 2023-10-18 10:57:55 +03:00
Ilya Averyanov c0ebaf9b6f
Merge pull request #11541 from savonarola/0829-duplicate-ft-responses
Support asynchronous mode of receiving responses to FT operations
2023-10-17 17:08:00 +03:00
Ilya Averyanov 6354f3b04f feat(authn): allow authn providers to define a separate schama for API 2023-10-17 13:19:11 +03:00
Zaiming (Stone) Shi 55618dd28a
Merge pull request #11731 from zmstone/1008-expose-file-transfer-host-conf
fix(api-docs): add file-transfer config to hot-config scope
2023-10-17 11:23:16 +02:00
Zaiming (Stone) Shi dfa5909392
Merge pull request #11752 from zmstone/1011-change-default-rpc-module-for-mria
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
2023-10-17 10:21:14 +02:00
Zaiming (Stone) Shi 162b901be6
Merge pull request #11763 from zmstone/1114-sync-release-53
Merge release-53 back to master
2023-10-16 21:42:02 +02:00
JianBo He 16cc816bd3
Merge pull request #11757 from zhongwencool/trace-api-return-500
fix: 500 error response when downloading non-existent trace files
2023-10-16 13:40:18 +08:00
Zaiming (Stone) Shi 6eb3bb7cff Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53 2023-10-14 10:16:38 +02:00
Zaiming (Stone) Shi 7c022c2c6a refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-10-14 10:13:31 +02:00
Ilya Averyanov cd2752117c
Merge pull request #11762 from savonarola/1013-fix-mnesia-authz-destroy
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 20:54:14 +03:00
Ilya Averyanov bde8800f2e fix(mnesia authz): destroy authz records on mnesia authz destroy 2023-10-13 18:29:45 +03:00
Ilya Averyanov 90a0c093bf
Merge pull request #11750 from savonarola/1011-redact-auth-http-request
Do not trace authn http request bodies
2023-10-13 17:25:28 +03:00
Ilya Averyanov 4f14e8df22 chore(source dir): split out postgresql connector 2023-10-13 16:58:57 +03:00
Thales Macedo Garitezi 7a6c756b7c
Merge pull request #11754 from thalesmg/fix-postgres-error-reason-log-m-20231011
fix(postgres): format unicode error messages from driver
2023-10-13 09:58:28 -03:00
Andrew Mayorov 0144ed9981
Merge pull request #11753 from keynslug/chore/smaller-ci-artifacts
chore(ci): try to reduce CI artifacts footprint
2023-10-13 16:56:47 +07:00
Andrew Mayorov 51c57a5f0a
fix(cass): avoid using aggregates in healthcheck query
Otherwise, this query will produce warning message per each
healthcheck, telling the user:
```
WARN  <...> - Aggregation query used without partition key
```
2023-10-13 16:50:19 +07:00
Andrew Mayorov 083e2da347
chore(bpapicheck): make some failures more user-friendly 2023-10-13 15:42:08 +07:00
Andrew Mayorov 0b9ac24c1e
fix(bpapicheck): stop depending on git index 2023-10-13 15:40:49 +07:00
lafirest 96c546c187
Merge pull request #11725 from lafirest/feat/os-ldap
chore: change the LDAP integration to opensource
2023-10-13 15:05:39 +08:00
Zaiming (Stone) Shi dd03be29a6
Merge pull request #11751 from zmstone/1011-refactor-move-emqx_license-app-to-apps
1011 refactor move emqx_license app to apps
2023-10-13 06:45:49 +02:00
firest 33ff5d5588 chore: update auth header file 2023-10-13 10:16:47 +08:00
Ilya Averyanov 68f31a9da2 fix(authn): do not trace authn requests 2023-10-12 16:32:16 +03:00
Ilya Averyanov 08795f559c
Merge pull request #11336 from savonarola/0724-trace-authz
Trace non-resultative authz calls
2023-10-12 14:51:15 +03:00
Andrew Mayorov 6413afd0bc
fix(gcpdev): restore original shard in mnesia tab definition
Changing the shard is not backward compatible, and leads to a crash
when upgrading from 5.2.x.
2023-10-12 17:30:07 +07:00
Ilya Averyanov 4ecd5e17a2 chore(authz): trace non-resultative authz calls to backend modules 2023-10-12 12:29:39 +03:00
firest 3e658b3da9 chore: update changes 2023-10-12 17:15:47 +08:00
firest 9b4def885a chore: change the LDAP integration to opensource 2023-10-12 16:37:32 +08:00
zhongwencool 1021088f5f fix: 500 error response when downloading non-existent trace files 2023-10-12 15:23:23 +08:00
Thales Macedo Garitezi b07dddd49e fix(postgres): format unicode error messages from driver
Fixes https://emqx.atlassian.net/browse/EMQX-11024

Sample error:

```
{error, error, <<"42501">>, insufficient_privilege,
<<229,175,185,232,161,168,32,109,113,116,116,95,117,115,101,114,32,230,157,131,233,153,144,228,184,141,229,164,159>>,
[]}
```
2023-10-11 17:56:12 -03:00
Ilya Averyanov 534c9bdc13 feat(ft): add additional operation status report channel 2023-10-11 19:51:36 +03:00
Zaiming (Stone) Shi d9859b6b49
Merge pull request #11749 from thalesmg/test-fix-flaky-http-suite-r53-20231011
test: attempt to fix flaky http bridge suite
2023-10-11 16:07:17 +02:00
William Yang e6358ca18c
Merge pull request #11747 from qzhuyan/dev/william/quicer-0.0.202
bump to quicer 0.0.202
2023-10-11 15:43:47 +02:00
Zaiming (Stone) Shi e069680bca refactor: move emqx_license app from lib-ee to apps 2023-10-11 15:41:35 +02:00
Thales Macedo Garitezi ab662f506d test: attempt to fix flaky http bridge suite
```
 =CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
  crasher:
    initial call: ehttpc:init/1
    pid: <0.23358.2>
    registered_name: []
    exception error: bad argument
      in function  persistent_term:get/1
         called as persistent_term:get({emqx_bridge_http_SUITE,
                                        do_t_async_retries,attempts})
         *** argument 1: no persistent term stored with this key
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-11 09:13:25 -03:00
Ilya Averyanov 57655854d1
Merge pull request #11743 from savonarola/1010-auth-leftovers
chore(auth): cleanup code
2023-10-11 15:09:07 +03:00
William Yang a69c53455f fix: bump to quicer 0.0.202
bring in urgent fixs from msquic 2.2.3
2023-10-11 13:12:32 +02:00
zhongwencool cc392cadb8
chore: update apps/emqx_retainer/src/emqx_retainer_mnesia_cli.erl
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-10-11 18:40:09 +08:00
Ilya Averyanov 03ae5bf3c8 chore(auth): cleanup code 2023-10-11 13:13:50 +03:00
zhongwencool f4a88f717b feat: support retainer Start Length command 2023-10-11 11:47:37 +08:00
zhongwencool 8e7ba16c3a fix: nothing show when run clients list command 2023-10-11 11:47:15 +08:00
zhongwencool 574dc2f243 fix: observer load command crash when loading noexist module 2023-10-11 10:36:33 +08:00
Zaiming (Stone) Shi a2e86c67db
Merge pull request #11733 from keynslug/fix/EMQX-10827/sesson-takeover
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
2023-10-10 19:03:29 +02:00
Zaiming (Stone) Shi 0f07f678e7
Merge pull request #11742 from zmstone/1010-sync-release-53-to-master
1010 sync release-53 to master
2023-10-10 18:56:23 +02:00
Thales Macedo Garitezi 196b24c0cb
Merge pull request #11741 from thalesmg/test-fix-flaky-http-test-m-20231010
test: attempt to fix flaky http bridge suite
2023-10-10 13:52:08 -03:00
Thales Macedo Garitezi ee45145fb5
Merge pull request #11724 from thalesmg/fix-kprodu-sync-metrics-m-20231006
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers
2023-10-10 12:13:33 -03:00
Thales Macedo Garitezi ec588f94e2 test: attempt to fix flaky http bridge suite
```
 =CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
  crasher:
    initial call: ehttpc:init/1
    pid: <0.23358.2>
    registered_name: []
    exception error: bad argument
      in function  persistent_term:get/1
         called as persistent_term:get({emqx_bridge_http_SUITE,
                                        do_t_async_retries,attempts})
         *** argument 1: no persistent term stored with this key
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-10 09:50:53 -03:00
Ivan Dyachkov bfb2218392 chore: fix release version 2023-10-10 14:50:49 +02:00
Thales Macedo Garitezi cf2075d7d8 chore: remove mention of `is_buffer_supported` from typespec 2023-10-10 09:49:18 -03:00
Zaiming (Stone) Shi 143673b703 chore: bump release version to 5.3.1-alpha.1 2023-10-10 14:35:46 +02:00
Zaiming (Stone) Shi 0a9e1dfa88
Merge pull request #11734 from zmstone/1002-support-ipv6-only-clustering
Make IPv6 clustering options configurable
2023-10-10 14:29:52 +02:00
Andrew Mayorov 5fff2ffe45
test(kafka): try to stabilize `t_dynamic_mqtt_topic/1` testcase 2023-10-10 17:18:24 +07:00
Andrew Mayorov 9b573834f9
Merge pull request #11736 from keynslug/fix/eval-erl-audit
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 15:49:19 +07:00
Zaiming (Stone) Shi 2626d793a7 test: try to resolve schema module from PROFILE in tests 2023-10-10 10:24:10 +02:00
Zaiming (Stone) Shi 808237364c fix(api-docs): add file-transfer config to hot-config scope 2023-10-10 10:24:10 +02:00
Zaiming (Stone) Shi e7e696cd66 feat(rpc): add ipv6_only config 2023-10-10 10:05:14 +02:00
firest e07937a3ef fix(ldap): escape the escape character (\) 2023-10-10 13:49:36 +08:00
Andrew Mayorov 2a291dfd27
fix(audit): handle abstract forms when logging `eval_erl` 2023-10-10 12:31:19 +07:00
Andrew Mayorov 522302fee1
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
Which was accidentally broken in bf164175.
2023-10-10 01:34:10 +07:00
Zaiming (Stone) Shi 1e93d2f1fc chore: upgrade to ekka 0.15.16 gen_rpc 3.2.0
* ekka 0.15.16 supports 'inet6_tls' as ekka.proto_dist
* gen_rpc 3.2.0 supports true | false as gen_rpc.ipv6_only
2023-10-09 20:03:21 +02:00
Zaiming (Stone) Shi 497e08448d feat(cluster): support ipv6 and tls on ipv6 for clustering
Made possible to configure inet6_tls for Erlang distribution
Also, added support to configure ipv6 listener for gen_rpc
2023-10-09 20:03:21 +02:00
Zaiming (Stone) Shi 03d8e06ff7 chore(emqx_rule_funcs): regroup export functions 2023-10-09 20:03:21 +02:00
Thales Macedo Garitezi d6781efee2 fix(resource): change how buffer workers are started 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 902b1d6ec5 fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi eebfb44f72 fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
Zaiming (Stone) Shi c60915293a
Merge pull request #11634 from savonarola/0919-auth-refactor
Auth refactor
2023-10-09 19:51:26 +02:00
lafirest 8f4cdc3fcf
Merge pull request #11732 from lafirest/test/sso_cli
test(ldap): add test suite for SSO CLI
2023-10-09 20:35:23 +08:00
Zaiming (Stone) Shi 9ab49a7ae3 Merge remote-tracking branch 'origin/master' into 0919-auth-refactor 2023-10-09 14:18:10 +02:00
lafirest 2c7e5eb1cb
Merge pull request #11728 from lafirest/fix/ldap_filter
fix(ldap): improve the filter lex && parse
2023-10-09 18:24:57 +08:00
firest e3550fc07b test(ldap): add test suite for SSO CLI 2023-10-09 16:05:56 +08:00
zhongwencool 0c89b6b213 docs: swagger api tags should camel case 2023-10-09 15:21:51 +08:00
firest b2a6724dc2 fix(ldap): improve the filter lex && parse
1. auto escape special chars in the filter
2. fix a bug that the value can't be `dn`
2023-10-08 18:39:20 +08:00
zhongwencool 482e82f914 fix: cli's args in audit log should be array 2023-10-07 17:38:02 +08:00
Thales Macedo Garitezi 34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
Ilya Averyanov 398a62031e chore: update app versions 2023-10-05 13:41:50 +03:00
Ilya Averyanov 5dff36474d chore(auth): get rid of hardcoded schema modules in auth 2023-10-05 13:41:50 +03:00
Ilya Averyanov c2c56ba481 chore(auth): update tests 2023-10-05 13:41:50 +03:00
Ilya Averyanov 1eb75b43c4 chore(auth): split emqx_authn and emqx_authz apps 2023-10-05 13:41:50 +03:00
Zaiming (Stone) Shi fd75dc895d fix(authn): "authentication" importance should not be "hidden" 2023-10-03 11:18:21 +02:00
Zaiming (Stone) Shi ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
0928 sync release 53
2023-10-02 21:57:32 +02:00
Zaiming (Stone) Shi 3f6e0e890b chore: bump emqx_durable_storage and emqx_license app vsn 2023-10-02 20:13:09 +02:00
Thales Macedo Garitezi 2358d67908 refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
Zaiming (Stone) Shi bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Zaiming (Stone) Shi ed5b456d62 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 18:24:44 +02:00
Zaiming (Stone) Shi a852400fb3 chore: add 5.3.bpapi 2023-09-29 16:34:04 +02:00
Zaiming (Stone) Shi fe01aaff4f chore: prepare for e5.3.0 promotion 2023-09-29 13:32:57 +02:00
Zaiming (Stone) Shi ce5bd0a3ce Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 11:36:32 +02:00
Zaiming (Stone) Shi c64e599e81 docs: document how to retrieve peercert 2023-09-29 10:33:57 +02:00
Zaiming (Stone) Shi dc147fd310 fix(rule-engine): console action has no args field 2023-09-29 10:33:33 +02:00
Zaiming (Stone) Shi 1177a32310 chore: bump version to 5.3.0-rc.2 2023-09-29 09:49:27 +02:00
Zaiming (Stone) Shi 6891234390 chore: return simplified error reason for less logging 2023-09-29 09:48:15 +02:00
Zaiming (Stone) Shi c2d750aa09 fix(resource): redact query args in exception log 2023-09-29 09:20:42 +02:00
Zaiming (Stone) Shi 02ef854f0f fix(ldap): no crash when no query result is empty list 2023-09-29 08:54:41 +02:00
Zaiming (Stone) Shi b59a7ff2dd
Merge pull request #11706 from zmstone/0928-fix-ldap-sso-logging-level
0928 fix ldap sso logging level and reject muti-match results
2023-09-29 02:40:37 +02:00
Zaiming (Stone) Shi 4a4730ad46 fix(ldap): handle invalidCredentials in ldap authn 2023-09-29 00:51:05 +02:00
Zaiming (Stone) Shi 9ee2cb9c79 fix(ldap): return unrecoverable_error if more than on match found 2023-09-28 23:58:34 +02:00
Zaiming (Stone) Shi cc5dab1dc7 chore: fix code style 2023-09-28 21:29:59 +02:00
Zaiming (Stone) Shi b267fc2588 chore: bump release version to 5.3.0 2023-09-28 21:22:33 +02:00
Zaiming (Stone) Shi 922d5a9a83 fix(ldap): do not allow multi-matches to proceed
if ldap query returns more than on match
we should reject the auth request instead of picking
the first one
2023-09-28 21:20:50 +02:00
Zaiming (Stone) Shi d858f8af39 test: fix openldap docker runs 2023-09-28 18:40:03 +02:00
Zaiming (Stone) Shi b28e781c50 fix(ldap-sso): do not log error level when invalid user credentials 2023-09-28 18:37:11 +02:00
Zaiming (Stone) Shi 36f3052be1
Merge pull request #11698 from zmstone/0928-disable-audit-log-by-default
fix(audit): disable audit log by default
2023-09-28 17:28:23 +02:00
Zaiming (Stone) Shi 39820be5ff
Merge pull request #11703 from lafirest/fix/ldap_bind
fix(ldap): use the search result as bind target
2023-09-28 14:48:53 +02:00
Zaiming (Stone) Shi 1a13b2ac56 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-28 13:53:03 +02:00
firest 43ea367df8 fix(ldap): use the search result as bind target 2023-09-28 18:49:00 +08:00
Zaiming (Stone) Shi 98409c9b1e
Merge pull request #11702 from zmstone/0928-minor-chore
0928 minor chore
2023-09-28 12:39:55 +02:00
Zaiming (Stone) Shi 26cadb2248 fix(audit): disable audit log by default 2023-09-28 12:36:14 +02:00
Zaiming (Stone) Shi b14425d865 fix(sso/saml): ensure lower case http header name 2023-09-28 11:40:21 +02:00
Zaiming (Stone) Shi 7ddcd71412 docs(ssl_dist.conf): update doc link 2023-09-28 11:16:18 +02:00
firest dca8fdb17f fix(resource): respect the start_timeout 2023-09-28 16:36:41 +08:00
ieQu1 b77e5e880a
Merge pull request #11697 from ieQu1/dev/gen-rpc-3.1.1
chore(gen_rpc): Bump version to 3.1.1
2023-09-28 10:03:36 +02:00
JianBo He 6d2adfc259 chore(audit): emit the message field 2023-09-28 11:04:42 +08:00
JianBo He 859b122cdd chore: format codes 2023-09-28 10:25:37 +08:00
JianBo He b52e4ac99d chore: update apps/emqx_machine/src/emqx_restricted_shell.erl
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-28 10:25:37 +08:00
JianBo He bdf24d0ec8 chore: remove the `unauthorized` type for `from` field 2023-09-28 10:25:37 +08:00
JianBo He 71acf121ba chore(audit): distinguish requests from rest_api or dashboard 2023-09-28 10:25:37 +08:00
JianBo He dd9938114c chore(audit): add from field 2023-09-28 10:25:37 +08:00
JianBo He ece7d5b52a
Merge pull request #11690 from lafirest/fix/ldap_parse_cfg
fix(ldap): improve the LDAP `parse_config` function
2023-09-28 10:24:52 +08:00
lafirest b0d86eecd6
Merge pull request #11691 from lafirest/fix/sso_ssl
fix(sso): support for SSL update && ensure update is atomic
2023-09-28 10:22:34 +08:00
firest 57781d0544 fix(ldap): remove the parse_config, it never work 2023-09-28 09:38:16 +08:00
firest 9dee2dc31e fix(sso): clear last error first before update && fix the `running` 2023-09-28 08:56:16 +08:00
ieQu1 0aa3ccdd65 feat(gen_rpc): Add schema for the TLS versions and cipher suites 2023-09-28 00:36:59 +02:00
Zaiming (Stone) Shi afdda107af fix(logger): json format log encode binary list as string array 2023-09-27 23:40:01 +02:00
Zaiming (Stone) Shi 6f7a4344dc fix: do not gc sso saml SP singing keys 2023-09-27 23:19:39 +02:00
Zaiming (Stone) Shi 34367fc4ec fix(audit_log): pretty print shell args 2023-09-27 23:19:39 +02:00
Zaiming (Stone) Shi bb49914fd6 fix(sso): add convet_certs callback for sso backends
must convert certs in pre_config_update so the cert path refernces
are stored in raw config, otherwise the files might get gc:ed
2023-09-27 22:41:39 +02:00
Zaiming (Stone) Shi 45caa3bf01 fix(sso): make sp_private_key sensitive
so it will not be logged
2023-09-27 21:27:59 +02:00
Zaiming (Stone) Shi c8cbbff044 fix(logger): no need for special handling of empty string
when formating json logs, there is no need to handle empty strings
special, already covered by unicode handling
2023-09-27 21:26:47 +02:00
ieQu1 3c37f19105 chore(gen_rpc): Bump version to 3.1.1 2023-09-27 21:09:00 +02:00
firest 66d2107007 fix(sso): refactor update logic 2023-09-28 00:09:09 +08:00
Zaiming (Stone) Shi bd3277c51b
Merge pull request #11683 from zmstone/0925-test-refactor-emqx-rpc-test
test: refactor emqx_rpc unit tests
2023-09-27 15:46:27 +02:00
firest 08ad09a68f fix(sso): refactor backen update logic
1. valid config always can update successfully
2. the `running` endpoint only return successfully created backend
3. enhancement of the `/sso` endpoint, and will check is the resource online
2023-09-27 20:53:10 +08:00
Zaiming (Stone) Shi ea8d54fd8b test: ensure atom exists in test module 2023-09-27 12:58:06 +02:00
Zaiming (Stone) Shi 6e8c73258f test: refactor emqx_rpc unit tests 2023-09-27 12:58:06 +02:00
JimMoen c9194cd6b2
fix(saml_sso): donot load IDP metadata when disabling saml 2023-09-27 18:46:24 +08:00
JimMoen af9e87c025
fix: saml callback should check saml state 2023-09-27 18:34:48 +08:00
firest b2699c687b fix(sso): support for SSL update && ensure update is atomic
1. support update SSL key and cert files
2. increase connection timeout
3. ensure the update is atomicity, everything will be consistent
2023-09-27 15:48:11 +08:00
Andrew Mayorov 1d0e789e4d
Merge pull request #11679 from keynslug/ft/EMQX-10942/custom-session-timers
feat(session): add custom session timers mechanism
2023-09-26 19:09:53 +03:00
firest 0c33df3912 fix(ldap): improve the LDAP `parse_config` function 2023-09-26 23:28:37 +08:00
JimMoen 3fa18d6935
fix: lower sso saml redirect http header 2023-09-26 23:25:59 +08:00
JianBo He 6a9bb7c3ae
Merge pull request #11681 from zhongwencool/audit-log-fix-2
fix: don't need to change audit log's level
2023-09-26 23:12:28 +08:00
lafirest ac5eb5bc29
Merge pull request #11687 from lafirest/fix/sso_timeout
fix(sso): Handle backend update timeout and fix create errors
2023-09-26 22:43:38 +08:00
JianBo He 1a96a5990b
chore: Update apps/emqx_enterprise/src/emqx_enterprise_schema.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 21:44:40 +08:00
firest 403714d44e fix(sso): Handle backend update timeout and fix create errors
1. correctly handle the timeout when call update on a backend
2. fix that config update always returns success
3. do not ignore start failures and ensure start is atomic
2023-09-26 21:43:30 +08:00
Thales Macedo Garitezi aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
Andrew Mayorov bce2142df8
test(session): add custom session timers testcase 2023-09-26 16:22:06 +03:00
Zaiming (Stone) Shi 4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
Thales Macedo Garitezi fdcd73e20c fix(file_logger): change file logger type depending on rotation size
Fixes https://emqx.atlassian.net/browse/EMQX-11036

From `logger_disk_log_h:open_disk_log`:

```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
    case filelib:ensure_dir(File) of
        ok ->
            Size =
                if Type==halt -> MaxNoBytes;
                   Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
                end,
            Opts = [{name,   Name},
                    {file,   File},
                    {size,   Size},
                    {type,   Type},
                    {linkto, self()},
                    {repair, false},
                    {format, external},
                    {notify, true},
                    {quiet,  true},
                    {mode,   read_write}],
            case disk_log:open(Opts) of
```

Affects all file loggers (audit included):

```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
                                                                             {badarg,
                                                                              [{size,
                                                                                {infinity,
                                                                                 10}},
                                                                               {type,
                                                                                wrap},
                                                                               {linkto,
                                                                                <0.1952.0>},
                                                                               {repair,
                                                                                false},
                                                                               {format,
                                                                                external},
                                                                               {notify,
                                                                                true},
                                                                               {quiet,
                                                                                true},
                                                                               {mode,
                                                                                read_write}]}}
```
2023-09-26 09:04:44 -03:00
Ilya Averyanov 14983ec14a chore(hooks): validate hookpoints and document hook callbacks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
JianBo He 5e6397b843 chore: remove result field for CLI audit logs 2023-09-26 19:24:35 +08:00
JianBo He a73c3b8e1e Merge remote-tracking branch 'ce/release-53' into audit-log-fix-2 2023-09-26 19:05:47 +08:00
JianBo He e9785a6863
Merge pull request #11680 from thalesmg/audit-not-on-ce-r53-20230925
fix(audit): only support audit log on enterprise edition
2023-09-26 18:57:22 +08:00
Zaiming (Stone) Shi 7d810c2107 test: fix test case match pattern 2023-09-26 11:40:30 +02:00
Zaiming (Stone) Shi d31bfc70fb
Merge pull request #11659 from zhongwencool/fix-listener-ssl-create-500
fix: create ssl listener return 500 crash
2023-09-26 11:38:07 +02:00
lafirest 8cc626d33f
Merge pull request #11686 from lafirest/fix/sso_user_backend
fix: fix update SSO user && improve SSO user deletion commnad
2023-09-26 16:04:51 +08:00
firest 56917fee5f fix(sso): Improved SSO user deletion command 2023-09-26 14:15:41 +08:00
firest d1af81c86e fix(sso): fix the failure to update SSO users 2023-09-26 12:14:45 +08:00
firest 8135e27586 fix: redact sensitive data in SSO and LDAP 2023-09-26 11:06:47 +08:00
Thales Macedo Garitezi 17206f8c75
Merge pull request #11662 from thalesmg/port-scan-check-proto-dist-m-20230922
chore: check ekka proto dist module type when resolving node address
2023-09-25 17:01:43 -03:00
Thales Macedo Garitezi 5d212e1086 fix(audit): only support audit log on enterprise edition
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
Thales Macedo Garitezi 0498e59c45 test: fix flaky test 2023-09-25 13:35:17 -03:00
zhongwencool 95060302fd fix: don't need to change audit log's level 2023-09-26 00:28:47 +08:00
Thales Macedo Garitezi ff7f37ccf5 test(cth): allow defining schema to load for app 2023-09-25 13:22:41 -03:00
Andrew Mayorov b1f144ab8b
feat(session): add custom session timers mechanism
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
lafirest 1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
Zaiming (Stone) Shi 5e6996dc05 refactor: log formatter format mfa+line as m:f/a(line)
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 6a557980e4 refactor(logger): ensure JSON log field order
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 5f45ba50ff refactor: delete log formatter config for audit log handler
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 1fed38c248 fix(logger): write 'json' format logs as JSON 2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi aea1e80290 feat: add 'format' as alias for log 'formatter' 2023-09-25 16:55:22 +02:00
William Yang 9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify 2023-09-25 15:48:01 +02:00
Thales Macedo Garitezi 806017ef90 chore: check ekka proto dist module type when resolving node address
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917

Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
firest cfdb25b213 fix(sso): updates the SSO backend when the `[dashboard]` has updated 2023-09-25 20:30:47 +08:00
Ivan Dyachkov f1bc3b68b6
Merge pull request #11674 from id/0925-e5.3.0-alpha.2
e5.3.0 alpha.2
2023-09-25 14:21:46 +02:00
JianBo He 6f29bbf997
Merge pull request #11673 from HJianBo/redact_audit_log_from_cli
Redact audit log from CLI
2023-09-25 20:03:45 +08:00
Andrew Mayorov 81cf619f07
fix(ftconf): also mark `secret_access_key` key as sensitive 2023-09-25 14:52:57 +03:00
William Yang 36d3a3a524 fix: bump to quicer 0.0.201
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
William Yang 22193d273a fix(quic): ignore undefined cacertfile 2023-09-25 12:57:55 +02:00
Ivan Dyachkov 8c93c79b44 chore: e5.3.0-alpha.2 2023-09-25 12:52:45 +02:00
JimMoen 7d58f6c61e
Merge pull request #11668 from JimMoen/saml-login-redirect
fix: saml login acs redirect to dashboard overview
2023-09-25 18:11:03 +08:00
JianBo He b970a34ee0 chore: redact some audit logs from CLI 2023-09-25 18:06:39 +08:00
JianBo He 89e15e9134
Merge pull request #11665 from HJianBo/typo-fixes
Typo fixes
2023-09-25 17:23:40 +08:00
lafirest 1cd9df6461
Merge pull request #11669 from lafirest/fix/dashboard_cli
fix(sso): refactor the `admins` CLI
2023-09-25 16:57:18 +08:00
firest e63d484632 fix(sso): move the config path of the SSO feature to `dashboard.sso` 2023-09-25 15:23:52 +08:00
Ivan Dyachkov e1f1c64ecd Merge branch 'release-53' into 0925-sync-r53-to-master 2023-09-25 08:12:45 +02:00
firest e94192d1fa fix(sso): refactor the `admins` CLI
1. revert the opensource version
2. allow delete the SSO user via CLI
2023-09-25 13:53:01 +08:00
firest 4f4868a46c fix(sso): Disable access to `logout` endpoint by the `API key` 2023-09-25 10:31:38 +08:00
JianBo He 30862a94c6 chore: typo fixes 2023-09-24 20:19:05 +08:00
JimMoen ad4fadc2fa
fix: saml login acs redirect to dashboard overview 2023-09-23 17:29:02 +08:00
JianBo He f8d06614c0 chore: fix dialyzer warnings 2023-09-23 07:34:04 +08:00
JimMoen 1dddccb448
fix(saml): cert files cleanup when destroy 2023-09-22 22:49:08 +08:00
JimMoen 80a6c1150d
fix(saml): saml login reply role `viewer` as default 2023-09-22 22:39:02 +08:00
JimMoen cc3e4e4dc5
fix(saml): drop cert and key content and return path 2023-09-22 22:37:04 +08:00
JimMoen 6349cd3910
fix(saml): sp sign request 2023-09-22 21:57:50 +08:00
JimMoen 2a8f3f9eaa
fix: saml xml metedata format 2023-09-22 21:50:43 +08:00
JimMoen a318ad486a
refactor: behavior login/2 use all http request 2023-09-22 21:50:43 +08:00
JianBo He 9181ec844f
chore: split out sso_saml_api module 2023-09-22 21:50:43 +08:00
JianBo He df94426ee3
chore: make static_check happy 2023-09-22 21:50:42 +08:00
JianBo He ec0894ca0b
chore: update esaml vsn 2023-09-22 21:50:42 +08:00
JianBo He 4a26f63bd6
chore: fix bugs 2023-09-22 21:50:41 +08:00
JianBo He 47badc3181
chore: make dialyzer happy 2023-09-22 21:49:12 +08:00
JianBo He 1c78c6bf6d
chore: fix 500 crashes when backend not existed 2023-09-22 21:49:12 +08:00
JimMoen b4fb5196cb
fix(sso): SSO management API 500 2023-09-22 21:49:12 +08:00
JimMoen bba5cc44a8
fix: keep same API path style 2023-09-22 21:49:12 +08:00
JimMoen 8300cd42d4
fix: acl url ignore auth check 2023-09-22 21:49:12 +08:00
JimMoen 44836ef5ee
chore: bump esaml vsn to v1.1.1 2023-09-22 21:49:12 +08:00
JimMoen 13666fa9f9
refactor: avoid dynamic call 2023-09-22 21:49:12 +08:00
JimMoen c9e0d4fc30
feat: saml integration for dashboard sso 2023-09-22 21:49:10 +08:00
JimMoen d9466eef63
chore: fix Dashboard RBAC license and rebar.config 2023-09-22 21:19:45 +08:00
Zaiming (Stone) Shi fd932c9bf7
Merge pull request #11660 from zmstone/0922-fix-audit-eval-command
fix(audit): make emqx eval command auditable
2023-09-22 14:02:47 +02:00
Zaiming (Stone) Shi a34ab19d93 fix(audit): make emqx eval command auditable 2023-09-22 12:05:45 +02:00
zhongwencool 0eed01abee fix: create ssl listener return 500 crash 2023-09-22 17:34:19 +08:00
lafirest 13b5e4dbc9
Merge pull request #11658 from lafirest/fix/sso_misc
Fix/sso misc
2023-09-22 14:44:39 +08:00
firest 7286d15ca6 chore(sso): adjust the schema of the SSO LDAP backend 2023-09-22 13:48:20 +08:00
firest 9e55ae240a feat(sso): add `role` into the result of login endpoints 2023-09-22 13:48:07 +08:00
firest 681e57dee6 fix(RBAC): allow read-only users to logout 2023-09-22 11:06:24 +08:00
Thales Macedo Garitezi d6935b6a67 feat: add port scan diagnostics to mria waiting for tables checks
Fixes https://emqx.atlassian.net/browse/EMQX-10944

Also updates ekka -> 0.15.15, mria -> 0.6.4

How to test
===========

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.

Or, more easily:

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.

Example output
==============

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status =>
                                                                 bad_ports},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```

After one node is back:

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status => ok},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```
2023-09-21 14:29:01 -03:00
Ivan Dyachkov 7cf60c5a91 chore: e5.3.0-alpha.1 2023-09-21 19:29:00 +02:00
Andrew Mayorov acf4227fc6
test(session): fix quic testgroup in persistent session suite
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
Thales Macedo Garitezi 76f614e9c5
Merge pull request #11653 from thalesmg/fix-dash-rbac-dialyzer-error-m-20230921
chore: fix dialyzer errors on ce version
2023-09-21 13:06:09 -03:00
Thales Macedo Garitezi 85a8c174d9 chore: fix dialyzer errors on ce version
```
apps/emqx_dashboard/src/emqx_dashboard.erl
Line 225 Column 17: The pattern {'error', 'unauthorized_role'} can never match the type {'error','not_found' | 'token_timeout'} | {'ok',binary()}
```
2023-09-21 11:07:31 -03:00
Andrew Mayorov b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov 0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
sync r52 to master
2023-09-21 12:31:24 +02:00
William Yang 3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
quicer 0.0.200
2023-09-21 12:20:14 +02:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
Ivan Dyachkov 105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
Andrew Mayorov a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently 2023-09-21 11:53:40 +04:00
firest 9a128517d8 fix(sso): use the correct way to mark the API as authorization-free 2023-09-21 15:52:05 +08:00
zhongwencool dfc14a39bc test: add emqx_management to telemetry SUITE 2023-09-21 14:15:06 +08:00
firest 2283db62ec fix(sso): add a API with authorization-free to list running backends 2023-09-21 11:52:30 +08:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool 9521e11d39
Merge pull request #11584 from zhongwencool/telemetry-error-on-windows
fix: telemetry report error on windows
2023-09-21 09:08:55 +08:00
JianBo He b924fb618a
Merge pull request #11639 from lafirest/test/sso
test(sso): add test case for the integration of SSO with LDAP
2023-09-21 09:01:13 +08:00
zhongwencool 1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify 2023-09-21 07:32:39 +08:00
Andrew Mayorov a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
Andrew Mayorov 3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy 2023-09-20 22:55:54 +04:00
Andrew Mayorov 21e82b9534
test(sessmem): make retry delivery testcase more involved 2023-09-20 22:55:54 +04:00
Andrew Mayorov 69889d14a3
fix(sessds): fix use of undefined types 2023-09-20 22:55:54 +04:00
Andrew Mayorov c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
Andrew Mayorov 9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
Andrew Mayorov 98706cd215
chore: ensure comments follow code style consistently 2023-09-20 22:55:25 +04:00
Andrew Mayorov 8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability 2023-09-20 22:55:25 +04:00
Andrew Mayorov 7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov 045d8b7f10
refactor(ds): reorder functions to improve readability 2023-09-20 22:55:24 +04:00
Andrew Mayorov 540ca6d60f
chore: drop few commented out and irrelevant pieces 2023-09-20 22:55:24 +04:00
Andrew Mayorov 9d145890cc
refactor(sessmem): pass log context as part of session event
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov e422f492ef
test(sessds): reuse and expand persistent session test suite 2023-09-20 22:55:24 +04:00
Andrew Mayorov 3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling 2023-09-20 22:55:23 +04:00
Andrew Mayorov 7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Andrew Mayorov e713fc38aa
feat(broker): reflect persisted messages in publish result
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
Andrew Mayorov f4953e719b
fix(cmproto): fix few typespecs 2023-09-20 22:53:45 +04:00
Andrew Mayorov 8af107e28d
test(ds): simplify cluster test setups 2023-09-20 22:53:45 +04:00
Andrew Mayorov 7c4f68dd3d
fix(session): make utility function names consistent
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
Andrew Mayorov 2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem` 2023-09-20 22:53:44 +04:00
Andrew Mayorov abeff0bc4f
chore(session): try to describe what happens after session takeover 2023-09-20 22:53:44 +04:00
Andrew Mayorov 45d44df11d
refactor(session): update eviction channel session logic
The changes partially reflect `emqx_channel` changes with respect to
in-memory session specific logic. The difference is that eviction
channel does not replay post-takeover, instead enqueues messages.
2023-09-20 22:53:44 +04:00
Andrew Mayorov ab1c4c4222
refactor(session): drop `lookup/1` session API
Due to the fact it's not used for anything right now.
2023-09-20 22:53:44 +04:00
Andrew Mayorov 97881ff3ca
refactor(session): bring back common session timers 2023-09-20 22:53:43 +04:00
Andrew Mayorov bf16417513
feat(session): introduce session implementation concept 2023-09-20 22:53:42 +04:00
Andrew Mayorov 780ca15298
chore: bump application versions 2023-09-20 22:52:14 +04:00
Andrew Mayorov 04731b7ef7
test(takeover): randomize messages for random natural ordering 2023-09-20 22:52:14 +04:00
Andrew Mayorov f022c9b1a4
feat(emqx): add `emqx_inflight:fold/3` generic function 2023-09-20 22:52:14 +04:00
Andrew Mayorov 596ce157fd
refactor(exproto): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov 57ae5b14f1
refactor(mqttsn): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov e4866adc2f
refactor(chan): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov cfb1bf1fa4
chore(emqx): drop remnants of former session persistence impl
1. It is not functional anyway.
2. It blocks `emqx_session` refactoring in a few places.
2023-09-20 22:52:13 +04:00
William Yang e43b87f18e test(quic): update emqx_broker_SUITE 2023-09-20 20:20:18 +02:00
Thales Macedo Garitezi 2965fa6fcb
Merge pull request #11613 from thalesmg/ds-keyspace-m-20230914
feat(ds): introduce keyspace concept
2023-09-20 14:21:27 -03:00
William Yang da626f6d29 feat(quic): bump to quicer 0.0.200 and emqtt 1.9.0 2023-09-20 16:43:15 +02:00
zhongwencool dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool 9e75ff88af
chore: apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:15:39 +08:00
zhongwencool f789a7504f
chore: update apps/emqx/src/emqx_quic_connection.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:00:44 +08:00
Ivan Dyachkov 09aea47f40 chore: 5.2.1 2023-09-20 14:04:28 +02:00
firest 21c37a3f14 fix(sso): adjust the schema of LDAP API && add more logs 2023-09-20 18:15:42 +08:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
zhongwencool c26a18e949 fix: always return ok when remove local resource 2023-09-20 18:02:42 +08:00
firest b11c5d26ce test(sso): add test case for the integration of SSO with LDAP 2023-09-20 16:25:19 +08:00
lafirest bc6edac45f
Merge pull request #11631 from lafirest/feat/dashboard_ldap
feat(dashboard): add SSO feature and integrate with LDAP
2023-09-20 11:40:42 +08:00
firest 61311081d8 fix(sso): fix RBAC test errors && spellcheck 2023-09-20 10:42:17 +08:00
Thales Macedo Garitezi 5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
firest 5b00998050 fix(sso): fix invalid_dynamic_call 2023-09-19 22:47:48 +08:00
firest 285e529766 fix(sso): fix sso errors found when manual test 2023-09-19 19:15:52 +08:00
Zaiming (Stone) Shi 65818740f3
Merge pull request #11622 from zmstone/0917-upgrade-gen_rpc-to-3.1.0
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-19 10:22:53 +02:00
Ivan Dyachkov a69c26b86d chore: e5.2.1-alpha.3 2023-09-19 08:17:29 +02:00
firest 2cddce5479 feat(dashboard): add SSO feature and integrate with LDAP 2023-09-19 14:10:29 +08:00
JianBo He de1bc04843 chore: bump ekka to 0.15.14, mria to 0.6.3
Inclued ekka's updates:
- https://github.com/emqx/ekka/pull/214
- https://github.com/emqx/ekka/pull/215
- https://github.com/emqx/ekka/pull/216
- https://github.com/emqx/ekka/pull/217

mria's updates:
- https://github.com/emqx/mria/pull/160
- https://github.com/emqx/mria/pull/161
- https://github.com/emqx/mria/pull/162
2023-09-19 09:43:19 +08:00
Thales Macedo Garitezi a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Zaiming (Stone) Shi 9f8a7a96af
Merge pull request #11628 from zmstone/0918-upgrade-emqtt-to-1.8.7
0918 upgrade emqtt to 1.8.7
2023-09-18 21:56:04 +02:00
Zaiming (Stone) Shi de93d71014 test: fix emqx_sahred_sub_SUITE:t_remote
prior to this fix, the peer ct node is actually node itself
2023-09-18 21:54:43 +02:00
Zaiming (Stone) Shi 99b60849b4 test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer 2023-09-18 20:44:38 +02:00
SergeTupchiy 0b0e322473
Merge pull request #11627 from SergeTupchiy/EMQX-10962-fix_hstreamdb-bridge-on-stop
fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
2023-09-18 19:57:07 +03:00
Zaiming (Stone) Shi d1e5b69eac test: increase wait time 2023-09-18 17:33:09 +02:00
Zaiming (Stone) Shi 4f1ec795eb test: delete special handling of emqtt race condition
the race should have been fixed in emqtt 1.8.7
2023-09-18 17:27:34 +02:00
Zaiming (Stone) Shi 46557c3462 fix(emqtt): upgrade from 1.8.6 to 1.8.7 2023-09-18 17:04:23 +02:00
Serge Tupchii 80cf60e9be fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
fixes: EMQX-10962
2023-09-18 17:48:05 +03:00
Zaiming (Stone) Shi 45f0d0dfce
Merge pull request #11623 from zmstone/0917-upgrade-esockd-to-5.9.7
0917 upgrade esockd to 5.9.7
2023-09-18 16:30:54 +02:00
zhongwencool c868264f65
Merge pull request #11617 from zhongwencool/rule-engine-console-args
fix: keep builtin_console actions not complain args
2023-09-18 20:40:02 +08:00
zhongwencool 8b6d5d4ca9 test: add test for build_console's args 2023-09-18 16:48:38 +08:00
zhongwencool 5490807b20
chore: update apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-18 16:11:39 +08:00
Zaiming (Stone) Shi f66d9e76fe fix(emqx_channel): aovid logging frame errors as exceptions 2023-09-18 09:04:00 +02:00
lafirest 6fe846bf0e
Merge pull request #11610 from lafirest/feat/rbac
feat(dashboard): add RBAC feature for Dashboard
2023-09-18 14:41:47 +08:00
firest ca6547c05b fix(rbac): change default role and update changes 2023-09-18 11:40:35 +08:00
Zaiming (Stone) Shi abbee4bc47 fix(esockd): upgrade esockd from 5.9.6 to 5.9.7 2023-09-17 19:44:04 +02:00
Zaiming (Stone) Shi 985f0e6994 chore(gen_rpc): upgrade from 2.8.1 to 3.1.0 2023-09-17 19:10:13 +02:00
Zaiming (Stone) Shi a9682c2989
Merge pull request #11605 from zmstone/0913-lower-severity-level-for-CMD-override-warnings
0913 lower severity level for cmd override warnings
2023-09-16 11:32:07 +02:00
Zaiming (Stone) Shi ac41f497b4 feat: add bytesize rule sql function 2023-09-15 16:04:21 +02:00
Thales Macedo Garitezi 2aae3856a3
Merge pull request #11596 from thalesmg/fix-typespec-typos-m-20230912
fix: fix typespec typos
2023-09-15 09:24:42 -03:00
Ilya Averyanov f0d2b2e6c3 chore: update emqx_eviction_agent version 2023-09-15 13:24:38 +03:00
firest 4b97d3f57d test(rbac): add test cases for RBAC && update changes 2023-09-15 18:16:28 +08:00
Ilya Averyanov 1959e15703 chore: simplify session eviction for node rebalance 2023-09-15 11:46:51 +02:00
zhongwencool 32d838c85b fix: format console action crash 2023-09-15 17:40:56 +08:00
zhongwencool 640dd130fa fix: don't urldecode twice clientid 2023-09-15 17:36:48 +08:00
zhongwencool d72348c8ed fix: keep builtin_console actions not complain args 2023-09-15 16:38:04 +08:00
firest 45ccc66474 fix(dashbaord): fix test cases to support RBAC 2023-09-15 15:49:37 +08:00
lafirest aace9215a4
Merge pull request #11608 from lafirest/feat/ldap_bind
feat(ldap): integrate authentication with LDAP bind operation
2023-09-15 14:06:39 +08:00
firest 0846939760 fix(ldap): remove unused code and mark sensitive field 2023-09-15 10:13:07 +08:00
Thales Macedo Garitezi b30bcf32bd feat(ds): introduce keyspace concept
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
Ilya Averyanov 11c39c4b6a chore: simplify session eviction for node rebalance 2023-09-14 18:28:25 +03:00
Thales Macedo Garitezi 6be0e14e8c fix: fix typespec typos 2023-09-14 10:21:05 -03:00
Thales Macedo Garitezi e41f7dd68c
Merge pull request #11598 from thalesmg/ds-fix-ps-router-m-20230912
fixes and improvements to persistent session router
2023-09-14 10:00:58 -03:00
firest 5c31c5ce76 feat(dashboard): add RBAC feature for Dashboard 2023-09-14 19:28:37 +08:00
firest afbf13b8a2 feat(ldap): integrate authentication with LDAP bind operation 2023-09-14 14:10:43 +08:00
firest 3eb46be2ae fix(banned): fix typos 2023-09-14 14:01:59 +08:00
Zaiming (Stone) Shi 95f1312df7 chore(emqx_ctl): lower command override log from warning to info
prior to this change this warning message is always logged if a node
joins cluster.

the intention of this warning is to detect potential command name
clash (provided by different plugins), so arguably it should be
at debug level
2023-09-13 20:52:08 +02:00
Thales Macedo Garitezi 803b69d878
refactor: use more consistent return types
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2023-09-13 10:49:45 -03:00