Commit Graph

2558 Commits

Author SHA1 Message Date
Shawn 9aa63358e6 fix(resource): type spec for resource_data() 2021-11-23 10:04:31 +08:00
Shawn 29ad6d215e feat(resource): add metrics to emqx_resource 2021-11-23 10:04:31 +08:00
Shawn 4c149f92c1 feat(bridge): support metrics for bridges 2021-11-23 09:36:10 +08:00
JianBo He 50542ec441
Merge pull request #6248 from HJianBo/gw-improve-apis-1
Add authentication data management APIs for gateway
2021-11-23 09:12:04 +08:00
zhouzb 5aa8ba45d3 fix(authn): add the case when the instance does not exist 2021-11-22 23:30:05 +01:00
zhouzb 2b282b9666 fix(ssl): Return the content of certificates in HTTP response 2021-11-22 23:30:05 +01:00
zhouzb 81d425cb27 fix(ssl): fix the issue that the content of TLS certificate is not converted to path 2021-11-22 23:30:05 +01:00
Spycsh c89a663b53 chore: refactor slog under emqx, conf and exhook 2021-11-22 23:12:17 +01:00
Zaiming (Stone) Shi c202688abf
Merge pull request #6258 from tigercl/chore/dashboard
chore(dashboard): disable by default
2021-11-22 22:34:26 +01:00
Ilya Averyanov 071c2c99e8 refactor(authn resources): add `emqx_resource` and `emqx_authn` tests 2021-11-22 21:08:04 +03:00
Zaiming (Stone) Shi b59ad6b186
Merge pull request #6252 from JimMoen/fix-clients-cluster-query
fix(mgmt): fix broken `ets:continuation` been sent between nodes
2021-11-22 15:28:16 +01:00
zhouzb 63f1d88ed3 chore(dashboard): disable by default 2021-11-22 22:23:06 +08:00
Shawn a0e81226d7 fix(elvis): improve some code format 2021-11-22 19:10:00 +08:00
Shawn 37aeb28a79 fix(connector): test case conflict 2021-11-22 18:52:43 +08:00
JianBo He 1e2eac0fce test(gw): add tests for authm data-mgmt 2021-11-22 17:38:59 +08:00
JianBo He 4eb27ce25b fix(gw): fix bad paging params 2021-11-22 17:38:59 +08:00
JianBo He 7d9930b64e feat(gw): support data-management apis gw's authn 2021-11-22 17:38:59 +08:00
JianBo He 4f752fb5ab chore(gw): fix elvis warnings 2021-11-22 17:38:59 +08:00
JianBo He f033fad7b3 refactor(gw): deps on emqx_dasboard_swagger 2021-11-22 17:38:59 +08:00
Shawn f0c61068e8 fix(authn): test cases issues 2021-11-22 17:35:33 +08:00
Shawn ea3c809287 fix(emqx_conf): merge issues 2021-11-22 17:35:33 +08:00
Shawn 07e3e43a21 fix(authn): xref issues 2021-11-22 17:35:33 +08:00
Shawn 9c93ea0338 feat(connector): add API for /connectors_test 2021-11-22 17:35:33 +08:00
Shawn c66c6b53b2 feat(connector): add APIs for connector 2021-11-22 17:35:33 +08:00
Shawn bcf5f499da feat(connector): add APIs for connector 2021-11-22 17:35:33 +08:00
Shawn 4dac90f4a7 refactor(bridges): add POST /bridges for creating 2021-11-22 17:35:32 +08:00
Shawn c4668b4047 fix(bridge): load emqx_conf in testcases 2021-11-22 17:35:32 +08:00
Shawn 339749764d fix(bridge): update the testcases for HTTP bridges 2021-11-22 17:35:32 +08:00
Shawn 7d64013edd refactor(connector): add connector configs 2021-11-22 17:35:32 +08:00
Shawn cf3f2c3057 fix(dialyzer): remote the extra str/1 2021-11-22 17:35:32 +08:00
Shawn e41736f5cb fix(bridge): update testcases for bridge APIs 2021-11-22 17:35:32 +08:00
Shawn 7996d8f709 refactor(bridge): rename the APIs for emqx_bridge 2021-11-22 17:35:32 +08:00
Shawn 1e6884ee7d fix(bridge): convert confs for http bridge when creating 2021-11-22 17:35:32 +08:00
Shawn f01f9632c1 fix(bridge): cannot start mqtt bridge from config 2021-11-22 17:35:32 +08:00
Shawn 63f942a1b8 feat(bridge): add descriptions to the mqtt bridge schema 2021-11-22 17:35:32 +08:00
Shawn 0cbdaa0f40 refactor(bridge): update config struct for mqtt bridge 2021-11-22 17:35:32 +08:00
Shawn d046f9c6e7 fix(bridge): improve the schema of connector_http 2021-11-22 17:35:32 +08:00
Shawn dd9e2c4b24 refactor(bridge): the configs for http bridges 2021-11-22 17:35:32 +08:00
Zaiming (Stone) Shi 90a65b8d04
Merge pull request #6242 from emqx/update-otp-version-and-pkg-name-scheme
Update otp version and pkg name scheme
2021-11-22 10:34:24 +01:00
JimMoen 08899552fe fix(mgmt): fix broken `ets:continuation` been sent between nodes 2021-11-22 17:16:12 +08:00
tigercl 6da039cf4f
Merge pull request #6231 from tigercl/fix/http-authn
fix(authn): fix pick worker error and match more http response
2021-11-22 09:10:44 +08:00
Ilya Averyanov 3d449b5acf fix(authn): create authn chain tab in start_link 2021-11-21 19:52:29 +03:00
Zaiming Shi 6ca6d60858 fix(dashboard): change hash from sha3_256 to sha256
old version os has no support for it
2021-11-20 21:21:11 +01:00
Zaiming Shi a705bc1fc7 build: change package name scheme
`${PROFILE}-${PKG_VSN}-otp${OTP_VSN}-${SYSTEM}-${ARCH}.zip`
2021-11-20 20:44:48 +01:00
Zaiming Shi f84d1aa75e style: fix some elvis style check 2021-11-20 20:41:06 +01:00
Zaiming Shi 6edd862dd5 refactor: no more EMQX_ENTERPRISE compile flag
The compile flag was introduced in EQM X 4.3 series
where CE and EE code was diverged large enough which made
non-practicle to determin edition at runtime.

such approach made testing quite challenging as we'll have to
build with different compile flags inorder to run per-edition
test cases

In this commit, we try to retrieve edition info from EMQX's
description text, (put to PT for fast access) at runtime
so we can test ALL editions from a super-set edition (EE).
2021-11-20 20:41:06 +01:00
Ilya Averyanov 8621257809
Merge pull request #6240 from savonarola/reinit-authn-1
fix(authn): transfer chain tab to emqx_authentication_sup
2021-11-19 19:57:14 +03:00
Ilya Averyanov c86887491c fix(authn): transfer chain tab to emqx_authentication_sup
To preserve configuration after emqx_authentication crashes.
2021-11-19 16:41:22 +03:00
k32 c9a59968ef refactor(emqx_schema): Code style 2021-11-19 11:41:29 +01:00
k32 deccba3406 fix(schema): Improve configuration of the internal DB 2021-11-19 11:41:08 +01:00
tigercl a074cf24d0
Merge pull request #6219 from tigercl/fix/authz-certs
fix(authz): fix badmatch when reading certs
2021-11-19 11:25:29 +08:00
zhouzb 980a6ceae9 fix(authn): fix pick worker error and match more http response 2021-11-19 11:19:52 +08:00
JianBo He 905e58c93a
Merge pull request #5997 from HJianBo/gw-fixes-part3
Gateway improvement part.3
2021-11-19 09:29:51 +08:00
zhouzb 47851c2430 fix(authz): fix badmatch when reading certs 2021-11-18 14:36:59 +08:00
JimMoen e361cd5733
Fix dashboard APIs return (#6177)
* fix(dashboard_api): delete non-exist user wrongly return 204

* fix(dashboard): dashboard user should use `tags` not `tag`

* fix(dashboard): create/update user return 200 with full users list

* fix(dashboard): logout status code 204

* fix(dashboard): update pwd status code 204

* test: test suite for dashboard APIs

* refactor(dashboard): user info mnesia record name use description

* style: make elvis happy

* fix(api): dashboard swagger check request should not override env

* fix(dashboard): add/modify dashboard returns single record

* ci: update emqx-fvt version to new tag 1.0.2-dev1
2021-11-18 14:36:12 +08:00
JimMoen 53b51e31f0 fix(api_schema): add authn mongo missing fields 2021-11-18 12:41:40 +08:00
zhongwencool 33f5eec802
feat: set keepalive via http api (#6171)
* feat: set keepalive via http api

* fix: make cluster_rpc test case solider
2021-11-18 10:56:58 +08:00
Zaiming (Stone) Shi 8d653b6d39
Merge pull request #6173 from zmstone/refactor-retainer-use-ordered-set
[5.0] refactor(retainer): use ordered_set Mnesia table type
2021-11-18 00:28:57 +01:00
Tobias Lindahl e52d8138bf test(persistent_session_SUITE): fix another flaky test
Make sure that the connection process dies before terminating the
snabbkaffe trace.
2021-11-17 14:04:06 +01:00
Tobias Lindahl 5914b8ad3d
Fix flaky tests for persistent sessions (#6202)
* test(persistent_session_SUITE): remove redundant sleep and trap_exits

* test(persistent_session_SUITE): fix race for started snabbkaffe

Sometimes snabbkaffe was not started when publishing from a different
process. Wrap the publishing code in a ?check_trace to make sure it is
started, and make sure the publish process is truly down before ending
the trace.

* test(persistent_session_SUITE): fix takeover race

Make sure the previous session is unregistered before trying to
connect again. Sometimes the new session was trying to take over the
session that was shutting down still.
2021-11-17 18:55:11 +08:00
tigercl 7950e26774
Merge pull request #6193 from tigercl/feat/mongo-srv-5.0
feat(mongo srv): support srv record for mongo connector
2021-11-17 15:42:01 +08:00
JianBo He ad2dbb5a49 chore(gw-sn): append messge headers 2021-11-17 14:51:27 +08:00
zhongwencool 04c204374a
chore: All placeholders are placed in the same header file (#6190)
* chore: All placeholders are placed in the same header file

* feat: Separate placeholders from the rules engine
2021-11-17 14:40:48 +08:00
zhongwencool a7a5e2d1f4
fix: can't get default key if all key if default (#6181) 2021-11-17 14:27:51 +08:00
JimMoen 57075902a3
fix(mgmt_api): kickout non-existing clientid should return code `404` (#6180) 2021-11-17 12:29:27 +08:00
zhouzb f8a625a67f feat(mongo srv): support srv record for mongo connector 2021-11-17 12:12:30 +08:00
JianBo He d0bdf27e0c chore: fix elvis warnings 2021-11-17 11:04:09 +08:00
DDDHuang 7c48bcabed
fixt: generate api (#6124)
* fix: generate api

* fix: banned suite

* fix: generate api bad rpc

* fix: bad message suite

* fix: create banned with check existed
2021-11-17 10:55:48 +08:00
JianBo He f23d2f2c92 chore: refine case side-effect 2021-11-17 10:46:46 +08:00
JianBo He 814e01c0cb test(authz): deps on emqx_conf instead of meck emqx_schema 2021-11-17 10:46:38 +08:00
Tobias Lindahl ae5792fa64 chore(emqx_cm): remove unused tracepoints 2021-11-16 14:37:15 +01:00
Tobias Lindahl c9b86a8375 test: make testcases not rely on timer:sleep 2021-11-16 13:50:01 +01:00
Tobias Lindahl 05f3bc8c90 fix(emqx_cm): make takeover session less likely to hit a race 2021-11-16 13:49:45 +01:00
JimMoen c0e388e52a style: make elvis happy 2021-11-16 19:35:35 +08:00
Zaiming Shi ac693c0dcb refactor(retainer): use ordered_set Mnesia table type
ordered_set is compatible to set in all the existing APIs
and in addition, it provides a more deterministic traversing
cursor in case we want to change the current dirty_select
based "select ALL" implementation
2021-11-16 12:33:59 +01:00
JimMoen 1dc0a2e8b5 fix(mgmt_api): kickout non-existing clientid should return code `404` 2021-11-16 17:32:12 +08:00
JianBo He 883c1b2e59 chore(gw): remove needless lines 2021-11-16 10:00:54 +08:00
JianBo He cf1d98adc6 chore(gw): save cert files 2021-11-16 10:00:54 +08:00
JianBo He 5a08fa6a03 docs(gw): add gateway comment 2021-11-16 10:00:54 +08:00
JianBo He 1f067d6f3b docs(gw): add docs for gateway schema 2021-11-16 10:00:54 +08:00
JianBo He bc3f3b4c55 test(gw): more robust testing 2021-11-16 10:00:53 +08:00
JianBo He 287859fe36 chore(gw): integrate emqx-conf 2021-11-16 09:58:37 +08:00
JianBo He 0b36b73ee1 fix(emqx_conf): fix badmatch for returned error value 2021-11-16 09:58:37 +08:00
JianBo He cf17a55079 chore(gw): rename cm-registry shard name 2021-11-16 09:58:37 +08:00
JimMoen c05ecdbcb8
fix(api): DELETE success wrongly returned code 200 (#6135) 2021-11-16 09:28:57 +08:00
Zaiming Shi ff23e9dde1 test: sync with emqx_cm process before flushing emqx_pool 2021-11-15 22:27:16 +01:00
JianBo He a113b9b65d chore: fix elvis warnings 2021-11-15 22:26:46 +01:00
JianBo He 9e333ac8b3 fix(emqx_cm): fix typos 2021-11-15 22:26:46 +01:00
Zaiming Shi dd771c5568 fix(emqx_mgmt_cli): idempontent kick. now it always returns ok 2021-11-15 22:26:46 +01:00
Zaiming Shi 7e2ca9e287 fix(session): force kill session for 'kick' and 'discard'
Prior to this fix, 'kick' and 'discard' calls may timeout (or
fail for other reason), failures lead to only a log, then
continue to allow the new session to get registered.

As a result, in case a client is stuck, there is no way to
force it to step down, end up with multiple connections (sessions)
for the client ID in dashboard.

After this fix, the stale pids are notified to shutdown
via a gen_server:call, and forced with a exit(Pid, kill) for any
exception happend to the gen_server:call
2021-11-15 22:26:46 +01:00
Zaiming (Stone) Shi 650905667b
Merge pull request #6172 from JimMoen/fix-emqx_cm-ct
test: add tasks flushing in emqx_cm cases
2021-11-15 17:17:27 +01:00
JimMoen cf80aaacc3 test: add tasks flushing in emqx_cm cases 2021-11-15 20:40:43 +08:00
Zaiming (Stone) Shi 9761fe2f6d
Merge pull request #6009 from tigercl/fix/authn
fix(authn): fix handling of query result
2021-11-15 13:32:56 +01:00
JimMoen dcc5ae0642 test: API return code section 2021-11-15 16:30:44 +08:00
JimMoen b3f9220d02 style: make elvis happy 2021-11-15 16:30:44 +08:00
JimMoen 106aa559e0 fix(api): DELETE success wrongly returned code 200 2021-11-15 16:30:44 +08:00
JimMoen 59e6b6ee5b
test(cm): add sleep time for multicore scheduling (#6164) 2021-11-15 15:43:26 +08:00
DDDHuang 1476accd63
fix: monitor bad mnesia write & rpc call (#6060) 2021-11-15 15:28:19 +08:00
zhongwencool 01f29ab052
chore(swagger): lwm2m2 http api to hocon (#6153) 2021-11-15 15:08:50 +08:00
JimMoen 5a928ad8c3 test(cm): add sleep time for multicore scheduling 2021-11-15 13:52:15 +08:00
DDDHuang f0107868d4
fix: test suite stable (#6136)
* fix: test suite stable
2021-11-12 10:53:30 +08:00
JimMoen 9b17757ae8 style: make elvis happy 2021-11-12 09:29:26 +08:00
JimMoen 37cdd429b2 fix(api): remove sessionID from client information 2021-11-12 09:29:26 +08:00
DDDHuang bc6e49804f
feat: support CORS request
feat: support CORS request
2021-11-11 20:27:39 +08:00
zhongwencool 7a5da76197
Mgmt http api banned (#5998)
* fix(swagger): don't transform [log,publish] to a list.pwd

* chore: replace banned-api by hocon schema

* fix(api): code style warning
2021-11-11 16:35:29 +08:00
JimMoen a65597e302
Authz api mnesia refactor (#6082)
* refactor(authz): macro move to hrl and rename, add meta example

* refactor(authz_api_mnesia): use config schemas for request validations

* chore(authz_api_mnesia): suite module typerefl fit

* chore(style): authz_api_mnesia style format
2021-11-11 16:29:36 +08:00
Zaiming Shi b96a106463 style: make elvis happy 2021-11-10 22:54:58 +01:00
Zaiming Shi 9c1972020c test: fix flaky test emqx_stomp_SUITE:t_rest_clienit_inf 2021-11-10 22:51:18 +01:00
Ilya Averyanov 4f0311b6f8
Merge pull request #6093 from savonarola/add-authn-tests
chore(authn mnesia): reimplement emqx_authn_mnesia provider tests
2021-11-10 21:20:46 +03:00
Ilya Averyanov a0a88e23b0 chore(authn mnesia): reimplement emqx_authn_mnesia provider tests 2021-11-10 17:13:04 +03:00
Thales Macedo Garitezi 60d5017eea
style(elvis): fix elvis style complaints 2021-11-10 11:12:10 -03:00
Thales Macedo Garitezi 8fe342a02d
feat(stats): track live channel / client count
Track connected client count

In order to correctly display the number of _connected_ clients in our
monitor dashboard, we need to track those connections that are
actually connected to clients, not considering connections from
persistent sessions that are disconnected. Today, the
`connections.count` that is displayed in the dashboards considers
those disconnected persistent sessions as well.

The new statistics can be found in the
[`emqx_management`](https://github.com/emqx/emqx/tree/main-v4.4/apps/emqx_management)
plugin, under `/api/v4/stats`, in the keys
`live_connections.{max,count}`.
2021-11-10 11:12:10 -03:00
lafirest e8f6035c34
feat(emqx_lwm2m): add some lwm2m api (#6047) 2021-11-10 14:20:33 +08:00
zhongwencool 2d159ad9a1
fix(test): flaky mqtt expiry test case. (#6099) 2021-11-10 09:55:36 +08:00
JimMoen ac23214447
fix(api): config api on sepcific node (#6100)
* fix(api): config api on sepcific node

* chore(style):  code style for elvis checking
2021-11-10 09:38:45 +08:00
zhongwencool eea789451b
feat: sha3_256 dashboard account's password (#6084) 2021-11-09 15:41:28 +08:00
Ilya Averyanov 0f8ad29e91
Merge pull request #6078 from savonarola/auth-api-schema-examples
refactor(authn api): add more schema examples
2021-11-08 19:44:02 +03:00
zhongwencool b873b92717
fix(test): cluster_rpc retry interval incorrect (#6038) 2021-11-08 23:23:17 +08:00
k32 5726444645
Merge pull request #6066 from k32/remove-nosuspend-5
fix(emqx_connection): Add backpressure to TCP connections
2021-11-08 13:39:38 +01:00
Ilya Averyanov a84b84aac9 refactor(authn api): reformat for elvis compliance 2021-11-05 19:36:26 +03:00
Ilya Averyanov 1e036bf74d refactor(authn api): add more schema examples 2021-11-05 16:12:14 +03:00
Zaiming (Stone) Shi 3b02366a5b
Merge pull request #6064 from zmstone/style-check-newline-at-eof
Style: check newline at EOF
2021-11-04 18:01:45 +01:00
k32 0a5a9bd7d0 fix(emqx_connection): Add backpressure to TCP connections
Fixes #5494
2021-11-04 16:24:47 +01:00
Zaiming Shi d1abb30818 ci: fix elvis check 2021-11-04 15:27:10 +01:00
Zaiming Shi 56e2a9741f style: ensure newline at EOF for all files 2021-11-04 14:40:14 +01:00
zhouzb 48ddd056b5 test(authn): add test cases for authn 2021-11-04 10:03:34 +08:00
zhouzb ca4bb100ec fix(authn): fix bad parsing for postgresql SQL 2021-11-04 10:01:54 +08:00
Ilya Averyanov aed9ad3f7d
Merge pull request #6037 from savonarola/fix-authn-api-atom-leak
fix(authn api): eliminate possible atom leak
2021-11-03 23:17:47 +03:00
Zaiming Shi a139a0d453 fix(config): pin hocon 0.20.6
fix translation error
2021-11-03 11:15:35 +01:00
Tobias Lindahl 89cd68d36f refactor(persistent_sessions): fix coding style 2021-11-02 09:33:54 +01:00
Tobias Lindahl 8385eff98e fix(persistent_sessions): we only need to persist if the session expired 2021-11-02 09:27:50 +01:00
Tobias Lindahl b7ed649185 test(persistent_session): wait in test to avoid race 2021-11-01 14:56:10 +01:00
Ilya Averyanov 796553b5ea fix(authn api): eliminate possible atom leak 2021-11-01 16:51:48 +03:00
Tobias Lindahl ce49a281ed fix(persistent_sessions): protect against looking up stale data 2021-11-01 14:21:01 +01:00
Tobias Lindahl 329dd4d780 test(persistent_session): try to fix flaky snabbkaffe failure 2021-11-01 14:21:01 +01:00
Tobias Lindahl 1f13a6caad chore(persistent_sessions): tune mnesia parameters for better dump behavior 2021-11-01 14:21:01 +01:00
Tobias Lindahl ec68d7fc58 test(persistent_sessions): stabilize flaky tests 2021-11-01 14:21:01 +01:00
Tobias Lindahl 7ae6e04582 fix(persistent_sessions): channels can terminate without a session 2021-11-01 14:21:01 +01:00
zhouzb 29fb9b3361 fix(authn): fix bad type of hash 2021-11-01 18:49:13 +08:00
Ilya Averyanov 800b4b32c7
refactor(authn api): use config schemas for request validations (#5999) 2021-11-01 17:52:03 +08:00
JimMoen d9cb0283f3 fix(alarm): duration unit in dashboard, microsecond => millisecond 2021-10-29 16:48:13 +08:00
zhouzb c64637ca39 test(authn): add test case of version checking 2021-10-29 14:18:25 +08:00
zhouzb 966348db05 fix(authn): fix version switching error when updating multiple times 2021-10-29 10:12:29 +08:00
zhongwencool e62fde321c
Emqx alarm (#5994)
* chore(alarm): normalize_message outside emqx_alarm

* chore(alarm): don't cache config in emqx_alarm; remove dirty_write/read; add desc/example to alarm; add more test

* chore(alarm_api): alarm_api with hocon schema

* fix: activted's nullable is true

* fix(swagger): translate map to object

* fix(cluster_rpc): debug failed cluster_rpc test

* fix: Update schema description

Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2021-10-28 18:03:51 +08:00
William Yang 2c69c00906
Merge pull request #6010 from qzhuyan/perf/William/pool-worker-pickup
During the benchmark test of ehttpc, we found the gproc's random pool or round-robin pool doesn't scale well.

This PR makes emqx prefer to use the hash pool.

changed:

retainer pool: from random to hash
connector : from random to hash (default)
bridge : from random to hash (default)
authn_http : from random to hash (example)

unchanged:
worker pool type: hash
broker pool type: hash
resource type: hash
exproto type: hash
dashboard type: hash
2021-10-28 09:40:14 +02:00
Tobias Lindahl 48c7788d6f
Merge pull request #5875 from emqx/extend-persistent-sessions
Extend persistent sessions
2021-10-27 09:12:13 +02:00
zhouzb a712daaebc fix(authn): fix bad list comprehension 2021-10-27 15:08:02 +08:00
William Yang 5e6dab435a perf(pool): authz http example pool_type to hash 2021-10-27 08:48:41 +02:00
William Yang 9c8cd6c437 perf(pool): change emqx_connector default pool type to hash 2021-10-27 08:47:02 +02:00
William Yang 34979c51d7 perf(config): emqx_bridge default hash pool 2021-10-27 08:45:13 +02:00