a44f8eb583
Merge pull request #11861 from zmstone/1101-do-not-repeat-print-shell-warning
...
fix(shell): do not repeat print shell warnings
2023-11-02 10:28:56 +01:00
34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293
...
fix(bridge_v2 API): optional cascading delete operation
2023-11-02 10:28:43 +01:00
ec2d339355
Merge pull request #11857 from kjellwinblad/kjell/shared_con/EMQX-11270
...
fix(bridge_v2): channels should not be removed when status is connecting
2023-11-01 16:46:47 +01:00
5b964ef415
Merge pull request #11858 from id/1101-emqx-11288-fix-kafka-replaq-dir-conflict
...
emqx 11288 fix kafka replaq dir conflict
2023-11-01 16:45:02 +01:00
4bea65bf97
fix(bridge_v2_api): don't disable schema check
2023-11-01 12:44:40 -03:00
2dd5061643
fix(kafka): kafka bridge replaq dir conflict
2023-11-01 15:44:28 +01:00
357b664c8d
fix(bridge_v2): more fixes thanks to PR comments from @thalesmg
2023-11-01 15:27:54 +01:00
b06d05eaac
test(bridge_v2): fix test case after new API
2023-11-01 15:27:54 +01:00
96d6c6db49
test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change
2023-11-01 15:27:53 +01:00
edb1d37e67
chore(bridge_v2): make fixes thanks to PR comments from @thalesmg
2023-11-01 15:27:53 +01:00
95f3b94ac3
fix(bridge_v2): channels should not be removed when status is connecting
...
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
1e935e9eb4
fix(bridge_v2 API): optional cascading delete operation
...
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
2023-11-01 15:03:18 +01:00
c38725facd
fix(shell): do not repeat print shell warnings
2023-11-01 14:44:18 +01:00
c03979a48a
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-01 09:51:26 -03:00
10b2894b6e
chore(bridge_v2_api): remove misplaced handling code
2023-11-01 11:31:17 +01:00
ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched
...
perf: no need to format event message if no bridge matched
2023-11-01 10:22:06 +01:00
aa8a6f2e26
Merge pull request #11853 from thalesmg/test-bridge-v1-compat-layer-r53-20231030
...
test(bridges): add bridge v1 compatibility layer test suite, and other fixes
2023-11-01 09:51:10 +01:00
29683072a1
fix(emqx_connector): remove `stop` and `restart` operations
2023-11-01 09:18:50 +01:00
45a39d97c6
fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1
...
Fixes https://emqx.atlassian.net/browse/EMQX-11291
2023-10-31 16:39:47 -03:00
b420b53075
fix(bridges_v2): check bridge name length before attempting atom conversion
...
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
8eb822d898
test(bridges): add bridge v1 compatibility layer test suite
2023-10-31 13:21:01 -03:00
db5cdb6742
perf: no need to format event message if no bridge matched
2023-10-31 16:48:10 +01:00
3e9f48d10f
fix: remove unneeded clause
2023-10-31 16:05:31 +01:00
2a4eaf41ba
docs: fix examples for PUT operations on bridge_v2 and connectors
...
Examples would show `type` and `name` properties in the request body, which is
not accepted by the schema.
Also fixes some minor inconsistencies in the example names of connectors and
bridges.
2023-10-31 15:10:11 +01:00
84e78f5d2e
fix(bridge_api): fix response status code for some operations
2023-10-31 10:40:56 -03:00
c90ca1ea53
fix(bridge_v1): always delete connector when deleting v1 bridge
...
Fixes https://emqx.atlassian.net/browse/EMQX-11287
2023-10-31 10:40:06 -03:00
b1ab213081
chore: merge 'upstream/release-53' into 1031-sync-r53
2023-10-31 11:06:25 +01:00
39f49d425c
test(emqx_bridge): fix test for bridge_v2_type_to_connector_type
2023-10-31 08:34:05 +01:00
63606922f2
refactor(emqx_bridge): only one place to generate same error
2023-10-31 08:34:05 +01:00
b256241650
test(emqx_bridge_azure_event_hub): fix legacy v1 bridge API tests
2023-10-31 08:34:05 +01:00
176bbe88bc
fix: use `_producer` for AEH bridge type
2023-10-31 08:34:03 +01:00
aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import
...
EMQX-11272 fix kafka bridge v1 import
2023-10-31 07:58:54 +01:00
9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271
...
fix: crash when listing non bridge_v1 compatible bridge_v2
2023-10-30 20:29:36 +01:00
c07cf9051e
Merge pull request #11844 from zmstone/1030-downgrade-bridge-type-for-old-api
...
1030 downgrade bridge type for old api
2023-10-30 19:47:24 +01:00
124d79a1ca
Merge pull request #11843 from thalesmg/test-kafka-producer-nits-r53-20231030
...
test(kafka_producer): minor adjustments to test suite
2023-10-30 18:27:10 +01:00
350e8433ab
fix(bridge_v2): downgrade bridge type for GET api results
2023-10-30 18:17:35 +01:00
0562585c8f
fix(emqx_mgmt_data_backup): implement importer modules ordering
...
`emqx_bridge_v2` depends on `emqx_connector`, so connectors must be imported first.
2023-10-30 19:14:58 +02:00
bc8c291820
fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2
2023-10-30 19:14:58 +02:00
d94193ac15
fix(emqx_connector_schema): add kafka alias for kafka_producer
2023-10-30 19:14:58 +02:00
0935bb6225
fix(emqx_connector): fix badarity error
2023-10-30 17:52:39 +02:00
d1cd5dd817
fix(emqx_mgmt_data_backup): upgrade raw conf before validating and importing
2023-10-30 17:52:39 +02:00
8fb426e57f
fix: crash when listing non bridge_v1 compatible bridge_v2
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
2023-10-30 16:32:04 +01:00
a60b96c5fd
test(kafka_producer): minor adjustments to test suite
...
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
3f6c09b195
Merge pull request #11780 from savonarola/1017-fix-pbkdf2-validation
...
fix(authn): fix pbkdf2 option validation
2023-10-30 16:37:37 +02:00
5a6e55e7af
fix(emqx_bridge_kafka): schema fixes for kafka_producer in bridge_v2
2023-10-30 14:59:06 +01:00
f463eff02d
chore: e5.3.1-alpha.2
2023-10-30 14:49:08 +01:00
0656b6be3c
refactor(emqx_config): use dynamic callback to upgrade raw config
2023-10-30 14:49:08 +01:00
9dc3a169b3
feat: split bridges into a connector part and a bridge part
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.
Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
97601171a5
Merge pull request #11818 from zmstone/1025-fix-flaky-test
...
test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked
2023-10-25 20:46:31 +02:00
76a8b0efd4
test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked
2023-10-25 16:08:31 +02:00
f52b4a9e32
fix: gen_rpc tcp client port align with server port
2023-10-25 12:11:35 +02:00
37dd44c3d7
Merge pull request #11813 from qzhuyan/fix/william/genrpc-ssl-port-align
...
Fix/william/genrpc ssl port align
2023-10-25 09:04:19 +02:00
6346e0d28a
fix(gen_rpc): ssl client port align with server port
2023-10-24 22:33:52 +02:00
abcb3166f6
fix: use binary string for raw default values
2023-10-24 14:46:01 +02:00
6e12569260
Merge pull request #11797 from sstrigler/EMQX-1057-fix-authz-api-donot-return-200
...
fix: return 404 if built_in_database not configured as auth source
2023-10-24 12:08:53 +02:00
ea9228108b
fix: use minirest filter
2023-10-24 09:57:08 +02:00
99fab8dc5d
chore: missing change log for duplicated apikey fixing
2023-10-24 09:44:39 +08:00
d392c9989f
Merge pull request #11793 from thalesmg/test-rule-engine-foreach-m-20231019
...
test(rule_engine): add test to exemplify `foreach` with json array payload
2023-10-23 13:43:33 -03:00
4e0e755b28
fix: return 404 if built_in_database not configured as auth source
2023-10-23 16:26:41 +02:00
8c6f6b56b8
chore(tests): report more friendly about unset PROFILE variable
2023-10-23 13:48:42 +03:00
edde661da3
fix(authn): fix pbkdf2 option validation
2023-10-23 10:26:11 +03:00
a9b42af8b3
Merge pull request #11796 from qzhuyan/fix/william/gen_rpc-driver
...
fix(gen_rpc): ensure client/server use same driver
2023-10-23 08:24:17 +02:00
e6576951ef
test: cleanup duplicated apikey with different name
2023-10-20 22:33:26 +08:00
d467289bb2
fix: avoid duplicated apikey from data import
2023-10-20 21:13:04 +08:00
dd5807b797
fix(gen_rpc): ensure client/server use same driver
...
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
8c4beec6f0
test(rule_engine): add test to exemplify `foreach` with json array payload
...
Relates to https://emqx.atlassian.net/browse/EMQX-11174
2023-10-19 18:19:42 -03:00
0b4600c293
feat(auth): improve redis command parsing and validation
2023-10-19 23:01:35 +03:00
eda82caf65
Merge pull request #11791 from thalesmg/fix-coap-heartbeat-m-20231018
...
fix(coap): increase received packet counter for keepalive
2023-10-19 13:12:41 -03:00
8d82c30b00
Merge pull request #11771 from savonarola/1015-validate-bcrypt-schema-in-api
...
feat(authn): allow authn providers to define a separate schama for API
2023-10-19 15:34:34 +03:00
8f9ac6c1f3
Merge pull request #11776 from savonarola/1017-auth-cleanup-2
...
chore(authn): remove dead code
2023-10-19 15:28:35 +03:00
5b9866f630
fix(coap): increase received packet counter for keepalive
...
Fixes https://emqx.atlassian.net/browse/EMQX-11193
Fixes https://github.com/emqx/emqx/issues/11779
2023-10-18 16:35:33 -03:00
2388d36b09
fix: allow viewers to change their own passwords
2023-10-18 18:40:02 +08:00
e1c8317779
chore(authn): remove dead code
2023-10-18 10:57:55 +03:00
c0ebaf9b6f
Merge pull request #11541 from savonarola/0829-duplicate-ft-responses
...
Support asynchronous mode of receiving responses to FT operations
2023-10-17 17:08:00 +03:00
6354f3b04f
feat(authn): allow authn providers to define a separate schama for API
2023-10-17 13:19:11 +03:00
55618dd28a
Merge pull request #11731 from zmstone/1008-expose-file-transfer-host-conf
...
fix(api-docs): add file-transfer config to hot-config scope
2023-10-17 11:23:16 +02:00
dfa5909392
Merge pull request #11752 from zmstone/1011-change-default-rpc-module-for-mria
...
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
2023-10-17 10:21:14 +02:00
162b901be6
Merge pull request #11763 from zmstone/1114-sync-release-53
...
Merge release-53 back to master
2023-10-16 21:42:02 +02:00
16cc816bd3
Merge pull request #11757 from zhongwencool/trace-api-return-500
...
fix: 500 error response when downloading non-existent trace files
2023-10-16 13:40:18 +08:00
6eb3bb7cff
Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53
2023-10-14 10:16:38 +02:00
7c022c2c6a
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
...
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-10-14 10:13:31 +02:00
cd2752117c
Merge pull request #11762 from savonarola/1013-fix-mnesia-authz-destroy
...
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 20:54:14 +03:00
bde8800f2e
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 18:29:45 +03:00
90a0c093bf
Merge pull request #11750 from savonarola/1011-redact-auth-http-request
...
Do not trace authn http request bodies
2023-10-13 17:25:28 +03:00
4f14e8df22
chore(source dir): split out postgresql connector
2023-10-13 16:58:57 +03:00
7a6c756b7c
Merge pull request #11754 from thalesmg/fix-postgres-error-reason-log-m-20231011
...
fix(postgres): format unicode error messages from driver
2023-10-13 09:58:28 -03:00
0144ed9981
Merge pull request #11753 from keynslug/chore/smaller-ci-artifacts
...
chore(ci): try to reduce CI artifacts footprint
2023-10-13 16:56:47 +07:00
51c57a5f0a
fix(cass): avoid using aggregates in healthcheck query
...
Otherwise, this query will produce warning message per each
healthcheck, telling the user:
```
WARN <...> - Aggregation query used without partition key
```
2023-10-13 16:50:19 +07:00
083e2da347
chore(bpapicheck): make some failures more user-friendly
2023-10-13 15:42:08 +07:00
0b9ac24c1e
fix(bpapicheck): stop depending on git index
2023-10-13 15:40:49 +07:00
96c546c187
Merge pull request #11725 from lafirest/feat/os-ldap
...
chore: change the LDAP integration to opensource
2023-10-13 15:05:39 +08:00
dd03be29a6
Merge pull request #11751 from zmstone/1011-refactor-move-emqx_license-app-to-apps
...
1011 refactor move emqx_license app to apps
2023-10-13 06:45:49 +02:00
33ff5d5588
chore: update auth header file
2023-10-13 10:16:47 +08:00
68f31a9da2
fix(authn): do not trace authn requests
2023-10-12 16:32:16 +03:00
08795f559c
Merge pull request #11336 from savonarola/0724-trace-authz
...
Trace non-resultative authz calls
2023-10-12 14:51:15 +03:00
6413afd0bc
fix(gcpdev): restore original shard in mnesia tab definition
...
Changing the shard is not backward compatible, and leads to a crash
when upgrading from 5.2.x.
2023-10-12 17:30:07 +07:00
4ecd5e17a2
chore(authz): trace non-resultative authz calls to backend modules
2023-10-12 12:29:39 +03:00
3e658b3da9
chore: update changes
2023-10-12 17:15:47 +08:00
9b4def885a
chore: change the LDAP integration to opensource
2023-10-12 16:37:32 +08:00
1021088f5f
fix: 500 error response when downloading non-existent trace files
2023-10-12 15:23:23 +08:00
b07dddd49e
fix(postgres): format unicode error messages from driver
...
Fixes https://emqx.atlassian.net/browse/EMQX-11024
Sample error:
```
{error, error, <<"42501">>, insufficient_privilege,
<<229,175,185,232,161,168,32,109,113,116,116,95,117,115,101,114,32,230,157,131,233,153,144,228,184,141,229,164,159>>,
[]}
```
2023-10-11 17:56:12 -03:00
534c9bdc13
feat(ft): add additional operation status report channel
2023-10-11 19:51:36 +03:00
d9859b6b49
Merge pull request #11749 from thalesmg/test-fix-flaky-http-suite-r53-20231011
...
test: attempt to fix flaky http bridge suite
2023-10-11 16:07:17 +02:00
e6358ca18c
Merge pull request #11747 from qzhuyan/dev/william/quicer-0.0.202
...
bump to quicer 0.0.202
2023-10-11 15:43:47 +02:00
e069680bca
refactor: move emqx_license app from lib-ee to apps
2023-10-11 15:41:35 +02:00
ab662f506d
test: attempt to fix flaky http bridge suite
...
```
=CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
crasher:
initial call: ehttpc:init/1
pid: <0.23358.2>
registered_name: []
exception error: bad argument
in function persistent_term:get/1
called as persistent_term:get({emqx_bridge_http_SUITE,
do_t_async_retries,attempts})
*** argument 1: no persistent term stored with this key
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-11 09:13:25 -03:00
57655854d1
Merge pull request #11743 from savonarola/1010-auth-leftovers
...
chore(auth): cleanup code
2023-10-11 15:09:07 +03:00
a69c53455f
fix: bump to quicer 0.0.202
...
bring in urgent fixs from msquic 2.2.3
2023-10-11 13:12:32 +02:00
cc392cadb8
chore: update apps/emqx_retainer/src/emqx_retainer_mnesia_cli.erl
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-10-11 18:40:09 +08:00
03ae5bf3c8
chore(auth): cleanup code
2023-10-11 13:13:50 +03:00
f4a88f717b
feat: support retainer Start Length command
2023-10-11 11:47:37 +08:00
8e7ba16c3a
fix: nothing show when run clients list command
2023-10-11 11:47:15 +08:00
574dc2f243
fix: observer load command crash when loading noexist module
2023-10-11 10:36:33 +08:00
a2e86c67db
Merge pull request #11733 from keynslug/fix/EMQX-10827/sesson-takeover
...
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
2023-10-10 19:03:29 +02:00
0f07f678e7
Merge pull request #11742 from zmstone/1010-sync-release-53-to-master
...
1010 sync release-53 to master
2023-10-10 18:56:23 +02:00
196b24c0cb
Merge pull request #11741 from thalesmg/test-fix-flaky-http-test-m-20231010
...
test: attempt to fix flaky http bridge suite
2023-10-10 13:52:08 -03:00
ee45145fb5
Merge pull request #11724 from thalesmg/fix-kprodu-sync-metrics-m-20231006
...
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers
2023-10-10 12:13:33 -03:00
ec588f94e2
test: attempt to fix flaky http bridge suite
...
```
=CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
crasher:
initial call: ehttpc:init/1
pid: <0.23358.2>
registered_name: []
exception error: bad argument
in function persistent_term:get/1
called as persistent_term:get({emqx_bridge_http_SUITE,
do_t_async_retries,attempts})
*** argument 1: no persistent term stored with this key
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-10 09:50:53 -03:00
bfb2218392
chore: fix release version
2023-10-10 14:50:49 +02:00
cf2075d7d8
chore: remove mention of `is_buffer_supported` from typespec
2023-10-10 09:49:18 -03:00
143673b703
chore: bump release version to 5.3.1-alpha.1
2023-10-10 14:35:46 +02:00
0a9e1dfa88
Merge pull request #11734 from zmstone/1002-support-ipv6-only-clustering
...
Make IPv6 clustering options configurable
2023-10-10 14:29:52 +02:00
5fff2ffe45
test(kafka): try to stabilize `t_dynamic_mqtt_topic/1` testcase
2023-10-10 17:18:24 +07:00
9b573834f9
Merge pull request #11736 from keynslug/fix/eval-erl-audit
...
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 15:49:19 +07:00
2626d793a7
test: try to resolve schema module from PROFILE in tests
2023-10-10 10:24:10 +02:00
808237364c
fix(api-docs): add file-transfer config to hot-config scope
2023-10-10 10:24:10 +02:00
e7e696cd66
feat(rpc): add ipv6_only config
2023-10-10 10:05:14 +02:00
e07937a3ef
fix(ldap): escape the escape character (\)
2023-10-10 13:49:36 +08:00
2a291dfd27
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 12:31:19 +07:00
522302fee1
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
...
Which was accidentally broken in bf164175
2023-10-10 01:34:10 +07:00
1e93d2f1fc
chore: upgrade to ekka 0.15.16 gen_rpc 3.2.0
...
* ekka 0.15.16 supports 'inet6_tls' as ekka.proto_dist
* gen_rpc 3.2.0 supports true | false as gen_rpc.ipv6_only
2023-10-09 20:03:21 +02:00
497e08448d
feat(cluster): support ipv6 and tls on ipv6 for clustering
...
Made possible to configure inet6_tls for Erlang distribution
Also, added support to configure ipv6 listener for gen_rpc
2023-10-09 20:03:21 +02:00
03d8e06ff7
chore(emqx_rule_funcs): regroup export functions
2023-10-09 20:03:21 +02:00
d6781efee2
fix(resource): change how buffer workers are started
2023-10-09 15:02:25 -03:00
902b1d6ec5
fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
...
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
eebfb44f72
fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
...
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
79cf0a2ced
fix(kafka_producer): correctly handle metrics for connector that have internal buffers
...
Fixes https://emqx.atlassian.net/browse/EMQX-11086
There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).
We use simple_sync_query to call the Kafka Producer bridge. If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
c60915293a
Merge pull request #11634 from savonarola/0919-auth-refactor
...
Auth refactor
2023-10-09 19:51:26 +02:00
8f4cdc3fcf
Merge pull request #11732 from lafirest/test/sso_cli
...
test(ldap): add test suite for SSO CLI
2023-10-09 20:35:23 +08:00
9ab49a7ae3
Merge remote-tracking branch 'origin/master' into 0919-auth-refactor
2023-10-09 14:18:10 +02:00
2c7e5eb1cb
Merge pull request #11728 from lafirest/fix/ldap_filter
...
fix(ldap): improve the filter lex && parse
2023-10-09 18:24:57 +08:00
e3550fc07b
test(ldap): add test suite for SSO CLI
2023-10-09 16:05:56 +08:00
0c89b6b213
docs: swagger api tags should camel case
2023-10-09 15:21:51 +08:00
b2a6724dc2
fix(ldap): improve the filter lex && parse
...
1. auto escape special chars in the filter
2. fix a bug that the value can't be `dn`
2023-10-08 18:39:20 +08:00
482e82f914
fix: cli's args in audit log should be array
2023-10-07 17:38:02 +08:00
34186fcc74
fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
...
Fixes https://emqx.atlassian.net/browse/EMQX-11085
Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
398a62031e
chore: update app versions
2023-10-05 13:41:50 +03:00
5dff36474d
chore(auth): get rid of hardcoded schema modules in auth
2023-10-05 13:41:50 +03:00
c2c56ba481
chore(auth): update tests
2023-10-05 13:41:50 +03:00
1eb75b43c4
chore(auth): split emqx_authn and emqx_authz apps
2023-10-05 13:41:50 +03:00
fd75dc895d
fix(authn): "authentication" importance should not be "hidden"
2023-10-03 11:18:21 +02:00
ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
...
0928 sync release 53
2023-10-02 21:57:32 +02:00
3f6e0e890b
chore: bump emqx_durable_storage and emqx_license app vsn
2023-10-02 20:13:09 +02:00
2358d67908
refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
...
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
bce8fd2fbc
chore: bump app versions
2023-09-29 18:56:52 +02:00
ed5b456d62
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-29 18:24:44 +02:00
a852400fb3
chore: add 5.3.bpapi
2023-09-29 16:34:04 +02:00
fe01aaff4f
chore: prepare for e5.3.0 promotion
2023-09-29 13:32:57 +02:00
ce5bd0a3ce
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-29 11:36:32 +02:00
c64e599e81
docs: document how to retrieve peercert
2023-09-29 10:33:57 +02:00
dc147fd310
fix(rule-engine): console action has no args field
2023-09-29 10:33:33 +02:00
1177a32310
chore: bump version to 5.3.0-rc.2
2023-09-29 09:49:27 +02:00
6891234390
chore: return simplified error reason for less logging
2023-09-29 09:48:15 +02:00
c2d750aa09
fix(resource): redact query args in exception log
2023-09-29 09:20:42 +02:00
02ef854f0f
fix(ldap): no crash when no query result is empty list
2023-09-29 08:54:41 +02:00
b59a7ff2dd
Merge pull request #11706 from zmstone/0928-fix-ldap-sso-logging-level
...
0928 fix ldap sso logging level and reject muti-match results
2023-09-29 02:40:37 +02:00
4a4730ad46
fix(ldap): handle invalidCredentials in ldap authn
2023-09-29 00:51:05 +02:00
9ee2cb9c79
fix(ldap): return unrecoverable_error if more than on match found
2023-09-28 23:58:34 +02:00
cc5dab1dc7
chore: fix code style
2023-09-28 21:29:59 +02:00
b267fc2588
chore: bump release version to 5.3.0
2023-09-28 21:22:33 +02:00
922d5a9a83
fix(ldap): do not allow multi-matches to proceed
...
if ldap query returns more than on match
we should reject the auth request instead of picking
the first one
2023-09-28 21:20:50 +02:00
d858f8af39
test: fix openldap docker runs
2023-09-28 18:40:03 +02:00
b28e781c50
fix(ldap-sso): do not log error level when invalid user credentials
2023-09-28 18:37:11 +02:00
36f3052be1
Merge pull request #11698 from zmstone/0928-disable-audit-log-by-default
...
fix(audit): disable audit log by default
2023-09-28 17:28:23 +02:00
39820be5ff
Merge pull request #11703 from lafirest/fix/ldap_bind
...
fix(ldap): use the search result as bind target
2023-09-28 14:48:53 +02:00
1a13b2ac56
Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53
2023-09-28 13:53:03 +02:00
43ea367df8
fix(ldap): use the search result as bind target
2023-09-28 18:49:00 +08:00
98409c9b1e
Merge pull request #11702 from zmstone/0928-minor-chore
...
0928 minor chore
2023-09-28 12:39:55 +02:00
26cadb2248
fix(audit): disable audit log by default
2023-09-28 12:36:14 +02:00
b14425d865
fix(sso/saml): ensure lower case http header name
2023-09-28 11:40:21 +02:00
7ddcd71412
docs(ssl_dist.conf): update doc link
2023-09-28 11:16:18 +02:00
dca8fdb17f
fix(resource): respect the start_timeout
2023-09-28 16:36:41 +08:00
b77e5e880a
Merge pull request #11697 from ieQu1/dev/gen-rpc-3.1.1
...
chore(gen_rpc): Bump version to 3.1.1
2023-09-28 10:03:36 +02:00
6d2adfc259
chore(audit): emit the message field
2023-09-28 11:04:42 +08:00
859b122cdd
chore: format codes
2023-09-28 10:25:37 +08:00
b52e4ac99d
chore: update apps/emqx_machine/src/emqx_restricted_shell.erl
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-28 10:25:37 +08:00
bdf24d0ec8
chore: remove the `unauthorized` type for `from` field
2023-09-28 10:25:37 +08:00
71acf121ba
chore(audit): distinguish requests from rest_api or dashboard
2023-09-28 10:25:37 +08:00
dd9938114c
chore(audit): add from field
2023-09-28 10:25:37 +08:00
ece7d5b52a
Merge pull request #11690 from lafirest/fix/ldap_parse_cfg
...
fix(ldap): improve the LDAP `parse_config` function
2023-09-28 10:24:52 +08:00
b0d86eecd6
Merge pull request #11691 from lafirest/fix/sso_ssl
...
fix(sso): support for SSL update && ensure update is atomic
2023-09-28 10:22:34 +08:00
57781d0544
fix(ldap): remove the parse_config, it never work
2023-09-28 09:38:16 +08:00
9dee2dc31e
fix(sso): clear last error first before update && fix the `running`
2023-09-28 08:56:16 +08:00
0aa3ccdd65
feat(gen_rpc): Add schema for the TLS versions and cipher suites
2023-09-28 00:36:59 +02:00
afdda107af
fix(logger): json format log encode binary list as string array
2023-09-27 23:40:01 +02:00
6f7a4344dc
fix: do not gc sso saml SP singing keys
2023-09-27 23:19:39 +02:00
34367fc4ec
fix(audit_log): pretty print shell args
2023-09-27 23:19:39 +02:00
bb49914fd6
fix(sso): add convet_certs callback for sso backends
...
must convert certs in pre_config_update so the cert path refernces
are stored in raw config, otherwise the files might get gc:ed
2023-09-27 22:41:39 +02:00
45caa3bf01
fix(sso): make sp_private_key sensitive
...
so it will not be logged
2023-09-27 21:27:59 +02:00
c8cbbff044
fix(logger): no need for special handling of empty string
...
when formating json logs, there is no need to handle empty strings
special, already covered by unicode handling
2023-09-27 21:26:47 +02:00
3c37f19105
chore(gen_rpc): Bump version to 3.1.1
2023-09-27 21:09:00 +02:00
66d2107007
fix(sso): refactor update logic
2023-09-28 00:09:09 +08:00
bd3277c51b
Merge pull request #11683 from zmstone/0925-test-refactor-emqx-rpc-test
...
test: refactor emqx_rpc unit tests
2023-09-27 15:46:27 +02:00
08ad09a68f
fix(sso): refactor backen update logic
...
1. valid config always can update successfully
2. the `running` endpoint only return successfully created backend
3. enhancement of the `/sso` endpoint, and will check is the resource online
2023-09-27 20:53:10 +08:00
ea8d54fd8b
test: ensure atom exists in test module
2023-09-27 12:58:06 +02:00
6e8c73258f
test: refactor emqx_rpc unit tests
2023-09-27 12:58:06 +02:00
c9194cd6b2
fix(saml_sso): donot load IDP metadata when disabling saml
2023-09-27 18:46:24 +08:00
af9e87c025
fix: saml callback should check saml state
2023-09-27 18:34:48 +08:00
b2699c687b
fix(sso): support for SSL update && ensure update is atomic
...
1. support update SSL key and cert files
2. increase connection timeout
3. ensure the update is atomicity, everything will be consistent
2023-09-27 15:48:11 +08:00
1d0e789e4d
Merge pull request #11679 from keynslug/ft/EMQX-10942/custom-session-timers
...
feat(session): add custom session timers mechanism
2023-09-26 19:09:53 +03:00
0c33df3912
fix(ldap): improve the LDAP `parse_config` function
2023-09-26 23:28:37 +08:00
3fa18d6935
fix: lower sso saml redirect http header
2023-09-26 23:25:59 +08:00
6a9bb7c3ae
Merge pull request #11681 from zhongwencool/audit-log-fix-2
...
fix: don't need to change audit log's level
2023-09-26 23:12:28 +08:00
ac5eb5bc29
Merge pull request #11687 from lafirest/fix/sso_timeout
...
fix(sso): Handle backend update timeout and fix create errors
2023-09-26 22:43:38 +08:00
1a96a5990b
chore: Update apps/emqx_enterprise/src/emqx_enterprise_schema.erl
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 21:44:40 +08:00
403714d44e
fix(sso): Handle backend update timeout and fix create errors
...
1. correctly handle the timeout when call update on a backend
2. fix that config update always returns success
3. do not ignore start failures and ensure start is atomic
2023-09-26 21:43:30 +08:00
aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
...
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
bce2142df8
test(session): add custom session timers testcase
2023-09-26 16:22:06 +03:00
4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
...
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
fdcd73e20c
fix(file_logger): change file logger type depending on rotation size
...
Fixes https://emqx.atlassian.net/browse/EMQX-11036
From `logger_disk_log_h:open_disk_log`:
```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
case filelib:ensure_dir(File) of
ok ->
Size =
if Type==halt -> MaxNoBytes;
Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
end,
Opts = [{name, Name},
{file, File},
{size, Size},
{type, Type},
{linkto, self()},
{repair, false},
{format, external},
{notify, true},
{quiet, true},
{mode, read_write}],
case disk_log:open(Opts) of
```
Affects all file loggers (audit included):
```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
{badarg,
[{size,
{infinity,
10}},
{type,
wrap},
{linkto,
<0.1952.0>},
{repair,
false},
{format,
external},
{notify,
true},
{quiet,
true},
{mode,
read_write}]}}
```
2023-09-26 09:04:44 -03:00
14983ec14a
chore(hooks): validate hookpoints and document hook callbacks
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
5e6397b843
chore: remove result field for CLI audit logs
2023-09-26 19:24:35 +08:00
a73c3b8e1e
Merge remote-tracking branch 'ce/release-53' into audit-log-fix-2
2023-09-26 19:05:47 +08:00
e9785a6863
Merge pull request #11680 from thalesmg/audit-not-on-ce-r53-20230925
...
fix(audit): only support audit log on enterprise edition
2023-09-26 18:57:22 +08:00
7d810c2107
test: fix test case match pattern
2023-09-26 11:40:30 +02:00
d31bfc70fb
Merge pull request #11659 from zhongwencool/fix-listener-ssl-create-500
...
fix: create ssl listener return 500 crash
2023-09-26 11:38:07 +02:00
8cc626d33f
Merge pull request #11686 from lafirest/fix/sso_user_backend
...
fix: fix update SSO user && improve SSO user deletion commnad
2023-09-26 16:04:51 +08:00
56917fee5f
fix(sso): Improved SSO user deletion command
2023-09-26 14:15:41 +08:00
d1af81c86e
fix(sso): fix the failure to update SSO users
2023-09-26 12:14:45 +08:00
8135e27586
fix: redact sensitive data in SSO and LDAP
2023-09-26 11:06:47 +08:00
17206f8c75
Merge pull request #11662 from thalesmg/port-scan-check-proto-dist-m-20230922
...
chore: check ekka proto dist module type when resolving node address
2023-09-25 17:01:43 -03:00
5d212e1086
fix(audit): only support audit log on enterprise edition
...
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
0498e59c45
test: fix flaky test
2023-09-25 13:35:17 -03:00
95060302fd
fix: don't need to change audit log's level
2023-09-26 00:28:47 +08:00
ff7f37ccf5
test(cth): allow defining schema to load for app
2023-09-25 13:22:41 -03:00
b1f144ab8b
feat(session): add custom session timers mechanism
...
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
...
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
5e6996dc05
refactor: log formatter format mfa+line as m:f/a(line)
...
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
6a557980e4
refactor(logger): ensure JSON log field order
...
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
5f45ba50ff
refactor: delete log formatter config for audit log handler
...
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
1fed38c248
fix(logger): write 'json' format logs as JSON
2023-09-25 16:55:22 +02:00
aea1e80290
feat: add 'format' as alias for log 'formatter'
2023-09-25 16:55:22 +02:00
9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify
2023-09-25 15:48:01 +02:00
806017ef90
chore: check ekka proto dist module type when resolving node address
...
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917
Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
cfdb25b213
fix(sso): updates the SSO backend when the `[dashboard]` has updated
2023-09-25 20:30:47 +08:00
f1bc3b68b6
Merge pull request #11674 from id/0925-e5.3.0-alpha.2
...
e5.3.0 alpha.2
2023-09-25 14:21:46 +02:00
6f29bbf997
Merge pull request #11673 from HJianBo/redact_audit_log_from_cli
...
Redact audit log from CLI
2023-09-25 20:03:45 +08:00
81cf619f07
fix(ftconf): also mark `secret_access_key` key as sensitive
2023-09-25 14:52:57 +03:00
36d3a3a524
fix: bump to quicer 0.0.201
...
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
22193d273a
fix(quic): ignore undefined cacertfile
2023-09-25 12:57:55 +02:00
8c93c79b44
chore: e5.3.0-alpha.2
2023-09-25 12:52:45 +02:00
7d58f6c61e
Merge pull request #11668 from JimMoen/saml-login-redirect
...
fix: saml login acs redirect to dashboard overview
2023-09-25 18:11:03 +08:00
b970a34ee0
chore: redact some audit logs from CLI
2023-09-25 18:06:39 +08:00
89e15e9134
Merge pull request #11665 from HJianBo/typo-fixes
...
Typo fixes
2023-09-25 17:23:40 +08:00
1cd9df6461
Merge pull request #11669 from lafirest/fix/dashboard_cli
...
fix(sso): refactor the `admins` CLI
2023-09-25 16:57:18 +08:00
e63d484632
fix(sso): move the config path of the SSO feature to `dashboard.sso`
2023-09-25 15:23:52 +08:00
e1f1c64ecd
Merge branch 'release-53' into 0925-sync-r53-to-master
2023-09-25 08:12:45 +02:00
e94192d1fa
fix(sso): refactor the `admins` CLI
...
1. revert the opensource version
2. allow delete the SSO user via CLI
2023-09-25 13:53:01 +08:00
4f4868a46c
fix(sso): Disable access to `logout` endpoint by the `API key`
2023-09-25 10:31:38 +08:00
30862a94c6
chore: typo fixes
2023-09-24 20:19:05 +08:00
ad4fadc2fa
fix: saml login acs redirect to dashboard overview
2023-09-23 17:29:02 +08:00
f8d06614c0
chore: fix dialyzer warnings
2023-09-23 07:34:04 +08:00
1dddccb448
fix(saml): cert files cleanup when destroy
2023-09-22 22:49:08 +08:00
80a6c1150d
fix(saml): saml login reply role `viewer` as default
2023-09-22 22:39:02 +08:00
cc3e4e4dc5
fix(saml): drop cert and key content and return path
2023-09-22 22:37:04 +08:00
6349cd3910
fix(saml): sp sign request
2023-09-22 21:57:50 +08:00
2a8f3f9eaa
fix: saml xml metedata format
2023-09-22 21:50:43 +08:00
a318ad486a
refactor: behavior login/2 use all http request
2023-09-22 21:50:43 +08:00
9181ec844f
chore: split out sso_saml_api module
2023-09-22 21:50:43 +08:00
df94426ee3
chore: make static_check happy
2023-09-22 21:50:42 +08:00
ec0894ca0b
chore: update esaml vsn
2023-09-22 21:50:42 +08:00
4a26f63bd6
chore: fix bugs
2023-09-22 21:50:41 +08:00
47badc3181
chore: make dialyzer happy
2023-09-22 21:49:12 +08:00
1c78c6bf6d
chore: fix 500 crashes when backend not existed
2023-09-22 21:49:12 +08:00
b4fb5196cb
fix(sso): SSO management API 500
2023-09-22 21:49:12 +08:00
bba5cc44a8
fix: keep same API path style
2023-09-22 21:49:12 +08:00
8300cd42d4
fix: acl url ignore auth check
2023-09-22 21:49:12 +08:00
44836ef5ee
chore: bump esaml vsn to v1.1.1
2023-09-22 21:49:12 +08:00
13666fa9f9
refactor: avoid dynamic call
2023-09-22 21:49:12 +08:00
c9e0d4fc30
feat: saml integration for dashboard sso
2023-09-22 21:49:10 +08:00
d9466eef63
chore: fix Dashboard RBAC license and rebar.config
2023-09-22 21:19:45 +08:00
fd932c9bf7
Merge pull request #11660 from zmstone/0922-fix-audit-eval-command
...
fix(audit): make emqx eval command auditable
2023-09-22 14:02:47 +02:00
a34ab19d93
fix(audit): make emqx eval command auditable
2023-09-22 12:05:45 +02:00
0eed01abee
fix: create ssl listener return 500 crash
2023-09-22 17:34:19 +08:00
13b5e4dbc9
Merge pull request #11658 from lafirest/fix/sso_misc
...
Fix/sso misc
2023-09-22 14:44:39 +08:00
7286d15ca6
chore(sso): adjust the schema of the SSO LDAP backend
2023-09-22 13:48:20 +08:00
9e55ae240a
feat(sso): add `role` into the result of login endpoints
2023-09-22 13:48:07 +08:00
681e57dee6
fix(RBAC): allow read-only users to logout
2023-09-22 11:06:24 +08:00
d6935b6a67
feat: add port scan diagnostics to mria waiting for tables checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-10944
Also updates ekka -> 0.15.15, mria -> 0.6.4
How to test
===========
1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.
Or, more easily:
1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.
Example output
==============
```
Check check_open_ports should get ok but got #{msg =>
"some ports are unreachable",
results =>
#{'emqx@172.100.239.4' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
4}],
status =>
bad_ports},
'emqx@172.100.239.5' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
5}],
status =>
bad_ports}}}
```
After one node is back:
```
Check check_open_ports should get ok but got #{msg =>
"some ports are unreachable",
results =>
#{'emqx@172.100.239.4' =>
#{ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
4}],
status => ok},
'emqx@172.100.239.5' =>
#{open_ports =>
#{4370 => false,
5370 =>
false},
ports_to_check =>
[4370,5370],
resolved_ips =>
[{172,100,239,
5}],
status =>
bad_ports}}}
```
2023-09-21 14:29:01 -03:00
7cf60c5a91
chore: e5.3.0-alpha.1
2023-09-21 19:29:00 +02:00
acf4227fc6
test(session): fix quic testgroup in persistent session suite
...
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
76f614e9c5
Merge pull request #11653 from thalesmg/fix-dash-rbac-dialyzer-error-m-20230921
...
chore: fix dialyzer errors on ce version
2023-09-21 13:06:09 -03:00
85a8c174d9
chore: fix dialyzer errors on ce version
...
```
apps/emqx_dashboard/src/emqx_dashboard.erl
Line 225 Column 17: The pattern {'error', 'unauthorized_role'} can never match the type {'error','not_found' | 'token_timeout'} | {'ok',binary()}
```
2023-09-21 11:07:31 -03:00
b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
...
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log
2023-09-21 18:39:23 +08:00
0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
...
sync r52 to master
2023-09-21 12:31:24 +02:00
3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
...
quicer 0.0.200
2023-09-21 12:20:14 +02:00
dafd7c6085
chore: bump apps versions
2023-09-21 10:58:42 +02:00
21bb209fb1
feat: added support for auditing API and CLI activity in logs
2023-09-21 16:35:46 +08:00
105bebc250
chore: merge release-52 into master
2023-09-21 10:22:47 +02:00
Zaiming (Stone) Shi