Commit Graph

11226 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 25d17ae68d test(ft/s3_uploader): wrong test case was put to wrong group 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi b7531839cd fix(dialyzer): delete stale code 2023-12-14 22:20:01 +01:00
Zaiming (Stone) Shi bcae3d67c7
Merge pull request #12169 from ieQu1/dev/bpapi-separate-for-otp26
fix(static_checks): Prepare for OTP26
2023-12-14 20:32:45 +01:00
JimMoen da537a794b
fix(gw_ocpp): handle subscriptions call 2023-12-15 01:57:09 +08:00
Thales Macedo Garitezi 57a8f2351d feat(rule_engine): also bump `unknown` counter for unrecoverable action errors
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-12-14 13:42:28 -03:00
Thales Macedo Garitezi d5d05a1701
Merge pull request #12162 from thalesmg/fix-lookup-timeout-r54-20231213
fix(connector_api): avoid calling resource process to get channels
2023-12-14 13:36:26 -03:00
Zaiming (Stone) Shi c22a3686ae fix(emqx_durable_storage): fix type specs 2023-12-14 17:35:32 +01:00
Thales Macedo Garitezi a5d848515b fix: project sub-fields in action/connector `resource_opts` for some bridges
Fixes https://emqx.atlassian.net/browse/EMQX-11589
2023-12-14 12:09:32 -03:00
Zaiming (Stone) Shi c1f2287b86 Merge remote-tracking branch 'origin/release-54' 2023-12-14 15:26:49 +01:00
ieQu1 02e2231734 fix(static_checks): Prepare for OTP26 2023-12-14 15:22:09 +01:00
Thales Macedo Garitezi dbc8141930
Merge pull request #12125 from thalesmg/ds-cache-m-20231206
chore(ds): return DS key from `next` and add `update_iterator` callback
2023-12-14 10:23:25 -03:00
Thales Macedo Garitezi a7fe5da8d6 fix(connector_api): avoid calling resource process to get channels
Fixes https://emqx.atlassian.net/browse/EMQX-11586
2023-12-14 10:22:46 -03:00
Thales Macedo Garitezi ecd5c288e5
Merge pull request #12164 from zhongwencool/flaky-audit-max-size-test
test: audit max_size flaky test
2023-12-14 10:21:28 -03:00
lafirest e9ee2a669f
Merge pull request #12160 from lafirest/fix/sso_ldap
fix(sso): Correctly handle wrapped passwords when updating
2023-12-14 11:56:24 +08:00
firest f1bde41bf6 fix(sso): Correctly handle wrapped passwords when updating 2023-12-14 10:07:04 +08:00
zhongwencool b35765708d test: audit max_size flaky test 2023-12-14 09:48:17 +08:00
firest d278d3afb5 feat(ds): implemented the replayer scheduler 2023-12-13 23:52:02 +08:00
zhongwencool b46438efc9
Merge pull request #12161 from zhongwencool/config-update-error-text
fix: ensure config update error text is readable
2023-12-13 21:03:25 +08:00
Thales Macedo Garitezi edec431a6c
Merge pull request #12144 from thalesmg/fix-conn-schema-query-mode-r54-20231211
fix({action,connector}_schema): fix `resource_opts` subfields for connectors and actions, and remove redundant subfields from actions, use query mode from actions
2023-12-13 09:19:46 -03:00
zhongwencool d8796ea985 test: flaky audit log test 2023-12-13 18:01:47 +08:00
zhongwencool 96d21d4dbe fix: ensure config update error text is readable 2023-12-13 17:07:28 +08:00
zhongwencool 60d9c1113d fix: audit create_at/duration_ms filter not working 2023-12-13 12:16:52 +08:00
zhongwencool af3bbbbd44
Merge pull request #12153 from zhongwencool/http-headers-allow-delete
feat: don't merge default headers if user already setting
2023-12-13 09:58:22 +08:00
Shawn 61538d96b6 chore: update eredis_cluster to 0.8.3 2023-12-13 09:39:15 +08:00
zhongwencool ccb947a0a1
Merge pull request #12152 from zhongwencool/license-watermark
fix: license watermark percent support float
2023-12-13 09:14:06 +08:00
zhongwencool 8e34d185ac
Merge pull request #12149 from zhongwencool/audit-boots-log
fix: incorrect behavior in audit logs during stop and start of EMQX
2023-12-13 09:13:51 +08:00
zhongwencool a1cd1f7c63
Merge pull request #12148 from zhongwencool/bridge-http-action-without-path
fix: don't require path in http action(fill default path)
2023-12-13 09:13:34 +08:00
zhongwencool 3c43db2828
Merge pull request #12147 from zhongwencool/bridge-description
fix:add descriptions fields to bridge_v1
2023-12-13 09:13:17 +08:00
zhongwencool c73b371a7a feat: don't merge default headers if user already setting one 2023-12-13 08:47:55 +08:00
Thales Macedo Garitezi 7f0cbcc323
Merge pull request #12155 from thalesmg/fix-conn-inconsistent-status-r54-20231212
fix(connector_api): return status reason when status is inconsistent
2023-12-12 17:23:22 -03:00
Thales Macedo Garitezi 4a71aa58ce fix(connector_api): return status reason when status is inconsistent
Fixes https://emqx.atlassian.net/browse/EMQX-11581
2023-12-12 11:29:34 -03:00
Thales Macedo Garitezi 6e49cce055 feat(connector_api): list disabled channels too
Fixes https://emqx.atlassian.net/browse/EMQX-11583
2023-12-12 11:20:33 -03:00
Thales Macedo Garitezi d7b96af893
Merge pull request #12145 from thalesmg/fix-postgres-res-opts-schema-r54-20231211
fix(postgres_schema): fix usages of `resource_opts` in connector and action schemas
2023-12-12 11:02:55 -03:00
Serge Tupchii 3c3452c1dd fix(emqx_otel_schema): add `enable` field to SSL opts 2023-12-12 13:07:08 +02:00
Serge Tupchii 36b12a01c9 chore: fix typos in comments 2023-12-12 13:06:20 +02:00
Serge Tupchii f529bda8b7 fix(opentelemetry_schema): decrease scheduled_delay/interval importance to medium
These fields should be shown in API schema.
2023-12-12 12:07:59 +02:00
Serge Tupchii ea6f5764bb test: remove debug message 2023-12-12 12:05:46 +02:00
zhongwencool 1807df157c fix: license watermark percent support float 2023-12-12 17:27:31 +08:00
Stefan Strigler 71011ad084
Merge pull request #12142 from sstrigler/EMQX-11487-e-5-4-0-mongodb-test-connection-failed
mongodb test connection failed
2023-12-12 09:05:04 +01:00
zhongwencool e2e7f96515 fix: incorrect behavior in audit logs during stop and start of EMQX 2023-12-12 14:29:39 +08:00
zhongwencool 363055a32e fix: add descriptions fields to bridge_v1 2023-12-12 10:18:18 +08:00
zhongwencool cd1365d753 fix: don't require path in http action(fill default path) 2023-12-12 10:15:53 +08:00
Thales Macedo Garitezi b80c9b0863 fix(actions_schema): remove redundant `resource_opts` subfields for actions
Buffer workers don't use those fields.
2023-12-11 16:58:43 -03:00
Thales Macedo Garitezi 30719d286a fix(http_bridge_schema): use correct `resource_opts` subfields for connector and action 2023-12-11 16:58:43 -03:00
Thales Macedo Garitezi 593283df93 fix(redis_bridge_schema): use correct `resource_opts` subfields for connector and action 2023-12-11 16:58:43 -03:00
Thales Macedo Garitezi 4fc14fb4ba
Merge pull request #12143 from thalesmg/fix-reply-to-async-r54-20231211
fix(actions): increment rule statistics even if channel is not installed
2023-12-11 15:35:35 -03:00
Thales Macedo Garitezi f7adaa8911 fix(postgres_schema): fix usages of `resource_opts` in connector and action schemas
Fixes https://emqx.atlassian.net/browse/EMQX-11569
2023-12-11 15:03:20 -03:00
Thales Macedo Garitezi fbbb55633d fix(connector_schema): remove `query_mode` from `resource_opts`
The connector query mode is inferred during its creation, and later it must be overridden
by an action, anyway.
2023-12-11 14:13:40 -03:00
Thales Macedo Garitezi b4a5c141ad fix(actions): use action query mode instead of connector's query mode 2023-12-11 14:10:51 -03:00
Thales Macedo Garitezi 2495f59c91 fix(actions): increment rule statistics even if channel is not installed
Fixes new bug posted after https://emqx.atlassian.net/browse/EMQX-11494 was already fixed.

Also reduces the usage of error throwing for flow control a bit.
2023-12-11 14:01:30 -03:00
Stefan Strigler a7c344ba29 fix(emqx_bridge): remove unused 2023-12-11 14:58:09 +01:00
Stefan Strigler 8e9cb6a928 style(emqx_bridge_mysql): minor cleanup 2023-12-11 14:54:10 +01:00
Stefan Strigler 31a7301f6f refactor(emqx_bridge): be more generic in to_bridge_v1_config 2023-12-11 14:50:38 +01:00
JimMoen 6552e0523a
Merge pull request #12141 from JimMoen/fix-shared-sub-api-internal-error
Fix shared sub api internal error
2023-12-11 21:43:09 +08:00
Stefan Strigler be31486983 fix(emqx_utils): use deep_merge in unindent 2023-12-11 14:40:24 +01:00
Stefan Strigler 6143b3a2be fix(emqx_bridge_mongodb): use correct config when calling downgrade_type
Also limit result in mongodb ..._to_bridge_v1_config to not include more fields
than what's in the schema (most notably `description`).
2023-12-11 14:08:45 +01:00
Stefan Strigler 5d7ae4b980
Merge pull request #12132 from sstrigler/EMQX-11154-bridge-v-2-my-sql-support
feat(emqx_bridge_mysql): port to shared connectors
2023-12-11 13:56:34 +01:00
Zaiming (Stone) Shi b6c11be159
Merge pull request #12138 from zmstone/1208-fix-dialyzer
fix(dialyzer): batch 3
2023-12-11 12:52:56 +01:00
zhongwencool 6ce2d96ae3
Merge pull request #12139 from zhongwencool/prometheus-api-get
fix: only return recommend setting when get prometheus api
2023-12-11 17:29:36 +08:00
zhongwencool 64c843088a
Merge pull request #12140 from zhongwencool/redis-schema-example
chore: update redis bridge example
2023-12-11 17:29:10 +08:00
lafirest 11622031c5
Merge pull request #12133 from lafirest/fix/ldap
fix(authn_ldap): Improve the type inference of method union
2023-12-11 17:05:51 +08:00
JianBo He 5b70baf624
Merge pull request #12121 from zhongwencool/cluster-rpc
fix: occasionally return stale view when updating configurations
2023-12-11 16:59:17 +08:00
Zaiming (Stone) Shi d560366c14 test: fix some compile warnings 2023-12-11 09:43:13 +01:00
JimMoen 8c25371ff3
chore: bump gitignore for emqx standalone common test SUITES 2023-12-11 15:37:36 +08:00
JimMoen 7b60928288
fix(api): endpoint `/v5/topics` with invalid topic filter 2023-12-11 15:36:50 +08:00
zhongwencool 0595db3911 chore: update redis bridge example 2023-12-11 14:55:04 +08:00
zhongwencool 525a9e0adf fix: only return recommend setting when get prometheus api 2023-12-11 14:49:32 +08:00
JimMoen 47901c9fed
fix(exhook): `client.authorize` hook always uses real-topic
See: emqx_channel:do_check_sub_authzs/3, line: 1895
2023-12-11 14:11:18 +08:00
JimMoen cfe3b2dcee
feat(exhook): subopts in on_client_subscribe/on_client_unsubscribe 2023-12-11 14:11:18 +08:00
JimMoen 46201a8796
feat(exhook): provide the `peerport` field
- both in `ConnInfo` and `ClientInfo`
2023-12-11 14:11:18 +08:00
JimMoen 68da627b4d
feat(channel): add peerport field in ClientInfo 2023-12-11 14:11:18 +08:00
JianBo He b34e66bdf7
Merge pull request #12134 from lafirest/fix/gbt
fix(gbt): Add unimplemented command call
2023-12-11 13:46:07 +08:00
Zaiming (Stone) Shi 50f4aba5cd fix(dialyzer): batch 3 2023-12-09 15:50:09 +01:00
SergeTupchiy 107efda78f
Merge pull request #12136 from SergeTupchiy/upgrade-otel-1.4.7
Upgrade otel 1.4.7
2023-12-08 22:08:01 +02:00
Zaiming (Stone) Shi 5ad46ce077
Merge pull request #12129 from zmstone/1208-default-license-conn-25
refactor: update default enterprise license
2023-12-08 21:01:04 +01:00
Zaiming (Stone) Shi 1212ccc519
Merge pull request #12137 from zmstone/1207-fix-dialyzer
fix(dialyzer): batch 2
2023-12-08 20:31:17 +01:00
Thales Macedo Garitezi 66d043becd feat(ds): introduce `update_iterator` callback 2023-12-08 15:04:18 -03:00
Zaiming (Stone) Shi ddbb8560fa fix(dialyzer): batch 2 2023-12-08 17:59:55 +01:00
Serge Tupchii b5de58df2d chore: upgrade opentelemetry lib to 1.4.7-emqx 2023-12-08 18:44:19 +02:00
Stefan Strigler 71607aa2ad feat(emqx_bridge_mysql): port to shared connectors 2023-12-08 16:15:18 +01:00
Andrew Mayorov 76970c268e
fix(sessds): stop overwriting QoS0-only pubrange checkpoints
Through making `StreamRef` part of a record key. Otherwise with
QoS0-only streams the seqno doesn't change, thus ranges over
different streams will have the same `FirstSeqno`.
2023-12-08 14:33:47 +01:00
Andrew Mayorov 24710d0f56
fix(sessds): avoid accumulating QoS0-only ranges in memory 2023-12-08 14:31:18 +01:00
firest 1a8b59045b fix(gbt): Add unimplemented command call 2023-12-08 20:55:20 +08:00
firest f4f45cf634 fix(authn_ldap): Improve the type inference of method union 2023-12-08 19:26:29 +08:00
Kjell Winblad 0c80c74554 fix: remove unnecessary prepare_statement from pgsql action 2023-12-08 09:59:19 +01:00
Zaiming (Stone) Shi 41679808fe chore: add version number in license apply link 2023-12-08 08:35:58 +01:00
Zaiming (Stone) Shi e5cf9e40ac refactor: update default enterprise license 2023-12-08 08:25:20 +01:00
firest e57d796df5 fix(ldap): Adjust logger level for authentication failures 2023-12-08 12:47:32 +08:00
zhongwencool 45c4fb33a8 fix: occasionally return stale view when updating configurations on different nodes concurrently 2023-12-08 09:56:31 +08:00
Thales Macedo Garitezi f37c86afb9 Merge remote-tracking branch 'origin/release-54' into sync-r54-m-20231207 2023-12-07 13:30:04 -03:00
Thales Macedo Garitezi 2a6d72878f chore(ds): return DS message key along with batch 2023-12-07 11:36:08 -03:00
JimMoen 17e6703ba2
Merge pull request #12120 from qzhuyan/dev/william/bump-new-quicer
chore: bump emqtt 1.10.1 and quicer 0.0.307
2023-12-07 22:10:43 +08:00
William Yang 499f4d4bc5 chore(quicer): dialyzer 2023-12-07 13:52:48 +01:00
William Yang 583645d97b chore: bump emqtt 1.10.1 and quicer 0.0.308 2023-12-07 13:52:44 +01:00
Thales Macedo Garitezi c45ed7b888 fix(connectors_api): fill default raw config values before returning
Fixes https://emqx.atlassian.net/browse/EMQX-11562
2023-12-07 09:44:48 -03:00
Thales Macedo Garitezi c274450c5e
Merge pull request #12118 from thalesmg/fix-missing-type-name-api-actions-r54-20231206
fix(actions_api): add missing fields to API spec and returned data
2023-12-07 09:34:29 -03:00
Zaiming (Stone) Shi 7e3ed05f87
Merge pull request #12115 from zmstone/1206-fix-dialyzer
1206 fix dialyzer
2023-12-07 12:24:47 +01:00
JianBo He bd985e52d7
Merge pull request #12072 from killme2008/feature/greptimedb-async-write
feat: implements async query mode for GreptimeDB data bridge
2023-12-07 19:21:44 +08:00
ieQu1 9b612cb0e7
Merge pull request #12119 from ieQu1/EMQX-10342
Make stream renew timer configurable
2023-12-07 10:42:23 +01:00
zhongwencool 253ea6093a
Merge pull request #12111 from zhongwencool/fix-token-not-found
fix: use sync_transaction when update admin/token
2023-12-07 09:56:20 +08:00
zhongwencool 850c6027d4
Merge pull request #12113 from zhongwencool/log-fix
fix: audit log format
2023-12-07 09:56:01 +08:00
Zaiming (Stone) Shi 7560016f0e fix(hocon): upgrade hocon from 0.40.0 to 0.40.1
fixed type spec, exported some public types from emqx_schema module
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 33a7282cdd fix(dialyzer): only include eunit when TEST is defined 2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 9a6056ff2f refactor(emqx_machine): move system_monitor to included_applications
system_monitor has to be found in application (or included_applications)
otherwise dialyzer fails with Unknown function

Adding to 'applications' would cause a deadlock because one
cannot call application:ensure_all_started in the application/2
callback.
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 6b553e37c0 chore: pin ekka 0.17.0 previously 0.15.16
change summary:
1. otp 26
2. type spec fixes
3. dropped UDP mcast clustering
2023-12-06 20:39:26 +01:00
Zaiming (Stone) Shi 423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Zaiming (Stone) Shi 6affda8194 refactor(emqx_gateway_cm): no need to keep locker pid in state
For two reasons
1. Every other places computes the process register name from gwname
2. The ekka_locker api spec only accepts atom but not pid
2023-12-06 19:20:24 +01:00
Zaiming (Stone) Shi 71f9838fd7 chore: upgrade to emqtt 1.10.0 2023-12-06 19:18:15 +01:00
Zaiming (Stone) Shi 83206daa20 refactor(emqx_cm_locker): delete dead code 2023-12-06 19:18:15 +01:00
Zaiming (Stone) Shi de61d9d609 chore: upgrade to jiffy from 1.0.5 to 1.0.6
1.0.5 has an unexported type which causes dialyzer to fail on otp 26
2023-12-06 19:18:15 +01:00
William Yang 64401cb6cc
Merge pull request #12042 from qzhuyan/dev/william/rel54/esockd-robustnes
fix: bump to esockd 8.9.9  improve robustness
2023-12-06 17:39:38 +01:00
Andrew Mayorov 074dc8ce11
test(emqx): switch rest of test suite to use `emqx_cth_suite` 2023-12-06 16:47:04 +01:00
ieQu1 371ec34992 feat(sessds): Make stream renew interval configurable 2023-12-06 16:04:53 +01:00
ieQu1 c43b3eb535 fix(sessds): Add debug logs for the session garbage collection 2023-12-06 15:37:23 +01:00
Thales Macedo Garitezi fdeedc360e fix(actions_api): add missing fields to `POST` API schema 2023-12-06 11:13:09 -03:00
Thales Macedo Garitezi 2757b95850 fix(actions_api): fill raw config defaults before returning
Fixes https://emqx.atlassian.net/browse/EMQX-11541
2023-12-06 11:13:09 -03:00
Andrew Mayorov c2c9de69b4
test(emqx): switch `emqx_crl_cache_SUITE` to use `emqx_cth_suite` 2023-12-06 14:34:41 +01:00
Andrew Mayorov f06a1f10ef
test(emqx): switch most of testsuites to use `emqx_cth_suite`
For increased robustness and isolation.
2023-12-06 14:34:41 +01:00
Andrew Mayorov d643330cb9
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-06 14:34:40 +01:00
Andrew Mayorov 7d80511b3a
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-06 14:34:40 +01:00
Thales Macedo Garitezi 8a5b0d252d
Merge pull request #12110 from thalesmg/test-plugin-flaky-r54-20231205
test(cth_peer): use an exclusive current dir for each peer
2023-12-06 10:16:13 -03:00
Thales Macedo Garitezi 17b6143c30
Merge pull request #12102 from thalesmg/nit-postgres-log-m-20231205
chore(postgres): prettify logged errors
2023-12-06 10:10:02 -03:00
William Yang c4956d99d1 fix: bump to esockd 8.9.9 improve listener robustness 2023-12-06 13:55:13 +01:00
Thales Macedo Garitezi b82189fb4d test(cth_peer): use an exclusive current dir for each peer 2023-12-06 09:35:13 -03:00
Andrew Mayorov 43813e29c2
chore(emqx): drop legacy session persistence impl leftovers 2023-12-06 10:53:57 +01:00
SergeTupchiy 28ff53e99c
Merge pull request #11984 from SergeTupchiy/EMQX-10535-openetelmetry-tracing-new
Integrate OpenTelmetry tracing
2023-12-06 11:45:41 +02:00
Zaiming (Stone) Shi d945a26c06
Merge pull request #12112 from zmstone/1206-refactor-delete-mcast-cluster-strategy
refactor: delete UDP multicast clustering strategy
2023-12-06 10:01:26 +01:00
zhongwencool 8965aa2a21 fix: audit log format 2023-12-06 16:58:04 +08:00
Andrew Mayorov 3798060543
Merge pull request #12109 from keynslug/test/emqx-cth-suite
test(emqx): switch select test suites to use `emqx_cth_suite`
2023-12-06 09:48:53 +01:00
Zaiming (Stone) Shi d0732aa2a6 refactor: delete UDP multicast clustering strategy 2023-12-06 08:39:08 +01:00
zhongwencool b015d08cba fix: use sync_transation when update admin/token 2023-12-06 09:04:27 +08:00
Thales Macedo Garitezi e03b8fd80e chore(postgres): prettify logged errors
Fixes https://emqx.atlassian.net/browse/EMQX-11490
2023-12-05 15:55:18 -03:00
Andrew Mayorov 83bea2254d
test(ocsp): switch test suite to use `emqx_cth_suite`
And simplify it slightly in the process.
2023-12-05 19:16:37 +01:00
Andrew Mayorov 6f5228e991
test(emqx): switch select test suites to use `emqx_cth_suite` 2023-12-05 19:16:37 +01:00
Thales Macedo Garitezi 99399c6e95
Merge pull request #12098 from thalesmg/revert-downgrade-bridge-id-m-20231205
chore: stop downgrading action id in rule API responses
2023-12-05 14:26:23 -03:00
Thales Macedo Garitezi f489de8860
Merge pull request #12100 from thalesmg/fix-password-action-probe-m-20231205
fix(bridges/actions api): correctly deobfuscate passwords when probing
2023-12-05 14:25:32 -03:00
Thales Macedo Garitezi 2579f8ea92
Merge pull request #12096 from thalesmg/fix-delete-deps-rules-m-20231204
fix(api): return list of dependent rule ids when trying to delete bridge/action
2023-12-05 14:24:50 -03:00
Serge Tupchii 938508b270 refactor(emqx_opentelemetry): pass only channel info to trace functions 2023-12-05 19:01:08 +02:00
Ivan Dyachkov f5e184ec24
Merge pull request #12101 from id/1205-scheduled-build-of-release54
ci: run scheduled packages build of release-54
2023-12-05 17:37:38 +01:00
Thales Macedo Garitezi f30c97f190 chore: bump app vsn 2023-12-05 13:26:02 -03:00
Thales Macedo Garitezi deb3fcd606 Merge remote-tracking branch 'origin/master' into sync-m-r54-20231205 2023-12-05 13:25:45 -03:00
Serge Tupchii 195a23ae27 chore(emqx_opentelemetry): bump opentelemetry to v1.4.6-emqx 2023-12-05 18:18:31 +02:00
Serge Tupchii 85441fda0f test(emqx_opentelemetry): add trace test suite 2023-12-05 18:18:31 +02:00
Serge Tupchii c8e69357cc feat(emqx_opentelemetry): use one global exporter config for all otel signals 2023-12-05 18:18:31 +02:00
Serge Tupchii 2a3f6b749c fix(emqx_opentelemetry): avoid using `application:ensure_all_started/1` for better deadlock safety 2023-12-05 18:18:31 +02:00
Serge Tupchii 8d3f98eff2 fix(emqx_opentelemetry): fix dialyzer warnings 2023-12-05 18:18:31 +02:00
Serge Tupchii 60da6427ca fix: move openetelmetry deps to emqx_openetelmetry app
This is to fix elixir build, as mix can't find included opentelemetry hrl files.
2023-12-05 18:18:31 +02:00
ieQu1 2a50ec3f7d
Merge pull request #12104 from ieQu1/dev/gcp-logs
fix(bridge_gcp): Don't print config to the logs
2023-12-05 17:15:11 +01:00
Serge Tupchii 7fdc650448 feat: integrate OpenTelemetry traces 2023-12-05 18:11:00 +02:00
Thales Macedo Garitezi e42f4155d3 fix(bridges/actions api): correctly deobfuscate passwords when probing
Fixes https://emqx.atlassian.net/browse/EMQX-11533
2023-12-05 11:49:50 -03:00
ieQu1 b2ca59dfa7 fix(bridge_gcp): Don't print config to the logs 2023-12-05 15:48:53 +01:00
Thales Macedo Garitezi e5a4be11f5 chore: stop downgrading action id in rule API responses
Fixes https://emqx.atlassian.net/browse/EMQX-11405
2023-12-05 11:48:46 -03:00
Thales Macedo Garitezi 1b68fbff05 fix(api): return list of dependent rule ids when trying to delete bridge/action
Fixes https://emqx.atlassian.net/browse/EMQX-11523
2023-12-05 11:47:26 -03:00
Andrew Mayorov 286d483a3a
Merge pull request #12055 from keynslug/ft/EMQX-11474/subopts
fix(sessds): respect subscription options when publishing
2023-12-05 15:45:25 +01:00
Ivan Dyachkov e995b3948e chore: bump emqx_gateway_coap version 2023-12-05 15:17:53 +01:00
Thales Macedo Garitezi b6e8f5a37c
Merge pull request #12095 from thalesmg/fix-rule-test-api-m-20231204
fix(rule_api): fix sql test when testing against events
2023-12-05 08:57:54 -03:00
SergeTupchiy 7b9d12f20e
Merge pull request #12048 from SergeTupchiy/fix-coap_sub-opts
Fix COAP sub opts
2023-12-05 10:27:04 +02:00
Thales Macedo Garitezi c1c7d1246f
Merge pull request #12084 from thalesmg/refactor-resource-status-state-m-20231201
refactor(resource_manager): use macros and better differentiate status from state
2023-12-04 17:55:27 -03:00
Thales Macedo Garitezi d594b38ceb
Merge pull request #12083 from thalesmg/fix-connector-bws-m-20231201
fix(connector): don't start buffer workers for the connector itself
2023-12-04 16:14:15 -03:00
Thales Macedo Garitezi 984e2ccc74
Merge pull request #12082 from thalesmg/test-ensure-res-opts-conn-m-20231201
add missing `resource_opts` fields to connectors schemas
2023-12-04 16:13:32 -03:00
Thales Macedo Garitezi cea857cb8e fix(rule_api): fix sql test when testing against events
Fixes https://emqx.atlassian.net/browse/EMQX-11449
2023-12-04 15:24:09 -03:00
Thales Macedo Garitezi b2cd8aaab7 refactor: use single clause
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi ac2c99f996 test(connector): add test to ensure connector schema has `resource_opts`
https://emqx.atlassian.net/browse/EMQX-11509
2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi bb04a2d89e fix(bridge_redis): add `resource_opts` to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 87a13aa316 fix(bridge_mongo): add `resource_opts` to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 7a1288e802 fix(bridge_postgres): add `resource_opts` to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 3078de6e97 fix(bridge_syskeeper): add `resource_opts` to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 5bdfac52b8 fix(bridge_kafka_producer): add missing `resource_opts` sub-fields to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 626bcc1c62 fix(bridge_http): add `resource_opts` to connector schema 2023-12-04 10:02:05 -03:00
Thales Macedo Garitezi 10b1a8bbda feat(connector): add some `resource_opts` schema facilities 2023-12-04 10:02:05 -03:00
Dennis Zhuang 9920841deb chore: upgrade greptimedb client to v0.1.6 to fix the failed test 2023-12-04 20:30:33 +08:00
Dennis Zhuang 1d6ea685ef fix: another try to fix t_write_failure 2023-12-04 20:30:33 +08:00
Dennis Zhuang 97592e2e40 fix: test case t_write_failure 2023-12-04 20:30:33 +08:00
Dennis Zhuang fbeb56e475 fix: test case t_write_failure 2023-12-04 20:30:33 +08:00
Dennis Zhuang 547acef01d feat: upgrade greptimedb client to v0.1.5 2023-12-04 20:30:33 +08:00
Dennis Zhuang a694f3c92d fix: tests 2023-12-04 20:30:33 +08:00
Dennis Zhuang 8bc874c261 feat: supports async mode for greptimedb data bridge 2023-12-04 20:30:33 +08:00
Dennis Zhuang 90d0081df4 feat: upgrade greptimedb and client 2023-12-04 20:30:33 +08:00
Andrew Mayorov 130a5a5442
fix(ds): pass topics to `emqx_topic:words/1` before feeding LTS tree
So that empty levels in topics will be properly mapped into `''` atoms.
2023-12-04 13:39:01 +03:00
Andrew Mayorov 92c4b29a4c
feat(topic): match empty topic levels more loosely
So that the result of `emqx_topic:tokens/1` would be perfectly
matchable with the result of `emqx_topic:words/1` of a topic filter
with empty levels.
2023-12-04 13:38:55 +03:00
Andrew Mayorov b5c485cd55
test(sessds): add empty level topic subscription testcase 2023-12-04 13:38:42 +03:00
Serge Tupchii e455433694 test(emqx_gateway): fix test failures caused by grpc-erl upgrade 2023-12-04 12:17:58 +02:00
JimMoen cb992fcf55
Merge pull request #12059 from JimMoen/1130-fix-vm-time-warp
fix: vm.args use `multi_time_warp`
2023-12-04 11:56:08 +08:00
Andrew Mayorov 627c58b07d
fix(sessds): handle `expire_awaiting_rel` common timer with no-op
Otherwise, the channel crashes when the timer is triggered.
2023-12-02 15:00:30 +03:00
Andrew Mayorov 47bc747323
fix(sessds): drop everything related to session on CleanStart 2023-12-02 15:00:30 +03:00
Andrew Mayorov ee9a98f0a4
chore: bump emqx_gateway_mqttsn to 0.1.7 2023-12-02 15:00:29 +03:00
Andrew Mayorov fd26e690b8
refactor(sessds): move parts of message processing to replayer
To simplify the processing flow, reducing the number of back-and-forth
between the session and the replayer.
2023-12-02 15:00:29 +03:00
Andrew Mayorov 29ec73847a
fix(utils): make `flattermap/2` results less variative 2023-12-02 14:38:04 +03:00
Andrew Mayorov 6255ee0833
fix(sessmem): avoid defining unsafe defaults in `#session` 2023-12-02 14:38:04 +03:00
Andrew Mayorov b5ad0f9815
fix(test): subscribe with QoS 2 to receive QoS 2 message 2023-12-02 14:38:04 +03:00
Andrew Mayorov 1948719eec
test(sessds): unskip testcase related to QoS2 handling 2023-12-02 14:38:04 +03:00
Andrew Mayorov 3265d2f2aa
fix(sessds): respect subscription options when publishing 2023-12-02 14:38:03 +03:00
Andrew Mayorov b5f39f89e3
feat(utils): add `flattermap/2` as slightly more generic `flatmap/2` 2023-12-02 14:36:21 +03:00
Andrew Mayorov 508346f095
refactor(topicidx): split persistent term stuff off gbt-based index 2023-12-02 14:36:21 +03:00
Andrew Mayorov 88103c5f0e
refactor(session): pass config to `SessionImpl:open/3` as well
* Anticipate that connection info may change during session takeover.
* Avoid persisting session conf as part of persistent session state.
2023-12-02 14:36:21 +03:00
Andrew Mayorov 6eba082992
Merge pull request #12075 from keynslug/fix/EMQX-11474/function-clause
fix(channel): expect there's no session yet on disconnect
2023-12-02 14:32:50 +03:00
Thales Macedo Garitezi dc5e3b939c refactor(resource_manager): use macros and better differentiate status from state
Internally in `emqx_resource_manager`, there seems to be many points where the
`gen_statem` states are conflated with resource status, since their names coincide.  While
that works for now, introducing a new `gen_statem` state, an internal state, shouldn't
necessarily imply a new, externally facing resource status.

Here we also introduce the usage of some macros to avoid the pitfalls of making a typo in
a state/status name.
2023-12-01 18:23:05 -03:00
Zaiming (Stone) Shi 55621fbb86
Merge pull request #12074 from zmstone/1201-user-peer-module-for-all
test: replace 'slave' and 'ct_slave' with 'peer'
2023-12-01 21:35:40 +01:00
Thales Macedo Garitezi 29ae45c39d fix(connector): don't start buffer workers for the connector itself
Fixes https://emqx.atlassian.net/browse/EMQX-11448
2023-12-01 17:20:38 -03:00
Zaiming (Stone) Shi 6e30718818 test: allow join cluster later 2023-12-01 20:33:39 +01:00
ieQu1 3d2e95fe3e
Merge pull request #12081 from ieQu1/dev/gen-rpc-3.3.0
chore(gen_rpc): Bump version to 3.3.0
2023-12-01 17:57:33 +01:00
ieQu1 476c300ecf
Merge pull request #12070 from ieQu1/dev/ds-min-max-batch-size
Various minor fixes
2023-12-01 17:57:15 +01:00
ieQu1 d3b7478d70 chore(gen_rpc): Bump version to 3.3.0 2023-12-01 17:10:12 +01:00
Zaiming (Stone) Shi ab7d57e1c5 test: delete debug level logging 2023-12-01 17:07:14 +01:00
Ivan Dyachkov 0e27d031a8 chore: 5.4.0-alpha.1 2023-12-01 16:54:03 +01:00
Zaiming (Stone) Shi 6a9cc20d4d test: wait for all nodes clustered 2023-12-01 16:37:39 +01:00
Zaiming (Stone) Shi 956b25cc4c
Merge pull request #12071 from zmstone/1201-sync-release-54
1201 sync release 54
2023-12-01 14:38:47 +01:00
Thales Macedo Garitezi cb60880bab
Merge pull request #12067 from thalesmg/fix-kafka-check-client-m-20231130
fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
2023-12-01 09:55:12 -03:00
Andrew Mayorov 6b17920fea
fix(channel): expect there's no session yet on disconnect 2023-12-01 15:29:05 +03:00
Andrew Mayorov cce474ce98
fix(session): include session term in stacktrace 2023-12-01 15:29:05 +03:00
Zaiming (Stone) Shi 1a78e7ae79 fix(connector): ensure webhook bridge convert to http connector 2023-12-01 11:56:37 +01:00
Zaiming (Stone) Shi 90b7486624 fix(redis): ensure schema namespace 2023-12-01 11:37:06 +01:00
ieQu1 e1ec560639 test(emqx): Fix flaky emqx_takeover_SUITE:t_takeover testcase
This testcase uses QoS1, so it must account for possible duplication
of the messages.
2023-12-01 10:06:18 +01:00
Zaiming (Stone) Shi 60d70b22dc chore: bump app vsn for emqx_auth_redis 2023-12-01 08:37:37 +01:00
Zaiming (Stone) Shi 7f4d91d490 Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54 2023-12-01 08:32:22 +01:00
ieQu1 e238602533 fix(ds): Update README 2023-12-01 08:27:05 +01:00
ieQu1 0ae618d010 fix(ds): Use emqx_rpc for calls that work with large binaries 2023-12-01 08:27:05 +01:00
ieQu1 0e625d814a feat(sessds): Make batch size configurable
This change affects flow control. It allows to configure maximum size
of a batch, as well as fetch threshold.
2023-12-01 08:27:05 +01:00
ieQu1 4717e56fb6 fix(sessds): Schedule poll immediately upon receiving an ack
This commit affects the flow control, and improves the throughput by
removing a delay between freeing up space in the in-flight window and
polling new messages.
2023-12-01 08:27:05 +01:00
ieQu1 38800c0260 refactor(sessds): Store timers in the session 2023-12-01 08:27:05 +01:00
Zaiming (Stone) Shi 22f7cc1622 test: replace 'slave' and 'ct_slave' with 'peer' 2023-12-01 08:07:09 +01:00
JimMoen ee25d9bd9e
fix: vm.args use `multi_time_warp`
See also:
https://www.erlang.org/doc/apps/erts/time_correction#multi-time-warp-mode
2023-12-01 14:52:31 +08:00
ieQu1 69f1ca43c3 fix(sessds): Create a timer even if it's present in the map 2023-12-01 06:11:13 +01:00
ieQu1 1897e5c31b refactor(session): Use common naming conventions 2023-12-01 06:11:08 +01:00
ieQu1 d9e7544070 refactor(sessds): Introduce macros for the timers 2023-12-01 06:11:08 +01:00
ieQu1 0d245acdc1
Merge pull request #12066 from keynslug/fix/sessds-tc-flaky
test(sessds): wait client disconnect propagates to broker
2023-12-01 00:13:38 +01:00
Thales Macedo Garitezi 113a4ad4b1
Merge pull request #12057 from thalesmg/ds-session-gc-m-20231128
feat(ds): add session gc process
2023-11-30 17:51:52 -03:00
Thales Macedo Garitezi 0388e1c1c4 fix(kafka_producer): add `resource_opts` to connector schema, and check for client connectivity
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-11-30 17:51:32 -03:00
Ivan Dyachkov 9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
Thales Macedo Garitezi 684d637fed test(bridge_api): workaround strange config syncing problem
For some unknown reason, this test has difficulties in syncing the config correctly
between the nodes, while the equivalent in bridge_v2_api_SUITE doesn't.
2023-11-30 19:51:35 +01:00
Ivan Dyachkov ec10c51073 Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53 2023-11-30 19:51:12 +01:00
Zaiming (Stone) Shi dcb1c0680b
Merge pull request #12053 from zmstone/1129-prepare-for-otp26
1129 prepare for OTP 26
2023-11-30 19:38:40 +01:00
Andrew Mayorov ec471c0557
test(sessds): wait client disconnect propagates to broker 2023-11-30 20:14:57 +03:00
Serge Tupchii f54bda7063 fix(emqx_mgmt_api_data_backup): add count and hasnext meta paging params 2023-11-30 19:10:12 +02:00
Zaiming (Stone) Shi 81e75cf068 test: fix bulk-kick test case flakyness 2023-11-30 17:30:25 +01:00
Zaiming (Stone) Shi 5427ebc5f1 fix: log error when failed to install plugin
the error return is a map with details.
prior to this change only the reason is returned in the api response,
now the error map is logged at error level to help troubleshooting
2023-11-30 16:49:30 +01:00
Ivan Dyachkov 6dd92c382f chore: 5.3.2 release 2023-11-30 16:18:44 +01:00
Thales Macedo Garitezi cf6cb3e4ad chore: set low importance to config 2023-11-30 11:55:57 -03:00
Thales Macedo Garitezi 1ab009f081 refactor: rename supervisor 2023-11-30 11:55:48 -03:00
Zaiming (Stone) Shi eed253af82 test: implement a new node restart helper func 2023-11-30 15:11:38 +01:00
Thales Macedo Garitezi 880f5e8f89 feat(ds): add session gc process
Fixes https://emqx.atlassian.net/browse/EMQX-9744
2023-11-30 09:38:59 -03:00
Zaiming (Stone) Shi cf72c04fdd test: fix peer node stop and plugin SUITE typo 2023-11-30 09:53:36 +01:00
Zaiming (Stone) Shi f2db4cc7fc chore: upgrade to gen_rpc 3.2.2 2023-11-30 09:42:44 +01:00
Zaiming (Stone) Shi 640b0df319 test: do not add -master erl flag for peer nodes 2023-11-30 08:49:57 +01:00
zhongwencool 773bd8ecb2
Merge pull request #12047 from thalesmg/test-more-flaky-tests-r53-20231128
test(flaky): more adjustments
2023-11-30 09:33:55 +08:00
Zaiming (Stone) Shi 4ecfe2be30 test: use peer module for slave and ct_slave 2023-11-29 21:45:53 +01:00
Thales Macedo Garitezi 62b763a8f8 test(gcp_pubsub_consumer): even more adjustments 2023-11-29 15:12:47 -03:00
Ivan Dyachkov 7f5433f6dd chore: 5.3.2-rc.1 2023-11-29 18:20:15 +01:00
Thales Macedo Garitezi fdafe2493e
Merge pull request #12041 from thalesmg/ds-estimate-last-alive-m-20231128
fix(ds_session): take conservative estimate of `last_alive_at` when bumping
2023-11-29 13:43:59 -03:00
Zaiming (Stone) Shi 6f35f25163 chore: upgrade esockd to 5.9.8 for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi 1a563b4f65 chore: fix 0.0 match for OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi 14644988e0 chore: change triple-quotes to single-quotes 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi e6eb97e104 chore: upgrade dependency brod_gssapi to work with OTP 26 2023-11-29 16:15:18 +01:00
Zaiming (Stone) Shi 41194cacc8
Merge pull request #12030 from zhongwencool/redis-connector-v2-540
Redis connector v2 540
2023-11-29 16:13:09 +01:00
Kjell Winblad a750fbbc52
Merge pull request #12051 from HJianBo/fix-http-bridge-bugs
Fix http bridge bugs
2023-11-29 15:48:51 +01:00
Zaiming (Stone) Shi 89cdfbca63 fix(emqx_vm): trim new-line in otp version string 2023-11-29 15:32:25 +01:00
Andrew Mayorov ccef91437d
Merge pull request #12029 from keynslug/ft/EMQX-11049/qos2
feat(sessds): provide QoS2 message replay support
2023-11-29 14:10:16 +03:00
Serge Tupchii 009a15c7d0 fix(emqx_mgmt_api_data_backup): validate empty file in upload API
fixes EMQX-11488
2023-11-29 12:12:42 +02:00
Andrew Mayorov d8691f1d64
refactor(sessds): rename marker → committed offset
For better clarity.
2023-11-29 13:04:42 +03:00
zhongwencool 0aec2f7605 feat: redis bridge v2 2023-11-29 16:06:46 +08:00
zhongwencool 8ec3857f4d
Merge pull request #12046 from savonarola/1128-fix-redis-54
[5.4] Fix Redis connector
2023-11-29 16:06:23 +08:00
William Yang e90e843ff3
Merge pull request #12040 from qzhuyan/dev/william/quic-0.0.300
feat(quic): bump to quicer-0.0.303
2023-11-29 08:48:47 +01:00
JianBo He d2e5f302a8 chore(http): add description for bridges v1 2023-11-29 13:47:59 +08:00
JianBo He 72bc046063 chore: avoid unnecessary default values being filled when querying via the v1 api.
see: https://emqx.atlassian.net/browse/EMQX-11482
2023-11-29 12:01:59 +08:00
JianBo He d7bf8e97d2 chore: add tests case 2023-11-29 11:18:51 +08:00
Serge Tupchii b0d4ba183e test: use `emqx_cth_suite` in `emqx_coap_api_SUITE`
`emqx_coap_api_SUITE` was failing with `emqx_authn_schema_not_injected`
(missing authn schema mods in persistent_term) if it was run standalone.
2023-11-28 20:02:21 +02:00
Serge Tupchii 3530ed66d5 fix(emqx_gateway_coap): take subscribe options from uri_query
Closes #12031
2023-11-28 20:02:21 +02:00
Thales Macedo Garitezi 095e7c4ecb test(flaky): more adjustments 2023-11-28 13:41:31 -03:00
Thales Macedo Garitezi f6f61f4353
Merge pull request #12036 from thalesmg/test-more-flaky-tests-r54-20231127
test: attempting to stabilize more flaky tests
2023-11-28 13:40:49 -03:00
Ilya Averyanov e5508105c7 fix(redis): start and load eredis app 2023-11-28 18:51:53 +03:00
Ilya Averyanov 55218e2df2 fix(redis): start and load eredis app 2023-11-28 18:51:19 +03:00
Zaiming (Stone) Shi d93fdd8e9b
Merge pull request #12039 from zmstone/1128-sync-release-54
Sync release-54 to master
2023-11-28 15:52:49 +01:00
William Yang 64f6220cf5 feat(quic): bump to quicer-0.0.303 and emqtt 1.9.7 2023-11-28 14:43:47 +01:00
ieQu1 8a065ba6d0
Merge pull request #12027 from ieQu1/dev/ds-meta-persist-shard
fix(ds): Persist shard table
2023-11-28 14:33:57 +01:00
Thales Macedo Garitezi f5c4fb5860 fix(ds_session): take conservative estimate of `last_alive_at` when bumping
Addresses https://github.com/emqx/emqx/pull/12024#discussion_r1407432154
2023-11-28 09:58:03 -03:00
Thales Macedo Garitezi bd40b5c553 test(flaky): more adjustments 2023-11-28 09:48:40 -03:00
Ivan Dyachkov 2e11ab6a16 chore: 5.3.2-alpha.2 2023-11-28 13:46:28 +01:00
ieQu1 05e47254e2 fix(ds): Fixes related to the shards table 2023-11-28 12:59:09 +01:00
Zaiming (Stone) Shi 6c85e62d26 fix(schema): add namespaces 2023-11-28 12:07:42 +01:00
Zaiming (Stone) Shi 1b1cea24f3 Merge remote-tracking branch 'origin/release-54' into 1128-sync-release-54 2023-11-28 11:43:30 +01:00
Andrew Mayorov 34c9c022d0
chore(replayer): add comment describing what "until" means 2023-11-28 11:47:54 +03:00
JianBo He bd6e9503e6 fix(http): compose the url and path in correctly format 2023-11-28 16:15:46 +08:00
zhongwencool 6e6e68c902
Merge pull request #12037 from zhongwencool/license-watermark-check
fix: validate error when set license's watermark to 100%
2023-11-28 16:04:39 +08:00
zhongwencool 29bcdb9a4a fix: validate error when set license's watermark to 100% 2023-11-28 13:16:16 +08:00
JianBo He 891ecc179d chore: fix flaky tests 2023-11-28 10:34:44 +08:00
JianBo He c8b5c51bbc chore: fix failed test cases 2023-11-28 09:53:46 +08:00
JianBo He cdb90ebe6b feat: rename webhook bridge to http bridge 2023-11-28 09:53:46 +08:00
JianBo He dc99651690 test(bridge): ensure almost test cases passed 2023-11-28 09:53:46 +08:00
JianBo He 8954450c0b chore: fix compile warnings 2023-11-28 09:53:46 +08:00
JianBo He 96af7a74e8 feat: impl the http bridge v2 2023-11-28 09:53:46 +08:00
JianBo He f9a1e747fd chore(http): break the bridge confs to connector and action parts 2023-11-28 09:53:46 +08:00
SergeTupchiy 3b1ae0f3df
Merge pull request #12017 from SergeTupchiy/EMQX-11319-data-backup-http-api
EMQX-11319 data backup http api
2023-11-27 19:23:54 +02:00
Thales Macedo Garitezi e3fbf6c958 test: attempting to stabilize more flaky tests 2023-11-27 14:12:48 -03:00
Thales Macedo Garitezi 26e59f9508
Merge pull request #12024 from thalesmg/ds-session-expiry-m-20231124
feat(ds): session expiry
2023-11-27 14:11:38 -03:00
Stefan Strigler ac370fed61
Merge pull request #12035 from sstrigler/EMQX-11382-a-new-field-should-be-added-to-the-interface-of-the-connector-to-return-the-actions-associated-with-it
feat(emqx_connector): add field 'actions' in API response
2023-11-27 17:39:44 +01:00
Zaiming (Stone) Shi d933d0b9e0
Merge pull request #12016 from zmstone/1122-load-license-file
1122 load license file
2023-11-27 16:43:51 +01:00
Stefan Strigler 70978df5bb fix: use parse_id 2023-11-27 16:32:15 +01:00
Serge Tupchii 89fdbf3e73 fix(emqx_mgmt_cli): remove duplicated word in the log message 2023-11-27 17:30:58 +02:00
Serge Tupchii 64ee29af81 feat(emqx_management): implement data backup API 2023-11-27 17:30:58 +02:00
Andrew Mayorov 86685bdce2
feat(sessds): use integer tags for pubrange types 2023-11-27 18:27:15 +03:00
Andrew Mayorov bb05281adb
refactor(sessds): add dedicated `#ds_pubrange.tracks` field
This slightly simplifies the replayer code.
2023-11-27 18:24:57 +03:00
Thales Macedo Garitezi d88deb9ceb feat(ds): add session timer to bump last alive at timestamp 2023-11-27 12:19:52 -03:00
Stefan Strigler 048f4724a9 feat(emqx_connector): add field 'actions' in API response
Also unify schemas, use emqx_connector_schema for the generic parts.
2023-11-27 16:09:17 +01:00
Thales Macedo Garitezi 09c4e40511 refactor(ds): rename `disconnected_at` to `last_alive_at`, add more assertions 2023-11-27 11:48:44 -03:00
Thales Macedo Garitezi f2dbddc315 test: attempting to stabilize more flaky tests 2023-11-27 11:36:32 -03:00
Thales Macedo Garitezi 6acdc6e432 feat: add `/actions/:id/metrics/`, `/actions/:id/metrics/reset` APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11381
2023-11-27 11:33:11 -03:00
Andrew Mayorov 923898eadf
chore(chan): leave a TODO note for PUBREC handler 2023-11-27 09:55:46 +03:00
Andrew Mayorov 41973ee1fa
fix(sessds): please dialyzer with well-typed dummy msgs 2023-11-27 09:55:46 +03:00
Andrew Mayorov 46475fac66
feat(sessds): provide QoS2 message replay support 2023-11-27 09:55:45 +03:00
Andrew Mayorov ce59cb71bb
chore: bump emqtt to 1.9.6 2023-11-27 09:55:45 +03:00
zhongwencool b5a00ec6b2
Merge pull request #12023 from emqx/master
chore: sync master to release-54
2023-11-27 09:31:47 +08:00
ieQu1 bd7a84fe3e revert(ds): Don't duplicate the clean start in session_ds 2023-11-26 19:18:59 +01:00
Kjell Winblad c85004b7ef chore: remove obsolete TODO 2023-11-24 21:22:27 +01:00
Kjell Winblad c6c1d886f0 fix: make pgsql action schema properly 2023-11-24 21:19:26 +01:00
Thales Macedo Garitezi 839f9dbedb feat(ds): session expiry
Fixes https://emqx.atlassian.net/browse/EMQX-11048
2023-11-24 16:31:54 -03:00
Kjell Winblad 66945dcc5c fix: address more comments from @thalesmg 2023-11-24 20:31:29 +01:00
Kjell Winblad 30e248061f
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-24 20:07:37 +01:00
Kjell Winblad f070d80b1a fix: swagger examples for PostgreSQL, Matrix and Timescale 2023-11-24 18:50:52 +01:00
Kjell Winblad d03674a505 fix: duplicate key in example 2023-11-24 17:59:56 +01:00
Kjell Winblad fc7bedb81a fix: remove duplicated entry 2023-11-24 17:37:58 +01:00
Kjell Winblad e920160805 fix: add enable and description fields to PostgreSQL connector 2023-11-24 17:33:39 +01:00
Kjell Winblad 3dca83c854 fix: all missing descriptions 2023-11-24 17:33:39 +01:00
Kjell Winblad c5e281b84b fix: emqx_auth_postgresql test suites 2023-11-24 17:33:39 +01:00
Kjell Winblad 6ef9c6fe4a fix: ops 2023-11-24 17:33:39 +01:00
Kjell Winblad f7296d549f fix: elvis problem 2023-11-24 17:33:39 +01:00
Kjell Winblad 64c015cf6f fix: unify the schema modules in emqx_bridge_pgsql 2023-11-24 17:33:39 +01:00
Kjell Winblad b8f510d956 fix: add env variables about action info 2023-11-24 17:33:39 +01:00
Kjell Winblad 2e3028a8f8 fix(emqx_postgresql): fix lifecycle test 2023-11-24 17:33:39 +01:00
Kjell Winblad dbe73c70b1 fix: dialyzer problem 2023-11-24 17:33:38 +01:00
Kjell Winblad d5b62eead0 feat: split pgsql, matrix and timescale into connector action
This commit splits the bridges pgsql, matrix and timescale into
connector and action.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11155
2023-11-24 17:33:35 +01:00
Stefan Strigler f8f8cf9f30
Merge pull request #11963 from sstrigler/EMQX-11156-bridge-v-2-mongo-db-support
EMQX 11156 bridge v2 mongo db support
2023-11-24 16:49:57 +01:00
zhongwencool 8f548f4cbb
Merge pull request #12020 from thalesmg/sync-r53-m-20231124
chore: sync `release-53` to `master`
2023-11-24 22:54:53 +08:00
Stefan Strigler c1ef773e77 fix: check for sane state after regular shutdown 2023-11-24 15:09:35 +01:00
Stefan Strigler 4e077c951b feat(emqx_bridge_mongodb): port mongodb to shared connector and actions 2023-11-24 15:09:35 +01:00
Thales Macedo Garitezi 261fe8a831 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 10:10:09 -03:00
ieQu1 2d4b9a7b9d
Merge pull request #12019 from ieQu1/ds-clean-start
Discard sessions when client connects with clean start = true
2023-11-24 13:35:39 +01:00
Thales Macedo Garitezi e95ec5b150 test: fix another flaky test 2023-11-24 09:24:21 -03:00
Thales Macedo Garitezi f8fd95c683 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231124 2023-11-24 09:22:24 -03:00
Thales Macedo Garitezi 6a3c438194
Merge pull request #12009 from thalesmg/fix-kafka-get-bridges-r53-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-24 08:56:24 -03:00
Stefan Strigler 9ebbc9bbea refactor(emqx_bridge): use more simplistic function to validate connector 2023-11-24 10:07:11 +01:00
Stefan Strigler d2901afd1b fix(emqx_bridge_kafka): match example in api schema 2023-11-24 10:07:11 +01:00
ieQu1 e616e0746a feat(ds): Implement a function for dumping persistent session state 2023-11-24 04:39:46 +01:00
ieQu1 8dfcb69e52 feat(ds): Discard session when client connects with CleanStart=1 2023-11-24 04:17:28 +01:00
ieQu1 e8d18b0e09
Merge pull request #12014 from ieQu1/ds-qos0
Support QoS0 messages in persistent_session_ds
2023-11-23 23:43:18 +01:00
ieQu1 449bafc27e fix(ds): LTS trie handles empty topic levels 2023-11-23 22:16:38 +01:00
ieQu1 a158f25a40 fix(ds): Fix return type of emqx_persistent_session_ds:publish 2023-11-23 22:16:38 +01:00
ieQu1 c5bb86db67 feat(ds): Support QoS 0 2023-11-23 22:16:38 +01:00
ieQu1 1da9ec1d7d
Merge pull request #12015 from ieQu1/ds-lts-dont-store-message
feat(ds): Don't store #message record in the DB
2023-11-23 18:56:45 +01:00
Stefan Strigler 1c11a02a76
Merge pull request #12010 from sstrigler/EMQX-11415-e-5-3-2-api-should-return-404-when-the-built-in-database-authentication-doesnt-exist
fix(emqx_auth): check authenticator exists in /authenticator/:id/users
2023-11-23 18:08:48 +01:00
ieQu1 c69b82455e feat(ds): Don't store #message record in the DB 2023-11-23 17:56:35 +01:00
Zaiming (Stone) Shi 14077ec43b feat(license): allow setting 'default' license key 2023-11-23 17:52:17 +01:00
Zaiming (Stone) Shi caaf8113fc feat(license): support loading license from file 2023-11-23 17:52:17 +01:00
Thales Macedo Garitezi 4597e1c5b9
Merge pull request #12011 from thalesmg/fix-kafka-get-bridges-m-20231123
fix(bridge_api): don't mangle configs, use correct type as argument
2023-11-23 13:15:39 -03:00
ieQu1 1ced8786fd feat(ds): Make session poll interval configurable 2023-11-23 17:10:03 +01:00
ieQu1 82e74d0201 feat(ds): Add a flag that forces all sessions to become durable 2023-11-23 17:10:03 +01:00
Stefan Strigler 8ba116d378 fix(emqx_auth): check authenticator exists in /authenticator/:id/users 2023-11-23 16:15:03 +01:00
Ilya Averyanov ce77a12cd3
Merge pull request #12012 from savonarola/1123-fix-callback-types
fix(ds): impose more flexible types for ds implementations
2023-11-23 17:03:56 +02:00
Thales Macedo Garitezi fdfa3213cc fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 11:41:26 -03:00
Zaiming (Stone) Shi 7375bc5f9b fix(license): allow CRLF in license keys 2023-11-23 15:39:18 +01:00
Zaiming (Stone) Shi 48b69bd60c chore: delete a TODO from perfectionism 2023-11-23 15:39:18 +01:00
Ilya Averyanov 29d42506d5 fix(ds): impose more flexible types for ds implementations 2023-11-23 17:16:11 +03:00
Thales Macedo Garitezi 8e8d6d2192
Merge pull request #12006 from thalesmg/test-debug-flaky-test-r53-20231122
test(gcp_pubsub_consumer): fix flaky test
2023-11-23 10:57:22 -03:00
Thales Macedo Garitezi c89ec0b1f7 fix(bridge_api): don't mangle configs, use correct type as argument
Fixes https://emqx.atlassian.net/browse/EMQX-11412

- The wrong type was being used in a list lookup function, resulting in the automatic
transformation being called erroneously and mangling the config.
- There was a left-over workaround still around which could still mangle the config.
2023-11-23 10:25:36 -03:00
Thales Macedo Garitezi f3693e5dbc fix(gcp_pubsub_producer): add missing references to api specs 2023-11-23 09:02:52 -03:00
Zaiming (Stone) Shi 209077593d
Merge pull request #11970 from zmstone/1108-add-v2-schema-dump
1108 add v2 schema dump
2023-11-22 23:42:54 +01:00
Thales Macedo Garitezi db83457d13 test: fix flaky test
The cause was that the call `sys:terminate/2` was timing out...

`exit/2` doens't always work:

```
 2023-11-22 19:14:40.974
killed async workers

Error: -22T19:14:40.974563+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15908.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 987, stack_size: 28, reductions: 1822; neighbours:
Error: -22T19:14:40.998051+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15908.7>.
2023-11-22T19:15:41.088752+00:00 [critical] Run stage failed: error:{badmatch,{timeout,#{expected_remaining => 1,mailbox => {messages,[]},msgs_so_far => []}}}, Stacktrace: [{emqx_bridge_gcp_pubsub_consumer_SUITE,'-t_async_worker_death_mid_pull/1-fun-17-',3,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1576}]},{emqx_bridge_gcp_pubsub_consumer_SUITE,t_async_worker_death_mid_pull,1,[{file,"/emqx/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_consumer_SUITE.erl"},{line,1505}]}], Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-11-22_19.14.27/snabbkaffe/1700680540975786370.log", mfa: undefined
Error: -22T19:15:46.095702+00:00 [error] crasher: initial call: gun:proc_lib_hack/5, pid: <0.15934.7>, registered_name: [], exit: {{{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,654}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [gun_sup,<0.15387.7>], message_queue_len: 0, messages: [], links: [<0.15388.7>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 28, reductions: 1471; neighbours:
Error: -22T19:15:46.095192+00:00 [error] Supervisor: {local,ehttpc_sup}. Context: shutdown_error. Reason: killed. Offender: id={ehttpc_pool_sup,<<98,114,105,100,103,101,58,103,99,112,95,112,117,98,115,117,98,95,99,111,110,115,117,109,101,114,58,116,95,97,115,121,110,99,95,119,111,114,107,101,114,95,100,101,97,116,104,95,109,105,100,95,112,117,108,108,45,53,55,54,52,54,48,55,53,50,51,48,51,52,50,50,55,53,49>>},pid=<0.15903.7>.
Error: -22T19:15:46.095470+00:00 [error] Supervisor: {<0.15906.7>,ehttpc_worker_sup}. Context: shutdown_error. Reason: killed. Offender: id={worker,1},pid=<0.15924.7>.
Error: -22T19:15:46.096762+00:00 [error] Supervisor: {local,gun_sup}. Context: child_terminated. Reason: {{owner_gone,killed},[{gun,owner_gone,1,[{file,"gun.erl"},{line,970}]},{gun,proc_lib_hack,5,[{file,"gun.erl"},{line,649}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}. Offender: id=gun,pid=<0.15934.7>.
Warning: 2T19:15:46.098278+00:00 [warning] msg: remove_local_resource_failed, mfa: emqx_resource:remove_local/1(362), error: {error,timeout}, resource_id: <<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>
Error: -22T19:15:46.149090+00:00 [error] Generic server <0.15904.7> terminating. Reason: killed. Last message: {'EXIT',<0.15903.7>,killed}. State: {state,<<"bridge:gcp_pubsub_consumer:t_async_worker_death_mid_pull-576460752303422751">>,1,random}.
Error: -22T19:15:46.149525+00:00 [error] crasher: initial call: ehttpc_pool:init/1, pid: <0.15904.7>, registered_name: [], exit: {killed,[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,909}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}, ancestors: [<0.15903.7>,ehttpc_sup,<0.15731.7>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: true, status: running, heap_size: 376, stack_size: 28, reductions: 3428; neighbours:
```
2023-11-22 17:41:32 -03:00
Zaiming (Stone) Shi d9f964a44f test: fix test cases after schema type namespace change 2023-11-22 16:58:05 +01:00
Thales Macedo Garitezi fc849f0c05 ci(test): add info to help diagnose flaky test 2023-11-22 12:36:10 -03:00
Thales Macedo Garitezi 6c9417efe0 Merge remote-tracking branch 'origin/release-53' into sync-r53-m-20231122 2023-11-22 12:02:34 -03:00
Zaiming (Stone) Shi e065217e9a
Merge pull request #11997 from zmstone/1121-do-not-allow-special-char-as-prefix-for-resource-name
1121 do not allow special char as prefix for resource name
2023-11-22 15:46:11 +01:00
Thales Macedo Garitezi 274a254624
Merge pull request #12002 from thalesmg/multiple-action-infos-per-app-m-20231122
feat(actions): allow multiple action info modules per application
2023-11-22 10:42:56 -03:00
Thales Macedo Garitezi bb549cdf8b feat(actions): allow multiple action info modules per application 2023-11-22 09:33:01 -03:00
Thales Macedo Garitezi 38d3a1d7d0 feat(actions): allow multiple action info modules per application 2023-11-22 09:30:37 -03:00
Zaiming (Stone) Shi db33bc616a feat(schema): Add v2 scheam JSON dump 2023-11-22 13:12:35 +01:00
Zaiming (Stone) Shi 1b2c052646 docs: add type namespaces 2023-11-22 13:12:35 +01:00
Serge Tupchii d9f95cdc56 feat: integrate OpenTelemetry log handler 2023-11-22 13:59:30 +02:00
Ilya Averyanov a1b9a14fa1 feat(ds): allow fdb implementation for durable storage 2023-11-22 14:49:38 +03:00
Zaiming (Stone) Shi 3261a12140 fix(emqx_resource): do not allow leading _ or - as resource name 2023-11-22 10:58:54 +01:00
ieQu1 ee191803ea
Merge pull request #11990 from ieQu1/dev/ds-shards
Implement a prototype of durable message sharding
2023-11-22 00:57:54 +01:00
Zaiming (Stone) Shi 869e73d637
Merge pull request #11966 from thalesmg/fix-kafka-parameters-r53-20231117
fix(kafka): don't return `parameters` from `/bridges` API
2023-11-22 00:36:39 +01:00
Zaiming (Stone) Shi 4fb2d3a503
Merge pull request #11996 from zmstone/1121-prepare-for-e5.3.2-alpha.1
chore: bump release version to e5.3.2-alpha.1
2023-11-21 22:01:05 +01:00
Thales Macedo Garitezi 39791511fc chore: remove obsolete workaround 2023-11-21 17:38:13 -03:00
Thales Macedo Garitezi 01895fd660
Merge pull request #11995 from thalesmg/fix-action-api-bad-name-r53-20231121
fix(actions_api): don't crash on validation errors
2023-11-21 17:27:38 -03:00
Zaiming (Stone) Shi 5ebd954b16 chore: bump release version to e5.3.2-alpha.1 2023-11-21 21:19:05 +01:00
Zaiming (Stone) Shi 8fbdcab118
Merge pull request #11987 from zmstone/1120-do-not-crash-on-einval-after-check-cache
fix(emqx_connection): handle socket activation error return
2023-11-21 20:31:56 +01:00
Thales Macedo Garitezi 3a8c332805 fix(actions_api): don't crash on validation errors
Fixes https://emqx.atlassian.net/browse/EMQX-11394
2023-11-21 16:27:54 -03:00
ieQu1 3165b4f645 fix(ds): Abort application startup when rocksdb is not avialable 2023-11-21 20:27:07 +01:00
Thales Macedo Garitezi 11ec1a30a0 test(flaky): fix flaky pulsar test 2023-11-21 16:00:19 -03:00
Thales Macedo Garitezi b3dffa4390 fix(kafka): don't return `parameters` from `/bridges` API
Fixes https://emqx.atlassian.net/browse/EMQX-11412
2023-11-21 16:00:15 -03:00
ieQu1 3d823beb11 fix(ds): Apply review remarks 2023-11-21 19:37:01 +01:00
ieQu1 4d47490734 chore(ds): Rebase configuration 2023-11-21 19:37:01 +01:00
ieQu1 f5c71e8068 refactor(ds): Add a wrapper to the store batch API 2023-11-21 19:37:01 +01:00
ieQu1 2a1f7d946a feat(ds): Shard messages by publisher client ID 2023-11-21 19:37:01 +01:00
ieQu1 62542e5844 feat(ds): Metadata storage for the replication layer 2023-11-21 19:37:01 +01:00
Kjell Winblad b0d670aaa9
Merge pull request #11992 from kjellwinblad/kjell/backport/action_upgrade_downgrade_hooks
fix(action): upgrade and downgrade strategy
2023-11-21 19:14:37 +01:00
Thales Macedo Garitezi 9e1796ec4f feat(gcp_pubsub_producer): migrate GCP PubSub producer to actions
Fixes https://emqx.atlassian.net/browse/EMQX-11157
2023-11-21 14:22:42 -03:00
Zaiming (Stone) Shi fa91bacdfb
Merge pull request #11975 from zmstone/1119-fix-socket-close-race-condition
fix(emqx_channel): do not log stale sock_close event as error
2023-11-21 16:47:03 +01:00
Kjell Winblad fa7151f255 fix: port emqx_utils_maps:rename function from master
The emqx_utils_maps:rename function is needed by action
upgrade/downgrade hoos.
2023-11-21 15:18:22 +01:00
Thales Macedo Garitezi 6c9a8461f7
Merge pull request #11989 from thalesmg/confluent-action-m-20231120
feat: add confluent connector/action
2023-11-21 11:01:40 -03:00
Kjell Winblad cd72dc11dd fix: missing emqx_action_info module mapping 2023-11-21 14:22:29 +01:00
Thales Macedo Garitezi 068d151b14
Merge pull request #11983 from thalesmg/sync-r53-to-m-20231120
sync `release-53` to `master`
2023-11-21 09:02:51 -03:00
Thales Macedo Garitezi 759468efcf
Merge pull request #11986 from thalesmg/split-resource-opts-actions-r53-20231120
refactor: split `resource_opts` fields between connector and actions
2023-11-21 09:01:29 -03:00
Thales Macedo Garitezi 75ee3ced68 feat: add confluent connector/action
Fixes https://emqx.atlassian.net/browse/EMQX-11044

See also: https://emqx.atlassian.net/wiki/spaces/EMQX/pages/712179934/Confluent+Integration
2023-11-21 08:59:42 -03:00
Kjell Winblad 6030bf6fa5 fix(action): upgrade and downgrade strategy
This commit adds upgrade and downgrade hooks that are called when
upgrading from a bridge V1 to connector and action or the other way
around. The automatic translation is used if the callback is not
defined.

NOTE: Backported from master
2023-11-21 11:18:21 +01:00
Ilya Averyanov c6fd1e4c75
Merge pull request #11971 from savonarola/1116-no-auth-rebalance-status
feat(rebalance): improve rebalance usability
2023-11-21 12:10:18 +02:00
Andrew Mayorov cd48c283db
Merge pull request #11988 from keynslug/ft/EMQX-11222/basic-config
feat(schema): introduce separate root for new session persistence
2023-11-21 15:02:21 +07:00
JianBo He 9716b051d9 fix(ocpp): fix bad webscoket schema for HTTP API 2023-11-21 11:24:23 +08:00
Thales Macedo Garitezi 6f8630304d
Merge pull request #11980 from kjellwinblad/kjell/fix_fixup_callbacks/EMQX-11428
fix: bridge to action upgrade fixup hook should run after upgrade
2023-11-20 16:19:43 -03:00
Kjell Winblad 7fb5ade832 fix(kafka_producer): hocon renames kafka field 2023-11-20 19:20:13 +01:00
Kjell Winblad d214ae8772
fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-20 18:59:57 +01:00
Andrew Mayorov 110a5a4896
feat(schema): introduce separate root for new session persistence
With some knobs to choose a storage backend. Support only builtin
RocksDB-based backend with minimal configuration for now.
2023-11-20 23:49:14 +07:00
Zaiming (Stone) Shi b02711af79 refactor(emqx_ws_connection): rename cache to buffer for limiter 2023-11-20 17:47:20 +01:00
Andrew Mayorov 8e107ffe45
Merge pull request #11958 from keynslug/ft/EMQX-9745/preserve-acks
feat(sessds): preserve acks / replays in session state
2023-11-20 23:40:22 +07:00
Kjell Winblad edbfe090c9 feat: add custom callback for creating connector from bridge V1 config 2023-11-20 17:23:51 +01:00
Zaiming (Stone) Shi ec19247271 refactor: rename limiter buffer related messages and var names 2023-11-20 16:55:26 +01:00
Andrew Mayorov 9684e79ee0
fix(sessds): ensure dup flag is on for replayed messages 2023-11-20 22:54:40 +07:00
Zaiming (Stone) Shi 8ec3b1db5d fix(emqx_connection): handle socket activation error return 2023-11-20 16:51:39 +01:00
Kjell Winblad 3aa8044475 fix(action): upgrade and downgrade strategy
This commit fixes the upgrade and downgrade strategy when upgrading
from a bridge V1 to connector and action or the other way around so that
the custom callbacks get the complete unchanged input instead of the
result of the automatic translation. The automatic translation is used
if the callback is not defined.
2023-11-20 16:40:27 +01:00
Thales Macedo Garitezi f40f6bc5dd refactor: split `resource_opts` fields between connector and actions 2023-11-20 12:37:36 -03:00
Stefan Strigler 1f1d9e58c6 fix(emqx_connector): don't crash in API on delete with active channels 2023-11-20 16:23:46 +01:00
Thales Macedo Garitezi 53e796bbd0 Merge remote-tracking branch 'origin/release-53' into sync-r53-to-m-20231120 2023-11-20 11:49:52 -03:00
Andrew Mayorov ef7cfd0202
feat(sessds): add field to `ds_pubrange` for forward-compat 2023-11-20 19:56:55 +07:00
Thales Macedo Garitezi b42d4e5ab0
Merge pull request #11950 from thalesmg/fix-bridge-v1-examples-r53-20231114
fix(bridges_v1): avoid merging action examples for non-v1 bridges
2023-11-20 09:12:23 -03:00
Thales Macedo Garitezi 926078b82c
Merge pull request #11960 from thalesmg/fix-kafka-producer-channel-status-r53-20231116
fix(kafka_producer): make status `connecting` while the client fails to connect
2023-11-20 09:12:05 -03:00
Kjell Winblad 79a764f117 fix: bridge to action upgrade fix up hook should run after upgrade
This commit changes how the `emqx_action_info` callback
`bridge_v1_to_action_fixup/1` works. It is now called after the
automatic upgrade instead of before. Since the full Bridge V1 config
might be needed to do the fixup, it is provided in a special field
`<<"__bridge_v1_conf__">>`. The `<<"__bridge_v1_conf__">>` field is
removed after the callback is called and can thus be ignored if it is
not needed.
2023-11-20 12:06:21 +01:00
Stefan Strigler 84ff7b0b38 feat(emqx_bridge): action_info with dynamic lookup
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
Andrew Mayorov d7974d835f
fix(sharesub): anticipate messages w/o redispatch header
For instance, `emqx_session_mem` will push almost all of the messages
that are still in the state to `emqx_shared_sub:redispatch/1`. After
this commit, the session will no longer crash during channel terminate.
2023-11-20 15:53:24 +07:00
Andrew Mayorov 7081f1951f
refactor(sessds): use `ds_pubrange` record as is
Instead of converting it into almost similar runtime representation.
2023-11-20 15:40:43 +07:00
Andrew Mayorov a5ff4144fe
test(sessds): add complex testcase for session replay 2023-11-20 15:40:43 +07:00
Andrew Mayorov 1246d714c5
feat(sessds): preserve acks / ranges in mnesia for replays 2023-11-20 13:33:48 +07:00
Andrew Mayorov 5b40304d1f
chore(sessds): simplify subscriptions handling
There's currently no point in storing parsed topic filters in the
subscriptions table.
2023-11-20 13:33:48 +07:00
Andrew Mayorov 648b6ac63e
chore(sessds): rename iterators -> subscriptions
Also try to make clearer the difference between 2 flavors of topic
filter representation in use.
2023-11-20 13:33:48 +07:00
Zaiming (Stone) Shi e73bf716ae fix(emqx_channel): do not log stale sock_close event as error
In some cases, EMQX may decide to close socket and mark connection
at 'disconnected' state, for example, when DISCONNECTE packet is
received, or, when failed to write data to socket.
However, by the time EMQX decided to close the socket, the socket
might have already been closed by peer, and the `tcp_closed` envet
is already delivered to the process mailbox -- causing EMQX to
handle sock_close event at 'disconnected' state.
2023-11-19 22:27:25 +01:00
Ilya Averyanov e93e9ed108 feat(rebalance): improve rebalance usability
* make availability API endpoint public
* allow connections during wait_health_check interval
* make availability status calculation more consistent and lightweight
* refactor test to get rid of some mocks and to use cth
2023-11-18 21:29:16 +03:00
Thales Macedo Garitezi eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Kjell Winblad 86c126ffcd feat: callbacks for fixup after automatic Bridge V1 upgrade/downgrade
This commit adds callbacks to the emqx_action_info module for
doing fixes (such as changing a field name) after the automatic
split of a Bridge V1 config or the merge of connector and action
configs for the compatibility layer.
2023-11-17 16:10:08 -03:00
Kjell Winblad 9feba802e9 chore: add convenience function for creating action schemas 2023-11-17 16:10:08 -03:00
Ilya Averyanov 58437cd35a fix(mongodb): fix deadlock while stopping mongodb resource 2023-11-17 12:34:36 +03:00
zhongwencool b2e168d508 feat: add prometheus.push_gateway.enable config 2023-11-17 09:48:39 +08:00
zhongwencool c4eb9f86e7 chore: add refactor prometheus refactor changelog 2023-11-17 08:23:51 +08:00
zhongwencool b343653769 chore: support list(tuple()) for schema_with_examples/3 2023-11-17 08:23:51 +08:00
zhongwencool 0c2610fa61 feat: support prometheus.enable_basic_auth 2023-11-17 08:23:51 +08:00
zhongwencool c376a5db29 refactor: prometheus config 2023-11-17 08:23:51 +08:00
Thales Macedo Garitezi b92821188b fix(kafka_producer): make status `connecting` while the client fails to connect
Fixes https://emqx.atlassian.net/browse/EMQX-11408

To make it consistent with the previous bridge behavior.

Also, introduces macros for resource status to avoid problems with typos.
2023-11-16 14:50:23 -03:00
Ilya Averyanov 4c54efd8b1 fix(auth): fix deadlock while stopping mongodb resource 2023-11-16 19:29:08 +03:00
Thales Macedo Garitezi 36b5d58957 test: reorganize test suite a bit 2023-11-16 13:16:02 -03:00
Thales Macedo Garitezi 90571b7d8e test: fix noise about undefined unofficial callbacks 2023-11-16 13:16:02 -03:00
Andrew Mayorov 656d575e2d
ci(router): fix flaky testcase 2023-11-16 17:53:20 +07:00
Andrew Mayorov d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default
feat(router): switch to v2 routing store by default
2023-11-16 15:56:26 +07:00
Andrew Mayorov 893e90b372
fix(kafka): use safe publish in consumer
Routing with v2 schema is actually more strict with respect to input to
`emqx_router` module routines. This causes Kafka consumer bridge to
crash when it tries to publish a message to a topic that looks like a
topic filter.
2023-11-15 22:37:00 +07:00
Andrew Mayorov 98f947f4f3
ci(router): fix flaky testcase 2023-11-15 22:28:52 +07:00
Andrew Mayorov 8919b08207
fix(utils): rename `emqx_utils_stream:take/2` to `consume/2`
Which is more neutral and harder to confuse with a destructive `take` in
collections.
2023-11-15 17:20:40 +07:00
Ivan Dyachkov 65ba381fd8 Merge remote-tracking branch 'upstream/master' into 1115-sync-master-to-r53 2023-11-15 09:25:03 +01:00
Thales Macedo Garitezi f5545bbdf3 fix(schema_types): add missing `secret()` type 2023-11-14 16:08:13 -03:00
Thales Macedo Garitezi 5275a01727 fix(syskeeper): refactor action to new action info scheme 2023-11-14 19:49:28 +01:00
Thales Macedo Garitezi d2cf29fae4 fix(schema): use `hoconsc:union/1` 2023-11-14 19:49:08 +01:00
Thales Macedo Garitezi 423fb32171 test: fix noise about undefined unofficial callbacks 2023-11-14 19:48:39 +01:00
Ivan Dyachkov 7c0e345d3a Merge remote-tracking branch 'upstream/release-54' 2023-11-14 19:38:21 +01:00
Thales Macedo Garitezi 2f1d88d414 fix(bridges_v1): avoid merging action examples for non-v1 bridges
Since some new bridges might not have a V1 equivalent (i.e. they are not registered in
`emqx_bridge_enterprise`), we should avoid displaying their examples in the V1 API spec.
2023-11-14 13:56:50 -03:00
Zaiming (Stone) Shi 88637f81d1
Merge pull request #11936 from zmstone/1112-readable-types
refactor(schema): keep type converters close
2023-11-14 17:56:07 +01:00
ieQu1 124c0e2dba fix(ds): Respect receive_maximum from the connection info 2023-11-14 16:06:19 +01:00
Ivan Dyachkov bbf00c2f71
Merge pull request #11946 from id/1114-sync-r53
sync release-53 to master
2023-11-14 13:38:24 +01:00
Thales Macedo Garitezi 01a2a3b1c0
Merge pull request #11930 from thalesmg/ds-session-discard-m-20231110
feat(ds): implement session discard
2023-11-14 09:30:34 -03:00
Thales Macedo Garitezi d6e9bbb95c fix(connector): validate connector name before converting ssl certs
Fixes https://emqx.atlassian.net/browse/EMQX-11336

See also: https://github.com/emqx/emqx/pull/11540
2023-11-14 09:29:59 -03:00
Andrew Mayorov 6812ee9d0f
fix(mgmt): hide route selection behind router interface
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
2023-11-14 18:36:25 +07:00
Ivan Dyachkov 28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
Ivan Dyachkov 1c57993c91 Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53 2023-11-14 10:44:35 +01:00
Andrew Mayorov d1c3b1c659
fix(bridges): handle secrets more consistently
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 16:05:54 +07:00
Andrew Mayorov d4dd4508c9
feat(greptimedb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 5d620465e2
feat(influx): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 7d6c9d2daa
feat(iotdb): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 7817502b8b
feat(rocketmq): accept wrapped secrets as passwords 2023-11-14 16:05:53 +07:00
Andrew Mayorov 2449d54b1f
feat(pulsar): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov e2b7b33d14
feat(kinesis): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 93eaf0caee
feat(dynamo): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 7250059c60
feat(sqlserver): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov 06861e377f
feat(ldap): accept wrapped secrets as passwords 2023-11-14 16:05:52 +07:00
Andrew Mayorov fda395014c
test(kafka): verify file-based secrets work 2023-11-14 16:05:25 +07:00
Andrew Mayorov aa458b65d6
test(kafka): simplify consumer testsuite matrix setup 2023-11-14 16:05:25 +07:00
Andrew Mayorov 18cd98def6
chore(test): fix formatting quirks 2023-11-14 16:05:25 +07:00
Andrew Mayorov c300eb41a7
feat(kafka): accept wrapped secrets as passwords 2023-11-14 16:05:25 +07:00
Andrew Mayorov 9c5856029f
feat(cassa): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov 8b4ac8eb4f
feat(oracle): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov 34aeeab041
feat(rabbitmq): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov a69a78d024
feat(clickhouse): accept wrapped secrets as passwords 2023-11-14 16:05:24 +07:00
Andrew Mayorov fc340a276e
feat(mongo): accept wrapped secrets as passwords
Also test authorization with mongo in bridge / auth test suites.
2023-11-14 16:05:24 +07:00
Andrew Mayorov f827df2821
test(mysql): slightly simplify bridge testsuite 2023-11-14 16:05:23 +07:00
Andrew Mayorov 36e57a479d
feat(mysql): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov ad72ab7a8c
feat(redis): accept wrapped secrets as passwords 2023-11-14 16:05:23 +07:00
Andrew Mayorov 4385b2f020
feat(pgsql): accept wrapped secrets as passwords
That are coming from `emqx_schema_secret`. Also adapt pgsql-related
connectors.
2023-11-14 16:05:23 +07:00
Andrew Mayorov 2b36b48f98
feat(connector): use `emqx_schema_secret` as password field schema 2023-11-14 16:05:23 +07:00
Ivan Dyachkov a49aea3b56 chore: bump app versions 2023-11-14 09:27:04 +01:00
Kjell Winblad d682e6e23c test: fix test mock by calling exported function 2023-11-14 09:20:46 +01:00
Kjell Winblad 3bea3496af test: fix test case mock that broke due to removal 2023-11-14 09:20:46 +01:00
Kjell Winblad 093c8b0c6e docs: add missing copyright headers 2023-11-14 09:20:46 +01:00
Kjell Winblad ca3e5eab11 refactor: remove emqx_bridge_v2_enterprise.erl 2023-11-14 09:20:46 +01:00
Kjell Winblad e93b71d8d5 fix: problems found by @thalesmg in code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-14 09:20:46 +01:00
Kjell Winblad 49fdfef8c3 fix: azure event hub names are defined in one place 2023-11-14 09:20:46 +01:00
Kjell Winblad 5e8e407017 refactor: action schema retrival after PR feedback 2023-11-14 09:20:46 +01:00
Kjell Winblad ab078647a5 chore: fix formatting problem 2023-11-14 09:20:46 +01:00
Kjell Winblad bdab421885 fix(bridge_v2): name and copyright header 2023-11-14 09:20:46 +01:00
Kjell Winblad ca2cdbc08d fix: bapi static check error 2023-11-14 09:20:46 +01:00
Kjell Winblad 7839f0cbc8 chore(bridge_v2): make dialyzer and bpapi happy 2023-11-14 09:20:46 +01:00
Kjell Winblad d26a1b9afb fix(bridge_v1): no hard coded downgrade and upgrade type functions 2023-11-14 09:20:46 +01:00
Kjell Winblad a6aa81b548 fix(emqx_bridge_v2): dialyzer found bad format parameter error 2023-11-14 09:20:46 +01:00
Kjell Winblad 3c778121a5 fix: bridge V1 type lookup issue 2023-11-14 09:20:46 +01:00
Kjell Winblad 77aaff137a test(emqx_bridge_v2_SUITE): fix incorrect assumption about return type
This commit fixes an incorrect assumption about the return type of
emqx_bridge_v2:lookup/2 in emqx_bridge_v2_SUITE:t_load_config_success/1.
2023-11-14 09:20:46 +01:00
Kjell Winblad 14e305e2a6 fix(emqx_bridge_v2): xref warning 2023-11-14 09:20:46 +01:00
Kjell Winblad 9eaee8f333 refactor(emqx_bridge_v2): make independent of Kafka
This removes the Kafka specific knowledge from emqx_bridge_v2 and
makes it possible to add new Bridge V2 bridges without modifying
the emqx_bridge application.
2023-11-14 09:20:46 +01:00
Kjell Winblad cd5b1f9b96 docs(bridge_V2): type specs for operations 2023-11-14 09:20:46 +01:00
Kjell Winblad 99031f0dae refactor(bridge_v2): prefix compatibility functions with bridge_v1 2023-11-14 09:20:46 +01:00
Kjell Winblad 729e7df0d5 docs(bridge_v2): add specs for CRUD functions 2023-11-14 09:20:46 +01:00
Ivan Dyachkov a9cf8fe78c chore: 5.3.1 2023-11-14 08:44:41 +01:00
Zaiming (Stone) Shi ab1f3b6b2f fix(emqx_conf_schema_types): address review comments 2023-11-13 21:55:03 +01:00
Stefan Strigler fc2891d6de
Merge pull request #11937 from sstrigler/EMQX-11318-crash-in-get-bridges-v-2-if-a-broken-bridge-is-configured
Emqx 11318 crash in get bridges v 2 if a broken bridge is configured
2023-11-13 17:28:56 +01:00
Zaiming (Stone) Shi 4c5d64abc2 refactor(schema): keep type converters close 2023-11-13 15:26:30 +01:00
Stefan Strigler 17b18849de test(emqx_bridge): test broken config 2023-11-13 14:52:06 +01:00
Zaiming (Stone) Shi 9b9908780c
Merge pull request #11926 from kjellwinblad/kjell/fix_bridge_v2_to_action_cluster_hocon/EMQX-11353
fix: reflect bridge V1 upgreade in cluster.hocon
2023-11-13 14:18:57 +01:00
Stefan Strigler 0b3645057d fix(emqx_bridge): don't try to handle what we don't understand 2023-11-13 14:08:54 +01:00
Stefan Strigler 0d33284672 fix(emqx_bridge): don't crash uninstallling broken action 2023-11-13 14:08:54 +01:00
Stefan Strigler 4fe1d332b3 fix(emqx_bridge): don't crash if there's no status field 2023-11-13 14:08:54 +01:00
Thales Macedo Garitezi 45dad2ed3a feat(ds): implement session discard
Fixes https://emqx.atlassian.net/browse/EMQX-9739

Fixes some issues to ensure the session is discarded when the client connects with
`clean_start = true`, and added some cleanup to subscriptions/routes/iterators/streams.

> There is an API that session garbage collector can use to perform cleaning

We already have `emqx_session:destroy/1`, which could serve as an API for a periodic
session GC to use.
2023-11-13 09:06:22 -03:00
Kjell Winblad 26fc9ed89d
fix: log warning message after suggestion from @zmstone
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-11-13 12:53:09 +01:00
Andrew Mayorov 03843c6071
feat(router): switch to v2 routing store by default 2023-11-13 17:43:30 +07:00
Kjell Winblad 5838568d5d fix: hard coded call to schema module
The user can configure the configuration loader and schema module so we
should not hard code a call to the default schema module.
2023-11-13 11:41:40 +01:00
Zaiming (Stone) Shi 518b02fc70
Merge pull request #11931 from zmstone/1110-delete-stale-type-converters
1110 delete stale type converters
2023-11-13 10:27:51 +01:00
Zaiming (Stone) Shi 65efa2672e test(schema): delete stale reference to emqx_schema:file() type 2023-11-13 09:28:26 +01:00
Zaiming (Stone) Shi 2fda91ec59 refactor: delete default type converter for dashboard schema json
prior to this commit, dashboard scheam json type conversion
has a 'fallback' type for all unknown types, which is 'string'

this commit removes this fallback and add all possible types
to the function clauses.

also, some of the old types which are no longer in use
are deleted from both swagger spec converter and dashboard
sechema converter
2023-11-12 20:08:53 +01:00
JimMoen 952ab08991
fix: make dialyzer happy 2023-11-12 21:44:24 +08:00
JimMoen 20bf9ed26d
fix(gw_gbt32960): mountpoint should not with `/` suffix 2023-11-12 21:44:23 +08:00
JimMoen 867d5aaea6
chore: rm dbg funcs in SUITE 2023-11-12 21:44:23 +08:00
JimMoen 70fcfdddcb
test(gw_jt808): fix test auth server bootstrap 2023-11-12 21:44:23 +08:00
JimMoen dc64068e41
test(gw_jt808): frame parser and channel SUITE 2023-11-12 21:44:23 +08:00
JimMoen 4d493292fb
feat(gw_jt808): port jt808 gateway from 4.4 2023-11-12 21:44:18 +08:00
JimMoen 526faa0362
fix(gw_gbt32960): clean frame parser state acc data 2023-11-12 21:43:12 +08:00
JimMoen d30f0aa4a0
fix(gw_gbt32960): parse state atom key error 2023-11-12 21:43:11 +08:00
JimMoen a4da1d260a
docs(gw_jt808): data exchange guide cn 2023-11-12 21:43:11 +08:00
JimMoen 17edeead6e
fix: erlang_ls complains gateway hrl file can not be found 2023-11-12 21:43:11 +08:00
JimMoen a218ff6dd8
fix(gw_gbt32960): undefined type `emqx_gbt32960_frame:frame()` 2023-11-12 21:43:11 +08:00
JimMoen aef5b17ed5
fix(gw_lwm2m): lwm2m channel behaviour 2023-11-12 21:43:11 +08:00
JimMoen caade4a7e5
chore: emacs mode entry comment for non-erl files 2023-11-12 21:43:11 +08:00
JimMoen 25e277bdc6
chore: fix schema desc and README.md typo 2023-11-12 21:43:10 +08:00
JimMoen 523bb17fc8
test: mark proper types gateway connection modules need be fixed 2023-11-12 21:43:10 +08:00
Zaiming (Stone) Shi b7d2c38213 refactor: delete stale type converters
Some of the types are refactored to use typerefl alias
or hoconsc:enum, no need to keep the special function
clauses to translate them to swagger spec
2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi a32cd20758 refactor(emqx_authn_jwt_schema): add alias for verify_claims
verifiy_claims is a list after converted, but input is a map
2023-11-12 11:39:44 +01:00
Zaiming (Stone) Shi 021f7e6b49 refactor(schema): comma_separated_list is list(string()) not list(any()) 2023-11-12 11:37:58 +01:00
Zaiming (Stone) Shi 06e440260f refactor(schema): add args to map alias 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi 8be718b22f refactor(emqx_schema): delete unused type bar_separated_list 2023-11-12 11:29:33 +01:00
Zaiming (Stone) Shi ff5916e33f docs: fix bad html tag 2023-11-12 10:31:37 +01:00
Xinyu Liu 7bb3ca5519
Merge pull request #11909 from emqx/bugfix-plugins
fix: emqx failed to start if plugin dir is removed by the user
2023-11-11 10:51:26 +08:00
Thales Macedo Garitezi b4e45f0189
Merge pull request #11923 from thalesmg/sync-m-to-r54-20231109
chore: sync `master` to `release-54`
2023-11-10 13:04:55 -03:00
Thales Macedo Garitezi 088de9476c fix(ds): use integer for tag values
Follow up to https://github.com/emqx/emqx/pull/11906#discussion_r1389115973
2023-11-10 11:26:32 -03:00
ieQu1 5170267fef
Merge pull request #11924 from ieQu1/dev/emqx-ds-behavior
feat(ds): Introduce emqx_ds behavior
2023-11-10 15:23:28 +01:00
Kjell Winblad 6ebf0d1bdd fix: reflect bridge V1 upgrade in override configurations
Fixes:
https://emqx.atlassian.net/browse/EMQX-11353
2023-11-10 14:52:29 +01:00
Zaiming (Stone) Shi 101990b8fc test: fix schema tests after types change 2023-11-10 14:23:43 +01:00
Zaiming (Stone) Shi e653c6b4e0 chore: bump app versions 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 855b3c5b29 test: ensure atom exists 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi f1de0aa176 fix(schema): add namespace to authn schemas 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 86110824eb feat: upgrade hocon to 0.40.0 which supports union type display name 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 8a4fba431e refactor(emqx_limiter_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi d603de10e6 refactor(schema): add namespace/0 to all schema modules 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 7b59d46854 fix(schema): add namesapce to sso schema 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi 6b3aaf5b24 refactor(emqx_bridge_gcp_pubsub): use typerefl alias 2023-11-10 13:41:51 +01:00
Zaiming (Stone) Shi b24b66081a refactor(authn/authz_http_schema): use typerefl alias 2023-11-10 13:41:51 +01:00
ieQu1 2ade6da7a6 fix(ds): Remove redundand type 2023-11-10 12:53:30 +01:00
ieQu1 feef23fc08 feat(ds): Pass store_batch through RPC 2023-11-10 12:16:39 +01:00
ieQu1 8dc8237331 feat(ds): Introduce emqx_ds behavior 2023-11-10 12:16:39 +01:00
William Yang 0fc4bcaf0d
Merge pull request #11912 from qzhuyan/test/william/quic-multistream-flaky
test(quic): fix flaky
2023-11-10 11:13:35 +01:00
JianBo He d1623f491c
Merge pull request #11925 from lafirest/docs/gbt
chore(gbt32960): add EN documentation of the data exchange guide
2023-11-10 16:43:49 +08:00
Zaiming (Stone) Shi 3e2ce57cb1 refactor(emqx_bridge_greptimedb): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 46fffa401e refactor(emqx_prometheus_schema): use typerefl alias 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi ae24b45d1b fix(schema): add type namespce to emqx_dashboard_sso_ldap 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi a2700771c9 refactor(emqx_s3_schema): make type readable 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi b3df8604fc refactor(emqx_conf_schema): use hocon enum type for log levels 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi df96798c53 refactor(http_bridge): use more readable type for pool_type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 40c2ca134d refactor(schema): delete unused type cipher() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 7a0a0bee98 refactor(schema): change file() type to string() 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi af3dfc91b1 refactor(schema): change log_lvel to enum type 2023-11-10 08:45:20 +01:00
Zaiming (Stone) Shi 3d20e566d8 docs: add namespace to authn and authz schema 2023-11-10 08:45:20 +01:00
firest 0299e9033a chore(gbt32960): add EN documentation of the data exchange guide 2023-11-10 15:22:28 +08:00
Zaiming (Stone) Shi 9c1a1c7de8
Merge pull request #11919 from thalesmg/action-api-swagger-nits-r53-20231109
fix(actions_api): change bridge v2 occurrences to action
2023-11-10 07:35:35 +01:00
firest 447c933ff8 fix(syskeeper): homogenize the Syskeeper bridge and update license data 2023-11-10 10:42:53 +08:00
ieQu1 38d004e06e
Merge pull request #11913 from ieQu1/dev/fix-hooks-comment
chore(hooks): Fix comment
2023-11-10 03:15:54 +01:00
firest 548e39e799 refactor(sysk): refactor the syskeeper to v2 style 2023-11-10 09:52:38 +08:00
firest abc7233a77 test(sysk): add test suite for syskeeper 2023-11-10 09:52:38 +08:00
firest e93216fa62 chore(sysk): fix spellchecks & update change & more logs 2023-11-10 09:52:32 +08:00
firest ce83079c6b feat(sysk): integrated Nari Syskeeper 2000 as a new bridge backend 2023-11-10 09:51:43 +08:00
JianBo He 6500d21d98
Merge pull request #11885 from HJianBo/ocpp-gw
feat: Port OCPP gateway from v4
2023-11-10 09:32:40 +08:00
ieQu1 ced7be61f0
Merge pull request #11917 from ieQu1/dev/fix-packet-id-to-seqno-trans
fix(ds): Fix packet id -> sequence number translation
2023-11-09 22:57:16 +01:00
Thales Macedo Garitezi 7a237f6a3a Merge branch 'master' into sync-m-to-r54-20231109 2023-11-09 18:49:52 -03:00
Thales Macedo Garitezi 371a49304d Merge branch 'release-53' into sync-r53-to-m-20231109 2023-11-09 17:19:23 -03:00
ieQu1 5aa9d026df fix(ds): Apply review remarks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-09 21:16:59 +01:00
ieQu1 b812db1e3c fix(ds): Fix packet id -> sequence number translation 2023-11-09 20:12:39 +01:00
Thales Macedo Garitezi 2d539463cd test: fix flaky test
```
 Testing lib.emqx.emqx_cm_SUITE: *** FAILED test case 24 of 31 ***
%%% emqx_cm_SUITE ==> t_stepdown_session_takeover_begin_timeout: FAILED
%%% emqx_cm_SUITE ==>
Failure/Error: ?assertEqual([<0.6635.8>,<0.6520.8>], lists : sort ( emqx_cm : lookup_channels ( ClientId ) ))
  expected: [<0.6635.8>,<0.6520.8>]
       got: [<0.6520.8>,<0.6635.8>]
      line: 309
```
2023-11-09 15:32:52 -03:00
Thales Macedo Garitezi ee87f90b0d
Merge pull request #11906 from thalesmg/ds-avoid-records-m-20231108
chore(ds): avoid using records in persistence / rpc
2023-11-09 14:25:27 -03:00
Thales Macedo Garitezi 7977ae95c0 fix(actions_api): change bridge v2 occurrences to action
Fixes https://emqx.atlassian.net/browse/EMQX-11338
2023-11-09 14:20:08 -03:00
Thales Macedo Garitezi eabd09051a feat(actions_api): add `/action_types` API 2023-11-09 13:54:57 -03:00
ieQu1 42a9f0eb6b
Merge pull request #11914 from ieQu1/dev/speed-up-poll
fix(ds): Speed up polling for the new messages
2023-11-09 16:40:06 +01:00
Thales Macedo Garitezi 42536bb5f7 chore(ds): avoid using records in persistence / rpc
Fixes https://emqx.atlassian.net/browse/EMQX-11279
2023-11-09 12:00:25 -03:00
ieQu1 cf4a46a78b fix(ds): Speed up polling for the new messages
Poll immediately if the previous poll returned non-empty result
2023-11-09 14:22:32 +01:00
ieQu1 3350bda446 chore(hooks): Fix comment 2023-11-09 14:05:13 +01:00
firest 2b97800c8b fix(limiter): simplify the descriptions for fields of the limiter 2023-11-09 18:00:09 +08:00
William Yang f7a2359329 test(quic): fix flaky 2023-11-09 10:56:16 +01:00
JianBo He d55f1e0813 chore: fix mix compiling failures 2023-11-09 16:54:41 +08:00
Shawn e0f4a2014d fix: emqx failed to start if plugin dir is removed by the user 2023-11-09 16:01:20 +08:00
Ivan Dyachkov 0bf5deaed9 chore: merge master into release-54 2023-11-09 08:32:26 +01:00
JianBo He d6104b37cf test: fix flaky tests 2023-11-09 15:01:12 +08:00
JianBo He 4b18631d49 chore: fix gateway failed test cases 2023-11-09 13:44:20 +08:00
JianBo He 2d958beae9 chore: try to fix spellcheck 2023-11-09 09:51:08 +08:00
Zaiming (Stone) Shi 42fa1289a1
Merge pull request #11905 from thalesmg/tag-e531-alpha5-r53-20231108
chore: prepare to tag `e5.3.1-alpha.5`
2023-11-08 23:39:33 +01:00
Zaiming (Stone) Shi f95058a3e7
Merge pull request #11897 from zmstone/1107-fix-cluster-conf-sync-wait-loop
1107 fix cluster conf sync wait loop
2023-11-08 23:37:09 +01:00
Zaiming (Stone) Shi 1e3500ffd2
fix(emqx_conf_app): ensure log data fields
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-11-08 21:19:22 +01:00
Thales Macedo Garitezi 8540566eba chore: prepare to tag `e5.3.1-alpha.5` 2023-11-08 13:02:19 -03:00
Zaiming (Stone) Shi f9e9748cec fix(cluster-rpc): boot from local config if table loaded
When EMQX boots up, it tries to get latest config from peer (core type)
nodes, if none of the nodes are replying, the node will decide
to boot with local config (and replay the committed changes) if
the commit table is loaded from disk locally (an indication of the
data being latest), otherwise it will sleep for 1-2 seconds and
retry.

This lead to a race condition, e.g. in a two nodes cluster:

1. node1 boots up
2. node2 boots up and copy mnesia table from node1
3. node1 restart before node2 can sync cluster.hocon from it
4. node1 boots up and copy mnesia table from node2

Now that both node1 and node2 has the mnesia `load_node` pointing
to each other (i.e. not a local disk load).

Prior to this fix, the nodes would wait for each other in a dead loop.

This commit fixes the issue by allowing node to boot
with local config if it does not have a lagging.
2023-11-08 15:06:47 +01:00
ieQu1 788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session
Refactor emqx_durable storage application and introduce learned topic structure storage
2023-11-08 14:27:40 +01:00
Thales Macedo Garitezi f5456135aa fix(bridge_v2_api): take status and error from bridge, not the connector
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-08 09:53:42 -03:00
Ilya Averyanov d061d64c70 fix(plugins): fix backward compatibility 2023-11-08 12:36:09 +03:00
Ivan Dyachkov 0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
ieQu1 a1cdbaa76d fix(ds): Address code review remarks 2023-11-08 04:31:27 +01:00
ieQu1 c030188eb7 chore(ds): Add rebar.config file to app/emqx_durable_storage 2023-11-08 03:04:19 +01:00
ieQu1 7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
ieQu1 74cb43f8b1 fix(ds): Add unique ID to the key 2023-11-08 03:04:19 +01:00
Andrew Mayorov 54951c273f feat(ds): mix safe cutoff into keymapper filter during iteration 2023-11-08 03:04:19 +01:00
Andrew Mayorov 7a94db25c3 fix(ds): don't iterate over incomplete epoch in bitmask lts storage 2023-11-08 03:04:19 +01:00
Andrew Mayorov 46d8301bc0 feat(emqx): expose timestamp function in `emqx_message`
So that the code that relies on it would not need to guess clock source
and precision.
2023-11-08 03:04:19 +01:00
ieQu1 38b7f516bc fix(ds): Change the directory where message database is stored 2023-11-08 03:04:19 +01:00
ieQu1 8e5dda40be fix(ds): Fix static checks 2023-11-08 03:04:19 +01:00
ieQu1 99329e1243 refactor(ds): Address review remarks 2023-11-08 03:04:19 +01:00
ieQu1 2de79dd9ac feat(ds): Replay QoS1 messages 2023-11-08 03:04:19 +01:00
ieQu1 e745e42093 test(ds): Explore full range of keys when testing ratchet function 2023-11-08 03:04:19 +01:00
ieQu1 87689890ff chore(ds): Fix linter and compilation warnings 2023-11-08 03:04:19 +01:00
ieQu1 465e8a90dd revert(ds): Remove change from the old protocol file 2023-11-08 03:04:19 +01:00
ieQu1 ef46c09caf feat(ds): Implement ratchet function for bitmask keymapper 2023-11-08 03:04:19 +01:00
ieQu1 164ae9e94a feat(ds): LTS bitfield storage passes all tests 2023-11-08 03:04:19 +01:00
ieQu1 56b6b176c2 fix(ds): LTS shall keeps the concrete topic indexes 2023-11-08 03:04:19 +01:00
ieQu1 7428e7037b feat(ds): Bitfield + Learned Topic Structure 2023-11-08 03:04:19 +01:00
ieQu1 ac91dbc58f feat(ds): Restore LTS trie from a dump 2023-11-08 03:04:19 +01:00
ieQu1 f1ab7c8a7c feat(ds): Add persist callback to LTS trie 2023-11-08 03:04:19 +01:00
ieQu1 c149e0e2df fix(ds): Pass topic filter to emqx_ds:make_iterator call 2023-11-08 03:04:19 +01:00
ieQu1 51a6f623fd refactor(ds): Split out bitfield keymapper to a different module 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 903b3863d1 chore(ps_ds): make persistent session module use new `emqx_ds` APIs 2023-11-08 03:04:19 +01:00
ieQu1 2972bf14ee refactor(ds): Implement create_generation gen_rpc storage layer call 2023-11-08 03:04:19 +01:00
ieQu1 6d65707d41 refactor(ds): Implement drop_db function 2023-11-08 03:04:19 +01:00
ieQu1 c6a721a7eb refactor(ds): Passthrough open_db and get_channels to storage layer 2023-11-08 03:04:19 +01:00
ieQu1 59d01dc823 refactor(ds): Implement emqx_ds:open_db 2023-11-08 03:04:19 +01:00
ieQu1 7095cb8583 refactor(ds): Refactor storage layer 2023-11-08 03:04:19 +01:00
ieQu1 c91df2f5cd refactor(ds): Create a prototype of replication layer 2023-11-08 03:04:19 +01:00
ieQu1 f1454bb57e feat(ds): learned topic structure 2023-11-08 03:04:19 +01:00
Thales Macedo Garitezi 7ab57824dc chore(ds): change return type of `storage_layer:next/{1,2}`
Part of https://emqx.atlassian.net/browse/EMQX-10942

The goal is to help make it clear to the caller of `next` what to do next: if the iterator
should still be used or if no new messages will ever come out of it.

From:

```erlang
-spec next(iterator()) -> {value, binary(), iterator()} | none | {error, closed}.
```

To:

```erlang
-spec next(iterator()) -> {ok, iterator(), [binary()]} | end_of_stream.

-spec next(iterator(), pos_integer()) -> {ok, iterator(), [binary()]} | end_of_stream.
```
2023-11-08 03:04:19 +01:00
Ilya Averyanov 75704513d8 fix(plugins): fix backward compatibility 2023-11-07 19:06:37 +03:00
Thales Macedo Garitezi c4c9d7ceb3 chore: prepara to tag `e5.3.1-alpha.4` 2023-11-07 13:04:46 -03:00
Thales Macedo Garitezi b255836cbd
Merge pull request #11890 from thalesmg/fix-kafka-unhealthy-r53-20231106
fix(resource): take error from action/connector before attempting query
2023-11-07 12:38:57 -03:00
Thales Macedo Garitezi 7dcdbc9e51 fix(resource): take error from action/connector before attempting query
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixes https://emqx.atlassian.net/browse/EMQX-11298
2023-11-07 10:04:04 -03:00
Stefan Strigler fa740a1927 docs: fix inconsistencies for bridge and connector operations
It's only `start` allowed for now. Also fixed some wordings.
2023-11-07 13:58:23 +01:00
Zaiming (Stone) Shi f19904d43c
Merge pull request #11877 from zmstone/1102-rename-kafka-to-config
Rename connector channel related configs in bridge_v2 to 'parameters'
2023-11-07 13:41:55 +01:00
Thales Macedo Garitezi 9742dae30d
Merge pull request #11889 from thalesmg/rename-bridgev2-to-action-apis-r53-20231106
chore: rename `bridges_v2` -> `actions` in the public facing APIs
2023-11-07 09:29:06 -03:00
JianBo He d76ed585b0 chore: correct the application vsn 2023-11-07 11:11:20 +08:00
JianBo He 7bd5579980 chore: fix the documentation generation and example conf checking 2023-11-07 11:09:39 +08:00
JianBo He 4ef156d69e chore: fix dialyzer warnings 2023-11-07 09:04:50 +08:00
JianBo He 0d9e0bd3fe chore: fix lot of running bugs 2023-11-07 08:52:10 +08:00
Zaiming (Stone) Shi bb8f80f256 test: add test case to pin bridge schema json version 0.1.0 2023-11-06 21:59:36 +01:00
Zaiming (Stone) Shi 9a26c03a5c refactor: unify top level field names for bridge v2 2023-11-06 21:45:08 +01:00
Thales Macedo Garitezi 2b8cf50a1d chore: rename `bridges_v2` -> `actions` in the public facing APIs
Fixes https://emqx.atlassian.net/browse/EMQX-11330

After feedback from Product team, we should rename `bridges_v2` to `actions` everywhere.
We'll start with the public facing APIs.

- HTTP API
- Hocon schema root key
2023-11-06 15:37:07 -03:00
Zaiming (Stone) Shi 6af6309301 chore: upgrade to hocon 0.39.19 2023-11-06 19:16:19 +01:00
Thales Macedo Garitezi f17b762596 chore: don't disable rule that references non-existent bridge
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
2023-11-06 10:09:14 -03:00
JianBo He 6f51b9f842 chore: ensure elvis pass 2023-11-06 12:31:54 +08:00
JianBo He 8871738756 chore: ensure emqx_gateway_ocpp tests pass 2023-11-06 12:26:58 +08:00
JianBo He bea0acd929 fix(ocpp): ensure ocpp gateway options works 2023-11-06 11:56:50 +08:00
Zaiming (Stone) Shi 5881e34d4e
Merge pull request #11874 from zmstone/1103-validate-bridge-type-and-name-at-entry
fix(bridge): do not allow dot in bridge name
2023-11-03 23:04:10 +01:00
Thales Macedo Garitezi 0ff4465c78
Merge pull request #11875 from thalesmg/fix-kafka-connecting-r53-20231103
fix(kafka_producer): don't return `disconnected` when there are connections issues while starting the bridge
2023-11-03 17:56:00 -03:00
Thales Macedo Garitezi a064eecb68
Merge pull request #11879 from thalesmg/fix-rule-api-downgrade-bridge-r53-20231103
fix(rules_api): downgrade bridge id
2023-11-03 17:26:09 -03:00
Zaiming (Stone) Shi 21a167e1a5
Merge pull request #11873 from zmstone/1103-unhide-nodelay-field
fix(kafka): do not hide nodelay field
2023-11-03 20:49:29 +01:00
Zaiming (Stone) Shi 600747b7e5 fix(bridge): do not allow dot in bridge name
also validate name at the API entry
2023-11-03 20:44:57 +01:00
Thales Macedo Garitezi 0ee74cd8ae fix(rules_api): downgrade bridge id
Fixes https://emqx.atlassian.net/browse/EMQX-11312
2023-11-03 15:57:13 -03:00
Thales Macedo Garitezi 4265ef66cc fix(kafka_producer): don't return `disconnected` when there are connection issues while starting the bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11284
Fixex https://emqx.atlassian.net/browse/EMQX-11298

We don't enforce the connection to be up when starting/creating the bridge, otherwise the
status will be `disconnected` for a possibly transient reason such as network issues or
Kafka broker restart.

Same applies for Azure Event Hub Producer bridge, as they share the same module.
2023-11-03 15:49:36 -03:00
Kjell Winblad dd8085d3a4 test(emqx_bridge_v2_api_SUITE): fix broken test case 2023-11-03 17:49:25 +01:00
Stefan Strigler 8a1b370a5e
Merge pull request #11876 from sstrigler/EMQX-11317-creating-a-bridge-for-a-non-existent-or-non-matching-connector-results-in-a-crash
handle errors from pre/post_config_update
2023-11-03 16:26:21 +01:00
Thales Macedo Garitezi 953d483c24
Merge pull request #11864 from thalesmg/test-enable-rule-check-deps-r53-20231101
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 11:01:38 -03:00
Stefan Strigler 45bfcea6ae
Merge pull request #11866 from sstrigler/EMQX-11294-e-5-3-1-swaggers-several-misdescriptions-of-ap-is
EMQX 11294 swaggers several misdescriptions of APIs
2023-11-03 14:55:46 +01:00
Stefan Strigler dc11008993 fix(emqx_bridge): handle errors from pre/post_config_update 2023-11-03 14:11:25 +01:00
Zaiming (Stone) Shi fcecd3e9d4 fix(kafka): do not hide nodelay field
otherwise:
it's returned in the GET response,
but it's not allowed in the POST request
2023-11-03 13:35:52 +01:00
Thales Macedo Garitezi c84e4a4187 fix(rule_engine): don't enable a rule that references non-existent bridge 2023-11-03 09:24:53 -03:00
Kjell Winblad 65ba0b9de4
Merge pull request #11869 from kjellwinblad/kjell/shared_con/start_feedback/EMQX-11304
fix(bridge_v2): start operation should return an error when unsuccessful
2023-11-03 13:16:32 +01:00
Zaiming (Stone) Shi 4dd054b0a2 test(emqx_bridge_kafka_impl_consumer_SUITE): fix flaky 2023-11-03 11:48:50 +01:00
Stefan Strigler dd11a30383 docs(emqx_bridge_azure_event_hub): remove timestamp from message template example 2023-11-03 11:18:54 +01:00
Stefan Strigler a89b0c8cf4 fix(emqx_bridge_azure_event_hub): make ssl mandatory (and give an example) 2023-11-03 11:18:38 +01:00
Kjell Winblad ebb5997a8c test(emqx_bridge_v2_api_SUITE): fix cluster test 2023-11-03 11:11:27 +01:00
JianBo He 7cab269e0b feat: port the ocpp gateway from version 4 2023-11-03 16:51:21 +08:00
Ivan Dyachkov fdbf439df8
Merge pull request #11859 from cisiqo/patch-1
fix content type returned by Stomp
2023-11-03 08:21:08 +01:00
Zaiming (Stone) Shi afe6f79a66 feat(bridge): add description field to bridge and connector 2023-11-03 06:53:42 +01:00
Kjell Winblad d14d0fbcb1 fix(bridge_v2 operations): better error message 2023-11-03 06:35:42 +01:00
Kjell Winblad 36640263ba fix(bridge_v2): start operation should return an error when unsuccessful
The bridge V2 HTTP API start operation should return a 400 error if the
start is unsuccessful.

The bridge V1 HTTP API compatibility layer for Bridge V2 should return a
400 error if the start or restart operation is unsuccessful.

This commit fixes the above and adds tests that checks this for the V2
HTTP API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11304
2023-11-02 18:28:22 +01:00
Andrew Mayorov 910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder
feat(tpl): split `emqx_placeholder` into a couple of modules
2023-11-02 22:50:05 +07:00
Andrew Mayorov 729c6edff6
chore(repub): simplify error handling in MQTT props templating 2023-11-02 20:13:32 +07:00
Andrew Mayorov f1847fe494
chore(tpl): drop unnecessary binding 2023-11-02 20:13:32 +07:00
Andrew Mayorov e521a9f5fc
fix(utils): denote `emqx_jsonish` follows access module behaviour
Defined in `emqx_template`.
2023-11-02 20:13:32 +07:00
Andrew Mayorov b5b6c3f8cc
fix(tpl): ensure full backward compat in basic connectors 2023-11-02 20:13:32 +07:00
Andrew Mayorov 02c1bd70b6
feat(tpl): factor out loose json concept into a separate module
Which is called `emqx_jsonish`. Also introduce an _access module_
abstraction to extract information from such data during rendering.
2023-11-02 20:13:31 +07:00
chengshq fbd27eda6a fix(Stomp): content type returned 2023-11-02 11:21:27 +00:00
Andrew Mayorov 69cfa740ea
fix(ruleeng): ensure full backward compatibility 2023-11-02 17:11:12 +07:00
Andrew Mayorov 75cc663786
chore(ruleeng): streamline application dependencies 2023-11-02 17:11:12 +07:00
Andrew Mayorov 8e4585d64f
chore: move template modules to `emqx_utils`
Even though most of the time these modules will be used by
connectors, there are exceptions (namely, `emqx_rule_engine`).
Besides, they are general enough to land there, more so given
that `emqx_placeholder` is already there.
2023-11-02 17:11:12 +07:00
Andrew Mayorov 343b679741
feat(tpl): make escaping mechanism more foolproof
Treat "${$}" as literal "$". This allows to template express
strings, for example, of the form "${some_var_value}" where
`some_var_value` is interpolated from bindings.
2023-11-02 17:11:11 +07:00
Andrew Mayorov f689d6c233
fix(tpl): ensure backward compat in `emqx_rule_engine`
Missing bindings in string templates will be rendered as "undefined",
as before. Rendering still assumes that missing binding with implicit
default (`undefined`) is an error.

This will also restore complete backward compat in `emqx_prometheus`.
2023-11-02 17:11:11 +07:00
Andrew Mayorov 1fcdfe991c
fix(tpl): add few missing `@doc`s + correct typespecs 2023-11-02 17:11:11 +07:00
Andrew Mayorov a9693eada7
fix(tpl): rename `trivial` -> `is_const`
This is clearer. Former naming was a bit misleading.
2023-11-02 17:11:11 +07:00
Andrew Mayorov 7bb995f0c6
feat(tpl): support `:n` SQL parameters 2023-11-02 17:11:11 +07:00
Andrew Mayorov dfb7faf6a8
fix(tpl): ensure rough backward compat in `emqx_prometheus`
But also emit warning when job name template renders with errors.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 49fba40ee7
fix(tpl): ensure backward compat with authz / authn templates
This commit leans heavy into discouraging the former approach where
only part of placeholders were interpolated, depending on `placeholders`
option.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 49f5325c67
feat(tpl): unify validations / errors var representations 2023-11-02 17:11:10 +07:00
Andrew Mayorov b812f9af5a
feat(tpl): use `emqx_connector_template` in `emqx_rule_engine` app 2023-11-02 17:11:10 +07:00
Andrew Mayorov e1bca5844f
feat(tpl): use `emqx_connector_template` in `emqx_prometheus` app 2023-11-02 17:11:10 +07:00
Andrew Mayorov 0538a77700
feat(tpl): use `emqx_connector_template` in `emqx_authn`, `emqx_authz`
This slightly changes semantics: now the attempt to create authenticator
with illegal bindings in templates will fail, instead of treating them
as literals. The runtime behaviour on the other hand should be the same.
2023-11-02 17:11:10 +07:00
Andrew Mayorov 35902dc72d
feat(tpl): switch basic connectors to `emqx_connector_template`
Also avoid `filename:join/2` in HTTP connector since it's both OS specific
and an overkill.
2023-11-02 17:11:09 +07:00
Andrew Mayorov 28d55d72ca
feat(tpl): split `emqx_placeholder` into a couple of modules
Located under `emqx_connector` for now. The APIs of the new modules
are slightly different from the old ones. The new ones are more
explicit in terms of error handling. Also copy the according testsuite
from `emqx_plugin_libs` and enrich it for better coverage.
2023-11-02 17:11:09 +07:00
Zaiming (Stone) Shi 73dd2f0ffd chore: bump version to e5.3.1-alpha.3 2023-11-02 10:36:41 +01:00
Zaiming (Stone) Shi f5818a86e3
Merge pull request #11860 from thalesmg/fix-delete-shared-bridgev2-r53-20231101
fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1
2023-11-02 10:30:47 +01:00
Zaiming (Stone) Shi a44f8eb583
Merge pull request #11861 from zmstone/1101-do-not-repeat-print-shell-warning
fix(shell): do not repeat print shell warnings
2023-11-02 10:28:56 +01:00
Zaiming (Stone) Shi 34ec7375ba
Merge pull request #11862 from kjellwinblad/kjell/shared_con/del_rules/EMQX-11293
fix(bridge_v2 API): optional cascading delete operation
2023-11-02 10:28:43 +01:00
Kjell Winblad ec2d339355
Merge pull request #11857 from kjellwinblad/kjell/shared_con/EMQX-11270
fix(bridge_v2): channels should not be removed when status is connecting
2023-11-01 16:46:47 +01:00
Ivan Dyachkov 5b964ef415
Merge pull request #11858 from id/1101-emqx-11288-fix-kafka-replaq-dir-conflict
emqx 11288 fix kafka replaq dir conflict
2023-11-01 16:45:02 +01:00
Thales Macedo Garitezi 4bea65bf97 fix(bridge_v2_api): don't disable schema check 2023-11-01 12:44:40 -03:00
Ivan Dyachkov 2dd5061643 fix(kafka): kafka bridge replaq dir conflict 2023-11-01 15:44:28 +01:00
Kjell Winblad 357b664c8d fix(bridge_v2): more fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:54 +01:00
Kjell Winblad b06d05eaac test(bridge_v2): fix test case after new API 2023-11-01 15:27:54 +01:00
Kjell Winblad 96d6c6db49 test(bridge_v2): emqx_bridge_v2_kafka_producer_SUITE fix after API change 2023-11-01 15:27:53 +01:00
Kjell Winblad edb1d37e67 chore(bridge_v2): make fixes thanks to PR comments from @thalesmg 2023-11-01 15:27:53 +01:00
Kjell Winblad 95f3b94ac3 fix(bridge_v2): channels should not be removed when status is connecting
This fixes so that channels are not removed from the resource state when
their status is connecting. This is needed for Kafka since Kafka's message
buffer is stored in the resource state.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11270
2023-11-01 15:27:53 +01:00
Kjell Winblad 1e935e9eb4 fix(bridge_v2 API): optional cascading delete operation
This commit makes the delete HTTP API operation for Bridge V2 behave in
the same way as in the Bridge V1 API.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11293
2023-11-01 15:03:18 +01:00
Zaiming (Stone) Shi c38725facd fix(shell): do not repeat print shell warnings 2023-11-01 14:44:18 +01:00
lafirest 17544dc410
Merge pull request #11852 from lafirest/feat/gbt_gw
feat(gbt32960): Port the GBT32960 gateway from v4
2023-11-01 21:03:18 +08:00
Thales Macedo Garitezi c03979a48a fix(bridge_v1): return 400 if trying to delete shared bridge via http api v1 2023-11-01 09:51:26 -03:00
firest 6b50fdcf9e chore(gbt32960): update changes 2023-11-01 19:35:44 +08:00
firest 30a72f557f fix(gateway): improve gateway schema modules
1. enhances the gateway name as an enum
2. make the schema more flexible and extensible without some hardcode
2023-11-01 18:55:36 +08:00
Kjell Winblad 10b2894b6e chore(bridge_v2_api): remove misplaced handling code 2023-11-01 11:31:17 +01:00
Zaiming (Stone) Shi ae760a4ca1
Merge pull request #11855 from zmstone/1031-perf-no-need-to-format-sys-message-if-no-bridge-matched
perf: no need to format event message if no bridge matched
2023-11-01 10:22:06 +01:00
Ivan Dyachkov aa8a6f2e26
Merge pull request #11853 from thalesmg/test-bridge-v1-compat-layer-r53-20231030
test(bridges): add bridge v1 compatibility layer test suite, and other fixes
2023-11-01 09:51:10 +01:00
Stefan Strigler 29683072a1 fix(emqx_connector): remove `stop` and `restart` operations 2023-11-01 09:18:50 +01:00
firest 5e314d4ef1 test(gbt32960): add test suites 2023-11-01 13:17:01 +08:00
Thales Macedo Garitezi 45a39d97c6 fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1
Fixes https://emqx.atlassian.net/browse/EMQX-11291
2023-10-31 16:39:47 -03:00
Andrew Mayorov 7092c75597
Merge pull request #11809 from keynslug/ft/EMQX-10808/file-secrets
feat(mqttbridge): support file-sourced secrets as passwords
2023-11-01 00:29:30 +07:00
Thales Macedo Garitezi b420b53075 fix(bridges_v2): check bridge name length before attempting atom conversion
Fixes https://emqx.atlassian.net/browse/EMQX-11289
2023-10-31 13:21:26 -03:00
Thales Macedo Garitezi 8eb822d898 test(bridges): add bridge v1 compatibility layer test suite 2023-10-31 13:21:01 -03:00
Zaiming (Stone) Shi db5cdb6742 perf: no need to format event message if no bridge matched 2023-10-31 16:48:10 +01:00
Stefan Strigler 3e9f48d10f fix: remove unneeded clause 2023-10-31 16:05:31 +01:00
Stefan Strigler 2a4eaf41ba docs: fix examples for PUT operations on bridge_v2 and connectors
Examples would show `type` and `name` properties in the request body, which is
not accepted by the schema.

Also fixes some minor inconsistencies in the example names of connectors and
bridges.
2023-10-31 15:10:11 +01:00
Thales Macedo Garitezi 84e78f5d2e fix(bridge_api): fix response status code for some operations 2023-10-31 10:40:56 -03:00
Thales Macedo Garitezi c90ca1ea53 fix(bridge_v1): always delete connector when deleting v1 bridge
Fixes https://emqx.atlassian.net/browse/EMQX-11287
2023-10-31 10:40:06 -03:00
Andrew Mayorov d278486416
fix(secret): dedicate a specific loader module for file secrets
To make code employing `emqx_secret` easier to follow.
2023-10-31 19:26:04 +07:00
firest b35cd57034 fix(gbt32960): make elvis happy 2023-10-31 19:04:21 +08:00
firest 3ec811e828 feat(gbt32960): Port the GBT32960 gateway from v4 2023-10-31 18:25:43 +08:00
Ivan Dyachkov b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Stefan Strigler 39f49d425c test(emqx_bridge): fix test for bridge_v2_type_to_connector_type 2023-10-31 08:34:05 +01:00
Stefan Strigler 63606922f2 refactor(emqx_bridge): only one place to generate same error 2023-10-31 08:34:05 +01:00
Stefan Strigler b256241650 test(emqx_bridge_azure_event_hub): fix legacy v1 bridge API tests 2023-10-31 08:34:05 +01:00
Stefan Strigler 176bbe88bc fix: use `_producer` for AEH bridge type 2023-10-31 08:34:03 +01:00
Kjell Winblad aea449306a
Merge pull request #11842 from SergeTupchiy/EMQX-11272-fix-kafka-bridge-v1-import
EMQX-11272 fix kafka bridge v1 import
2023-10-31 07:58:54 +01:00
JianBo He 21e4f918aa
Merge pull request #11833 from JimMoen/fix-shared-sub-topic-or-subscription-searching
Fix shared sub topic or subscription searching
2023-10-31 08:16:36 +08:00
Kjell Winblad 9d7ed31c3a
Merge pull request #11845 from kjellwinblad/kjell/shared_con/EMQX-11271
fix: crash when listing non bridge_v1 compatible bridge_v2
2023-10-30 20:29:36 +01:00
Zaiming (Stone) Shi c07cf9051e
Merge pull request #11844 from zmstone/1030-downgrade-bridge-type-for-old-api
1030 downgrade bridge type for old api
2023-10-30 19:47:24 +01:00
Zaiming (Stone) Shi 124d79a1ca
Merge pull request #11843 from thalesmg/test-kafka-producer-nits-r53-20231030
test(kafka_producer): minor adjustments to test suite
2023-10-30 18:27:10 +01:00
Zaiming (Stone) Shi 350e8433ab fix(bridge_v2): downgrade bridge type for GET api results 2023-10-30 18:17:35 +01:00
Serge Tupchii 0562585c8f fix(emqx_mgmt_data_backup): implement importer modules ordering
`emqx_bridge_v2` depends on `emqx_connector`, so connectors must be imported first.
2023-10-30 19:14:58 +02:00
Serge Tupchii bc8c291820 fix(emqx_bridge): add import_config/1 cb to emqx_bridge_v2 2023-10-30 19:14:58 +02:00
Serge Tupchii d94193ac15 fix(emqx_connector_schema): add kafka alias for kafka_producer 2023-10-30 19:14:58 +02:00
Serge Tupchii 0935bb6225 fix(emqx_connector): fix badarity error 2023-10-30 17:52:39 +02:00
Serge Tupchii d1cd5dd817 fix(emqx_mgmt_data_backup): upgrade raw conf before validating and importing 2023-10-30 17:52:39 +02:00
Kjell Winblad 8fb426e57f fix: crash when listing non bridge_v1 compatible bridge_v2
Fixes:
https://emqx.atlassian.net/browse/EMQX-11271
2023-10-30 16:32:04 +01:00
Thales Macedo Garitezi a60b96c5fd test(kafka_producer): minor adjustments to test suite
- Use `emqx_cth_suite`.
- Use `query_mode` matrix value when setting up bridge in a couple test cases.
2023-10-30 12:13:40 -03:00
Ilya Averyanov 3f6c09b195
Merge pull request #11780 from savonarola/1017-fix-pbkdf2-validation
fix(authn): fix pbkdf2 option validation
2023-10-30 16:37:37 +02:00
Stefan Strigler 5a6e55e7af fix(emqx_bridge_kafka): schema fixes for kafka_producer in bridge_v2 2023-10-30 14:59:06 +01:00
Zaiming (Stone) Shi f463eff02d chore: e5.3.1-alpha.2 2023-10-30 14:49:08 +01:00
Zaiming (Stone) Shi 0656b6be3c refactor(emqx_config): use dynamic callback to upgrade raw config 2023-10-30 14:49:08 +01:00
Kjell Winblad 9dc3a169b3 feat: split bridges into a connector part and a bridge part
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
zhongwencool e896957803
Merge pull request #11824 from zhongwencool/audit-log-reviewed
refactor: audit log review refactor
2023-10-30 17:51:15 +08:00
JimMoen 3a09fdc495
refactor: check match topic before do subscriptions query 2023-10-30 16:45:29 +08:00
JimMoen e9de7316b6
test: shared-sub topics/subscription api 2023-10-30 14:42:59 +08:00
JimMoen 814e22feb3
fix: topics/subscripton mgmt api searching 2023-10-30 14:42:47 +08:00
zhongwencool 0634ff61c0 fix: dialyzer warning 2023-10-30 09:57:30 +08:00
JimMoen a2015f37ae
Merge pull request #10976 from JimMoen/fix-shared-sub-unsub 2023-10-27 21:22:27 +08:00
lafirest e63602c4b0
Merge pull request #11811 from lafirest/fix/api_rbac
fix(rbac): for compatibility with old data schema, extend the existing field as an extra
2023-10-27 17:56:54 +08:00
JimMoen d563121284
refactor: move ?REDISPATCH_TO macro to emqx_mqtt.hrl 2023-10-27 17:54:14 +08:00
firest 4ba34f8f3e chore(rbac): fix CI errors & update change 2023-10-27 17:17:04 +08:00
JimMoen 3b5cc912e7
fix: add `redispatch_to` header to all msgs when deliver shared-sub
- to find correct SubOpts for shared-sub dispatch
- use previous key `redispatch_to` to ensure rolling upgrade compatibility
2023-10-27 16:15:23 +08:00
zhongwencool 22223dc536 fix: return 400 when audit log is disabled 2023-10-27 16:11:30 +08:00
zhongwencool 995948f0e8 refactor: remove seq from audit record 2023-10-27 14:07:19 +08:00
zhongwencool 9bb22507df fix: dont use transation on audit log 2023-10-27 11:41:12 +08:00
JimMoen 53383991d9
fix: rewrite #share{} and return #share{}, not formated $share/group/topic 2023-10-27 09:23:44 +08:00
JimMoen afec6fa2f6
fix: TopicFilter may modified by `client.subscribe` hook 2023-10-27 09:23:44 +08:00
JimMoen 802a36c670
fix: find SubOpts by shared_record, not deliver topic 2023-10-27 09:23:44 +08:00
JimMoen eaa5459509
chore: bump apps vsn 2023-10-27 09:23:43 +08:00
JimMoen 0ca725ff25
fix: deprecated `share` in `subopts` 2023-10-27 09:23:43 +08:00
JimMoen b5411da770
refactor: subscribe process to fix shared-sub 2023-10-27 09:23:43 +08:00
Andrew Mayorov 44b4205561
fix(secret): do not treat missing file secrets as config error
They are intended to be used mostly in the context of resources, which
have their own feedback mechanism: statuses, retries, etc.

Also turn the error into a throw exception, so that it can be
interpreted as a regular error condition, for example by the resource
manager.
2023-10-26 14:37:14 +07:00
firest cbfd02d1b0 refactor(ldap): merge the `ldap-bind` backend as a type for the `ldap` backend 2023-10-26 11:04:44 +08:00
Zaiming (Stone) Shi 97601171a5
Merge pull request #11818 from zmstone/1025-fix-flaky-test
test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked
2023-10-25 20:46:31 +02:00
Zaiming (Stone) Shi 76a8b0efd4 test: fix a flaky test case emqx_crl_cache_SUITE:t_revoked 2023-10-25 16:08:31 +02:00
William Yang f52b4a9e32 fix: gen_rpc tcp client port align with server port 2023-10-25 12:11:35 +02:00
William Yang 37dd44c3d7
Merge pull request #11813 from qzhuyan/fix/william/genrpc-ssl-port-align
Fix/william/genrpc ssl port align
2023-10-25 09:04:19 +02:00
William Yang 6346e0d28a fix(gen_rpc): ssl client port align with server port 2023-10-24 22:33:52 +02:00
firest ec41479633 feat(rbac): supports setting role in API bootstrap file 2023-10-24 23:03:18 +08:00
firest e175c213a1 fix(rbac): for compatibility with old data schema, extend the existing field as extra 2023-10-24 22:28:53 +08:00
Zaiming (Stone) Shi abcb3166f6 fix: use binary string for raw default values 2023-10-24 14:46:01 +02:00
firest 7a8a5926ab fix(rbac): adjust the role names 2023-10-24 19:01:12 +08:00
Stefan Strigler 6e12569260
Merge pull request #11797 from sstrigler/EMQX-1057-fix-authz-api-donot-return-200
fix: return 404 if built_in_database not configured as auth source
2023-10-24 12:08:53 +02:00
Andrew Mayorov 52f4519eeb
feat(mqttbridge): support file-sourced secrets as passwords 2023-10-24 16:24:52 +07:00
Andrew Mayorov 1c2f9321d1
feat(emqx): add file-sourced generic secrets
These secrets follow the same `emqx_secret` convention of 0-arity
functions. Also provide a simple HOCON schema module for use in
application schemas.
2023-10-24 16:24:51 +07:00
Stefan Strigler ea9228108b fix: use minirest filter 2023-10-24 09:57:08 +02:00
JimMoen 99fab8dc5d
chore: missing change log for duplicated apikey fixing 2023-10-24 09:44:39 +08:00
Thales Macedo Garitezi d392c9989f
Merge pull request #11793 from thalesmg/test-rule-engine-foreach-m-20231019
test(rule_engine): add test to exemplify `foreach` with json array payload
2023-10-23 13:43:33 -03:00
Stefan Strigler 4e0e755b28 fix: return 404 if built_in_database not configured as auth source 2023-10-23 16:26:41 +02:00
Ilya Averyanov 8c6f6b56b8 chore(tests): report more friendly about unset PROFILE variable 2023-10-23 13:48:42 +03:00
Ilya Averyanov edde661da3 fix(authn): fix pbkdf2 option validation 2023-10-23 10:26:11 +03:00
William Yang a9b42af8b3
Merge pull request #11796 from qzhuyan/fix/william/gen_rpc-driver
fix(gen_rpc): ensure client/server use same driver
2023-10-23 08:24:17 +02:00
JimMoen e6576951ef
test: cleanup duplicated apikey with different name 2023-10-20 22:33:26 +08:00
JimMoen d467289bb2
fix: avoid duplicated apikey from data import 2023-10-20 21:13:04 +08:00
William Yang dd5807b797 fix(gen_rpc): ensure client/server use same driver
Also bump to gen_rpc 3.2.1
2023-10-20 12:17:49 +02:00
zhongwencool ef692596f7 fix: don't crash when 401 and 403 unauthorize 2023-10-20 15:22:30 +08:00
zhongwencool 6a8b2dc1f9 fix: bad high frequency request name 2023-10-20 14:57:31 +08:00
zhongwencool c97fe796e3 chore: update emqx_audit's license 2023-10-20 14:47:24 +08:00
JimMoen f381961108 fix: macro EMQX_RELEASE_EDITION when `emqx` run as standalnoe app 2023-10-20 14:47:24 +08:00
zhongwencool 32c1f1cca6 chore: inlude emqx as emqx_audit's deps 2023-10-20 14:47:24 +08:00
zhongwencool 1d7aa9495a chore: make spellcheck happy 2023-10-20 14:47:24 +08:00
zhongwencool befc484544 chore: add changelog entry and fix xref warning 2023-10-20 14:47:24 +08:00
zhongwencool 141061c1e2 feat: add max_filter_size and ignore_high_frequency_request config 2023-10-20 14:47:24 +08:00
zhongwencool 926c804314 feat: add /audit http api to filter audit log 2023-10-20 14:47:24 +08:00
zhongwencool 8b5ec2a4d5 fix: xref failed 2023-10-20 11:46:31 +08:00
zhongwencool 281add1464 Merge branch 'master' into sync-master-to-release-54 2023-10-20 10:11:39 +08:00
lafirest 74442b0d31
Merge pull request #11766 from lafirest/feat/rbac
feat(api_key): add RBAC feature for the API key
2023-10-20 09:24:41 +08:00
Thales Macedo Garitezi 8c4beec6f0 test(rule_engine): add test to exemplify `foreach` with json array payload
Relates to https://emqx.atlassian.net/browse/EMQX-11174
2023-10-19 18:19:42 -03:00
Ilya Averyanov 0b4600c293 feat(auth): improve redis command parsing and validation 2023-10-19 23:01:35 +03:00
Thales Macedo Garitezi eda82caf65
Merge pull request #11791 from thalesmg/fix-coap-heartbeat-m-20231018
fix(coap): increase received packet counter for keepalive
2023-10-19 13:12:41 -03:00
Ilya Averyanov 8d82c30b00
Merge pull request #11771 from savonarola/1015-validate-bcrypt-schema-in-api
feat(authn): allow authn providers to define a separate schama for API
2023-10-19 15:34:34 +03:00
Ilya Averyanov 8f9ac6c1f3
Merge pull request #11776 from savonarola/1017-auth-cleanup-2
chore(authn): remove dead code
2023-10-19 15:28:35 +03:00
Thales Macedo Garitezi 5b9866f630 fix(coap): increase received packet counter for keepalive
Fixes https://emqx.atlassian.net/browse/EMQX-11193
Fixes https://github.com/emqx/emqx/issues/11779
2023-10-18 16:35:33 -03:00
zhongwencool 2388d36b09 fix: allow viewers to change their own passwords 2023-10-18 18:40:02 +08:00
Ilya Averyanov e1c8317779 chore(authn): remove dead code 2023-10-18 10:57:55 +03:00
JimMoen 1f8478623f
chore: bump esaml to support Entra ID && bump app.src 2023-10-18 10:01:34 +08:00
JimMoen b2948666fb
chore: add saml sso redirect login log 2023-10-18 10:01:34 +08:00
Ilya Averyanov c0ebaf9b6f
Merge pull request #11541 from savonarola/0829-duplicate-ft-responses
Support asynchronous mode of receiving responses to FT operations
2023-10-17 17:08:00 +03:00
Ilya Averyanov 6354f3b04f feat(authn): allow authn providers to define a separate schama for API 2023-10-17 13:19:11 +03:00
Zaiming (Stone) Shi 55618dd28a
Merge pull request #11731 from zmstone/1008-expose-file-transfer-host-conf
fix(api-docs): add file-transfer config to hot-config scope
2023-10-17 11:23:16 +02:00
firest 467a2bc9b1 test(rbac): add test cases for RBAC in REST API 2023-10-17 16:23:38 +08:00
firest 26ec860d96 feat(backup): add migration mechanism when import backup data 2023-10-17 16:23:32 +08:00
Zaiming (Stone) Shi dfa5909392
Merge pull request #11752 from zmstone/1011-change-default-rpc-module-for-mria
refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
2023-10-17 10:21:14 +02:00
firest db3915d472 fix(rbac): update changes && fix CI errors 2023-10-17 13:44:37 +08:00
Zaiming (Stone) Shi 162b901be6
Merge pull request #11763 from zmstone/1114-sync-release-53
Merge release-53 back to master
2023-10-16 21:42:02 +02:00
firest e095de7367 feat(api_key): add RBAC feature for the API key 2023-10-16 14:10:01 +08:00
JianBo He 16cc816bd3
Merge pull request #11757 from zhongwencool/trace-api-return-500
fix: 500 error response when downloading non-existent trace files
2023-10-16 13:40:18 +08:00
Zaiming (Stone) Shi 6eb3bb7cff Merge remote-tracking branch 'origin/release-53' into 1114-sync-release-53 2023-10-14 10:16:38 +02:00
Zaiming (Stone) Shi 7c022c2c6a refactor: change mria default rpc module from 'gen_rpc' to 'rpc'
Erlang distribution seems to outperform gen_rpc (unless gen_rpc clients
are scaled up, but this is not easy to achive for shard transport as it
may reorder events).
2023-10-14 10:13:31 +02:00
Ilya Averyanov cd2752117c
Merge pull request #11762 from savonarola/1013-fix-mnesia-authz-destroy
fix(mnesia authz): destroy authz records on mnesia authz destroy
2023-10-13 20:54:14 +03:00
Ilya Averyanov bde8800f2e fix(mnesia authz): destroy authz records on mnesia authz destroy 2023-10-13 18:29:45 +03:00
Ilya Averyanov 90a0c093bf
Merge pull request #11750 from savonarola/1011-redact-auth-http-request
Do not trace authn http request bodies
2023-10-13 17:25:28 +03:00
Ilya Averyanov 4f14e8df22 chore(source dir): split out postgresql connector 2023-10-13 16:58:57 +03:00
Thales Macedo Garitezi 7a6c756b7c
Merge pull request #11754 from thalesmg/fix-postgres-error-reason-log-m-20231011
fix(postgres): format unicode error messages from driver
2023-10-13 09:58:28 -03:00
Andrew Mayorov 0144ed9981
Merge pull request #11753 from keynslug/chore/smaller-ci-artifacts
chore(ci): try to reduce CI artifacts footprint
2023-10-13 16:56:47 +07:00
Andrew Mayorov 51c57a5f0a
fix(cass): avoid using aggregates in healthcheck query
Otherwise, this query will produce warning message per each
healthcheck, telling the user:
```
WARN  <...> - Aggregation query used without partition key
```
2023-10-13 16:50:19 +07:00
Andrew Mayorov 083e2da347
chore(bpapicheck): make some failures more user-friendly 2023-10-13 15:42:08 +07:00
Andrew Mayorov 0b9ac24c1e
fix(bpapicheck): stop depending on git index 2023-10-13 15:40:49 +07:00
lafirest 96c546c187
Merge pull request #11725 from lafirest/feat/os-ldap
chore: change the LDAP integration to opensource
2023-10-13 15:05:39 +08:00
Zaiming (Stone) Shi dd03be29a6
Merge pull request #11751 from zmstone/1011-refactor-move-emqx_license-app-to-apps
1011 refactor move emqx_license app to apps
2023-10-13 06:45:49 +02:00
firest 33ff5d5588 chore: update auth header file 2023-10-13 10:16:47 +08:00
Ilya Averyanov 68f31a9da2 fix(authn): do not trace authn requests 2023-10-12 16:32:16 +03:00
Ilya Averyanov 08795f559c
Merge pull request #11336 from savonarola/0724-trace-authz
Trace non-resultative authz calls
2023-10-12 14:51:15 +03:00
Andrew Mayorov 6413afd0bc
fix(gcpdev): restore original shard in mnesia tab definition
Changing the shard is not backward compatible, and leads to a crash
when upgrading from 5.2.x.
2023-10-12 17:30:07 +07:00
Ilya Averyanov 4ecd5e17a2 chore(authz): trace non-resultative authz calls to backend modules 2023-10-12 12:29:39 +03:00
firest 3e658b3da9 chore: update changes 2023-10-12 17:15:47 +08:00
firest 9b4def885a chore: change the LDAP integration to opensource 2023-10-12 16:37:32 +08:00
zhongwencool 1021088f5f fix: 500 error response when downloading non-existent trace files 2023-10-12 15:23:23 +08:00
Thales Macedo Garitezi b07dddd49e fix(postgres): format unicode error messages from driver
Fixes https://emqx.atlassian.net/browse/EMQX-11024

Sample error:

```
{error, error, <<"42501">>, insufficient_privilege,
<<229,175,185,232,161,168,32,109,113,116,116,95,117,115,101,114,32,230,157,131,233,153,144,228,184,141,229,164,159>>,
[]}
```
2023-10-11 17:56:12 -03:00
Ilya Averyanov 534c9bdc13 feat(ft): add additional operation status report channel 2023-10-11 19:51:36 +03:00
Zaiming (Stone) Shi d9859b6b49
Merge pull request #11749 from thalesmg/test-fix-flaky-http-suite-r53-20231011
test: attempt to fix flaky http bridge suite
2023-10-11 16:07:17 +02:00
William Yang e6358ca18c
Merge pull request #11747 from qzhuyan/dev/william/quicer-0.0.202
bump to quicer 0.0.202
2023-10-11 15:43:47 +02:00
Zaiming (Stone) Shi e069680bca refactor: move emqx_license app from lib-ee to apps 2023-10-11 15:41:35 +02:00
Thales Macedo Garitezi ab662f506d test: attempt to fix flaky http bridge suite
```
 =CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
  crasher:
    initial call: ehttpc:init/1
    pid: <0.23358.2>
    registered_name: []
    exception error: bad argument
      in function  persistent_term:get/1
         called as persistent_term:get({emqx_bridge_http_SUITE,
                                        do_t_async_retries,attempts})
         *** argument 1: no persistent term stored with this key
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-11 09:13:25 -03:00
Ilya Averyanov 57655854d1
Merge pull request #11743 from savonarola/1010-auth-leftovers
chore(auth): cleanup code
2023-10-11 15:09:07 +03:00
William Yang a69c53455f fix: bump to quicer 0.0.202
bring in urgent fixs from msquic 2.2.3
2023-10-11 13:12:32 +02:00
zhongwencool cc392cadb8
chore: update apps/emqx_retainer/src/emqx_retainer_mnesia_cli.erl
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-10-11 18:40:09 +08:00
Ilya Averyanov 03ae5bf3c8 chore(auth): cleanup code 2023-10-11 13:13:50 +03:00
zhongwencool f4a88f717b feat: support retainer Start Length command 2023-10-11 11:47:37 +08:00
zhongwencool 8e7ba16c3a fix: nothing show when run clients list command 2023-10-11 11:47:15 +08:00
zhongwencool 574dc2f243 fix: observer load command crash when loading noexist module 2023-10-11 10:36:33 +08:00
Zaiming (Stone) Shi a2e86c67db
Merge pull request #11733 from keynslug/fix/EMQX-10827/sesson-takeover
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
2023-10-10 19:03:29 +02:00
Zaiming (Stone) Shi 0f07f678e7
Merge pull request #11742 from zmstone/1010-sync-release-53-to-master
1010 sync release-53 to master
2023-10-10 18:56:23 +02:00
Thales Macedo Garitezi 196b24c0cb
Merge pull request #11741 from thalesmg/test-fix-flaky-http-test-m-20231010
test: attempt to fix flaky http bridge suite
2023-10-10 13:52:08 -03:00
Thales Macedo Garitezi ee45145fb5
Merge pull request #11724 from thalesmg/fix-kprodu-sync-metrics-m-20231006
fix({kafka,pulsar}_producer): correctly handle metrics for connectors that have internal buffers
2023-10-10 12:13:33 -03:00
Thales Macedo Garitezi ec588f94e2 test: attempt to fix flaky http bridge suite
```
 =CRASH REPORT==== 10-Oct-2023::05:35:30.911371 ===
  crasher:
    initial call: ehttpc:init/1
    pid: <0.23358.2>
    registered_name: []
    exception error: bad argument
      in function  persistent_term:get/1
         called as persistent_term:get({emqx_bridge_http_SUITE,
                                        do_t_async_retries,attempts})
         *** argument 1: no persistent term stored with this key
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-2-'/0 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 697)
      in call from emqx_bridge_http_SUITE:'-do_t_async_retries/3-fun-4-'/6 (/__w/emqx/emqx/apps/emqx_bridge_http/test/emqx_bridge_http_SUITE.erl, line 705)
```
2023-10-10 09:50:53 -03:00
Ivan Dyachkov bfb2218392 chore: fix release version 2023-10-10 14:50:49 +02:00
Thales Macedo Garitezi cf2075d7d8 chore: remove mention of `is_buffer_supported` from typespec 2023-10-10 09:49:18 -03:00
Zaiming (Stone) Shi 143673b703 chore: bump release version to 5.3.1-alpha.1 2023-10-10 14:35:46 +02:00
Zaiming (Stone) Shi 0a9e1dfa88
Merge pull request #11734 from zmstone/1002-support-ipv6-only-clustering
Make IPv6 clustering options configurable
2023-10-10 14:29:52 +02:00
Andrew Mayorov 5fff2ffe45
test(kafka): try to stabilize `t_dynamic_mqtt_topic/1` testcase 2023-10-10 17:18:24 +07:00
Andrew Mayorov 9b573834f9
Merge pull request #11736 from keynslug/fix/eval-erl-audit
fix(audit): handle abstract forms when logging `eval_erl`
2023-10-10 15:49:19 +07:00
Zaiming (Stone) Shi 2626d793a7 test: try to resolve schema module from PROFILE in tests 2023-10-10 10:24:10 +02:00
Zaiming (Stone) Shi 808237364c fix(api-docs): add file-transfer config to hot-config scope 2023-10-10 10:24:10 +02:00
Zaiming (Stone) Shi e7e696cd66 feat(rpc): add ipv6_only config 2023-10-10 10:05:14 +02:00
firest e07937a3ef fix(ldap): escape the escape character (\) 2023-10-10 13:49:36 +08:00
Andrew Mayorov 2a291dfd27
fix(audit): handle abstract forms when logging `eval_erl` 2023-10-10 12:31:19 +07:00
Andrew Mayorov 522302fee1
fix(cm): bring back pre-v5.3.0 compat in `takeover_session_begin/1`
Which was accidentally broken in bf164175.
2023-10-10 01:34:10 +07:00
Zaiming (Stone) Shi 1e93d2f1fc chore: upgrade to ekka 0.15.16 gen_rpc 3.2.0
* ekka 0.15.16 supports 'inet6_tls' as ekka.proto_dist
* gen_rpc 3.2.0 supports true | false as gen_rpc.ipv6_only
2023-10-09 20:03:21 +02:00
Zaiming (Stone) Shi 497e08448d feat(cluster): support ipv6 and tls on ipv6 for clustering
Made possible to configure inet6_tls for Erlang distribution
Also, added support to configure ipv6 listener for gen_rpc
2023-10-09 20:03:21 +02:00
Zaiming (Stone) Shi 03d8e06ff7 chore(emqx_rule_funcs): regroup export functions 2023-10-09 20:03:21 +02:00
Thales Macedo Garitezi d6781efee2 fix(resource): change how buffer workers are started 2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 902b1d6ec5 fix(pulsar_producer): use `simple_async_internal_buffer` query mode for Pulsar
Since it has internal buffering, it necessitates the same fix as Kafka producer.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi eebfb44f72 fix(resource): create `simple_async_internal_buffer` query mode for bridges with internal buffering
Since authn/authz backends also use simple async/sync queries, we may want to avoid them
calling the connector when it's not connected.
2023-10-09 15:02:25 -03:00
Thales Macedo Garitezi 79cf0a2ced fix(kafka_producer): correctly handle metrics for connector that have internal buffers
Fixes https://emqx.atlassian.net/browse/EMQX-11086

There’s currently a metric inconsistency due to the internal buffering nature of Kafka
Producer (wolff).

We use simple_sync_query to call the Kafka Producer bridge.  If that times out, the call
is accounted as failed, even though the message is buffered in wolff and later sent
successfully.
2023-10-09 15:02:25 -03:00
Zaiming (Stone) Shi c60915293a
Merge pull request #11634 from savonarola/0919-auth-refactor
Auth refactor
2023-10-09 19:51:26 +02:00
lafirest 8f4cdc3fcf
Merge pull request #11732 from lafirest/test/sso_cli
test(ldap): add test suite for SSO CLI
2023-10-09 20:35:23 +08:00
Zaiming (Stone) Shi 9ab49a7ae3 Merge remote-tracking branch 'origin/master' into 0919-auth-refactor 2023-10-09 14:18:10 +02:00
lafirest 2c7e5eb1cb
Merge pull request #11728 from lafirest/fix/ldap_filter
fix(ldap): improve the filter lex && parse
2023-10-09 18:24:57 +08:00
firest e3550fc07b test(ldap): add test suite for SSO CLI 2023-10-09 16:05:56 +08:00
zhongwencool 0c89b6b213 docs: swagger api tags should camel case 2023-10-09 15:21:51 +08:00
firest b2a6724dc2 fix(ldap): improve the filter lex && parse
1. auto escape special chars in the filter
2. fix a bug that the value can't be `dn`
2023-10-08 18:39:20 +08:00
zhongwencool 482e82f914 fix: cli's args in audit log should be array 2023-10-07 17:38:02 +08:00
Thales Macedo Garitezi 34186fcc74 fix(kafka_producer): send messages to wolff producer to buffer even when connector is in `connecting` state
Fixes https://emqx.atlassian.net/browse/EMQX-11085

Messages would not be sent to wolff if the connection was down, so they were effectively lost.
2023-10-06 11:43:29 -03:00
Ilya Averyanov 398a62031e chore: update app versions 2023-10-05 13:41:50 +03:00
Ilya Averyanov 5dff36474d chore(auth): get rid of hardcoded schema modules in auth 2023-10-05 13:41:50 +03:00
Ilya Averyanov c2c56ba481 chore(auth): update tests 2023-10-05 13:41:50 +03:00
Ilya Averyanov 1eb75b43c4 chore(auth): split emqx_authn and emqx_authz apps 2023-10-05 13:41:50 +03:00
Zaiming (Stone) Shi fd75dc895d fix(authn): "authentication" importance should not be "hidden" 2023-10-03 11:18:21 +02:00
Zaiming (Stone) Shi ca8da5723a
Merge pull request #11704 from zmstone/0928-sync-release-53
0928 sync release 53
2023-10-02 21:57:32 +02:00
Zaiming (Stone) Shi 3f6e0e890b chore: bump emqx_durable_storage and emqx_license app vsn 2023-10-02 20:13:09 +02:00
Thales Macedo Garitezi 2358d67908 refactor: move session stuff from `emqx_ds` to `emqx_persistent_session_ds`
Part of https://emqx.atlassian.net/browse/EMQX-10942
2023-09-29 18:00:24 -03:00
Zaiming (Stone) Shi bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Zaiming (Stone) Shi ed5b456d62 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 18:24:44 +02:00
Zaiming (Stone) Shi a852400fb3 chore: add 5.3.bpapi 2023-09-29 16:34:04 +02:00
Zaiming (Stone) Shi fe01aaff4f chore: prepare for e5.3.0 promotion 2023-09-29 13:32:57 +02:00
Zaiming (Stone) Shi ce5bd0a3ce Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-29 11:36:32 +02:00
Zaiming (Stone) Shi c64e599e81 docs: document how to retrieve peercert 2023-09-29 10:33:57 +02:00
Zaiming (Stone) Shi dc147fd310 fix(rule-engine): console action has no args field 2023-09-29 10:33:33 +02:00
Zaiming (Stone) Shi 1177a32310 chore: bump version to 5.3.0-rc.2 2023-09-29 09:49:27 +02:00
Zaiming (Stone) Shi 6891234390 chore: return simplified error reason for less logging 2023-09-29 09:48:15 +02:00
Zaiming (Stone) Shi c2d750aa09 fix(resource): redact query args in exception log 2023-09-29 09:20:42 +02:00
Zaiming (Stone) Shi 02ef854f0f fix(ldap): no crash when no query result is empty list 2023-09-29 08:54:41 +02:00
Zaiming (Stone) Shi b59a7ff2dd
Merge pull request #11706 from zmstone/0928-fix-ldap-sso-logging-level
0928 fix ldap sso logging level and reject muti-match results
2023-09-29 02:40:37 +02:00
Zaiming (Stone) Shi 4a4730ad46 fix(ldap): handle invalidCredentials in ldap authn 2023-09-29 00:51:05 +02:00
Zaiming (Stone) Shi 9ee2cb9c79 fix(ldap): return unrecoverable_error if more than on match found 2023-09-28 23:58:34 +02:00
Zaiming (Stone) Shi cc5dab1dc7 chore: fix code style 2023-09-28 21:29:59 +02:00
Zaiming (Stone) Shi b267fc2588 chore: bump release version to 5.3.0 2023-09-28 21:22:33 +02:00
Zaiming (Stone) Shi 922d5a9a83 fix(ldap): do not allow multi-matches to proceed
if ldap query returns more than on match
we should reject the auth request instead of picking
the first one
2023-09-28 21:20:50 +02:00
Zaiming (Stone) Shi d858f8af39 test: fix openldap docker runs 2023-09-28 18:40:03 +02:00
Zaiming (Stone) Shi b28e781c50 fix(ldap-sso): do not log error level when invalid user credentials 2023-09-28 18:37:11 +02:00
Zaiming (Stone) Shi 36f3052be1
Merge pull request #11698 from zmstone/0928-disable-audit-log-by-default
fix(audit): disable audit log by default
2023-09-28 17:28:23 +02:00
Zaiming (Stone) Shi 39820be5ff
Merge pull request #11703 from lafirest/fix/ldap_bind
fix(ldap): use the search result as bind target
2023-09-28 14:48:53 +02:00
Zaiming (Stone) Shi 1a13b2ac56 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-28 13:53:03 +02:00
firest 43ea367df8 fix(ldap): use the search result as bind target 2023-09-28 18:49:00 +08:00
Zaiming (Stone) Shi 98409c9b1e
Merge pull request #11702 from zmstone/0928-minor-chore
0928 minor chore
2023-09-28 12:39:55 +02:00
Zaiming (Stone) Shi 26cadb2248 fix(audit): disable audit log by default 2023-09-28 12:36:14 +02:00
Zaiming (Stone) Shi b14425d865 fix(sso/saml): ensure lower case http header name 2023-09-28 11:40:21 +02:00
Zaiming (Stone) Shi 7ddcd71412 docs(ssl_dist.conf): update doc link 2023-09-28 11:16:18 +02:00
firest dca8fdb17f fix(resource): respect the start_timeout 2023-09-28 16:36:41 +08:00
ieQu1 b77e5e880a
Merge pull request #11697 from ieQu1/dev/gen-rpc-3.1.1
chore(gen_rpc): Bump version to 3.1.1
2023-09-28 10:03:36 +02:00
JianBo He 6d2adfc259 chore(audit): emit the message field 2023-09-28 11:04:42 +08:00
JianBo He 859b122cdd chore: format codes 2023-09-28 10:25:37 +08:00
JianBo He b52e4ac99d chore: update apps/emqx_machine/src/emqx_restricted_shell.erl
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-28 10:25:37 +08:00
JianBo He bdf24d0ec8 chore: remove the `unauthorized` type for `from` field 2023-09-28 10:25:37 +08:00
JianBo He 71acf121ba chore(audit): distinguish requests from rest_api or dashboard 2023-09-28 10:25:37 +08:00
JianBo He dd9938114c chore(audit): add from field 2023-09-28 10:25:37 +08:00
JianBo He ece7d5b52a
Merge pull request #11690 from lafirest/fix/ldap_parse_cfg
fix(ldap): improve the LDAP `parse_config` function
2023-09-28 10:24:52 +08:00
lafirest b0d86eecd6
Merge pull request #11691 from lafirest/fix/sso_ssl
fix(sso): support for SSL update && ensure update is atomic
2023-09-28 10:22:34 +08:00
firest 57781d0544 fix(ldap): remove the parse_config, it never work 2023-09-28 09:38:16 +08:00
firest 9dee2dc31e fix(sso): clear last error first before update && fix the `running` 2023-09-28 08:56:16 +08:00
ieQu1 0aa3ccdd65 feat(gen_rpc): Add schema for the TLS versions and cipher suites 2023-09-28 00:36:59 +02:00
Zaiming (Stone) Shi afdda107af fix(logger): json format log encode binary list as string array 2023-09-27 23:40:01 +02:00
Zaiming (Stone) Shi 6f7a4344dc fix: do not gc sso saml SP singing keys 2023-09-27 23:19:39 +02:00
Zaiming (Stone) Shi 34367fc4ec fix(audit_log): pretty print shell args 2023-09-27 23:19:39 +02:00
Zaiming (Stone) Shi bb49914fd6 fix(sso): add convet_certs callback for sso backends
must convert certs in pre_config_update so the cert path refernces
are stored in raw config, otherwise the files might get gc:ed
2023-09-27 22:41:39 +02:00
Zaiming (Stone) Shi 45caa3bf01 fix(sso): make sp_private_key sensitive
so it will not be logged
2023-09-27 21:27:59 +02:00
Zaiming (Stone) Shi c8cbbff044 fix(logger): no need for special handling of empty string
when formating json logs, there is no need to handle empty strings
special, already covered by unicode handling
2023-09-27 21:26:47 +02:00
ieQu1 3c37f19105 chore(gen_rpc): Bump version to 3.1.1 2023-09-27 21:09:00 +02:00
firest 66d2107007 fix(sso): refactor update logic 2023-09-28 00:09:09 +08:00
Zaiming (Stone) Shi bd3277c51b
Merge pull request #11683 from zmstone/0925-test-refactor-emqx-rpc-test
test: refactor emqx_rpc unit tests
2023-09-27 15:46:27 +02:00
firest 08ad09a68f fix(sso): refactor backen update logic
1. valid config always can update successfully
2. the `running` endpoint only return successfully created backend
3. enhancement of the `/sso` endpoint, and will check is the resource online
2023-09-27 20:53:10 +08:00
Zaiming (Stone) Shi ea8d54fd8b test: ensure atom exists in test module 2023-09-27 12:58:06 +02:00
Zaiming (Stone) Shi 6e8c73258f test: refactor emqx_rpc unit tests 2023-09-27 12:58:06 +02:00
JimMoen c9194cd6b2
fix(saml_sso): donot load IDP metadata when disabling saml 2023-09-27 18:46:24 +08:00
JimMoen af9e87c025
fix: saml callback should check saml state 2023-09-27 18:34:48 +08:00
firest b2699c687b fix(sso): support for SSL update && ensure update is atomic
1. support update SSL key and cert files
2. increase connection timeout
3. ensure the update is atomicity, everything will be consistent
2023-09-27 15:48:11 +08:00
Andrew Mayorov 1d0e789e4d
Merge pull request #11679 from keynslug/ft/EMQX-10942/custom-session-timers
feat(session): add custom session timers mechanism
2023-09-26 19:09:53 +03:00
firest 0c33df3912 fix(ldap): improve the LDAP `parse_config` function 2023-09-26 23:28:37 +08:00
JimMoen 3fa18d6935
fix: lower sso saml redirect http header 2023-09-26 23:25:59 +08:00
JianBo He 6a9bb7c3ae
Merge pull request #11681 from zhongwencool/audit-log-fix-2
fix: don't need to change audit log's level
2023-09-26 23:12:28 +08:00
lafirest ac5eb5bc29
Merge pull request #11687 from lafirest/fix/sso_timeout
fix(sso): Handle backend update timeout and fix create errors
2023-09-26 22:43:38 +08:00
JianBo He 1a96a5990b
chore: Update apps/emqx_enterprise/src/emqx_enterprise_schema.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 21:44:40 +08:00
firest 403714d44e fix(sso): Handle backend update timeout and fix create errors
1. correctly handle the timeout when call update on a backend
2. fix that config update always returns success
3. do not ignore start failures and ensure start is atomic
2023-09-26 21:43:30 +08:00
Thales Macedo Garitezi aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
Andrew Mayorov bce2142df8
test(session): add custom session timers testcase 2023-09-26 16:22:06 +03:00
Zaiming (Stone) Shi 4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
Thales Macedo Garitezi fdcd73e20c fix(file_logger): change file logger type depending on rotation size
Fixes https://emqx.atlassian.net/browse/EMQX-11036

From `logger_disk_log_h:open_disk_log`:

```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
    case filelib:ensure_dir(File) of
        ok ->
            Size =
                if Type==halt -> MaxNoBytes;
                   Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
                end,
            Opts = [{name,   Name},
                    {file,   File},
                    {size,   Size},
                    {type,   Type},
                    {linkto, self()},
                    {repair, false},
                    {format, external},
                    {notify, true},
                    {quiet,  true},
                    {mode,   read_write}],
            case disk_log:open(Opts) of
```

Affects all file loggers (audit included):

```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
                                                                             {badarg,
                                                                              [{size,
                                                                                {infinity,
                                                                                 10}},
                                                                               {type,
                                                                                wrap},
                                                                               {linkto,
                                                                                <0.1952.0>},
                                                                               {repair,
                                                                                false},
                                                                               {format,
                                                                                external},
                                                                               {notify,
                                                                                true},
                                                                               {quiet,
                                                                                true},
                                                                               {mode,
                                                                                read_write}]}}
```
2023-09-26 09:04:44 -03:00
Ilya Averyanov 14983ec14a chore(hooks): validate hookpoints and document hook callbacks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
JianBo He 5e6397b843 chore: remove result field for CLI audit logs 2023-09-26 19:24:35 +08:00
JianBo He a73c3b8e1e Merge remote-tracking branch 'ce/release-53' into audit-log-fix-2 2023-09-26 19:05:47 +08:00
JianBo He e9785a6863
Merge pull request #11680 from thalesmg/audit-not-on-ce-r53-20230925
fix(audit): only support audit log on enterprise edition
2023-09-26 18:57:22 +08:00
Zaiming (Stone) Shi 7d810c2107 test: fix test case match pattern 2023-09-26 11:40:30 +02:00
Zaiming (Stone) Shi d31bfc70fb
Merge pull request #11659 from zhongwencool/fix-listener-ssl-create-500
fix: create ssl listener return 500 crash
2023-09-26 11:38:07 +02:00
lafirest 8cc626d33f
Merge pull request #11686 from lafirest/fix/sso_user_backend
fix: fix update SSO user && improve SSO user deletion commnad
2023-09-26 16:04:51 +08:00
firest 56917fee5f fix(sso): Improved SSO user deletion command 2023-09-26 14:15:41 +08:00
firest d1af81c86e fix(sso): fix the failure to update SSO users 2023-09-26 12:14:45 +08:00
firest 8135e27586 fix: redact sensitive data in SSO and LDAP 2023-09-26 11:06:47 +08:00
Thales Macedo Garitezi 17206f8c75
Merge pull request #11662 from thalesmg/port-scan-check-proto-dist-m-20230922
chore: check ekka proto dist module type when resolving node address
2023-09-25 17:01:43 -03:00
Thales Macedo Garitezi 5d212e1086 fix(audit): only support audit log on enterprise edition
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
Thales Macedo Garitezi 0498e59c45 test: fix flaky test 2023-09-25 13:35:17 -03:00
zhongwencool 95060302fd fix: don't need to change audit log's level 2023-09-26 00:28:47 +08:00
Thales Macedo Garitezi ff7f37ccf5 test(cth): allow defining schema to load for app 2023-09-25 13:22:41 -03:00
Andrew Mayorov b1f144ab8b
feat(session): add custom session timers mechanism
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
lafirest 1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
Zaiming (Stone) Shi 5e6996dc05 refactor: log formatter format mfa+line as m:f/a(line)
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 6a557980e4 refactor(logger): ensure JSON log field order
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 5f45ba50ff refactor: delete log formatter config for audit log handler
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 1fed38c248 fix(logger): write 'json' format logs as JSON 2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi aea1e80290 feat: add 'format' as alias for log 'formatter' 2023-09-25 16:55:22 +02:00
William Yang 9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify 2023-09-25 15:48:01 +02:00
Thales Macedo Garitezi 806017ef90 chore: check ekka proto dist module type when resolving node address
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917

Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
firest cfdb25b213 fix(sso): updates the SSO backend when the `[dashboard]` has updated 2023-09-25 20:30:47 +08:00
Ivan Dyachkov f1bc3b68b6
Merge pull request #11674 from id/0925-e5.3.0-alpha.2
e5.3.0 alpha.2
2023-09-25 14:21:46 +02:00
JianBo He 6f29bbf997
Merge pull request #11673 from HJianBo/redact_audit_log_from_cli
Redact audit log from CLI
2023-09-25 20:03:45 +08:00
Andrew Mayorov 81cf619f07
fix(ftconf): also mark `secret_access_key` key as sensitive 2023-09-25 14:52:57 +03:00
William Yang 36d3a3a524 fix: bump to quicer 0.0.201
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
William Yang 22193d273a fix(quic): ignore undefined cacertfile 2023-09-25 12:57:55 +02:00
Ivan Dyachkov 8c93c79b44 chore: e5.3.0-alpha.2 2023-09-25 12:52:45 +02:00
JimMoen 7d58f6c61e
Merge pull request #11668 from JimMoen/saml-login-redirect
fix: saml login acs redirect to dashboard overview
2023-09-25 18:11:03 +08:00
JianBo He b970a34ee0 chore: redact some audit logs from CLI 2023-09-25 18:06:39 +08:00
JianBo He 89e15e9134
Merge pull request #11665 from HJianBo/typo-fixes
Typo fixes
2023-09-25 17:23:40 +08:00
lafirest 1cd9df6461
Merge pull request #11669 from lafirest/fix/dashboard_cli
fix(sso): refactor the `admins` CLI
2023-09-25 16:57:18 +08:00
firest e63d484632 fix(sso): move the config path of the SSO feature to `dashboard.sso` 2023-09-25 15:23:52 +08:00
Ivan Dyachkov e1f1c64ecd Merge branch 'release-53' into 0925-sync-r53-to-master 2023-09-25 08:12:45 +02:00
firest e94192d1fa fix(sso): refactor the `admins` CLI
1. revert the opensource version
2. allow delete the SSO user via CLI
2023-09-25 13:53:01 +08:00
firest 4f4868a46c fix(sso): Disable access to `logout` endpoint by the `API key` 2023-09-25 10:31:38 +08:00
JianBo He 30862a94c6 chore: typo fixes 2023-09-24 20:19:05 +08:00
JimMoen ad4fadc2fa
fix: saml login acs redirect to dashboard overview 2023-09-23 17:29:02 +08:00
JianBo He f8d06614c0 chore: fix dialyzer warnings 2023-09-23 07:34:04 +08:00
JimMoen 1dddccb448
fix(saml): cert files cleanup when destroy 2023-09-22 22:49:08 +08:00
JimMoen 80a6c1150d
fix(saml): saml login reply role `viewer` as default 2023-09-22 22:39:02 +08:00
JimMoen cc3e4e4dc5
fix(saml): drop cert and key content and return path 2023-09-22 22:37:04 +08:00
JimMoen 6349cd3910
fix(saml): sp sign request 2023-09-22 21:57:50 +08:00
JimMoen 2a8f3f9eaa
fix: saml xml metedata format 2023-09-22 21:50:43 +08:00
JimMoen a318ad486a
refactor: behavior login/2 use all http request 2023-09-22 21:50:43 +08:00
JianBo He 9181ec844f
chore: split out sso_saml_api module 2023-09-22 21:50:43 +08:00
JianBo He df94426ee3
chore: make static_check happy 2023-09-22 21:50:42 +08:00
JianBo He ec0894ca0b
chore: update esaml vsn 2023-09-22 21:50:42 +08:00
JianBo He 4a26f63bd6
chore: fix bugs 2023-09-22 21:50:41 +08:00
JianBo He 47badc3181
chore: make dialyzer happy 2023-09-22 21:49:12 +08:00
JianBo He 1c78c6bf6d
chore: fix 500 crashes when backend not existed 2023-09-22 21:49:12 +08:00
JimMoen b4fb5196cb
fix(sso): SSO management API 500 2023-09-22 21:49:12 +08:00
JimMoen bba5cc44a8
fix: keep same API path style 2023-09-22 21:49:12 +08:00
JimMoen 8300cd42d4
fix: acl url ignore auth check 2023-09-22 21:49:12 +08:00
JimMoen 44836ef5ee
chore: bump esaml vsn to v1.1.1 2023-09-22 21:49:12 +08:00
JimMoen 13666fa9f9
refactor: avoid dynamic call 2023-09-22 21:49:12 +08:00
JimMoen c9e0d4fc30
feat: saml integration for dashboard sso 2023-09-22 21:49:10 +08:00
JimMoen d9466eef63
chore: fix Dashboard RBAC license and rebar.config 2023-09-22 21:19:45 +08:00
Zaiming (Stone) Shi fd932c9bf7
Merge pull request #11660 from zmstone/0922-fix-audit-eval-command
fix(audit): make emqx eval command auditable
2023-09-22 14:02:47 +02:00
Zaiming (Stone) Shi a34ab19d93 fix(audit): make emqx eval command auditable 2023-09-22 12:05:45 +02:00
zhongwencool 0eed01abee fix: create ssl listener return 500 crash 2023-09-22 17:34:19 +08:00
lafirest 13b5e4dbc9
Merge pull request #11658 from lafirest/fix/sso_misc
Fix/sso misc
2023-09-22 14:44:39 +08:00
firest 7286d15ca6 chore(sso): adjust the schema of the SSO LDAP backend 2023-09-22 13:48:20 +08:00
firest 9e55ae240a feat(sso): add `role` into the result of login endpoints 2023-09-22 13:48:07 +08:00
firest 681e57dee6 fix(RBAC): allow read-only users to logout 2023-09-22 11:06:24 +08:00
Thales Macedo Garitezi d6935b6a67 feat: add port scan diagnostics to mria waiting for tables checks
Fixes https://emqx.atlassian.net/browse/EMQX-10944

Also updates ekka -> 0.15.15, mria -> 0.6.4

How to test
===========

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.

Or, more easily:

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.

Example output
==============

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status =>
                                                                 bad_ports},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```

After one node is back:

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status => ok},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```
2023-09-21 14:29:01 -03:00
Ivan Dyachkov 7cf60c5a91 chore: e5.3.0-alpha.1 2023-09-21 19:29:00 +02:00
Andrew Mayorov acf4227fc6
test(session): fix quic testgroup in persistent session suite
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
Thales Macedo Garitezi 76f614e9c5
Merge pull request #11653 from thalesmg/fix-dash-rbac-dialyzer-error-m-20230921
chore: fix dialyzer errors on ce version
2023-09-21 13:06:09 -03:00
Thales Macedo Garitezi 85a8c174d9 chore: fix dialyzer errors on ce version
```
apps/emqx_dashboard/src/emqx_dashboard.erl
Line 225 Column 17: The pattern {'error', 'unauthorized_role'} can never match the type {'error','not_found' | 'token_timeout'} | {'ok',binary()}
```
2023-09-21 11:07:31 -03:00
Andrew Mayorov b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov 0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
sync r52 to master
2023-09-21 12:31:24 +02:00
William Yang 3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
quicer 0.0.200
2023-09-21 12:20:14 +02:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
Ivan Dyachkov 105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
Andrew Mayorov a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently 2023-09-21 11:53:40 +04:00
firest 9a128517d8 fix(sso): use the correct way to mark the API as authorization-free 2023-09-21 15:52:05 +08:00
zhongwencool dfc14a39bc test: add emqx_management to telemetry SUITE 2023-09-21 14:15:06 +08:00
firest 2283db62ec fix(sso): add a API with authorization-free to list running backends 2023-09-21 11:52:30 +08:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool 9521e11d39
Merge pull request #11584 from zhongwencool/telemetry-error-on-windows
fix: telemetry report error on windows
2023-09-21 09:08:55 +08:00
JianBo He b924fb618a
Merge pull request #11639 from lafirest/test/sso
test(sso): add test case for the integration of SSO with LDAP
2023-09-21 09:01:13 +08:00
zhongwencool 1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify 2023-09-21 07:32:39 +08:00
Andrew Mayorov a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
Andrew Mayorov 3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy 2023-09-20 22:55:54 +04:00
Andrew Mayorov 21e82b9534
test(sessmem): make retry delivery testcase more involved 2023-09-20 22:55:54 +04:00
Andrew Mayorov 69889d14a3
fix(sessds): fix use of undefined types 2023-09-20 22:55:54 +04:00
Andrew Mayorov c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
Andrew Mayorov 9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
Andrew Mayorov 98706cd215
chore: ensure comments follow code style consistently 2023-09-20 22:55:25 +04:00
Andrew Mayorov 8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability 2023-09-20 22:55:25 +04:00
Andrew Mayorov 7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov 045d8b7f10
refactor(ds): reorder functions to improve readability 2023-09-20 22:55:24 +04:00
Andrew Mayorov 540ca6d60f
chore: drop few commented out and irrelevant pieces 2023-09-20 22:55:24 +04:00
Andrew Mayorov 9d145890cc
refactor(sessmem): pass log context as part of session event
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov e422f492ef
test(sessds): reuse and expand persistent session test suite 2023-09-20 22:55:24 +04:00
Andrew Mayorov 3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling 2023-09-20 22:55:23 +04:00
Andrew Mayorov 7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Andrew Mayorov e713fc38aa
feat(broker): reflect persisted messages in publish result
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
Andrew Mayorov f4953e719b
fix(cmproto): fix few typespecs 2023-09-20 22:53:45 +04:00
Andrew Mayorov 8af107e28d
test(ds): simplify cluster test setups 2023-09-20 22:53:45 +04:00
Andrew Mayorov 7c4f68dd3d
fix(session): make utility function names consistent
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
Andrew Mayorov 2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem` 2023-09-20 22:53:44 +04:00
Andrew Mayorov abeff0bc4f
chore(session): try to describe what happens after session takeover 2023-09-20 22:53:44 +04:00