Commit Graph

10340 Commits

Author SHA1 Message Date
Thales Macedo Garitezi aae59f1efd
Merge pull request #11682 from thalesmg/fix-file-logger-type-r53-20230925
fix(file_logger): change file logger type depending on rotation size
2023-09-26 10:22:36 -03:00
Andrew Mayorov bce2142df8
test(session): add custom session timers testcase 2023-09-26 16:22:06 +03:00
Zaiming (Stone) Shi 4e15edb5e4
Merge pull request #11661 from zmstone/0922-fix-json-log-formatter
0922 fix json log formatter
2023-09-26 14:13:41 +02:00
Thales Macedo Garitezi fdcd73e20c fix(file_logger): change file logger type depending on rotation size
Fixes https://emqx.atlassian.net/browse/EMQX-11036

From `logger_disk_log_h:open_disk_log`:

```erlang
open_disk_log(Name,File,Type,MaxNoBytes,MaxNoFiles) ->
    case filelib:ensure_dir(File) of
        ok ->
            Size =
                if Type==halt -> MaxNoBytes;
                   Type==wrap -> {MaxNoBytes,MaxNoFiles} %% <-------
                end,
            Opts = [{name,   Name},
                    {file,   File},
                    {size,   Size},
                    {type,   Type},
                    {linkto, self()},
                    {repair, false},
                    {format, external},
                    {notify, true},
                    {quiet,  true},
                    {mode,   read_write}],
            case disk_log:open(Opts) of
```

Affects all file loggers (audit included):

```
% emqx_config_logger:update_log_handler/1 -> ok
iex(emqx@127.0.0.1)14> Config override: log.file.emqx_audit is updated, but failed to add handler: {handler_not_added,
                                                                             {badarg,
                                                                              [{size,
                                                                                {infinity,
                                                                                 10}},
                                                                               {type,
                                                                                wrap},
                                                                               {linkto,
                                                                                <0.1952.0>},
                                                                               {repair,
                                                                                false},
                                                                               {format,
                                                                                external},
                                                                               {notify,
                                                                                true},
                                                                               {quiet,
                                                                                true},
                                                                               {mode,
                                                                                read_write}]}}
```
2023-09-26 09:04:44 -03:00
Ilya Averyanov 14983ec14a chore(hooks): validate hookpoints and document hook callbacks
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-26 14:27:42 +03:00
JianBo He 5e6397b843 chore: remove result field for CLI audit logs 2023-09-26 19:24:35 +08:00
JianBo He a73c3b8e1e Merge remote-tracking branch 'ce/release-53' into audit-log-fix-2 2023-09-26 19:05:47 +08:00
JianBo He e9785a6863
Merge pull request #11680 from thalesmg/audit-not-on-ce-r53-20230925
fix(audit): only support audit log on enterprise edition
2023-09-26 18:57:22 +08:00
Zaiming (Stone) Shi 7d810c2107 test: fix test case match pattern 2023-09-26 11:40:30 +02:00
Zaiming (Stone) Shi d31bfc70fb
Merge pull request #11659 from zhongwencool/fix-listener-ssl-create-500
fix: create ssl listener return 500 crash
2023-09-26 11:38:07 +02:00
lafirest 8cc626d33f
Merge pull request #11686 from lafirest/fix/sso_user_backend
fix: fix update SSO user && improve SSO user deletion commnad
2023-09-26 16:04:51 +08:00
firest 56917fee5f fix(sso): Improved SSO user deletion command 2023-09-26 14:15:41 +08:00
firest d1af81c86e fix(sso): fix the failure to update SSO users 2023-09-26 12:14:45 +08:00
firest 8135e27586 fix: redact sensitive data in SSO and LDAP 2023-09-26 11:06:47 +08:00
Thales Macedo Garitezi 17206f8c75
Merge pull request #11662 from thalesmg/port-scan-check-proto-dist-m-20230922
chore: check ekka proto dist module type when resolving node address
2023-09-25 17:01:43 -03:00
Thales Macedo Garitezi 5d212e1086 fix(audit): only support audit log on enterprise edition
Fixes https://emqx.atlassian.net/browse/EMQX-11039
2023-09-25 15:59:35 -03:00
Thales Macedo Garitezi 0498e59c45 test: fix flaky test 2023-09-25 13:35:17 -03:00
zhongwencool 95060302fd fix: don't need to change audit log's level 2023-09-26 00:28:47 +08:00
Thales Macedo Garitezi ff7f37ccf5 test(cth): allow defining schema to load for app 2023-09-25 13:22:41 -03:00
Andrew Mayorov b1f144ab8b
feat(session): add custom session timers mechanism
That are managed exclusively by the session implementation, unlike
common session timers that are managed by the channel itself.
2023-09-25 18:19:26 +03:00
lafirest 1df8326fb8
Merge pull request #11677 from lafirest/fix/sso_cfg
adjust the config path for the SSO feature and improve the update logic
2023-09-25 23:00:46 +08:00
Zaiming (Stone) Shi 5e6996dc05 refactor: log formatter format mfa+line as m:f/a(line)
also improve json formatter when a field is iolist
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 6a557980e4 refactor(logger): ensure JSON log field order
Ensure that the log fileds in JSON format are ordered as
time, level, msg, mfal, ...
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 5f45ba50ff refactor: delete log formatter config for audit log handler
It is using 'json' formatter, the template is useless
2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi 1fed38c248 fix(logger): write 'json' format logs as JSON 2023-09-25 16:55:22 +02:00
Zaiming (Stone) Shi aea1e80290 feat: add 'format' as alias for log 'formatter' 2023-09-25 16:55:22 +02:00
William Yang 9106eb92d2
Merge pull request #11675 from qzhuyan/fix/william/quic-respect-param-verify 2023-09-25 15:48:01 +02:00
Thales Macedo Garitezi 806017ef90 chore: check ekka proto dist module type when resolving node address
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917

Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
firest cfdb25b213 fix(sso): updates the SSO backend when the `[dashboard]` has updated 2023-09-25 20:30:47 +08:00
Ivan Dyachkov f1bc3b68b6
Merge pull request #11674 from id/0925-e5.3.0-alpha.2
e5.3.0 alpha.2
2023-09-25 14:21:46 +02:00
JianBo He 6f29bbf997
Merge pull request #11673 from HJianBo/redact_audit_log_from_cli
Redact audit log from CLI
2023-09-25 20:03:45 +08:00
Andrew Mayorov 81cf619f07
fix(ftconf): also mark `secret_access_key` key as sensitive 2023-09-25 14:52:57 +03:00
William Yang 36d3a3a524 fix: bump to quicer 0.0.201
do not load cacertfile if verify_none
2023-09-25 13:02:01 +02:00
William Yang 22193d273a fix(quic): ignore undefined cacertfile 2023-09-25 12:57:55 +02:00
Ivan Dyachkov 8c93c79b44 chore: e5.3.0-alpha.2 2023-09-25 12:52:45 +02:00
JimMoen 7d58f6c61e
Merge pull request #11668 from JimMoen/saml-login-redirect
fix: saml login acs redirect to dashboard overview
2023-09-25 18:11:03 +08:00
JianBo He b970a34ee0 chore: redact some audit logs from CLI 2023-09-25 18:06:39 +08:00
JianBo He 89e15e9134
Merge pull request #11665 from HJianBo/typo-fixes
Typo fixes
2023-09-25 17:23:40 +08:00
lafirest 1cd9df6461
Merge pull request #11669 from lafirest/fix/dashboard_cli
fix(sso): refactor the `admins` CLI
2023-09-25 16:57:18 +08:00
firest e63d484632 fix(sso): move the config path of the SSO feature to `dashboard.sso` 2023-09-25 15:23:52 +08:00
Ivan Dyachkov e1f1c64ecd Merge branch 'release-53' into 0925-sync-r53-to-master 2023-09-25 08:12:45 +02:00
firest e94192d1fa fix(sso): refactor the `admins` CLI
1. revert the opensource version
2. allow delete the SSO user via CLI
2023-09-25 13:53:01 +08:00
firest 4f4868a46c fix(sso): Disable access to `logout` endpoint by the `API key` 2023-09-25 10:31:38 +08:00
JianBo He 30862a94c6 chore: typo fixes 2023-09-24 20:19:05 +08:00
JimMoen ad4fadc2fa
fix: saml login acs redirect to dashboard overview 2023-09-23 17:29:02 +08:00
JianBo He f8d06614c0 chore: fix dialyzer warnings 2023-09-23 07:34:04 +08:00
JimMoen 1dddccb448
fix(saml): cert files cleanup when destroy 2023-09-22 22:49:08 +08:00
JimMoen 80a6c1150d
fix(saml): saml login reply role `viewer` as default 2023-09-22 22:39:02 +08:00
JimMoen cc3e4e4dc5
fix(saml): drop cert and key content and return path 2023-09-22 22:37:04 +08:00
JimMoen 6349cd3910
fix(saml): sp sign request 2023-09-22 21:57:50 +08:00
JimMoen 2a8f3f9eaa
fix: saml xml metedata format 2023-09-22 21:50:43 +08:00
JimMoen a318ad486a
refactor: behavior login/2 use all http request 2023-09-22 21:50:43 +08:00
JianBo He 9181ec844f
chore: split out sso_saml_api module 2023-09-22 21:50:43 +08:00
JianBo He df94426ee3
chore: make static_check happy 2023-09-22 21:50:42 +08:00
JianBo He ec0894ca0b
chore: update esaml vsn 2023-09-22 21:50:42 +08:00
JianBo He 4a26f63bd6
chore: fix bugs 2023-09-22 21:50:41 +08:00
JianBo He 47badc3181
chore: make dialyzer happy 2023-09-22 21:49:12 +08:00
JianBo He 1c78c6bf6d
chore: fix 500 crashes when backend not existed 2023-09-22 21:49:12 +08:00
JimMoen b4fb5196cb
fix(sso): SSO management API 500 2023-09-22 21:49:12 +08:00
JimMoen bba5cc44a8
fix: keep same API path style 2023-09-22 21:49:12 +08:00
JimMoen 8300cd42d4
fix: acl url ignore auth check 2023-09-22 21:49:12 +08:00
JimMoen 44836ef5ee
chore: bump esaml vsn to v1.1.1 2023-09-22 21:49:12 +08:00
JimMoen 13666fa9f9
refactor: avoid dynamic call 2023-09-22 21:49:12 +08:00
JimMoen c9e0d4fc30
feat: saml integration for dashboard sso 2023-09-22 21:49:10 +08:00
JimMoen d9466eef63
chore: fix Dashboard RBAC license and rebar.config 2023-09-22 21:19:45 +08:00
Zaiming (Stone) Shi fd932c9bf7
Merge pull request #11660 from zmstone/0922-fix-audit-eval-command
fix(audit): make emqx eval command auditable
2023-09-22 14:02:47 +02:00
Zaiming (Stone) Shi a34ab19d93 fix(audit): make emqx eval command auditable 2023-09-22 12:05:45 +02:00
zhongwencool 0eed01abee fix: create ssl listener return 500 crash 2023-09-22 17:34:19 +08:00
lafirest 13b5e4dbc9
Merge pull request #11658 from lafirest/fix/sso_misc
Fix/sso misc
2023-09-22 14:44:39 +08:00
firest 7286d15ca6 chore(sso): adjust the schema of the SSO LDAP backend 2023-09-22 13:48:20 +08:00
firest 9e55ae240a feat(sso): add `role` into the result of login endpoints 2023-09-22 13:48:07 +08:00
firest 681e57dee6 fix(RBAC): allow read-only users to logout 2023-09-22 11:06:24 +08:00
Thales Macedo Garitezi d6935b6a67 feat: add port scan diagnostics to mria waiting for tables checks
Fixes https://emqx.atlassian.net/browse/EMQX-10944

Also updates ekka -> 0.15.15, mria -> 0.6.4

How to test
===========

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.

Or, more easily:

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.

Example output
==============

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status =>
                                                                 bad_ports},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```

After one node is back:

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status => ok},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```
2023-09-21 14:29:01 -03:00
Ivan Dyachkov 7cf60c5a91 chore: e5.3.0-alpha.1 2023-09-21 19:29:00 +02:00
Andrew Mayorov acf4227fc6
test(session): fix quic testgroup in persistent session suite
Which broker after quicer 0.0.200 upgrade.
2023-09-21 20:52:04 +04:00
Thales Macedo Garitezi 76f614e9c5
Merge pull request #11653 from thalesmg/fix-dash-rbac-dialyzer-error-m-20230921
chore: fix dialyzer errors on ce version
2023-09-21 13:06:09 -03:00
Thales Macedo Garitezi 85a8c174d9 chore: fix dialyzer errors on ce version
```
apps/emqx_dashboard/src/emqx_dashboard.erl
Line 225 Column 17: The pattern {'error', 'unauthorized_role'} can never match the type {'error','not_found' | 'token_timeout'} | {'ok',binary()}
```
2023-09-21 11:07:31 -03:00
Andrew Mayorov b563e30615
Merge pull request #11650 from keynslug/fix/simplify-takeover-suite
test(session): make testsuite trigger takeover logic consistently
2023-09-21 18:02:38 +04:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov 0a61d08a0f
Merge pull request #11651 from id/0921-sync-r52-to-master
sync r52 to master
2023-09-21 12:31:24 +02:00
William Yang 3a5227198e
Merge pull request #11642 from qzhuyan/dev/william/quicer-0.0.200
quicer 0.0.200
2023-09-21 12:20:14 +02:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
Ivan Dyachkov 105bebc250 chore: merge release-52 into master 2023-09-21 10:22:47 +02:00
Andrew Mayorov a8f4b5bf86
test(session): make testsuite trigger takeover logic consistently 2023-09-21 11:53:40 +04:00
firest 9a128517d8 fix(sso): use the correct way to mark the API as authorization-free 2023-09-21 15:52:05 +08:00
zhongwencool dfc14a39bc test: add emqx_management to telemetry SUITE 2023-09-21 14:15:06 +08:00
firest 2283db62ec fix(sso): add a API with authorization-free to list running backends 2023-09-21 11:52:30 +08:00
zhongwencool 123d31fa7d
Merge pull request #11640 from zhongwencool/ensure-destory-resource
fix: always return ok when remove local resource
2023-09-21 09:21:45 +08:00
zhongwencool 9521e11d39
Merge pull request #11584 from zhongwencool/telemetry-error-on-windows
fix: telemetry report error on windows
2023-09-21 09:08:55 +08:00
JianBo He b924fb618a
Merge pull request #11639 from lafirest/test/sso
test(sso): add test case for the integration of SSO with LDAP
2023-09-21 09:01:13 +08:00
zhongwencool 1b29e4b189
Merge pull request #11641 from zhongwencool/slog-msg-unify 2023-09-21 07:32:39 +08:00
Andrew Mayorov a2ddd9d5f5
fix(session): respect existing session even if expiry interval = 0
If the original connection had Session-Expiry-Interval > 0, and the
new connection set Session-Expiry-Interval = 0, the MQTTv5 spec says
that (supposedly) we still have to continue with the existing session
(if it hasn't expired yet).

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:54 +04:00
Andrew Mayorov 3945f08f8f
fix(sessds): try to ensure iterators are closed on destroy 2023-09-20 22:55:54 +04:00
Andrew Mayorov 21e82b9534
test(sessmem): make retry delivery testcase more involved 2023-09-20 22:55:54 +04:00
Andrew Mayorov 69889d14a3
fix(sessds): fix use of undefined types 2023-09-20 22:55:54 +04:00
Andrew Mayorov c1583f7f9d
fix(ds): refine `topic()` type to describe parsed topics
And separate it from `topic_filter()` type, which describes parsed
topic filters.
2023-09-20 22:55:52 +04:00
Andrew Mayorov 9362ef6f73
test(sessmem): drop unnecessary nesting in testcase
Also get rid of sneaky binding assignment.
2023-09-20 22:55:25 +04:00
Andrew Mayorov 98706cd215
chore: ensure comments follow code style consistently 2023-09-20 22:55:25 +04:00
Andrew Mayorov 8670efbfa0
chore(chan): rename `Name` → `TimerName` for better readability 2023-09-20 22:55:25 +04:00
Andrew Mayorov 7a9916c84d
fix(sessds): convert ds iterator topics upon opening ds session 2023-09-20 22:55:25 +04:00
Andrew Mayorov 045d8b7f10
refactor(ds): reorder functions to improve readability 2023-09-20 22:55:24 +04:00
Andrew Mayorov 540ca6d60f
chore: drop few commented out and irrelevant pieces 2023-09-20 22:55:24 +04:00
Andrew Mayorov 9d145890cc
refactor(sessmem): pass log context as part of session event
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov adc29e15cc
refactor(session): make typespecsa and flow a bit more clear
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:55:24 +04:00
Andrew Mayorov e422f492ef
test(sessds): reuse and expand persistent session test suite 2023-09-20 22:55:24 +04:00
Andrew Mayorov 3383ae19a9
test(session): switch `emqx_persistent_session_SUITE` to cth tooling 2023-09-20 22:55:23 +04:00
Andrew Mayorov 7326ef550b
fix(sessds): make existing parts of persistent session impl work 2023-09-20 22:55:22 +04:00
Andrew Mayorov e713fc38aa
feat(broker): reflect persisted messages in publish result
In order for callers to distinguish between silently dropped and
durably persisted message w/o matching subscribers.
2023-09-20 22:53:45 +04:00
Andrew Mayorov f4953e719b
fix(cmproto): fix few typespecs 2023-09-20 22:53:45 +04:00
Andrew Mayorov 8af107e28d
test(ds): simplify cluster test setups 2023-09-20 22:53:45 +04:00
Andrew Mayorov 7c4f68dd3d
fix(session): make utility function names consistent
Before this commit behavior of `is_banned_msg/1` / `should_discard/1`
were actually the exact opposite of their names.

Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:53:44 +04:00
Andrew Mayorov 2dae8020ec
refactor(cm): avoid deep indirection in `emqx_session_mem` 2023-09-20 22:53:44 +04:00
Andrew Mayorov abeff0bc4f
chore(session): try to describe what happens after session takeover 2023-09-20 22:53:44 +04:00
Andrew Mayorov 45d44df11d
refactor(session): update eviction channel session logic
The changes partially reflect `emqx_channel` changes with respect to
in-memory session specific logic. The difference is that eviction
channel does not replay post-takeover, instead enqueues messages.
2023-09-20 22:53:44 +04:00
Andrew Mayorov ab1c4c4222
refactor(session): drop `lookup/1` session API
Due to the fact it's not used for anything right now.
2023-09-20 22:53:44 +04:00
Andrew Mayorov 97881ff3ca
refactor(session): bring back common session timers 2023-09-20 22:53:43 +04:00
Andrew Mayorov bf16417513
feat(session): introduce session implementation concept 2023-09-20 22:53:42 +04:00
Andrew Mayorov 780ca15298
chore: bump application versions 2023-09-20 22:52:14 +04:00
Andrew Mayorov 04731b7ef7
test(takeover): randomize messages for random natural ordering 2023-09-20 22:52:14 +04:00
Andrew Mayorov f022c9b1a4
feat(emqx): add `emqx_inflight:fold/3` generic function 2023-09-20 22:52:14 +04:00
Andrew Mayorov 596ce157fd
refactor(exproto): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov 57ae5b14f1
refactor(mqttsn): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov e4866adc2f
refactor(chan): make timer names equal to messages they send
Because keeping timer names different from the messages they send
complicates understanding of the control flow, and spends few
reductions per timer operation unnecessarily.
2023-09-20 22:52:13 +04:00
Andrew Mayorov cfb1bf1fa4
chore(emqx): drop remnants of former session persistence impl
1. It is not functional anyway.
2. It blocks `emqx_session` refactoring in a few places.
2023-09-20 22:52:13 +04:00
William Yang e43b87f18e test(quic): update emqx_broker_SUITE 2023-09-20 20:20:18 +02:00
Thales Macedo Garitezi 2965fa6fcb
Merge pull request #11613 from thalesmg/ds-keyspace-m-20230914
feat(ds): introduce keyspace concept
2023-09-20 14:21:27 -03:00
William Yang da626f6d29 feat(quic): bump to quicer 0.0.200 and emqtt 1.9.0 2023-09-20 16:43:15 +02:00
zhongwencool dd687d9582 fix: dialyzer warning 2023-09-20 22:41:26 +08:00
zhongwencool 9e75ff88af
chore: apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:15:39 +08:00
zhongwencool f789a7504f
chore: update apps/emqx/src/emqx_quic_connection.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-20 22:00:44 +08:00
Ivan Dyachkov 09aea47f40 chore: 5.2.1 2023-09-20 14:04:28 +02:00
firest 21c37a3f14 fix(sso): adjust the schema of LDAP API && add more logs 2023-09-20 18:15:42 +08:00
zhongwencool 2f1fa2e961 chore: unified slog message formatting to improve logging consistency 2023-09-20 18:13:00 +08:00
zhongwencool c26a18e949 fix: always return ok when remove local resource 2023-09-20 18:02:42 +08:00
firest b11c5d26ce test(sso): add test case for the integration of SSO with LDAP 2023-09-20 16:25:19 +08:00
lafirest bc6edac45f
Merge pull request #11631 from lafirest/feat/dashboard_ldap
feat(dashboard): add SSO feature and integrate with LDAP
2023-09-20 11:40:42 +08:00
firest 61311081d8 fix(sso): fix RBAC test errors && spellcheck 2023-09-20 10:42:17 +08:00
Thales Macedo Garitezi 5ed5ac48ee refactor: combine shard id and keyspace into a single value 2023-09-19 12:31:32 -03:00
firest 5b00998050 fix(sso): fix invalid_dynamic_call 2023-09-19 22:47:48 +08:00
firest 285e529766 fix(sso): fix sso errors found when manual test 2023-09-19 19:15:52 +08:00
Zaiming (Stone) Shi 65818740f3
Merge pull request #11622 from zmstone/0917-upgrade-gen_rpc-to-3.1.0
chore(gen_rpc): upgrade from 2.8.1 to 3.1.0
2023-09-19 10:22:53 +02:00
Ivan Dyachkov a69c26b86d chore: e5.2.1-alpha.3 2023-09-19 08:17:29 +02:00
firest 2cddce5479 feat(dashboard): add SSO feature and integrate with LDAP 2023-09-19 14:10:29 +08:00
JianBo He de1bc04843 chore: bump ekka to 0.15.14, mria to 0.6.3
Inclued ekka's updates:
- https://github.com/emqx/ekka/pull/214
- https://github.com/emqx/ekka/pull/215
- https://github.com/emqx/ekka/pull/216
- https://github.com/emqx/ekka/pull/217

mria's updates:
- https://github.com/emqx/mria/pull/160
- https://github.com/emqx/mria/pull/161
- https://github.com/emqx/mria/pull/162
2023-09-19 09:43:19 +08:00
Thales Macedo Garitezi a511088fd4 refactor: address review comments 2023-09-18 17:47:56 -03:00
Zaiming (Stone) Shi 9f8a7a96af
Merge pull request #11628 from zmstone/0918-upgrade-emqtt-to-1.8.7
0918 upgrade emqtt to 1.8.7
2023-09-18 21:56:04 +02:00
Zaiming (Stone) Shi de93d71014 test: fix emqx_sahred_sub_SUITE:t_remote
prior to this fix, the peer ct node is actually node itself
2023-09-18 21:54:43 +02:00
Zaiming (Stone) Shi 99b60849b4 test: fix flaky test emqx_falpping_SUITE:t_conf_update_timer 2023-09-18 20:44:38 +02:00
SergeTupchiy 0b0e322473
Merge pull request #11627 from SergeTupchiy/EMQX-10962-fix_hstreamdb-bridge-on-stop
fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
2023-09-18 19:57:07 +03:00
Zaiming (Stone) Shi d1e5b69eac test: increase wait time 2023-09-18 17:33:09 +02:00
Zaiming (Stone) Shi 4f1ec795eb test: delete special handling of emqtt race condition
the race should have been fixed in emqtt 1.8.7
2023-09-18 17:27:34 +02:00
Zaiming (Stone) Shi 46557c3462 fix(emqtt): upgrade from 1.8.6 to 1.8.7 2023-09-18 17:04:23 +02:00
Serge Tupchii 80cf60e9be fix(emqx_bridge_hstreamdb): fix resources cleanup in on_stop/2 cb
fixes: EMQX-10962
2023-09-18 17:48:05 +03:00
Zaiming (Stone) Shi 45f0d0dfce
Merge pull request #11623 from zmstone/0917-upgrade-esockd-to-5.9.7
0917 upgrade esockd to 5.9.7
2023-09-18 16:30:54 +02:00
zhongwencool c868264f65
Merge pull request #11617 from zhongwencool/rule-engine-console-args
fix: keep builtin_console actions not complain args
2023-09-18 20:40:02 +08:00
zhongwencool 8b6d5d4ca9 test: add test for build_console's args 2023-09-18 16:48:38 +08:00
zhongwencool 5490807b20
chore: update apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-18 16:11:39 +08:00
Zaiming (Stone) Shi f66d9e76fe fix(emqx_channel): aovid logging frame errors as exceptions 2023-09-18 09:04:00 +02:00
lafirest 6fe846bf0e
Merge pull request #11610 from lafirest/feat/rbac
feat(dashboard): add RBAC feature for Dashboard
2023-09-18 14:41:47 +08:00
firest ca6547c05b fix(rbac): change default role and update changes 2023-09-18 11:40:35 +08:00
Zaiming (Stone) Shi abbee4bc47 fix(esockd): upgrade esockd from 5.9.6 to 5.9.7 2023-09-17 19:44:04 +02:00
Zaiming (Stone) Shi 985f0e6994 chore(gen_rpc): upgrade from 2.8.1 to 3.1.0 2023-09-17 19:10:13 +02:00
Zaiming (Stone) Shi a9682c2989
Merge pull request #11605 from zmstone/0913-lower-severity-level-for-CMD-override-warnings
0913 lower severity level for cmd override warnings
2023-09-16 11:32:07 +02:00
Zaiming (Stone) Shi ac41f497b4 feat: add bytesize rule sql function 2023-09-15 16:04:21 +02:00
Thales Macedo Garitezi 2aae3856a3
Merge pull request #11596 from thalesmg/fix-typespec-typos-m-20230912
fix: fix typespec typos
2023-09-15 09:24:42 -03:00
Ilya Averyanov f0d2b2e6c3 chore: update emqx_eviction_agent version 2023-09-15 13:24:38 +03:00
firest 4b97d3f57d test(rbac): add test cases for RBAC && update changes 2023-09-15 18:16:28 +08:00
Ilya Averyanov 1959e15703 chore: simplify session eviction for node rebalance 2023-09-15 11:46:51 +02:00
zhongwencool 32d838c85b fix: format console action crash 2023-09-15 17:40:56 +08:00
zhongwencool 640dd130fa fix: don't urldecode twice clientid 2023-09-15 17:36:48 +08:00
zhongwencool d72348c8ed fix: keep builtin_console actions not complain args 2023-09-15 16:38:04 +08:00
firest 45ccc66474 fix(dashbaord): fix test cases to support RBAC 2023-09-15 15:49:37 +08:00
lafirest aace9215a4
Merge pull request #11608 from lafirest/feat/ldap_bind
feat(ldap): integrate authentication with LDAP bind operation
2023-09-15 14:06:39 +08:00
firest 0846939760 fix(ldap): remove unused code and mark sensitive field 2023-09-15 10:13:07 +08:00
Thales Macedo Garitezi b30bcf32bd feat(ds): introduce keyspace concept
Fixes https://emqx.atlassian.net/browse/EMQX-10579

This introduces the concept of "keyspaces" to our durable storage (DS) implementation, and
also refactors some places where "shard" and "keyspace" would be mixed up.

We might want to tune the storage options differently for distinct sets of topics, the
keyspaces.  The keyspace is composed by one or more shards.

- Keyspaces are identified simply by binary strings.
- DS configuration is scoped by keyspaces instead of shards.
- Starting a new DS shard requires definining to which keyspace the shard belongs.
2023-09-14 16:39:00 -03:00
Ilya Averyanov 11c39c4b6a chore: simplify session eviction for node rebalance 2023-09-14 18:28:25 +03:00
Thales Macedo Garitezi 6be0e14e8c fix: fix typespec typos 2023-09-14 10:21:05 -03:00
Thales Macedo Garitezi e41f7dd68c
Merge pull request #11598 from thalesmg/ds-fix-ps-router-m-20230912
fixes and improvements to persistent session router
2023-09-14 10:00:58 -03:00
firest 5c31c5ce76 feat(dashboard): add RBAC feature for Dashboard 2023-09-14 19:28:37 +08:00
firest afbf13b8a2 feat(ldap): integrate authentication with LDAP bind operation 2023-09-14 14:10:43 +08:00
firest 3eb46be2ae fix(banned): fix typos 2023-09-14 14:01:59 +08:00
Zaiming (Stone) Shi 95f1312df7 chore(emqx_ctl): lower command override log from warning to info
prior to this change this warning message is always logged if a node
joins cluster.

the intention of this warning is to detect potential command name
clash (provided by different plugins), so arguably it should be
at debug level
2023-09-13 20:52:08 +02:00
Thales Macedo Garitezi 803b69d878
refactor: use more consistent return types
Co-authored-by: Andrew Mayorov <encube.ul@gmail.com>
2023-09-13 10:49:45 -03:00
Thales Macedo Garitezi f55f6dc779 docs(ps_session): add notes explaining rationale about operation order and consistency 2023-09-13 10:12:27 -03:00
Thales Macedo Garitezi e081abb9e6 perf(ps_router): only check if there are any routes when deciding whether to persist message 2023-09-13 10:02:39 -03:00
Thales Macedo Garitezi cae9ae1fab fix(ps_router): use `disc_copies` for storing persistent session routes 2023-09-13 10:02:39 -03:00
William Yang ea73d674b9 chore: bump version to e5.2.1-alpha.2 2023-09-13 14:55:17 +02:00
ieQu1 69d17f981d
Merge pull request #11595 from ieQu1/dev/dont-prevent-overlapping-partitions
fix(distribution): Set prevent_overlapping_partitions to false
2023-09-13 14:22:34 +02:00
Serge Tupchii e1456c1c04 feat(emqx_conf_schema): add node.default_bootstrap_batch_size config parameter 2023-09-13 10:11:14 +03:00
Serge Tupchii af015bc2bb chore(ekka): bump version to 0.15.13 2023-09-13 10:11:14 +03:00
ieQu1 b41310af9c
fix: Add prevent_overlappiing_partitions to vm_args
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-12 20:01:56 +02:00
ieQu1 1a86c21e20 fix(distribution): Set prevent_overlapping_partitions to false 2023-09-12 17:41:51 +02:00
William Yang 6ef296268e chore: bump version to e5.2.1-alpha.1 2023-09-12 15:24:02 +02:00
Thales Macedo Garitezi 59b94788bf refactor: address review comments 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi 64fcdb671d test: fix a bunch of inter-suite interferences 2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi f1294736b7 feat(ds): add filter for message persistence
Fixes https://emqx.atlassian.net/browse/EMQX-10520
2023-09-11 16:25:34 -03:00
Thales Macedo Garitezi fe4640922d feat(ds): add delete callback 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi 7805cc8c9b fix(session): check if session is persistent 2023-09-11 16:25:33 -03:00
Thales Macedo Garitezi f490da71cd chore: prepare v5.2.0-build.1 2023-09-11 16:09:17 -03:00
Thales Macedo Garitezi d0f1c75651 chore: bump apps vsns 2023-09-11 16:06:53 -03:00
Thales Macedo Garitezi 223d47a6ac Merge branch 'master' into sync-m-r52-20230911 2023-09-11 16:04:28 -03:00
Thales Macedo Garitezi 063913f245 fix(session): remove recently added `iterators` field from `#session{}` record
Fixes https://emqx.atlassian.net/browse/EMQX-10945

This could lead to `badrecord` errors being raised if a takeover were to happen during a
rolling cluster upgrade, as the old nodes could receive a record with more fields than
expected.
2023-09-11 14:46:12 -03:00
zhongwencool 887a74a6cb fix: telemetry report error on windows 2023-09-11 16:48:30 +08:00
Zaiming (Stone) Shi 7010ac358c chore: bump release version to v5.2.0 2023-09-08 16:31:52 +02:00
Andrew Mayorov 3d133801af
fix(router): use `ets:new/2` options compatible with OTP 24
While essentially keeping the same effect.
2023-09-08 18:04:29 +04:00
Thales Macedo Garitezi e089fda260
Merge pull request #11568 from thalesmg/republish-props-m-20230905
feat(republish): allow templating mqtt properties
2023-09-07 16:43:00 -03:00
Zaiming (Stone) Shi f3bebb0ba2 Merge remote-tracking branch 'origin/master' into 0907-sync-master-to-release-52 2023-09-07 18:51:30 +02:00
Thales Macedo Garitezi 68293231b8 test: add cases for more complex placeholders 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 014bc64d3b chore: ensure general prop is a binary 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 02b8bbf76a chore: drop templating support for `Subscription-Identifier`
This is highly-dependent on the session state, as is `Topic-Alias`.
2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 47bd19dee4 chore: lower error log messages down to debug 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi 080cb73da1 fix: handle badmap inside `emqx_placeholder:proc_tmpl` 2023-09-07 13:40:03 -03:00
Zaiming (Stone) Shi db2ec55aa0 chore: bump release version to official e5.2.0 2023-09-07 09:40:15 +02:00
Zaiming (Stone) Shi 7c2f87fabe test: merge broker and router boot modules 2023-09-06 21:36:16 +02:00
Thales Macedo Garitezi bb55b04d46 refactor: inline fn clauses, improve error logging 2023-09-06 13:54:40 -03:00
Thales Macedo Garitezi c9100c7591
refactor: improve logging information and descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-06 13:33:59 -03:00
Thales Macedo Garitezi 78c5a779d7 feat(republish): allow templating mqtt properties
Fixes https://emqx.atlassian.net/browse/EMQX-10912
2023-09-06 09:54:02 -03:00
Kjell Winblad 7fe704b6dd
Merge pull request #11563 from kjellwinblad/kjell/fix/release-52/EMQX-10775
fix: update gpb library to fix type error
2023-09-06 14:47:23 +02:00
Zaiming (Stone) Shi 18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Zaiming (Stone) Shi e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
Thales Macedo Garitezi 954b73a9aa
Merge pull request #11533 from thalesmg/ds-unsubscribe-m-20230828
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
2023-09-05 09:47:17 -03:00
ieQu1 01a128878f chore(mria): Bump mria to 0.6.1 2023-09-05 11:08:32 +02:00
Andrew Mayorov d26995a2aa
Merge pull request #11524 from keynslug/ft/EMQX-10713/unified-route-tab
feat(router): add routing storage schema with 2 tables
2023-09-05 12:38:00 +04:00
Kjell Winblad 2b2e84b83c fix: update gpb library to fix type error
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-09-05 09:52:17 +02:00
Thales Macedo Garitezi 23d63f5e01 refactor: fork clientid type for `emqx_ds:session_id/0` 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 60ae3c15c8 refactor: disregard impossible case 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi 2cfe5ba057 fix(plugins_api): fix response types and test 2023-09-04 14:37:25 -03:00
Thales Macedo Garitezi 02c3d6a0a7 fix(data_import): rm duplicate import call and fix test 2023-09-04 14:19:59 -03:00
Thales Macedo Garitezi 73d4248334 test(cth_cluster): always allocate listener ports 2023-09-04 11:22:26 -03:00
Andrew Mayorov 9f0f218387
test(router): simplify cluster-related testcase in helper testsuite 2023-09-04 18:05:59 +04:00
Andrew Mayorov 7d26b7bc1a
fix(router): emit clearer error when conflicting schemas in use 2023-09-04 16:37:38 +04:00
Thales Macedo Garitezi d33ad8a46f feat: cluster purge (r5.2)
Port of https://github.com/emqx/emqx/pull/11447

targeting `release-52`

Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-09-04 09:15:28 -03:00
Andrew Mayorov 893f69617a
chore(test): drop obsolete TODOs 2023-09-04 14:40:49 +04:00
Andrew Mayorov f0a0c7d4b1
fix(cthsuite): avoid duplicate application env entries
Because `application:set_env/1` complains loudly if there are
duplicates.
2023-09-04 14:40:49 +04:00
Andrew Mayorov 5024304bf9
fix(router): wait for tables replicate before choosing schema vsn 2023-09-04 14:40:48 +04:00
Ivan Dyachkov 24230a64df chore: bump app versions 2023-09-04 11:39:21 +02:00
Ivan Dyachkov b2f57636bd Merge remote-tracking branch 'upstream/release-51' into 0904-sync-release-51 2023-09-04 11:34:22 +02:00
Kjell Winblad 52546ac50f
Merge pull request #11522 from kjellwinblad/kjell/fix/EMQX-10778
fix: bad error message when rule engine schema name is too long
2023-09-04 11:05:55 +02:00
Ilya Averyanov 240afecd69
Merge pull request #11487 from savonarola/0821-optimize-bcrypt
feat: reduce bcrypt rounds to a usable value
2023-09-04 10:12:40 +03:00
zhongwencool ce571d892e chore: ping opentelemetry v1.3.0-emqx 2023-09-04 09:34:09 +08:00
Serge Tupchii ed9afe3345 refactor(emqx_connector): use `hocon_schema:override/2` to make pgsql 'username' field required 2023-09-01 19:56:07 +03:00
Serge Tupchii 607705518b test(emqx_bridge_cassandra): add connector test case for Cassandra configured without authentication 2023-09-01 19:56:07 +03:00
Serge Tupchii f0c75d97e1 chore: upgrade emqx_http_lib to 0.5.3
0.5.3 release includes a fix to parse <host>:<port> URLs using the default http scheme.
2023-09-01 19:56:07 +03:00
Serge Tupchii c3b3460173 fix(emqx_oracle): make username a required field in emqx_oracle_schema 2023-09-01 19:56:07 +03:00
Andrew Mayorov 4ab5f8374b
chore(schema): mark routing storage schema readOnly
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-01 19:58:31 +04:00
Andrew Mayorov ebf13c90ea
test(router): make sort stable in e2e routing testcase 2023-09-01 19:48:56 +04:00
Ilya Averyanov 90156befb5 feat: improve bcrypt usability
* limit salt rounds to usable values
* update bcrypt library to enable concurrent bcrypt hash calculation
2023-09-01 17:36:11 +03:00
Serge Tupchii c54527857a fix(emqx_bridge_sqlserver): use empty password by default as it is not a required field 2023-09-01 16:57:35 +03:00
Serge Tupchii 1bf86250dd fix(emqx_bridge_cassandra): allow cassandra bridge without username/password
Cassandra can be used without credentials, if it is configured with AllowAllAuthenticator (default).
2023-09-01 16:57:35 +03:00
Andrew Mayorov 545f1c84a6
fix(cth): do not allocate ports for `emqx` app by default
This causes tricky and impressively hard to track side effects down
the line. Namely, loading `emqx_schema` _after_ `emqx_conf_schema`
(as part of cluster node startup sequence) leads to a couple of
schema root rewrites, because `emqx_schema` defines similar config
roots yet slightly differently (e.g. `authorization`).
2023-09-01 17:23:34 +04:00
Andrew Mayorov 0a879bffd1
test(router): add e2e testcase for cluster-wide routing
Which verifies that both router storage schemas work correctly.
2023-09-01 13:49:12 +04:00
Andrew Mayorov eb0385a28f
chore(router): separate test-only function exports 2023-09-01 13:49:11 +04:00
zhongwencool 3d859453f4
Merge pull request #11554 from lafirest/chore/exhook_prop
chore: shutdown reason will never be a UTF8 atom
2023-09-01 16:53:26 +08:00
Kjell Winblad 3bb65e6b05
Merge pull request #11542 from kjellwinblad/kjell/fix/EMQX-10775
fix: update gpb library to fix type error
2023-09-01 10:07:03 +02:00
firest 53940754b7 chore: shutdown reason will never be a UTF8 atom 2023-09-01 14:17:33 +08:00
Paulo Zulato 17fe406605
Merge pull request #11551 from paulozulato/refactor-connector-config-callback
refactor: remove BridgeName from connector_config callback
2023-08-31 17:05:49 -03:00
Thales Macedo Garitezi bf0f77b6dc
Merge pull request #11548 from thalesmg/fix-plugin-order-update-r52-20230830
fix(plugins): update plugin order on whole cluster
2023-08-31 16:18:28 -03:00
Paulo Zulato 7b607c3300 refactor: remove BridgeName from connector_config callback
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.

Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
2023-08-31 11:01:49 -03:00
Zaiming (Stone) Shi c20ba0572a chore: bump to 5.2.0-alpha.4 2023-08-31 15:47:39 +02:00
Zaiming (Stone) Shi 876d539336 Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table 2023-08-31 15:46:33 +02:00
Andrew Mayorov fb094e1d47
test(router): avoid testsuite collisions 2023-08-31 13:36:01 +04:00
Paulo Zulato 727fd296ee
Merge pull request #11546 from paulozulato/fix-kafka-producer-unique-name
fix(kafka): avoid producer name collision between Kafka and AEH bridges
2023-08-30 18:08:54 -03:00
Thales Macedo Garitezi 4dfbc859f9 fix(plugins): update plugin order on whole cluster
Fixes https://emqx.atlassian.net/browse/EMQX-10879
2023-08-30 16:39:52 -03:00
Paulo Zulato ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-30 14:48:10 -03:00
Andrew Mayorov 2da604dfa4
test(mgmt): simplify flaky `t_verify_imported_mnesia_tab_on_cluster`
And also rest of the test setup in the suite.
2023-08-30 20:24:18 +04:00
Andrew Mayorov b1defa29d7
feat(ruleeng): avoid storing whole rules in topic index
Because it doesn't really give any benefit, but wastes memory by
duplication.
2023-08-30 20:07:56 +04:00
Andrew Mayorov 6b9cb06334
fix(router): add / refine some comments and log messages
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-30 19:51:31 +04:00
Thales Macedo Garitezi 922ca5e141 feat(ds): close iterators when handling `UNSUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9742
2023-08-30 11:14:28 -03:00
Thales Macedo Garitezi 9c6dd30f44 feat(session): store iterator ids in session record 2023-08-30 11:13:55 -03:00
Paulo Zulato dfcede8794 fix: increment matched counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-30 10:52:53 -03:00
Andrew Mayorov f8923ab419
Merge pull request #11539 from keynslug/ft/cth-desc-workdir
feat(cthsuite): add function to determine workdir of testrun
2023-08-30 17:15:55 +04:00
Thales Macedo Garitezi 1cab687153
Merge pull request #11540 from thalesmg/fix-cert-path-utf8-r52-20230829
fix(bridge): validate bridge name before attempting to convert certificates
2023-08-30 09:45:55 -03:00
lafirest 8b3a9072f9
Merge pull request #11532 from lafirest/fix/frame_parse
fix(frame): improve some error reasons when parsing invalid packet
2023-08-30 19:10:45 +08:00
Kjell Winblad d911f7fbea fix: update gpb library to fix type error
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
2023-08-30 11:12:33 +02:00
JianBo He 9fe4382d98
Merge pull request #11530 from JimMoen/chore-bump-hstreamdb-driver
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-30 14:37:17 +08:00
firest 804443ba40 chore: fix typos 2023-08-30 11:22:09 +08:00
JianBo He 06fad14d07
Merge pull request #11536 from zhongwencool/comma_separated_binary
chore: ciphers should allow empty space(comma_separated_binary)
2023-08-30 10:20:06 +08:00
Ilya Averyanov 2ef60db147
Merge pull request #11481 from savonarola/0817-simplify-async-responses
chore(ft): refactor async reply mechanism
2023-08-30 01:20:09 +03:00
Ilya Averyanov 54ac4a8527 chore(ft): tidy up the code according to the review 2023-08-29 23:21:36 +03:00
Thales Macedo Garitezi 0f297ffef4 fix(bridge): validate bridge name before attempting to convert certificates
Fixes https://emqx.atlassian.net/browse/EMQX-10865
2023-08-29 16:32:01 -03:00
Andrew Mayorov 0e770bdc95
test: switch `emqx_broker_SUITE` to use new cth tooling 2023-08-29 22:05:36 +04:00
Andrew Mayorov 3268093881
feat(cth): add module-level documenation 2023-08-29 22:05:36 +04:00
Andrew Mayorov f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
Andrew Mayorov 063d6200c8
feat(router): enable using 2 tables to store routes
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.

Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
2023-08-29 17:41:59 +04:00
Andrew Mayorov 270fd107b2
feat(triesearch): allow reusing filter tokenization result
As a kind of micro-optimization.
2023-08-29 17:31:26 +04:00
zhongwencool 501456efec
chore: more safe with iolist_to_binary
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-29 17:10:43 +08:00
zhongwencool 4e7ba5d35f chore: ciphers should allow empty space(comma_separated_binary) 2023-08-29 11:34:40 +08:00
firest 6722722522 fix(frame): improve some error reasons when parsing invalid packet 2023-08-29 10:53:20 +08:00
zhongwencool 4245a4d8c6 fix: authz clean-cache clientid always return not_found 2023-08-29 08:17:47 +08:00
Paulo Zulato cc3ba18734 fix: increment dropped message counter when bridge is unhealthy
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
Paulo Zulato 2c89be04d1
Merge pull request #11527 from paulozulato/fix-kafka-header-template-handling
fix(kafka): fix template processing for header
2023-08-28 18:55:33 -03:00
Paulo Zulato 0b86f04bae fix(kafka): fix template processing for header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
Ilya Averyanov 279895b8fd chore(ft): add read/write concurrency to emqx_ft_async_reply tabs 2023-08-28 22:46:30 +03:00
SergeTupchiy c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log
fix(emqx_trace): don't download empty trace log file
2023-08-28 22:17:00 +03:00
Serge Tupchii 0072749143 fix(emqx_trace): don't download empty trace log file
Closes: EMQX-10274
2023-08-28 19:25:55 +03:00
Thales Macedo Garitezi 5908b69353 chore: hide cluster purge from cli and api for now 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi f988de4ff4 feat(purge): clear delayed messages 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi 9bded07834 feat: cluster purge
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
Thales Macedo Garitezi dec21ffc95
Merge pull request #11362 from thalesmg/ds-handle-sub-20230725
feat(ds): open iterators when handling `SUBSCRIBE` packets
2023-08-28 10:21:28 -03:00
Andrew Mayorov 2d931a0512
test(evict): do not disable router module in cluster nodes 2023-08-28 14:07:46 +04:00
Andrew Mayorov 8d2ebdea7e
fix(router): generalize config option and make effects visible 2023-08-28 14:07:46 +04:00
Andrew Mayorov e85789306b
chore(router): drop test-only `has_routes/1`
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
Andrew Mayorov 5d51687dbf
feat(router): add unified routing table config option 2023-08-28 14:07:45 +04:00
Andrew Mayorov 33e5e1ba57
feat(router): add unified routing table 2023-08-28 14:07:44 +04:00
Andrew Mayorov 166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 84e40fb6fe
test(trie): add more involved route add + delete case
That shows how current local trie implementation breaks because
of lack of refcounting.
2023-08-28 13:52:59 +04:00
Andrew Mayorov dcb63440bc
refactor(mgmt): avoid dealing with router tab directly
Instead, contain all this behind `emqx_router` module interface.
2023-08-28 13:52:59 +04:00
Andrew Mayorov 9eccfa0909
refactor(router): isolate cleanup logic in router module 2023-08-28 13:52:59 +04:00
JimMoen f3467f44b2
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1` 2023-08-28 17:28:50 +08:00
Ilya Averyanov 4488e9e591 chore(ft): stop hook chain when doing actual handling 2023-08-28 11:57:44 +03:00
Ilya Averyanov 39a48179ea chore(emqx_channel): use macros for reply construction 2023-08-28 11:57:44 +03:00
Ilya Averyanov b8cacd2833 chore(ft): add tests for async reply registry 2023-08-28 11:57:44 +03:00
Andrew Mayorov b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
zhongwencool 053b7fb94a test: add more test for conf_cli 2023-08-28 11:14:36 +08:00
zhongwencool d8be248a3d fix: improve the suggest msg for update conf failed 2023-08-28 08:53:23 +08:00
Ilya Averyanov b0d4a22aa8 chore(ft): refactor async reply mechanism 2023-08-27 21:54:23 +03:00
zhongwencool b874926de7
Merge pull request #11514 from zhongwencool/stop-load-application-before-reboot
fix: stop otel deps appication before reboot
2023-08-27 20:46:31 +08:00
Paulo Zulato 2bb415f897
Merge pull request #11526 from paulozulato/fix-validator-return-msg
fix: flatten error message on resource validator
2023-08-25 17:27:31 -03:00
Thales Macedo Garitezi 33a0048155 refactor: move logic to `ensure_iterator` 2023-08-25 15:24:53 -03:00
Paulo Zulato 84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
Paulo Zulato 42877e282d fix: flatten error message on resource validator
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
Thales Macedo Garitezi f15f59650d test: rm obselete workaround code 2023-08-25 13:49:33 -03:00
Paulo Zulato cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id
Fix Kafka/AEH ClientId uniqueness
2023-08-25 10:14:44 -03:00
Andrew Mayorov d5cff533e3
chore(topicidx): drop TODO comment 2023-08-25 16:44:03 +04:00
Andrew Mayorov ca59a87d47
chore(topicidx): refine example of wildcard compare 2023-08-25 16:44:03 +04:00
Zaiming (Stone) Shi 558402a68e
chore(topic_index): add topic validation 2023-08-25 16:44:03 +04:00
Andrew Mayorov 33ed53bb6a
refactor(topicidx): simplify `compare/3` further
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
2023-08-25 16:34:45 +04:00
Andrew Mayorov fc37d235c7
refactor(topicidx): simplify seek instructions
Which also avoids comparing filter tail repeatedly when evaluating
it.
2023-08-25 15:24:06 +04:00
Kjell Winblad c65db82b07 fix: bad error message when rule engine schema name is too long
Fixes:
https://emqx.atlassian.net/browse/EMQX-10778
2023-08-25 11:02:38 +02:00
zhongwencool c8af5a702c test: remove connection_closed from test 2023-08-25 15:53:35 +08:00
zhongwencool fc1738188e fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero 2023-08-25 15:18:34 +08:00
lafirest ba9cb0a1ed
Merge pull request #11512 from lafirest/fix/ldap_def_port
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-25 10:22:44 +08:00
zhongwencool 8ab6eb5638 test: start emqx_opentelemetry in emqx_machine SUITE 2023-08-25 10:09:00 +08:00
Andrew Mayorov cf45e80c71
feat(topicidx): iterate on trie search implementation
This improves matching performance and decreases GC pressure on
synthetic workloads.
2023-08-25 01:47:11 +04:00
Paulo Zulato 535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi d9a5a9ea2b
Merge pull request #11513 from thalesmg/kafka-fix-ts-template-r52-20230824
fix: use correct timestamp template for kafka and remove such fields from azure event hub producer
2023-08-24 16:26:52 -03:00
Thales Macedo Garitezi 016ae0524f fix(aeh_producer): remove timestamp template field
Fixes https://emqx.atlassian.net/browse/EMQX-10847

Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.

Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type

Example message consumed from AEH:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test0",
  "ts" => 1692879703006,
  "ts_type" => "append",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Note the ts_type above is append.

Example message from a Kafka broker whose ts type is create:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test-topic-three-partitions",
  "ts" => 1692881883668,
  "ts_type" => "create",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
2023-08-24 14:20:42 -03:00
Thales Macedo Garitezi 0381ac0410 fix(kafka_producer): use correct timestamp template field
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
Kjell Winblad 9fe5080705 build: bump versions before release 2023-08-24 16:46:27 +02:00
zhongwencool c215fe3736 fix: stop otel deps appication before reboot 2023-08-24 22:14:14 +08:00
Paulo Zulato 60e6217496 fix(kafka): fix result handling when sending message with invalid header
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi e5041de9cc fix(gcp_consumer): handle 401 errors
Fixes https://emqx.atlassian.net/browse/EMQX-10852
2023-08-24 09:03:34 -03:00
Zaiming (Stone) Shi 62423b0b12 refactor(topic_index): less special handling for leading $ words 2023-08-24 13:30:10 +02:00
Zaiming (Stone) Shi a30d87e14f refactor(topic_index): remove more unnecessary next calls
also avoid using records (setelement) for recursive return values
2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi a1e6635614 refactor(topic_index): no forced ceiling entry in index table 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi f4c8c6be55 refactor(topic_index): optimize trie-search performance 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi 6b152b3cb7 test: add more debug output 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi ae094e363c chore: fix a typo in log message 2023-08-24 12:25:04 +02:00
firest 6085442958 fix(ldap): expose request_timeout to prevent infinite hang 2023-08-24 16:50:38 +08:00
zhongwencool 2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics
Prometheus metrics
2023-08-24 12:06:32 +08:00
zhongwencool 8d2be27ade chore: add changlog for 11466 2023-08-24 06:46:50 +08:00
zhongwencool 84dd3b83bb fix: hocon converter not working when make_serializable is true(check_plain) 2023-08-24 06:46:47 +08:00
zhongwencool f72ba6819f
Merge pull request #11498 from zhongwencool/fix-otel-stop-error-log
fix: stop otel deps application in terminate crashed
2023-08-24 06:40:24 +08:00
zhongwencool d287b736ec chore: add changelog for 11497 2023-08-23 17:02:30 +08:00
zhongwencool 58be7f709b fix: stop otel deps application in terminate crashed 2023-08-23 16:45:41 +08:00
zhongwencool 6bb15541c7 feat: rename olp metrics to overload_protection metrics 2023-08-23 15:25:54 +08:00
zhongwencool b9a97923c4 feat: add some authz/connection missing metrics 2023-08-23 14:47:53 +08:00
zhongwencool 141676185d feat: add authentication metrics to prometheus 2023-08-23 14:15:16 +08:00
zhongwencool 4928c69758 feat: add authorization metrics to prometheus 2023-08-23 14:11:59 +08:00
zhongwencool df9f8da178 feat: add overload_protection metrics to prometheus 2023-08-23 14:07:37 +08:00
zhongwencool 188a81913e test: add prometheus api test 2023-08-23 11:23:43 +08:00
zhongwencool ae10415fc3 feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security 2023-08-23 09:03:22 +08:00
Paulo Zulato 61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators
fix(kinesis): set validators for static constraints
2023-08-22 18:28:35 -03:00
Paulo Zulato 20b46091f9 fix(kinesis): set validators for static constraints
Fixes https://emqx.atlassian.net/browse/EMQX-10833
2023-08-22 14:42:44 -03:00
Thales Macedo Garitezi f007b44426 fix(data_import): rm duplicate import call and fix test 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi ee2897e5de test(refactor): move test to integration tests dir 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c1f49abad2 test: fix inter-suite flakiness 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi dad27091be test: rm custom option 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi dbfacae283 fix: reinstate transactions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 65085d012b refactor: rename fn 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 3344bfb0bd refactor: rm `emqx_ds_replay` 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c46b8de938 test: remove unused things, refactor some functions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8c73b06e1 docs: add comment about future test failure 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 6de0bbe76a test(refactor): always allocate listeners for emqx app 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi a15405a800 test: fix assertions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c74abe79d0 refactor: reduce arity 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 021755b82b refactor: rm iterators from DS `#session{}` record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 3239f5ac5b feat: rm unnecessary transactions, use separate table for iterator references 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 8eab389ae1 perf: avoid unnecessary transaction 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi c28c6d1b7e fix: ensure iterator is opened 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 33ddbe80ad refactor: remove persistence leakeage from emqx_cm level 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi e8d7bb9a67 refactor: rename module 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 9463e271c0 feat(ds): open iterators when handling `SUBSCRIBE` packets
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi ca697a4e14 fix: rename `emqx_ds{,_replay}:replay_id()` 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi 45e2e687e5 fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication
Fixes https://emqx.atlassian.net/browse/EMQX-10843
2023-08-22 12:42:48 -03:00
Kjell Winblad 3ffbb7decf fix: HTTP API /api/v5/publish schema
The schema for the /api/v5/publish HTTP API endpoint was incorrect.
For 400 (Bad Request) error it cannot return a list but the incorrect
schema declared that the response could include a list.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10837
https://github.com/emqx/emqx/issues/11488
2023-08-22 15:21:20 +02:00
lafirest 7b8a599d17
Merge pull request #11486 from lafirest/fix/ldap_autn_wrong_pw
fix(ldap): Return correct reason code when password is wrong
2023-08-22 18:53:17 +08:00
lafirest f8f39bf223
Merge pull request #11490 from lafirest/fix/absent_pw
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 18:52:58 +08:00
firest e20b804e6b chore: update changes && test cases 2023-08-22 17:41:08 +08:00
firest 184e03387a fix(ldap): fix status detection and auto reconnecting errors 2023-08-22 17:06:00 +08:00
Kinple 663d03112e
Merge pull request #11476 from Kinplemelon/kinple/for-upgrade-dashboard
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-22 14:28:40 +08:00
firest b0c5b447a4 fix(authn): quickly return when the password is absent in password-based authentication 2023-08-22 11:49:32 +08:00
JimMoen 13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support
Hstreamdb tls support
2023-08-22 10:40:54 +08:00
firest 037b0a7599 fix(ldap): Return correct reason code when password is wrong 2023-08-22 09:42:04 +08:00
Paulo Zulato 4e4c93a758
Merge pull request #11469 from paulozulato/feat-redis-auth-username
Add support for AUTH with username on Redis
2023-08-21 16:44:21 -03:00
Thales Macedo Garitezi 74d4a8325e
Merge pull request #11470 from thalesmg/gcp-consumer-rm-worker-pool-size-r52-20230817
fix(gcp_consumer): remove unused resource_opts fields
2023-08-21 14:49:59 -03:00
Thales Macedo Garitezi bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816
fix(bridge): make dryrun health check timeout more malleable
2023-08-21 14:49:46 -03:00
Ivan Dyachkov 999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
Thales Macedo Garitezi 4163f06611
Merge pull request #11480 from thalesmg/rule-funcs-func-clause-r52-20230818
fix(rule_engine): capture function_clause errors
2023-08-21 11:16:25 -03:00
Kinplemelon 443d44aaa3 fix: set the `endpoint` in kinesis bridge as a required field 2023-08-21 11:49:57 +08:00
JimMoen f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
JimMoen 649647190e
test: new batch mechanism of hstreamdb_erl 2023-08-21 10:40:51 +08:00
JimMoen 5527edf442
chore: bump hstreamdb_erl to `0.4.3+v0.16.1` 2023-08-21 10:40:36 +08:00
Ilya Averyanov ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
Zaiming (Stone) Shi 01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7
chore: upgrade Kafka client wolff to 1.7.7
2023-08-20 10:14:13 +02:00
Paulo Zulato 05fdf52d8e fix(redis): set bridge as unhealthy on authentication error 2023-08-18 14:51:03 -03:00
Paulo Zulato fc35c74eab fix(redis): use eredis_secret to avoid password leaking 2023-08-18 14:51:03 -03:00
Paulo Zulato 2c458b62f5 feat(redis): add support for `username` on AUTH command
Fixes https://emqx.atlassian.net/browse/EMQX-9911
2023-08-18 14:50:06 -03:00
Thales Macedo Garitezi 66dec69d09 fix(rule_engine): capture function_clause errors
Fixes https://emqx.atlassian.net/browse/EMQX-10798
2023-08-18 13:51:19 -03:00
Thales Macedo Garitezi befc260555 fix(gcp_consumer): remove unused resource_opts fields
Fixes https://emqx.atlassian.net/browse/EMQX-10789
2023-08-17 17:02:02 -03:00
Thales Macedo Garitezi e48bfb52bb Merge branch 'release-52' into sync-r52-master-20230817 2023-08-17 17:01:04 -03:00
Paulo Zulato afffdbbaa1 fix(kinesis): replace default payload template
Fixes https://emqx.atlassian.net/browse/EMQX-10766
2023-08-17 17:00:14 -03:00
Ivan Dyachkov e2b9d905d7
Merge pull request #11465 from id/0718-e5.2.0-alpha.3 2023-08-17 20:59:13 +02:00
Andrew Mayorov 75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-17 20:43:09 +04:00
Ivan Dyachkov a0c2fe8cc1 chore: e5.2.0-alpha.3 2023-08-17 17:55:56 +02:00
Thales Macedo Garitezi ebecbd1545 fix(bridge): make dryrun health check timeout more malleable
Fixes https://emqx.atlassian.net/browse/EMQX-10773

- Makes the timeout for probing a bridge more malleable to account for differences between
  each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
  slowness/throttling.
2023-08-17 09:21:19 -03:00
Thales Macedo Garitezi e3572eb758
Merge pull request #11459 from thalesmg/kafka-health-check-interval-r52-20230816
feat(kafka): add option to configure health check interval
2023-08-17 09:04:30 -03:00
Thales Macedo Garitezi 029b461a13
Merge pull request #11448 from thalesmg/gcp-consumer-403-20230815
fix(gcp_consumer): handle 403 responses
2023-08-17 09:04:00 -03:00
ieQu1 7466024bca
Merge pull request #11458 from ieQu1/fix-excluded-apps-230916
fix(emqx_machine_boot): Fix excluded_apps
2023-08-17 11:36:23 +02:00
Ivan Dyachkov 7469222a17 Merge remote-tracking branch 'upstream/master' into 0817-e5.2.0-code-freeze 2023-08-17 08:16:04 +02:00
lafirest 5e448f5a02
Merge pull request #11446 from lafirest/refactor/calendar
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-17 07:40:44 +08:00
Andrew Mayorov 5d79823891
perf(topicidx): preserve next key on the stack to reuse later
This should further optimize the number of `ets:next/2` calls
required for a single match query.
2023-08-17 01:13:19 +04:00
Paulo Zulato 740680d54b
Merge pull request #11444 from paulozulato/fix-kinesis-msg-access-denied
fix(kinesis): return error message on access denied
2023-08-16 18:02:46 -03:00
Thales Macedo Garitezi f03733ba6b
Merge pull request #11460 from thalesmg/integration-tests-dir-20230816
ci: add `integration_test` dir to `emqx` application
2023-08-16 17:47:14 -03:00
Ilya Averyanov d17bcc94f6 chore(auth): update emqx_gcp_device tests 2023-08-16 22:37:42 +03:00
Paulo Zulato a64386ef82 fix(kinesis): return error message on access denied
Fixes https://emqx.atlassian.net/browse/EMQX-10764
2023-08-16 16:04:23 -03:00
Ilya Averyanov 3943830eae chore(auth): remove unnecessary atom conversions 2023-08-16 20:50:35 +03:00
Ilya Averyanov 0dae040047 chore(auth): restore injected schema key position 2023-08-16 20:50:35 +03:00
Ilya Averyanov db31e5f0d9 chore(auth): tidy up the code 2023-08-16 20:50:35 +03:00
Ilya Averyanov 67e06b3171 chore(auth): make schema injection be more universal 2023-08-16 20:41:05 +03:00
Ilya Averyanov abcdf18ca4 chore(auth): update ldap authn test 2023-08-16 20:41:05 +03:00
Ilya Averyanov f00f4568dc chore(auth): simplify schema injection 2023-08-16 20:41:05 +03:00
Ilya Averyanov 8a3e8ee528 chore(auth): add config propagation tests for authn 2023-08-16 20:41:05 +03:00
Ilya Averyanov cd88dfbbb8 chore(auth): update authn includes 2023-08-16 20:41:05 +03:00
Ilya Averyanov ae3355880e chore(auth): update emqx_authn tests 2023-08-16 20:41:05 +03:00
Ilya Averyanov 128aa19d47 chore(auth): improve emqx_config_handler to replace adhoc config updates 2023-08-16 20:41:05 +03:00
Ilya Averyanov 8213aa42c9 chore(auth): move authn leftovers to the authn app 2023-08-16 18:52:14 +03:00
Ilya Averyanov ca8c1e3ef8 chore(auth): removed direct usages of authn from emqx core app 2023-08-16 18:52:14 +03:00
Ilya Averyanov ce2b159022 chore(auth): make pre_hook_authenticate be part of emqx_access_control domain 2023-08-16 18:52:14 +03:00
zhongwencool e6f0dead9e
Merge pull request #11445 from zhongwencool/remove-os-mon-from-windows
fix: remove os_mon application in Windows release
2023-08-16 23:30:24 +08:00
Thales Macedo Garitezi a6a023eff7 ci: add `integration_test` dir to `emqx` application
This gives us a place where to put tests that exercise multiple umbrella application,
which is more closely related to `emqx`, but in a way that doesn't affect the standalone
app tests.
2023-08-16 11:48:47 -03:00
Thales Macedo Garitezi ffca581229 feat(kafka): add option to configure health check interval
Fixes https://emqx.atlassian.net/browse/EMQX-10781
2023-08-16 11:08:06 -03:00
ieQu1 7b44caeae4 fix(emqx_machine_boot): Fix excluded_apps 2023-08-16 16:04:35 +02:00
Andrew Mayorov c84ee66d80
Merge pull request #11450 from keynslug/fix/dirty-testsuite-flapping
fix(test): switch `emqx_flapping_SUITE` to cth tooling
2023-08-16 17:20:31 +04:00
zhongwencool e406cd7868
Merge pull request #11456 from zhongwencool/allow-empty-cacertfile-pem
fix: allow empty cacertfile pem
2023-08-16 21:10:52 +08:00
zhongwencool 78356d2625 chore: start os_mon in handle_continue/2 2023-08-16 20:54:53 +08:00
Thales Macedo Garitezi 82f27eaf78
Merge pull request #11429 from thalesmg/mongodb-legacy-opt-20230810
feat(mongodb): add configurable option to override legacy protocol usage
2023-08-16 09:54:53 -03:00
zhongwencool 8b23ee86b3 fix: hide cpu_status if os_check is not supported 2023-08-16 20:49:28 +08:00
zhongwencool b817e03c08 fix: start os_mon application temporary 2023-08-16 20:49:27 +08:00
zhongwencool 848eb7e3c3 fix: remove os_mon application in Windows release 2023-08-16 20:49:27 +08:00
zhongwencool b733adca06 fix: allow empty cacertfile pem 2023-08-16 20:15:21 +08:00
firest 040d28c35e fix(cli): fix two typos in the time unit 2023-08-16 18:46:17 +08:00
zhongwencool 1af0407c48 fix: don't crash when debug huge payload 2023-08-16 15:35:09 +08:00
firest 8cd21da94b chore: update apps version 2023-08-16 10:55:54 +08:00
firest b08102269a refactor(calendar): refactor datetime-related code and remove redundant 2023-08-16 10:55:46 +08:00
Thales Macedo Garitezi 4e80d669b0 fix(influxdb_bridge): avoid double-parsing write syntax during probe
Fixes https://emqx.atlassian.net/browse/EMQX-10771
2023-08-15 17:05:57 -03:00
Andrew Mayorov 0b21b87296
fix(test): switch `emqx_flapping_SUITE` to cth tooling 2023-08-15 21:29:42 +04:00
Andrew Mayorov 5560e6ed4f
fix(cthsuite): also prevent `emqx` from loading default conf 2023-08-15 21:25:39 +04:00
Thales Macedo Garitezi 5d707c8b7d
Merge pull request #11403 from thalesmg/gcp-produ-attr-20230728
feat(gcp_producer): add support for defining message attributes and ordering key
2023-08-15 13:22:31 -03:00
Thales Macedo Garitezi ba956ebe88 fix(gcp_consumer): handle 403 responses
Fixes https://emqx.atlassian.net/browse/EMQX-10736
2023-08-15 13:20:20 -03:00
Zaiming (Stone) Shi fa9e9f3348
Merge pull request #11442 from zmstone/0814-sync-release-52-to-master
0814 sync release 52 to master
2023-08-15 16:21:55 +02:00
Andrew Mayorov e39bbf4c49
chore(topicidx): add more descriptive comments and specs
To (hopefully) better illustrate what is happening there.
2023-08-15 16:55:48 +04:00
Thales Macedo Garitezi 23f5cea482 feat: handle strange key values when resolving placeholders 2023-08-14 13:39:38 -03:00
Thales Macedo Garitezi d93e1bbf08 feat(mongodb): add configurable option to override legacy protocol usage
Fixes https://emqx.atlassian.net/browse/EMQX-10750

Fixes https://github.com/emqx/emqx/discussions/11428

See https://github.com/emqx/mongodb-erlang/pull/39
2023-08-14 13:19:50 -03:00
Thales Macedo Garitezi 926eb4e3dd test: rm unused var warning 2023-08-14 10:33:24 -03:00
Thales Macedo Garitezi 82b8538041 feat(gcp_producer): add support for defining message attributes and ordering key
Fixes https://emqx.atlassian.net/browse/EMQX-10652
2023-08-14 10:33:17 -03:00
lafirest 1bda8020f5
Merge pull request #11436 from lafirest/feat/banned_clear
feat(banned): add a new API used to clear all banned data
2023-08-14 20:48:43 +08:00
lafirest 6bea894205
Merge pull request #11440 from lafirest/fix/ldap_dep
fix(ldap): fix dependency problem
2023-08-14 20:34:58 +08:00
Andrew Mayorov fe9477f92e
chore: bump applications versions
* emqx_rule_engine 5.0.23
2023-08-14 15:36:58 +04:00
Andrew Mayorov 47dfba4341
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching 2023-08-14 15:36:58 +04:00
Andrew Mayorov 9a249e4b01
test(topicidx): increase test coverage 2023-08-14 15:36:58 +04:00
Andrew Mayorov fd0986071c
perf(topicidx): implement fast-forwarding prefixes
This should give less `ets:next/2` calls in general and much less
when index has relatively small number of long non-wildcard topics.
2023-08-14 15:36:58 +04:00
Andrew Mayorov 0c7bdbdab4
test(topicidx): add property test
Co-Authored-By: JianBo He <heeejianbo@gmail.com>
2023-08-14 15:36:57 +04:00
Andrew Mayorov 48a50c9137
fix(topicidx): fix missing matches when 'a/b' and 'a/b/#' both exist
Thanks to @HJianBo for spotting this issue. The approach to fix it
is different though: we try to keep the "recurrency branch factor"
to a minimum, instead introducing new condition for the case when
filter does not match, but iteration with `ets:next/2` is not yet
finished for the prefix.

Co-Authored-By: JianBo He <heeejianbo@gmail.com>
2023-08-14 15:36:57 +04:00
Andrew Mayorov 6a13406363
fix(topicidx): use custom topic words to keep required ordering
Otherwise, topic with empty tokens (e.g. `a/b///c`) would have
some of their tokens ordered before `#` / `+`, because empty
token was represented as empty atom (`''`).
2023-08-14 15:36:57 +04:00
Andrew Mayorov 28bcb394d1
fix(topicidx): allow to return matches unique by record id 2023-08-14 15:36:57 +04:00
Andrew Mayorov 8feda315f6
feat(index): add topic index facility
Somewhat similar to `emqx_trie` in design and logic, yet built on
top of a single, potentially pre-existing table.
2023-08-14 15:36:54 +04:00
Zaiming (Stone) Shi 970ff28200 Merge remote-tracking branch 'origin/master' into release-52 2023-08-14 11:50:40 +02:00
firest 0b066fa20c fix(ldap): fix dependency problem 2023-08-14 16:59:37 +08:00
zhongwencool 41180170f4 fix: update error msg 2023-08-14 15:45:57 +08:00
zhongwencool e10f9e5e9b feat: change mqtt.max_packet_size type from string to bytesize 2023-08-14 10:10:46 +08:00
lafirest b8d3dcf59a
Merge pull request #11424 from lafirest/fix/epoch_maximum_val
fix(datetime): make sure the epoch is not larger than the maximum supported value
2023-08-11 20:37:56 +08:00
firest 749c2d075f chore: update changes && bump app version 2023-08-11 17:57:27 +08:00
firest bc417a6764 fix(datetime): make sure the epoch is not larger than the maximum supported value 2023-08-11 17:57:21 +08:00
firest 2fc0468e0c feat(banned): add a new API used to clear all banned data 2023-08-11 17:25:49 +08:00
Ivan Dyachkov 7852af896c chore: e5.2.0-alpha.1 2023-08-11 11:16:30 +02:00
Ivan Dyachkov bab05b6613
Merge pull request #11433 from id/0811-ci-upgrade-push-helm-action 2023-08-11 11:03:00 +02:00
Ivan Dyachkov b6084b0001 chore: bump apps/emqx version 2023-08-11 09:19:51 +02:00
lafirest 4eaced241d
Merge pull request #11423 from lafirest/chore/limiter_utils
fix(limiter): simplify the code of the limiter
2023-08-11 15:17:42 +08:00
JimMoen 060074e04a
fix: api total accumulate ignore `undefined` by down nodes result 2023-08-11 11:27:55 +08:00
Thales Macedo Garitezi 65aee8870b
Merge pull request #11427 from id/0810-cut-v5.1.5-build.3
v5.1.5-build.3
2023-08-10 09:52:58 -03:00
Paulo Zulato 9ca9c65af2
Merge pull request #11367 from paulozulato/feat-gcp-devices
feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
2023-08-10 09:44:55 -03:00
Ivan Dyachkov 801c93dd46 chore: v5.1.5-build.3 2023-08-10 12:56:28 +02:00
Ivan Dyachkov 6ead09b28d chore: merge 'upstream/master' 2023-08-10 12:56:00 +02:00
firest 6dbddfb089 fix(limiter): simplify the code of the limiter
move runtime code out from the schema
2023-08-10 18:20:05 +08:00
zhongwencool aa798e531e
Merge pull request #11409 from zhongwencool/opentelemetry-metrics
feat: support opentelemetry metrics
2023-08-10 14:35:25 +08:00
zhongwencool d1dc37af4c
Merge pull request #11279 from zhongwencool/trace-log
fix: don't be kill when send large payload when log is debug
2023-08-10 11:58:31 +08:00
lafirest b2394cf92d
Merge pull request #11399 from lafirest/fix/ph_utf8
fix(placeholder): porting fix to support utf8 key in placeholder
2023-08-10 09:41:39 +08:00
zhongwencool 5a4dd3a5e5 fix: truncate large payload 2023-08-10 09:35:12 +08:00
zhongwencool d5fe891961 fix: don't be kill when send large payload when log is debug 2023-08-10 09:22:30 +08:00
Thales Macedo Garitezi bb8ce68c2e docs: Generate changelog for v5.1.5-patch.2 2023-08-09 16:38:28 -03:00
Thales Macedo Garitezi 95df958f10 docs: Generate changelog for v5.1.5-patch.1 2023-08-09 13:18:09 -03:00
Thales Macedo Garitezi cce0d144f9 Merge branch 'master' into release-v515-patch1-20230809 2023-08-09 13:08:43 -03:00
Paulo Zulato a85c948e23 feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
Fixes https://emqx.atlassian.net/browse/EMQX-10341
2023-08-09 10:48:25 -03:00
zhongwencool 3933227636 feat: support opentelemetry metrics 2023-08-09 20:23:40 +08:00
lafirest bef8eddad3
Merge pull request #11405 from lafirest/fix/kal_kialo
fix(calendar): make date parse error reason more sense
2023-08-09 14:42:08 +08:00
lafirest 524d33a1fa
Merge pull request #11407 from lafirest/fix/config_name_example
fix(ldap): improve configuration name and docs
2023-08-09 14:41:53 +08:00
firest 3e9155fdb1 chore: bump emqx_utils version && changes 2023-08-09 10:54:56 +08:00
firest 7567d211da fix(placeholder): porting fix to support utf8 key in placeholder 2023-08-09 10:54:41 +08:00
firest 177ec161a1 fix(ldap): improve configuration name and docs 2023-08-09 09:39:57 +08:00
Thales Macedo Garitezi fdfdb105f0 docs: Generate changelog for v5.1.5 2023-08-08 10:33:34 -03:00
Thales Macedo Garitezi a1ffd0263d fix(cacerts): add `cacerts` back as deprecated to allow upgrading from older versions
See https://github.com/emqx/emqx/issues/11380
2023-08-08 10:13:13 -03:00
Kjell Winblad b38461e50a fix: mongo_date/2 shall give user friendly value in the test environment 2023-08-08 14:47:07 +02:00
Ilya Averyanov 9245162394 chore(rebalance): fix flaky test 2023-08-08 13:06:25 +03:00
firest 9f38f5f26a fix(calendar): make date parse error reason more sense 2023-08-08 14:28:38 +08:00
Thales Macedo Garitezi b6decf9592
Merge pull request #11402 from thalesmg/dynamic-kconsu-mqtt-topics-20230807
feat(kafka_consumer): add mqtt topic placeholder support
2023-08-07 15:10:29 -03:00
Thales Macedo Garitezi 29e706c83d refactor: move catch to dry run fn 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi 4b8cea4498 fix: make error handling more flexible 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi 963d63b899 refactor: rename temporary name
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi 5c8dc092a1 fix(http_bridge): don't attempt to convert headers to atoms
Fixes https://emqx.atlassian.net/browse/EMQX-10653
2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi d5847f33a8 test: attempt to stabilize flaky tests 2023-08-07 13:08:34 -03:00
Thales Macedo Garitezi 9900a32850 feat(kafka_consumer): add mqtt topic placeholder support
Fixes https://emqx.atlassian.net/browse/EMQX-10678
2023-08-07 11:42:34 -03:00
Kjell Winblad 3ed031db70 fix: rule SQL mongo_date function should return a string in test mode
The rule SQL mongo_date function should return a string with the format
ISODate(*), where * is an ISO date string when running the rule in test
mode.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10727
2023-08-07 16:10:30 +02:00
lafirest 2b03436552
Merge pull request #11392 from lafirest/feat/ldap_authz
feat(ldap-authz): integrate the LDAP authorization
2023-08-07 11:12:05 +08:00
zhongwencool 665695a977
Merge pull request #11388 from SergeTupchiy/EMQX-10703-fix-replicant-crash-when-core-terminates-abnormally
fix: increase emqx_router_sup restart intensity
2023-08-06 14:40:32 +08:00
Serge Tupchii 2b7798608d fix(emqx_router_helper): don't cleanup down node on a replicant
The cleanup on a replicant node is redundant, as Mria would delegate this delete op
to a core node (via RPC), and the core node is expected to receive the same
`nodedown` message and process it.
2023-08-04 16:55:36 +03:00
SergeTupchiy 2044326734
Merge pull request #11390 from SergeTupchiy/EMQX-10661-improve-perf-high-latency-cluster-network
EMQX-10661 improve performance on high latency cluster network
2023-08-04 16:41:08 +03:00
Zaiming (Stone) Shi 115ab85670 chore: upgrade Kafka client wolff to 1.7.7 2023-08-04 15:31:34 +02:00
firest b24a9d5343 chore(ldap-authz): update apps version && changes 2023-08-04 18:14:09 +08:00
firest 0571fd8cac feat(ldap-authz): integrate the LDAP authorization 2023-08-04 18:14:01 +08:00
Serge Tupchii f276ea9e91 fix: increase emqx_router_sup restart intensity
The goal is to tolerate occasional crashes that can happen under relatively normal conditions
and don't seem critical to shutdown the whole app (emqx).
For example, mria write/delete call delegated from a replicant to a core node may fail,
if the core node is being stopped / restarted / not ready.

Fixes: EMQX-10703, #11310
2023-08-04 10:49:59 +03:00
firest 7055eafb91 fix(ldap): fix license date and some minor problems 2023-08-04 11:01:13 +08:00
lafirest 1b0b15786c
Merge pull request #11386 from lafirest/feat/ldap_authn
feat(authn): integrate the LDAP authentication
2023-08-04 09:37:37 +08:00
Andrew Mayorov 420653e5a2
Merge pull request #11389 from keynslug/perf/EMQX-10706/squash-retained-index
perf(retainer): squash index updates into single dirty activity
2023-08-03 21:32:04 +04:00
Serge Tupchii 466fe7e009 perf: add broker_pool_size, generic_pool_size and channel_cleanup_batch_size config options
Tuning these options can improve performance if cluster interconnect network latency is high.

Fixes: EMQX-10661
2023-08-03 19:44:00 +03:00
Serge Tupchii ed28c12a66 chore(emqx_conf): deprecate node.tlog_push_mode setting
It is not used since mria 0.2.10
2023-08-03 17:46:14 +03:00
Andrew Mayorov 0e0c0d0e1e
chore: bump `emqx_retainer` to 5.0.16 2023-08-03 15:35:21 +04:00
Andrew Mayorov 534a741b0e
perf(retainer): do not GC on replicant nodes 2023-08-03 15:34:24 +04:00
firest efff585b82 feat(ldap-authn): add test suite for the LDAP authenticator 2023-08-03 18:56:41 +08:00
Andrew Mayorov 42be694d40
perf(retainer): squash index updates into single dirty activity
In order to minimize number of round trips to core nodes and between
them, improving publishing latency. This shouldn't make consistency
worse than it was before.
2023-08-03 12:51:36 +04:00
Andrew Mayorov 39bb9d5136
chore: bump ekka to 0.15.10
Which sports mria 0.6.0 with dirty activities support.
2023-08-03 11:38:53 +04:00
firest 18b3db336a fix(ldap-authn): remove slat-related configs since they conflict with RFC 3123 2023-08-03 11:00:42 +08:00
Paulo Zulato 2c63fe3bcd fix(greptimedb): add sync_start to avoid false negative healthy check
Fixes https://emqx.atlassian.net/browse/EMQX-10709
2023-08-02 19:40:23 -03:00
Paulo Zulato 97fdc834de
Merge pull request #11381 from paulozulato/fix-oracle-doc-link
doc(oracle): add link to online documentation into README
2023-08-02 12:10:59 -03:00
firest c041216ec0 feat(authn): integrate the LDAP authentication 2023-08-02 19:19:17 +08:00
firest 9bb5c9de33 fix(ldap): make elvis happy 2023-08-02 11:02:02 +08:00
firest b2f3ac9967 fix(ldap): fix newline error && correcting files header 2023-08-02 10:07:27 +08:00
Paulo Zulato b8be080ffb docs(oracle): add link to online documentation into README
Fixes https://emqx.atlassian.net/browse/EMQX-10694
2023-08-01 16:20:31 -03:00
firest f98f97f37e feat(ldap): set test env and add test suites 2023-08-01 18:49:54 +08:00
firest 8c9b136d15 fix(ldap): integrate parser and fix lexer errors 2023-08-01 18:49:54 +08:00
firest fa6343cc80 feat(ldap): add LDAP connector 2023-08-01 18:49:54 +08:00
Ivan Dyachkov 0f161a81c3 chore: v5.1.4 2023-08-01 09:20:46 +02:00
Thales Macedo Garitezi 41606aa840
Merge pull request #11373 from thalesmg/fix-spellcheck-master-20230731
fix(spellcheck): fix several spellcheck issues not caught in CI
2023-07-31 18:00:31 -03:00
Thales Macedo Garitezi 620b5fc048 fix(spellcheck): fix several spellcheck issues not caught in CI
Fixes some issus later found by CI that somehow didn't block the originating PRs.

Example failure:

https://github.com/emqx/emqx/actions/runs/5715470851/job/15485284918?pr=11372
2023-07-31 14:32:12 -03:00
Thales Macedo Garitezi 7687770821 fix(tls): remove `cacerts` config for now
Fixes https://github.com/emqx/emqx/issues/11370

Related: https://github.com/emqx/emqx/pull/11371
2023-07-31 10:44:05 -03:00
Thales Macedo Garitezi a3be1529c6 fix(schema): ensure enterprise module is correctly loaded
Fixes https://emqx.atlassian.net/browse/EMQX-10654

Despite loading the application in `nodetool`, we need to invoke `:module_info()` to force
the module to be actually loaded, especially when it's called in `call_hocon` to generate
the initial configurations.  Otherwise, it'll fail to list all the bridge schemas.
2023-07-28 16:27:43 -03:00
Thales Macedo Garitezi d6344ab709
Merge pull request #11363 from thalesmg/rabbit-tls-20230727
feat(rabbitmq_bridge): add TLS support
2023-07-28 13:23:30 -03:00
Thales Macedo Garitezi 2ee1aa6d60
Merge pull request #11347 from thalesmg/fix-ocsp-path-encoding-20230725
fix(ocsp): URL encode request path
2023-07-28 09:01:39 -03:00
Thales Macedo Garitezi 399f849f7b feat(rabbitmq_bridge): add TLS support
Fixes https://emqx.atlassian.net/browse/EMQX-10605
2023-07-27 15:52:46 -03:00
Ivan Dyachkov d51b5b739b chore: v5.1.3 2023-07-27 17:00:20 +02:00
Ivan Dyachkov 5ac01c9b85
Merge pull request #11360 from id/0727-sync-release-51-to-master 2023-07-27 16:58:09 +02:00
Thales Macedo Garitezi ee3d002fca
Merge pull request #11329 from thalesmg/aeh-bridge-20230720
feat: add Azure Event Hub Producer bridge
2023-07-27 11:18:40 -03:00
Ivan Dyachkov 63adeabf72 chore: bump app versions 2023-07-27 15:29:03 +02:00
Ivan Dyachkov cbfca8c043 chore: merge master into release-51 2023-07-27 15:19:57 +02:00
Thales Macedo Garitezi 5abe4bed88 feat: add Azure Event Hub Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10338
2023-07-27 09:22:39 -03:00
Ivan Dyachkov 50a0900d92 chore: e5.1.1 2023-07-27 12:18:03 +02:00
JianBo He 951a96457b
Revert "feat(index): add topic index facility " 2023-07-27 13:42:43 +08:00
JianBo He 5e4855334e
Revert "Fix(topicidx): allow to return matches unique by record id" 2023-07-27 13:39:37 +08:00
firest abf8d8b391 fix(ds): make dialyzer happy 2023-07-26 16:35:07 +08:00
firest 14b76916df fix(ds): avoid crashes when starting on Windows 2023-07-26 15:37:58 +08:00
Thales Macedo Garitezi 2ac8ba2add test: fix flaky test 2023-07-25 17:43:33 -03:00
Thales Macedo Garitezi d3d52695d5 feat(tls): automatically add `cacerts` to client opts
`public_key:cacerts_get/0` was introduced in OTP 25 and allows us to
load the system trusted CA certificates.

https://www.erlang.org/doc/man/public_key.html#cacerts_get-0
2023-07-25 17:43:33 -03:00
Thales Macedo Garitezi d0deaca00d fix(ocsp): URL encode request path
Fixes https://emqx.atlassian.net/browse/EMQX-10624
2023-07-25 17:39:38 -03:00
Serge Tupchii 77184c00ff chore(ekka): Bump version to 0.15.9 2023-07-25 19:29:18 +03:00
Ivan Dyachkov f9d3d3325b chore: e5.1.1-rc.1 2023-07-25 16:34:25 +02:00
Ilya Averyanov 2909b1de2d
Merge pull request #11339 from savonarola/0724-fix-subscribe-qos-authz
fix(authz): correctly identify qos of subscribe actions
2023-07-25 17:07:57 +03:00
Thales Macedo Garitezi c7bf3ece34
Merge pull request #11338 from thalesmg/fix-pgsql-error-r51-20230724
fix(postgres_bridge): fix table existence check and handle sync_required
2023-07-25 10:53:03 -03:00
Paulo Zulato 7ee17553d5
Merge pull request #11337 from paulozulato/fix-error-code-topic-wildcard
fix(topic_rewrite): handle error when target contains wildcards
2023-07-25 10:37:20 -03:00
Paulo Zulato 8ec14bb07e fix(topic_rewrite): handle error when target contains wildcards
Fixes https://emqx.atlassian.net/browse/EMQX-10565
2023-07-25 09:43:14 -03:00
Ilya Averyanov deaac9bd73 fix(authz): correctly identify qos of subscribe actions 2023-07-25 15:20:52 +03:00
Thales Macedo Garitezi 7a16ff4f04 fix(postgres_bridge): fix table existence check and handle sync_required
Fixes https://emqx.atlassian.net/browse/EMQX-10629

During health checking, we check whether tables in the SQL statement
exist.  Such check was done by asking the backend to parse the
statement using a named prepared statements.  Concurrent health checks
could then result in the error:

```erlang
{error,{error,error,<<"42P05">>,duplicate_prepared_statement,<<"prepared statement \"get_status\" already exists">>,[{file,<<"prepare.c">>},{line,<<"451">>},{routine,<<"StorePreparedStatement">>},{severity,<<"ERROR">>}]}}
```

This could lead to an inconsistent state in the driver process, which
would crash later when a message from the backend (`READY_FOR_QUERY`, "idle"):

```
  2023-07-24T13:05:58.892043+00:00 [error] Generic server <0.2134.0> terminating. Reason: {'module could not be loaded',[{undefined,handle_message,[90,<<"I">>,...
```

Added calls to `epgsql:sync/1` for functions that could return
`{error, sync_required}`.

Also, redundant calls to `parse2` were removed to reduce the number of requests.
2023-07-25 09:15:14 -03:00
zhongwencool 2c193f26e6
Merge pull request #11332 from HJianBo/fix-topic-index-unique-bugs 2023-07-25 19:07:07 +08:00
JianBo He d05a5cfe0f fix(rule): fix the `matches/2` for some edge cases 2023-07-25 14:35:18 +08:00
Paulo Zulato 69f4275871 fix(oracle): fix return error checking on table validation
Fixes https://emqx.atlassian.net/browse/EMQX-10622
2023-07-24 14:50:38 -03:00
JianBo He e630331de1 fix(rule): fix a quering problem when 'a/b' and 'a/b/#' exist at the same time.
When using `ets:next` to query the next level of topic words, we should prioritize the next level
of '#', '+'.
2023-07-24 23:04:53 +08:00
JianBo He f848c0b87a
Merge pull request #11333 from JimMoen/chore-hide-hstreamdb
chore: hidden hstreamdb bridge schema
2023-07-24 21:54:10 +08:00
JianBo He 511d1b6ca1 chore: hide the hstreamdb http api 2023-07-24 20:11:44 +08:00
JianBo He dcf4819c04 test(rule): add tests to ensure the rules ordering 2023-07-24 19:30:34 +08:00
firest 8439ce0e84 chore: update app version && reboot_lists 2023-07-24 19:08:41 +08:00
Dennis Zhuang 9f200120c2 feat: use http port to detect server availability 2023-07-24 17:58:48 +08:00
Dennis Zhuang cd9d5f287e chore: adds auto_reconnect for ecpool 2023-07-24 17:58:48 +08:00
Dennis Zhuang 2ea903c5ac fix: static checks failures 2023-07-24 17:58:48 +08:00
Dennis Zhuang b34374c26f chore: by CR comments 2023-07-24 17:58:48 +08:00
Dennis Zhuang c9550cc2e5 refactor: rename bridge greptimedb_grpc_v1 to greptimedb 2023-07-24 17:58:48 +08:00
Dennis Zhuang a1c7eb337b fix: dependency name 2023-07-24 17:58:48 +08:00
Dennis Zhuang 50c10dd919 chore: update greptimedb-client-erl to v0.1.2 2023-07-24 17:58:48 +08:00
Dennis Zhuang 3b1363dbb7 chore: change license to BCL and adds emqx_bridge_greptimedb_connector_SUITE 2023-07-24 17:58:48 +08:00
Dennis Zhuang 6f7fbcf693 fix: compile error 2023-07-24 17:58:48 +08:00
Dennis Zhuang 4921856950 test: make all emqx_bridge_greptimedb_SUITE tests passing 2023-07-24 17:58:48 +08:00
Dennis Zhuang c6a7f3e2ad test: make test passed 21/29 2023-07-24 17:58:48 +08:00
Dennis Zhuang 975795a6e0 feat: add ci test 2023-07-24 17:58:48 +08:00
Dennis Zhuang 89bce99870 test: greptimedb data brige 2023-07-24 17:58:48 +08:00
Dennis Zhuang 91ebd90442 fix: batch write 2023-07-24 17:58:48 +08:00
Dennis Zhuang 6d9944a8e8 feat: update greptimedb dependencies 2023-07-24 17:58:48 +08:00
Dennis Zhuang 417e017498 feat: begin to impl connector 2023-07-24 17:58:48 +08:00
Dennis Zhuang c5078980f3 feat: adds the greptimedb bridge to emqx modules 2023-07-24 17:58:48 +08:00
Dennis Zhuang 4d581d0b30 feat: init commit 2023-07-24 17:58:48 +08:00
Andrew Mayorov 0496038361 fix(ruleeng): ensure topic index matched rules evalauted once 2023-07-24 17:44:12 +08:00
Andrew Mayorov 6432c9c8fc fix(topicidx): allow to return matches unique by record id 2023-07-24 17:43:11 +08:00
JimMoen 613dc1646c
chore: hidden hstreamdb bridge schema 2023-07-24 17:22:29 +08:00
firest af6405fa25 fix(nested_put): fix a data loss bug introduced by #11172 2023-07-24 16:39:37 +08:00
Andrew Mayorov 81793c31fc
Merge pull request #11315 from keynslug/ft/EMQX-9593/peek-mqueue-info
refactor(session): allow peeking at mqueue less intrusively
2023-07-24 09:57:05 +02:00
JimMoen c393c2e091
test: ets table cleanup after cases 2023-07-24 09:33:28 +08:00
JimMoen 4e4b1ac115
refactor: module move to app emqx_rule_engine
- Rename to emqx_rule_index.erl
- Remove test funcs from src -> test dir
2023-07-24 09:33:28 +08:00
Andrew Mayorov b821bdee00
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching 2023-07-24 09:27:31 +08:00
Andrew Mayorov 28dad5d7a9
feat(index): add topic index facility
Somewhat similar to `emqx_trie` in design and logic, yet built on
top of a single, potentially pre-existing table.
2023-07-24 09:27:31 +08:00
Paulo Zulato 31cf05ec66
Merge pull request #11316 from paulozulato/fix-oracle-pool-size
fix(oracle): fix Pool Size parameter retrieval
2023-07-21 17:51:21 -03:00
Thales Macedo Garitezi 2531c3e7d1
Merge pull request #11306 from thalesmg/rule-actions-reply-dropped-r51-20230719
fix(rule_metrics): notify rule metrics of late replies and expired requests
2023-07-21 16:51:14 -03:00
Ivan Dyachkov 29432009db chore: 5.1.1-alpha.4 2023-07-21 18:42:14 +02:00
ieQu1 485e495b98 chore(ekka): Bump version to 0.15.8 (mria 0.5.10) 2023-07-21 18:31:02 +02:00
Paulo Zulato 85ab97970f fix(oracle): fix Pool Size parameter retrieval
Fixes https://emqx.atlassian.net/browse/EMQX-10599
2023-07-21 11:02:27 -03:00
Ivan Dyachkov c2c3da073f chore: v5.1.2 2023-07-21 13:26:29 +02:00
Ivan Dyachkov 243b8f5b67 chore: merge 'upstream/master' into v5.1.2 2023-07-21 13:25:46 +02:00
Ivan Dyachkov 3a3b843f4a fix(machine): add emqx_bridge_kinesis to reboot_lists.eterm 2023-07-21 13:20:50 +02:00
SergeTupchiy 04bd7c7eed
Merge pull request #11322 from SergeTupchiy/EMQX-10590-data-import-fix-missing-config-rel51
EMQX-10590 data import fix missing config relealse 51
2023-07-21 12:36:21 +03:00
Serge Tupchii ee97cf8e33 fix(emqx_modules): ignore already_existed/not_found errors for added/removed topic metrics 2023-07-21 12:17:47 +03:00
zhongwencool 520ae0e9d3 chore: remove zone desc from document 2023-07-21 15:51:21 +08:00
Serge Tupchii b37920d2e5 feat: add `topic_metrics` and `slow_subs` configuration to data import/export
Fixes: EMQX-10590
2023-07-21 10:41:22 +03:00
Serge Tupchii a1a7c4fab6 fix(emqx_rule_engine): fix typo in `behaviour` module attribute
The type resulted in missing `rule_engine` config after importing data from a backup file.

Fixes: EMQX-10590
2023-07-21 10:33:56 +03:00
zhongwencool d3e4fda21d fix: hocon_pp quote string for special char 2023-07-21 12:31:22 +08:00
Paulo Zulato 0827a45c89
Merge pull request #11307 from paulozulato/fix-oracle-table-check
fix(oracle): discard nested tokens when checking table
2023-07-20 19:19:50 -03:00
Andrew Mayorov e1e4c64a30
refactor(session): allow peeking at mqueue less intrusively 2023-07-20 21:21:30 +02:00
Thales Macedo Garitezi 54efc04a6f
Merge pull request #11309 from thalesmg/fix-machine-boot-reboot-r51-20230719
fix(machine_boot): use shared list of reboot apps and add bridges to reboot list
2023-07-20 14:33:16 -03:00
Thales Macedo Garitezi 6cd503865b fix(machine_boot): ensure `emqx_bridge` starts after its companion apps
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
2023-07-20 13:11:44 -03:00
Thales Macedo Garitezi 3b1e436d3f refactor: use `emqx_pool:async_submit` to avoid excessive spawning 2023-07-20 10:17:43 -03:00
JianBo He b9f0cd7ba4 chore: pretty the cli output style 2023-07-20 21:01:28 +08:00
Thales Macedo Garitezi 77e2d852e5 fix: add emqx_durable_storage to app list 2023-07-20 09:55:05 -03:00
Andrew Mayorov f2a32e8ed2
Merge pull request #11304 from keynslug/ft/EMQX-9593/ds-session-connect
refactor(session): minimize `emqx_session` interface surface
2023-07-20 14:33:21 +02:00
Thales Macedo Garitezi 89ea40a8b7 refactor: apply review remarks 2023-07-20 09:31:13 -03:00
zhongwencool 3cd223ff5b chore: add more detail msg for merging failed 2023-07-20 17:48:26 +08:00
SergeTupchiy 8480a26b66
Merge pull request #11065 from SergeTupchiy/EMQX-9992-fix-async-cleanup-error-logs
fix: avoid logging unnecessary errors in async cleanup functions
2023-07-20 12:20:57 +03:00
Thales Macedo Garitezi b9b11d8f4d fix(machine_boot): use shared list of reboot apps and add bridges to reboot list 2023-07-19 20:15:42 -03:00
Paulo Zulato e76ba760c9 fix(oracle): discard nested tokens when checking table
Fixes https://emqx.atlassian.net/browse/EMQX-10597
2023-07-19 15:03:49 -03:00
Thales Macedo Garitezi eb41b77de4 fix(rule_metrics): notify rule metrics of late replies and expired requests
Fixes https://emqx.atlassian.net/browse/EMQX-10600
2023-07-19 11:39:28 -03:00
Serge Tupchii ff4119e8b3 chore(emqx_config): fix typo in the log message 2023-07-19 16:58:21 +03:00
Thales Macedo Garitezi 41b780f7f2
Merge pull request #11280 from thalesmg/unhide-gcp-consumer-master-20230717
chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
2023-07-19 10:26:42 -03:00
Thales Macedo Garitezi 6003c34ce5
Merge pull request #11283 from thalesmg/flaky-test-20230717
test(http): attempt to stabilize flaky tests
2023-07-19 10:24:07 -03:00
ieQu1 57e39f42c5
Merge pull request #11294 from ieQu1/mria-fixes
Mria fixes
2023-07-19 15:10:42 +02:00
Andrew Mayorov dd31487b4a
refactor(session): drop `is_session/1` helper as useless
This is a part of effort to minimize `emqx_session` module
interface to simplify adding alternative session implementations.
2023-07-19 14:55:26 +02:00
Andrew Mayorov 0cd2351134
refactor(session): hide `no_local` logic behind enqueue / deliver
This is a part of effort to minimize `emqx_session` module
interface to simplify adding alternative session implementations.
2023-07-19 14:55:26 +02:00
Ivan Dyachkov f29a9ed9d5
Merge pull request #11301 from id/0719-sync-release-51-to-master 2023-07-19 14:21:26 +02:00
lafirest d4550f911c
Merge pull request #11299 from lafirest/hs
fix(hstream): handle the `unavailable` error code
2023-07-19 17:47:30 +08:00
Andrew Mayorov 92166f287a
Merge pull request #11293 from keynslug/fix/EMQX-9591/phase-out-persistence
chore(ps): phase out existing session persistence mechanism
2023-07-19 10:36:46 +02:00
Ivan Dyachkov b7f0f7b32d chore: merge master into release-51 2023-07-19 09:28:05 +02:00
firest 7ac8715358 fix(hstream): handle the `unavailable` error code
The "unavailable" error code just means that the HStream is warming up or busy, HStream guarantees that a retry will work well.
2023-07-19 14:43:50 +08:00
zhongwencool 0524c3b3b1 test: configs SUITE crash 2023-07-19 11:05:14 +08:00
JianBo He 439eb4d029 test: cover the node params for `get /configs` endpoint 2023-07-19 11:05:14 +08:00
JianBo He 74532a64e6 chore: update apps/emqx_management/src/emqx_mgmt_api_configs.erl 2023-07-19 11:05:14 +08:00
zhongwencool 402f75592e chore: move rpc call from emqx_management_proto to emqx_conf_proto 2023-07-19 11:05:14 +08:00
zhongwencool 8a0f7bfd99 chore: update version bpapi.versions 2023-07-19 11:05:14 +08:00
zhongwencool 59ed65787f fix: get_config return target node's node conf 2023-07-19 11:05:14 +08:00
Paulo Zulato 27630ca215
Merge pull request #11261 from paulozulato/feat-bridge-kinesis
Implement Amazon Kinesis Producer bridge
2023-07-18 16:48:34 -03:00
ieQu1 3a5112829c fix(mgmt_cli): Don't print cluster status on the replicant 2023-07-18 21:45:00 +02:00
ieQu1 ce76d4ab04 chore(ekka): Bump version to 0.15.7 (mria 0.5.9) 2023-07-18 21:45:00 +02:00
ieQu1 103b89f60c fix(mgmt_cli): Don't crash when mria is not running 2023-07-18 21:45:00 +02:00
ieQu1 293700773f refactor(mgmt_cli): Call mria directly 2023-07-18 21:45:00 +02:00
Serge Tupchii 2954ff7300 feat: add `topic_metrics` and `slow_subs` configuration to data import/export
Fixes: EMQX-10590
2023-07-18 20:53:10 +03:00
Serge Tupchii c9985758d9 fix(emqx_rule_engine): fix typo in `behaviour` module attribute
The type resulted in missing `rule_engine` config after importing data from a backup file.

Fixes: EMQX-10590
2023-07-18 20:35:21 +03:00
Ivan Dyachkov de4cc492d4 chore: e5.1.1-alpha.2 2023-07-18 18:55:24 +02:00
Thales Macedo Garitezi 4ec4041f99
Merge pull request #11281 from thalesmg/restore-queue-shared-sub-r51-20230717
fix(shared_sub): restore support for `$queue/` shared subscription
2023-07-18 13:52:44 -03:00
Ivan Dyachkov ca51732d8e
Merge pull request #11292 from emqx/update-rocksdb 2023-07-18 16:14:50 +02:00
Thales Macedo Garitezi cb1e0cdc89 test(http): attempt to stabilize flaky tests 2023-07-18 10:51:04 -03:00
Andrew Mayorov 3ba6d34511
chore(eviction): phase out session persistence concerns 2023-07-18 15:47:12 +02:00
Andrew Mayorov 0a00c39282
chore(ps): phase out existing session persistence mechanism
* It's not really working currently in a cluster setup.
* It's not really working well.
* It would be much easier to integrate `emqx_durable_storage` w/o
  the need to support existing solution.
2023-07-18 15:46:04 +02:00
Thales Macedo Garitezi 91cd88c136 fix(api_mgmt): use `emqx_topic:join` for correct construction of topics 2023-07-18 10:19:55 -03:00
Thales Macedo Garitezi 4a889c0994 chore: bump ekka -> 0.15.6
https://github.com/emqx/erlang-rocksdb/releases/tag/1.8.0-emqx-1
2023-07-18 09:00:29 -03:00
firest 4f4a813f04 fix(tdengine): update the default template and the changelog 2023-07-18 10:31:18 +00:00
Thales Macedo Garitezi 1fdcfba629 fix(shared_sub): restore support for `$queue/` shared subscription
Partially addresses: https://emqx.atlassian.net/browse/EMQX-4589

There's still a problem with the handling of shared groups that is
_not particular to the `$queue/` syntax_ and preexistent that the same
client cannot subscribe to _different groups_ that have the same
_topic filter_.
2023-07-17 16:41:24 -03:00
Paulo Zulato 484519dcf9 feat(kinesis): implement Amazon Kinesis Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10474
Fixes https://emqx.atlassian.net/browse/EMQX-10475
2023-07-17 11:51:42 -03:00
Thales Macedo Garitezi 05c3e023a9 chore(gcp_pubsub_consumer): unhide GCP PubSub Consumer bridge for e5.2.0
Fixes https://emqx.atlassian.net/browse/EMQX-10506
2023-07-17 11:24:21 -03:00
Thales Macedo Garitezi ab5fd1e5c3
Merge pull request #11260 from thalesmg/fix-rule-maps-nested-put-r51
fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
2023-07-14 15:02:24 -03:00
zhongwencool 0a35b7875f
Merge pull request #11269 from zhongwencool/fix-process-limit-hidden
fix: hide node.process_limit completely
2023-07-14 22:32:41 +08:00
lafirest f6834b33a0
Merge pull request #11266 from lafirest/fix/influxdb-ph
fix(tdengine): fix SQL template errors
2023-07-14 22:29:16 +08:00
zhongwencool f853f3d3bd
Merge pull request #11271 from zhongwencool/fix-precent-check
fix: keep percentage type's range is 0%~100%
2023-07-14 22:07:41 +08:00
Thales Macedo Garitezi f9452241bd test(rules): add a few more tests to assert our rule evaluation behavior 2023-07-14 10:23:20 -03:00
Serge Tupchii 950d5edc41 fix: avoid logging unnecessary errors in async cleanup functions
Cleanup functions that access ETS tables may fail with `badarg` error during EMQX shutdown.
They are called asynchronously by `emqx_pool` workers and accessed ETS tables
may be already destroyed as their owners are shut down.
This fix catches ETS `badarg` errors before they can be caught and logged by `emqx_pool`.

Fixes: EMQX-9992
2023-07-14 14:22:43 +03:00
firest fea73cf17d fix(log): fix the `pubrel` typo in the log 2023-07-14 17:19:50 +08:00
zhongwencool 11ee6abf02 fix: keep precent range is 0%~100% 2023-07-14 16:48:50 +08:00
zhongwencool 131511b10e fix: hide process_limit completely 2023-07-14 11:04:20 +08:00
Thales Macedo Garitezi 01b143c5ad fix(resource): don't destruct error tuple
Otherwise, `emqx_resource:query` won't correctly deem the resource to
be unhealthy when there's an extra message.
2023-07-13 16:12:33 -03:00
Thales Macedo Garitezi 0dff428efb
Merge pull request #11262 from thalesmg/fix-gcp-consumer-hc-20230712-master
fix(gcp_pubsub_consumer): fail health check when there are no workers
2023-07-13 15:09:02 -03:00
Thales Macedo Garitezi 4034bcbb26 fix(rule_runtime): avoid rewriting select clause sources (by @keynslug)
Thanks to @keynslug for the insight and the patch.

With this, we avoid rewriting the parsed select clause sources, and
attempt to reuse the result context when reading values during
evaluation.
2023-07-13 14:46:08 -03:00
Thales Macedo Garitezi 924ed06760
Merge pull request #11256 from thalesmg/bridge-api-ecpool-400-errors-20230712-r51
fix(bridge_api): return 400 for all ecpool start errors
2023-07-13 10:05:07 -03:00
firest 749b8cd2ed test(tdengine): cover multi-table insert && subtable name generation 2023-07-13 07:39:01 +00:00
firest fef5087c41 fix(tdengine): fix SQL template errors
1. supports multi-table insert
2. supports constructing subtable names by mixed prefixes and placeholders
2023-07-13 04:44:48 +00:00
JimMoen c23c3a1f78
fix: hstreamdb connector use unified schema 2023-07-13 10:47:45 +08:00
Thales Macedo Garitezi 06010f7ca9 Merge remote-tracking branch 'origin/release-51' into sync-r51-20230712 2023-07-12 16:47:45 -03:00
Thales Macedo Garitezi be7918aa41 fix(gcp_pubsub_consumer): fail health check when there are no workers
`ecpool` already returns an error even if the worker process is dead,
but we add the empty worker list clause here just for completeness.
2023-07-12 16:31:21 -03:00
Thales Macedo Garitezi bbff1d6ae9 fix(bridge_api): return 400 for all ecpool start errors
Fixes https://emqx.atlassian.net/browse/EMQX-10460
(https://emqx.atlassian.net/browse/EMQX-10460?focusedCommentId=30126)
2023-07-12 15:32:34 -03:00
Thales Macedo Garitezi bffef386c1 fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
Fixes https://emqx.atlassian.net/browse/EMQX-10541
2023-07-12 14:28:18 -03:00
Ivan Dyachkov 8ebe099653
Merge pull request #11253 from id/kjell/refactor/http_bridge/EMQX-9568
refactor: HTTP bridge into the emqx_bridge_http application
2023-07-12 15:58:02 +02:00
Ivan Dyachkov 6180e775f4 test: fix mocking and application start 2023-07-12 14:47:52 +02:00
Ivan Dyachkov 8815e58149 chore: remove docker-ct from emqx_bridge_http since toxiproxy is not used in CT 2023-07-12 14:47:46 +02:00
Ivan Dyachkov 45f94df7b1 fix: add ehttpc as emqx_bridge_http dependency 2023-07-12 14:47:39 +02:00
Thales Macedo Garitezi c94dc2e63f fix: add emqx_dashboard as a dependency of emqx_machine
```
===> Compiling emqx_machine
===> Compiling src/user_default.erl failed
src/user_default.erl:{24,14}: can't find include lib "emqx_dashboard/include/emqx_dashboard.hrl"; Make sure emqx_dashboard is in your app file's 'applications' list

==> emqx_mix
** (Mix) Could not compile dependency :emqx_machine, "/github/home/.mix/elixir/1-14/rebar3 bare compile --paths /__w/emqx/emqx/_build/emqx/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile emqx_machine", update it with "mix deps.update emqx_machine" or clean it with "mix deps.clean emqx_machine"
make: *** [Makefile:276: emqx-elixir] Error 1
```
2023-07-12 14:47:28 +02:00
Kjell Winblad 6ce47614d2 refactor: HTTP bridge into the emqx_bridge_http app
Fixes:
https://emqx.atlassian.net/browse/EMQX-9568
2023-07-12 14:47:02 +02:00
Kjell Winblad f28510b3ad refactor: HTTP connector into emqx_bridge_http app 2023-07-12 14:46:43 +02:00
Thales Macedo Garitezi 541d03a0ba refactor(schema_registry): refactor schema registry app and modules
Fixes https://emqx.atlassian.net/browse/EMQX-10361

- Moves `lib-ee/emqx_ee_schema_registry` to `apps/emqx_schema_registry`.
- Removes the `_ee_` segment from module names.
  - Exceptions are the table names which are kept to avoid backwards incompatibilities.
2023-07-12 09:24:15 -03:00
Serge Tupchii cacfb51280 fix(emqx_machine): add emqx_ee_schema_registry to the reboot apps list
As emqx_ee_schema_registry uses Mria tables (schema_registry_shard),
a node joining a cluster needs to restart this application in order to
restart relevant Mria shard processes.
2023-07-12 09:23:28 -03:00
zhongwencool 5b0695ca19
Merge pull request #11249 from zhongwencool/license-setting-api
feat: add license setting get/put api
2023-07-12 18:34:23 +08:00
zhongwencool f40d8318b7 fix: license validations is not working 2023-07-12 14:40:38 +08:00
SergeTupchiy fce3060634
Merge pull request #11251 from SergeTupchiy/EMQX-10392-cluster-topology-api
Add cluster topology HTTP API endpoint
2023-07-11 17:45:00 +03:00
Serge Tupchii 19de10be7c feat(emqx_management): add /cluster/topology HTTP API endpoint
The endpoint shows Mria RLOG cluster topology info:
connections between core and replicant nodes.

Closes: EMQX-10392
2023-07-11 16:25:36 +03:00
lafirest 903d81b532
Merge pull request #11250 from lafirest/fix/ws_order
fix(ws): fix MQTT packets may be reversed in the WS
2023-07-11 20:09:55 +08:00
firest 211486a946 fix(ws): fix MQTT packets may be reversed in the WS
Fix when a WS packet contains many MQTT packets, the order of MQTT packets will be reversed
2023-07-11 18:40:06 +08:00
JianBo He e51a4ae81e chore: avoid dialyzer warnings 2023-07-11 10:41:10 +08:00
JianBo He 15c91373fe perf(http): improve the speed of clients querying in default parameters
In https://github.com//emqx/emqx/pull/9749 we fixed the problem of
incorrect total calculation. In this PR we added an option to improve
the query speed when the query condition is empty
2023-07-11 10:41:10 +08:00
Andrew Mayorov 21fbf79c29
Merge pull request #11191 from keynslug/ft/EMQX-9591/msg-persistence-minimal
feat(ds): hardwire emqx_durable_storage message persistence
2023-07-10 21:05:04 +02:00
Thales Macedo Garitezi beb5be4de2
Merge pull request #11244 from thalesmg/ci-sorted-reboot-apps-check-master
ci(machine_boot): add ci check for missing reboot apps
2023-07-10 15:56:10 -03:00
Thales Macedo Garitezi a53768c1d4 fix(machine_boot): add `emqx_durable_storage` to reboot apps list 2023-07-10 14:39:44 -03:00
SergeTupchiy 721dbad300
Merge pull request #11242 from SergeTupchiy/add-schema_registry-app-to-reboot-list
fix(emqx_machine): add emqx_ee_schema_registry to the reboot apps list
2023-07-10 20:17:59 +03:00
Andrew Mayorov d91ab7dec9
test(ds): make `emqx_persistent_messages_SUITE` setup cleaner
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-07-10 18:12:16 +02:00
Ivan Dyachkov fb96c1e20d chore: bump to e5.1.1-alpha.1 2023-07-10 17:25:07 +02:00
Serge Tupchii 7f04fff2a8 fix(emqx_machine): add emqx_ee_schema_registry to the reboot apps list
As emqx_ee_schema_registry uses Mria tables (schema_registry_shard),
a node joining a cluster needs to restart this application in order to
restart relevant Mria shard processes.
2023-07-10 18:04:22 +03:00
JianBo He ae59a26659
Merge pull request #11223 from JimMoen/fix-influxdb-float-value-trans
fix: influxdb float serialization error
2023-07-10 23:04:01 +08:00
Stefan Strigler 93a9772743
Merge pull request #11211 from sstrigler/EMQX-9963-refactor-all-http-ap-is-to-respond-404-on-the-deletion-of-the-resource-is-non-exist
consistently return 404 if resource not found
2023-07-10 16:25:31 +02:00
Ilya Averyanov 58a83739b8
Merge pull request #10986 from savonarola/ft/EMQX-9523/config-api
feat(ft-api): provide configuration API
2023-07-10 15:32:30 +03:00
Ilya Averyanov 12e237c36d
Merge pull request #11132 from savonarola/0531-authz-qos-retain
feat(authz): use extensible map format for actions in authz rules
2023-07-10 15:32:02 +03:00
JimMoen e30bc6afa8
test(influxdb): write raw as float for all number value 2023-07-10 20:00:09 +08:00
zhongwencool f9e54ed8b1
Merge pull request #11237 from zhongwencool/prometheus-api-schema
fix: bad prometheus schema for headers
2023-07-10 18:13:42 +08:00
Andrew Mayorov 576eab9717
Merge pull request #11193 from keynslug/fix/emqx-ft-test-flaps
fix(ft-test): use new cth tooling in emqx_ft_storage_fs_SUITE
2023-07-10 11:55:16 +02:00
JianBo He f144bb0571
Merge pull request #11226 from zhongwencool/listener-conf-enabled-to-enable
fix: alias listeners.Type.Name.enabled as listeners.Type.Name.enable
2023-07-10 17:15:24 +08:00
zhongwencool 1b5d55cdbb fix: delete bad link in emqx_prometheus 2023-07-10 16:48:13 +08:00
zhongwencool 0632d629cb fix: bad prometheus schema for headers 2023-07-10 16:48:13 +08:00
JimMoen e9f1d7f2bf
fix: influxdb float serialization error 2023-07-10 13:53:11 +08:00
zhongwencool 802a50601a chore: add comment for init_loader_done 2023-07-10 12:03:36 +08:00
zhongwencool c4ba558ee3 fix: unset config_loader in emqx's env when stop emqx app 2023-07-10 11:06:08 +08:00
zhongwencool d661b10355 fix: emqx CI failed 2023-07-10 10:34:36 +08:00
zhongwencool 86d3984025 fix: mgmt api SUITE failed 2023-07-10 10:34:36 +08:00
zhongwencool c0ee47dc08 chore: add changelog for 11226 pr 2023-07-10 10:34:36 +08:00
zhongwencool d781346efc fix: alias listeners.Type.Name.enabled as listeners.Type.Name.enable 2023-07-10 10:34:36 +08:00
Stefan Strigler 1110b5d8f5 fix(emqx_retainer): return 404 in delete if topic not found 2023-07-08 14:43:43 +02:00
Stefan Strigler 80e4ffff75 fix(emqx_management): return 404 if plugin does not exist 2023-07-08 14:43:43 +02:00
Stefan Strigler 3fd28f9e18 fix(emqx_management): return 404 for unknown listener id 2023-07-08 14:43:43 +02:00
Stefan Strigler d65d690c17 fix(emqx_gateway): return 404 for unknown client id 2023-07-08 14:43:43 +02:00
Stefan Strigler 3bc419ee64 fix(emqx_gateway): return 404 for unknown listener id 2023-07-08 14:10:23 +02:00
Andrew Mayorov 8c883feb54
chore: bump `emqx_durable_storage` to 0.1.1 2023-07-07 21:56:21 +02:00
Andrew Mayorov 030beb3e6a
fix(ds): drop debug printouts in test suite 2023-07-07 21:56:20 +02:00
Andrew Mayorov daf4e86da1
feat(ds): add `ensure_shard/2` shortcut
And use it in `emqx_persistent_session_ds:init()` backend initialization
function.
2023-07-07 21:56:20 +02:00
Andrew Mayorov ac56de9fc5
feat(ds): allow to redefine directory for shard's rocksdb 2023-07-07 21:56:20 +02:00
Ilya Averyanov 19f9fc5089 feat(authz): bump app versions 2023-07-07 19:41:34 +03:00
Ilya Averyanov 77895f2555 feat(authz): fix typos and style for Retain & QoS authz feature
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-07-07 19:40:34 +03:00
Ilya Averyanov 6db02d6b49 feat(authz): enable feature for both CE and EE 2023-07-07 19:40:34 +03:00
Ilya Averyanov 1ce6a225ae feat(authz): add tests for authz extended actions 2023-07-07 19:40:34 +03:00
Ilya Averyanov 7de26a1776 feat(authz): use extensible map format for actions in authz rules
* support authorization on retain, qos fields
* refactored authz tests heavily
2023-07-07 19:40:34 +03:00
Ilya Averyanov b568bda098 feat(ft-api): simplify config import test 2023-07-07 19:39:05 +03:00
Thales Macedo Garitezi 6337f52cf9 fix(plugins): start/stop plugins when loading config from CLI
Fixes https://emqx.atlassian.net/browse/EMQX-10288
2023-07-07 13:37:51 -03:00
Ilya Averyanov 56c81c2c25 feat(ft-api): bump app versions 2023-07-07 19:17:22 +03:00
Ilya Averyanov e0353ab750 fix(ft-api): update FT API tests to use emqx_cth_suite 2023-07-07 19:15:42 +03:00
Ilya Averyanov fde506838a fix(ft-api): implement import config behaviour 2023-07-07 19:15:42 +03:00
Ilya Averyanov fe691e8330 fix(ft-api): fix swagger schema dump for ft schema 2023-07-07 19:15:42 +03:00
Ilya Averyanov a2eb658cd9 feat(ft-api): do cleanup certs explicitly 2023-07-07 19:15:42 +03:00
Andrew Mayorov a2b03716be feat(ft-api): provide configuration API
To configure `emqx_ft` during the runtime.
2023-07-07 19:15:42 +03:00
Andrew Mayorov 34793c5ed0 feat(ft-conf): preprocess TLS configuration on updates
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
2023-07-07 19:15:42 +03:00
Andrew Mayorov 27ab9c62d8 fix(ft): unload conf hooks in `prep_stop`
In order to avoid situations when the root supervisor is stopped
already.
2023-07-07 19:15:42 +03:00
Ilya Averyanov aba52c6a25 chore(rebalance test): fix API tests 2023-07-07 18:04:32 +03:00
Ilya Averyanov 9bb159cf1e fix(rebalance): fix changelog and version 2023-07-07 18:04:11 +03:00
zhongwencool 5b1c265485
Merge pull request #11224 from zhongwencool/fix-psk-conf-load
fix: psk_authentication is updated failed
2023-07-07 21:27:49 +08:00
Ilya Averyanov a6019a2846
Merge pull request #11198 from savonarola/0705-fix-rebalance-swagger
fix(rebalance): fix rebalance swagger examples
2023-07-07 15:55:16 +03:00
lafirest 70bd7a59e4
Merge pull request #11225 from lafirest/fix/pgsql_function_clause
fix(bridge): ensure the username of pgsql must exists
2023-07-07 20:39:07 +08:00
zhongwencool 8b80a2f754
Merge pull request #11192 from zhongwencool/bump-hocon-to-0.39.13
chore: update hocon to 0.39.13
2023-07-07 20:02:46 +08:00
zhongwencool 0ef00d5919 fix: psk_authentication is updated failed 2023-07-07 20:02:06 +08:00
Andrew Mayorov 97b6c43062
fix(ft-test): use new cth tooling in `emqx_ft_storage_fs_SUITE`
In attempt to battle test flakiness.
2023-07-07 14:00:33 +02:00
Andrew Mayorov 7e76914599
feat(ds): hardwire emqx_durable_storage message persistence
Only message persistence is currently implemented, irrespectively of
whether there are persistent sessions around or not.
2023-07-07 13:46:12 +02:00
firest d01eee7fe4 chore: update changes && app version 2023-07-07 18:22:54 +08:00
firest 8cee75139d fix(bridge): ensure the username of pgsql must exists 2023-07-07 18:06:28 +08:00
Stefan Strigler 055a1e8d75
Merge pull request #11210 from sstrigler/EMQX-9569-provide-the-ce-ee-version-info-and-the-current-version-number-in-the-info-field
feat(emqx_dashboard): include edition and version in swagger.json info
2023-07-07 11:55:44 +02:00
zhongwencool b5cc8fb3c3 fix: start_after_created's default value 2023-07-07 16:39:26 +08:00
zhongwencool 352d818e3b chore: update hocon to 0.39.13 2023-07-07 16:39:26 +08:00
JimMoen 88fd8d0582
Merge pull request #11219 from JimMoen/fix-hstreamdb-deps-app
chore: hstreamdb_erl as deps for hstreamdb bridge
2023-07-07 16:39:12 +08:00
JianBo He 2f33fd9bac
Merge pull request #11208 from HJianBo/update-lwm2m-stats
fix(lwm2m): update the stats correctly and timely
2023-07-07 15:32:50 +08:00
JianBo He daa6cf14fe
Merge pull request #11206 from HJianBo/coap-bugs
Fix(coap): Make the username and password params to optional
2023-07-07 15:32:00 +08:00
zhongwencool ab07fd0547
Merge pull request #11214 from zhongwencool/fix-cluster-rpc-call-tnx-id
fix: bad tnx-id when rejoin cluster
2023-07-07 15:08:06 +08:00
JimMoen 63c6427037
chore: hstreamdb_erl as deps for hstreamdb bridge 2023-07-07 14:50:44 +08:00
JimMoen 30c931ae62
fix: pulsar flaky cluster tests 2023-07-07 12:25:37 +08:00
JimMoen 4ee44972b2
feat: refactor hstreamdb connector to to avoid resources leaking 2023-07-07 12:25:37 +08:00
JimMoen b089fba100
refactor: rm ee_bridge and ee_connector application 2023-07-07 12:25:37 +08:00
JimMoen 583ccfaafd
test(hstreamdb): create stream and wirte data 2023-07-07 12:25:36 +08:00
JimMoen 1587f038a5
feat: hstreamdb bridge with batch query 2023-07-07 12:25:36 +08:00
JimMoen b9bfdfd583
chore: compatibility with hstreamdb v0.15.0 2023-07-07 12:25:36 +08:00
JimMoen d31a305bcb
refactor(hstreamdb): HStreamDB bridge to its own application 2023-07-07 12:25:36 +08:00
zhongwencool f7513b900a fix: set load config done after update tnx_id 2023-07-07 12:25:34 +08:00
zhongwencool 9f57ba510e chore: add 11214 changelog 2023-07-07 12:25:34 +08:00
zhongwencool 221f6eba06 fix: bad tnx-id when rejoin cluster 2023-07-07 12:25:34 +08:00
JianBo He d3d2c8a532 chore(lwm2m): bump version 2023-07-07 12:24:31 +08:00
JianBo He 31a240ba63 fix(lwm2m): update the stats correctly and timely 2023-07-07 12:24:31 +08:00
JianBo He 791b8ef671 chore: bump versions 2023-07-07 12:23:57 +08:00
JianBo He 6d2222318d chore(coap): update the subscriptions_cnt stats in time 2023-07-07 12:23:57 +08:00
JianBo He 800b154582 test(coap): cover the params optional logic to create connection 2023-07-07 12:23:57 +08:00
JianBo He 5c901a52bd fix(coap): make username/password optinal in connection 2023-07-07 12:23:57 +08:00
JianBo He ab809c8b85
Merge pull request #11207 from lafirest/feat/keep_obfuscated
feat: ensure data backends don't leak sensitive data
2023-07-07 12:22:11 +08:00
JianBo He b11fa5f3bf
Merge pull request #11195 from HJianBo/stomp-topic-id-unique
Fix(stomp): avoid to create duplicated subscription by HTTP or Client
2023-07-07 11:18:30 +08:00
firest a1d2b2ca51 chore: bump app versions 2023-07-07 11:17:54 +08:00
firest 18dec53d8b feat: ensure data backends don't leak sensitive data 2023-07-07 11:04:13 +08:00
JianBo He 05048dbf93
Merge pull request #11204 from HJianBo/avoid-case-clause
Allow the `*/*` or multiple `Accept` values in header for the `/configs`  API
2023-07-07 09:46:28 +08:00
Stefan Strigler 8faaa86c07 chore: bump vsn for emqx and emqx_bridge 2023-07-06 14:26:11 +02:00
Stefan Strigler 22b25fbbb3 feat(emqx_dashboard): include edition and version in swagger.json info 2023-07-06 14:14:21 +02:00
Thales Macedo Garitezi 353cc065db
Merge pull request #11175 from thalesmg/fix-mysql-start-error
fix(mysql_bridge): make nxdomain a 400 API error
2023-07-06 08:26:12 -03:00
Stefan Strigler 9bcabe0a9e
Merge pull request #11184 from sstrigler/EMQX-10270-e-5-1-0-missing-value-range-validation-for-the-max-packet-size-of-mqtt-settings
fix: limit mqtt max_packet_size to 256MB
2023-07-06 10:20:24 +02:00
JianBo He 71d1f80530 test(api): cover the accept logic for `/configs` API 2023-07-06 11:30:38 +08:00
JianBo He 966b2affc2 chore: allow handle */* or multiple values in Accept headers 2023-07-06 10:53:07 +08:00
SergeTupchiy 1a79e34510
Merge pull request #11202 from SergeTupchiy/fix-rule-type-acl-conf
chore(emqx_authz): fix `rule()` type example
2023-07-05 21:02:36 +03:00
Serge Tupchii 4dfd3421e7 chore(emqx_authz): fix `rule()` type example
Use `action()` type which is a correct and defined type.
2023-07-05 20:24:26 +03:00
Ilya Averyanov 9af0030538 chore: bump to v5.1.1 2023-07-05 18:43:45 +03:00
Ilya Averyanov f174cb656c fix(rebalance): fix swagger examples in api-doc 2023-07-05 15:38:41 +03:00
Ilya Averyanov dca6fe6200 fix(rebalance): fix global status evaluation on replicant nodes 2023-07-05 12:48:44 +03:00
JianBo He aaf015c06f test(stomp): cover subscription in use 2023-07-05 15:22:44 +08:00
lafirest bf5167a8bf
Merge pull request #11172 from lafirest/fix/dup_payload
fix(ruleengine): fix duplicate `payload` problems
2023-07-05 14:20:26 +08:00
JianBo He 0483c6afef fix(stomp): not allow to create same subscriptions 2023-07-05 14:13:25 +08:00
Stefan Strigler 263890be47 fix: limit mqtt max_packet_size to 256MB 2023-07-04 16:03:11 +02:00
Andrew Mayorov 8cc7022760
Merge pull request #11120 from ft/EMQX-10329/tooling
test: add tooling to make testruns more isolated and predictable
2023-07-04 14:36:39 +02:00
Ilya Averyanov 0bd0a16693
Merge pull request #11039 from savonarola/0613-fix-redis-db-schema
fix(redis): validate database number
2023-07-04 15:32:02 +03:00
zhongwencool f2c5f4932c
Merge pull request #11148 from lafirest/fix/mcall_error_report
fix(emqx_conf): don't sync the MFA to a leaved node
2023-07-04 19:56:52 +08:00
Andrew Mayorov 9ed75085b7
chore: drop dead commented out code 2023-07-04 13:20:04 +02:00
Andrew Mayorov c7ba300efc
fix(cthsuite): use correct type for "smth serializable as HOCON" 2023-07-04 13:20:04 +02:00
Andrew Mayorov d568d43fe4
test(bridge): switch `emqx_bridge_api_SUITE` to use new tooling 2023-07-04 13:20:03 +02:00
Andrew Mayorov fc709a14cc
test: switch `emqx_authz_file_SUITE` to use new tooling 2023-07-04 13:20:03 +02:00
Andrew Mayorov 2c50e61cee
fix(cth): reduce amount of logging + streamline logging
Also make sure that code running on `emqx_cth_cluster` nodes don't
crash when using `ct:pal` and friends.
2023-07-04 13:20:03 +02:00
Andrew Mayorov 9bda4192e5
fix(cthsuite): disable default `acl.conf` authz source
So that users of `emqx_cth_suite` won't have to deal with providing
correct filepath each time they start `emqx_conf` and/or `emqx_authz`
up.
2023-07-04 13:20:03 +02:00
Andrew Mayorov 24a16ea247
fix(cthsuite): bump to hocon 0.39.11 with atom prettyprint fix 2023-07-04 13:20:03 +02:00
Andrew Mayorov 6a7254f55e
fix(cthcluster): match correct driver in `stop_node/2`
Co-authored-by: Ilya Averyanov <av@rubybox.dev>
2023-07-04 13:20:02 +02:00
Andrew Mayorov fe0b8cfbaf
test(conf): respect `init_config_load_done` in emqx_conf itself
This should make task of configuring various applications (emqx_conf
/ emqx) during testsuites setup much simpler.
2023-07-04 13:20:02 +02:00
Andrew Mayorov f76f3b77d7
test: add tooling to make testruns more isolated and predictable
Also showcase their usage in some of the existing testsuites.
2023-07-04 12:53:39 +02:00
firest afabdc3440 fix(emqx_conf): call clean callback before ekka stop 2023-07-04 18:04:33 +08:00
zhongwencool d509c47344 feat: add conf load api 2023-07-04 14:17:11 +08:00
Kjell Winblad 33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
Ilya Averyanov 9701c3f7a1 fix(redis): validate database number 2023-07-03 22:44:54 +03:00
Andrew Mayorov a127fd6571
Merge pull request #11183 from keynslug/perf/EMQX-9591/routing
perf(broker): avoid usorting aggregated routes in a loop
2023-07-03 19:16:25 +02:00
Thales Macedo Garitezi 2cc0ba1217
Merge pull request #11174 from thalesmg/make-rule-server-a-binary
fix(mqtt_bridge): ensure `server` key is a binary
2023-07-03 10:38:39 -03:00
Thales Macedo Garitezi 59d71a1147
Merge pull request #11173 from thalesmg/fix-flaky-gcp-consu-test
test(gcp_consumer): attempts to reduce flakiness
2023-07-03 10:04:31 -03:00
Andrew Mayorov 0f25df3aa8
perf(router): employ `maps:foreach/2` instead 2023-07-03 15:01:47 +02:00
Andrew Mayorov 57da71eb1d
perf(broker): avoid usorting aggregated routes in a loop 2023-07-03 15:01:33 +02:00
firest 902603780b test(ruleengine): add test for the fix of duplicate payload 2023-07-03 18:22:35 +08:00
firest bd4f207365 fix(ruleengine): fix duplicate `payload` problems 2023-07-03 14:21:56 +08:00
Thales Macedo Garitezi c216dfd96b fix(mysql_bridge): make nxdomain a 400 API error
Fixes https://emqx.atlassian.net/browse/EMQX-10460
2023-06-30 16:10:09 -03:00
Stefan Strigler 07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
Thales Macedo Garitezi 425eba8b13 fix(mqtt_bridge): ensure `server` key is a binary
Fixes https://emqx.atlassian.net/browse/EMQX-10461

So that it can be JSON encoded correctly.

```
2023-06-30T02:00:41.160110+00:00 [debug] msg: publish_to, mfa: emqx_trace:publish/1, line: 73, topic: b/$, payload: {"topic":"t/1","server":[49,48,46,52,50,46,51,46,49,56,48,58,49,56,56,51],"retain":false,"qos":1,"pub_props":{},"payload":"{\"msg\": \"hello\"}","message_received_at":1688090441159,"id":"0005FF4F2F181488103417000C2E0000","dup":false}, tag: PUBLISH
```
2023-06-30 13:46:24 -03:00
Thales Macedo Garitezi 6fe6aa7997
Merge pull request #11162 from thalesmg/treat-404-as-failure-master
fix(webhook): treat 404 and other error replies as errors in async requests
2023-06-30 13:13:18 -03:00
Thales Macedo Garitezi 7cf1dd7e86
Merge pull request #11163 from thalesmg/fix-rm-mongo-topology-pool-size
fix(mongodb): hide `topology.pool_size` and fix it at 1
2023-06-30 13:13:02 -03:00
Thales Macedo Garitezi 0201481c70
Merge pull request #11164 from thalesmg/fix-placeholder-sql-nested
fix(placeholder): add back support for nested keys inside binary encoded json
2023-06-30 13:12:51 -03:00
Thales Macedo Garitezi 1a058f6890 test(gcp_consumer): attempts to reduce flakiness 2023-06-30 12:49:23 -03:00
Kjell Winblad 714363bd01 fix: decouple emqx_rule_engine application from emqx_ee_schema_registry
This commit decouples the emqx_rule_engine application from the
emqx_ee_schema_registry application by making it possible to register a
callback module that defines extra rule engine SQL functions instead of
calling a module in emqx_ee_schema_registry directly from the
emqx_rule_engine application.
2023-06-30 16:56:16 +02:00
Ilya Averyanov 724efcb557
Merge pull request #11168 from savonarola/0630-fix-rebalance-schema
Specify correct response schema for global rebalance status
2023-06-30 17:00:35 +03:00
Thales Macedo Garitezi 3c8f591cc4 fix(mongodb): hide `topology.pool_size` and fix it at 1
Fixes https://emqx.atlassian.net/browse/EMQX-10408

From an old conversation with @kjellwinblad:

> There are 3 pool_sizes
> - The buffer workers pool size, just exposed here: https://github.com/emqx/emqx/pull/9742
> - The topology.pool_size, which controls the pool size for the poolboy_pool in Kjell's
  diagram (on mongodb's side).
> - The pool_size from emqx_connector_mongo:mongo_fields that controls the ecpool pool
  size (on EMQX's side).
> So we actually want to set topology.pool_size = 1 and hide it from users.
2023-06-30 10:17:33 -03:00
Thales Macedo Garitezi 631f4ceac9 fix(placeholder): add back support for nested keys inside binary encoded json
Fixes https://emqx.atlassian.net/browse/EMQX-10459
2023-06-30 10:05:49 -03:00
Thales Macedo Garitezi ca435975de fix(webhook): treat http status code 429 as recoverable 2023-06-30 09:46:03 -03:00
lafirest 1ea92d4009
Merge pull request #11166 from lafirest/feat/porti_uuid_el_v4
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 18:36:45 +08:00
zhongwencool c8d1bd3bd5
Merge pull request #11138 from zhongwencool/cluster-conf-k8s-default
fix: update cluster.conf's k8s default api_server value
2023-06-30 17:01:52 +08:00
Kjell Winblad 039e27a153 feat: add Sparkplug encode and decode functions to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
Ilya Averyanov b5c0951584 fix(rebalance): specify correct response schema for global rebalance status 2023-06-30 10:53:08 +03:00
firest 872698d19e feat(ruleengine): port random && uuid_v4 functions from v4 2023-06-30 14:38:27 +08:00
zhongwencool bb1ae808b4
Merge pull request #11165 from zhongwencool/hide-global-limiter-from-swagger-api
chore: hide /configs/limiter from swagger api doc
2023-06-30 11:54:01 +08:00
zhongwencool 3eefa1b2a3 chore: hide /configs/limiter from swagger api doc 2023-06-30 10:59:18 +08:00
zhongwencool 8b679cf358
Merge pull request #11137 from zhongwencool/dashboard-https-ssl-options
feat: refactor dashboard https ssl_options
2023-06-30 09:57:07 +08:00
zhongwencool bd83365d45 fix: update cluster.conf's k8s default api_server value 2023-06-30 09:55:36 +08:00
Thales Macedo Garitezi 59b109eb5c fix(webhook): treat 404 and other error replies as errors in async requests
Fixes https://emqx.atlassian.net/browse/EMQX-10405

The problem here was that, for async requests, ehttpc responses of the form `{ok, 4__, _,
_}` and similar were being treated as successes.
2023-06-29 15:45:23 -03:00
Thales Macedo Garitezi 4506eff3ff
Merge pull request #11140 from thalesmg/gcp-consumer-tests
test(gcp_pubsub_consumer): add more tests and improve bridge
2023-06-29 13:02:47 -03:00
Stefan Strigler 14da61194e fix: handle resource errors 2023-06-29 16:09:45 +02:00
Stefan Strigler 321fd53132 fix: use ReplyTo in QUERY for async 2023-06-29 16:09:45 +02:00
Stefan Strigler d71d70c889 fix: allow to pass in Opts rather than just a ReplyTo 2023-06-29 16:09:45 +02:00
Stefan Strigler 40dd34a704 fix: use reply_to instead of async_reply_fun 2023-06-29 16:09:45 +02:00
Stefan Strigler 1363108678 fix: fix simple_sync_query 2023-06-29 16:09:45 +02:00
Stefan Strigler 422c79583a test(emqx_rule_engine): fix simple sync 2023-06-29 16:09:45 +02:00
Stefan Strigler 837acd4234 test(emqx_rule_engine): test rule metrics 2023-06-29 16:09:45 +02:00
Stefan Strigler 2274a192cc fix(emqx_resource): call async reply fun in simple_aysnc_query 2023-06-29 16:09:45 +02:00
Stefan Strigler ae636a52d7 fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun 2023-06-29 16:09:45 +02:00
lafirest 9708a02680
Merge pull request #11134 from lafirest/fix/auth_header_sensitive
fix: mask the value of the authorization header whatever its case
2023-06-29 21:43:16 +08:00
ieQu1 f2074e9c9b
Merge pull request #11143 from ieQu1/master
docs(ds): Update IMPLEMENTATION.md
2023-06-29 11:14:55 +02:00
lafirest be1a4c89a5
Merge pull request #11158 from lafirest/fix/wait_table_in_retainer
fix(emqx_retainer): wait for Mria tables
2023-06-29 16:01:54 +08:00
firest 31e914f6a2 fix(emqx_utils): improve the checking for the `authorization` key 2023-06-29 15:57:48 +08:00
firest 6fedd0328f fix(retainer): Fix missing changes in configuration 2023-06-29 14:06:17 +08:00
firest 1202bef6db fix(emqx_retainer): wait for Mria tables 2023-06-29 03:16:05 +00:00
zhongwencool 9dae408b5a
Merge pull request #11142 from zhongwencool/list-listener-group-with-wrong-bind
List listener group with wrong bind
2023-06-29 11:04:35 +08:00
zhongwencool 73487ecdd5 chore: add desc for ssl_options 2023-06-29 10:49:39 +08:00
zhongwencool 82ea9e5a04 test: dashbaord api ct failed 2023-06-29 10:49:39 +08:00
zhongwencool e26c30ff0c feat: refactor dashboard https ssl_options 2023-06-29 10:49:39 +08:00
firest 71d34504bf fix(cluster_rpc): clean self-data by mria stop callback 2023-06-29 01:45:47 +00:00
firest 8e68af14ef chore: bump ekka version 2023-06-29 01:45:47 +00:00
firest 51ec8cb8fa fix(emqx_conf): don't sync the MFA to a leaved node 2023-06-29 01:45:47 +00:00
Thales Macedo Garitezi 30e0b4be54 test(gcp_pubsub_consumer): add more tests and improve bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10309
2023-06-28 14:08:40 -03:00
Kjell Winblad f9ea924cd0
Merge pull request #11151 from kjellwinblad/kjell/refactor/mysql_bridge/EMQX-9533
refactor: MySQL bridge and connector to separate applications
2023-06-28 15:15:55 +02:00
ieQu1 940ad12108 docs(ds): Apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-28 10:04:07 +02:00
Kjell Winblad 6d6287ef26 test: start emqx_mongodb when running the MongoDB tests 2023-06-28 08:03:41 +02:00
SergeTupchiy 9cf874e37e
Merge pull request #11150 from SergeTupchiy/wait-for-emqx_psk_tab
fix(emqx_psk): wait for Mria table on app start
2023-06-28 08:57:32 +03:00
zhongwencool 30b67add05 chore: add more example for clients api 2023-06-28 10:55:55 +08:00
zhongwencool 93cbb5ebd8 fix: show the local listener's settings first 2023-06-28 10:55:55 +08:00
zhongwencool d556adf86c
Merge pull request #11146 from zhongwencool/hide-node-process-limit
feat: hide node.process_limit
2023-06-28 10:45:37 +08:00
Thales Macedo Garitezi c4fc0e767e feat: allow specifying more helpful messages for unhealthy targets 2023-06-27 17:13:43 -03:00
Serge Tupchii 8e877caf47 fix(emqx_psk): wait for Mria table on app start
This ensures emqx_psk table is synced to a replicant even if it doesn't have init psk file.
2023-06-27 20:22:40 +03:00
Andrew Mayorov 30d78aaac2
Merge pull request #11152 from keynslug/test/ft-prop-aggressive
test(ft): make proptests even less aggressive
2023-06-27 17:44:54 +02:00
Andrew Mayorov 7d171de02b
test(ft): make proptests even less aggressive 2023-06-27 17:17:56 +02:00
Kjell Winblad 3d1712c00c refactor: MySQL bridge to its own application
This commit refactor the MySQL bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9533
2023-06-27 17:13:26 +02:00
Andrew Mayorov ff327609db
test(ft): generate segments manually in proptests
In order to avoid spamming proptest logs with HUGE type instances,
and make proptests themselves simpler. Proper now should refuse
shrinking altogether, instances that cause prop function to timeout
do not shrink well for some reason, making proptests stuck.
2023-06-27 17:13:23 +02:00
Kjell Winblad 85a130f68b refactor: move MySQL open source connector to its own app 2023-06-27 16:15:18 +02:00
Zaiming (Stone) Shi 4ed422cb9a
Merge pull request #11127 from zmstone/0622-refine-examples
0622 refine examples
2023-06-27 15:30:30 +02:00
Andrew Mayorov 0e55aeada5
Merge pull request #11147 from keynslug/test/ft-timetrap
test(ft): set safer suite-wide timetraps in cluster-related suites
2023-06-27 15:11:14 +02:00
Kjell Winblad d3a0b3e183
Merge pull request #11139 from kjellwinblad/kjell/refactor/redis/EMQX-9534
refactor: redis bridge to its own application
2023-06-27 14:55:55 +02:00
Zaiming (Stone) Shi 90c35c3ff8 chore(prometheus): change default vaue enum the first symbol 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 9b6ed09513 refactor: call hocon_tconf:check_plain/4 directly for example conf check
the attemp to re-use emqx_hocon:check failed because
hocon_tconf:check_plain/3 and check_plain/4 behaves slightly different
for unknown root fields.

To keep the old logic unchanged, for example config checks, we call
check_plain/4 directly
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 92ed7d7639 docs: move some prometheus docs from schema desc to code comments 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 41de679557 ci: run example config check in ci 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi a069b351fd test: add test script to verify config example files 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 03d05825db docs: hide core_nodes config doc
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi e0d05a02ff refactor: refine examples
Also turns off vm_dist_collector metrics collector by default
2023-06-27 14:23:18 +02:00
Andrew Mayorov 31e08a6046
test(ft): set safer suite-wide timetraps in cluster-related suites 2023-06-27 13:03:19 +02:00
Serge Tupchii 116675e86f chore: bump ekka to 0.15.4 2023-06-27 13:24:49 +03:00
zhongwencool b4ae4260c0 feat: hide node.process_limit 2023-06-27 18:09:26 +08:00
ieQu1 e0c11f54aa docs(ds): Update IMPLEMENTATION.md 2023-06-27 10:53:38 +02:00
Kjell Winblad a7db85c1e6 style: fix code style issue 2023-06-27 10:52:24 +02:00
Kjell Winblad f99ccd4d76 fix: fix issues found in review by @paulozulato and @thalesmg 2023-06-27 10:49:19 +02:00
Kjell Winblad e76daa0f3b refactor: redis bridge to its own application
This commit refactor the redis bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9534
2023-06-27 10:24:44 +02:00
Kjell Winblad dd6aa96c99 refactor: move Redis open source connector to its own app 2023-06-27 10:24:40 +02:00
Kjell Winblad 36b7aef04c chore: bump versions 2023-06-26 14:35:29 +02:00
Kjell Winblad a91a478d42 build: fix order in mix.exs list due to CI script 2023-06-26 14:32:59 +02:00
Kjell Winblad a65d768873 style: fix format issues 2023-06-26 14:32:59 +02:00
Kjell Winblad fa2c2a300a refactor: rename emqx_connector_mongo to emqx_mongodb
This renaming is consistent with the naming in the only other connector
only application emqx_oracle.
2023-06-26 14:32:59 +02:00
Kjell Winblad b78a75bb5c refactor: the MongoDB connector into its own application 2023-06-26 14:32:59 +02:00
Kjell Winblad e3199ef813 refactor: MongoDB bridge to use the new app structure 2023-06-26 14:32:53 +02:00
lafirest f2e3446e29
Merge pull request #11135 from lafirest/pref/calendar_timeoffset
perf(rule_engine): improve the time offset parser
2023-06-26 18:54:00 +08:00
lafirest 850d642ec2
Merge pull request #11108 from ieQu1/emqx-ds
Durable storage
2023-06-25 11:36:34 +08:00
firest c8c2793944 perf(rule_engine): improve the time offset parser 2023-06-24 13:05:27 +00:00
firest d485f8fe96 fix: mask the value of the authorization header whatever its case 2023-06-24 09:20:26 +00:00
firest 60a1f5a38d chore: update app version && changes 2023-06-23 17:01:22 +00:00
firest 6bb8bf58ea fix(retainer): rename deliver_rate to delivery_rate 2023-06-23 16:51:19 +00:00
Serge Tupchii 639fe276e1 chore(emqx_authz): fix `ipaddrs` type and style in etc/acl.conf 2023-06-23 14:53:26 +03:00
ieQu1 9a81f22105 fix(ds): Apply review remarks 2023-06-22 17:07:30 +02:00
Paulo Zulato f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
Thales Macedo Garitezi 22356b7c25 chore: hide gcp pubsub consumer until e5.2.0 2023-06-22 10:05:52 -03:00
Thales Macedo Garitezi f01de238b2 chore: bump app vsns 2023-06-22 10:01:12 -03:00
Thales Macedo Garitezi 2f00cf7f84
Merge pull request #11107 from thalesmg/fix-mongo-health-check-reason-master
fix(mongo): return health check failure reason
2023-06-22 09:30:34 -03:00
Thales Macedo Garitezi 7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi 75eb18bc81 chore: bump app vsns 2023-06-22 11:09:09 +02:00
ieQu1 cfcd81e17b fix(ds): Fix build 2023-06-22 11:07:52 +02:00
Zaiming (Stone) Shi c58a98954b Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-22 11:05:51 +02:00
ieQu1 b29c5ad23c feat(emqx_ds): Add API draft for logic layer 2023-06-22 10:20:18 +02:00
ieQu1 a343cdb1d5 refactor(ds): message_storage -> message_storage_bitmask 2023-06-22 10:20:18 +02:00
ieQu1 a4219db163 refactor(ds): emqx_ds_local_store -> emqx_ds_storage_layer 2023-06-22 10:20:18 +02:00
ieQu1 1159f99432 refactor(ds): emqx_replay -> emqx_ds 2023-06-22 10:20:18 +02:00
ieQu1 8d6bcc1414 refactor(ds): emqx_replay -> emqx_durable_storage 2023-06-22 10:20:18 +02:00
ieQu1 c4f721f214 chore(ds): Add callback definitions to local_store 2023-06-22 10:20:18 +02:00
ieQu1 efed7cdc2f chore(ds): behavior -> behaviour 2023-06-22 10:20:18 +02:00
ieQu1 0f6c81e42d chore(ds): Change license to BSL 2023-06-22 10:20:18 +02:00
ieQu1 04adb65c09 refactor(ds): zone -> shard
Also bump erlang-rocksdb version
2023-06-22 10:20:18 +02:00
Andrew Mayorov 8ac0bba958 test(ds): verify preserve / restore works with stored iterators 2023-06-22 10:20:18 +02:00
Andrew Mayorov d2065e0c1b feat(ds): allow to preserve / restore iterators in the db
So that we could guarantee replay consistency / availability under
the assumption that nodes may be restarted or even lost occasionally.
2023-06-22 10:20:18 +02:00
Andrew Mayorov cbded9af39 feat(ds): Make `create_generation` safer against bad input 2023-06-22 10:20:18 +02:00
Andrew Mayorov 83d1606d00 feat(ds): Make iteration fully generation-aware 2023-06-22 10:20:18 +02:00
Andrew Mayorov bf9d57f6a9 chore(ds): Simplify iteration-related typespecs 2023-06-22 10:20:18 +02:00
Andrew Mayorov cf6a5e1643 feat(ds): Allow to create new storage generations 2023-06-22 10:20:18 +02:00
Andrew Mayorov f80e906553 fix(ds): Correct typespec 2023-06-22 10:20:18 +02:00
Andrew Mayorov f5a7b49f57 feat(ds): Wire iteration options up to the app config 2023-06-22 10:20:18 +02:00
Andrew Mayorov 2bf8a07b05 test(ds): Split unit tests off into a full-fledged suite 2023-06-22 10:20:18 +02:00
Andrew Mayorov cf70184ebd test(ds): Provide more general `keymapper_info/1` 2023-06-22 10:20:18 +02:00
Andrew Mayorov 2f385022cd chore(ds): Rename testsuite to reflect test subject better 2023-06-22 10:20:18 +02:00
Andrew Mayorov d875fa49d3 feat(ds): Enable periodic iterator refresh
This might be helpful during replays taking multiple tens of seconds so
that underlying iterators won't hold onto in-memory / on-disk data
structures for too long, preventing rocksdb from recycling them.
2023-06-22 10:20:18 +02:00
Andrew Mayorov 7f408da251 refactor(ds): Use inline functions instead of macros 2023-06-22 10:20:07 +02:00
Andrew Mayorov a11e75d189 fix(ds): clear bitmask of topic filter tail containing wildcards 2023-06-22 10:16:55 +02:00
Andrew Mayorov ac0935ef91 test(ds): Proptest that iteration is exhaustive
Compare iteration results against what an extremely simplified model
produces.
2023-06-22 10:16:45 +02:00
Andrew Mayorov 60e3070328 test(ds): Scale up number of messages per topic in proptests 2023-06-22 10:16:41 +02:00
Andrew Mayorov 82a7292851 test(ds): Use `_build/test/proper` as a scratch dir for testruns 2023-06-22 10:16:32 +02:00
Andrew Mayorov 440fecd268 test(ds): Add a proptest on iterator preserve / restore
Which verifies that preservation and restoration of iterators does not
affect the outcome of an iteration (under the precondition that the
state of database is constant during an iteration).
2023-06-22 10:16:23 +02:00
Andrew Mayorov 1f5425975f feat(ds): add an ability to preserve and restore iterators
This will allow to persist iteration state and to periodically recreate
iterators during long replays.
2023-06-22 10:16:20 +02:00
Andrew Mayorov 94e8dbfe7f test(ds): Move proptests into a separate module
Following conventions. Also add few proptests on keyspace filters.
2023-06-22 10:16:16 +02:00
Andrew Mayorov 4c643aa7d5 refactor(ds): Introduce keyspace filter concept
So we could conveniently test it separately.
2023-06-22 10:16:10 +02:00
Andrew Mayorov c5826cee56 refactor(ds): rename `compute_hash_*` → `compute_topic_*` 2023-06-22 10:16:05 +02:00
Andrew Mayorov 85d3783ff1 chore(ds): Drop few unused macrodefinitions 2023-06-22 10:15:56 +02:00
Andrew Mayorov a0bcdb5104 chore(ds): Attempt to make `compute_next_seek`'s logic clearer 2023-06-22 10:15:49 +02:00
ieQu1 f1e39d977a fix(ds): Fix dialyzer warnings 2023-06-22 10:15:47 +02:00
ieQu1 adcbf40d27 refactor(ds): tau -> epoch 2023-06-22 10:15:44 +02:00
ieQu1 120d4e66ae refactor(ds): Factor out configuration to a separate module 2023-06-22 10:15:42 +02:00
ieQu1 0bbc5ecb32 refactor(ds): Introduce bitwise_concat function 2023-06-22 10:15:40 +02:00
ieQu1 b5bb77dd58 feat(ds): Add metadata storage and supervision tree 2023-06-22 10:15:37 +02:00
Andrew Mayorov 5e30a5d3dd feat(ds): Implement keyspace partitioning across time 2023-06-22 10:15:32 +02:00
Andrew Mayorov 8c9c9cc669 feat(ds): Allow to specify message store options
* Keymapper
* Column family name + DB options
* DB write / read options
2023-06-22 10:15:25 +02:00
Andrew Mayorov 5b6c4d766b test(ds): Add basic property tests 2023-06-22 10:15:20 +02:00
Andrew Mayorov a1e29e4e24 refactor(ds): Improve function naming 2023-06-22 10:15:03 +02:00
Andrew Mayorov cd12338c3f feat(ds): Smoke tests for wildcard iterator scans 2023-06-22 10:14:46 +02:00
Andrew Mayorov 9c1cd4911d feat(ds): Implement iterator next 2023-06-22 10:14:43 +02:00
ieQu1 27b925405b test(ds): Add testcase 2023-06-22 10:14:41 +02:00
Andrew Mayorov 691a27883d feat(ds): Initial commit 2023-06-22 10:14:37 +02:00
Paulo Zulato f47cc2a458 feat(kafka): add support for kafka headers and ext headers
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
Paulo Zulato 62d3766726
Merge pull request #10645 from paulozulato/data-bridge-target-unavailable
Data bridge target unavailable
2023-06-21 18:19:23 -03:00
Andrew Mayorov 62b832be45
Merge pull request #11118 from fix/EMQX-9964/bump-hocon
chore: bump hocon to 0.39.10
2023-06-21 23:13:35 +02:00
Andrew Mayorov 86d787eced
chore: bump hocon to 0.39.10
Which comes with a fix for slightly more user-friendly validation error
messages.
2023-06-21 21:25:43 +02:00
Thales Macedo Garitezi 18f0510353 fix(mongo): return health check failure reason
Fixes https://emqx.atlassian.net/browse/EMQX-10335
2023-06-21 15:09:37 -03:00
Paulo Zulato 8430ec673c feat(kafka): check whether target topic exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-21 15:00:35 -03:00
Thales Macedo Garitezi decfd6df2b feat(buffer_worker): log expired message count
Fixes https://emqx.atlassian.net/browse/EMQX-10165

```
iex(emqx@127.0.0.1)38> 2023-06-21T11:09:35.569404-03:00 [info] msg: buffer_worker_dropped_expired_messages, mfa: emqx_resource_buffer_worker:log_expired_messge_count/1, line: 982, expired_count: 900, resource_id: <<"bridge:webhook:webhook">>, worker_index: 3
```
2023-06-21 14:38:51 -03:00
Zaiming (Stone) Shi 7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Zaiming (Stone) Shi 5fa87091c4 Merge remote-tracking branch 'origin/master' into 0621-merge-release-51-to-master 2023-06-21 16:08:41 +02:00
Ilya Averyanov 7382c6e6e1
Merge pull request #11103 from savonarola/0620-update-erlcloud
chore(deps): update erlcloud and remove unnecessary overrides
2023-06-21 15:07:45 +03:00
zhongwencool f50d7334d6
Merge pull request #11077 from zhongwencool/fix-crash-ip-port-type 2023-06-21 17:36:59 +08:00
Zaiming (Stone) Shi 5f47649d4d chore: downgrade log severity level of 'not_ready' nodes to info 2023-06-21 09:45:45 +02:00
Zaiming (Stone) Shi 045ed340dd fix(emqx_conf_app): fix release version detect during cluster conf sync 2023-06-21 09:28:46 +02:00
Thales Macedo Garitezi 0463828e84 refactor(gcp_pubsub): transform connector into opaque client 2023-06-20 15:27:42 -03:00
Thales Macedo Garitezi 1d791d7a8c fix(resource): validate maximum worker pool size
Fixes https://emqx.atlassian.net/browse/EMQX-10297
2023-06-20 14:26:42 -03:00
Thales Macedo Garitezi ffce6fefa8 docs: note that connector state must be immutable to avoid errors 2023-06-20 13:50:53 -03:00
Zaiming (Stone) Shi dddccfdc5c chore: bump to v5.1.0 official cut 2023-06-20 17:09:17 +02:00
Zaiming (Stone) Shi 728656d4f6 chore: bump to e5.1.0 official cut 2023-06-20 17:08:18 +02:00
Thales Macedo Garitezi ba96c725e5 feat: add time-based pull for safety 2023-06-20 11:36:35 -03:00
Thales Macedo Garitezi 57460e83b9 test(fix): rm unused var warnings 2023-06-20 11:15:37 -03:00
Thales Macedo Garitezi 2ac2d4c037 refactor: addressing review comments 2023-06-20 11:15:13 -03:00
Ilya Averyanov 54d96488a0 chore(deps): update erlcloud and remove unnecessary overrides 2023-06-20 16:46:23 +03:00
Zaiming (Stone) Shi b588b88a60 chore: change autoclean interval default value to 24h
There is currently a bug in OTP causing external backends to
fail when merge schema (mnesia_schema:change_storage_type/3).

To avoid forcing users to clean up the DB after the node is
stopped for a short while (5m before this change), we increase
the default auto-clean interval to 24 hours.

This PR also has the autoclean config removed from examples.
2023-06-20 15:43:37 +02:00
lafirest 8f595cbeca
Merge pull request #11098 from lafirest/feat/doc-example
chore: add examples of how to configure
2023-06-20 20:09:44 +08:00
Zaiming (Stone) Shi f95787d229 test: make possible to test core/replica locally 2023-06-20 12:42:00 +02:00
ieQu1 e5803a61f1 fix(management): Don't wait for an empty shard 2023-06-20 10:08:41 +02:00
firest bb2bf059ac chore: modify the build to copy examples into the release 2023-06-20 14:53:14 +08:00
zhongwencool 36ee872361 chore: hide plugins from conf load cli 2023-06-20 10:34:09 +08:00
zhongwencool 856de78698 chore: remove the is_list guard 2023-06-20 08:39:23 +08:00
zhongwencool 093cdab838 chore: to_integer to make sure integer is converted 2023-06-20 08:39:23 +08:00
zhongwencool 07172e42f0 test: integer CI check failed 2023-06-20 08:39:23 +08:00
zhongwencool a4be9b8281 chore: bump hocon to 0.39.9 2023-06-20 08:39:23 +08:00
zhongwencool 45963b6a95 fix: ip_port schema type crash 2023-06-20 08:39:23 +08:00
Zaiming (Stone) Shi ead511a128 chore: bump to 5.1.0-rc.1 2023-06-19 23:19:27 +02:00
ieQu1 86b78981c4
Merge pull request #11092 from ieQu1/mria-0.5.5
chore(mria): Bump version to 0.5.5
2023-06-19 23:13:36 +02:00
Thales Macedo Garitezi 13746c2cdf fix(resource): check status when (re)starting a resource
Fixes https://emqx.atlassian.net/browse/EMQX-10290
2023-06-19 18:01:02 -03:00
Thales Macedo Garitezi 01d758e4c0 fix(bpapi): fix `query_mode` type(spec)
The previous spec was wrong due to a mismatch between type declaration and usage in specs:
it would never be `dynamic`, and it was lacking the `simple_async`, `simple_sync` and
`no_queries` constructors.
2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi bb49482529 refactor(gcp_pubsub): move logging from connector to bridge 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi dabefdb01b fix(gcp_pubsub): redact request when errors occur 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi 7f850f7499 fix(resource): fix `query_mode/0` type and usage 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi a1690b3684 chore(gcp_pubsub): add stricter compile/xref flags 2023-06-19 15:59:00 -03:00
Thales Macedo Garitezi 5375421954 refactor(gcp_pubsub): split connector into producer and reusable parts 2023-06-19 15:59:00 -03:00
Paulo Zulato 5f10936091 feat(oracle): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
Paulo Zulato c9a2ddf98c feat(mysql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 15:22:27 -03:00
ieQu1 f82c56ae16 chore(mria): Bump version to 0.5.5 2023-06-19 19:28:17 +02:00
JianBo He cc939d2bda
Merge pull request #11087 from zhongwencool/bump-v5.1.0.alpha.8
chore: bump to v5.1.0.alpha.8
2023-06-19 23:00:53 +08:00
JianBo He b0a6c18917 chore: bump vsn to 5.1.0-alpha.8 2023-06-19 22:59:05 +08:00
Paulo Zulato 9454af9a8b feat(postgresql): check whether target table exists
Fixes https://emqx.atlassian.net/browse/EMQX-9026
2023-06-19 11:12:10 -03:00
Andrew Mayorov af7b7ebf7c
Merge pull request #11088 from fix/EMQX-10068/nicer-doc-tags
fix(ft-api): make schema tags look nicer in the docs
2023-06-19 16:00:04 +02:00
Thales Macedo Garitezi 7f1e8baa76
Merge pull request #11078 from thalesmg/fix-pmap-alias-master
fix(nolink_apply): avoid sending late replies to caller
2023-06-19 10:32:43 -03:00
Andrew Mayorov ae5846be29
fix(ft-api): make schema tags look nicer in the docs 2023-06-19 13:35:21 +02:00
zhongwencool 1f1d61cef6
Merge pull request #11084 from zhongwencool/hide-plugins-from-cli
chore: hide plugins from conf load cli
2023-06-19 19:11:52 +08:00
zhongwencool 64612cab14 chore: bump to v5.1.0.alpha.8 2023-06-19 18:22:51 +08:00
zhongwencool 6efb07f6eb fix: crash on emqx_ctl listeners 2023-06-19 18:12:33 +08:00
zhongwencool a578d6fa1d chore: release e5.1.0-alpha.8 2023-06-19 17:19:03 +08:00
zhongwencool 8b20a70344 chore: hide plugins from conf load cli 2023-06-19 15:57:40 +08:00
zhongwencool 803cd6c812
Merge pull request #11082 from emqx/release-51
sync release-51 to master
2023-06-19 15:12:37 +08:00
JimMoen 30ac6c4e48
Merge pull request #11074 from JimMoen/fix-share-sub-nl-protocol-error
fix: shared-sub with nl sub-option should cause protocol error
2023-06-19 10:39:08 +08:00
Kjell Winblad 59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216
fix: rule engine different behavior for div and mod
2023-06-17 07:20:37 +02:00
Andrew Mayorov 13d9f5c3e7
Merge pull request #11053 from fix/EMQX-10231/cluster-view
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
2023-06-17 00:54:24 +02:00
Thales Macedo Garitezi 50e7d5d2ec fix(nolink_apply): avoid sending late replies to caller
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.

e.g. when a timeout occurred while resource manager was checking a resource's health:

```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```

Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277))
does), we avoid polluting the caller's mailbox with late replies.
2023-06-16 17:03:39 -03:00
Thales Macedo Garitezi 8801ae9e5a
Merge pull request #11004 from paulozulato/fix-rewrite-wildcard
fix(rewrite): avoid wildcards on destination topic on publish
2023-06-16 09:57:46 -03:00
JimMoen 178711f742
fix: shared-sub with nl sub-option should cause protocol error 2023-06-16 18:19:23 +08:00
zhongwencool 5f7e938b20
Merge pull request #11070 from JimMoen/use-mria-autoclean
fix: use mria.cluster_autoclean
2023-06-16 14:56:52 +08:00
JimMoen c4e7129cb6
fix: use mria.cluster_autoclean and mria.cluster_autoheal 2023-06-16 14:04:24 +08:00
Serge Tupchii b49b80d36c fix(emqx_schema): use non negative integer type for 'depth' SSL option
Closes: EMQX-10276
2023-06-16 10:59:39 +08:00
Paulo Zulato b2f4dbd232
Merge pull request #11067 from paulozulato/chore-bump-influx
chore(influxdb): bump version for influxdb client
2023-06-15 20:16:28 -03:00
Paulo Zulato e127df3912 chore(influxdb): bump version for influxdb client 2023-06-15 17:53:08 -03:00
zhongwencool 9b56f116f4
Merge pull request #11062 from zhongwencool/log-file-rename 2023-06-15 20:51:52 +08:00
zhongwencool 3406a5d916
Merge pull request #11063 from zhongwencool/bump-to-alpha.7
chore: bump to e5.1.0-alpha.7
2023-06-15 19:22:37 +08:00
zhongwencool fc1d78134a test: replace 'to' with 'path' 2023-06-15 19:13:51 +08:00
zhongwencool 9060a2dc05 chore: bump to e5.1.0-alpha.7 2023-06-15 19:05:13 +08:00
zhongwencool e6c2da8d19 feat: rename log.file.to to log.file.path 2023-06-15 18:24:21 +08:00
zhongwencool 84a5d0c3c1 fix: delete a default listener then recreate it, it's disabled 2023-06-15 17:19:11 +08:00
zhongwencool 0adbb0deec feat: try to save config when post_config_update failed not in init mfa status 2023-06-15 17:19:11 +08:00
zhongwencool c160301dbe fix: don't log enoent error 2023-06-15 17:06:09 +08:00
zhongwencool 7dade3a52c fix: tls_certfile_gc notice log don't print abspath 2023-06-15 17:06:09 +08:00
Serge Tupchii 64bbe21209 fix(emqx_schema): use non negative integer type for 'depth' SSL option
Closes: EMQX-10276
2023-06-15 11:22:58 +03:00
JimMoen 903d8d899e
fix: .js files are required by dashboard as static
revert parts of 8c1334c421
2023-06-15 16:10:16 +08:00
firest c28346d67b chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-15 10:35:18 +08:00
zhongwencool e42cc58694
Merge pull request #11050 from emqx/release-51
chore: sync release-51 back to master
2023-06-15 09:45:31 +08:00
Paulo Zulato 8c7c0877fd fix(rewrite): avoid wildcards on destination topic on publish
Fixes https://emqx.atlassian.net/browse/EMQX-9247
2023-06-14 20:02:03 -03:00
Andrew Mayorov 6dc281345c
test(ft-fs): run API tests with replicant nodes 2023-06-15 00:50:18 +03:00
Andrew Mayorov acd6e5635b
fix(ct): adapt cluster setup helper to replicant nodes
Before this change, trying to start cluster consisting of cores +
replicants resulted in error joining replicant node to the cluster.
2023-06-15 00:50:15 +03:00
Andrew Mayorov 52354bf58a
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
When iterating over complete exports cluster-wide.
2023-06-14 22:54:56 +03:00
Paulo Zulato 36d6350880
Merge pull request #11031 from paulozulato/fix-influx-wrong-username
fix(influxdb): check authentication
2023-06-14 14:41:50 -03:00
zhongwencool 41004e86da
Merge pull request #10959 from qzhuyan/dev/william/remove-depre-quic-config
chore: remove deprecated QUIC listener config
2023-06-14 23:01:00 +08:00
zhongwencool bd269e259e
Merge pull request #11047 from zhongwencool/dashboard-listeners-deprecate-enable
feat: deprecated dashboard.listeners.name.enable
2023-06-14 22:58:39 +08:00
Serge Tupchii 7882755114 chore: bump cowboy to 2.9.2
cowboy 2.9.2 uses ranch 1.8.1-emqx that fixes incompatible TLS options
2023-06-14 17:35:17 +03:00
Serge Tupchii 4425942030 fix(emqx_schema): move tls version gap validation to emqx_schema 2023-06-14 17:29:56 +03:00
Ivan Dyachkov 6d88cd7a20 fix(tls): fix incompatible tls options and issue with tls version gap 2023-06-14 17:29:56 +03:00
Paulo Zulato a573b9b38c fix(influxdb): check authentication
Checks authentication on bridge start and get status. Also, handle
authentication error when sending message.

Fixes https://emqx.atlassian.net/browse/EMQX-10213
2023-06-14 11:27:39 -03:00
zhongwencool 9209e0bed5 test: delete dashboard.listeners.xx.enable 2023-06-14 20:51:17 +08:00
zhongwencool 4b82d4d927 feat: deprecated dashboard.listeners.name.enable 2023-06-14 20:51:17 +08:00
zhongwencool 267053cc35
Merge pull request #11045 from zhongwencool/hide-zones-authn-in-listeners
feat: hide zone/authn in listeners and remove listeners's authn api
2023-06-14 20:32:27 +08:00
Stefan Strigler bf62cdc3e3
Merge pull request #11037 from sstrigler/EMQX-10174-e-5-1-0-reconnect-does-not-show-any-errors-for-apache-io-tdb-or-webhook-or-kafka
fix(emqx_connector): report errors in on_start handler
2023-06-14 13:20:47 +02:00
zhongwencool f6df52020c test: conf_cli failed with replace 2023-06-14 18:43:41 +08:00
Andrew Mayorov 1beb588207
fix(ft): give more failproof example in README
Without that `ssl` defaults to being disabled, and interacting
with AWS S3 with such configuration consistently times out.
2023-06-14 13:15:00 +03:00
zhongwencool 9a0856fd6b fix: authn/authz's keys should be binary 2023-06-14 18:00:36 +08:00
zhongwencool bd33dcb049 feat: remove zone from /clients api 2023-06-14 17:11:20 +08:00
Stefan Strigler 0d6d441f4c test(emqx_connector): start/stop test for webhook bridge 2023-06-14 09:56:50 +02:00
zhongwencool 121cd9124a chore: improve show_keys output 2023-06-14 15:53:55 +08:00
zhongwencool 7668753f50 feat: hide zone/authn in listeners and remove /listeners/:id/authentication api 2023-06-14 15:17:12 +08:00
JianBo He 80e2c2c955 fix(http): avoid crash if the old sample data dont contain `live_connections`
In version 5.1.0, a new metric called "live_connections" was added. However, we try to merge
some data from the Mria disk table, which may include old data formats.

Therefore, maps:get/3 is needed to avoid crashes.
2023-06-14 15:14:20 +08:00
JianBo He 8ce0380704 chore: bump vsn to e5.1.0-alpha.6 2023-06-14 12:49:02 +08:00
JianBo He 86d3b702d7
Merge pull request #11042 from zhongwencool/max-connections-string
fix: api_listener crash when setting max_connnections as string
2023-06-14 12:46:40 +08:00
zhongwencool feb08b1181 chore: update 11042 changelog 2023-06-14 11:40:20 +08:00
zhongwencool 091c49277f fix: api_listener crash when setting max_connnections as string 2023-06-14 11:26:41 +08:00
JianBo He 26ad06b9ca chore: update changes 2023-06-14 10:49:17 +08:00
JianBo He c4c32f5032 chore: cover stomp mountpoint tests 2023-06-14 10:49:17 +08:00
JianBo He 4212d90672 fix(stomp): unmount topic prefix in delivering proccess 2023-06-14 10:49:17 +08:00
JianBo He e3d208f678 fix(gw): close the stomp connections once an error frame occured
According to the Stomp v1.2 specification:

> The server MAY send ERROR frames if something goes wrong. In this case,
> it MUST then close the connection just after sending the ERROR frame

Additional, fixes the `is_superuser` is not working for all gateways
2023-06-14 10:49:17 +08:00
zhongwencool d41a23b7d0
Merge pull request #11036 from zhongwencool/conf-load-cli
feat: support --replace|merge with conf load
2023-06-14 10:48:13 +08:00
zhongwencool 7f0f40cb58
Merge pull request #11034 from zhongwencool/mqtt-conf
feat:  move shared_subscription_strategy from broker to mqtt
2023-06-14 10:00:38 +08:00
zhongwencool 9b5c7065f2 feat: conf reload support --replace|merge 2023-06-14 09:12:39 +08:00
zhongwencool 095fb040c0 chore: format conf cli usage text 2023-06-14 09:12:39 +08:00
zhongwencool a8959900d0 feat: support --replace|merge with conf load 2023-06-14 09:12:39 +08:00
zhongwencool f57ff802e4
Merge pull request #11038 from thalesmg/fix-pulsar-get-status-r51
fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
2023-06-14 08:32:25 +08:00
Thales Macedo Garitezi 25b0e31035 fix(kafka_producer): do not return disconnected when checking status (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10279

Related: https://github.com/emqx/emqx/pull/11038

Since wolff client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 13:54:57 -03:00
Stefan Strigler b2a5065641 fix(emqx_connector): report errors in on_start handler 2023-06-13 16:57:08 +02:00
Thales Macedo Garitezi b609792a90 fix(pulsar_producer): do not return `disconnected` when checking status (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10278

Since Pulsar client has its own replayq that lives outside the management of the buffer
workers, we must not return disconnected status for such bridge: otherwise, the resource
manager will eventually kill the producers and data may be lost.
2023-06-13 11:44:45 -03:00
zhongwencool 946de4a303 chore: dialyzer warning 2023-06-13 20:11:49 +08:00
zhongwencool 50041fdddf feat: hide broker rootkey and move broker.shared_subscription_strategy to mqtt.shared_subscription_strategy 2023-06-13 19:35:52 +08:00
zhongwencool beee35bdea
Merge pull request #11033 from HJianBo/fix-exproto-mountpoint
feat(exproto): abandon the mountpoint field in AuthenticateRequest
2023-06-13 19:18:17 +08:00
JianBo He 7afd9d2b06 chore: update changes 2023-06-13 16:38:23 +08:00
firest 536f7ba423 chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-13 16:06:28 +08:00
JianBo He 7684cefa86 feat(exproto): abandon the mountpoint field in AuthenticateRequest
This field was introduced in v4.x, but in fact, in 5.0 we have provided
`gateway.exproto.mountpoint` for configuration, so there is no need to override
it through the Authenticate request
2023-06-13 14:10:14 +08:00
William Yang 8fa743fa22 chore: mark deprecated QUIC listener config for 5.1 2023-06-13 06:00:57 +02:00
zhongwencool 3331af4757 feat: make gateway and slow_subs low level and hide rule_engine 2023-06-13 11:47:38 +08:00
Thales Macedo Garitezi 5ef7102cba fix(listener_mgmt_api): use union member selector fn for better error messages (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10247
2023-06-12 16:48:13 -03:00
Thales Macedo Garitezi 73fe671dba test: rm unused var warnings 2023-06-12 15:19:47 -03:00
zhongwencool 58dc042279 chore: bump to e/v5.1.0-alpha.5 2023-06-13 00:06:54 +08:00
Thales Macedo Garitezi c11011d857
Merge pull request #11024 from thalesmg/pulsar-cosmetic-connecting-check-r51
feat(pulsar): retry health check a bit before returning (r5.1)
2023-06-12 13:00:54 -03:00
Kjell Winblad cf31b65076 fix: rule engine different behavior for div and mod
Previously, the div operation could only be used as an infix operation
while mod could only be used as a function call. After this commit, one
can use both div and mod using function call syntax and infix syntax.

Fixes: https://emqx.atlassian.net/browse/EMQX-10216
2023-06-12 17:20:52 +02:00
SergeTupchiy 5ef19ecf58
Merge pull request #10993 from SergeTupchiy/improve_mnesia_backup_import
fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
2023-06-12 17:26:26 +03:00
Thales Macedo Garitezi 741c1f091e fix(pulsar): update pulsar -> 0.8.3
Fixes https://emqx.atlassian.net/browse/EMQX-10229

See https://github.com/emqx/pulsar-client-erl/pull/59

Fixes a case_clause error that could arise from race conditions.
2023-06-12 10:29:40 -03:00
Thales Macedo Garitezi db5d14d5bf feat(pulsar): retry health check a bit before returning (r5.1)
Fixes https://emqx.atlassian.net/browse/EMQX-10228

This is a cosmetic fix for the Pulsar Producer bridge health check status.

Pulsar connection process is asynchronous, therefore, when a bridge of this type is
created or updated (which is the same as stopping and re-creating it), the immediate
status will be connecting because it’s indeed still connecting.  The bridge will connect
very soon afterwards (assuming there are no true network/config issue), but having to
refresh the UI to see the new status and/or seeing the resource alarm might annoy users.

This workaround adds a few retries to account for that effect to reduce the probability of
seeing the `connecting` state on such happy-paths.
2023-06-12 10:26:07 -03:00
firest 0ed420f81f fix: obfuscated the value of JWT 2023-06-12 18:01:02 +08:00
Ilya Averyanov 93733c1ccc fix(docs): fix typo in ft README 2023-06-12 11:38:17 +02:00
zhongwencool 0fc10ad239
Merge pull request #10984 from zhongwencool/reload-cli
feat: support reload cli
2023-06-12 17:21:29 +08:00
Zaiming (Stone) Shi 12b73ba1c0 Merge remote-tracking branch 'origin/master' into release-51 2023-06-12 10:35:47 +02:00
firest 947d99218b chore: upgrade emqtt to avoid sensitive data leakage in the debug log 2023-06-12 16:32:24 +08:00
Ilya Averyanov bf2c768182
Merge pull request #11010 from zhongwencool/fs-delete-SUITE-data
chore: delete emqx_utils_fs_SUITE_data dir
2023-06-12 10:40:48 +03:00
JianBo He d14d87b443
Merge pull request #10955 from HJianBo/clean-mqttsn-topic-registry
Fix(mqttsn): clean predefined topics once gateway unload
2023-06-12 09:37:47 +08:00
Zaiming (Stone) Shi e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb
fix(mongodb): hide batch_size for mongodb resource
2023-06-11 21:26:12 +02:00
zhongwencool 89ddd54c68
Merge pull request #11013 from HJianBo/fix-max-conns-in-gateway-lists-api
fix(gateway): fix 500 crash for '/gateways' endpoint
2023-06-11 22:29:24 +08:00
JianBo He 57a3b6fdda fix(gateway): fix 500 crash for '/gateways' endpoint
It introduced by https://github.com/emqx/emqx/pull/10961
2023-06-11 18:43:38 +08:00
zhongwencool 1550158ddd fix: update emqx_flapping process's state when emqx:update_config([flapping_detect],Conf) 2023-06-11 18:23:15 +08:00
JianBo He 001e2976ec chore: fix typos 2023-06-11 18:13:25 +08:00
JianBo He 9c4a576fb3 feat: reserved 1-1024 topic id for predefined topics 2023-06-11 18:12:34 +08:00
JianBo He ac6693c8cc refactor(mqttsn): takeover can resume the registrations of session 2023-06-11 18:12:34 +08:00
JianBo He 3c6afee690 refactor(mqttsn): make the topic registration mechanism simpler 2023-06-11 18:12:34 +08:00
JianBo He 9760181ed9 test: cover mqttsn predefined topics 2023-06-11 18:12:34 +08:00
JianBo He 6fc50d11f8 fix(mqttsn): clean pre-defined topics once gateway reloaded 2023-06-11 18:12:34 +08:00
Zaiming (Stone) Shi 654a9dcab5
Merge pull request #11006 from zmstone/0610-fix-quic-default-certs-path
fix(quic): environment variables as cert file prefix for quic listener
2023-06-11 11:52:56 +02:00
zhongwencool c8cf8d1def
Merge pull request #11009 from zhongwencool/persistent_session_store-default-value
chore: correct persistent_session_store's default value type to boolean
2023-06-11 17:26:58 +08:00
Zaiming (Stone) Shi ddef751527 fix(mongodb): hide batch_size for mongodb resource
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
2023-06-11 11:08:58 +02:00
zhongwencool a10107045f test: delete emqx_ft_fs_utils_SUITE_data from source code 2023-06-11 15:50:45 +08:00
zhongwencool aed64aae60 chore: delete emqx_utils_fs_SUITE_data dir 2023-06-11 15:50:45 +08:00
zhongwencool 072f6fac8d chore: update hocon to 0.39.8 2023-06-11 09:11:17 +08:00
zhongwencool 346b363cb5 test: config SUITE failed 2023-06-11 09:11:17 +08:00
zhongwencool 505d677685 chore: move reload_local_etc_config from emqx_conf to emqx_conf_cli 2023-06-11 09:11:17 +08:00
zhongwencool 1ca1ba9e7a fix: get_sources return default authz when sources key not found 2023-06-11 09:11:17 +08:00
zhongwencool 01671d82c7 feat: delete authorization form etc/emqx.conf 2023-06-11 09:11:17 +08:00
zhongwencool 6c6ff4ea1f chore: rename {mod} to module 2023-06-11 09:11:17 +08:00
zhongwencool cf9a207743 feat: check all config schema before loading 2023-06-11 09:11:17 +08:00
zhongwencool b829d8edc2 feat: support --auth-chains merge|replace mod 2023-06-11 09:11:17 +08:00
zhongwencool 21ffd958b4 feat: support authn merge_authenticators 2023-06-11 09:11:17 +08:00
zhongwencool aac5b85d26 feat: authz support merge sources 2023-06-11 09:11:17 +08:00
zhongwencool 994db58cbf fix: load authz's default sources 2023-06-11 09:11:17 +08:00
zhongwencool c819ac27f4 feat: add conf reload cli 2023-06-11 09:11:17 +08:00
zhongwencool 61bbe19eba
Merge pull request #10961 from HJianBo/support-infinity-max-conns-in-gateway
Adds support for unlimited max connections for gateway listeners
2023-06-11 09:09:34 +08:00
zhongwencool 599eb9fef5 chore: correct persistent_session_store's default value type to boolean 2023-06-10 19:36:05 +08:00
Zaiming (Stone) Shi 97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi 9f135d1f2b fix(quic): environment variables as cert file prefix for quic listener 2023-06-10 12:23:20 +02:00
JianBo He aea3ae4024
Merge pull request #11005 from HJianBo/fix-tracing-logs-typos
fix(authn): fix the hardcode method in trace logs
2023-06-10 18:08:28 +08:00
Zaiming (Stone) Shi 3901ed3255
Merge pull request #10996 from sstrigler/EMQX-9435-request-the-non-existent-api-v-4-nodes-endpoint-in-emqx-e-5-0-1-returns-http-code-200
Return 404 for everything we don't know
2023-06-10 11:20:25 +02:00
Zaiming (Stone) Shi f98cdd4983
Merge pull request #10994 from sstrigler/EMQX-10003-e-5-0-4-auth-header-value-of-webhook-data-bridge-can-be-found-in-emqx-log
fix(emqx_utils): redact proxy-authorization headers
2023-06-10 11:18:58 +02:00
JianBo He fdaac9800c fix(authn): fix the hardcode method in trace logs 2023-06-10 13:04:24 +08:00
Serge Tupchii 48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option
Closes: EMQX-8725
2023-06-09 22:10:17 +03:00
Kjell Winblad 43292c06e4
Merge pull request #10997 from kjellwinblad/kjell/fix/clickhouse_lost_messages/EMQX-10217
fix: lost messages when clickhouse closes while sending messages
2023-06-09 21:08:25 +02:00
Serge Tupchii 87b57112df refactor: move tcp keepalive options helper to emqx_utils 2023-06-09 21:43:50 +03:00
Paulo Zulato 7c63f20aa8
Merge pull request #10999 from paulozulato/fix-kafka-time-field-too-large
fix(kafka): fix timeout field for some kafka fields
2023-06-09 15:29:59 -03:00
Paulo Zulato 720a84d48b fix(kafka): fix timeout field for some kafka fields
Change type of fields 'Partition Count Refresh Interval' and 'Offset
Commit Interval' to avoid accepting values larger than allowed.

Fixes https://emqx.atlassian.net/browse/EMQX-10196
Fixes https://emqx.atlassian.net/browse/EMQX-10199
2023-06-09 13:38:04 -03:00
Kjell Winblad 1b3af2ac91 refactor: nicer handling of errors and warning log for recoverable errors 2023-06-09 17:22:16 +02:00
Stefan Strigler 57d72ed23e fix(emqx_utils): redact proxy-authorization headers 2023-06-09 17:13:53 +02:00
Stefan Strigler 8c1334c421 fix(emqx_dashboard): return 404 for everything we don't know 2023-06-09 17:13:18 +02:00
Andrew Mayorov b930f4cc73
Merge pull request #10987 from keynslug/fix/EMQX-9257/sep-placeholder
refactor: tear `emqx_plugin_libs` application apart
2023-06-09 17:02:14 +02:00
Thales Macedo Garitezi 4a9f2247ca test(conf_app): fix bad merge conflict 2023-06-09 11:06:54 -03:00
Kjell Winblad 61488ec407 fix: lost messages when clickhouse closes while sending messages
Fixes:
https://emqx.atlassian.net/browse/EMQX-10217
2023-06-09 16:04:38 +02:00
Thales Macedo Garitezi 40675843a1 fix(iotdb): remove root `request_timeout` option
Fixes https://emqx.atlassian.net/browse/EMQX-10175

Since IoTDB bridge is not used as an authn/authz source and reuses the
webhook connector, we should have only one request timeout field (the
`resource_opts.request_ttl` one) like the webhook bridge does, to
avoid confusion and potentially bad configuration combinations.
2023-06-09 10:39:01 -03:00
Zaiming (Stone) Shi 643da0564b
Merge pull request #10990 from zmstone/0609-fix-schema-for-dashboard-use
0609 fix schema for dashboard use
2023-06-09 15:07:51 +02:00
Thales Macedo Garitezi 20a7ce64cd
Merge pull request #10919 from thalesmg/test-flakiness-20230601-a
test(pulsar_producer): attempt to fix flaky test
2023-06-09 09:30:30 -03:00
Andrew Mayorov c8360f86b7
chore: bump application versions 2023-06-09 14:44:38 +03:00
Andrew Mayorov 830ba54721
refactor(pluglib): remove `emqx_plugin_libs` application 2023-06-09 14:44:37 +03:00
Andrew Mayorov e6fb0203b4
refactor(pluglib): move connectivity checks to `emqx_connector_lib` 2023-06-09 14:44:37 +03:00
Andrew Mayorov 8919a6ef93
refactor(pluglib): provide SQL related utils in `emqx_utils_sql` 2023-06-09 14:44:37 +03:00
Andrew Mayorov a51baaa206
refactor(pluglib): move conversion utils to `emqx_utils_conv` 2023-06-09 14:44:37 +03:00
Andrew Mayorov 7d0abb6146
feat(emqx): add `emqx_topic:match_any/2` utility 2023-06-09 14:44:37 +03:00
Andrew Mayorov d6c1ee183f
refactor(pluglib): move `emqx_placeholder` to utils app
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
Serge Tupchii 2aad6671ac fix(emqx_mgmt_data_backup): avoid writing temporary Mnesia backup file
As we use `keep_tables` option, we don't need to modify a backup schema before importing the backup file.
With this option, `mnesia:restore/2` ignores backup schema and keeps current table schema unchanged.
2023-06-09 14:41:21 +03:00
SergeTupchiy e61b2100a3
Merge pull request #10676 from SergeTupchiy/EMQX-9203-config-backup
feat: implement configuration and user data export/import
2023-06-09 14:18:39 +03:00
Serge Tupchii e4d09d4ad4 feat: implement configuration and user data export/import CLI
Closes: EMQX-9203
2023-06-09 14:11:47 +03:00
Kjell Winblad 4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631
feat: add sync/async option to the Kafka producer bridge
2023-06-09 12:51:23 +02:00
Kjell Winblad 2671e8ecf9 fix: dialyzer type problem 2023-06-09 11:00:05 +02:00
Ilya Averyanov c2635e938b
Merge pull request #10974 from keynslug/ft/EMQX-9523/backend-enable-flag
fix(ft): add `enable` flag to every backend
2023-06-09 11:44:37 +03:00
JianBo He 53eb8d7f1b test: fix flaky tests introduced by https://github.com/emqx/emqx/pull/10948 2023-06-09 15:48:23 +08:00