yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,763 Commits 393 Branches 684 Tags 74 MiB
b80513e941
Commit Graph

244 Commits

Author SHA1 Message Date
Serge Tupchii
d2a1a7f7cf chore: rename message_queue_too_long error reason to mailbox_overflow 
`mailbox_overflow` is consistent with the corresponding config parameter:
 'force_shutdown.max_mailbox_size'
 2024-03-22 12:20:20 +02:00
Serge Tupchii
cb5fdb3c79 fix: rework In-flight / Mqueue API 
- use timestamp 'position' to find the next chunk of data
- add 'start' position to response meta
- sort In-flight messages by insertion time
- sort Mqueue messages by priority
 2024-03-20 15:25:33 +02:00
Andrew Mayorov
d30c99512a
feat(utils-stream): add a few more stream combinators 2024-03-19 20:21:54 +01:00
Ivan Dyachkov
923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56 
sync release-56
 2024-03-19 19:09:41 +01:00
Thales Macedo Garitezi
f84a996671 feat: implement message validation 
Fixes https://emqx.atlassian.net/browse/EMQX-11980
 2024-03-18 13:11:39 -03:00
Shawn
29111a2192 fix: dialyzer problems 2024-03-08 14:25:19 +08:00
Shawn
163d095dca fix: port the changes for date_to_unix_ts SQL fun from 4.4 2024-03-08 10:48:08 +08:00
zmstone
f57f2fa1b7 chore: bump app version numbers 2024-03-06 19:37:06 +01:00
zmstone
9929025820 fix(rule_func): time zone shift at wrong precision 2024-03-04 21:25:49 +01:00
Thales Macedo Garitezi
577ef41e45 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 12:48:06 -03:00
lafirest
79f0720d14
Merge pull request #12635 from lafirest/fix/r551 
fix(http): fix that sensitive headers may be printed in log when querying
 2024-03-04 22:32:14 +08:00
Thales Macedo Garitezi
0dd9990f24 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 09:44:30 -03:00
firest
c5eb09a72f fix(http): fix that sensitive headers may be printed in log when querying 2024-03-04 20:36:01 +08:00
JimMoen
0a33f9d027
fix(calendar): leap year time to unix timestamp 2024-03-04 15:37:49 +08:00
firest
1c1c4e497d chore: bump version && update change 2024-03-01 12:42:19 +08:00
firest
5a3a34cce7 fix(redact): enhanced the redact for sensitive headers 2024-03-01 12:30:26 +08:00
firest
0670272188 fix: Revert "fix: redact all headers from logs" 
This reverts commit d8032f47ca.
 2024-03-01 10:32:57 +08:00
Andrew Mayorov
367ffa8e80
feat(utils-fs): add function to compute relpaths naively 2024-02-27 23:41:50 +01:00
Thales Macedo Garitezi
d8032f47ca fix: redact all headers from logs 
Fixes https://emqx.atlassian.net/browse/EMQX-11904

Since headers are usually used for authentication and the headers used for that are very
flexible, we redact all headers from logs to avoid leaking anything.
 2024-02-26 18:04:12 -03:00
Zaiming (Stone) Shi
00c2aeb1ab refactor: delete mfa from log metadata 2024-02-26 12:15:00 +01:00
Zaiming (Stone) Shi
46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Andrew Mayorov
ae8f59979d
fix(utils): handle improper lists as well in redact/1 2024-02-22 13:35:11 +01:00
Thales Macedo Garitezi
d469f4158e chore: bump app vsns 2024-02-20 16:53:57 -03:00
zhongwencool
e9c8446d57 feat: upgrade erlfmt to support maybe syntax 2024-01-31 20:06:49 +08:00
JianBo He
d12335c4c6 chore: add tests 2024-01-29 18:10:42 +08:00
JianBo He
ab99a17c99 chore(utils_stream): simplfy the csv reader implementation 2024-01-29 11:10:50 +08:00
JianBo He
2e35024df1 test: update eunit tests 2024-01-29 10:49:07 +08:00
JianBo He
9915c85b0b chore(authn_mnesia): use emqx_utils_stream module to imporve reusability 2024-01-29 10:49:07 +08:00
Serge Tupchii
d3a6870097 feat(emqx_utils): add pforeach/2,3 2024-01-19 21:44:31 +02:00
Serge Tupchii
54457b7093 feat(emqx_utils): allow infinity timeout in pmap/3 2024-01-19 19:34:04 +02:00
Andrew Mayorov
eca2f972bc
Merge pull request #12237 from keynslug/fix/EMQX-11483/broker 
feat(routing): add route sync process pool
 2024-01-12 14:44:31 +01:00
Andrew Mayorov
f92b5b3f32
feat(stream): add simple stream over process message queue 2024-01-12 11:57:23 +01:00
Thales Macedo Garitezi
79a4a041e4 fix(bridge_api): correctly deobfuscate secrets during dry run 
Fixes https://emqx.atlassian.net/browse/EMQX-11733
 2024-01-11 15:42:14 -03:00
Ivan Dyachkov
06117c3a33 Merge remote-tracking branch 'upstream/release-54' into 0105-sync-r54 2024-01-05 14:20:38 +01:00
zhongwencool
90404b62a4 fix: incorrectly attempt to update the field value when the previous field don't have value 2024-01-04 10:12:34 +08:00
Zaiming (Stone) Shi
23ded313ec chore: update app versions 2023-12-22 15:29:22 +01:00
Zaiming (Stone) Shi
891fd972bd Merge remote-tracking branch 'origin/release-54' into 1221-prepare-bpapi-on-otp-26 2023-12-22 15:19:25 +01:00
zhongwencool
a16bce8c24 fix: deobfuscate ft's secret keys in api 2023-12-22 13:13:44 +01:00
Zaiming (Stone) Shi
20543d55ef chore: bump app vsn 2023-12-22 13:13:30 +01:00
Zaiming (Stone) Shi
322b7bb7d2 chore: bump app vsn 2023-12-22 13:00:37 +01:00
Thales Macedo Garitezi
cf9331a95f Merge branch 'release-54' into sync-r54-m-20231218 2023-12-18 17:21:08 -03:00
Thales Macedo Garitezi
2c61b2bfbb fix(mysql_bridge): forbid update statements with batch operations 
Fixes https://emqx.atlassian.net/browse/EMQX-11605
 2023-12-18 15:05:14 -03:00
Zaiming (Stone) Shi
35504bd323 refactor: move ntoa (ip address formatting) code to emqx_utils 2023-12-17 21:03:16 +01:00
zhongwencool
d9e179b364 fix: use microsecond precision as the primary key for audit logs to prevent duplication 2023-12-15 09:07:04 +08:00
Stefan Strigler
be31486983 fix(emqx_utils): use deep_merge in unindent 2023-12-11 14:40:24 +01:00
Stefan Strigler
5d7ae4b980
Merge pull request #12132 from sstrigler/EMQX-11154-bridge-v-2-my-sql-support 
feat(emqx_bridge_mysql): port to shared connectors
 2023-12-11 13:56:34 +01:00
Zaiming (Stone) Shi
ddbb8560fa fix(dialyzer): batch 2 2023-12-08 17:59:55 +01:00
Stefan Strigler
71607aa2ad feat(emqx_bridge_mysql): port to shared connectors 2023-12-08 16:15:18 +01:00
Zaiming (Stone) Shi
423b586c56 fix(dialyzer): fix some dialyzer issues found on otp 26 2023-12-06 20:32:49 +01:00
Zaiming (Stone) Shi
de61d9d609 chore: upgrade to jiffy from 1.0.5 to 1.0.6 
1.0.5 has an unexported type which causes dialyzer to fail on otp 26
 2023-12-06 19:18:15 +01:00
SergeTupchiy
28ff53e99c
Merge pull request #11984 from SergeTupchiy/EMQX-10535-openetelmetry-tracing-new 
Integrate OpenTelmetry tracing
 2023-12-06 11:45:41 +02:00
Andrew Mayorov
3798060543
Merge pull request #12109 from keynslug/test/emqx-cth-suite 
test(emqx): switch select test suites to use `emqx_cth_suite`
 2023-12-06 09:48:53 +01:00
Thales Macedo Garitezi
f489de8860
Merge pull request #12100 from thalesmg/fix-password-action-probe-m-20231205 
fix(bridges/actions api): correctly deobfuscate passwords when probing
 2023-12-05 14:25:32 -03:00
Serge Tupchii
7fdc650448 feat: integrate OpenTelemetry traces 2023-12-05 18:11:00 +02:00
Thales Macedo Garitezi
e42f4155d3 fix(bridges/actions api): correctly deobfuscate passwords when probing 
Fixes https://emqx.atlassian.net/browse/EMQX-11533
 2023-12-05 11:49:50 -03:00
Andrew Mayorov
286d483a3a
Merge pull request #12055 from keynslug/ft/EMQX-11474/subopts 
fix(sessds): respect subscription options when publishing
 2023-12-05 15:45:25 +01:00
Andrew Mayorov
b5c485cd55
test(sessds): add empty level topic subscription testcase 2023-12-04 13:38:42 +03:00
Andrew Mayorov
29ec73847a
fix(utils): make flattermap/2 results less variative 2023-12-02 14:38:04 +03:00
Andrew Mayorov
b5f39f89e3
feat(utils): add flattermap/2 as slightly more generic flatmap/2 2023-12-02 14:36:21 +03:00
Ivan Dyachkov
9fd2fa95a8 chore: bump apps versions 2023-11-30 20:01:12 +01:00
zhongwencool
b5a00ec6b2
Merge pull request #12023 from emqx/master 
chore: sync master to release-54
 2023-11-27 09:31:47 +08:00
Thales Macedo Garitezi
eb3f54184e refactor: address review comments and avoid transformations without schema knowledge 2023-11-17 16:16:34 -03:00
Andrew Mayorov
d019be5806
Merge pull request #11935 from keynslug/feat/EMQX-10713/routing-v2-default 
feat(router): switch to v2 routing store by default
 2023-11-16 15:56:26 +07:00
Andrew Mayorov
8919b08207
fix(utils): rename emqx_utils_stream:take/2 to consume/2 
Which is more neutral and harder to confuse with a destructive `take` in
collections.
 2023-11-15 17:20:40 +07:00
Andrew Mayorov
6812ee9d0f
fix(mgmt): hide route selection behind router interface 
Also introduce a generic _stream_ concept, mostly to deal with
iterating over 2 ETS tables at once with `ets:match_object/3`.
 2023-11-14 18:36:25 +07:00
Ivan Dyachkov
28a577ad09 chore: bump apps versions 2023-11-14 11:02:26 +01:00
ieQu1
788698f157
Merge pull request #11720 from ieQu1/dev/refactor-persistent-session 
Refactor emqx_durable storage application and introduce learned topic structure storage
 2023-11-08 14:27:40 +01:00
Ivan Dyachkov
0c91bec98d chore: merge 'upstream/release-53' 2023-11-08 09:24:38 +01:00
ieQu1
7cb0322856 fix(emqx): Move bpapi and emqx message record to emqx_utils app 2023-11-08 03:04:19 +01:00
Andrew Mayorov
910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder 
feat(tpl): split `emqx_placeholder` into a couple of modules
 2023-11-02 22:50:05 +07:00
Andrew Mayorov
f1847fe494
chore(tpl): drop unnecessary binding 2023-11-02 20:13:32 +07:00
Andrew Mayorov
e521a9f5fc
fix(utils): denote emqx_jsonish follows access module behaviour 
Defined in `emqx_template`.
 2023-11-02 20:13:32 +07:00
Andrew Mayorov
b5b6c3f8cc
fix(tpl): ensure full backward compat in basic connectors 2023-11-02 20:13:32 +07:00
Andrew Mayorov
02c1bd70b6
feat(tpl): factor out loose json concept into a separate module 
Which is called `emqx_jsonish`. Also introduce an _access module_
abstraction to extract information from such data during rendering.
 2023-11-02 20:13:31 +07:00
Andrew Mayorov
69cfa740ea
fix(ruleeng): ensure full backward compatibility 2023-11-02 17:11:12 +07:00
Andrew Mayorov
8e4585d64f
chore: move template modules to emqx_utils 
Even though most of the time these modules will be used by
connectors, there are exceptions (namely, `emqx_rule_engine`).
Besides, they are general enough to land there, more so given
that `emqx_placeholder` is already there.
 2023-11-02 17:11:12 +07:00
Thales Macedo Garitezi
45a39d97c6 fix(bridges_v1): avoid create dangling connectors when updating bridges via api v1 
Fixes https://emqx.atlassian.net/browse/EMQX-11291
 2023-10-31 16:39:47 -03:00
Ivan Dyachkov
b1ab213081 chore: merge 'upstream/release-53' into 1031-sync-r53 2023-10-31 11:06:25 +01:00
Kjell Winblad
9dc3a169b3 feat: split bridges into a connector part and a bridge part 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>

Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.

Issues:
https://emqx.atlassian.net/browse/EMQX-10805
 2023-10-30 14:48:47 +01:00
Ilya Averyanov
6354f3b04f feat(authn): allow authn providers to define a separate schama for API 2023-10-17 13:19:11 +03:00
Zaiming (Stone) Shi
bce8fd2fbc chore: bump app versions 2023-09-29 18:56:52 +02:00
Zaiming (Stone) Shi
1a13b2ac56 Merge remote-tracking branch 'origin/release-53' into 0928-sync-release-53 2023-09-28 13:53:03 +02:00
Zaiming (Stone) Shi
45caa3bf01 fix(sso): make sp_private_key sensitive 
so it will not be logged
 2023-09-27 21:27:59 +02:00
Zaiming (Stone) Shi
5e6996dc05 refactor: log formatter format mfa+line as m:f/a(line) 
also improve json formatter when a field is iolist
 2023-09-25 16:55:22 +02:00
Andrew Mayorov
81cf619f07
fix(ftconf): also mark secret_access_key key as sensitive 2023-09-25 14:52:57 +03:00
Ivan Dyachkov
dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
firest
0846939760 fix(ldap): remove unused code and mark sensitive field 2023-09-15 10:13:07 +08:00
Thales Macedo Garitezi
080cb73da1 fix: handle badmap inside emqx_placeholder:proc_tmpl 2023-09-07 13:40:03 -03:00
firest
8cd21da94b chore: update apps version 2023-08-16 10:55:54 +08:00
firest
b08102269a refactor(calendar): refactor datetime-related code and remove redundant 2023-08-16 10:55:46 +08:00
firest
3e9155fdb1 chore: bump emqx_utils version && changes 2023-08-09 10:54:56 +08:00
firest
7567d211da fix(placeholder): porting fix to support utf8 key in placeholder 2023-08-09 10:54:41 +08:00
Ilya Averyanov
56c81c2c25 feat(ft-api): bump app versions 2023-07-07 19:17:22 +03:00
Andrew Mayorov
a2b03716be feat(ft-api): provide configuration API 
To configure `emqx_ft` during the runtime.
 2023-07-07 19:15:42 +03:00
Thales Macedo Garitezi
631f4ceac9 fix(placeholder): add back support for nested keys inside binary encoded json 
Fixes https://emqx.atlassian.net/browse/EMQX-10459
 2023-06-30 10:05:49 -03:00
firest
31e914f6a2 fix(emqx_utils): improve the checking for the authorization key 2023-06-29 15:57:48 +08:00
firest
d485f8fe96 fix: mask the value of the authorization header whatever its case 2023-06-24 09:20:26 +00:00
Thales Macedo Garitezi
7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer 
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
 2023-06-22 09:17:45 -03:00
Zaiming (Stone) Shi
7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Thales Macedo Garitezi
b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-10281
 2023-06-19 16:04:12 -03:00
Thales Macedo Garitezi
50e7d5d2ec fix(nolink_apply): avoid sending late replies to caller 
Due to race conditions, it's possible that the caller to `pmap`/`nolink_apply` might
receive a late reply.

e.g. when a timeout occurred while resource manager was checking a resource's health:

```
19:18:23.084 [error] [data: ..., event_data: {#Reference<0.3247872820.3887857670.131018>, {:normal, [false, true, true, true, true, true]}}, event_type: :info, msg: :ignore_all_other_events, state: :connected]
```

Using an alias and also checking for the race condition in the `after` block (like
[`gen`](a76bf63197/lib/stdlib/src/gen.erl (L270-L277))
does), we avoid polluting the caller's mailbox with late replies.
 2023-06-16 17:03:39 -03:00
firest
0ed420f81f fix: obfuscated the value of JWT 2023-06-12 18:01:02 +08:00
zhongwencool
a10107045f test: delete emqx_ft_fs_utils_SUITE_data from source code 2023-06-11 15:50:45 +08:00
zhongwencool
aed64aae60 chore: delete emqx_utils_fs_SUITE_data dir 2023-06-11 15:50:45 +08:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi
f98cdd4983
Merge pull request #10994 from sstrigler/EMQX-10003-e-5-0-4-auth-header-value-of-webhook-data-bridge-can-be-found-in-emqx-log 
fix(emqx_utils): redact proxy-authorization headers
 2023-06-10 11:18:58 +02:00
Serge Tupchii
87b57112df refactor: move tcp keepalive options helper to emqx_utils 2023-06-09 21:43:50 +03:00
Stefan Strigler
57d72ed23e fix(emqx_utils): redact proxy-authorization headers 2023-06-09 17:13:53 +02:00
Andrew Mayorov
8919a6ef93
refactor(pluglib): provide SQL related utils in emqx_utils_sql 2023-06-09 14:44:37 +03:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Andrew Mayorov
d6c1ee183f
refactor(pluglib): move emqx_placeholder to utils app 
Also make user that existing code calls it directly.
 2023-06-09 14:44:36 +03:00
Serge Tupchii
e4d09d4ad4 feat: implement configuration and user data export/import CLI 
Closes: EMQX-9203
 2023-06-09 14:11:47 +03:00
Zaiming (Stone) Shi
ccd2589ff2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-07 21:43:29 +02:00
Andrew Mayorov
750b7158d4
fix(tlsgc): make computing orphans safe against symlinking 
Before this commit, `orphans/1` considered managed file an orphan based
on its absolute filename, which is not safe when, for example, emqx has
a symlink configured as `data_dir` while config references certfiles
through realpaths.
 2023-06-05 22:41:08 +03:00
Andrew Mayorov
468fd98173
test(utilfs): add few more traverse_dir/3 testcases 2023-06-05 22:41:08 +03:00
Andrew Mayorov
8a31e5639b
fix(emqx): ensure that standalone build works 
Turns out rebar3 `path` resource has troubles with dangling
symlinks and non-ASCII filenames. 🥲
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
1cb226dffb
chore: bump applications versions 
* emqx_authz 0.1.22
* emqx_exhook 5.0.13
* emqx_utils 5.0.3
 2023-06-05 22:41:07 +03:00
Andrew Mayorov
afbf8cb32f
chore(util): drop obsolete stuff from emqx_utils_maps 2023-06-05 22:41:07 +03:00
Andrew Mayorov
99ea9b86c2
feat(tlslib): add separate managed certfiles GC process 
Which periodically inpects managed certificates directory and tries
to collect "orphans" here, in other words files that aren't
referenced anywhere in the current emqx config.
 2023-06-05 22:41:07 +03:00
某文
bd29433997 feat: support emqx_conf:update([exhook],Conf) 2023-06-04 09:30:17 +08:00
JimMoen
dcfe985ee9
fix: redact api request to hide auth token 2023-05-29 11:02:02 +08:00
Ilya Averyanov
e683d28973 chore(rebalance): rebase and review fixes 2023-05-09 20:51:24 +05:00
Zhongwen Deng
04e62f6a2d test: check_oom's max_mailbox_size 2023-05-08 20:27:52 +08:00
Zhongwen Deng
4f396a36a9 Merge remote-tracking branch 'upstream/master' into release-50 2023-05-08 14:58:03 +08:00
firest
277deee616 fix: add new sensitive keywords to redact checklist 2023-05-04 16:32:32 +08:00
William Yang
5ed3c3a92c perf(config): eliminate make_ref() calls in config get calls 2023-04-26 10:58:08 +02:00
Stefan Strigler
17d84fb5e0 Merge branch 'EMQX-9549-new-emqx-utils-app-to-collect-utility-modules' of github.com:sstrigler/emqx into EMQX-9549-new-emqx-utils-app-to-collect-utility-modules 2023-04-14 18:53:47 +02:00
Stefan Strigler
9ccfa643ce
style: fix wording 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-14 18:51:51 +02:00
Stefan Strigler
a20797160e style: remove unnecessary ifdef(TEST) 2023-04-14 18:45:25 +02:00
Stefan Strigler
24df1045de fix: test not updated after rebase 2023-04-14 16:40:25 +02:00
Stefan Strigler
a9976287b5 fix: force proper calling decode to return proplist 2023-04-14 16:26:32 +02:00
Stefan Strigler
92ca2f66f5 style: add original copyright header 2023-04-14 13:41:34 +02:00
Stefan Strigler
0f162fb50a test: add tests for emqx_utils_binary from original site 2023-04-14 13:41:34 +02:00
Stefan Strigler
badf962800 fix: stale call to emqx_misc 2023-04-14 13:41:34 +02:00
Stefan Strigler
d98f7222ff style: add comment to binary_string/1 2023-04-14 13:41:34 +02:00
Stefan Strigler
90520a5382 docs: add an actual README 2023-04-14 13:41:34 +02:00
Stefan Strigler
4f80690162 fix: byebye jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler
19981757ae fix: add is_json 2023-04-14 13:41:34 +02:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
6e8665365b refactor: rename emqx_tables to emqx_utils_ets 2023-04-14 13:41:33 +02:00
Stefan Strigler
d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
1880da0a2e refactor: move binary_util to emqx_utils_binary 2023-04-14 13:35:15 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00