yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,429 Commits 393 Branches 669 Tags 74 MiB
92ff2b7d6e
Commit Graph

6848 Commits

Author SHA1 Message Date
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0d39546080 feat(wdgraph): add fold/3 which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c821cd3bd test(ft): inject configs through hocon subsystem 
So that relevant parts of config would be initialized with defaults.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov
715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e9f98adca2 test: make content_gen emit meta with each chunk 
Which will also contain total content size, to increase this tool's
utility.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
788e76ed2d test: make assembly proptests waste less memory 2023-04-07 17:25:22 +03:00
Andrew Mayorov
8e6f960c09 refactor(ft): employ emqx_wdgraph for coverage computation 
Also describe how coverage problem maps to shortest path problem.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b189ee463c feat: add weighted directional graph ADT with shortest path 
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
130601376a perf(ft-asm): express coverage through shortest path on graph 
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity 
Through squashing segments table into consecutive "runs".
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
97cfdf8eef test(ft-asm): add property tests for file assembly 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c feat(ft): add segment checksum validation 
Also downgrade validation errors to mere info messages.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5014cc15f4 test(ft): do not rely on fs backend implementation details 2023-04-07 17:25:22 +03:00
Andrew Mayorov
93865a79e9 fix(ft): disallow empty fileids 2023-04-07 17:25:22 +03:00
Andrew Mayorov
75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov
7ff253e80d test(ft): attempt to avoid gen_rpc config reset hacks 2023-04-07 17:25:22 +03:00
Andrew Mayorov
a3f47641f3 test(ft): describe the complex test case with comments 
In order to make the test client model and behavior clearer.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
10d4c4305a feat(maybe): add basic tests for the new module 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2cf2a2d952 fix(ft): make fin packet parser stricter + safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
58115715dd fix(ft): require final size in fin packet 
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov
228bf1a0ce chore(ft): fix typing issues 2023-04-07 17:25:22 +03:00
Ilya Averyanov
2c9cd1397d chore(ft): fix ?tp calls 2023-04-07 17:25:22 +03:00
Ilya Averyanov
1c61165a91 fix(ft): fix emqx_cm takover 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c6b3f2c2ca feat(ft): add schema descriptions 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bc0a15afd7 feat(ft): fix test for CI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
16fa55e19a feat(ft): add API tests 2023-04-07 17:25:21 +03:00
Andrew Mayorov
6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov
f896fefa59 feat(ft): make storage backend fully async-aware 
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
2cdf486bf4 feat(ft): simplify responder mechanism 
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
 2023-04-07 17:25:21 +03:00
Ilya Averyanov
f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
2e889f4ac7 feat(ft): add emqx_ft tests and fixes 2023-04-07 17:25:21 +03:00
Ilya Averyanov
1d48a97fd2 feat(ft): fix remote reader handling of gen_rpc errors 2023-04-07 17:25:21 +03:00
Ilya Averyanov
c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
eae3dc7b4b feat(ft): update BPAPI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bcfa22f343 fix(ft): use correct supervison strategy for emqx_ft_sup 2023-04-07 17:25:21 +03:00
Ilya Averyanov
9e4a37a398 fix(ft): fix typespecs 2023-04-07 17:25:21 +03:00
Ilya Averyanov
836ec213c9 feat(ft): add responder tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
8038a3fd4a feat(ft): add tests for remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov
b7d0bad970 feat(ft): improve remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov
0aefd4a8c7 feat(ft): add streaming of file content when downloading 2023-04-07 17:25:21 +03:00
Ilya Averyanov
197ce32669 feat(ft): add proxy module for emqx_ft_storage_fs 2023-04-07 17:25:21 +03:00
Ilya Averyanov
04e5378bda feat(ft): add API 2023-04-07 17:25:21 +03:00
Andrew Mayorov
92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Andrew Mayorov
7ed06b0a2a feat(ft-fs): allow to list all transfers in storage 
This is rather simplistic and thus, temporary solution.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
429eeaf029 feat(ft-fs): make list / read more generic 
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
1308fa0e6b fix(ft-fs): put fragments into separate directories 
In order to avoid potential filename collisions.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
8298236908 refactor(ft): bring back userdata to filemeta schema 2023-04-07 17:25:21 +03:00
Ilya Averyanov
b4a42a447c feat(ft): removed replicated data 2023-04-07 17:25:21 +03:00
Ilya Averyanov
72e3eee6c9 feat(ft): add config & backend behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov
97b831a160 fix(ft-fs): add missing read_segment/5 + fix atomic write 2023-04-07 17:25:21 +03:00
Andrew Mayorov
14b2a1013b fix(ft-asm): follow proper segment fragment type 2023-04-07 17:25:21 +03:00
Andrew Mayorov
1fedae8a16 fix(ft-asm): ensure module follows statem behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov
7b77e96ab9 test(ft): add some basic assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
cbff2e2309 feat(ft): improve robustness of asynchronous acks 
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
 2023-04-07 17:25:21 +03:00
Ilya Averyanov
d36ca18bff feat(ft): tie file transfer frontend and backend together 2023-04-07 17:25:21 +03:00
Andrew Mayorov
81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00
Ilya Averyanov
aaaef30be6 feat(ft): add file transfer app and bootstrap replicated ft data structure 2023-04-07 17:25:21 +03:00
Ilya Averyanov
d976943f99 feat(ft): add basic hooks 2023-04-07 17:25:21 +03:00
Kjell Winblad
1938882f16
Merge pull request #10318 from kjellwinblad/kjell/feat/rule_engine_from_clause_support_both_string_types 
feat(rule engine sql): enable both ' and " strings in FROM clause
 2023-04-06 09:19:44 +02:00
Serge Tupchii
9810c9f7e3 refactor(rule_engine): test 'ok' results in one dedicated function 2023-04-05 16:55:43 +03:00
Stefan Strigler
f3446c48f7
Merge pull request #10315 from sstrigler/EMQX-8945-crash-for-bad-param-value-to-mqtt-delayed-messages-api-call 
EMQX 8945 crash for bad param value to mqtt delayed messages api call
 2023-04-05 13:44:22 +02:00
Stefan Strigler
a39312e892
Merge pull request #10292 from sstrigler/EMQX-9253-readme-refine-emqx-management 
EMQX Management README
 2023-04-05 09:00:34 +02:00
Zaiming (Stone) Shi
a9bf633e03
Merge pull request #10320 from zmstone/0403-sync-release-50-back-to-master 
0403 sync release 50 back to master
 2023-04-04 23:31:24 +02:00
Thales Macedo Garitezi
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50 
feat(bridges): enable async query mode for all bridges with buffer workers
 2023-04-04 17:10:46 -03:00
Zaiming (Stone) Shi
68c15ffd48 Merge remote-tracking branch 'origin/release-50' into 0403-sync-release-50-back-to-master 2023-04-04 16:42:58 +02:00
Zaiming (Stone) Shi
d25db3ace4 chore: bump version to e5.0.2-rc.4 2023-04-04 16:38:25 +02:00
Zaiming (Stone) Shi
9e17064e47
Merge pull request #10325 from zmstone/0404-delay-config-change-replay-until-handler-is-ready 
0404 delay config change replay until handler is ready
 2023-04-04 16:37:35 +02:00
SergeTupchiy
877b828d4a
Merge pull request #10327 from SergeTupchiy/EMQX-8786-fix-unknown-counter-inc-on-unrecoverable-err 
fix(rule_engine): don't increment unknown counter on unrecoverable er…
 2023-04-04 16:51:34 +03:00
Zaiming (Stone) Shi
5925ff07c2 test(emqx_cluster_rpc): fix test cases 2023-04-04 15:20:30 +02:00
Zaiming (Stone) Shi
eeb7b32bc8
Merge pull request #10317 from zmstone/0403-refactor-hide-listener-level-authentication 
0403 refactor hide listener level authentication
 2023-04-04 15:04:35 +02:00
Serge Tupchii
aca65ca2d4 fix(rule_engine): don't increment unknown counter on unrecoverable errors 
Closes: EMQX-8786
 2023-04-04 15:59:53 +03:00
Zaiming (Stone) Shi
196ca43fbb fix(emqx_conf_app): call the right API to retrieve core nodes 2023-04-04 14:19:48 +02:00
Thales Macedo Garitezi
0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval 
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
 2023-04-04 08:58:36 -03:00
Zaiming (Stone) Shi
8fd9dd741e fix(emqx_conf_app): wait for tables ready beofre starting apps 2023-04-04 13:50:50 +02:00
Kjell Winblad
0e66eb5f3f feat(rule engine sql): enable both ' and " strings in FROM clause 
This commit upgrades the rulesql dependency to version 1.5 instead of
1.4. The difference between these two versions is that strings surrounded
by ' and " are supported in FROM clauses in version 1.5, but in version
1.4, only strings surrounded by " are supported.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9243
 2023-04-04 10:36:48 +02:00
Zaiming (Stone) Shi
b16c516e6b refactor: rename cluster_rpc_handler to cluster_rpc_cleaner 
this reflects what is actually does
 2023-04-04 10:28:47 +02:00
firest
6b2419998d chore: bump emqx_authn version 2023-04-04 16:06:23 +08:00
firest
11b3264251 fix: redact the password to ****** in API examples 
fix #10222
 2023-04-04 16:02:45 +08:00
Thales Macedo Garitezi
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50 
test(peer): increase init and startup timeout for peer nodes
 2023-04-03 15:46:13 -03:00
Zaiming (Stone) Shi
429b3d9efd Merge remote-tracking branch 'origin/master' into 0403-sync-release-50-back-to-master 2023-04-03 20:32:33 +02:00
Zaiming (Stone) Shi
2f2e8b8218 chore: bump version to e5.0.2-rc.3 2023-04-03 20:23:31 +02:00
Thales Macedo Garitezi
06048ae4ff
Merge pull request #10316 from thalesmg/flaky-crl-emqtt-test-v50 
test(crl): fix flaky test (v5.0)
 2023-04-03 15:12:36 -03:00
Thales Macedo Garitezi
f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers 
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
 2023-04-03 14:49:51 -03:00
Thales Macedo Garitezi
8b5a717a1f test(peer): increase init and startup timeout for peer nodes 
Attempt to stabilize tests that use cluster nodes.
 2023-04-03 13:20:22 -03:00
Zaiming (Stone) Shi
2d6ca69ffb refactor: no support for listener level authentication for now 2023-04-03 16:49:35 +02:00
Zaiming (Stone) Shi
e978d86c86 chore: add doc_lift for authorization.sources 
doc_lift is to make the doc render application to lift
this field to the root level and force the field's doc
to refernec it instead of expanding the structs in a nested way
 2023-04-03 16:49:35 +02:00
Thales Macedo Garitezi
ed25ee6fec test(crl): fix flaky test (v5.0) 2023-04-03 10:58:31 -03:00
Stefan Strigler
0efa9c7a11 fix: pretty format error responses 2023-04-03 15:48:52 +02:00
Stefan Strigler
c1cb5357e1 fix: enable schema check 2023-04-03 15:48:33 +02:00
Thales Macedo Garitezi
ec1871ffde test(janitor): catch each callback invocation 2023-04-03 10:20:19 -03:00
JimMoen
53712e6146
fix: running nodes should not include replica nodes 2023-04-03 20:01:37 +08:00