yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,379 Commits 393 Branches 684 Tags 74 MiB
2707b4500f
Commit Graph

14379 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
9aec01e7a3 fix(ft-asm): use regular map for meta fragments 
There's actually no need to use an ordered data structure at all.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4132f5a5fb feat(ft): introduce exporter concept in local storage backend 
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45f00e14a9 feat(ft): ensure that clientid is always binary 
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0d39546080 feat(wdgraph): add fold/3 which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c821cd3bd test(ft): inject configs through hocon subsystem 
So that relevant parts of config would be initialized with defaults.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov
715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e9f98adca2 test: make content_gen emit meta with each chunk 
Which will also contain total content size, to increase this tool's
utility.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
788e76ed2d test: make assembly proptests waste less memory 2023-04-07 17:25:22 +03:00
Andrew Mayorov
8e6f960c09 refactor(ft): employ emqx_wdgraph for coverage computation 
Also describe how coverage problem maps to shortest path problem.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b189ee463c feat: add weighted directional graph ADT with shortest path 
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
130601376a perf(ft-asm): express coverage through shortest path on graph 
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity 
Through squashing segments table into consecutive "runs".
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
97cfdf8eef test(ft-asm): add property tests for file assembly 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Ilya Averyanov
ccba65e1ea chore(ft): add code owners for ft 2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c feat(ft): add segment checksum validation 
Also downgrade validation errors to mere info messages.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5014cc15f4 test(ft): do not rely on fs backend implementation details 2023-04-07 17:25:22 +03:00
Andrew Mayorov
93865a79e9 fix(ft): disallow empty fileids 2023-04-07 17:25:22 +03:00
Andrew Mayorov
75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov
7ff253e80d test(ft): attempt to avoid gen_rpc config reset hacks 2023-04-07 17:25:22 +03:00
Andrew Mayorov
a3f47641f3 test(ft): describe the complex test case with comments 
In order to make the test client model and behavior clearer.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
10d4c4305a feat(maybe): add basic tests for the new module 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2cf2a2d952 fix(ft): make fin packet parser stricter + safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
58115715dd fix(ft): require final size in fin packet 
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov
228bf1a0ce chore(ft): fix typing issues 2023-04-07 17:25:22 +03:00
Ilya Averyanov
2c9cd1397d chore(ft): fix ?tp calls 2023-04-07 17:25:22 +03:00
Ilya Averyanov
1c61165a91 fix(ft): fix emqx_cm takover 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c6b3f2c2ca feat(ft): add schema descriptions 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bc0a15afd7 feat(ft): fix test for CI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
16fa55e19a feat(ft): add API tests 2023-04-07 17:25:21 +03:00
Andrew Mayorov
6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov
f896fefa59 feat(ft): make storage backend fully async-aware 
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
2cdf486bf4 feat(ft): simplify responder mechanism 
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
 2023-04-07 17:25:21 +03:00
Ilya Averyanov
f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
2e889f4ac7 feat(ft): add emqx_ft tests and fixes 2023-04-07 17:25:21 +03:00
Ilya Averyanov
1d48a97fd2 feat(ft): fix remote reader handling of gen_rpc errors 2023-04-07 17:25:21 +03:00
Ilya Averyanov
c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
eae3dc7b4b feat(ft): update BPAPI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bcfa22f343 fix(ft): use correct supervison strategy for emqx_ft_sup 2023-04-07 17:25:21 +03:00