Commit Graph

11 Commits

Author SHA1 Message Date
Andrew Mayorov a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
Andrew Mayorov 9aec01e7a3 fix(ft-asm): use regular map for meta fragments
There's actually no need to use an ordered data structure at all.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4f2600b9f1 feat(ft-gc): treat all transfer as incomplete
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 8e6f960c09 refactor(ft): employ `emqx_wdgraph` for coverage computation
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 130601376a perf(ft-asm): express coverage through shortest path on graph
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov 75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Andrew Mayorov 14b2a1013b fix(ft-asm): follow proper `segment` fragment type 2023-04-07 17:25:21 +03:00
Andrew Mayorov 81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00