e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
bcd2099ce1
fix(fs-gc): make deletion empty transfer directories safer
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
8e6f960c09
refactor(ft): employ `emqx_wdgraph` for coverage computation
...
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
130601376a
perf(ft-asm): express coverage through shortest path on graph
...
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
0c84fc28b0
perf(asm-ft): tradeoff optimality for computational complexity
...
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
0af7e2a002
perf(ft-asm): optimize away remote segments identical to local
2023-04-07 17:25:22 +03:00
15d967459c
feat(ft): add segment checksum validation
...
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
5998961f9f
fix(ft): log errors where they might get lost
2023-04-07 17:25:22 +03:00
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
75070102ec
fix(ft): improve typespecs
2023-04-07 17:25:22 +03:00
2b925aa60b
fix(ft): drop unrelated TODO
2023-04-07 17:25:22 +03:00
2cf2a2d952
fix(ft): make `fin` packet parser stricter + safer
2023-04-07 17:25:22 +03:00
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
228bf1a0ce
chore(ft): fix typing issues
2023-04-07 17:25:22 +03:00
2c9cd1397d
chore(ft): fix `?tp` calls
2023-04-07 17:25:22 +03:00
c6b3f2c2ca
feat(ft): add schema descriptions
2023-04-07 17:25:21 +03:00
6d9f780313
chore: remove unused code
2023-04-07 17:25:21 +03:00
f896fefa59
feat(ft): make storage backend fully async-aware
...
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
2cdf486bf4
feat(ft): simplify responder mechanism
...
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
f6a0598f27
feat(ft): add file transfer tests
2023-04-07 17:25:21 +03:00
2e889f4ac7
feat(ft): add emqx_ft tests and fixes
2023-04-07 17:25:21 +03:00
1d48a97fd2
feat(ft): fix remote reader handling of gen_rpc errors
2023-04-07 17:25:21 +03:00
c44fe92ef1
feat(ft): add assembler tests
2023-04-07 17:25:21 +03:00
bcfa22f343
fix(ft): use correct supervison strategy for emqx_ft_sup
2023-04-07 17:25:21 +03:00
9e4a37a398
fix(ft): fix typespecs
2023-04-07 17:25:21 +03:00
836ec213c9
feat(ft): add responder tests
2023-04-07 17:25:21 +03:00
8038a3fd4a
feat(ft): add tests for remote reader
2023-04-07 17:25:21 +03:00
b7d0bad970
feat(ft): improve remote reader
2023-04-07 17:25:21 +03:00
0aefd4a8c7
feat(ft): add streaming of file content when downloading
2023-04-07 17:25:21 +03:00
197ce32669
feat(ft): add proxy module for emqx_ft_storage_fs
2023-04-07 17:25:21 +03:00
04e5378bda
feat(ft): add API
2023-04-07 17:25:21 +03:00
92670bfe3d
feat(ft): add fs storage bpapi and use it in assembler
2023-04-07 17:25:21 +03:00
7ed06b0a2a
feat(ft-fs): allow to list all transfers in storage
...
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
429eeaf029
feat(ft-fs): make `list` / `read` more generic
...
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
1308fa0e6b
fix(ft-fs): put fragments into separate directories
...
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
8298236908
refactor(ft): bring back userdata to filemeta schema
2023-04-07 17:25:21 +03:00
b4a42a447c
feat(ft): removed replicated data
2023-04-07 17:25:21 +03:00
72e3eee6c9
feat(ft): add config & backend behaviour
2023-04-07 17:25:21 +03:00
97b831a160
fix(ft-fs): add missing `read_segment/5` + fix atomic write
2023-04-07 17:25:21 +03:00
14b2a1013b
fix(ft-asm): follow proper `segment` fragment type
2023-04-07 17:25:21 +03:00
1fedae8a16
fix(ft-asm): ensure module follows statem behaviour
2023-04-07 17:25:21 +03:00
cbff2e2309
feat(ft): improve robustness of asynchronous acks
...
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
2023-04-07 17:25:21 +03:00
d36ca18bff
feat(ft): tie file transfer frontend and backend together
2023-04-07 17:25:21 +03:00
81e04ce93a
feat(ft): introduce simple filesystem storage backend + assembler
2023-04-07 17:25:21 +03:00
aaaef30be6
feat(ft): add file transfer app and bootstrap replicated ft data structure
2023-04-07 17:25:21 +03:00
Andrew Mayorov