Andrew Mayorov
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
Andrew Mayorov
e9f98adca2
test: make content_gen emit meta with each chunk
...
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
Andrew Mayorov
788e76ed2d
test: make assembly proptests waste less memory
2023-04-07 17:25:22 +03:00
Andrew Mayorov
97cfdf8eef
test(ft-asm): add property tests for file assembly
2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c
feat(ft): add segment checksum validation
...
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
Andrew Mayorov
5014cc15f4
test(ft): do not rely on fs backend implementation details
2023-04-07 17:25:22 +03:00
Andrew Mayorov
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
Andrew Mayorov
7ff253e80d
test(ft): attempt to avoid `gen_rpc` config reset hacks
2023-04-07 17:25:22 +03:00
Andrew Mayorov
a3f47641f3
test(ft): describe the complex test case with comments
...
In order to make the test client model and behavior clearer.
2023-04-07 17:25:22 +03:00
Andrew Mayorov
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Ilya Averyanov
c073914f75
chore(ft): fix gen_rpc flakyness
2023-04-07 17:25:22 +03:00
Ilya Averyanov
bc0a15afd7
feat(ft): fix test for CI
2023-04-07 17:25:21 +03:00
Ilya Averyanov
16fa55e19a
feat(ft): add API tests
2023-04-07 17:25:21 +03:00
Andrew Mayorov
6d9f780313
chore: remove unused code
2023-04-07 17:25:21 +03:00
Andrew Mayorov
f896fefa59
feat(ft): make storage backend fully async-aware
...
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
Andrew Mayorov
2cdf486bf4
feat(ft): simplify responder mechanism
...
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
Ilya Averyanov
f6a0598f27
feat(ft): add file transfer tests
2023-04-07 17:25:21 +03:00
Ilya Averyanov
2e889f4ac7
feat(ft): add emqx_ft tests and fixes
2023-04-07 17:25:21 +03:00
Ilya Averyanov
c44fe92ef1
feat(ft): add assembler tests
2023-04-07 17:25:21 +03:00
Ilya Averyanov
836ec213c9
feat(ft): add responder tests
2023-04-07 17:25:21 +03:00
Ilya Averyanov
8038a3fd4a
feat(ft): add tests for remote reader
2023-04-07 17:25:21 +03:00
Andrew Mayorov
7ed06b0a2a
feat(ft-fs): allow to list all transfers in storage
...
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
Andrew Mayorov
429eeaf029
feat(ft-fs): make `list` / `read` more generic
...
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
Andrew Mayorov
1308fa0e6b
fix(ft-fs): put fragments into separate directories
...
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
Ilya Averyanov
b4a42a447c
feat(ft): removed replicated data
2023-04-07 17:25:21 +03:00
Andrew Mayorov
7b77e96ab9
test(ft): add some basic assembler tests
2023-04-07 17:25:21 +03:00