Commit Graph

135 Commits

Author SHA1 Message Date
Ilya Averyanov 228bf1a0ce chore(ft): fix typing issues 2023-04-07 17:25:22 +03:00
Ilya Averyanov 2c9cd1397d chore(ft): fix `?tp` calls 2023-04-07 17:25:22 +03:00
Ilya Averyanov c6b3f2c2ca feat(ft): add schema descriptions 2023-04-07 17:25:21 +03:00
Ilya Averyanov bc0a15afd7 feat(ft): fix test for CI 2023-04-07 17:25:21 +03:00
Ilya Averyanov 16fa55e19a feat(ft): add API tests 2023-04-07 17:25:21 +03:00
Andrew Mayorov 6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov f896fefa59 feat(ft): make storage backend fully async-aware
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 2cdf486bf4 feat(ft): simplify responder mechanism
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
Ilya Averyanov f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov 2e889f4ac7 feat(ft): add emqx_ft tests and fixes 2023-04-07 17:25:21 +03:00
Ilya Averyanov 1d48a97fd2 feat(ft): fix remote reader handling of gen_rpc errors 2023-04-07 17:25:21 +03:00
Ilya Averyanov c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov bcfa22f343 fix(ft): use correct supervison strategy for emqx_ft_sup 2023-04-07 17:25:21 +03:00
Ilya Averyanov 9e4a37a398 fix(ft): fix typespecs 2023-04-07 17:25:21 +03:00
Ilya Averyanov 836ec213c9 feat(ft): add responder tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov 8038a3fd4a feat(ft): add tests for remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov b7d0bad970 feat(ft): improve remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov 0aefd4a8c7 feat(ft): add streaming of file content when downloading 2023-04-07 17:25:21 +03:00
Ilya Averyanov 197ce32669 feat(ft): add proxy module for emqx_ft_storage_fs 2023-04-07 17:25:21 +03:00
Ilya Averyanov 04e5378bda feat(ft): add API 2023-04-07 17:25:21 +03:00
Andrew Mayorov 92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Andrew Mayorov 7ed06b0a2a feat(ft-fs): allow to list all transfers in storage
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 429eeaf029 feat(ft-fs): make `list` / `read` more generic
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 1308fa0e6b fix(ft-fs): put fragments into separate directories
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 8298236908 refactor(ft): bring back userdata to filemeta schema 2023-04-07 17:25:21 +03:00
Ilya Averyanov b4a42a447c feat(ft): removed replicated data 2023-04-07 17:25:21 +03:00
Ilya Averyanov 72e3eee6c9 feat(ft): add config & backend behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov 97b831a160 fix(ft-fs): add missing `read_segment/5` + fix atomic write 2023-04-07 17:25:21 +03:00
Andrew Mayorov 14b2a1013b fix(ft-asm): follow proper `segment` fragment type 2023-04-07 17:25:21 +03:00
Andrew Mayorov 1fedae8a16 fix(ft-asm): ensure module follows statem behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov 7b77e96ab9 test(ft): add some basic assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov cbff2e2309 feat(ft): improve robustness of asynchronous acks
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
2023-04-07 17:25:21 +03:00
Ilya Averyanov d36ca18bff feat(ft): tie file transfer frontend and backend together 2023-04-07 17:25:21 +03:00
Andrew Mayorov 81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00
Ilya Averyanov aaaef30be6 feat(ft): add file transfer app and bootstrap replicated ft data structure 2023-04-07 17:25:21 +03:00