Commit Graph

25 Commits

Author SHA1 Message Date
Ilya Averyanov 534c9bdc13 feat(ft): add additional operation status report channel 2023-10-11 19:51:36 +03:00
Andrew Mayorov 52354bf58a
fix(ft-fs): use `emqx:running_nodes()` as default cluster view
When iterating over complete exports cluster-wide.
2023-06-14 22:54:56 +03:00
Andrew Mayorov 0293b54211
fix(ft): respect checksum in `fin` packets 2023-05-31 18:50:38 +03:00
Andrew Mayorov a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
Andrew Mayorov 6b688d6646
fix(ft): anticipate repeated `kickoff`s + fix testcase 2023-05-10 17:35:53 +03:00
Ilya Averyanov 8ac881a140 chore(ft): handle multiple/concurrent fins more gracefully 2023-05-09 23:31:39 +05:00
Andrew Mayorov 5efd590ca4
feat(ft): properly propagate config updates
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
Ilya Averyanov 818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Andrew Mayorov 23cd78b8d6 refactor(ft-asm): turn state data record into a map
Which should be more future-proof.
2023-04-07 17:25:22 +03:00
Andrew Mayorov b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov 4132f5a5fb feat(ft): introduce exporter concept in local storage backend
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4f2600b9f1 feat(ft-gc): treat all transfer as incomplete
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov 75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov 2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Andrew Mayorov 6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov f896fefa59 feat(ft): make storage backend fully async-aware
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
Ilya Averyanov c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov 9e4a37a398 fix(ft): fix typespecs 2023-04-07 17:25:21 +03:00
Andrew Mayorov 92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Andrew Mayorov 429eeaf029 feat(ft-fs): make `list` / `read` more generic
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 1fedae8a16 fix(ft-asm): ensure module follows statem behaviour 2023-04-07 17:25:21 +03:00
Ilya Averyanov d36ca18bff feat(ft): tie file transfer frontend and backend together 2023-04-07 17:25:21 +03:00
Andrew Mayorov 81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00