Commit Graph

10 Commits

Author SHA1 Message Date
Andrew Mayorov 8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
Ilya Averyanov 4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Andrew Mayorov 6ad7ce55d2 fix(ft-conf): separate local segments storage settings
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4132f5a5fb feat(ft): introduce exporter concept in local storage backend
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0c821cd3bd test(ft): inject configs through hocon subsystem
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
Ilya Averyanov 15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Ilya Averyanov c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00