Commit Graph

23 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Ilya Averyanov 534c9bdc13 feat(ft): add additional operation status report channel 2023-10-11 19:51:36 +03:00
Ilya Averyanov 58a83739b8
Merge pull request #10986 from savonarola/ft/EMQX-9523/config-api
feat(ft-api): provide configuration API
2023-07-10 15:32:30 +03:00
Andrew Mayorov 34793c5ed0 feat(ft-conf): preprocess TLS configuration on updates
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
2023-07-07 19:15:42 +03:00
Andrew Mayorov 97b6c43062
fix(ft-test): use new cth tooling in `emqx_ft_storage_fs_SUITE`
In attempt to battle test flakiness.
2023-07-07 14:00:33 +02:00
Andrew Mayorov f76f3b77d7
test: add tooling to make testruns more isolated and predictable
Also showcase their usage in some of the existing testsuites.
2023-07-04 12:53:39 +02:00
Andrew Mayorov 6dc281345c
test(ft-fs): run API tests with replicant nodes 2023-06-15 00:50:18 +03:00
Andrew Mayorov ae97d516b8
fix(ft): add `enable` flag to every backend
And tolerate multiple configured backends, as long as only one of
them is enabled.
2023-06-08 14:54:13 +03:00
Andrew Mayorov 5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
2023-05-09 23:32:16 +05:00
Ilya Averyanov cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
Andrew Mayorov 811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Andrew Mayorov 5efd590ca4
feat(ft): properly propagate config updates
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
Ilya Averyanov 69c4ba2a62 feat(ft): use new utils application 2023-04-22 22:50:19 +03:00
Andrew Mayorov 8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
Ilya Averyanov 4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Andrew Mayorov 6ad7ce55d2 fix(ft-conf): separate local segments storage settings
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4132f5a5fb feat(ft): introduce exporter concept in local storage backend
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0c821cd3bd test(ft): inject configs through hocon subsystem
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
Ilya Averyanov 15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Ilya Averyanov c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00