d333187c5a
docs(s3): update readme
2023-04-07 17:36:47 +03:00
820e06d756
fix(ft-s3): make controllable use of max_retries and request_timeout arguments
2023-04-07 17:36:47 +03:00
be99242e32
fix(ft-s3): fix review comments
2023-04-07 17:36:47 +03:00
d7a85242de
fix(ft-s3): fix s3 listing
2023-04-07 17:36:47 +03:00
7d13862da5
fix(ft-s3): use uploader specific headers only for object creation
2023-04-07 17:36:47 +03:00
c7865e5eae
fix(ft-s3): fix dependency dependencies
2023-04-07 17:36:45 +03:00
4fbabe5a76
fix(ft-s3): fix atom and variable naming
2023-04-07 17:27:01 +03:00
71965d90e4
fix(ft-s3): synchronize erlcloud version across apps
2023-04-07 17:26:53 +03:00
07ae50a54b
fix(ft-s3): fix release settings
2023-04-07 17:25:23 +03:00
3ffa01e160
feat(ft-s3): fix logging level
2023-04-07 17:25:23 +03:00
ca22f11161
feat(ft-s3): fix logging
2023-04-07 17:25:23 +03:00
9bb72ee020
feat(ft-s3): fix behaviour specification
2023-04-07 17:25:23 +03:00
43f9737420
feat(ft-s3): integrate list API
2023-04-07 17:25:23 +03:00
5ac3543a76
feat(ft-s3): integrate exporter configs
2023-04-07 17:25:22 +03:00
8361223648
feat(ft-s3): extract checksum verification
2023-04-07 17:25:22 +03:00
818a5cacf2
feat(ft-s3): add initial integration
2023-04-07 17:25:22 +03:00
31b441a46e
feat(s3): add S3 client application
2023-04-07 17:25:22 +03:00
28d87ca62d
fix(ft-fs): move default config to the backend impl
2023-04-07 17:25:22 +03:00
258fabbf8b
fix(fs-exp): reply with error when listing failed everywhere
2023-04-07 17:25:22 +03:00
11edfc1c6a
fix(ft): comment out some clauses as "unreachable"
2023-04-07 17:25:22 +03:00
bef5cc9c0f
fix(fs-fold): avoid folding through symlinked directories
...
Also a testsuite that verifies multilevel fold behaviour.
2023-04-07 17:25:22 +03:00
6ad7ce55d2
fix(ft-conf): separate local segments storage settings
...
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
45e3b62dc4
refactor(ft): prefer plain `files` over `exports` in generic APIs
...
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
c24c7eca34
fix(ft-test): unbreak testcase by inhibiting local fs storage GC
2023-04-07 17:25:22 +03:00
23cd78b8d6
refactor(ft-asm): turn state data record into a map
...
Which should be more future-proof.
2023-04-07 17:25:22 +03:00
b6b044f429
feat(ft): move out exporter concept into dedicated modules
2023-04-07 17:25:22 +03:00
54e54cc63d
feat(api): provide utility to qualify API URIs
2023-04-07 17:25:22 +03:00
64f15f1fdb
fix(ft-gc): ensure directories of complete transfers are GCed
2023-04-07 17:25:22 +03:00
2707b4500f
feat(ft-fs): ensure filsystem-safe handling of client input
...
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
9aec01e7a3
fix(ft-asm): use regular map for meta fragments
...
There's actually no need to use an ordered data structure at all.
2023-04-07 17:25:22 +03:00
4132f5a5fb
feat(ft): introduce exporter concept in local storage backend
...
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.
Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
2880c8f4eb
fix(ft): unwrap error details when reader fails to start
2023-04-07 17:25:22 +03:00
45f00e14a9
feat(ft): ensure that clientid is always binary
...
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
2023-04-07 17:25:22 +03:00
4f2600b9f1
feat(ft-gc): treat all transfer as incomplete
...
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
0d39546080
feat(wdgraph): add `fold/3` which folds over graph edges
2023-04-07 17:25:22 +03:00
0c821cd3bd
test(ft): inject configs through hocon subsystem
...
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
bcd2099ce1
fix(fs-gc): make deletion empty transfer directories safer
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
15dc7c3e84
fix(test): fix node helper usage
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
e9f98adca2
test: make content_gen emit meta with each chunk
...
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
788e76ed2d
test: make assembly proptests waste less memory
2023-04-07 17:25:22 +03:00
8e6f960c09
refactor(ft): employ `emqx_wdgraph` for coverage computation
...
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
b189ee463c
feat: add weighted directional graph ADT with shortest path
...
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
2023-04-07 17:25:22 +03:00
130601376a
perf(ft-asm): express coverage through shortest path on graph
...
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
0c84fc28b0
perf(asm-ft): tradeoff optimality for computational complexity
...
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
97cfdf8eef
test(ft-asm): add property tests for file assembly
2023-04-07 17:25:22 +03:00
0af7e2a002
perf(ft-asm): optimize away remote segments identical to local
2023-04-07 17:25:22 +03:00
Ilya Averyanov