92ff2b7d6e
feat(ft): make timeouts configurable
2023-04-14 01:58:14 +03:00
37a520d797
fix(i18n): relocate i18n files to rel/i18n
2023-04-10 16:02:34 +03:00
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
...
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
b2fd2dcbc1
fix(ft): make configs user friendlier
2023-04-07 17:36:47 +03:00
7eeba32619
fix(s3): fix typings
2023-04-07 17:36:47 +03:00
52f3189779
fix(ft): fix default ft config
2023-04-07 17:36:47 +03:00
bd7250cb13
fix(s3): fix hash pool type
2023-04-07 17:36:47 +03:00
4a144044b7
fix(ft): set default ft config in tests
2023-04-07 17:36:47 +03:00
918bdcae7d
fix(ft): fix schema descriptions
2023-04-07 17:36:47 +03:00
8b13d249eb
fix(ft): fix detection of ee apps in mix
2023-04-07 17:36:47 +03:00
7bcb60a84a
fix(ft): fix config update tests
2023-04-07 17:36:47 +03:00
f6461fe287
fix(ft): fix typings
2023-04-07 17:36:47 +03:00
2f1970adbc
fix(ft): make ee schema aware of ee fields
2023-04-07 17:36:47 +03:00
be99242e32
fix(ft-s3): fix review comments
2023-04-07 17:36:47 +03:00
d7a85242de
fix(ft-s3): fix s3 listing
2023-04-07 17:36:47 +03:00
07ae50a54b
fix(ft-s3): fix release settings
2023-04-07 17:25:23 +03:00
3ffa01e160
feat(ft-s3): fix logging level
2023-04-07 17:25:23 +03:00
ca22f11161
feat(ft-s3): fix logging
2023-04-07 17:25:23 +03:00
9bb72ee020
feat(ft-s3): fix behaviour specification
2023-04-07 17:25:23 +03:00
43f9737420
feat(ft-s3): integrate list API
2023-04-07 17:25:23 +03:00
5ac3543a76
feat(ft-s3): integrate exporter configs
2023-04-07 17:25:22 +03:00
8361223648
feat(ft-s3): extract checksum verification
2023-04-07 17:25:22 +03:00
818a5cacf2
feat(ft-s3): add initial integration
2023-04-07 17:25:22 +03:00
28d87ca62d
fix(ft-fs): move default config to the backend impl
2023-04-07 17:25:22 +03:00
258fabbf8b
fix(fs-exp): reply with error when listing failed everywhere
2023-04-07 17:25:22 +03:00
11edfc1c6a
fix(ft): comment out some clauses as "unreachable"
2023-04-07 17:25:22 +03:00
bef5cc9c0f
fix(fs-fold): avoid folding through symlinked directories
...
Also a testsuite that verifies multilevel fold behaviour.
2023-04-07 17:25:22 +03:00
6ad7ce55d2
fix(ft-conf): separate local segments storage settings
...
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
45e3b62dc4
refactor(ft): prefer plain `files` over `exports` in generic APIs
...
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
c24c7eca34
fix(ft-test): unbreak testcase by inhibiting local fs storage GC
2023-04-07 17:25:22 +03:00
23cd78b8d6
refactor(ft-asm): turn state data record into a map
...
Which should be more future-proof.
2023-04-07 17:25:22 +03:00
b6b044f429
feat(ft): move out exporter concept into dedicated modules
2023-04-07 17:25:22 +03:00
64f15f1fdb
fix(ft-gc): ensure directories of complete transfers are GCed
2023-04-07 17:25:22 +03:00
2707b4500f
feat(ft-fs): ensure filsystem-safe handling of client input
...
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
9aec01e7a3
fix(ft-asm): use regular map for meta fragments
...
There's actually no need to use an ordered data structure at all.
2023-04-07 17:25:22 +03:00
4132f5a5fb
feat(ft): introduce exporter concept in local storage backend
...
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.
Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
2880c8f4eb
fix(ft): unwrap error details when reader fails to start
2023-04-07 17:25:22 +03:00
45f00e14a9
feat(ft): ensure that clientid is always binary
...
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
2023-04-07 17:25:22 +03:00
4f2600b9f1
feat(ft-gc): treat all transfer as incomplete
...
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
0c821cd3bd
test(ft): inject configs through hocon subsystem
...
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
bcd2099ce1
fix(fs-gc): make deletion empty transfer directories safer
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
15dc7c3e84
fix(test): fix node helper usage
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
e9f98adca2
test: make content_gen emit meta with each chunk
...
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
788e76ed2d
test: make assembly proptests waste less memory
2023-04-07 17:25:22 +03:00
8e6f960c09
refactor(ft): employ `emqx_wdgraph` for coverage computation
...
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
130601376a
perf(ft-asm): express coverage through shortest path on graph
...
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
Ilya Averyanov