yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,813 Commits 393 Branches 669 Tags 74 MiB
811e449357
Commit Graph

87 Commits

Author SHA1 Message Date
Andrew Mayorov
811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Andrew Mayorov
a420c92d28
fix(ft): fix typing issue 2023-04-24 21:51:54 +03:00
Andrew Mayorov
5efd590ca4
feat(ft): properly propagate config updates 
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
 2023-04-24 21:51:54 +03:00
Ilya Averyanov
69c4ba2a62 feat(ft): use new utils application 2023-04-22 22:50:19 +03:00
Andrew Mayorov
04523b3f81
fix(ft): restrict max filename length in transfers 
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
 2023-04-19 18:48:40 +03:00
Ilya Averyanov
92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
Andrew Mayorov
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format 
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
 2023-04-07 22:20:20 +03:00
Ilya Averyanov
b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov
918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov
f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov
07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov
3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov
ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov
9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov
43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov
5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov
8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov
818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Andrew Mayorov
28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov
258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov
11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories 
Also a testsuite that verifies multilevel fold behaviour.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
6ad7ce55d2 fix(ft-conf): separate local segments storage settings 
To make things less ambiguous.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45e3b62dc4 refactor(ft): prefer plain files over exports in generic APIs 
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov
23cd78b8d6 refactor(ft-asm): turn state data record into a map 
Which should be more future-proof.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov
64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
9aec01e7a3 fix(ft-asm): use regular map for meta fragments 
There's actually no need to use an ordered data structure at all.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4132f5a5fb feat(ft): introduce exporter concept in local storage backend 
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45f00e14a9 feat(ft): ensure that clientid is always binary 
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov
8e6f960c09 refactor(ft): employ emqx_wdgraph for coverage computation 
Also describe how coverage problem maps to shortest path problem.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
130601376a perf(ft-asm): express coverage through shortest path on graph 
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity 
Through squashing segments table into consecutive "runs".
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c feat(ft): add segment checksum validation 
Also downgrade validation errors to mere info messages.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00