yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,418 Commits 393 Branches 684 Tags 74 MiB
52f3189779
Commit Graph

6841 Commits

Author SHA1 Message Date
Ilya Averyanov
52f3189779 fix(ft): fix default ft config 2023-04-07 17:36:47 +03:00
Ilya Averyanov
eae2478678 fix(ft): add descriptions for missing ft schema fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov
4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov
8b13d249eb fix(ft): fix detection of ee apps in mix 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7bcb60a84a fix(ft): fix config update tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
6e50b168a0 fix(ft): add empty strings for zh translations 2023-04-07 17:36:47 +03:00
Ilya Averyanov
f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
18aa39f892 fix(ft): add emqx_ft_schema to ee schemas 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d333187c5a docs(s3): update readme 2023-04-07 17:36:47 +03:00
Ilya Averyanov
820e06d756 fix(ft-s3): make controllable use of max_retries and request_timeout arguments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7d13862da5 fix(ft-s3): use uploader specific headers only for object creation 2023-04-07 17:36:47 +03:00
Ilya Averyanov
c7865e5eae fix(ft-s3): fix dependency dependencies 2023-04-07 17:36:45 +03:00
Ilya Averyanov
4fbabe5a76 fix(ft-s3): fix atom and variable naming 2023-04-07 17:27:01 +03:00
Ilya Averyanov
07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov
3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov
ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov
9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov
43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov
5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov
8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov
818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Ilya Averyanov
31b441a46e feat(s3): add S3 client application 2023-04-07 17:25:22 +03:00
Andrew Mayorov
28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov
258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov
11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories 
Also a testsuite that verifies multilevel fold behaviour.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
6ad7ce55d2 fix(ft-conf): separate local segments storage settings 
To make things less ambiguous.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45e3b62dc4 refactor(ft): prefer plain files over exports in generic APIs 
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov
23cd78b8d6 refactor(ft-asm): turn state data record into a map 
Which should be more future-proof.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov
54e54cc63d feat(api): provide utility to qualify API URIs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
9aec01e7a3 fix(ft-asm): use regular map for meta fragments 
There's actually no need to use an ordered data structure at all.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4132f5a5fb feat(ft): introduce exporter concept in local storage backend 
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45f00e14a9 feat(ft): ensure that clientid is always binary 
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0d39546080 feat(wdgraph): add fold/3 which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c821cd3bd test(ft): inject configs through hocon subsystem 
So that relevant parts of config would be initialized with defaults.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00