yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,227 Commits 393 Branches 669 Tags 74 MiB
244188982b
Commit Graph

115 Commits

Author SHA1 Message Date
Ilya Averyanov
244188982b fix(ft): add missing translations 2023-05-09 23:32:17 +05:00
Ilya Averyanov
45875bfee5 fix(ft): update s3 exporter tests to the new config 2023-05-09 23:32:17 +05:00
Ilya Averyanov
9381c895bb fix(ft): fix dialyzer issues 2023-05-09 23:32:16 +05:00
Andrew Mayorov
079b8e9476 fix(ft): silence warnings when some root is not yet mkdired 2023-05-09 23:32:16 +05:00
Andrew Mayorov
5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends 
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
 2023-05-09 23:32:16 +05:00
Ilya Averyanov
cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
Andrew Mayorov
bd9f129bb8
test(ft-api): also run testcases under cluster setup 2023-04-28 13:50:23 +03:00
Andrew Mayorov
a9866fede4
feat(ft-api): support paging in S3 storage exporter 2023-04-28 13:49:17 +03:00
Andrew Mayorov
75cceffa06
fix(ft-test): rename testcases for consistency 2023-04-28 13:49:17 +03:00
Andrew Mayorov
ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks 
In order to support paging over filesystem contents, to serve REST
APIs effectively.
 2023-04-28 13:47:50 +03:00
Andrew Mayorov
811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Andrew Mayorov
a420c92d28
fix(ft): fix typing issue 2023-04-24 21:51:54 +03:00
Andrew Mayorov
5efd590ca4
feat(ft): properly propagate config updates 
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
 2023-04-24 21:51:54 +03:00
Ilya Averyanov
69c4ba2a62 feat(ft): use new utils application 2023-04-22 22:50:19 +03:00
Andrew Mayorov
04523b3f81
fix(ft): restrict max filename length in transfers 
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
 2023-04-19 18:48:40 +03:00
Ilya Averyanov
92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
Andrew Mayorov
37a520d797
fix(i18n): relocate i18n files to rel/i18n 2023-04-10 16:02:34 +03:00
Andrew Mayorov
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format 
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
 2023-04-07 22:20:20 +03:00
Ilya Averyanov
b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
52f3189779 fix(ft): fix default ft config 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov
4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov
8b13d249eb fix(ft): fix detection of ee apps in mix 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7bcb60a84a fix(ft): fix config update tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov
07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov
3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov
ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov
9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov
43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov
5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov
8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov
818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Andrew Mayorov
28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov
258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov
11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories 
Also a testsuite that verifies multilevel fold behaviour.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
6ad7ce55d2 fix(ft-conf): separate local segments storage settings 
To make things less ambiguous.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45e3b62dc4 refactor(ft): prefer plain files over exports in generic APIs 
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov
23cd78b8d6 refactor(ft-asm): turn state data record into a map 
Which should be more future-proof.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov
64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00