98ce7e8bbd
test(ft): ensure no division by zero in coverage proptest
2023-05-26 21:53:01 +03:00
d36c2c2928
fix(ft-gc): ensure GC of already complete transfers
2023-05-26 12:59:20 +03:00
f4047d3946
test(ft): add testcase for nasty tranfer filenames
...
* Emoji
* Chinese characters
2023-05-23 20:46:47 +03:00
2dbf84479c
fix(ft): handle wider class of jiffy decode errors
...
With malformed cursors in the File listing API.
Fixes EMQX-9965
2023-05-23 20:00:00 +03:00
cb14a3e08b
fix(ft): handle empty filepath in fs exporter API
...
Fixes EMQX-9973
2023-05-23 19:58:05 +03:00
966f748551
test(ft): make proptest less aggressive
...
So that the chance of getting huge coverage and as a result, high
chance of running out of memory, is reduced.
2023-05-22 10:45:27 +03:00
6b688d6646
fix(ft): anticipate repeated `kickoff`s + fix testcase
2023-05-10 17:35:53 +03:00
42f5433aaf
Merge pull request #10655 from savonarola/0510-idempotent-fin
...
Make FT fin command idempotent
2023-05-10 11:39:43 +05:00
45875bfee5
fix(ft): update s3 exporter tests to the new config
2023-05-09 23:32:17 +05:00
5e5f854ce1
feat(ft-conf): simplify schema of storage / exporter backends
...
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
2023-05-09 23:32:16 +05:00
8ac881a140
chore(ft): handle multiple/concurrent fins more gracefully
2023-05-09 23:31:39 +05:00
cff7788b2e
chore(ft): add s3 exporter tests
2023-05-05 11:55:09 +03:00
bd9f129bb8
test(ft-api): also run testcases under cluster setup
2023-04-28 13:50:23 +03:00
75cceffa06
fix(ft-test): rename testcases for consistency
2023-04-28 13:49:17 +03:00
ed3756ea09
feat(ft-api): add paging support through cursors
2023-04-28 13:49:15 +03:00
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks
...
In order to support paging over filesystem contents, to serve REST
APIs effectively.
2023-04-28 13:47:50 +03:00
811e449357
feat(ft-conf): provide global killswitch
2023-04-28 10:54:21 +03:00
5efd590ca4
feat(ft): properly propagate config updates
...
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.
Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
69c4ba2a62
feat(ft): use new utils application
2023-04-22 22:50:19 +03:00
04523b3f81
fix(ft): restrict max filename length in transfers
...
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
2023-04-19 18:48:40 +03:00
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
...
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
4a144044b7
fix(ft): set default ft config in tests
2023-04-07 17:36:47 +03:00
7bcb60a84a
fix(ft): fix config update tests
2023-04-07 17:36:47 +03:00
bef5cc9c0f
fix(fs-fold): avoid folding through symlinked directories
...
Also a testsuite that verifies multilevel fold behaviour.
2023-04-07 17:25:22 +03:00
6ad7ce55d2
fix(ft-conf): separate local segments storage settings
...
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
45e3b62dc4
refactor(ft): prefer plain `files` over `exports` in generic APIs
...
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
c24c7eca34
fix(ft-test): unbreak testcase by inhibiting local fs storage GC
2023-04-07 17:25:22 +03:00
b6b044f429
feat(ft): move out exporter concept into dedicated modules
2023-04-07 17:25:22 +03:00
64f15f1fdb
fix(ft-gc): ensure directories of complete transfers are GCed
2023-04-07 17:25:22 +03:00
2707b4500f
feat(ft-fs): ensure filsystem-safe handling of client input
...
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
4132f5a5fb
feat(ft): introduce exporter concept in local storage backend
...
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.
Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
4f2600b9f1
feat(ft-gc): treat all transfer as incomplete
...
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
0c821cd3bd
test(ft): inject configs through hocon subsystem
...
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
15dc7c3e84
fix(test): fix node helper usage
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
e9f98adca2
test: make content_gen emit meta with each chunk
...
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
788e76ed2d
test: make assembly proptests waste less memory
2023-04-07 17:25:22 +03:00
97cfdf8eef
test(ft-asm): add property tests for file assembly
2023-04-07 17:25:22 +03:00
15d967459c
feat(ft): add segment checksum validation
...
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
5014cc15f4
test(ft): do not rely on fs backend implementation details
2023-04-07 17:25:22 +03:00
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
7ff253e80d
test(ft): attempt to avoid `gen_rpc` config reset hacks
2023-04-07 17:25:22 +03:00
a3f47641f3
test(ft): describe the complex test case with comments
...
In order to make the test client model and behavior clearer.
2023-04-07 17:25:22 +03:00
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
c073914f75
chore(ft): fix gen_rpc flakyness
2023-04-07 17:25:22 +03:00
bc0a15afd7
feat(ft): fix test for CI
2023-04-07 17:25:21 +03:00
16fa55e19a
feat(ft): add API tests
2023-04-07 17:25:21 +03:00
Andrew Mayorov