6b688d6646
fix(ft): anticipate repeated `kickoff`s + fix testcase
2023-05-10 17:35:53 +03:00
42f5433aaf
Merge pull request #10655 from savonarola/0510-idempotent-fin
...
Make FT fin command idempotent
2023-05-10 11:39:43 +05:00
244188982b
fix(ft): add missing translations
2023-05-09 23:32:17 +05:00
9381c895bb
fix(ft): fix dialyzer issues
2023-05-09 23:32:16 +05:00
079b8e9476
fix(ft): silence warnings when some root is not yet `mkdir`ed
2023-05-09 23:32:16 +05:00
5e5f854ce1
feat(ft-conf): simplify schema of storage / exporter backends
...
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
2023-05-09 23:32:16 +05:00
8ac881a140
chore(ft): handle multiple/concurrent fins more gracefully
2023-05-09 23:31:39 +05:00
cff7788b2e
chore(ft): add s3 exporter tests
2023-05-05 11:55:09 +03:00
a9866fede4
feat(ft-api): support paging in S3 storage exporter
2023-04-28 13:49:17 +03:00
ed3756ea09
feat(ft-api): add paging support through cursors
2023-04-28 13:49:15 +03:00
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks
...
In order to support paging over filesystem contents, to serve REST
APIs effectively.
2023-04-28 13:47:50 +03:00
811e449357
feat(ft-conf): provide global killswitch
2023-04-28 10:54:21 +03:00
a420c92d28
fix(ft): fix typing issue
2023-04-24 21:51:54 +03:00
5efd590ca4
feat(ft): properly propagate config updates
...
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.
Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
69c4ba2a62
feat(ft): use new utils application
2023-04-22 22:50:19 +03:00
04523b3f81
fix(ft): restrict max filename length in transfers
...
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
2023-04-19 18:48:40 +03:00
92ff2b7d6e
feat(ft): make timeouts configurable
2023-04-14 01:58:14 +03:00
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
...
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
b2fd2dcbc1
fix(ft): make configs user friendlier
2023-04-07 17:36:47 +03:00
7eeba32619
fix(s3): fix typings
2023-04-07 17:36:47 +03:00
bd7250cb13
fix(s3): fix hash pool type
2023-04-07 17:36:47 +03:00
918bdcae7d
fix(ft): fix schema descriptions
2023-04-07 17:36:47 +03:00
f6461fe287
fix(ft): fix typings
2023-04-07 17:36:47 +03:00
2f1970adbc
fix(ft): make ee schema aware of ee fields
2023-04-07 17:36:47 +03:00
be99242e32
fix(ft-s3): fix review comments
2023-04-07 17:36:47 +03:00
d7a85242de
fix(ft-s3): fix s3 listing
2023-04-07 17:36:47 +03:00
07ae50a54b
fix(ft-s3): fix release settings
2023-04-07 17:25:23 +03:00
3ffa01e160
feat(ft-s3): fix logging level
2023-04-07 17:25:23 +03:00
ca22f11161
feat(ft-s3): fix logging
2023-04-07 17:25:23 +03:00
9bb72ee020
feat(ft-s3): fix behaviour specification
2023-04-07 17:25:23 +03:00
43f9737420
feat(ft-s3): integrate list API
2023-04-07 17:25:23 +03:00
5ac3543a76
feat(ft-s3): integrate exporter configs
2023-04-07 17:25:22 +03:00
8361223648
feat(ft-s3): extract checksum verification
2023-04-07 17:25:22 +03:00
818a5cacf2
feat(ft-s3): add initial integration
2023-04-07 17:25:22 +03:00
28d87ca62d
fix(ft-fs): move default config to the backend impl
2023-04-07 17:25:22 +03:00
258fabbf8b
fix(fs-exp): reply with error when listing failed everywhere
2023-04-07 17:25:22 +03:00
11edfc1c6a
fix(ft): comment out some clauses as "unreachable"
2023-04-07 17:25:22 +03:00
bef5cc9c0f
fix(fs-fold): avoid folding through symlinked directories
...
Also a testsuite that verifies multilevel fold behaviour.
2023-04-07 17:25:22 +03:00
6ad7ce55d2
fix(ft-conf): separate local segments storage settings
...
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
45e3b62dc4
refactor(ft): prefer plain `files` over `exports` in generic APIs
...
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
c24c7eca34
fix(ft-test): unbreak testcase by inhibiting local fs storage GC
2023-04-07 17:25:22 +03:00
23cd78b8d6
refactor(ft-asm): turn state data record into a map
...
Which should be more future-proof.
2023-04-07 17:25:22 +03:00
b6b044f429
feat(ft): move out exporter concept into dedicated modules
2023-04-07 17:25:22 +03:00
64f15f1fdb
fix(ft-gc): ensure directories of complete transfers are GCed
2023-04-07 17:25:22 +03:00
2707b4500f
feat(ft-fs): ensure filsystem-safe handling of client input
...
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
9aec01e7a3
fix(ft-asm): use regular map for meta fragments
...
There's actually no need to use an ordered data structure at all.
2023-04-07 17:25:22 +03:00
4132f5a5fb
feat(ft): introduce exporter concept in local storage backend
...
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.
Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
2880c8f4eb
fix(ft): unwrap error details when reader fails to start
2023-04-07 17:25:22 +03:00
45f00e14a9
feat(ft): ensure that clientid is always binary
...
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
2023-04-07 17:25:22 +03:00
4f2600b9f1
feat(ft-gc): treat all transfer as incomplete
...
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
bcd2099ce1
fix(fs-gc): make deletion empty transfer directories safer
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
8e6f960c09
refactor(ft): employ `emqx_wdgraph` for coverage computation
...
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
130601376a
perf(ft-asm): express coverage through shortest path on graph
...
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
0c84fc28b0
perf(asm-ft): tradeoff optimality for computational complexity
...
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
0af7e2a002
perf(ft-asm): optimize away remote segments identical to local
2023-04-07 17:25:22 +03:00
15d967459c
feat(ft): add segment checksum validation
...
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
5998961f9f
fix(ft): log errors where they might get lost
2023-04-07 17:25:22 +03:00
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
75070102ec
fix(ft): improve typespecs
2023-04-07 17:25:22 +03:00
2b925aa60b
fix(ft): drop unrelated TODO
2023-04-07 17:25:22 +03:00
2cf2a2d952
fix(ft): make `fin` packet parser stricter + safer
2023-04-07 17:25:22 +03:00
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
228bf1a0ce
chore(ft): fix typing issues
2023-04-07 17:25:22 +03:00
2c9cd1397d
chore(ft): fix `?tp` calls
2023-04-07 17:25:22 +03:00
c6b3f2c2ca
feat(ft): add schema descriptions
2023-04-07 17:25:21 +03:00
6d9f780313
chore: remove unused code
2023-04-07 17:25:21 +03:00
f896fefa59
feat(ft): make storage backend fully async-aware
...
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
2cdf486bf4
feat(ft): simplify responder mechanism
...
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
f6a0598f27
feat(ft): add file transfer tests
2023-04-07 17:25:21 +03:00
2e889f4ac7
feat(ft): add emqx_ft tests and fixes
2023-04-07 17:25:21 +03:00
1d48a97fd2
feat(ft): fix remote reader handling of gen_rpc errors
2023-04-07 17:25:21 +03:00
c44fe92ef1
feat(ft): add assembler tests
2023-04-07 17:25:21 +03:00
bcfa22f343
fix(ft): use correct supervison strategy for emqx_ft_sup
2023-04-07 17:25:21 +03:00
9e4a37a398
fix(ft): fix typespecs
2023-04-07 17:25:21 +03:00
836ec213c9
feat(ft): add responder tests
2023-04-07 17:25:21 +03:00
8038a3fd4a
feat(ft): add tests for remote reader
2023-04-07 17:25:21 +03:00
b7d0bad970
feat(ft): improve remote reader
2023-04-07 17:25:21 +03:00
0aefd4a8c7
feat(ft): add streaming of file content when downloading
2023-04-07 17:25:21 +03:00
197ce32669
feat(ft): add proxy module for emqx_ft_storage_fs
2023-04-07 17:25:21 +03:00
04e5378bda
feat(ft): add API
2023-04-07 17:25:21 +03:00
92670bfe3d
feat(ft): add fs storage bpapi and use it in assembler
2023-04-07 17:25:21 +03:00
7ed06b0a2a
feat(ft-fs): allow to list all transfers in storage
...
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
429eeaf029
feat(ft-fs): make `list` / `read` more generic
...
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
1308fa0e6b
fix(ft-fs): put fragments into separate directories
...
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
8298236908
refactor(ft): bring back userdata to filemeta schema
2023-04-07 17:25:21 +03:00
b4a42a447c
feat(ft): removed replicated data
2023-04-07 17:25:21 +03:00
72e3eee6c9
feat(ft): add config & backend behaviour
2023-04-07 17:25:21 +03:00
97b831a160
fix(ft-fs): add missing `read_segment/5` + fix atomic write
2023-04-07 17:25:21 +03:00
14b2a1013b
fix(ft-asm): follow proper `segment` fragment type
2023-04-07 17:25:21 +03:00
1fedae8a16
fix(ft-asm): ensure module follows statem behaviour
2023-04-07 17:25:21 +03:00
cbff2e2309
feat(ft): improve robustness of asynchronous acks
...
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
2023-04-07 17:25:21 +03:00
d36ca18bff
feat(ft): tie file transfer frontend and backend together
2023-04-07 17:25:21 +03:00
81e04ce93a
feat(ft): introduce simple filesystem storage backend + assembler
2023-04-07 17:25:21 +03:00
aaaef30be6
feat(ft): add file transfer app and bootstrap replicated ft data structure
2023-04-07 17:25:21 +03:00
Andrew Mayorov