ddbb8560fa
fix(dialyzer): batch 2
2023-12-08 17:59:55 +01:00
534c9bdc13
feat(ft): add additional operation status report channel
2023-10-11 19:51:36 +03:00
b08102269a
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-16 10:55:46 +08:00
34793c5ed0
feat(ft-conf): preprocess TLS configuration on updates
...
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
2023-07-07 19:15:42 +03:00
ae97d516b8
fix(ft): add `enable` flag to every backend
...
And tolerate multiple configured backends, as long as only one of
them is enabled.
2023-06-08 14:54:13 +03:00
69b98c1830
fix(ft): ensure temp filenames are under 255 bytes long
2023-06-06 10:38:31 +03:00
0293b54211
fix(ft): respect checksum in `fin` packets
2023-05-31 18:50:38 +03:00
d36c2c2928
fix(ft-gc): ensure GC of already complete transfers
2023-05-26 12:59:20 +03:00
8ac881a140
chore(ft): handle multiple/concurrent fins more gracefully
2023-05-09 23:31:39 +05:00
cff7788b2e
chore(ft): add s3 exporter tests
2023-05-05 11:55:09 +03:00
ed3756ea09
feat(ft-api): add paging support through cursors
2023-04-28 13:49:15 +03:00
a420c92d28
fix(ft): fix typing issue
2023-04-24 21:51:54 +03:00
5efd590ca4
feat(ft): properly propagate config updates
...
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.
Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
69c4ba2a62
feat(ft): use new utils application
2023-04-22 22:50:19 +03:00
818a5cacf2
feat(ft-s3): add initial integration
2023-04-07 17:25:22 +03:00
28d87ca62d
fix(ft-fs): move default config to the backend impl
2023-04-07 17:25:22 +03:00
6ad7ce55d2
fix(ft-conf): separate local segments storage settings
...
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
45e3b62dc4
refactor(ft): prefer plain `files` over `exports` in generic APIs
...
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
b6b044f429
feat(ft): move out exporter concept into dedicated modules
2023-04-07 17:25:22 +03:00
2707b4500f
feat(ft-fs): ensure filsystem-safe handling of client input
...
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
4132f5a5fb
feat(ft): introduce exporter concept in local storage backend
...
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.
Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
5998961f9f
fix(ft): log errors where they might get lost
2023-04-07 17:25:22 +03:00
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
75070102ec
fix(ft): improve typespecs
2023-04-07 17:25:22 +03:00
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
228bf1a0ce
chore(ft): fix typing issues
2023-04-07 17:25:22 +03:00
f896fefa59
feat(ft): make storage backend fully async-aware
...
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
f6a0598f27
feat(ft): add file transfer tests
2023-04-07 17:25:21 +03:00
2e889f4ac7
feat(ft): add emqx_ft tests and fixes
2023-04-07 17:25:21 +03:00
9e4a37a398
fix(ft): fix typespecs
2023-04-07 17:25:21 +03:00
0aefd4a8c7
feat(ft): add streaming of file content when downloading
2023-04-07 17:25:21 +03:00
197ce32669
feat(ft): add proxy module for emqx_ft_storage_fs
2023-04-07 17:25:21 +03:00
04e5378bda
feat(ft): add API
2023-04-07 17:25:21 +03:00
92670bfe3d
feat(ft): add fs storage bpapi and use it in assembler
2023-04-07 17:25:21 +03:00
7ed06b0a2a
feat(ft-fs): allow to list all transfers in storage
...
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
429eeaf029
feat(ft-fs): make `list` / `read` more generic
...
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
1308fa0e6b
fix(ft-fs): put fragments into separate directories
...
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
8298236908
refactor(ft): bring back userdata to filemeta schema
2023-04-07 17:25:21 +03:00
b4a42a447c
feat(ft): removed replicated data
2023-04-07 17:25:21 +03:00
72e3eee6c9
feat(ft): add config & backend behaviour
2023-04-07 17:25:21 +03:00
97b831a160
fix(ft-fs): add missing `read_segment/5` + fix atomic write
2023-04-07 17:25:21 +03:00
d36ca18bff
feat(ft): tie file transfer frontend and backend together
2023-04-07 17:25:21 +03:00
81e04ce93a
feat(ft): introduce simple filesystem storage backend + assembler
2023-04-07 17:25:21 +03:00
Zaiming (Stone) Shi