Commit Graph

24 Commits

Author SHA1 Message Date
zhongwencool e9c8446d57 feat: upgrade erlfmt to support maybe syntax 2024-01-31 20:06:49 +08:00
Ilya Averyanov b0d4a22aa8 chore(ft): refactor async reply mechanism 2023-08-27 21:54:23 +03:00
firest b08102269a refactor(calendar): refactor datetime-related code and remove redundant 2023-08-16 10:55:46 +08:00
Andrew Mayorov 34793c5ed0 feat(ft-conf): preprocess TLS configuration on updates
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
2023-07-07 19:15:42 +03:00
Andrew Mayorov ae97d516b8
fix(ft): add `enable` flag to every backend
And tolerate multiple configured backends, as long as only one of
them is enabled.
2023-06-08 14:54:13 +03:00
Andrew Mayorov 0293b54211
fix(ft): respect checksum in `fin` packets 2023-05-31 18:50:38 +03:00
Andrew Mayorov a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
Ilya Averyanov 9381c895bb fix(ft): fix dialyzer issues 2023-05-09 23:32:16 +05:00
Andrew Mayorov 5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
2023-05-09 23:32:16 +05:00
Ilya Averyanov cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
Andrew Mayorov ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov 5efd590ca4
feat(ft): properly propagate config updates
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
2023-04-24 21:51:54 +03:00
Ilya Averyanov f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Andrew Mayorov 45e3b62dc4 refactor(ft): prefer plain `files` over `exports` in generic APIs
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
Andrew Mayorov b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov 4132f5a5fb feat(ft): introduce exporter concept in local storage backend
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Andrew Mayorov f896fefa59 feat(ft): make storage backend fully async-aware
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
Ilya Averyanov 0aefd4a8c7 feat(ft): add streaming of file content when downloading 2023-04-07 17:25:21 +03:00
Ilya Averyanov 04e5378bda feat(ft): add API 2023-04-07 17:25:21 +03:00
Andrew Mayorov 92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Ilya Averyanov b4a42a447c feat(ft): removed replicated data 2023-04-07 17:25:21 +03:00
Ilya Averyanov 72e3eee6c9 feat(ft): add config & backend behaviour 2023-04-07 17:25:21 +03:00