yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
16,875 Commits 393 Branches 669 Tags 74 MiB
31cf05ec66
Commit Graph

127 Commits

Author SHA1 Message Date
Ilya Averyanov
56c81c2c25 feat(ft-api): bump app versions 2023-07-07 19:17:22 +03:00
Ilya Averyanov
e0353ab750 fix(ft-api): update FT API tests to use emqx_cth_suite 2023-07-07 19:15:42 +03:00
Ilya Averyanov
fde506838a fix(ft-api): implement import config behaviour 2023-07-07 19:15:42 +03:00
Andrew Mayorov
a2b03716be feat(ft-api): provide configuration API 
To configure `emqx_ft` during the runtime.
 2023-07-07 19:15:42 +03:00
Andrew Mayorov
34793c5ed0 feat(ft-conf): preprocess TLS configuration on updates 
Specifically, reify keys and certificates into files in the file system
and update the configuration to point to those files.
 2023-07-07 19:15:42 +03:00
Andrew Mayorov
27ab9c62d8 fix(ft): unload conf hooks in prep_stop 
In order to avoid situations when the root supervisor is stopped
already.
 2023-07-07 19:15:42 +03:00
Andrew Mayorov
f76f3b77d7
test: add tooling to make testruns more isolated and predictable 
Also showcase their usage in some of the existing testsuites.
 2023-07-04 12:53:39 +02:00
Zaiming (Stone) Shi
7cf8a6c892 chore: bump app vsns 2023-06-21 16:36:51 +02:00
Andrew Mayorov
ae5846be29
fix(ft-api): make schema tags look nicer in the docs 2023-06-19 13:35:21 +02:00
Andrew Mayorov
52354bf58a
fix(ft-fs): use emqx:running_nodes() as default cluster view 
When iterating over complete exports cluster-wide.
 2023-06-14 22:54:56 +03:00
Zaiming (Stone) Shi
97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Zaiming (Stone) Shi
7d2ea3dc76 docs: refine some desc 2023-06-08 13:58:31 +02:00
Andrew Mayorov
ce8cc05cc8
chore(ftschema): mention translate is for tests purpose only 2023-06-08 14:54:14 +03:00
Andrew Mayorov
e3c16be3c4
fix(ftschema): ensure translate follow emqx_config behavior 
So that atom keys won't unexpectedly get to converters / validators.
 2023-06-08 14:54:14 +03:00
Andrew Mayorov
ae97d516b8
fix(ft): add enable flag to every backend 
And tolerate multiple configured backends, as long as only one of
them is enabled.
 2023-06-08 14:54:13 +03:00
Andrew Mayorov
dcd59e4f1b
fix(ft): set more conservative filename length limit 
Otherwise, local fs exporter will have a hard time preserving the
filemeta, because its filename is even 13 bytes longer.
 2023-06-06 10:40:20 +03:00
Andrew Mayorov
69b98c1830
fix(ft): ensure temp filenames are under 255 bytes long 2023-06-06 10:38:31 +03:00
Thales Macedo Garitezi
46393343e2 chore: use timeout_duration types for timer fields 
Fixes https://emqx.atlassian.net/browse/EMQX-10020
 2023-06-05 11:46:38 -03:00
Andrew Mayorov
0293b54211
fix(ft): respect checksum in fin packets 2023-05-31 18:50:38 +03:00
Zaiming (Stone) Shi
d2ce7c22fd chore: bump app vsn, core app 'emqx' is now at 5.1 2023-05-30 11:32:18 +02:00
JimMoen
8569fac6eb
fix: make static_check happy 2023-05-30 14:00:20 +08:00
JimMoen
909d348570
chore: fix spec typo 2023-05-30 11:31:37 +08:00
Zaiming (Stone) Shi
36e268c933 chore: bump app versions 2023-05-26 16:05:37 +02:00
Andrew Mayorov
3d428a2e0e
fix(ft-s3): rely on asynchronous upload abort 
So that upload aborts would not block assemblers. This should not
affect the expected behavior since S3 API usually allows having
concurrent uploads, plus we couldn't really guarantee successful aborts
anyway.
 2023-05-26 12:59:20 +03:00
Andrew Mayorov
cc3275b389
fix(ft): use conservative timeouts for S3 uploader calls 2023-05-26 12:59:20 +03:00
Andrew Mayorov
d36c2c2928
fix(ft-gc): ensure GC of already complete transfers 2023-05-26 12:59:20 +03:00
Andrew Mayorov
2dbf84479c
fix(ft): handle wider class of jiffy decode errors 
With malformed cursors in the File listing API.

Fixes EMQX-9965
 2023-05-23 20:00:00 +03:00
Andrew Mayorov
cb14a3e08b
fix(ft): handle empty filepath in fs exporter API 
Fixes EMQX-9973
 2023-05-23 19:58:05 +03:00
Andrew Mayorov
a7595ff468
chore: fixup copyright years 2023-05-16 16:45:07 +03:00
Andrew Mayorov
6b688d6646
fix(ft): anticipate repeated kickoffs + fix testcase 2023-05-10 17:35:53 +03:00
Ilya Averyanov
42f5433aaf
Merge pull request #10655 from savonarola/0510-idempotent-fin 
Make FT fin command idempotent
 2023-05-10 11:39:43 +05:00
Ilya Averyanov
244188982b fix(ft): add missing translations 2023-05-09 23:32:17 +05:00
Ilya Averyanov
9381c895bb fix(ft): fix dialyzer issues 2023-05-09 23:32:16 +05:00
Andrew Mayorov
079b8e9476 fix(ft): silence warnings when some root is not yet mkdired 2023-05-09 23:32:16 +05:00
Andrew Mayorov
5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends 
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
 2023-05-09 23:32:16 +05:00
Ilya Averyanov
8ac881a140 chore(ft): handle multiple/concurrent fins more gracefully 2023-05-09 23:31:39 +05:00
Ilya Averyanov
cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
Andrew Mayorov
a9866fede4
feat(ft-api): support paging in S3 storage exporter 2023-04-28 13:49:17 +03:00
Andrew Mayorov
ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks 
In order to support paging over filesystem contents, to serve REST
APIs effectively.
 2023-04-28 13:47:50 +03:00
Andrew Mayorov
811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Andrew Mayorov
a420c92d28
fix(ft): fix typing issue 2023-04-24 21:51:54 +03:00
Andrew Mayorov
5efd590ca4
feat(ft): properly propagate config updates 
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
 2023-04-24 21:51:54 +03:00
Ilya Averyanov
69c4ba2a62 feat(ft): use new utils application 2023-04-22 22:50:19 +03:00
Andrew Mayorov
04523b3f81
fix(ft): restrict max filename length in transfers 
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
 2023-04-19 18:48:40 +03:00
Ilya Averyanov
92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
Andrew Mayorov
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format 
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
 2023-04-07 22:20:20 +03:00
Ilya Averyanov
b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00