45f00e14a9
feat(ft): ensure that clientid is always binary
...
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
2023-04-07 17:25:22 +03:00
4f2600b9f1
feat(ft-gc): treat all transfer as incomplete
...
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
0d39546080
feat(wdgraph): add `fold/3` which folds over graph edges
2023-04-07 17:25:22 +03:00
0c821cd3bd
test(ft): inject configs through hocon subsystem
...
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
e1dc48fa2b
feat(fs-gc): wire gc up with emqx config
2023-04-07 17:25:22 +03:00
50c6eef2bc
fix(fs-gc): do not hide `enoent`s
...
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
bcd2099ce1
fix(fs-gc): make deletion empty transfer directories safer
2023-04-07 17:25:22 +03:00
344799f100
fix(fs-gc): add testcase on incomplete transfers GC
...
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
15dc7c3e84
fix(test): fix node helper usage
2023-04-07 17:25:22 +03:00
715816e67b
feat(ft): add GC logic and process for the FS storage backend
2023-04-07 17:25:22 +03:00
e9f98adca2
test: make content_gen emit meta with each chunk
...
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
788e76ed2d
test: make assembly proptests waste less memory
2023-04-07 17:25:22 +03:00
8e6f960c09
refactor(ft): employ `emqx_wdgraph` for coverage computation
...
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
b189ee463c
feat: add weighted directional graph ADT with shortest path
...
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
2023-04-07 17:25:22 +03:00
130601376a
perf(ft-asm): express coverage through shortest path on graph
...
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
0c84fc28b0
perf(asm-ft): tradeoff optimality for computational complexity
...
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
97cfdf8eef
test(ft-asm): add property tests for file assembly
2023-04-07 17:25:22 +03:00
0af7e2a002
perf(ft-asm): optimize away remote segments identical to local
2023-04-07 17:25:22 +03:00
15d967459c
feat(ft): add segment checksum validation
...
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
5998961f9f
fix(ft): log errors where they might get lost
2023-04-07 17:25:22 +03:00
5014cc15f4
test(ft): do not rely on fs backend implementation details
2023-04-07 17:25:22 +03:00
93865a79e9
fix(ft): disallow empty fileids
2023-04-07 17:25:22 +03:00
75070102ec
fix(ft): improve typespecs
2023-04-07 17:25:22 +03:00
2b925aa60b
fix(ft): drop unrelated TODO
2023-04-07 17:25:22 +03:00
7ff253e80d
test(ft): attempt to avoid `gen_rpc` config reset hacks
2023-04-07 17:25:22 +03:00
a3f47641f3
test(ft): describe the complex test case with comments
...
In order to make the test client model and behavior clearer.
2023-04-07 17:25:22 +03:00
10d4c4305a
feat(maybe): add basic tests for the new module
2023-04-07 17:25:22 +03:00
2cf2a2d952
fix(ft): make `fin` packet parser stricter + safer
2023-04-07 17:25:22 +03:00
58115715dd
fix(ft): require final size in `fin` packet
...
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.
Also synthesize a testcase to show why it's needed.
Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
c073914f75
chore(ft): fix gen_rpc flakyness
2023-04-07 17:25:22 +03:00
228bf1a0ce
chore(ft): fix typing issues
2023-04-07 17:25:22 +03:00
2c9cd1397d
chore(ft): fix `?tp` calls
2023-04-07 17:25:22 +03:00
1c61165a91
fix(ft): fix emqx_cm takover
2023-04-07 17:25:22 +03:00
c6b3f2c2ca
feat(ft): add schema descriptions
2023-04-07 17:25:21 +03:00
bc0a15afd7
feat(ft): fix test for CI
2023-04-07 17:25:21 +03:00
16fa55e19a
feat(ft): add API tests
2023-04-07 17:25:21 +03:00
6d9f780313
chore: remove unused code
2023-04-07 17:25:21 +03:00
f896fefa59
feat(ft): make storage backend fully async-aware
...
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
2cdf486bf4
feat(ft): simplify responder mechanism
...
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
f6a0598f27
feat(ft): add file transfer tests
2023-04-07 17:25:21 +03:00
2e889f4ac7
feat(ft): add emqx_ft tests and fixes
2023-04-07 17:25:21 +03:00
1d48a97fd2
feat(ft): fix remote reader handling of gen_rpc errors
2023-04-07 17:25:21 +03:00
c44fe92ef1
feat(ft): add assembler tests
2023-04-07 17:25:21 +03:00
eae3dc7b4b
feat(ft): update BPAPI
2023-04-07 17:25:21 +03:00
bcfa22f343
fix(ft): use correct supervison strategy for emqx_ft_sup
2023-04-07 17:25:21 +03:00
9e4a37a398
fix(ft): fix typespecs
2023-04-07 17:25:21 +03:00
836ec213c9
feat(ft): add responder tests
2023-04-07 17:25:21 +03:00
8038a3fd4a
feat(ft): add tests for remote reader
2023-04-07 17:25:21 +03:00
b7d0bad970
feat(ft): improve remote reader
2023-04-07 17:25:21 +03:00
0aefd4a8c7
feat(ft): add streaming of file content when downloading
2023-04-07 17:25:21 +03:00
197ce32669
feat(ft): add proxy module for emqx_ft_storage_fs
2023-04-07 17:25:21 +03:00
04e5378bda
feat(ft): add API
2023-04-07 17:25:21 +03:00
92670bfe3d
feat(ft): add fs storage bpapi and use it in assembler
2023-04-07 17:25:21 +03:00
7ed06b0a2a
feat(ft-fs): allow to list all transfers in storage
...
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
429eeaf029
feat(ft-fs): make `list` / `read` more generic
...
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
1308fa0e6b
fix(ft-fs): put fragments into separate directories
...
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
8298236908
refactor(ft): bring back userdata to filemeta schema
2023-04-07 17:25:21 +03:00
b4a42a447c
feat(ft): removed replicated data
2023-04-07 17:25:21 +03:00
72e3eee6c9
feat(ft): add config & backend behaviour
2023-04-07 17:25:21 +03:00
97b831a160
fix(ft-fs): add missing `read_segment/5` + fix atomic write
2023-04-07 17:25:21 +03:00
14b2a1013b
fix(ft-asm): follow proper `segment` fragment type
2023-04-07 17:25:21 +03:00
1fedae8a16
fix(ft-asm): ensure module follows statem behaviour
2023-04-07 17:25:21 +03:00
7b77e96ab9
test(ft): add some basic assembler tests
2023-04-07 17:25:21 +03:00
cbff2e2309
feat(ft): improve robustness of asynchronous acks
...
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
2023-04-07 17:25:21 +03:00
d36ca18bff
feat(ft): tie file transfer frontend and backend together
2023-04-07 17:25:21 +03:00
81e04ce93a
feat(ft): introduce simple filesystem storage backend + assembler
2023-04-07 17:25:21 +03:00
aaaef30be6
feat(ft): add file transfer app and bootstrap replicated ft data structure
2023-04-07 17:25:21 +03:00
d976943f99
feat(ft): add basic hooks
2023-04-07 17:25:21 +03:00
1938882f16
Merge pull request #10318 from kjellwinblad/kjell/feat/rule_engine_from_clause_support_both_string_types
...
feat(rule engine sql): enable both ' and " strings in FROM clause
2023-04-06 09:19:44 +02:00
9810c9f7e3
refactor(rule_engine): test 'ok' results in one dedicated function
2023-04-05 16:55:43 +03:00
f3446c48f7
Merge pull request #10315 from sstrigler/EMQX-8945-crash-for-bad-param-value-to-mqtt-delayed-messages-api-call
...
EMQX 8945 crash for bad param value to mqtt delayed messages api call
2023-04-05 13:44:22 +02:00
a39312e892
Merge pull request #10292 from sstrigler/EMQX-9253-readme-refine-emqx-management
...
EMQX Management README
2023-04-05 09:00:34 +02:00
a9bf633e03
Merge pull request #10320 from zmstone/0403-sync-release-50-back-to-master
...
0403 sync release 50 back to master
2023-04-04 23:31:24 +02:00
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
...
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
68c15ffd48
Merge remote-tracking branch 'origin/release-50' into 0403-sync-release-50-back-to-master
2023-04-04 16:42:58 +02:00
d25db3ace4
chore: bump version to e5.0.2-rc.4
2023-04-04 16:38:25 +02:00
9e17064e47
Merge pull request #10325 from zmstone/0404-delay-config-change-replay-until-handler-is-ready
...
0404 delay config change replay until handler is ready
2023-04-04 16:37:35 +02:00
877b828d4a
Merge pull request #10327 from SergeTupchiy/EMQX-8786-fix-unknown-counter-inc-on-unrecoverable-err
...
fix(rule_engine): don't increment unknown counter on unrecoverable er…
2023-04-04 16:51:34 +03:00
5925ff07c2
test(emqx_cluster_rpc): fix test cases
2023-04-04 15:20:30 +02:00
eeb7b32bc8
Merge pull request #10317 from zmstone/0403-refactor-hide-listener-level-authentication
...
0403 refactor hide listener level authentication
2023-04-04 15:04:35 +02:00
aca65ca2d4
fix(rule_engine): don't increment unknown counter on unrecoverable errors
...
Closes: EMQX-8786
2023-04-04 15:59:53 +03:00
196ca43fbb
fix(emqx_conf_app): call the right API to retrieve core nodes
2023-04-04 14:19:48 +02:00
0b6fd7fe14
fix(buffer_worker): check request timeout and health check interval
...
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`
Fixes https://emqx.atlassian.net/browse/EMQX-9099
Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`. This had the problem that,
with default values, `health_check_interval = request_timeout`. This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.
Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
8fd9dd741e
fix(emqx_conf_app): wait for tables ready beofre starting apps
2023-04-04 13:50:50 +02:00
0e66eb5f3f
feat(rule engine sql): enable both ' and " strings in FROM clause
...
This commit upgrades the rulesql dependency to version 1.5 instead of
1.4. The difference between these two versions is that strings surrounded
by ' and " are supported in FROM clauses in version 1.5, but in version
1.4, only strings surrounded by " are supported.
Fixes:
https://emqx.atlassian.net/browse/EMQX-9243
2023-04-04 10:36:48 +02:00
b16c516e6b
refactor: rename cluster_rpc_handler to cluster_rpc_cleaner
...
this reflects what is actually does
2023-04-04 10:28:47 +02:00
6b2419998d
chore: bump emqx_authn version
2023-04-04 16:06:23 +08:00
11b3264251
fix: redact the password to `******` in API examples
...
fix #10222
2023-04-04 16:02:45 +08:00
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
...
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
429b3d9efd
Merge remote-tracking branch 'origin/master' into 0403-sync-release-50-back-to-master
2023-04-03 20:32:33 +02:00
2f2e8b8218
chore: bump version to e5.0.2-rc.3
2023-04-03 20:23:31 +02:00
06048ae4ff
Merge pull request #10316 from thalesmg/flaky-crl-emqtt-test-v50
...
test(crl): fix flaky test (v5.0)
2023-04-03 15:12:36 -03:00
f3ffc02bff
feat(bridges): enable async query mode for all bridges with buffer workers
...
Fixes https://emqx.atlassian.net/browse/EMQX-9130
Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.
For bridges that currently only allow sync query modes, we should
allow them to be configured with async. That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
8b5a717a1f
test(peer): increase init and startup timeout for peer nodes
...
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
2d6ca69ffb
refactor: no support for listener level authentication for now
2023-04-03 16:49:35 +02:00
e978d86c86
chore: add doc_lift for authorization.sources
...
doc_lift is to make the doc render application to lift
this field to the root level and force the field's doc
to refernec it instead of expanding the structs in a nested way
2023-04-03 16:49:35 +02:00
ed25ee6fec
test(crl): fix flaky test (v5.0)
2023-04-03 10:58:31 -03:00
0efa9c7a11
fix: pretty format error responses
2023-04-03 15:48:52 +02:00
c1cb5357e1
fix: enable schema check
2023-04-03 15:48:33 +02:00
ec1871ffde
test(janitor): catch each callback invocation
2023-04-03 10:20:19 -03:00
Andrew Mayorov