yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,818 Commits 393 Branches 669 Tags 74 MiB
75cceffa06
Commit Graph

7097 Commits

Author SHA1 Message Date
Ivan Dyachkov
bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov
9c9f39d0f7
feat(resman): also move out metrics collection for debugging 
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
 2023-04-12 16:14:42 +03:00
JianBo He
d16b5c40d6
Merge pull request #10356 from HJianBo/async-batch-cassa 
feat: support async and batch callback for cassandra connector
 2023-04-12 20:59:36 +08:00
JianBo He
9774ad2cdd
chore: update apps/emqx_gateway/README.md 2023-04-12 20:54:08 +08:00
Zhongwen Deng
45254c5936 chore: add changelog for create trace 2023-04-12 17:59:14 +08:00
Stefan Strigler
f668ad7b9e
Merge pull request #10336 from sstrigler/EMQX-8507-rule-engine-need-a-new-api-to-crud-rule-engines-setting-configs 
feat: add `/rule_engine` API endpoint
 2023-04-12 11:44:42 +02:00
Zhongwen Deng
f161399bcc feat: deprecated trace config 2023-04-12 17:07:45 +08:00
lafirest
a49eea5838
Merge pull request #10355 from lafirest/fix/auto_sub_doc 
chore: update README for auto subscribe
 2023-04-12 15:47:58 +08:00
JianBo He
80f964bdbe
chore: update apps/emqx_gateway/README.md 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-12 10:44:37 +08:00
Thales Macedo Garitezi
914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50 
feat: implement schema registry for 5.0 (avro)
 2023-04-11 16:46:27 -03:00
Stefan Strigler
e6f8682c47 fix: ensure we don't return 'rules' in rule_engine 2023-04-11 15:42:56 +02:00
ieQu1
bc463a1b1c
Merge pull request #10343 from ieQu1/emqx-machine-readme 
doc(emqx_machine): Add readme
 2023-04-11 15:15:32 +02:00
Stefan Strigler
8ef36f29ce fix: add 'rule_engine' as possible tag() value for spec 2023-04-11 12:03:42 +02:00
Stefan Strigler
b48fb17f4a fix: CHECK_PARAMS macro defines unused var 2023-04-11 12:03:42 +02:00
JianBo He
945c6bc757 docs: update gateway README 2023-04-11 17:26:33 +08:00
Andrew Mayorov
e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
JianBo He
aee85dc328 chore: fix flaky tests 2023-04-11 14:49:07 +08:00
JianBo He
d501c1ee9c chore: update changes 2023-04-11 14:04:41 +08:00
Ivan Dyachkov
a711ce6b93 chore: bump version to e5.0.2 2023-04-10 18:10:31 +02:00
Andrew Mayorov
37a520d797
fix(i18n): relocate i18n files to rel/i18n 2023-04-10 16:02:34 +03:00
firest
5a58dfc3a4 chore: update README for auto subscribe 2023-04-10 14:39:13 +08:00
Zhongwen Deng
c68909767a feat: more straightforward log for force_shutdown reason 2023-04-10 11:21:07 +08:00
ieQu1
c2ca9089ca docs(emqx_machine): Add readme 2023-04-08 13:22:52 +02:00
JianBo He
e186477531
Merge pull request #10278 from HJianBo/refactor-gw-dir 
Refactor gateway application dirs
 2023-04-08 09:42:14 +08:00
Andrew Mayorov
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format 
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
 2023-04-07 22:20:20 +03:00
Andrew Mayorov
0d86ef8d3a
fix(ft-s3): use AWS4 signed urls for S3 export URIs 
This will ensure that the S3 export URIs are valid in all AWS regions.
 2023-04-07 22:15:01 +03:00
Ilya Averyanov
b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
52f3189779 fix(ft): fix default ft config 2023-04-07 17:36:47 +03:00
Ilya Averyanov
eae2478678 fix(ft): add descriptions for missing ft schema fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov
4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov
8b13d249eb fix(ft): fix detection of ee apps in mix 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7bcb60a84a fix(ft): fix config update tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
6e50b168a0 fix(ft): add empty strings for zh translations 2023-04-07 17:36:47 +03:00
Ilya Averyanov
f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
18aa39f892 fix(ft): add emqx_ft_schema to ee schemas 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d333187c5a docs(s3): update readme 2023-04-07 17:36:47 +03:00
Ilya Averyanov
820e06d756 fix(ft-s3): make controllable use of max_retries and request_timeout arguments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7d13862da5 fix(ft-s3): use uploader specific headers only for object creation 2023-04-07 17:36:47 +03:00
Ilya Averyanov
c7865e5eae fix(ft-s3): fix dependency dependencies 2023-04-07 17:36:45 +03:00
Ilya Averyanov
4fbabe5a76 fix(ft-s3): fix atom and variable naming 2023-04-07 17:27:01 +03:00
Ilya Averyanov
07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov
3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov
ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov
9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov
43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov
5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov
8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov
818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Ilya Averyanov
31b441a46e feat(s3): add S3 client application 2023-04-07 17:25:22 +03:00
Andrew Mayorov
28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov
258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov
11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories 
Also a testsuite that verifies multilevel fold behaviour.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
6ad7ce55d2 fix(ft-conf): separate local segments storage settings 
To make things less ambiguous.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45e3b62dc4 refactor(ft): prefer plain files over exports in generic APIs 
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov
23cd78b8d6 refactor(ft-asm): turn state data record into a map 
Which should be more future-proof.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov
54e54cc63d feat(api): provide utility to qualify API URIs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
9aec01e7a3 fix(ft-asm): use regular map for meta fragments 
There's actually no need to use an ordered data structure at all.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4132f5a5fb feat(ft): introduce exporter concept in local storage backend 
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45f00e14a9 feat(ft): ensure that clientid is always binary 
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0d39546080 feat(wdgraph): add fold/3 which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c821cd3bd test(ft): inject configs through hocon subsystem 
So that relevant parts of config would be initialized with defaults.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov
715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e9f98adca2 test: make content_gen emit meta with each chunk 
Which will also contain total content size, to increase this tool's
utility.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
788e76ed2d test: make assembly proptests waste less memory 2023-04-07 17:25:22 +03:00
Andrew Mayorov
8e6f960c09 refactor(ft): employ emqx_wdgraph for coverage computation 
Also describe how coverage problem maps to shortest path problem.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b189ee463c feat: add weighted directional graph ADT with shortest path 
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
130601376a perf(ft-asm): express coverage through shortest path on graph 
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity 
Through squashing segments table into consecutive "runs".
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
97cfdf8eef test(ft-asm): add property tests for file assembly 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c feat(ft): add segment checksum validation 
Also downgrade validation errors to mere info messages.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5014cc15f4 test(ft): do not rely on fs backend implementation details 2023-04-07 17:25:22 +03:00
Andrew Mayorov
93865a79e9 fix(ft): disallow empty fileids 2023-04-07 17:25:22 +03:00
Andrew Mayorov
75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov
7ff253e80d test(ft): attempt to avoid gen_rpc config reset hacks 2023-04-07 17:25:22 +03:00
Andrew Mayorov
a3f47641f3 test(ft): describe the complex test case with comments 
In order to make the test client model and behavior clearer.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
10d4c4305a feat(maybe): add basic tests for the new module 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2cf2a2d952 fix(ft): make fin packet parser stricter + safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
58115715dd fix(ft): require final size in fin packet 
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov
228bf1a0ce chore(ft): fix typing issues 2023-04-07 17:25:22 +03:00
Ilya Averyanov
2c9cd1397d chore(ft): fix ?tp calls 2023-04-07 17:25:22 +03:00
Ilya Averyanov
1c61165a91 fix(ft): fix emqx_cm takover 2023-04-07 17:25:22 +03:00
Ilya Averyanov
c6b3f2c2ca feat(ft): add schema descriptions 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bc0a15afd7 feat(ft): fix test for CI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
16fa55e19a feat(ft): add API tests 2023-04-07 17:25:21 +03:00
Andrew Mayorov
6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov
f896fefa59 feat(ft): make storage backend fully async-aware 
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
2cdf486bf4 feat(ft): simplify responder mechanism 
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
 2023-04-07 17:25:21 +03:00
Ilya Averyanov
f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
2e889f4ac7 feat(ft): add emqx_ft tests and fixes 2023-04-07 17:25:21 +03:00
Ilya Averyanov
1d48a97fd2 feat(ft): fix remote reader handling of gen_rpc errors 2023-04-07 17:25:21 +03:00
Ilya Averyanov
c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
eae3dc7b4b feat(ft): update BPAPI 2023-04-07 17:25:21 +03:00
Ilya Averyanov
bcfa22f343 fix(ft): use correct supervison strategy for emqx_ft_sup 2023-04-07 17:25:21 +03:00
Ilya Averyanov
9e4a37a398 fix(ft): fix typespecs 2023-04-07 17:25:21 +03:00
Ilya Averyanov
836ec213c9 feat(ft): add responder tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
8038a3fd4a feat(ft): add tests for remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov
b7d0bad970 feat(ft): improve remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov
0aefd4a8c7 feat(ft): add streaming of file content when downloading 2023-04-07 17:25:21 +03:00
Ilya Averyanov
197ce32669 feat(ft): add proxy module for emqx_ft_storage_fs 2023-04-07 17:25:21 +03:00
Ilya Averyanov
04e5378bda feat(ft): add API 2023-04-07 17:25:21 +03:00
Andrew Mayorov
92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Andrew Mayorov
7ed06b0a2a feat(ft-fs): allow to list all transfers in storage 
This is rather simplistic and thus, temporary solution.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
429eeaf029 feat(ft-fs): make list / read more generic 
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
1308fa0e6b fix(ft-fs): put fragments into separate directories 
In order to avoid potential filename collisions.
 2023-04-07 17:25:21 +03:00
Andrew Mayorov
8298236908 refactor(ft): bring back userdata to filemeta schema 2023-04-07 17:25:21 +03:00
Ilya Averyanov
b4a42a447c feat(ft): removed replicated data 2023-04-07 17:25:21 +03:00
Ilya Averyanov
72e3eee6c9 feat(ft): add config & backend behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov
97b831a160 fix(ft-fs): add missing read_segment/5 + fix atomic write 2023-04-07 17:25:21 +03:00
Andrew Mayorov
14b2a1013b fix(ft-asm): follow proper segment fragment type 2023-04-07 17:25:21 +03:00
Andrew Mayorov
1fedae8a16 fix(ft-asm): ensure module follows statem behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov
7b77e96ab9 test(ft): add some basic assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov
cbff2e2309 feat(ft): improve robustness of asynchronous acks 
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
 2023-04-07 17:25:21 +03:00
Ilya Averyanov
d36ca18bff feat(ft): tie file transfer frontend and backend together 2023-04-07 17:25:21 +03:00
Andrew Mayorov
81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00
Ilya Averyanov
aaaef30be6 feat(ft): add file transfer app and bootstrap replicated ft data structure 2023-04-07 17:25:21 +03:00
Ilya Averyanov
d976943f99 feat(ft): add basic hooks 2023-04-07 17:25:21 +03:00
JianBo He
502cc2b8b8 chore: fix common tests 2023-04-07 15:34:16 +08:00
Thales Macedo Garitezi
7c05304ff4 chore: bump version to e5.0.2-rc.6 2023-04-06 16:53:55 -03:00
Thales Macedo Garitezi
33100ecca6 feat: implement schema registry for 5.0 (avro) 
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
 2023-04-06 16:28:45 -03:00
Thales Macedo Garitezi
4c24b08244 fix(rule_action): fix metrics for bridges returning async_return 
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
 2023-04-06 16:00:01 -03:00
Zaiming (Stone) Shi
11a7770d16 chore: bump version to e5.0.2-rc.5 2023-04-06 11:57:39 +02:00
JimMoen
bbb3fdb49e
fix: make emqx_api_lib compatible 2023-04-06 17:03:39 +08:00
Kjell Winblad
1938882f16
Merge pull request #10318 from kjellwinblad/kjell/feat/rule_engine_from_clause_support_both_string_types 
feat(rule engine sql): enable both ' and " strings in FROM clause
 2023-04-06 09:19:44 +02:00
JimMoen
ab807fba51
Merge pull request #10329 from kjellwinblad/kjell/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent 
fix: create consistent interface 'with_node' for API access
 2023-04-06 14:47:42 +08:00
JianBo He
702ecbcb6e Merge remote-tracking branch 'upstream/master' into refactor-gw-dir 2023-04-06 10:44:59 +08:00
Stefan Strigler
b799af1f71 fix: don't create virtual type "rule_engine_api" 2023-04-05 16:46:19 +02:00
Stefan Strigler
9cfe9cc709 fix: don't allow rules to be set from /rule_engine 2023-04-05 16:28:54 +02:00
Serge Tupchii
9810c9f7e3 refactor(rule_engine): test 'ok' results in one dedicated function 2023-04-05 16:55:43 +03:00
Stefan Strigler
5cd8865a93 fix: deprecate and hide jq_implementation_module 2023-04-05 15:34:42 +02:00
Stefan Strigler
aea870f319 feat: add /rule_engine API endpoint 2023-04-05 15:19:31 +02:00
Stefan Strigler
f3446c48f7
Merge pull request #10315 from sstrigler/EMQX-8945-crash-for-bad-param-value-to-mqtt-delayed-messages-api-call 
EMQX 8945 crash for bad param value to mqtt delayed messages api call
 2023-04-05 13:44:22 +02:00
JianBo He
a3262486e5 chore: delete all LICENSE files under gateway dirs 2023-04-05 15:25:08 +08:00
Stefan Strigler
a39312e892
Merge pull request #10292 from sstrigler/EMQX-9253-readme-refine-emqx-management 
EMQX Management README
 2023-04-05 09:00:34 +02:00
Zaiming (Stone) Shi
a9bf633e03
Merge pull request #10320 from zmstone/0403-sync-release-50-back-to-master 
0403 sync release 50 back to master
 2023-04-04 23:31:24 +02:00
Thales Macedo Garitezi
5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50 
feat(bridges): enable async query mode for all bridges with buffer workers
 2023-04-04 17:10:46 -03:00
Ivan Dyachkov
9d1a16aae1 feat: add emqx_rpc:multicall_on_running 
also move emqx:is_running multicall to emqx_proto_v2:are_running
 2023-04-04 20:40:47 +02:00
Ivan Dyachkov
3d7ceb01a0 fix(mgmt): fix stats api by applying filter to running_nodes 2023-04-04 19:58:43 +02:00
Ivan Dyachkov
c20da5ffa6 fix(emqx_dashboard): fix monitor_current api 2023-04-04 19:58:43 +02:00
Stefan Strigler
04626ce9cc fix: create consistent interface 'with_node' for API access 2023-04-04 16:54:14 +02:00
Zaiming (Stone) Shi
68c15ffd48 Merge remote-tracking branch 'origin/release-50' into 0403-sync-release-50-back-to-master 2023-04-04 16:42:58 +02:00
Zaiming (Stone) Shi
d25db3ace4 chore: bump version to e5.0.2-rc.4 2023-04-04 16:38:25 +02:00
Zaiming (Stone) Shi
9e17064e47
Merge pull request #10325 from zmstone/0404-delay-config-change-replay-until-handler-is-ready 
0404 delay config change replay until handler is ready
 2023-04-04 16:37:35 +02:00
SergeTupchiy
877b828d4a
Merge pull request #10327 from SergeTupchiy/EMQX-8786-fix-unknown-counter-inc-on-unrecoverable-err 
fix(rule_engine): don't increment unknown counter on unrecoverable er…
 2023-04-04 16:51:34 +03:00
Zaiming (Stone) Shi
5925ff07c2 test(emqx_cluster_rpc): fix test cases 2023-04-04 15:20:30 +02:00
Zaiming (Stone) Shi
eeb7b32bc8
Merge pull request #10317 from zmstone/0403-refactor-hide-listener-level-authentication 
0403 refactor hide listener level authentication
 2023-04-04 15:04:35 +02:00
Serge Tupchii
aca65ca2d4 fix(rule_engine): don't increment unknown counter on unrecoverable errors 
Closes: EMQX-8786
 2023-04-04 15:59:53 +03:00
Zaiming (Stone) Shi
196ca43fbb fix(emqx_conf_app): call the right API to retrieve core nodes 2023-04-04 14:19:48 +02:00
Thales Macedo Garitezi
0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval 
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
 2023-04-04 08:58:36 -03:00
Zaiming (Stone) Shi
8fd9dd741e fix(emqx_conf_app): wait for tables ready beofre starting apps 2023-04-04 13:50:50 +02:00
Kjell Winblad
0e66eb5f3f feat(rule engine sql): enable both ' and " strings in FROM clause 
This commit upgrades the rulesql dependency to version 1.5 instead of
1.4. The difference between these two versions is that strings surrounded
by ' and " are supported in FROM clauses in version 1.5, but in version
1.4, only strings surrounded by " are supported.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9243
 2023-04-04 10:36:48 +02:00
Zaiming (Stone) Shi
b16c516e6b refactor: rename cluster_rpc_handler to cluster_rpc_cleaner 
this reflects what is actually does
 2023-04-04 10:28:47 +02:00
firest
6b2419998d chore: bump emqx_authn version 2023-04-04 16:06:23 +08:00
firest
11b3264251 fix: redact the password to ****** in API examples 
fix #10222
 2023-04-04 16:02:45 +08:00
Thales Macedo Garitezi
a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50 
test(peer): increase init and startup timeout for peer nodes
 2023-04-03 15:46:13 -03:00
Zaiming (Stone) Shi
429b3d9efd Merge remote-tracking branch 'origin/master' into 0403-sync-release-50-back-to-master 2023-04-03 20:32:33 +02:00
Zaiming (Stone) Shi
2f2e8b8218 chore: bump version to e5.0.2-rc.3 2023-04-03 20:23:31 +02:00
Thales Macedo Garitezi
06048ae4ff
Merge pull request #10316 from thalesmg/flaky-crl-emqtt-test-v50 
test(crl): fix flaky test (v5.0)
 2023-04-03 15:12:36 -03:00
Thales Macedo Garitezi
f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers 
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
 2023-04-03 14:49:51 -03:00
Thales Macedo Garitezi
8b5a717a1f test(peer): increase init and startup timeout for peer nodes 
Attempt to stabilize tests that use cluster nodes.
 2023-04-03 13:20:22 -03:00
Zaiming (Stone) Shi
2d6ca69ffb refactor: no support for listener level authentication for now 2023-04-03 16:49:35 +02:00
Zaiming (Stone) Shi
e978d86c86 chore: add doc_lift for authorization.sources 
doc_lift is to make the doc render application to lift
this field to the root level and force the field's doc
to refernec it instead of expanding the structs in a nested way
 2023-04-03 16:49:35 +02:00
Thales Macedo Garitezi
ed25ee6fec test(crl): fix flaky test (v5.0) 2023-04-03 10:58:31 -03:00
Stefan Strigler
0efa9c7a11 fix: pretty format error responses 2023-04-03 15:48:52 +02:00
Stefan Strigler
c1cb5357e1 fix: enable schema check 2023-04-03 15:48:33 +02:00
Thales Macedo Garitezi
ec1871ffde test(janitor): catch each callback invocation 2023-04-03 10:20:19 -03:00
JimMoen
53712e6146
fix: running nodes should not include replica nodes 2023-04-03 20:01:37 +08:00
Zaiming (Stone) Shi
5f6d318cf0 fix(i18n): fix missing docs for gateway configs 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi
36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
JianBo He
205e97fdca chore(gw): update README files 2023-04-03 14:30:41 +08:00
zhongwencool
d63680cf25
Merge pull request #10307 from emqx/release-50 
Sync release-50 back to master
 2023-04-02 11:36:41 +08:00
Zaiming (Stone) Shi
b17548f382
Merge pull request #10265 from HJianBo/improve-some-options-desc 
docs: improve some mqtt options description
 2023-04-01 16:43:15 +02:00
JianBo He
b2d018f249 chore: fix test cases 2023-04-01 18:51:39 +08:00
JianBo He
3a3879f99f chore: update gateways deps 2023-04-01 12:59:43 +08:00
JianBo He
9577beaa4e chore: update rebar.conf in emqx_exproto 2023-04-01 12:24:09 +08:00
JianBo He
b24ff9bc6e test(gateway): refine all test cases 2023-04-01 12:24:09 +08:00
JianBo He
a70545b64a chore: split out exproto gateway 2023-04-01 12:24:09 +08:00
JianBo He
b58ce09658 chore: split out lwm2m 2023-04-01 12:24:09 +08:00
JianBo He
40c413ac05 chore: fix dialyzer warnings 2023-04-01 12:24:09 +08:00
JianBo He
0b6c5c4c91 refactor: split out emqx_coap application 2023-04-01 12:24:09 +08:00
JianBo He
786f030958 test: move tests into splited gateway dirs 2023-04-01 12:24:09 +08:00
JianBo He
159bcf329c refactor: make Stomp and MQTT-SN gateway as an independent apps 2023-04-01 12:24:09 +08:00
Thales Macedo Garitezi
53d760ec69
Merge pull request #10300 from thalesmg/fix-plugin-upload-v50 
fix(plugins): create directory before uploading
 2023-03-31 16:28:18 -03:00
Thales Macedo Garitezi
246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50 
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
 2023-03-31 16:25:26 -03:00
Thales Macedo Garitezi
5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-9422
 2023-03-31 11:33:15 -03:00
Zaiming (Stone) Shi
9bbca07943
Merge pull request #10298 from emqx/release-50 
Merge release-50 back to master
 2023-03-31 15:35:51 +02:00
Zaiming (Stone) Shi
abf0329b60 test(emqx_banned_SUITE): fix flaky test case 2023-03-31 13:03:00 +02:00
Zaiming (Stone) Shi
bcde52383b docs: fix max batch size desc 2023-03-31 12:35:27 +02:00
William Yang
d9bec63733
Merge pull request #10284 from qzhuyan/docs/william/shared-subs 
chore(docs): update shared_subscription_strategy
 2023-03-31 09:44:12 +02:00
Stefan Strigler
7e31e60e90
style: fix wording 2023-03-31 09:19:48 +02:00
Stefan Strigler
1ff96f5314
style: fix wording 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-31 09:18:39 +02:00
Thales Macedo Garitezi
a2b82dae69 fix(plugins): create directory before uploading 
Fixes https://emqx.atlassian.net/browse/EMQX-9434

Fixes https://github.com/emqx/emqx-elixir-plugin/issues/23
 2023-03-30 17:51:16 -03:00
Zaiming (Stone) Shi
36043dd651
Merge pull request #10286 from zmstone/0330-exit-with-non-zero-code-if-conf-init-failed 
0330 exit with non zero code if conf init failed
 2023-03-30 17:41:14 +02:00
Stefan Strigler
7c597bfaa9 docs: give a better idea of what this application is supposed to do 2023-03-30 16:57:16 +02:00
Stefan Strigler
87a262edff docs: add README for EMQX Dashboard 2023-03-30 16:53:27 +02:00
Thales Macedo Garitezi
632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
 2023-03-30 11:51:36 -03:00
Thales Macedo Garitezi
ad28d8418d
Merge pull request #10274 from thalesmg/fix-subs-api-egress-mqtt-v50 
fix: handle subscription list api when there are `emqx:subscribe` internal subscriptions
 2023-03-30 11:16:31 -03:00
lafirest
0af226dd5b
Merge pull request #10276 from lafirest/fix/banned_api 
fix(i18n): fix the description error about the default value of `until`
 2023-03-30 20:26:32 +08:00
William Yang
3c52faa55e chore(docs): update shared_subscription_strategy 2023-03-30 13:32:45 +02:00
William Yang
65ef9c9086
Merge pull request #10077 from qzhuyan/dev/william/quic-cert-password 
feat(quic): support TLS password protected keyfile
 2023-03-30 12:59:50 +02:00
Zaiming (Stone) Shi
494e4b639a chore: exit with non-zero status if config initialization failed 2023-03-30 12:52:51 +02:00
firest
f794666c7f fix(i18n): fix the description error about the default value of until 2023-03-30 17:12:45 +08:00
JianBo He
1ff8ecf604 chore: fix typos 2023-03-30 15:10:10 +08:00
JianBo He
3c495f8fd2
chore: update apps/emqx/i18n/emqx_schema_i18n.conf 2023-03-30 08:44:28 +08:00
JianBo He
758e610059
chore: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-03-30 08:42:08 +08:00
Thales Macedo Garitezi
806ee716ee fix(redis): remove required meta from schema 
Since there's a default value, this field shouldn't be marked as
required.
 2023-03-29 16:43:52 -03:00
Thales Macedo Garitezi
916ce99fd5 fix: handle subscription list api when there are emqx:subscribe subscriptions 2023-03-29 14:56:40 -03:00
Thales Macedo Garitezi
5089449203
Merge pull request #10260 from thalesmg/minor-refactor-basic-usage-rule-rv50 
refactor(rule_engine): use more helper functions
 2023-03-29 09:44:23 -03:00
Zaiming (Stone) Shi
80eb9d7542
Merge pull request #10252 from emqx/release-50 
0327 merge release-50 to master
 2023-03-29 12:33:17 +02:00
Kjell Winblad
8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell 
fix: add inflight window setting to the clickhouse bridge
 2023-03-29 09:38:24 +02:00
JianBo He
fee4ff6a07
Merge pull request #10257 from HJianBo/gw-lwm2m-fixes 
Fixed the issue where `auto_observe` was not working in LwM2M Gateway
 2023-03-29 15:20:46 +08:00
JianBo He
27aa3f49aa docs: improve some mqtt options description 2023-03-29 15:02:01 +08:00
Zaiming (Stone) Shi
11c120f87c chore: bump emqx app vsn 2023-03-28 21:19:03 +02:00
Zaiming (Stone) Shi
4b428f7a29 chore: bump version to e5.0.2-rc.2 2023-03-28 21:18:21 +02:00
Thales Macedo Garitezi
64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi
d126c7dc62 refactor(rule_engine): use more helper functions 
Follow up to
https://github.com/emqx/emqx/pull/10251#discussion_r1150710899 and
https://github.com/emqx/emqx/pull/10251#discussion_r1150720420
 2023-03-28 13:46:56 -03:00
William Yang
ecc2cd1a94 test: password cert for SSL listener 2023-03-28 17:15:59 +02:00
Thales Macedo Garitezi
1824e7efcc fix(rule_engine): count referenced bridges in from clauses as dependencies (rv5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-9325

Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.

When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
 2023-03-28 10:31:10 -03:00
Zaiming (Stone) Shi
f5e9d3eab2 test: use valid config for jq function timeout 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
3f9ed08308 fix(emqx_schema): ocsp field use 'importance' to replace 'hidden' 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
d07987288a chore: add some example annotations for config importance level 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
5a19b6f73e chore: pin hocon 0.38.0 (with config importance level annotation) 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
486352eb6f fix(swagger): do not generate scheam for hidden args 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
fb29f8035b feat: make possible to dump full schema (including hidden fields) 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
c302405408 fix: fail fast if not able to load config 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi
813ccd7df1 chore: hide node.perf configs 
these configs are rarely needed by regular users
and if they are to be changed, it should be done by
someone who really understands what they mean
 2023-03-28 14:29:24 +02:00
JianBo He
6db5a4872a test(lwm2m): cover auto_observe mechanism 2023-03-28 18:31:20 +08:00
JianBo He
6750618d9a fix(lwm2m): ignore unknown ObjectId to auto observe 2023-03-28 17:44:33 +08:00
Xinyu Liu
c607e4ed92
Merge pull request #10243 from terry-xiaoyu/remove-peercert-from-conninfo50 
chore: remove peercert from the state of connection processes
 2023-03-28 17:22:57 +08:00