Commit Graph

7376 Commits

Author SHA1 Message Date
zhongwencool 1e07f37ab0
Merge pull request #10354 from zhongwencool/straightforward-log-info
feat: more straightforward log for force_shutdown reason
2023-04-14 10:15:32 +08:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
JimMoen 8ceeafb0de
chore: fix file license in lib-ee 2023-04-14 09:56:59 +08:00
zhongwencool 0553facfca
Merge pull request #10391 from zhongwencool/04-13-hide-conf
feat: hide ex_hook/rewrite/topic_metric/persistent_session_store
2023-04-14 09:39:29 +08:00
zhongwencool 626e814e9a
Merge pull request #10385 from zhongwencool/hide-conf
feat: hide data items from configuration files and documentation.
2023-04-14 09:38:34 +08:00
zhongwencool 835f66a8d2
Merge pull request #10375 from zhongwencool/trace-conf
feat: deprecated trace config
2023-04-14 09:22:12 +08:00
Ilya Averyanov 92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
ieQu1 c5ad3c8f41 docs(emqx_machine): Improve README 2023-04-13 22:57:30 +02:00
Ilya Averyanov e22c1c01ec feat(s3): make acl optional 2023-04-13 19:47:12 +03:00
JianBo He 9f99a72594 chore: refine tests 2023-04-14 00:45:22 +08:00
Thales Macedo Garitezi 9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
JianBo He 1759dd1157
Merge pull request #10368 from HJianBo/improve-gateway-readme
docs: update gateway README
2023-04-14 00:14:01 +08:00
Kjell Winblad d3ccd8a65d feat: add date_to_unix_ts/3 function to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-9245
2023-04-13 16:14:03 +02:00
zhongwencool 2334917e35
chore: apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-04-13 21:59:26 +08:00
某文 58e31d5efd feat: hide ex_hook/rewrite/topic_metric/persistent_session_store/overload_protection 2023-04-13 21:52:35 +08:00
某文 c1163d1952 chore: share emqx_trace.hrl between code and test 2023-04-13 20:51:14 +08:00
某文 dd7dcfe373 fix: fix dialyzer warning 2023-04-13 20:04:03 +08:00
JianBo He c7054886b1 chore: refine test cases 2023-04-13 17:50:58 +08:00
JianBo He 4dd0080e82 chore: fix unreachable links 2023-04-13 17:19:47 +08:00
JianBo He 40e4419a62
chore: update apps/emqx_gateway/README.md
Co-authored-by: Stefan Strigler <stefan@strigler.de>
2023-04-13 16:18:46 +08:00
JianBo He aa9292e37e
chore: update apps/emqx_gateway/README.md
Co-authored-by: Stefan Strigler <stefan@strigler.de>
2023-04-13 16:18:34 +08:00
JianBo He 0a62d6c556 refactor(gw): rename all gateway application name 2023-04-13 14:53:47 +08:00
zhongwencool cf2f00c937
Merge pull request #10381 from zhongwencool/hiden-auto-subscribe-conf
feat: hiden auto_subscribe conf
2023-04-13 14:42:59 +08:00
Zhongwen Deng cc2beda37a feat: hiden rule_engine/bridge/authz/authn from doc/example 2023-04-13 14:42:21 +08:00
Zhongwen Deng 4ba4c6bf62 chore: only hiden root keys 2023-04-13 14:23:25 +08:00
Zhongwen Deng a70930fed0 feat: hiden auto_subscribe conf 2023-04-13 09:26:16 +08:00
Thales Macedo Garitezi 871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov 1e33600190 chore: bump version to v5.0.22 2023-04-12 17:33:53 +02:00
Ivan Dyachkov f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master
0411 sync release 50 back to master
2023-04-12 17:23:17 +02:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov 9c9f39d0f7
feat(resman): also move out metrics collection for debugging
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
2023-04-12 16:14:42 +03:00
JianBo He d16b5c40d6
Merge pull request #10356 from HJianBo/async-batch-cassa
feat: support async and batch callback for cassandra connector
2023-04-12 20:59:36 +08:00
JianBo He 9774ad2cdd
chore: update apps/emqx_gateway/README.md 2023-04-12 20:54:08 +08:00
Zhongwen Deng 45254c5936 chore: add changelog for create trace 2023-04-12 17:59:14 +08:00
Stefan Strigler f668ad7b9e
Merge pull request #10336 from sstrigler/EMQX-8507-rule-engine-need-a-new-api-to-crud-rule-engines-setting-configs
feat: add `/rule_engine` API endpoint
2023-04-12 11:44:42 +02:00
Zhongwen Deng f161399bcc feat: deprecated trace config 2023-04-12 17:07:45 +08:00
lafirest a49eea5838
Merge pull request #10355 from lafirest/fix/auto_sub_doc
chore: update README for auto subscribe
2023-04-12 15:47:58 +08:00
JianBo He 80f964bdbe
chore: update apps/emqx_gateway/README.md
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-04-12 10:44:37 +08:00
Thales Macedo Garitezi 914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50
feat: implement schema registry for 5.0 (avro)
2023-04-11 16:46:27 -03:00
Stefan Strigler e6f8682c47 fix: ensure we don't return 'rules' in rule_engine 2023-04-11 15:42:56 +02:00
ieQu1 bc463a1b1c
Merge pull request #10343 from ieQu1/emqx-machine-readme
doc(emqx_machine): Add readme
2023-04-11 15:15:32 +02:00
Stefan Strigler 8ef36f29ce fix: add 'rule_engine' as possible tag() value for spec 2023-04-11 12:03:42 +02:00
Stefan Strigler b48fb17f4a fix: CHECK_PARAMS macro defines unused var 2023-04-11 12:03:42 +02:00
JianBo He 945c6bc757 docs: update gateway README 2023-04-11 17:26:33 +08:00
Andrew Mayorov e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
JianBo He aee85dc328 chore: fix flaky tests 2023-04-11 14:49:07 +08:00
JianBo He d501c1ee9c chore: update changes 2023-04-11 14:04:41 +08:00
Ivan Dyachkov a711ce6b93 chore: bump version to e5.0.2 2023-04-10 18:10:31 +02:00
Andrew Mayorov 37a520d797
fix(i18n): relocate i18n files to rel/i18n 2023-04-10 16:02:34 +03:00
firest 5a58dfc3a4 chore: update README for auto subscribe 2023-04-10 14:39:13 +08:00
Zhongwen Deng c68909767a feat: more straightforward log for force_shutdown reason 2023-04-10 11:21:07 +08:00
ieQu1 c2ca9089ca docs(emqx_machine): Add readme 2023-04-08 13:22:52 +02:00
JianBo He e186477531
Merge pull request #10278 from HJianBo/refactor-gw-dir
Refactor gateway application dirs
2023-04-08 09:42:14 +08:00
Andrew Mayorov 8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
2023-04-07 22:20:20 +03:00
Andrew Mayorov 0d86ef8d3a
fix(ft-s3): use AWS4 signed urls for S3 export URIs
This will ensure that the S3 export URIs are valid in all AWS regions.
2023-04-07 22:15:01 +03:00
Ilya Averyanov b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov 7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov 52f3189779 fix(ft): fix default ft config 2023-04-07 17:36:47 +03:00
Ilya Averyanov eae2478678 fix(ft): add descriptions for missing ft schema fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov 4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov 918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov 8b13d249eb fix(ft): fix detection of ee apps in mix 2023-04-07 17:36:47 +03:00
Ilya Averyanov 7bcb60a84a fix(ft): fix config update tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov 6e50b168a0 fix(ft): add empty strings for zh translations 2023-04-07 17:36:47 +03:00
Ilya Averyanov f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov 2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov 18aa39f892 fix(ft): add emqx_ft_schema to ee schemas 2023-04-07 17:36:47 +03:00
Ilya Averyanov d333187c5a docs(s3): update readme 2023-04-07 17:36:47 +03:00
Ilya Averyanov 820e06d756 fix(ft-s3): make controllable use of max_retries and request_timeout arguments 2023-04-07 17:36:47 +03:00
Ilya Averyanov be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov 7d13862da5 fix(ft-s3): use uploader specific headers only for object creation 2023-04-07 17:36:47 +03:00
Ilya Averyanov c7865e5eae fix(ft-s3): fix dependency dependencies 2023-04-07 17:36:45 +03:00
Ilya Averyanov 4fbabe5a76 fix(ft-s3): fix atom and variable naming 2023-04-07 17:27:01 +03:00
Ilya Averyanov 07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov 3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov 9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov 43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov 5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov 8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov 818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Ilya Averyanov 31b441a46e feat(s3): add S3 client application 2023-04-07 17:25:22 +03:00
Andrew Mayorov 28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov 258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov 11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories
Also a testsuite that verifies multilevel fold behaviour.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 6ad7ce55d2 fix(ft-conf): separate local segments storage settings
To make things less ambiguous.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 45e3b62dc4 refactor(ft): prefer plain `files` over `exports` in generic APIs
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
2023-04-07 17:25:22 +03:00
Andrew Mayorov c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov 23cd78b8d6 refactor(ft-asm): turn state data record into a map
Which should be more future-proof.
2023-04-07 17:25:22 +03:00
Andrew Mayorov b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov 54e54cc63d feat(api): provide utility to qualify API URIs 2023-04-07 17:25:22 +03:00
Andrew Mayorov 64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov 2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 9aec01e7a3 fix(ft-asm): use regular map for meta fragments
There's actually no need to use an ordered data structure at all.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4132f5a5fb feat(ft): introduce exporter concept in local storage backend
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov 45f00e14a9 feat(ft): ensure that clientid is always binary
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
2023-04-07 17:25:22 +03:00
Andrew Mayorov 4f2600b9f1 feat(ft-gc): treat all transfer as incomplete
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0d39546080 feat(wdgraph): add `fold/3` which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov 0c821cd3bd test(ft): inject configs through hocon subsystem
So that relevant parts of config would be initialized with defaults.
2023-04-07 17:25:22 +03:00
Andrew Mayorov e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov 50c6eef2bc fix(fs-gc): do not hide `enoent`s
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
2023-04-07 17:25:22 +03:00
Andrew Mayorov bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov 344799f100 fix(fs-gc): add testcase on incomplete transfers GC
And fix logic that made GC delete incomplete segments prematurely.
2023-04-07 17:25:22 +03:00
Ilya Averyanov 15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov 715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov e9f98adca2 test: make content_gen emit meta with each chunk
Which will also contain total content size, to increase this tool's
utility.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 788e76ed2d test: make assembly proptests waste less memory 2023-04-07 17:25:22 +03:00
Andrew Mayorov 8e6f960c09 refactor(ft): employ `emqx_wdgraph` for coverage computation
Also describe how coverage problem maps to shortest path problem.
2023-04-07 17:25:22 +03:00
Andrew Mayorov b189ee463c feat: add weighted directional graph ADT with shortest path
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 130601376a perf(ft-asm): express coverage through shortest path on graph
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity
Through squashing segments table into consecutive "runs".
2023-04-07 17:25:22 +03:00
Andrew Mayorov 97cfdf8eef test(ft-asm): add property tests for file assembly 2023-04-07 17:25:22 +03:00
Andrew Mayorov 0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov 15d967459c feat(ft): add segment checksum validation
Also downgrade validation errors to mere info messages.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00
Andrew Mayorov 5014cc15f4 test(ft): do not rely on fs backend implementation details 2023-04-07 17:25:22 +03:00
Andrew Mayorov 93865a79e9 fix(ft): disallow empty fileids 2023-04-07 17:25:22 +03:00
Andrew Mayorov 75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov 2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov 7ff253e80d test(ft): attempt to avoid `gen_rpc` config reset hacks 2023-04-07 17:25:22 +03:00
Andrew Mayorov a3f47641f3 test(ft): describe the complex test case with comments
In order to make the test client model and behavior clearer.
2023-04-07 17:25:22 +03:00
Andrew Mayorov 10d4c4305a feat(maybe): add basic tests for the new module 2023-04-07 17:25:22 +03:00
Andrew Mayorov 2cf2a2d952 fix(ft): make `fin` packet parser stricter + safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov 58115715dd fix(ft): require final size in `fin` packet
Otherwise there are situations when it's not entirely clear if a
transfer is really ready to be assembled. Since the `size` field
in a filemeta is not required (and rightly so), we need client to
tell us the final transfer size at the end of the process.

Also synthesize a testcase to show why it's needed.

Also worth noting that right now `fin` packets require final size,
even if a client already told us the size through filemeta. The
latter is regarded as serving informational purposes only (which
means that, for example, it might differ from the final size, or
some tranfer progress might show >100% somewhere because of that).
2023-04-07 17:25:22 +03:00
Ilya Averyanov c073914f75 chore(ft): fix gen_rpc flakyness 2023-04-07 17:25:22 +03:00
Ilya Averyanov 228bf1a0ce chore(ft): fix typing issues 2023-04-07 17:25:22 +03:00
Ilya Averyanov 2c9cd1397d chore(ft): fix `?tp` calls 2023-04-07 17:25:22 +03:00
Ilya Averyanov 1c61165a91 fix(ft): fix emqx_cm takover 2023-04-07 17:25:22 +03:00
Ilya Averyanov c6b3f2c2ca feat(ft): add schema descriptions 2023-04-07 17:25:21 +03:00
Ilya Averyanov bc0a15afd7 feat(ft): fix test for CI 2023-04-07 17:25:21 +03:00
Ilya Averyanov 16fa55e19a feat(ft): add API tests 2023-04-07 17:25:21 +03:00
Andrew Mayorov 6d9f780313 chore: remove unused code 2023-04-07 17:25:21 +03:00
Andrew Mayorov f896fefa59 feat(ft): make storage backend fully async-aware
Introduce an ad-hoc concept of tasks that need to be kicked off
manually. Rework filesystem backend to accomodate for this change.
Adapt responder logic for that "kickoff" protocol.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 2cdf486bf4 feat(ft): simplify responder mechanism
Make responder short-lived process responsible for a single task, and
manage them with supervisor + gproc.
2023-04-07 17:25:21 +03:00
Ilya Averyanov f6a0598f27 feat(ft): add file transfer tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov 2e889f4ac7 feat(ft): add emqx_ft tests and fixes 2023-04-07 17:25:21 +03:00
Ilya Averyanov 1d48a97fd2 feat(ft): fix remote reader handling of gen_rpc errors 2023-04-07 17:25:21 +03:00
Ilya Averyanov c44fe92ef1 feat(ft): add assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov eae3dc7b4b feat(ft): update BPAPI 2023-04-07 17:25:21 +03:00
Ilya Averyanov bcfa22f343 fix(ft): use correct supervison strategy for emqx_ft_sup 2023-04-07 17:25:21 +03:00
Ilya Averyanov 9e4a37a398 fix(ft): fix typespecs 2023-04-07 17:25:21 +03:00
Ilya Averyanov 836ec213c9 feat(ft): add responder tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov 8038a3fd4a feat(ft): add tests for remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov b7d0bad970 feat(ft): improve remote reader 2023-04-07 17:25:21 +03:00
Ilya Averyanov 0aefd4a8c7 feat(ft): add streaming of file content when downloading 2023-04-07 17:25:21 +03:00
Ilya Averyanov 197ce32669 feat(ft): add proxy module for emqx_ft_storage_fs 2023-04-07 17:25:21 +03:00
Ilya Averyanov 04e5378bda feat(ft): add API 2023-04-07 17:25:21 +03:00
Andrew Mayorov 92670bfe3d feat(ft): add fs storage bpapi and use it in assembler 2023-04-07 17:25:21 +03:00
Andrew Mayorov 7ed06b0a2a feat(ft-fs): allow to list all transfers in storage
This is rather simplistic and thus, temporary solution.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 429eeaf029 feat(ft-fs): make `list` / `read` more generic
And usable in wider contexts as a consequence, for example querying and
fetching resulting files from remote nodes.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 1308fa0e6b fix(ft-fs): put fragments into separate directories
In order to avoid potential filename collisions.
2023-04-07 17:25:21 +03:00
Andrew Mayorov 8298236908 refactor(ft): bring back userdata to filemeta schema 2023-04-07 17:25:21 +03:00
Ilya Averyanov b4a42a447c feat(ft): removed replicated data 2023-04-07 17:25:21 +03:00
Ilya Averyanov 72e3eee6c9 feat(ft): add config & backend behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov 97b831a160 fix(ft-fs): add missing `read_segment/5` + fix atomic write 2023-04-07 17:25:21 +03:00
Andrew Mayorov 14b2a1013b fix(ft-asm): follow proper `segment` fragment type 2023-04-07 17:25:21 +03:00
Andrew Mayorov 1fedae8a16 fix(ft-asm): ensure module follows statem behaviour 2023-04-07 17:25:21 +03:00
Andrew Mayorov 7b77e96ab9 test(ft): add some basic assembler tests 2023-04-07 17:25:21 +03:00
Ilya Averyanov cbff2e2309 feat(ft): improve robustness of asynchronous acks
* add auto ack after timeout
* add fin file transfer packet registration to avoid
duplication and multiple acks
2023-04-07 17:25:21 +03:00
Ilya Averyanov d36ca18bff feat(ft): tie file transfer frontend and backend together 2023-04-07 17:25:21 +03:00
Andrew Mayorov 81e04ce93a feat(ft): introduce simple filesystem storage backend + assembler 2023-04-07 17:25:21 +03:00
Ilya Averyanov aaaef30be6 feat(ft): add file transfer app and bootstrap replicated ft data structure 2023-04-07 17:25:21 +03:00
Ilya Averyanov d976943f99 feat(ft): add basic hooks 2023-04-07 17:25:21 +03:00
JianBo He 502cc2b8b8 chore: fix common tests 2023-04-07 15:34:16 +08:00
Thales Macedo Garitezi 7c05304ff4 chore: bump version to e5.0.2-rc.6 2023-04-06 16:53:55 -03:00
Thales Macedo Garitezi 33100ecca6 feat: implement schema registry for 5.0 (avro)
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
2023-04-06 16:28:45 -03:00
Thales Macedo Garitezi 4c24b08244 fix(rule_action): fix metrics for bridges returning `async_return`
Kafka Producer, when called asynchronously, will return
`{async_return, {ok, pid()}}`, which currently counts as an unknown failure.
2023-04-06 16:00:01 -03:00
Zaiming (Stone) Shi 11a7770d16 chore: bump version to e5.0.2-rc.5 2023-04-06 11:57:39 +02:00
JimMoen bbb3fdb49e
fix: make emqx_api_lib compatible 2023-04-06 17:03:39 +08:00
Kjell Winblad 1938882f16
Merge pull request #10318 from kjellwinblad/kjell/feat/rule_engine_from_clause_support_both_string_types
feat(rule engine sql): enable both ' and " strings in FROM clause
2023-04-06 09:19:44 +02:00
JimMoen ab807fba51
Merge pull request #10329 from kjellwinblad/kjell/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent
fix: create consistent interface 'with_node' for API access
2023-04-06 14:47:42 +08:00
JianBo He 702ecbcb6e Merge remote-tracking branch 'upstream/master' into refactor-gw-dir 2023-04-06 10:44:59 +08:00
Stefan Strigler b799af1f71 fix: don't create virtual type "rule_engine_api" 2023-04-05 16:46:19 +02:00
Stefan Strigler 9cfe9cc709 fix: don't allow `rules` to be set from `/rule_engine` 2023-04-05 16:28:54 +02:00
Serge Tupchii 9810c9f7e3 refactor(rule_engine): test 'ok' results in one dedicated function 2023-04-05 16:55:43 +03:00
Stefan Strigler 5cd8865a93 fix: deprecate and hide jq_implementation_module 2023-04-05 15:34:42 +02:00
Stefan Strigler aea870f319 feat: add `/rule_engine` API endpoint 2023-04-05 15:19:31 +02:00
Stefan Strigler f3446c48f7
Merge pull request #10315 from sstrigler/EMQX-8945-crash-for-bad-param-value-to-mqtt-delayed-messages-api-call
EMQX 8945 crash for bad param value to mqtt delayed messages api call
2023-04-05 13:44:22 +02:00
JianBo He a3262486e5 chore: delete all LICENSE files under gateway dirs 2023-04-05 15:25:08 +08:00
Stefan Strigler a39312e892
Merge pull request #10292 from sstrigler/EMQX-9253-readme-refine-emqx-management
EMQX Management README
2023-04-05 09:00:34 +02:00
Zaiming (Stone) Shi a9bf633e03
Merge pull request #10320 from zmstone/0403-sync-release-50-back-to-master
0403 sync release 50 back to master
2023-04-04 23:31:24 +02:00
Thales Macedo Garitezi 5d5b7ea215
Merge pull request #10306 from thalesmg/enable-async-buffer-workers-all-bridges-v50
feat(bridges): enable async query mode for all bridges with buffer workers
2023-04-04 17:10:46 -03:00
Ivan Dyachkov 9d1a16aae1 feat: add emqx_rpc:multicall_on_running
also move emqx:is_running multicall to emqx_proto_v2:are_running
2023-04-04 20:40:47 +02:00
Ivan Dyachkov 3d7ceb01a0 fix(mgmt): fix stats api by applying filter to running_nodes 2023-04-04 19:58:43 +02:00
Ivan Dyachkov c20da5ffa6 fix(emqx_dashboard): fix monitor_current api 2023-04-04 19:58:43 +02:00
Stefan Strigler 04626ce9cc fix: create consistent interface 'with_node' for API access 2023-04-04 16:54:14 +02:00
Zaiming (Stone) Shi 68c15ffd48 Merge remote-tracking branch 'origin/release-50' into 0403-sync-release-50-back-to-master 2023-04-04 16:42:58 +02:00
Zaiming (Stone) Shi d25db3ace4 chore: bump version to e5.0.2-rc.4 2023-04-04 16:38:25 +02:00
Zaiming (Stone) Shi 9e17064e47
Merge pull request #10325 from zmstone/0404-delay-config-change-replay-until-handler-is-ready
0404 delay config change replay until handler is ready
2023-04-04 16:37:35 +02:00
SergeTupchiy 877b828d4a
Merge pull request #10327 from SergeTupchiy/EMQX-8786-fix-unknown-counter-inc-on-unrecoverable-err
fix(rule_engine): don't increment unknown counter on unrecoverable er…
2023-04-04 16:51:34 +03:00
Zaiming (Stone) Shi 5925ff07c2 test(emqx_cluster_rpc): fix test cases 2023-04-04 15:20:30 +02:00
Zaiming (Stone) Shi eeb7b32bc8
Merge pull request #10317 from zmstone/0403-refactor-hide-listener-level-authentication
0403 refactor hide listener level authentication
2023-04-04 15:04:35 +02:00
Serge Tupchii aca65ca2d4 fix(rule_engine): don't increment unknown counter on unrecoverable errors
Closes: EMQX-8786
2023-04-04 15:59:53 +03:00
Zaiming (Stone) Shi 196ca43fbb fix(emqx_conf_app): call the right API to retrieve core nodes 2023-04-04 14:19:48 +02:00
Thales Macedo Garitezi 0b6fd7fe14 fix(buffer_worker): check request timeout and health check interval
Port of https://github.com/emqx/emqx/pull/10154 for `release-50`

Fixes https://emqx.atlassian.net/browse/EMQX-9099

Originally, the `resume_interval`, which is what defines how often a
buffer worker will attempt to retry its inflight window, was set to
the same as the `health_check_interval`.  This had the problem that,
with default values, `health_check_interval = request_timeout`.  This
meant that, if a buffer worker with those configs were ever blocked,
all requests would have timed out by the time it retried them.

Here we change the default `resume_interval` to a reasonable value
dependent on `health_check_interval` and `request_timeout`, and also
expose that as a hidden parameter for fine tuning if necessary.
2023-04-04 08:58:36 -03:00
Zaiming (Stone) Shi 8fd9dd741e fix(emqx_conf_app): wait for tables ready beofre starting apps 2023-04-04 13:50:50 +02:00
Kjell Winblad 0e66eb5f3f feat(rule engine sql): enable both ' and " strings in FROM clause
This commit upgrades the rulesql dependency to version 1.5 instead of
1.4. The difference between these two versions is that strings surrounded
by ' and " are supported in FROM clauses in version 1.5, but in version
1.4, only strings surrounded by " are supported.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9243
2023-04-04 10:36:48 +02:00
Zaiming (Stone) Shi b16c516e6b refactor: rename cluster_rpc_handler to cluster_rpc_cleaner
this reflects what is actually does
2023-04-04 10:28:47 +02:00
firest 6b2419998d chore: bump emqx_authn version 2023-04-04 16:06:23 +08:00
firest 11b3264251 fix: redact the password to `******` in API examples
fix #10222
2023-04-04 16:02:45 +08:00
Thales Macedo Garitezi a8f8228a12
Merge pull request #10308 from thalesmg/test-increase-peer-timeout-v50
test(peer): increase init and startup timeout for peer nodes
2023-04-03 15:46:13 -03:00
Zaiming (Stone) Shi 429b3d9efd Merge remote-tracking branch 'origin/master' into 0403-sync-release-50-back-to-master 2023-04-03 20:32:33 +02:00
Zaiming (Stone) Shi 2f2e8b8218 chore: bump version to e5.0.2-rc.3 2023-04-03 20:23:31 +02:00
Thales Macedo Garitezi 06048ae4ff
Merge pull request #10316 from thalesmg/flaky-crl-emqtt-test-v50
test(crl): fix flaky test (v5.0)
2023-04-03 15:12:36 -03:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Thales Macedo Garitezi 8b5a717a1f test(peer): increase init and startup timeout for peer nodes
Attempt to stabilize tests that use cluster nodes.
2023-04-03 13:20:22 -03:00
Zaiming (Stone) Shi 2d6ca69ffb refactor: no support for listener level authentication for now 2023-04-03 16:49:35 +02:00
Zaiming (Stone) Shi e978d86c86 chore: add doc_lift for authorization.sources
doc_lift is to make the doc render application to lift
this field to the root level and force the field's doc
to refernec it instead of expanding the structs in a nested way
2023-04-03 16:49:35 +02:00
Thales Macedo Garitezi ed25ee6fec test(crl): fix flaky test (v5.0) 2023-04-03 10:58:31 -03:00
Stefan Strigler 0efa9c7a11 fix: pretty format error responses 2023-04-03 15:48:52 +02:00
Stefan Strigler c1cb5357e1 fix: enable schema check 2023-04-03 15:48:33 +02:00
Thales Macedo Garitezi ec1871ffde test(janitor): catch each callback invocation 2023-04-03 10:20:19 -03:00
JimMoen 53712e6146
fix: running nodes should not include replica nodes 2023-04-03 20:01:37 +08:00
Zaiming (Stone) Shi 5f6d318cf0 fix(i18n): fix missing docs for gateway configs 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi 36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
JianBo He 205e97fdca chore(gw): update README files 2023-04-03 14:30:41 +08:00
zhongwencool d63680cf25
Merge pull request #10307 from emqx/release-50
Sync release-50 back to master
2023-04-02 11:36:41 +08:00
Zaiming (Stone) Shi b17548f382
Merge pull request #10265 from HJianBo/improve-some-options-desc
docs: improve some mqtt options description
2023-04-01 16:43:15 +02:00
JianBo He b2d018f249 chore: fix test cases 2023-04-01 18:51:39 +08:00
JianBo He 3a3879f99f chore: update gateways deps 2023-04-01 12:59:43 +08:00
JianBo He 9577beaa4e chore: update rebar.conf in emqx_exproto 2023-04-01 12:24:09 +08:00
JianBo He b24ff9bc6e test(gateway): refine all test cases 2023-04-01 12:24:09 +08:00
JianBo He a70545b64a chore: split out exproto gateway 2023-04-01 12:24:09 +08:00
JianBo He b58ce09658 chore: split out lwm2m 2023-04-01 12:24:09 +08:00
JianBo He 40c413ac05 chore: fix dialyzer warnings 2023-04-01 12:24:09 +08:00
JianBo He 0b6c5c4c91 refactor: split out emqx_coap application 2023-04-01 12:24:09 +08:00
JianBo He 786f030958 test: move tests into splited gateway dirs 2023-04-01 12:24:09 +08:00
JianBo He 159bcf329c refactor: make Stomp and MQTT-SN gateway as an independent apps 2023-04-01 12:24:09 +08:00
Thales Macedo Garitezi 53d760ec69
Merge pull request #10300 from thalesmg/fix-plugin-upload-v50
fix(plugins): create directory before uploading
2023-03-31 16:28:18 -03:00
Thales Macedo Garitezi 246a792965
Merge pull request #10273 from thalesmg/refactor-kprod-start-error-msg-rv50
fix: return friendly message when kafka producer and consumer fails to start (rv5.0)
2023-03-31 16:25:26 -03:00
Thales Macedo Garitezi 5011486b18 fix(kafka_consumer): return better error messages when probing kafka consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-9422
2023-03-31 11:33:15 -03:00
Zaiming (Stone) Shi 9bbca07943
Merge pull request #10298 from emqx/release-50
Merge release-50 back to master
2023-03-31 15:35:51 +02:00
Zaiming (Stone) Shi abf0329b60 test(emqx_banned_SUITE): fix flaky test case 2023-03-31 13:03:00 +02:00
Zaiming (Stone) Shi bcde52383b docs: fix max batch size desc 2023-03-31 12:35:27 +02:00
William Yang d9bec63733
Merge pull request #10284 from qzhuyan/docs/william/shared-subs
chore(docs): update shared_subscription_strategy
2023-03-31 09:44:12 +02:00
Stefan Strigler 7e31e60e90
style: fix wording 2023-03-31 09:19:48 +02:00
Stefan Strigler 1ff96f5314
style: fix wording
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-31 09:18:39 +02:00
Thales Macedo Garitezi a2b82dae69 fix(plugins): create directory before uploading
Fixes https://emqx.atlassian.net/browse/EMQX-9434

Fixes https://github.com/emqx/emqx-elixir-plugin/issues/23
2023-03-30 17:51:16 -03:00
Zaiming (Stone) Shi 36043dd651
Merge pull request #10286 from zmstone/0330-exit-with-non-zero-code-if-conf-init-failed
0330 exit with non zero code if conf init failed
2023-03-30 17:41:14 +02:00
Stefan Strigler 7c597bfaa9 docs: give a better idea of what this application is supposed to do 2023-03-30 16:57:16 +02:00
Stefan Strigler 87a262edff docs: add README for EMQX Dashboard 2023-03-30 16:53:27 +02:00
Thales Macedo Garitezi 632bffd451 fix: return friendly message when kafka producer fails to start (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9392

The returned information does not allow to diagnose the issue (i.e.: a
connection issue due to the wrong host and port, the wrong password
failing authn).  However, such information is printed to the logs.

This changes the returned error to the API so that the user is hinted
at looking at the logs for further investigation of the error.
2023-03-30 11:51:36 -03:00
Thales Macedo Garitezi ad28d8418d
Merge pull request #10274 from thalesmg/fix-subs-api-egress-mqtt-v50
fix: handle subscription list api when there are `emqx:subscribe` internal subscriptions
2023-03-30 11:16:31 -03:00
lafirest 0af226dd5b
Merge pull request #10276 from lafirest/fix/banned_api
fix(i18n): fix the description error about the default value of `until`
2023-03-30 20:26:32 +08:00
William Yang 3c52faa55e chore(docs): update shared_subscription_strategy 2023-03-30 13:32:45 +02:00
William Yang 65ef9c9086
Merge pull request #10077 from qzhuyan/dev/william/quic-cert-password
feat(quic): support TLS password protected keyfile
2023-03-30 12:59:50 +02:00
Zaiming (Stone) Shi 494e4b639a chore: exit with non-zero status if config initialization failed 2023-03-30 12:52:51 +02:00
firest f794666c7f fix(i18n): fix the description error about the default value of `until` 2023-03-30 17:12:45 +08:00
JianBo He 1ff8ecf604 chore: fix typos 2023-03-30 15:10:10 +08:00
JianBo He 3c495f8fd2
chore: update apps/emqx/i18n/emqx_schema_i18n.conf 2023-03-30 08:44:28 +08:00
JianBo He 758e610059
chore: apply suggestions from code review
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-30 08:42:08 +08:00
Thales Macedo Garitezi 806ee716ee fix(redis): remove `required` meta from schema
Since there's a default value, this field shouldn't be marked as
required.
2023-03-29 16:43:52 -03:00
Thales Macedo Garitezi 916ce99fd5 fix: handle subscription list api when there are `emqx:subscribe` subscriptions 2023-03-29 14:56:40 -03:00
Thales Macedo Garitezi 5089449203
Merge pull request #10260 from thalesmg/minor-refactor-basic-usage-rule-rv50
refactor(rule_engine): use more helper functions
2023-03-29 09:44:23 -03:00
Zaiming (Stone) Shi 80eb9d7542
Merge pull request #10252 from emqx/release-50
0327 merge release-50 to master
2023-03-29 12:33:17 +02:00
Kjell Winblad 8e0d315b7b
Merge pull request #10197 from kjellwinblad/0321-fix-inflight-window-hand-over-to-kjell
fix: add inflight window setting to the clickhouse bridge
2023-03-29 09:38:24 +02:00
JianBo He fee4ff6a07
Merge pull request #10257 from HJianBo/gw-lwm2m-fixes
Fixed the issue where `auto_observe` was not working in LwM2M Gateway
2023-03-29 15:20:46 +08:00
JianBo He 27aa3f49aa docs: improve some mqtt options description 2023-03-29 15:02:01 +08:00
Zaiming (Stone) Shi 11c120f87c chore: bump emqx app vsn 2023-03-28 21:19:03 +02:00
Zaiming (Stone) Shi 4b428f7a29 chore: bump version to e5.0.2-rc.2 2023-03-28 21:18:21 +02:00
Thales Macedo Garitezi 64faccf50b test: fix flaky kafka consumer test 2023-03-28 14:50:55 -03:00
Thales Macedo Garitezi d126c7dc62 refactor(rule_engine): use more helper functions
Follow up to
https://github.com/emqx/emqx/pull/10251#discussion_r1150710899 and
https://github.com/emqx/emqx/pull/10251#discussion_r1150720420
2023-03-28 13:46:56 -03:00
William Yang ecc2cd1a94 test: password cert for SSL listener 2023-03-28 17:15:59 +02:00
Thales Macedo Garitezi 1824e7efcc fix(rule_engine): count referenced bridges in `from` clauses as dependencies (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9325

Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.

When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
2023-03-28 10:31:10 -03:00
Zaiming (Stone) Shi f5e9d3eab2 test: use valid config for jq function timeout 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi 3f9ed08308 fix(emqx_schema): ocsp field use 'importance' to replace 'hidden' 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi d07987288a chore: add some example annotations for config importance level 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi 5a19b6f73e chore: pin hocon 0.38.0 (with config importance level annotation) 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi 486352eb6f fix(swagger): do not generate scheam for hidden args 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi fb29f8035b feat: make possible to dump full schema (including hidden fields) 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi c302405408 fix: fail fast if not able to load config 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi 813ccd7df1 chore: hide node.perf configs
these configs are rarely needed by regular users
and if they are to be changed, it should be done by
someone who really understands what they mean
2023-03-28 14:29:24 +02:00
JianBo He 6db5a4872a test(lwm2m): cover auto_observe mechanism 2023-03-28 18:31:20 +08:00
JianBo He 6750618d9a fix(lwm2m): ignore unknown ObjectId to auto observe 2023-03-28 17:44:33 +08:00
Xinyu Liu c607e4ed92
Merge pull request #10243 from terry-xiaoyu/remove-peercert-from-conninfo50
chore: remove peercert from the state of connection processes
2023-03-28 17:22:57 +08:00
Stefan Strigler 193905c7b9
Merge pull request #10237 from sstrigler/EMQX-9138-e-5-0-2-ensure-that-the-error-messages-returned-are-consistent
fix: create consistent interface 'with_node' for API access
2023-03-28 10:41:10 +02:00
Andrew Mayorov 89108594f0
Merge pull request #10212 from keynslug/test/EMQX-9269/cluster-env
test(bridge-api): also run testcases in cluster environment
2023-03-28 11:36:48 +03:00
Andrew Mayorov 40f7fd691e
test(dashboard): reuse `emqx_mgmt_api_test_util` in testsuites
To reduce the init / teardown complexity.
2023-03-27 23:41:19 +03:00
Andrew Mayorov b14a420885
fix(test): delete mnesia schema in-between test suites
When using `emqx_common_test_helpers` at least. This should help
reduce the need for superfluous cleanups and test failures due to
unclean state.
2023-03-27 23:41:19 +03:00
Andrew Mayorov d942a531e6
fix(test): avoid unnecessary cleanups
This might make some noise in the logs, depending on the testsuite
configuration, because some test setup might be missing required
applications.
2023-03-27 23:41:19 +03:00
Andrew Mayorov 11b5b7b638
test(bridge-api): also run testcases in cluster environment
Excluding a couple of testcases which does not make much sense running
in the cluster. Also try to reduce amount of "noise" in the testcases,
making them easier to comprehend.

Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-27 23:41:08 +03:00
Zaiming (Stone) Shi 7a6494cda6
Merge pull request #10242 from zmstone/0326-fix-logging-avoid-using-pid-as-log-data-field
fix: avoid uisng 'pid' as log data field
2023-03-27 16:13:50 +02:00
Stefan Strigler 64a1d84a44 fix: create consistent interface 'with_node' for API access 2023-03-27 12:14:48 +02:00
William Yang cec77c2b65 test(quic): chasing flaky tc. 2023-03-27 11:54:08 +02:00
William Yang 169cc9f822 chore(quic): unhide TLS certfile password 2023-03-27 11:54:08 +02:00
Andrew Mayorov 9a93b745fc
fix(dashboard): use correct default for listeners map 2023-03-27 12:38:07 +03:00
firest 5c9538086f chore: bump emqx_plugin_libs.app.src 2023-03-27 13:54:38 +08:00
JianBo He bfa5922209
Merge pull request #10140 from HJianBo/cassa
feat: support cassandra data bridge
2023-03-27 10:23:02 +08:00
JimMoen 798597c834
Merge pull request #10217 from JimMoen/conn-health-check-with-worker
fix: for connection used ecpool, let worker do health check fun
2023-03-27 09:51:52 +08:00
JianBo He b77aeb69cb
Merge pull request #10172 from HJianBo/fix-typos-acl-file
fix(acl): fix wrong default ACL rules
2023-03-27 09:51:02 +08:00
Shawn 07ac2cd57a chore: remove peercert from the state of connection processes 2023-03-27 09:33:17 +08:00
Zaiming (Stone) Shi 19e136bb66 fix: avoid uisng 'pid' as log data field
Since it clashes with log metad data field
2023-03-26 10:33:18 +02:00
Zaiming (Stone) Shi dd996ad1dc chore: bump app vsns 2023-03-24 21:47:15 +01:00
Zaiming (Stone) Shi b37f186142 Merge remote-tracking branch 'origin/master' into 0324-merge-release-50-back-to-master 2023-03-24 21:47:15 +01:00
Thales Macedo Garitezi ff272a2071
Merge pull request #10206 from thalesmg/decouple-buffer-worker-query-call-mode-v50
feat(buffer_worker): decouple query mode from underlying connector call mode
2023-03-24 13:49:00 -03:00
Zaiming (Stone) Shi bf29919708
Merge pull request #10160 from thalesmg/hide-ocsp-v50
chore(ocsp): hide ocsp from schema for now
2023-03-24 15:32:47 +01:00
SergeTupchiy f4472f66de
Merge pull request #10225 from SergeTupchiy/EMQX-9290-fix-plugin-name-clash
fix(emqx_management): resolve plugin name clashes
2023-03-24 16:17:20 +02:00
Serge Tupchii b02d0088ea fix(emqx_management): resolve plugin name clashes
Allow installing a plugin if its name matches the beginning of another (already installed) plugin name.
For example: if plugin "emqx_plugin_template_a" is installed, it must not block installing plugin "emqx_plugin_template".

Closes: #10213, EMQX-9290
2023-03-24 16:11:31 +02:00
Stefan Strigler 8d084ed13b
Merge pull request #10226 from sstrigler/EMQX-9273-e-5-0-2-creating-a-data-bridge-the-name-verification-returned-an-incorrect-error-code-of-500
fix(emqx_bridge): don't crash on validation error
2023-03-24 14:48:03 +01:00
Thales Macedo Garitezi 166191abd6 chore(ocsp): hide ocsp from schema for now
We're going to release it in a later version.  So, for now, we'll just
hide it.
2023-03-24 10:04:16 -03:00
Stefan Strigler fbfdaf44e0 fix(emqx_bridge): don't crash on validation error 2023-03-24 11:46:41 +01:00
Stefan Strigler a8cfc1c7a1 fix(emqx_bridge): filter status_reason in config update 2023-03-24 11:46:41 +01:00
Stefan Strigler 4f63982d13
Merge pull request #10195 from sstrigler/EMQX-8982-html-tags-are-displayed-in-http-api-doc
fix: add summary to schemas where description has html
2023-03-24 08:56:34 +01:00
Ivan Dyachkov 0208fc8296 chore: e5.0.2-rc.1 2023-03-24 08:55:31 +01:00
Zaiming (Stone) Shi 9870f4834e
Merge pull request #10179 from kjellwinblad/kjell/fix/rule_engine_app_readme
docs: refine README.md for the emqx_rule_engine app
2023-03-24 08:28:11 +01:00
JimMoen 65f468f706
fix: for connection used ecpool, let worker do health check fun
let it be same with ecpool:pick_and_do/3 for checkfun use format as {M,F,A}
2023-03-24 10:39:58 +08:00
Thales Macedo Garitezi f8d5d53908 feat(buffer_worker): decouple query mode from underlying connector call mode
Fixes https://emqx.atlassian.net/browse/EMQX-9129

Currently, if an user configures a bridge with query mode sync, then
all calls to the underlying driver/connector ("inner calls") will
always be synchronous, regardless of its support for async calls.

Since buffer workers always support async queries ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

There are two situations when we want to force synchronous calls to
the underlying connector even if it supports async:

1) When using `simple_sync_query`, since we are bypassing the buffer
workers;
2) When retrying the inflight window, to avoid overwhelming the
driver.
2023-03-23 13:40:31 -03:00
Stefan Strigler 678400bc0a fix: add summary to schemas where description has html 2023-03-23 16:40:43 +01:00
Stefan Strigler ba49316ec5 fix: use lower-case 2023-03-23 16:08:27 +01:00
William Yang 7e15f90bf5 chore(test): check openssl cmd returns 2023-03-23 15:06:21 +01:00
William Yang 88731fd145 feat(quic): support TLS password protected keyfile 2023-03-23 15:06:21 +01:00
Stefan Strigler 2803aff798
Merge pull request #10207 from sstrigler/EMQX-9286-use-label-from-i-18-n-files-as-summary-for-open-api-spec
feat(emqx_dashboard): use label as summary in OpenAPI spec
2023-03-23 14:58:21 +01:00
Kjell Winblad 35474578ca refactor: rename async_inflight_window to inflight_window everywhere 2023-03-23 14:21:57 +01:00
Kjell Winblad 9d3f369cca
docs: fix spelling mistake
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-03-23 14:09:57 +01:00
Serge Tupchii e3541d8947 test: move dashboard specific cleanup call from emqx_common_test_helpers to the relevant test suite 2023-03-23 11:59:43 +02:00
Serge Tupchii 50bbf4f231 chore: bump ekka to 0.14.6
ekka 0.14.6 uses mria 0.4.2, which unregisters mnesia_hook upon stopping mria.
This fixes mnesia_hook errors on restarting mria.

Closes: EMQX-9101
2023-03-23 11:59:43 +02:00
JianBo He 8cbbc9f271 Merge remote-tracking branch 'upstream/master' into cassa 2023-03-23 11:53:17 +08:00
Thales Macedo Garitezi ddffba0355
Merge pull request #10154 from thalesmg/fix-buffer-worker-default-req-timeout
fix(buffer_worker): calculate default `resume_interval` based on `request_timeout` and `health_check_interval`
2023-03-22 20:21:04 -03:00
Thales Macedo Garitezi 84f7e9c320
Merge pull request #10209 from thalesmg/fix-lwt-banned-then-kicked-v50
fix(last_will_testament): don't publish LWT if client is banned when kicked out
2023-03-22 18:30:31 -03:00
Thales Macedo Garitezi cb65cded88 fix(last_will_testament): don't publish LWT if client is banned when kicked
Fixes https://emqx.atlassian.net/browse/EMQX-9288

Related issue:
https://github.com/emqx/emqx/issues/10192#issuecomment-1478809900
2023-03-22 16:47:58 -03:00
Thales Macedo Garitezi 8844b22c80
docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-22 15:32:09 -03:00
Thales Macedo Garitezi 127a075b66 test(dynamo): attempt to fix dynamo tests
Those tests in the `flaky` test are really flaky and require lots of
CI retries.

Apparently, the flakiness comes from race conditions from restarting
bridges with the same name too fast between test cases.  Previously,
all test cases were sharing the same bridge name (the module name).
2023-03-22 14:34:37 -03:00
Stefan Strigler 2662f79ec0 feat(emqx_dashboard): use label as summary in OpenAPI spec 2023-03-22 16:29:51 +01:00
Thales Macedo Garitezi 61cb03b45a fix(buffer_worker): change the default `resume_interval` value and expose it as hidden config
Also removes the previously added alarm for request timeout.

There are situations where having a short request timeout and a long
health check interval make sense, so we don't want to alarm the user
for those situations.  Instead, we automatically attempt to set a
reasonable `resume_interval` value.
2023-03-22 11:47:36 -03:00
lafirest 84def357a9
Merge pull request #10143 from lafirest/feat/rocketmq
feat(bridges): integrate RocketMQ into data bridges
2023-03-22 20:43:22 +08:00
Zaiming (Stone) Shi e43aa5d841
Merge pull request #10199 from zmstone/0321-merge-release-50-to-master
0321 merge release 50 to master
2023-03-22 10:20:48 +01:00
Zaiming (Stone) Shi db6afb40b0
Merge pull request #10174 from zmstone/0320-upgrade-esockd-to-5.9.6
fix: upgrade esockd from 5.9.4 to 5.9.6
2023-03-22 09:50:38 +01:00
Andrew Mayorov af7b2215ca
fix(bridge-api): unflatten nodes' bridge lists back
Bridge lists were erroneously flattened in cad6492c. This causes
bridge listing fail in emqx clusters consisting of more than 1 node.
2023-03-22 11:35:09 +03:00
firest 9853d00cad feat(bridges): integrate RocketMQ into data bridges 2023-03-22 10:36:52 +08:00
JianBo He 65c2da7ef5 Merge remote-tracking branch 'ce/master' into cassa 2023-03-22 09:30:50 +08:00
Andrew Mayorov bf35419b52
Merge pull request #10190 from keynslug/fix/EMQX-9269/bridge-list
fix(bridge-api): unflatten nodes' bridge lists back
2023-03-22 00:53:03 +03:00
Zaiming (Stone) Shi e6091db351 Merge remote-tracking branch 'origin/release-50' into 0321-merge-release-50-to-master 2023-03-21 22:03:31 +01:00
Stefan Strigler 6c24b3a77d
Merge pull request #10196 from sstrigler/EMQX-9201-the-case-is-not-correct-for-some-of-menu
Emqx 9201 the case is not correct for some of menu
2023-03-21 19:42:10 +01:00
Thales Macedo Garitezi 4bbbaf8b13
Merge pull request #10164 from thalesmg/crl-v50-mkIII
feat(crl): add crl check support
2023-03-21 14:42:01 -03:00
Kjell Winblad 27b8445337 fix: add inflight window setting to the clickhouse bridge
This commit makes sure the inflight window setting is present for the
clickhouse bridge. It also changes emqx_resource_schema that previously
removed the inflight window setting from resources with query mode
`always_sync`. We don't need to do that because all bridges that uses
the buffer worker queue will get async call handling even if the bridge
don't support the async callback.

Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 17:14:03 +01:00
Stefan Strigler 327b782210 style: lower case in descriptions 2023-03-21 17:04:42 +01:00
Andrew Mayorov d9b1b1c802
fix(bridge-api): unflatten nodes' bridge lists back
Bridge lists were erroneously flattened in cad6492c. This causes
bridge listing fail in emqx clusters consisting of more than 1 node.
2023-03-21 18:59:27 +03:00
Stefan Strigler 981268911a style: make summary lower-case 2023-03-21 16:56:37 +01:00
Stefan Strigler 3880862c81 fix(emqx_bridge): return 503 for inconsistency in bridge setup 2023-03-21 15:07:43 +01:00
Stefan Strigler 4b0ea562a2 refactor(emqx_bridge): consistently use macros for http response 2023-03-21 15:07:41 +01:00
Stefan Strigler 84fc64822e style: fix wording for nxdomain and malformed_username_or_password
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 15:02:29 +01:00
Stefan Strigler 8af3fb4ee7 feat: move human readable error translations to emqx_misc 2023-03-21 15:02:29 +01:00
Stefan Strigler c1384b6e6e feat(emqx_resource): include error with alarm for resource_down 2023-03-21 15:02:29 +01:00
Stefan Strigler 53825b9aba fix(emqx_bridge): propagate connection error to resource status 2023-03-21 15:02:29 +01:00
Zaiming (Stone) Shi fd23800370
Merge pull request #10180 from zmstone/0320-add-emqx_ctl-readme
docs: add readme for emqx_ctl app
2023-03-21 11:33:16 +01:00
William Yang 7d8fc3d2ad
Merge pull request #10185 from qzhuyan/dev/william/bump-quicer
chore(build): Support fetching binary for macOS M1 [quicer 0.0.114]
2023-03-21 11:32:33 +01:00
William Yang 092daf409a chore(testlib): update include for CI of emqtt lib 2023-03-21 09:03:31 +01:00
William Yang b6e583143f chore: bump quicer 0.0.114
for fetching quicer binary for macOS M1 builds.
2023-03-21 09:01:25 +01:00
Kjell Winblad 89782e6d21
docs: add info about that rule engine can handle different types of structured data
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-21 06:26:04 +01:00
JianBo He b059bad08a
chore(acl): fully match `dashboard` username
Co-authored-by: Ilya Averyanov <av@rubybox.dev>
2023-03-21 09:06:46 +08:00
Thales Macedo Garitezi f2b54588e7 docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-20 18:09:38 -03:00
Thales Macedo Garitezi 7a71d64686 feat(crl): add crl check support 2023-03-20 18:09:38 -03:00
Ilya Averyanov ee403423b9
Merge pull request #10166 from savonarola/0318-mix-build
chore(mix): fix mix builds
2023-03-20 22:37:21 +02:00
Zaiming (Stone) Shi c739d9957c docs: add readme for emqx_ctl app 2023-03-20 19:35:17 +01:00
Zaiming (Stone) Shi 6f71898546 fix: upgrade esockd from 5.9.4 to 5.9.6
changes in esockd are

* 5.9.5: added API to retrieve SNI
* 5.9.6: avoid error log if socket is closed before sending
         proxy protocol headers
2023-03-20 19:27:32 +01:00
Kjell Winblad 08e249dc4d docs: refine README.md for the emqx_rule_engine app
This commit adds more content to the readme file of the `emqx_rule_engine`
app. The aim of the changes are to make it easier to understand what the
rule engine is and what it does.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9229
2023-03-20 16:53:55 +01:00
Erik Timan 2d75c7d6d9 fix(emqx_bridge): remove metrics from non-dedicated bridge API endpoints
Metrics should only be exposed via the /bridges/:id/metrics endpoint,
and not in other operations such as getting the list of all bridges, or
in the response when a bridge has been created. This commit removes all
traces of metrics for the non-dedicated API endpoints.
2023-03-20 09:43:11 +01:00
JianBo He 377127ce72 fix(acl): fix wrong default ACL rules 2023-03-20 13:53:57 +08:00
JianBo He 12942b676d Merge remote-tracking branch 'upstream/master' into cassa 2023-03-20 09:50:27 +08:00
zhongwencool 6a668c35db
Merge pull request #10167 from zhongwencool/fix-swagger-doc-500-crash
fix: swagger doc 500 crash
2023-03-19 16:28:32 +08:00
Zhongwen Deng b529a0bce1 test: add swagger.json api test 2023-03-19 13:05:32 +08:00
Zhongwen Deng f8573ae8d7 fix: swagger doc 500 crash 2023-03-19 12:34:06 +08:00
Ilya Averyanov 39f22585d0 chore(mix): fix mix builds 2023-03-19 01:39:56 +02:00
Zaiming (Stone) Shi 149afe863f chore: e5.0.2/v5.0.21 code freeze 2023-03-17 13:19:44 +01:00
Thales Macedo Garitezi cf1cce011b
Merge pull request #9564 from thalesmg/kafka-source-ee50
feat: implement kafka consumer (ee5.0)
2023-03-17 09:04:53 -03:00
Zaiming (Stone) Shi 1bf72c4d2e
Merge pull request #10155 from zmstone/0316-build-schema-dump-per-lang
build: generate per-lang schema dump
2023-03-17 11:45:35 +01:00
Zhongwen Deng 3c1254d873 fix: newly created listeners have no limiter restrictions 2023-03-17 14:32:42 +08:00
JianBo He c0a216a740 feat(bridge): support cassandra bridge 2023-03-17 11:34:48 +08:00
Zaiming (Stone) Shi ad1deedd0e build: generate per-lang schema dump 2023-03-16 21:25:21 +01:00
Thales Macedo Garitezi 20414d7373 fix(buffer_worker): check request timeout and health check interval
Fixes https://emqx.atlassian.net/browse/EMQX-9099

The default value for `request_timeout` is 15 seconds, and the default
resume interval is also 15 seconds (the health check timeout, if
`resume_interval` is not explicitly given).  This means that, in
practice, if a buffer worker ever gets into the blocked state, then
almost all requests will timeout.

Proposed improvement:

- `request_timeout` should by default be twice as much as
health_check_interval.
- Emit a alarm if `request_timeout` is not greater than
`health_check_interval`.
2023-03-16 13:46:45 -03:00
Thales Macedo Garitezi 966276127e test: trying to make tests more stable 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 3954b7bde2 fix(bridges): function clause when a non-ingress bridge coexists with an egress bridge
This was not caught by our tests because we always test bridge types
in isolation.  So, if the config only contains ingress-only bridges,
the `on_message_publish` hook is never installed.

In a real system, if there are bridges of mixed types in the config,
the hook might be installed, and `emqx_bridge:get_matched_bridge_id`
would crash when iterating over the ingress bridges.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 947e014132 test: improve cluster node helpers
* Add option to start autocluster.  This is useful for scenarios where
a cluster is already running and has some configurations set (via
config handler/cluster rpc) and later another node joins.

* Improve mnesia data directory isolation for nodes.  A new dir is set
so that we may avoid intra and inter-suite flakiness due to dirty
tables and schema.

* The janitor is now called synchronously.  This ensures the cleanup
is done before the test pid dies.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi d464e2aad5 refactor: rename test resource prefix
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 66eb4ef069 test: fix inter-suite flakiness 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi f31f15e44e chore(kafka_producer): make schema changes more backwards compatible
This will still require fixes to the frontend.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 03342923b9 fix(bridge): use the same dry run prefix
Kafka Producer and Consumer bridges rely on this prefix for detecting
a dry run and avoid leaking atoms.  At some point, this prefix was
changed, effectively disabling the check in Kafka Producer.
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 1f31a87974 fix(bridge): improve macro var usage
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 969fa03ecc feat: implement kafka consumer 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi 094e4a2eeb chore: fix typespec 2023-03-16 13:43:01 -03:00
Thales Macedo Garitezi b9e92173cf test: improve cluster helper
1) Make each node have its own isolated data dir to avoid false
negatives.
2) Allow parameterizing the peer module.
3) Fix cluster RPC after a node joins the cluster.
2023-03-16 13:42:59 -03:00
Thales Macedo Garitezi 561c25f0e3 feat: update snabbkaffe -> 1.0.7 2023-03-16 13:42:38 -03:00
Thales Macedo Garitezi 91a57faa95
Merge pull request #10128 from thalesmg/ocsp-v50-mkII
feat: add ocsp stapling support to mqtt ssl listener (5.0)
2023-03-16 13:10:48 -03:00
JimMoen 1688cb7df2
Merge pull request #10152 from JimMoen/refine-changes
chore: refine changes for merged PRs
2023-03-16 21:11:52 +08:00
Thales Macedo Garitezi a614bdc94a chore(ocsp): catch unexpected error when fetching ocsp response 2023-03-16 09:55:52 -03:00
Thales Macedo Garitezi d1f58d6e2d refactor: replace macro by simple function 2023-03-16 09:32:40 -03:00
Thales Macedo Garitezi e5645a7b21 refactor: rename macros 2023-03-16 09:18:16 -03:00
Kjell Winblad 884dddec79
Merge pull request #10136 from kjellwinblad/kjell/fix/default_value_ui
fix: schema default value so that it shows up correctly in UI
2023-03-16 12:20:08 +01:00
JimMoen afd29b69c4
chore: refine i18n doc punctuation 2023-03-16 18:22:29 +08:00
Ivan Dyachkov e3d4deb8a0
Merge pull request #10147 from id/chore-minor-improvements
Chore minor improvements
2023-03-16 08:18:43 +00:00
Andrew Mayorov 816667d85c
Merge pull request #10123 from fix/EMQX-9136/lookup-busy-resources
perf(bridge-api): ask nodes' bridge listings in parallel
2023-03-16 00:07:12 +03:00
Ivan Dyachkov 5258b4c6e9 chore: bump ekka to 0.14.5
to make use of erlang-rocksdb 1.7.2-emqx-9
2023-03-15 19:54:55 +01:00
Thales Macedo Garitezi 164440fe83 test(resource): fix flaky test
Sometimes this test might retry more times, so we check the prefix
of the trace only.
2023-03-15 14:25:55 -03:00
Thales Macedo Garitezi 03b95073fc test: fix inter-suite flakiness 2023-03-15 14:25:41 -03:00
Andrew Mayorov a9bc8a4464
refactor(resman): rename `ets_lookup` → `lookup_cached`
That way we hide the impementation details + the interface becomes
cleaner and more obvious.
2023-03-15 19:17:30 +03:00
Andrew Mayorov 29907875bf
test(bufworker): set `batch_time` for batch-related testcases
By default it's `0` since e9d3fc51. This made a couple of tests prone
to flapping.
2023-03-15 19:17:30 +03:00
Andrew Mayorov e411c5d5f8
refactor(resman): work with state cache atomically
Also ensure that cache entries are always consistent with `Data`,
so that most of the code could rely on reading the cached entry
most of the time.
2023-03-15 19:17:30 +03:00
Andrew Mayorov b3e7e51094
test(bridge): drop unnecessary cleanup routines
Since `end_per_testcase` cleans out all the resources anyway.
2023-03-15 19:17:29 +03:00
Andrew Mayorov cad6492c99
perf(bridge-api): ask bridge listings in parallel
Also rename response formatting functions to better clarify their
purpose.
2023-03-15 19:17:29 +03:00
Thales Macedo Garitezi 0deb9925c1 docs: improve descriptions
Co-authored-by: William Yang <mscame@gmail.com>
2023-03-15 10:30:02 -03:00
Zhongwen Deng 60677bc400 chore: remove unuse code 2023-03-15 20:40:54 +08:00
Ivan Dyachkov c9b736ae23
Merge pull request #10141 from id/0315-chore-merge-release-50-to-master
0315 chore merge release 50 to master
2023-03-15 12:11:54 +00:00
Kjell Winblad 13a35cc045 fix: schema default value so that it shows up correctly in UI
This complements PR https://github.com/emqx/emqx/pull/10124.
The default values for duration_ms() fields needs to be formatted
as a binary string with unit to show up correctly in the dashboard
UI.
2023-03-15 11:04:16 +01:00
Thales Macedo Garitezi 04378f242c feat(ocsp_cache): give ocsp cache table an heir 2023-03-14 16:09:06 -03:00
Thales Macedo Garitezi f6707d1dd0 test: fix how ocsp client is run in tests
For some yet unknown reason the old test version using `open_port`
does not work in OTP 25, but works fine in OTP 24.  There are no
messages at all received from The openssl client port program in OTP
25.
2023-03-14 16:09:06 -03:00
Thales Macedo Garitezi 63ef2f9b79 feat: save uploaded OCSP issuer pem like other ssl certs 2023-03-14 16:09:06 -03:00
Thales Macedo Garitezi 158f054187 docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-03-14 16:09:06 -03:00
Thales Macedo Garitezi 57e38c8502 refactor(ocsp): add reusable type for normalized binary URLs 2023-03-14 16:09:05 -03:00
Thales Macedo Garitezi 067747c2de docs: improve descriptions
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-03-14 16:08:47 -03:00
Thales Macedo Garitezi 52263a0448 feat: add ocsp stapling and crl support to mqtt ssl listener 2023-03-14 16:08:47 -03:00
Thales Macedo Garitezi 422597a441 test: fix flaky tests 2023-03-14 16:08:47 -03:00
Thales Macedo Garitezi 65fee34fe4 test: fix inter-suite test teardowns 2023-03-14 16:08:47 -03:00
Thales Macedo Garitezi 4880a849b9 chore: update emqtt -> 1.8.5
Needed for OCSP / CRL tests because of a bug that makes emqtt hang
forever on TLS handshake errors.
2023-03-14 16:08:47 -03:00
SergeTupchiy 9f9d16dd48
Merge pull request #10117 from SergeTupchiy/EMQX-8889_copy_plugins_on_joining_a_cluster
fix: copy plugins to a new node joining a cluster
2023-03-14 21:07:12 +02:00
Zaiming (Stone) Shi 6cbc17f110
Merge pull request #10135 from zmstone/0314-upgrade-to-hocon-0.37.0
chore: upgrade to hocon 0.37.0
2023-03-14 19:55:41 +01:00
Stefan Strigler 255117f2c4 chore: bump vsn e5.0.2-alpha.1 2023-03-14 19:51:35 +01:00
Serge Tupchii bde4215a11 fix: copy plugins to a new node joining a cluster
Closes: EMQX-8889
2023-03-14 20:42:57 +02:00
William Yang fffb92ebb7 test(quic): make tc run independent 2023-03-14 11:46:24 +01:00
William Yang 4c28b9f839 chore(test): use power of 2 recvbuff size in quic test
Make msquic assertions happy while in debug mode.
2023-03-14 11:46:24 +01:00
Zaiming (Stone) Shi 18b145ab67 chore: upgrade to hocon 0.37.0
hocon 0.37.0 introduced schema field aliases
this change does not impact anything until we start using aliases
2023-03-14 11:39:55 +01:00
zhongwencool 4f1da5e9fa
Merge pull request #10132 from zhongwencool/systemd-stop-crash-log
fix: stop port apps when emqx_machine_terminator shutdown
2023-03-14 17:40:26 +08:00
zhongwencool d853085259
Merge pull request #10119 from zhongwencool/fix-statsd-empty-server
fix: don't allow empty string in server_port schema
2023-03-14 16:37:15 +08:00
Zhongwen Deng 0738444da9 fix: stop port apps when emqx_machine_terminator shutdown 2023-03-14 15:39:50 +08:00
Andrew Mayorov a530ccbe3d
Merge pull request #10095 from fix/EEC-782/mysql-prepstmt-exhaustion
fix(mysql): be explicit that batch queries are parameterless
2023-03-14 10:21:45 +03:00
Zhongwen Deng 80205d9704 chore: apply code review 2023-03-14 10:54:05 +08:00
Zhongwen Deng ee2847dcd9 test: make schema test happy 2023-03-14 09:17:45 +08:00
Zhongwen Deng 5be4d97c42 fix: don't allow empty string in server_port schema 2023-03-14 09:17:45 +08:00
Thales Macedo Garitezi 823128dfc7
Merge pull request #10127 from thalesmg/rm-rebar3-emqx-app-master
chore: remove committed rebar3 binary
2023-03-13 16:33:57 -03:00
Kjell Winblad 87ede1b1f7
Merge pull request #10124 from kjellwinblad/kjell/mongodb/default_heartbeat
fix: increase heartbeat time to avoid extreme MongoDB logging
2023-03-13 18:56:49 +01:00
Thales Macedo Garitezi 25c28eab81 chore: remove committed rebar3 binary
We already set up our custom rebar3 binary in CI, so this doesn't need
to be committed.  It is also outdated with our currently used version.
2023-03-13 14:11:39 -03:00
Stefan Strigler 62aec87c56
Merge pull request #10107 from sstrigler/EMQX-9147-bridges-api-fix-handling-unknown-bridge-id
Bridges API: fix handling unknown bridge-id
2023-03-13 16:40:46 +01:00
Kjell Winblad 1cf01197bb fix: increase heartbeat time to avoid extreme MongoDB logging
Our MongoDB driver creates a new temporary connection, for every active
connection, to just do a single heartbeat test. There is configurable
delay between every heartbeat test. When the user has an EMQX cluster
with a MongoDB bridge (to a MongoDB replica set), there will be a lot of
connections. Furthermore, as MongoDB creates a log entry every time a
new connection is created, the log will be flooded with info about new
connection. One user have reported more than 1MB of log data in a 10
minute period.

This commit tries to fix this by increasing the default delay between
heartbeats. A better fix would be to change the MongoDB driver so that
it does not create a new connection just to do a heartbeat check, but
this is more complicated so we leave this to the future. We might also
swap out the current MongoDB driver to something better.

Fixes:
https://github.com/emqx/emqx/issues/9851
2023-03-13 14:58:53 +01:00
Zaiming (Stone) Shi 0c5ab5183a
Merge pull request #10109 from zmstone/0308-stop-releasing-schema.json-in-rel-package
0308 stop releasing schema.json in rel package
2023-03-13 14:27:15 +01:00
Andrew Mayorov 53bc27e0f4
refactor(bridge): avoid unnecessary `maps:to_list/1` when listing 2023-03-13 14:49:38 +03:00
Andrew Mayorov 686bf8255b
fix(bridge): reply `emqx_resource:get_instance/1` from cache
The resource manager may be busy at times, so this change ensures that
getting resource instance state will not block. Currently, no users of
`emqx_resource:get_instance/1` do seem to be relying on state being
"as-actual-as-possible" guarantee it was providing.
2023-03-13 14:35:08 +03:00
Kjell Winblad baf39fe080
Merge pull request #10098 from kjellwinblad/kjell/fix/mongo_authz_crash
fix: mongodb authz crash
2023-03-13 10:46:49 +01:00
Stefan Strigler c1adf0de1f test(emqx_bridge): increase coverage by adding common edge cases 2023-03-13 10:40:47 +01:00
Stefan Strigler 478601ee41 fix(emqx_bridge): remove unreachable code paths 2023-03-13 10:40:47 +01:00
Stefan Strigler dba95ec0fa style(emqx_bridge): fix wording 2023-03-13 10:40:47 +01:00
Stefan Strigler 71ec77a2f2 fix(emqx_bridge): for node operation check if node is part of cluster
This fixes the case where we returned `501 NOT IMPLEMENTED` in the past.
2023-03-13 10:40:47 +01:00
Stefan Strigler 7124600a71 docs(emqx_bridge): add `start` as operation to bridge 2023-03-13 10:38:51 +01:00
Stefan Strigler 1bcc5623ed fix(emqx_bridge): check if bridge enabled before calling op 2023-03-13 10:38:49 +01:00
Stefan Strigler a325133391 fix(emqx_bridge): don't crash checking if bridge enabled 2023-03-13 10:37:28 +01:00
Stefan Strigler 80b81748df fix(emqx_bridge): handle bridge not found in call_operation 2023-03-13 10:37:28 +01:00
Stefan Strigler fb3d101b3a refactor(emqx_bridge): fix var names 2023-03-13 10:37:28 +01:00
Stefan Strigler e31f4d6091 refactor(emqx_bridge): add BAD_REQUEST macro and minor cleanups 2023-03-13 10:37:28 +01:00
lafirest e8cf604ad4
Merge pull request #9893 from lafirest/feat/filiter_banned
feat(banned): filter out banned messages from mqueue
2023-03-13 09:58:39 +08:00
ieQu1 e3595f2e79 chore(mria): Bump version to 0.4.0 2023-03-11 00:37:25 +01:00
Andrew Mayorov e571b602b8
chore: bump app versions + add changelog
Bump `emqx_plugin_libs` app version to 4.3.7.
2023-03-10 18:43:19 +03:00
Andrew Mayorov 781942cc4f
chore: drop unused macrodefs 2023-03-10 18:42:04 +03:00
Andrew Mayorov 0a7f6c7d03
fix(mysql): ensure proper escaping in batch inserts
Also hexencode non-utf8 binaries. This is essentially an heuristic.
We don't know column types in runtime, and there's no simple way
to find them out. Since we're already doing full binary scan during
escaping it should be cheap to bail out on non-utf8 strings and
hexencode them instead.

Also introduce separate function to highlight that this escaping
is MySQL-specific.
2023-03-10 18:42:04 +03:00
Andrew Mayorov fc37d9b3cd
fix(mysql): be explicit that batch queries are parameterless
So that mysql client won't attempt to prepare them automatically, thus
trashing the server's prepared statements table and making interaction
overall heavier.
2023-03-10 18:42:04 +03:00
Andrew Mayorov a86d06f043
chore: bump app versions following last merge-back 2023-03-10 16:44:15 +03:00
Ilya Averyanov 5650ad20f8
Merge pull request #10100 from savonarola/0308-fix-scram-auth
Fix channel crash for slow clients with enhanced authentication
2023-03-10 12:58:43 +02:00
Ivan Dyachkov 951e89134c Merge remote-tracking branch 'upstream/release-50' into 0310-merge-release-50-to-master 2023-03-10 08:48:32 +01:00
Zaiming (Stone) Shi 3f461f312a chore: bump version to e5.0.1 and v5.0.20 2023-03-09 18:34:26 +01:00
Zaiming (Stone) Shi 6ebc31a929
Merge pull request #10104 from JimMoen/refine-changes-for-release-50
Refine changes for release 50
2023-03-09 18:33:00 +01:00
Ilya Averyanov cba0287439 fix(mqtt): fix channel crash for slow clients with enhanced authn 2023-03-09 17:27:24 +02:00
Kjell Winblad aa57ea9ee1 fix: mongodb authz crash
This fixes a crash with an error in the log file (see below) that
happened when the MongoDB authorization module queried the database. The
reason is that the collection name that was sent to the mongodb
connection was an atom. This is fixed by making sure it is not an atom.

2023-03-08T17:16:34.215523+01:00 [error] msg: query_mongo_error, mfa:
emqx_authz_mongodb:authorize/4, line: 95, peername: 127.0.0.1:53212,
clientid: client123, collection: mqtt_acl, filter: #{username =>
<<"emqx_u">>}, reason: {resource_error,#{msg => #{error =>
{error,{error_cannot_parse_response,{op_msg_response,#{<<"code">> =>
73,<<"codeName">> => <<"InvalidNamespace">>,<<"errmsg">> => <<"Failed to
parse namespace element">>,<<"ok">> => 0.0}}}},id =>
<<"emqx_authz_mongodb:3">>,name => call_query,request =>
{find,mqtt_acl,#{username => <<"emqx_u">>},#{}},stacktrace =>
[{mc_connection_man,reply,1,[{file,"mc_connection_man.erl"},{line,123}],
...]}, reason => exception}}, resource_id: <<"emqx_authz_mongodb:3">>

Fixes: https://github.com/emqx/emqx/issues/9783
2023-03-09 16:01:23 +01:00
Zaiming (Stone) Shi 3463cd0d37 refactor: generate document related artifacts to _build/docgen 2023-03-09 09:32:03 +01:00
JimMoen 842cb7fadc
fix: debug log for rulesql parse failed
See also: PR#10059.
The other error logs will be printed in `emqx_rule_runtime:apply_rule/3`
2023-03-09 11:28:27 +08:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
Zaiming (Stone) Shi c773f5f5e1 chore: bump version to e5.0.1-rc.2 2023-03-08 13:55:37 +01:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Zaiming (Stone) Shi 158b4a95df
Merge pull request #10079 from qzhuyan/fix/william/i18n-broker_shared_subscription_strategy
fix: i18n desc of broker_shared_subscription_strategy
2023-03-07 21:57:07 +01:00
Zaiming (Stone) Shi 26b29185b2 test(emqx_bridge_webhook_SUITE): fix flakyness in test web server 2023-03-07 20:57:38 +01:00
Kjell Winblad 163b33ab28 test: remove unnecessary dependencies of ee apps 2023-03-07 20:57:38 +01:00
Kjell Winblad ca947e3e70 fix: lost messages when HTTP connection times out
When using async mode with the webhook bridge, queued messages that are
not fully processed when the connection times out could be lost. This
commit fixes this by letting the bridge return a recoverable_error when
this happen. The message send will then be retried in sync mode by the
emqx_resource_buffer_worker.

Fixes: https://emqx.atlassian.net/browse/EMQX-8974
2023-03-07 20:57:19 +01:00
Stefan Strigler 609cd01a35
Merge pull request #10085 from sstrigler/EMQX-8552-authorization-sources-type-status-move-shouldnt-exist-when-authorization-sources-type-doesnt-exist
emqx_authz API: return 404 for all requests on non existent source
2023-03-07 16:19:28 +01:00
Stefan Strigler b54f444263 fix(emqx_authz): return `404` for requests on non existent source 2023-03-07 13:51:06 +01:00
Stefan Strigler a7605fba94 test(emqx_authz): use snabbkaffe:retry instead of timer:sleep
also use emqx_json rather than jiffy or jsx directly
2023-03-07 13:49:46 +01:00
ieQu1 6f88cb7d9e chore(ekka): Bump version to 0.14.3 2023-03-07 12:07:07 +01:00
zhongwencool 0a5b221984
Merge pull request #10055 from zhongwencool/fix-max-awaiting-rel
fix: update max_awaiting_rel in session
2023-03-07 18:37:20 +08:00
SergeTupchiy 9949c87b91
Merge pull request #10069 from SergeTupchiy/ci/EMQX-9055-configuration-item-flush-time-interval-used-for-stats-d-integration-not-work
fix: EMQX-9055 configuration item flush time interval used for statsd integration not work
2023-03-07 11:43:57 +02:00
William Yang 2b1cf2fa52
Merge pull request #10058 from qzhuyan/dev/william/deprecated-unsupp-quic-TLS-options
fix: deprecate unsupp quic TLS options
2023-03-07 09:29:52 +01:00
William Yang 422f8e3be3
Merge pull request #10078 from qzhuyan/fix/william/inval-listener-options-segfault
fix: inval listener option cause segfault
2023-03-07 09:26:27 +01:00
Zaiming (Stone) Shi 133734b345 chore: bump version number to e5.0.1-rc.1 and v5.0.20 2023-03-07 08:41:13 +01:00
Thales Macedo Garitezi eef65fba60 fix(buffer_worker): handle `request_timeout = infinity` case
The current schema allows `infinity` for `request_timeout`, so we have
to take that into account.  It's not currently possible to set
`batch_time = infinity`, so there's no need to treat that case.
2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi 18ab7ed197 chore: bump app vsns 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi 0e707e837f docs(buffer_worker): improve description of `request_timeout` 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi e17ad320ee fix(bridge): do not log in converter 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi e9d3fc511f chore(buffer_worker): change default `batch_time` to 0 and improve docs 2023-03-06 15:31:28 -03:00
Thales Macedo Garitezi 2fa4e8e21a
Merge pull request #10068 from thalesmg/fix-flaky-mqtt-bridge-test
test: fix flaky mqtt bridge test
2023-03-06 13:06:37 -03:00
William Yang 7c65f6f9a9 fix: i18n desc of broker_shared_subscription_strategy 2023-03-06 15:09:56 +01:00
William Yang abb5271177 test(quic): start listener with invalid parameter 2023-03-06 14:58:35 +01:00
Stefan Strigler a81d1a4618
Merge pull request #10066 from sstrigler/EMQX-8932-bridge-api-reports-500-for-most-errors-when-starting-a-resource
fix: return human readable error message for most common cases
2023-03-06 14:13:34 +01:00
Thales Macedo Garitezi f95a30ae89 fix(webhook): convert `request_timeout`s in root and resource_opts 2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi 167b7a212f refactor(buffer_worker): avoid starting 0-time timers 2023-03-06 10:12:38 -03:00
Thales Macedo Garitezi e9ffabf936 fix(buffer_worker): add batch time automatic adjustment
To avoid message loss due to misconfigurations, we adjust `batch_time`
based on `request_timeout`.  If `batch_time` > `request_timeout`, all
requests will timeout before being sent if the message rate is low.
Even worse if `pool_size` is high.  We cap `batch_time` at
`request_timeout div 2` as a rule of thumb.
2023-03-06 10:12:38 -03:00
William Yang 27c5389fdc fix(quic): inval listener option casue segfault
bump quicer to 0.0.113
2023-03-06 14:05:33 +01:00
Thales Macedo Garitezi 8fbb948b6f test: fix flaky mqtt bridge test
Sometimes, this test fails because the metrics are still in the
inflight phase.
2023-03-06 09:09:33 -03:00
Stefan Strigler d0ea7f4647 fix(emqx_authz): check if type param matches type in body 2023-03-06 11:10:31 +01:00
Zhongwen Deng f498a3538b fix: update max_awaiting_rel in session 2023-03-06 17:06:40 +08:00
Zaiming (Stone) Shi a4aece396a
Merge pull request #10052 from zmstone/0301-optimize-boot-error-log
0301 optimize boot failure logs
2023-03-06 08:32:15 +01:00
Kjell Winblad a638cc6566
Merge pull request #9985 from kjellwinblad/kjell/feat/clickhouse_bridge_2/EMQX-8391
feat: add clickhouse database bridge
2023-03-04 07:05:05 +01:00
Serge Tupchii b3907128e8 fix: set statsd flush_time_interval = max(flush_time_interval, sample_time_interval)
flush_time_interval is used to calculate statsd sampling rate:
    rate = sample_time_interval / flush_time_interval
This means that flush_time_interval must always be greater than (or equal to)
sample_time_interval, otherwise, the sampling rate will be invalid (> 1).

Relates to EMQX-9055
2023-03-03 22:24:00 +02:00