yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,835 Commits 393 Branches 684 Tags 74 MiB
ea8d54fd8b
Commit Graph

9188 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
063913f245 fix(session): remove recently added iterators field from #session{} record 
Fixes https://emqx.atlassian.net/browse/EMQX-10945

This could lead to `badrecord` errors being raised if a takeover were to happen during a
rolling cluster upgrade, as the old nodes could receive a record with more fields than
expected.
 2023-09-11 14:46:12 -03:00
zhongwencool
887a74a6cb fix: telemetry report error on windows 2023-09-11 16:48:30 +08:00
Zaiming (Stone) Shi
7010ac358c chore: bump release version to v5.2.0 2023-09-08 16:31:52 +02:00
Andrew Mayorov
3d133801af
fix(router): use ets:new/2 options compatible with OTP 24 
While essentially keeping the same effect.
 2023-09-08 18:04:29 +04:00
Thales Macedo Garitezi
e089fda260
Merge pull request #11568 from thalesmg/republish-props-m-20230905 
feat(republish): allow templating mqtt properties
 2023-09-07 16:43:00 -03:00
Zaiming (Stone) Shi
f3bebb0ba2 Merge remote-tracking branch 'origin/master' into 0907-sync-master-to-release-52 2023-09-07 18:51:30 +02:00
Thales Macedo Garitezi
68293231b8 test: add cases for more complex placeholders 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi
014bc64d3b chore: ensure general prop is a binary 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi
02b8bbf76a chore: drop templating support for Subscription-Identifier 
This is highly-dependent on the session state, as is `Topic-Alias`.
 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi
47bd19dee4 chore: lower error log messages down to debug 2023-09-07 13:40:03 -03:00
Thales Macedo Garitezi
080cb73da1 fix: handle badmap inside emqx_placeholder:proc_tmpl 2023-09-07 13:40:03 -03:00
Zaiming (Stone) Shi
db2ec55aa0 chore: bump release version to official e5.2.0 2023-09-07 09:40:15 +02:00
Zaiming (Stone) Shi
7c2f87fabe test: merge broker and router boot modules 2023-09-06 21:36:16 +02:00
Thales Macedo Garitezi
bb55b04d46 refactor: inline fn clauses, improve error logging 2023-09-06 13:54:40 -03:00
Thales Macedo Garitezi
c9100c7591
refactor: improve logging information and descriptions 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-09-06 13:33:59 -03:00
Thales Macedo Garitezi
78c5a779d7 feat(republish): allow templating mqtt properties 
Fixes https://emqx.atlassian.net/browse/EMQX-10912
 2023-09-06 09:54:02 -03:00
Kjell Winblad
7fe704b6dd
Merge pull request #11563 from kjellwinblad/kjell/fix/release-52/EMQX-10775 
fix: update gpb library to fix type error
 2023-09-06 14:47:23 +02:00
Zaiming (Stone) Shi
18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Zaiming (Stone) Shi
e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
Thales Macedo Garitezi
954b73a9aa
Merge pull request #11533 from thalesmg/ds-unsubscribe-m-20230828 
feat(ds): close iterators when handling `UNSUBSCRIBE` packets
 2023-09-05 09:47:17 -03:00
ieQu1
01a128878f chore(mria): Bump mria to 0.6.1 2023-09-05 11:08:32 +02:00
Andrew Mayorov
d26995a2aa
Merge pull request #11524 from keynslug/ft/EMQX-10713/unified-route-tab 
feat(router): add routing storage schema with 2 tables
 2023-09-05 12:38:00 +04:00
Kjell Winblad
2b2e84b83c fix: update gpb library to fix type error 
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
 2023-09-05 09:52:17 +02:00
Thales Macedo Garitezi
23d63f5e01 refactor: fork clientid type for emqx_ds:session_id/0 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi
60ae3c15c8 refactor: disregard impossible case 2023-09-04 15:38:25 -03:00
Thales Macedo Garitezi
2cfe5ba057 fix(plugins_api): fix response types and test 2023-09-04 14:37:25 -03:00
Thales Macedo Garitezi
02c3d6a0a7 fix(data_import): rm duplicate import call and fix test 2023-09-04 14:19:59 -03:00
Thales Macedo Garitezi
73d4248334 test(cth_cluster): always allocate listener ports 2023-09-04 11:22:26 -03:00
Andrew Mayorov
9f0f218387
test(router): simplify cluster-related testcase in helper testsuite 2023-09-04 18:05:59 +04:00
Andrew Mayorov
7d26b7bc1a
fix(router): emit clearer error when conflicting schemas in use 2023-09-04 16:37:38 +04:00
Thales Macedo Garitezi
d33ad8a46f feat: cluster purge (r5.2) 
Port of https://github.com/emqx/emqx/pull/11447

targeting `release-52`

Fixes https://emqx.atlassian.net/browse/EMQX-10763
 2023-09-04 09:15:28 -03:00
Andrew Mayorov
893f69617a
chore(test): drop obsolete TODOs 2023-09-04 14:40:49 +04:00
Andrew Mayorov
f0a0c7d4b1
fix(cthsuite): avoid duplicate application env entries 
Because `application:set_env/1` complains loudly if there are
duplicates.
 2023-09-04 14:40:49 +04:00
Andrew Mayorov
5024304bf9
fix(router): wait for tables replicate before choosing schema vsn 2023-09-04 14:40:48 +04:00
Ivan Dyachkov
24230a64df chore: bump app versions 2023-09-04 11:39:21 +02:00
Ivan Dyachkov
b2f57636bd Merge remote-tracking branch 'upstream/release-51' into 0904-sync-release-51 2023-09-04 11:34:22 +02:00
Kjell Winblad
52546ac50f
Merge pull request #11522 from kjellwinblad/kjell/fix/EMQX-10778 
fix: bad error message when rule engine schema name is too long
 2023-09-04 11:05:55 +02:00
Ilya Averyanov
240afecd69
Merge pull request #11487 from savonarola/0821-optimize-bcrypt 
feat: reduce bcrypt rounds to a usable value
 2023-09-04 10:12:40 +03:00
zhongwencool
ce571d892e chore: ping opentelemetry v1.3.0-emqx 2023-09-04 09:34:09 +08:00
Serge Tupchii
ed9afe3345 refactor(emqx_connector): use hocon_schema:override/2 to make pgsql 'username' field required 2023-09-01 19:56:07 +03:00
Serge Tupchii
607705518b test(emqx_bridge_cassandra): add connector test case for Cassandra configured without authentication 2023-09-01 19:56:07 +03:00
Serge Tupchii
f0c75d97e1 chore: upgrade emqx_http_lib to 0.5.3 
0.5.3 release includes a fix to parse <host>:<port> URLs using the default http scheme.
 2023-09-01 19:56:07 +03:00
Serge Tupchii
c3b3460173 fix(emqx_oracle): make username a required field in emqx_oracle_schema 2023-09-01 19:56:07 +03:00
Andrew Mayorov
4ab5f8374b
chore(schema): mark routing storage schema readOnly 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-09-01 19:58:31 +04:00
Andrew Mayorov
ebf13c90ea
test(router): make sort stable in e2e routing testcase 2023-09-01 19:48:56 +04:00
Ilya Averyanov
90156befb5 feat: improve bcrypt usability 
* limit salt rounds to usable values
* update bcrypt library to enable concurrent bcrypt hash calculation
 2023-09-01 17:36:11 +03:00
Serge Tupchii
c54527857a fix(emqx_bridge_sqlserver): use empty password by default as it is not a required field 2023-09-01 16:57:35 +03:00
Serge Tupchii
1bf86250dd fix(emqx_bridge_cassandra): allow cassandra bridge without username/password 
Cassandra can be used without credentials, if it is configured with AllowAllAuthenticator (default).
 2023-09-01 16:57:35 +03:00
Andrew Mayorov
545f1c84a6
fix(cth): do not allocate ports for emqx app by default 
This causes tricky and impressively hard to track side effects down
the line. Namely, loading `emqx_schema` _after_ `emqx_conf_schema`
(as part of cluster node startup sequence) leads to a couple of
schema root rewrites, because `emqx_schema` defines similar config
roots yet slightly differently (e.g. `authorization`).
 2023-09-01 17:23:34 +04:00
Andrew Mayorov
0a879bffd1
test(router): add e2e testcase for cluster-wide routing 
Which verifies that both router storage schemas work correctly.
 2023-09-01 13:49:12 +04:00
Andrew Mayorov
eb0385a28f
chore(router): separate test-only function exports 2023-09-01 13:49:11 +04:00
zhongwencool
3d859453f4
Merge pull request #11554 from lafirest/chore/exhook_prop 
chore: shutdown reason will never be a UTF8 atom
 2023-09-01 16:53:26 +08:00
Kjell Winblad
3bb65e6b05
Merge pull request #11542 from kjellwinblad/kjell/fix/EMQX-10775 
fix: update gpb library to fix type error
 2023-09-01 10:07:03 +02:00
firest
53940754b7 chore: shutdown reason will never be a UTF8 atom 2023-09-01 14:17:33 +08:00
Paulo Zulato
17fe406605
Merge pull request #11551 from paulozulato/refactor-connector-config-callback 
refactor: remove BridgeName from connector_config callback
 2023-08-31 17:05:49 -03:00
Thales Macedo Garitezi
bf0f77b6dc
Merge pull request #11548 from thalesmg/fix-plugin-order-update-r52-20230830 
fix(plugins): update plugin order on whole cluster
 2023-08-31 16:18:28 -03:00
Paulo Zulato
7b607c3300 refactor: remove BridgeName from connector_config callback 
As `bridge_name` is already in the Config parameter, this callback
doesn't need to have it on API anymore.

Fixes https://github.com/emqx/emqx/pull/11546/files#r1310778734
 2023-08-31 11:01:49 -03:00
Zaiming (Stone) Shi
c20ba0572a chore: bump to 5.2.0-alpha.4 2023-08-31 15:47:39 +02:00
Zaiming (Stone) Shi
876d539336 Merge remote-tracking branch 'andrew/ft/EMQX-10713/unified-route-tab' into 0831-make-use-of-new-routing-table 2023-08-31 15:46:33 +02:00
Andrew Mayorov
fb094e1d47
test(router): avoid testsuite collisions 2023-08-31 13:36:01 +04:00
Paulo Zulato
727fd296ee
Merge pull request #11546 from paulozulato/fix-kafka-producer-unique-name 
fix(kafka): avoid producer name collision between Kafka and AEH bridges
 2023-08-30 18:08:54 -03:00
Thales Macedo Garitezi
4dfbc859f9 fix(plugins): update plugin order on whole cluster 
Fixes https://emqx.atlassian.net/browse/EMQX-10879
 2023-08-30 16:39:52 -03:00
Paulo Zulato
ee77976424 fix(kafka): avoid producer name collision between Kafka and AEH bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-30 14:48:10 -03:00
Andrew Mayorov
2da604dfa4
test(mgmt): simplify flaky t_verify_imported_mnesia_tab_on_cluster 
And also rest of the test setup in the suite.
 2023-08-30 20:24:18 +04:00
Andrew Mayorov
b1defa29d7
feat(ruleeng): avoid storing whole rules in topic index 
Because it doesn't really give any benefit, but wastes memory by
duplication.
 2023-08-30 20:07:56 +04:00
Andrew Mayorov
6b9cb06334
fix(router): add / refine some comments and log messages 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-08-30 19:51:31 +04:00
Thales Macedo Garitezi
922ca5e141 feat(ds): close iterators when handling UNSUBSCRIBE packets 
Fixes https://emqx.atlassian.net/browse/EMQX-9742
 2023-08-30 11:14:28 -03:00
Thales Macedo Garitezi
9c6dd30f44 feat(session): store iterator ids in session record 2023-08-30 11:13:55 -03:00
Paulo Zulato
dfcede8794 fix: increment matched counter when bridge is unhealthy 
Fixes https://emqx.atlassian.net/browse/EMQX-10767
 2023-08-30 10:52:53 -03:00
Andrew Mayorov
f8923ab419
Merge pull request #11539 from keynslug/ft/cth-desc-workdir 
feat(cthsuite): add function to determine workdir of testrun
 2023-08-30 17:15:55 +04:00
Thales Macedo Garitezi
1cab687153
Merge pull request #11540 from thalesmg/fix-cert-path-utf8-r52-20230829 
fix(bridge): validate bridge name before attempting to convert certificates
 2023-08-30 09:45:55 -03:00
lafirest
8b3a9072f9
Merge pull request #11532 from lafirest/fix/frame_parse 
fix(frame): improve some error reasons when parsing invalid packet
 2023-08-30 19:10:45 +08:00
Kjell Winblad
d911f7fbea fix: update gpb library to fix type error 
This fixes a bug in the protobuf schema registry functionality. Before
this fix one would get a badarith error if one tried to assign a float
value to an uint64 field. However, this commit fixes this by upgrading
gpb so we instead will get a gpb_type_error which is what we want.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10775
 2023-08-30 11:12:33 +02:00
JianBo He
9fe4382d98
Merge pull request #11530 from JimMoen/chore-bump-hstreamdb-driver 
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
 2023-08-30 14:37:17 +08:00
firest
804443ba40 chore: fix typos 2023-08-30 11:22:09 +08:00
JianBo He
06fad14d07
Merge pull request #11536 from zhongwencool/comma_separated_binary 
chore: ciphers should allow empty space(comma_separated_binary)
 2023-08-30 10:20:06 +08:00
Ilya Averyanov
2ef60db147
Merge pull request #11481 from savonarola/0817-simplify-async-responses 
chore(ft): refactor async reply mechanism
 2023-08-30 01:20:09 +03:00
Ilya Averyanov
54ac4a8527 chore(ft): tidy up the code according to the review 2023-08-29 23:21:36 +03:00
Thales Macedo Garitezi
0f297ffef4 fix(bridge): validate bridge name before attempting to convert certificates 
Fixes https://emqx.atlassian.net/browse/EMQX-10865
 2023-08-29 16:32:01 -03:00
Andrew Mayorov
0e770bdc95
test: switch emqx_broker_SUITE to use new cth tooling 2023-08-29 22:05:36 +04:00
Andrew Mayorov
3268093881
feat(cth): add module-level documenation 2023-08-29 22:05:36 +04:00
Andrew Mayorov
f57d16ba13
feat(cthsuite): add function to determine workdir of testrun 
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
 2023-08-29 21:34:01 +04:00
Andrew Mayorov
063d6200c8
feat(router): enable using 2 tables to store routes 
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.

Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
 2023-08-29 17:41:59 +04:00
Andrew Mayorov
270fd107b2
feat(triesearch): allow reusing filter tokenization result 
As a kind of micro-optimization.
 2023-08-29 17:31:26 +04:00
zhongwencool
501456efec
chore: more safe with iolist_to_binary 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-08-29 17:10:43 +08:00
zhongwencool
4e7ba5d35f chore: ciphers should allow empty space(comma_separated_binary) 2023-08-29 11:34:40 +08:00
firest
6722722522 fix(frame): improve some error reasons when parsing invalid packet 2023-08-29 10:53:20 +08:00
zhongwencool
4245a4d8c6 fix: authz clean-cache clientid always return not_found 2023-08-29 08:17:47 +08:00
Paulo Zulato
cc3ba18734 fix: increment dropped message counter when bridge is unhealthy 
Fixes https://emqx.atlassian.net/browse/EMQX-10767
 2023-08-28 19:47:11 -03:00
Paulo Zulato
2c89be04d1
Merge pull request #11527 from paulozulato/fix-kafka-header-template-handling 
fix(kafka): fix template processing for header
 2023-08-28 18:55:33 -03:00
Paulo Zulato
0b86f04bae fix(kafka): fix template processing for header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-28 17:28:05 -03:00
Ilya Averyanov
279895b8fd chore(ft): add read/write concurrency to emqx_ft_async_reply tabs 2023-08-28 22:46:30 +03:00
SergeTupchiy
c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log 
fix(emqx_trace): don't download empty trace log file
 2023-08-28 22:17:00 +03:00
Serge Tupchii
0072749143 fix(emqx_trace): don't download empty trace log file 
Closes: EMQX-10274
 2023-08-28 19:25:55 +03:00
Thales Macedo Garitezi
5908b69353 chore: hide cluster purge from cli and api for now 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi
f988de4ff4 feat(purge): clear delayed messages 2023-08-28 10:23:21 -03:00
Thales Macedo Garitezi
9bded07834 feat: cluster purge 
Fixes https://emqx.atlassian.net/browse/EMQX-10763
 2023-08-28 10:23:20 -03:00
Thales Macedo Garitezi
dec21ffc95
Merge pull request #11362 from thalesmg/ds-handle-sub-20230725 
feat(ds): open iterators when handling `SUBSCRIBE` packets
 2023-08-28 10:21:28 -03:00
Andrew Mayorov
2d931a0512
test(evict): do not disable router module in cluster nodes 2023-08-28 14:07:46 +04:00
Andrew Mayorov
8d2ebdea7e
fix(router): generalize config option and make effects visible 2023-08-28 14:07:46 +04:00
Andrew Mayorov
e85789306b
chore(router): drop test-only has_routes/1 
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
 2023-08-28 14:07:46 +04:00
Andrew Mayorov
5d51687dbf
feat(router): add unified routing table config option 2023-08-28 14:07:45 +04:00
Andrew Mayorov
33e5e1ba57
feat(router): add unified routing table 2023-08-28 14:07:44 +04:00
Andrew Mayorov
166375a000
fix(topicidx): make get_record/2 simpler to use in concurrent env 
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
 2023-08-28 13:52:59 +04:00
Andrew Mayorov
84e40fb6fe
test(trie): add more involved route add + delete case 
That shows how current local trie implementation breaks because
of lack of refcounting.
 2023-08-28 13:52:59 +04:00
Andrew Mayorov
dcb63440bc
refactor(mgmt): avoid dealing with router tab directly 
Instead, contain all this behind `emqx_router` module interface.
 2023-08-28 13:52:59 +04:00
Andrew Mayorov
9eccfa0909
refactor(router): isolate cleanup logic in router module 2023-08-28 13:52:59 +04:00
JimMoen
f3467f44b2
chore: bump hstreamdb_erl driver to 0.4.5+v0.16.1 2023-08-28 17:28:50 +08:00
Ilya Averyanov
4488e9e591 chore(ft): stop hook chain when doing actual handling 2023-08-28 11:57:44 +03:00
Ilya Averyanov
39a48179ea chore(emqx_channel): use macros for reply construction 2023-08-28 11:57:44 +03:00
Ilya Averyanov
b8cacd2833 chore(ft): add tests for async reply registry 2023-08-28 11:57:44 +03:00
Andrew Mayorov
b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3 
feat(topicidx): iterate on trie search implementation
 2023-08-28 12:24:16 +04:00
zhongwencool
053b7fb94a test: add more test for conf_cli 2023-08-28 11:14:36 +08:00
zhongwencool
d8be248a3d fix: improve the suggest msg for update conf failed 2023-08-28 08:53:23 +08:00
Ilya Averyanov
b0d4a22aa8 chore(ft): refactor async reply mechanism 2023-08-27 21:54:23 +03:00
zhongwencool
b874926de7
Merge pull request #11514 from zhongwencool/stop-load-application-before-reboot 
fix: stop otel deps appication before reboot
 2023-08-27 20:46:31 +08:00
Paulo Zulato
2bb415f897
Merge pull request #11526 from paulozulato/fix-validator-return-msg 
fix: flatten error message on resource validator
 2023-08-25 17:27:31 -03:00
Thales Macedo Garitezi
33a0048155 refactor: move logic to ensure_iterator 2023-08-25 15:24:53 -03:00
Paulo Zulato
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling 
fix(kafka): fix result handling when sending message with invalid header
 2023-08-25 14:57:09 -03:00
Paulo Zulato
42877e282d fix: flatten error message on resource validator 
Fixes https://emqx.atlassian.net/browse/EMQX-10864
 2023-08-25 13:53:52 -03:00
Thales Macedo Garitezi
f15f59650d test: rm obselete workaround code 2023-08-25 13:49:33 -03:00
Paulo Zulato
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id 
Fix Kafka/AEH ClientId uniqueness
 2023-08-25 10:14:44 -03:00
Andrew Mayorov
d5cff533e3
chore(topicidx): drop TODO comment 2023-08-25 16:44:03 +04:00
Andrew Mayorov
ca59a87d47
chore(topicidx): refine example of wildcard compare 2023-08-25 16:44:03 +04:00
Zaiming (Stone) Shi
558402a68e
chore(topic_index): add topic validation 2023-08-25 16:44:03 +04:00
Andrew Mayorov
33ed53bb6a
refactor(topicidx): simplify compare/3 further 
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
 2023-08-25 16:34:45 +04:00
Andrew Mayorov
fc37d235c7
refactor(topicidx): simplify seek instructions 
Which also avoids comparing filter tail repeatedly when evaluating
it.
 2023-08-25 15:24:06 +04:00
Kjell Winblad
c65db82b07 fix: bad error message when rule engine schema name is too long 
Fixes:
https://emqx.atlassian.net/browse/EMQX-10778
 2023-08-25 11:02:38 +02:00
zhongwencool
c8af5a702c test: remove connection_closed from test 2023-08-25 15:53:35 +08:00
zhongwencool
fc1738188e fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero 2023-08-25 15:18:34 +08:00
lafirest
ba9cb0a1ed
Merge pull request #11512 from lafirest/fix/ldap_def_port 
fix(ldap): expose request_timeout to prevent infinite hang
 2023-08-25 10:22:44 +08:00
zhongwencool
8ab6eb5638 test: start emqx_opentelemetry in emqx_machine SUITE 2023-08-25 10:09:00 +08:00
Andrew Mayorov
cf45e80c71
feat(topicidx): iterate on trie search implementation 
This improves matching performance and decreases GC pressure on
synthetic workloads.
 2023-08-25 01:47:11 +04:00
Paulo Zulato
535c7f8b43 fix(kafka): avoid ClientId collision between Kafka and Azure bridges 
Fixes https://emqx.atlassian.net/browse/EMQX-10860
 2023-08-24 18:46:22 -03:00
Thales Macedo Garitezi
d9a5a9ea2b
Merge pull request #11513 from thalesmg/kafka-fix-ts-template-r52-20230824 
fix: use correct timestamp template for kafka and remove such fields from azure event hub producer
 2023-08-24 16:26:52 -03:00
Thales Macedo Garitezi
016ae0524f fix(aeh_producer): remove timestamp template field 
Fixes https://emqx.atlassian.net/browse/EMQX-10847

Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.

Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type

Example message consumed from AEH:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test0",
  "ts" => 1692879703006,
  "ts_type" => "append",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Note the ts_type above is append.

Example message from a Kafka broker whose ts type is create:
```
%{
  "headers" => %{},
  "key" => "",
  "offset" => 4,
  "topic" => "test-topic-three-partitions",
  "ts" => 1692881883668,
  "ts_type" => "create",
  "value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```

Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
 2023-08-24 14:20:42 -03:00
Thales Macedo Garitezi
0381ac0410 fix(kafka_producer): use correct timestamp template field 
Fixes https://emqx.atlassian.net/browse/EMQX-10847
 2023-08-24 14:20:42 -03:00
Kjell Winblad
9fe5080705 build: bump versions before release 2023-08-24 16:46:27 +02:00
zhongwencool
c215fe3736 fix: stop otel deps appication before reboot 2023-08-24 22:14:14 +08:00
Paulo Zulato
60e6217496 fix(kafka): fix result handling when sending message with invalid header 
Fixes https://emqx.atlassian.net/browse/EMQX-10846
 2023-08-24 10:41:42 -03:00
Thales Macedo Garitezi
e5041de9cc fix(gcp_consumer): handle 401 errors 
Fixes https://emqx.atlassian.net/browse/EMQX-10852
 2023-08-24 09:03:34 -03:00
Zaiming (Stone) Shi
62423b0b12 refactor(topic_index): less special handling for leading $ words 2023-08-24 13:30:10 +02:00
Zaiming (Stone) Shi
a30d87e14f refactor(topic_index): remove more unnecessary next calls 
also avoid using records (setelement) for recursive return values
 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi
a1e6635614 refactor(topic_index): no forced ceiling entry in index table 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi
f4c8c6be55 refactor(topic_index): optimize trie-search performance 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi
6b152b3cb7 test: add more debug output 2023-08-24 12:25:04 +02:00
Zaiming (Stone) Shi
ae094e363c chore: fix a typo in log message 2023-08-24 12:25:04 +02:00
firest
6085442958 fix(ldap): expose request_timeout to prevent infinite hang 2023-08-24 16:50:38 +08:00
zhongwencool
2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics 
Prometheus metrics
 2023-08-24 12:06:32 +08:00
zhongwencool
8d2be27ade chore: add changlog for 11466 2023-08-24 06:46:50 +08:00
zhongwencool
84dd3b83bb fix: hocon converter not working when make_serializable is true(check_plain) 2023-08-24 06:46:47 +08:00
zhongwencool
f72ba6819f
Merge pull request #11498 from zhongwencool/fix-otel-stop-error-log 
fix: stop otel deps application in terminate crashed
 2023-08-24 06:40:24 +08:00
zhongwencool
d287b736ec chore: add changelog for 11497 2023-08-23 17:02:30 +08:00
zhongwencool
58be7f709b fix: stop otel deps application in terminate crashed 2023-08-23 16:45:41 +08:00
zhongwencool
6bb15541c7 feat: rename olp metrics to overload_protection metrics 2023-08-23 15:25:54 +08:00
zhongwencool
b9a97923c4 feat: add some authz/connection missing metrics 2023-08-23 14:47:53 +08:00
zhongwencool
141676185d feat: add authentication metrics to prometheus 2023-08-23 14:15:16 +08:00
zhongwencool
4928c69758 feat: add authorization metrics to prometheus 2023-08-23 14:11:59 +08:00
zhongwencool
df9f8da178 feat: add overload_protection metrics to prometheus 2023-08-23 14:07:37 +08:00
zhongwencool
188a81913e test: add prometheus api test 2023-08-23 11:23:43 +08:00
zhongwencool
ae10415fc3 feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security 2023-08-23 09:03:22 +08:00
Paulo Zulato
61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators 
fix(kinesis): set validators for static constraints
 2023-08-22 18:28:35 -03:00
Paulo Zulato
20b46091f9 fix(kinesis): set validators for static constraints 
Fixes https://emqx.atlassian.net/browse/EMQX-10833
 2023-08-22 14:42:44 -03:00
Thales Macedo Garitezi
f007b44426 fix(data_import): rm duplicate import call and fix test 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
ee2897e5de test(refactor): move test to integration tests dir 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
c1f49abad2 test: fix inter-suite flakiness 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
dad27091be test: rm custom option 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
dbfacae283 fix: reinstate transactions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
65085d012b refactor: rename fn 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
3344bfb0bd refactor: rm emqx_ds_replay 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
c46b8de938 test: remove unused things, refactor some functions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
e8c73b06e1 docs: add comment about future test failure 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
6de0bbe76a test(refactor): always allocate listeners for emqx app 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
a15405a800 test: fix assertions 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
c74abe79d0 refactor: reduce arity 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
021755b82b refactor: rm iterators from DS #session{} record 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
e4e88ebf36 test: add scenario for node stopping midway during subscribe 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
3239f5ac5b feat: rm unnecessary transactions, use separate table for iterator references 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
8eab389ae1 perf: avoid unnecessary transaction 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
c28c6d1b7e fix: ensure iterator is opened 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
33ddbe80ad refactor: remove persistence leakeage from emqx_cm level 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
e8d7bb9a67 refactor: rename module 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
9463e271c0 feat(ds): open iterators when handling SUBSCRIBE packets 
Fixes https://emqx.atlassian.net/browse/EMQX-9741
 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
ca697a4e14 fix: rename emqx_ds{,_replay}:replay_id() 2023-08-22 13:29:15 -03:00
Thales Macedo Garitezi
45e2e687e5 fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication 
Fixes https://emqx.atlassian.net/browse/EMQX-10843
 2023-08-22 12:42:48 -03:00
Kjell Winblad
3ffbb7decf fix: HTTP API /api/v5/publish schema 
The schema for the /api/v5/publish HTTP API endpoint was incorrect.
For 400 (Bad Request) error it cannot return a list but the incorrect
schema declared that the response could include a list.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10837
https://github.com/emqx/emqx/issues/11488
 2023-08-22 15:21:20 +02:00
lafirest
7b8a599d17
Merge pull request #11486 from lafirest/fix/ldap_autn_wrong_pw 
fix(ldap): Return correct reason code when password is wrong
 2023-08-22 18:53:17 +08:00
lafirest
f8f39bf223
Merge pull request #11490 from lafirest/fix/absent_pw 
fix(authn): quickly return when the password is absent in password-based authentication
 2023-08-22 18:52:58 +08:00
firest
e20b804e6b chore: update changes && test cases 2023-08-22 17:41:08 +08:00
firest
184e03387a fix(ldap): fix status detection and auto reconnecting errors 2023-08-22 17:06:00 +08:00
Kinple
663d03112e
Merge pull request #11476 from Kinplemelon/kinple/for-upgrade-dashboard 
fix: set the `endpoint` in kinesis bridge as a required field
 2023-08-22 14:28:40 +08:00
firest
b0c5b447a4 fix(authn): quickly return when the password is absent in password-based authentication 2023-08-22 11:49:32 +08:00
JimMoen
13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support 
Hstreamdb tls support
 2023-08-22 10:40:54 +08:00
firest
037b0a7599 fix(ldap): Return correct reason code when password is wrong 2023-08-22 09:42:04 +08:00
Paulo Zulato
4e4c93a758
Merge pull request #11469 from paulozulato/feat-redis-auth-username 
Add support for AUTH with username on Redis
 2023-08-21 16:44:21 -03:00
Thales Macedo Garitezi
74d4a8325e
Merge pull request #11470 from thalesmg/gcp-consumer-rm-worker-pool-size-r52-20230817 
fix(gcp_consumer): remove unused resource_opts fields
 2023-08-21 14:49:59 -03:00
Thales Macedo Garitezi
bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816 
fix(bridge): make dryrun health check timeout more malleable
 2023-08-21 14:49:46 -03:00
Ivan Dyachkov
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727 
fix: rule SQL mongo_date function should return a string in test mode
 2023-08-21 16:20:59 +02:00
Thales Macedo Garitezi
4163f06611
Merge pull request #11480 from thalesmg/rule-funcs-func-clause-r52-20230818 
fix(rule_engine): capture function_clause errors
 2023-08-21 11:16:25 -03:00
Kinplemelon
443d44aaa3 fix: set the endpoint in kinesis bridge as a required field 2023-08-21 11:49:57 +08:00
JimMoen
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0 2023-08-21 10:40:51 +08:00
JimMoen
649647190e
test: new batch mechanism of hstreamdb_erl 2023-08-21 10:40:51 +08:00
JimMoen
5527edf442
chore: bump hstreamdb_erl to 0.4.3+v0.16.1 2023-08-21 10:40:36 +08:00
Ilya Averyanov
ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor 
Auth refactor stage 1
 2023-08-21 00:24:15 +03:00
Zaiming (Stone) Shi
01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7 
chore: upgrade Kafka client wolff to 1.7.7
 2023-08-20 10:14:13 +02:00
Paulo Zulato
05fdf52d8e fix(redis): set bridge as unhealthy on authentication error 2023-08-18 14:51:03 -03:00
Paulo Zulato
fc35c74eab fix(redis): use eredis_secret to avoid password leaking 2023-08-18 14:51:03 -03:00
Paulo Zulato
2c458b62f5 feat(redis): add support for username on AUTH command 
Fixes https://emqx.atlassian.net/browse/EMQX-9911
 2023-08-18 14:50:06 -03:00
Thales Macedo Garitezi
66dec69d09 fix(rule_engine): capture function_clause errors 
Fixes https://emqx.atlassian.net/browse/EMQX-10798
 2023-08-18 13:51:19 -03:00
Thales Macedo Garitezi
befc260555 fix(gcp_consumer): remove unused resource_opts fields 
Fixes https://emqx.atlassian.net/browse/EMQX-10789
 2023-08-17 17:02:02 -03:00
Thales Macedo Garitezi
e48bfb52bb Merge branch 'release-52' into sync-r52-master-20230817 2023-08-17 17:01:04 -03:00
Paulo Zulato
afffdbbaa1 fix(kinesis): replace default payload template 
Fixes https://emqx.atlassian.net/browse/EMQX-10766
 2023-08-17 17:00:14 -03:00
Ivan Dyachkov
e2b9d905d7
Merge pull request #11465 from id/0718-e5.2.0-alpha.3 2023-08-17 20:59:13 +02:00
Andrew Mayorov
75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2 
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
 2023-08-17 20:43:09 +04:00
Ivan Dyachkov
a0c2fe8cc1 chore: e5.2.0-alpha.3 2023-08-17 17:55:56 +02:00
Thales Macedo Garitezi
ebecbd1545 fix(bridge): make dryrun health check timeout more malleable 
Fixes https://emqx.atlassian.net/browse/EMQX-10773

- Makes the timeout for probing a bridge more malleable to account for differences between
  each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
  slowness/throttling.
 2023-08-17 09:21:19 -03:00
Thales Macedo Garitezi
e3572eb758
Merge pull request #11459 from thalesmg/kafka-health-check-interval-r52-20230816 
feat(kafka): add option to configure health check interval
 2023-08-17 09:04:30 -03:00
Thales Macedo Garitezi
029b461a13
Merge pull request #11448 from thalesmg/gcp-consumer-403-20230815 
fix(gcp_consumer): handle 403 responses
 2023-08-17 09:04:00 -03:00
ieQu1
7466024bca
Merge pull request #11458 from ieQu1/fix-excluded-apps-230916 
fix(emqx_machine_boot): Fix excluded_apps
 2023-08-17 11:36:23 +02:00
Ivan Dyachkov
7469222a17 Merge remote-tracking branch 'upstream/master' into 0817-e5.2.0-code-freeze 2023-08-17 08:16:04 +02:00
lafirest
5e448f5a02
Merge pull request #11446 from lafirest/refactor/calendar 
refactor(calendar): refactor datetime-related code and remove redundant
 2023-08-17 07:40:44 +08:00
Andrew Mayorov
5d79823891
perf(topicidx): preserve next key on the stack to reuse later 
This should further optimize the number of `ets:next/2` calls
required for a single match query.
 2023-08-17 01:13:19 +04:00
Paulo Zulato
740680d54b
Merge pull request #11444 from paulozulato/fix-kinesis-msg-access-denied 
fix(kinesis): return error message on access denied
 2023-08-16 18:02:46 -03:00
Thales Macedo Garitezi
f03733ba6b
Merge pull request #11460 from thalesmg/integration-tests-dir-20230816 
ci: add `integration_test` dir to `emqx` application
 2023-08-16 17:47:14 -03:00
Ilya Averyanov
d17bcc94f6 chore(auth): update emqx_gcp_device tests 2023-08-16 22:37:42 +03:00
Paulo Zulato
a64386ef82 fix(kinesis): return error message on access denied 
Fixes https://emqx.atlassian.net/browse/EMQX-10764
 2023-08-16 16:04:23 -03:00
Ilya Averyanov
3943830eae chore(auth): remove unnecessary atom conversions 2023-08-16 20:50:35 +03:00
Ilya Averyanov
0dae040047 chore(auth): restore injected schema key position 2023-08-16 20:50:35 +03:00
Ilya Averyanov
db31e5f0d9 chore(auth): tidy up the code 2023-08-16 20:50:35 +03:00
Ilya Averyanov
67e06b3171 chore(auth): make schema injection be more universal 2023-08-16 20:41:05 +03:00
Ilya Averyanov
abcdf18ca4 chore(auth): update ldap authn test 2023-08-16 20:41:05 +03:00
Ilya Averyanov
f00f4568dc chore(auth): simplify schema injection 2023-08-16 20:41:05 +03:00
Ilya Averyanov
8a3e8ee528 chore(auth): add config propagation tests for authn 2023-08-16 20:41:05 +03:00
Ilya Averyanov
cd88dfbbb8 chore(auth): update authn includes 2023-08-16 20:41:05 +03:00
Ilya Averyanov
ae3355880e chore(auth): update emqx_authn tests 2023-08-16 20:41:05 +03:00
Ilya Averyanov
128aa19d47 chore(auth): improve emqx_config_handler to replace adhoc config updates 2023-08-16 20:41:05 +03:00
Ilya Averyanov
8213aa42c9 chore(auth): move authn leftovers to the authn app 2023-08-16 18:52:14 +03:00
Ilya Averyanov
ca8c1e3ef8 chore(auth): removed direct usages of authn from emqx core app 2023-08-16 18:52:14 +03:00
Ilya Averyanov
ce2b159022 chore(auth): make pre_hook_authenticate be part of emqx_access_control domain 2023-08-16 18:52:14 +03:00
zhongwencool
e6f0dead9e
Merge pull request #11445 from zhongwencool/remove-os-mon-from-windows 
fix: remove os_mon application in Windows release
 2023-08-16 23:30:24 +08:00
Thales Macedo Garitezi
a6a023eff7 ci: add integration_test dir to emqx application 
This gives us a place where to put tests that exercise multiple umbrella application,
which is more closely related to `emqx`, but in a way that doesn't affect the standalone
app tests.
 2023-08-16 11:48:47 -03:00
Thales Macedo Garitezi
ffca581229 feat(kafka): add option to configure health check interval 
Fixes https://emqx.atlassian.net/browse/EMQX-10781
 2023-08-16 11:08:06 -03:00
ieQu1
7b44caeae4 fix(emqx_machine_boot): Fix excluded_apps 2023-08-16 16:04:35 +02:00
Andrew Mayorov
c84ee66d80
Merge pull request #11450 from keynslug/fix/dirty-testsuite-flapping 
fix(test): switch `emqx_flapping_SUITE` to cth tooling
 2023-08-16 17:20:31 +04:00
zhongwencool
e406cd7868
Merge pull request #11456 from zhongwencool/allow-empty-cacertfile-pem 
fix: allow empty cacertfile pem
 2023-08-16 21:10:52 +08:00
zhongwencool
78356d2625 chore: start os_mon in handle_continue/2 2023-08-16 20:54:53 +08:00
Thales Macedo Garitezi
82f27eaf78
Merge pull request #11429 from thalesmg/mongodb-legacy-opt-20230810 
feat(mongodb): add configurable option to override legacy protocol usage
 2023-08-16 09:54:53 -03:00
zhongwencool
8b23ee86b3 fix: hide cpu_status if os_check is not supported 2023-08-16 20:49:28 +08:00
zhongwencool
b817e03c08 fix: start os_mon application temporary 2023-08-16 20:49:27 +08:00
zhongwencool
848eb7e3c3 fix: remove os_mon application in Windows release 2023-08-16 20:49:27 +08:00