0e770bdc95
test: switch `emqx_broker_SUITE` to use new cth tooling
2023-08-29 22:05:36 +04:00
3268093881
feat(cth): add module-level documenation
2023-08-29 22:05:36 +04:00
f57d16ba13
feat(cthsuite): add function to determine workdir of testrun
...
In a deterministic fashion, to lift the burden of undestanding where
the testrun's data should go from the test writer.
2023-08-29 21:34:01 +04:00
063d6200c8
feat(router): enable using 2 tables to store routes
...
Instead of a single unified table, to reap the benefits of cheap
`ets:lookup/2` per regular topic subscription route.
Change configuration option naming to reflect the change: user now has
an ability to choose _storage schema_.
2023-08-29 17:41:59 +04:00
270fd107b2
feat(triesearch): allow reusing filter tokenization result
...
As a kind of micro-optimization.
2023-08-29 17:31:26 +04:00
501456efec
chore: more safe with iolist_to_binary
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-08-29 17:10:43 +08:00
4e7ba5d35f
chore: ciphers should allow empty space(comma_separated_binary)
2023-08-29 11:34:40 +08:00
6722722522
fix(frame): improve some error reasons when parsing invalid packet
2023-08-29 10:53:20 +08:00
4245a4d8c6
fix: authz clean-cache clientid always return not_found
2023-08-29 08:17:47 +08:00
cc3ba18734
fix: increment dropped message counter when bridge is unhealthy
...
Fixes https://emqx.atlassian.net/browse/EMQX-10767
2023-08-28 19:47:11 -03:00
2c89be04d1
Merge pull request #11527 from paulozulato/fix-kafka-header-template-handling
...
fix(kafka): fix template processing for header
2023-08-28 18:55:33 -03:00
0b86f04bae
fix(kafka): fix template processing for header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-28 17:28:05 -03:00
279895b8fd
chore(ft): add read/write concurrency to emqx_ft_async_reply tabs
2023-08-28 22:46:30 +03:00
c04f770118
Merge pull request #11506 from SergeTupchiy/EMQX-10274-do-not-download-empty-trace-log
...
fix(emqx_trace): don't download empty trace log file
2023-08-28 22:17:00 +03:00
0072749143
fix(emqx_trace): don't download empty trace log file
...
Closes: EMQX-10274
2023-08-28 19:25:55 +03:00
5908b69353
chore: hide cluster purge from cli and api for now
2023-08-28 10:23:21 -03:00
f988de4ff4
feat(purge): clear delayed messages
2023-08-28 10:23:21 -03:00
9bded07834
feat: cluster purge
...
Fixes https://emqx.atlassian.net/browse/EMQX-10763
2023-08-28 10:23:20 -03:00
dec21ffc95
Merge pull request #11362 from thalesmg/ds-handle-sub-20230725
...
feat(ds): open iterators when handling `SUBSCRIBE` packets
2023-08-28 10:21:28 -03:00
2d931a0512
test(evict): do not disable router module in cluster nodes
2023-08-28 14:07:46 +04:00
8d2ebdea7e
fix(router): generalize config option and make effects visible
2023-08-28 14:07:46 +04:00
e85789306b
chore(router): drop test-only `has_routes/1`
...
Seems that it's too much support work only for test purposes, where
`lookup_routes/1` is nearly as usable.
2023-08-28 14:07:46 +04:00
5d51687dbf
feat(router): add unified routing table config option
2023-08-28 14:07:45 +04:00
33e5e1ba57
feat(router): add unified routing table
2023-08-28 14:07:44 +04:00
166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
...
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
84e40fb6fe
test(trie): add more involved route add + delete case
...
That shows how current local trie implementation breaks because
of lack of refcounting.
2023-08-28 13:52:59 +04:00
dcb63440bc
refactor(mgmt): avoid dealing with router tab directly
...
Instead, contain all this behind `emqx_router` module interface.
2023-08-28 13:52:59 +04:00
9eccfa0909
refactor(router): isolate cleanup logic in router module
2023-08-28 13:52:59 +04:00
f3467f44b2
chore: bump hstreamdb_erl driver to `0.4.5+v0.16.1`
2023-08-28 17:28:50 +08:00
4488e9e591
chore(ft): stop hook chain when doing actual handling
2023-08-28 11:57:44 +03:00
39a48179ea
chore(emqx_channel): use macros for reply construction
2023-08-28 11:57:44 +03:00
b8cacd2833
chore(ft): add tests for async reply registry
2023-08-28 11:57:44 +03:00
b74ff10705
Merge pull request #11517 from keynslug/ft/topic-index-v3
...
feat(topicidx): iterate on trie search implementation
2023-08-28 12:24:16 +04:00
053b7fb94a
test: add more test for conf_cli
2023-08-28 11:14:36 +08:00
d8be248a3d
fix: improve the suggest msg for update conf failed
2023-08-28 08:53:23 +08:00
b0d4a22aa8
chore(ft): refactor async reply mechanism
2023-08-27 21:54:23 +03:00
b874926de7
Merge pull request #11514 from zhongwencool/stop-load-application-before-reboot
...
fix: stop otel deps appication before reboot
2023-08-27 20:46:31 +08:00
2bb415f897
Merge pull request #11526 from paulozulato/fix-validator-return-msg
...
fix: flatten error message on resource validator
2023-08-25 17:27:31 -03:00
33a0048155
refactor: move logic to `ensure_iterator`
2023-08-25 15:24:53 -03:00
84e68f8ed8
Merge pull request #11508 from paulozulato/fix-azure-msg-error-handling
...
fix(kafka): fix result handling when sending message with invalid header
2023-08-25 14:57:09 -03:00
42877e282d
fix: flatten error message on resource validator
...
Fixes https://emqx.atlassian.net/browse/EMQX-10864
2023-08-25 13:53:52 -03:00
f15f59650d
test: rm obselete workaround code
2023-08-25 13:49:33 -03:00
cb1e105e19
Merge pull request #11516 from paulozulato/fix-kafka-aeh-client-unique-id
...
Fix Kafka/AEH ClientId uniqueness
2023-08-25 10:14:44 -03:00
d5cff533e3
chore(topicidx): drop TODO comment
2023-08-25 16:44:03 +04:00
ca59a87d47
chore(topicidx): refine example of wildcard compare
2023-08-25 16:44:03 +04:00
558402a68e
chore(topic_index): add topic validation
2023-08-25 16:44:03 +04:00
33ed53bb6a
refactor(topicidx): simplify `compare/3` further
...
It's possible to emit seek instruction just once, on the way back
out of the `compare/3` stack.
2023-08-25 16:34:45 +04:00
fc37d235c7
refactor(topicidx): simplify seek instructions
...
Which also avoids comparing filter tail repeatedly when evaluating
it.
2023-08-25 15:24:06 +04:00
c65db82b07
fix: bad error message when rule engine schema name is too long
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-10778
2023-08-25 11:02:38 +02:00
c8af5a702c
test: remove connection_closed from test
2023-08-25 15:53:35 +08:00
fc1738188e
fix: packets_connack_sent is not incremented if the ack_flag field in the CONNACK packet is non-zero
2023-08-25 15:18:34 +08:00
ba9cb0a1ed
Merge pull request #11512 from lafirest/fix/ldap_def_port
...
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-25 10:22:44 +08:00
8ab6eb5638
test: start emqx_opentelemetry in emqx_machine SUITE
2023-08-25 10:09:00 +08:00
cf45e80c71
feat(topicidx): iterate on trie search implementation
...
This improves matching performance and decreases GC pressure on
synthetic workloads.
2023-08-25 01:47:11 +04:00
535c7f8b43
fix(kafka): avoid ClientId collision between Kafka and Azure bridges
...
Fixes https://emqx.atlassian.net/browse/EMQX-10860
2023-08-24 18:46:22 -03:00
d9a5a9ea2b
Merge pull request #11513 from thalesmg/kafka-fix-ts-template-r52-20230824
...
fix: use correct timestamp template for kafka and remove such fields from azure event hub producer
2023-08-24 16:26:52 -03:00
016ae0524f
fix(aeh_producer): remove timestamp template field
...
Fixes https://emqx.atlassian.net/browse/EMQX-10847
Checking the whole Kafka message from AEH, it seems like the timestamp type is append,
which means that it’s the broker who controls the timestamp, and the timestamp defined by
the producer is ignored.
Ref: https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type
Example message consumed from AEH:
```
%{
"headers" => %{},
"key" => "",
"offset" => 4,
"topic" => "test0",
"ts" => 1692879703006,
"ts_type" => "append",
"value" => "{\"username\":\"undefined\",\"topic\":\"t/aeh/produ\",\"timestamp\":1692879692189,\"qos\":0,\"publish_received_at\":1692879692189,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaa\",\"node\":\"emqx@127.0.0.1\",\"metadata\":{\"rule_id\":\"rule_aehp\"},\"id\":\"000603AA44B34E08F4AF000006E30003\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```
Note the ts_type above is append.
Example message from a Kafka broker whose ts type is create:
```
%{
"headers" => %{},
"key" => "",
"offset" => 4,
"topic" => "test-topic-three-partitions",
"ts" => 1692881883668,
"ts_type" => "create",
"value" => "{\"username\":\"undefined\",\"topic\":\"t/kafka/produ\",\"timestamp\":1692881883668,\"qos\":0,\"publish_received_at\":1692881883668,\"pub_props\":{\"User-Property\":{}},\"peerhost\":\"undefined\",\"payload\":\"aaaaaa\",\"node\":\"emqx@127.0.0.1\",\"id\":\"000603AAC7529FEEF4AC000007050000\",\"flags\":{},\"event\":\"message.publish\",\"clientid\":\"undefined\"}"
}
```
Unfortunately, I couldn’t find anywhere in AEH where that configuration could be changed.
2023-08-24 14:20:42 -03:00
0381ac0410
fix(kafka_producer): use correct timestamp template field
...
Fixes https://emqx.atlassian.net/browse/EMQX-10847
2023-08-24 14:20:42 -03:00
9fe5080705
build: bump versions before release
2023-08-24 16:46:27 +02:00
c215fe3736
fix: stop otel deps appication before reboot
2023-08-24 22:14:14 +08:00
60e6217496
fix(kafka): fix result handling when sending message with invalid header
...
Fixes https://emqx.atlassian.net/browse/EMQX-10846
2023-08-24 10:41:42 -03:00
e5041de9cc
fix(gcp_consumer): handle 401 errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-10852
2023-08-24 09:03:34 -03:00
62423b0b12
refactor(topic_index): less special handling for leading $ words
2023-08-24 13:30:10 +02:00
a30d87e14f
refactor(topic_index): remove more unnecessary next calls
...
also avoid using records (setelement) for recursive return values
2023-08-24 12:25:04 +02:00
a1e6635614
refactor(topic_index): no forced ceiling entry in index table
2023-08-24 12:25:04 +02:00
f4c8c6be55
refactor(topic_index): optimize trie-search performance
2023-08-24 12:25:04 +02:00
6b152b3cb7
test: add more debug output
2023-08-24 12:25:04 +02:00
ae094e363c
chore: fix a typo in log message
2023-08-24 12:25:04 +02:00
6085442958
fix(ldap): expose request_timeout to prevent infinite hang
2023-08-24 16:50:38 +08:00
2322b27542
Merge pull request #11497 from zhongwencool/prometheus-metrics
...
Prometheus metrics
2023-08-24 12:06:32 +08:00
8d2be27ade
chore: add changlog for 11466
2023-08-24 06:46:50 +08:00
84dd3b83bb
fix: hocon converter not working when make_serializable is true(check_plain)
2023-08-24 06:46:47 +08:00
f72ba6819f
Merge pull request #11498 from zhongwencool/fix-otel-stop-error-log
...
fix: stop otel deps application in terminate crashed
2023-08-24 06:40:24 +08:00
d287b736ec
chore: add changelog for 11497
2023-08-23 17:02:30 +08:00
58be7f709b
fix: stop otel deps application in terminate crashed
2023-08-23 16:45:41 +08:00
6bb15541c7
feat: rename olp metrics to overload_protection metrics
2023-08-23 15:25:54 +08:00
b9a97923c4
feat: add some authz/connection missing metrics
2023-08-23 14:47:53 +08:00
141676185d
feat: add authentication metrics to prometheus
2023-08-23 14:15:16 +08:00
4928c69758
feat: add authorization metrics to prometheus
2023-08-23 14:11:59 +08:00
df9f8da178
feat: add overload_protection metrics to prometheus
2023-08-23 14:07:37 +08:00
188a81913e
test: add prometheus api test
2023-08-23 11:23:43 +08:00
ae10415fc3
feat: disabled the Erlang VM Prometheus exporter by default to improve performance and security
2023-08-23 09:03:22 +08:00
61536b5362
Merge pull request #11494 from paulozulato/fix-kinesis-validators
...
fix(kinesis): set validators for static constraints
2023-08-22 18:28:35 -03:00
20b46091f9
fix(kinesis): set validators for static constraints
...
Fixes https://emqx.atlassian.net/browse/EMQX-10833
2023-08-22 14:42:44 -03:00
f007b44426
fix(data_import): rm duplicate import call and fix test
2023-08-22 13:29:15 -03:00
ee2897e5de
test(refactor): move test to integration tests dir
2023-08-22 13:29:15 -03:00
c1f49abad2
test: fix inter-suite flakiness
2023-08-22 13:29:15 -03:00
dad27091be
test: rm custom option
2023-08-22 13:29:15 -03:00
dbfacae283
fix: reinstate transactions
2023-08-22 13:29:15 -03:00
65085d012b
refactor: rename fn
2023-08-22 13:29:15 -03:00
3344bfb0bd
refactor: rm `emqx_ds_replay`
2023-08-22 13:29:15 -03:00
c46b8de938
test: remove unused things, refactor some functions
2023-08-22 13:29:15 -03:00
e8c73b06e1
docs: add comment about future test failure
2023-08-22 13:29:15 -03:00
6de0bbe76a
test(refactor): always allocate listeners for emqx app
2023-08-22 13:29:15 -03:00
a15405a800
test: fix assertions
2023-08-22 13:29:15 -03:00
c74abe79d0
refactor: reduce arity
2023-08-22 13:29:15 -03:00
021755b82b
refactor: rm iterators from DS `#session{}` record
2023-08-22 13:29:15 -03:00
e4e88ebf36
test: add scenario for node stopping midway during subscribe
2023-08-22 13:29:15 -03:00
3239f5ac5b
feat: rm unnecessary transactions, use separate table for iterator references
2023-08-22 13:29:15 -03:00
8eab389ae1
perf: avoid unnecessary transaction
2023-08-22 13:29:15 -03:00
c28c6d1b7e
fix: ensure iterator is opened
2023-08-22 13:29:15 -03:00
33ddbe80ad
refactor: remove persistence leakeage from emqx_cm level
2023-08-22 13:29:15 -03:00
e8d7bb9a67
refactor: rename module
2023-08-22 13:29:15 -03:00
9463e271c0
feat(ds): open iterators when handling `SUBSCRIBE` packets
...
Fixes https://emqx.atlassian.net/browse/EMQX-9741
2023-08-22 13:29:15 -03:00
ca697a4e14
fix: rename `emqx_ds{,_replay}:replay_id()`
2023-08-22 13:29:15 -03:00
45e2e687e5
fix(greptimedb_bridge): avoid double-parsing write syntax during probe and remove code duplication
...
Fixes https://emqx.atlassian.net/browse/EMQX-10843
2023-08-22 12:42:48 -03:00
3ffbb7decf
fix: HTTP API /api/v5/publish schema
...
The schema for the /api/v5/publish HTTP API endpoint was incorrect.
For 400 (Bad Request) error it cannot return a list but the incorrect
schema declared that the response could include a list.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10837
https://github.com/emqx/emqx/issues/11488
2023-08-22 15:21:20 +02:00
7b8a599d17
Merge pull request #11486 from lafirest/fix/ldap_autn_wrong_pw
...
fix(ldap): Return correct reason code when password is wrong
2023-08-22 18:53:17 +08:00
f8f39bf223
Merge pull request #11490 from lafirest/fix/absent_pw
...
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 18:52:58 +08:00
e20b804e6b
chore: update changes && test cases
2023-08-22 17:41:08 +08:00
184e03387a
fix(ldap): fix status detection and auto reconnecting errors
2023-08-22 17:06:00 +08:00
663d03112e
Merge pull request #11476 from Kinplemelon/kinple/for-upgrade-dashboard
...
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-22 14:28:40 +08:00
b0c5b447a4
fix(authn): quickly return when the password is absent in password-based authentication
2023-08-22 11:49:32 +08:00
13ebcd6290
Merge pull request #11478 from JimMoen/hstreamdb-tls-support
...
Hstreamdb tls support
2023-08-22 10:40:54 +08:00
037b0a7599
fix(ldap): Return correct reason code when password is wrong
2023-08-22 09:42:04 +08:00
4e4c93a758
Merge pull request #11469 from paulozulato/feat-redis-auth-username
...
Add support for AUTH with username on Redis
2023-08-21 16:44:21 -03:00
74d4a8325e
Merge pull request #11470 from thalesmg/gcp-consumer-rm-worker-pool-size-r52-20230817
...
fix(gcp_consumer): remove unused resource_opts fields
2023-08-21 14:49:59 -03:00
bd5680db00
Merge pull request #11461 from thalesmg/bridge-dryrun-flex-timeout-r52-20230816
...
fix(bridge): make dryrun health check timeout more malleable
2023-08-21 14:49:46 -03:00
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
...
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
4163f06611
Merge pull request #11480 from thalesmg/rule-funcs-func-clause-r52-20230818
...
fix(rule_engine): capture function_clause errors
2023-08-21 11:16:25 -03:00
443d44aaa3
fix: set the `endpoint` in kinesis bridge as a required field
2023-08-21 11:49:57 +08:00
f45b5fac6f
chore: unhidden hstreamdb bridge for e5.2.0
2023-08-21 10:40:51 +08:00
649647190e
test: new batch mechanism of hstreamdb_erl
2023-08-21 10:40:51 +08:00
5527edf442
chore: bump hstreamdb_erl to `0.4.3+v0.16.1`
2023-08-21 10:40:36 +08:00
ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
...
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
01c9095982
Merge pull request #11394 from zmstone/0804-upgrade-wolff-to-1.7.7
...
chore: upgrade Kafka client wolff to 1.7.7
2023-08-20 10:14:13 +02:00
05fdf52d8e
fix(redis): set bridge as unhealthy on authentication error
2023-08-18 14:51:03 -03:00
fc35c74eab
fix(redis): use eredis_secret to avoid password leaking
2023-08-18 14:51:03 -03:00
2c458b62f5
feat(redis): add support for `username` on AUTH command
...
Fixes https://emqx.atlassian.net/browse/EMQX-9911
2023-08-18 14:50:06 -03:00
66dec69d09
fix(rule_engine): capture function_clause errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-10798
2023-08-18 13:51:19 -03:00
befc260555
fix(gcp_consumer): remove unused resource_opts fields
...
Fixes https://emqx.atlassian.net/browse/EMQX-10789
2023-08-17 17:02:02 -03:00
e48bfb52bb
Merge branch 'release-52' into sync-r52-master-20230817
2023-08-17 17:01:04 -03:00
afffdbbaa1
fix(kinesis): replace default payload template
...
Fixes https://emqx.atlassian.net/browse/EMQX-10766
2023-08-17 17:00:14 -03:00
e2b9d905d7
Merge pull request #11465 from id/0718-e5.2.0-alpha.3
2023-08-17 20:59:13 +02:00
75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2
...
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-17 20:43:09 +04:00
a0c2fe8cc1
chore: e5.2.0-alpha.3
2023-08-17 17:55:56 +02:00
ebecbd1545
fix(bridge): make dryrun health check timeout more malleable
...
Fixes https://emqx.atlassian.net/browse/EMQX-10773
- Makes the timeout for probing a bridge more malleable to account for differences between
each database.
- Increases GCP PubSub Consumer default health check timeout to account for GCP
slowness/throttling.
2023-08-17 09:21:19 -03:00
e3572eb758
Merge pull request #11459 from thalesmg/kafka-health-check-interval-r52-20230816
...
feat(kafka): add option to configure health check interval
2023-08-17 09:04:30 -03:00
029b461a13
Merge pull request #11448 from thalesmg/gcp-consumer-403-20230815
...
fix(gcp_consumer): handle 403 responses
2023-08-17 09:04:00 -03:00
7466024bca
Merge pull request #11458 from ieQu1/fix-excluded-apps-230916
...
fix(emqx_machine_boot): Fix excluded_apps
2023-08-17 11:36:23 +02:00
7469222a17
Merge remote-tracking branch 'upstream/master' into 0817-e5.2.0-code-freeze
2023-08-17 08:16:04 +02:00
5e448f5a02
Merge pull request #11446 from lafirest/refactor/calendar
...
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-17 07:40:44 +08:00
5d79823891
perf(topicidx): preserve next key on the stack to reuse later
...
This should further optimize the number of `ets:next/2` calls
required for a single match query.
2023-08-17 01:13:19 +04:00
740680d54b
Merge pull request #11444 from paulozulato/fix-kinesis-msg-access-denied
...
fix(kinesis): return error message on access denied
2023-08-16 18:02:46 -03:00
f03733ba6b
Merge pull request #11460 from thalesmg/integration-tests-dir-20230816
...
ci: add `integration_test` dir to `emqx` application
2023-08-16 17:47:14 -03:00
d17bcc94f6
chore(auth): update emqx_gcp_device tests
2023-08-16 22:37:42 +03:00
a64386ef82
fix(kinesis): return error message on access denied
...
Fixes https://emqx.atlassian.net/browse/EMQX-10764
2023-08-16 16:04:23 -03:00
3943830eae
chore(auth): remove unnecessary atom conversions
2023-08-16 20:50:35 +03:00
0dae040047
chore(auth): restore injected schema key position
2023-08-16 20:50:35 +03:00
db31e5f0d9
chore(auth): tidy up the code
2023-08-16 20:50:35 +03:00
Andrew Mayorov