f393d75a1e
Merge pull request #12771 from thalesmg/test-another-flaky-monitor-m-20240322
...
test: attempt to fix flaky test
2024-03-25 09:19:20 -03:00
e39fa5eaa9
docs(es_connector_info): add copyright header
2024-03-25 10:44:33 +01:00
3e97e5528e
Merge pull request #12765 from JimMoen/EMQX-12043/subscribers-count
...
fix(stats): `'subscribers.count'` contains shared-subscriber
2024-03-25 17:19:13 +08:00
5f7f9e43f9
fix(stats): `'subscribers.count'` contains shared-subscriber
2024-03-25 15:12:57 +08:00
2926518b3d
Merge pull request #12764 from zhongwencool/fix-bpapi-undef
...
fix: bpapi undef on old node
2024-03-25 13:37:37 +08:00
df913f2450
docs: refine changelog
2024-03-23 14:11:30 +01:00
22838f027a
fix: mountpoint template render should not replace unknown as undefined
...
For backward compatibility, the unknown vars used in mountpoint
is kept unchanged.
e.g. '${unknown}/foo/bar' should be rendered as '${unknown}/foo/bar'
but not 'undefined/foo/bar'
2024-03-23 10:16:05 +01:00
3136ec5958
feat: allow mountpoint to use client_attrs
2024-03-23 10:16:05 +01:00
8e8fc6a3d1
test: fix config test
2024-03-23 10:16:05 +01:00
5e9814d171
fix: add debug level logging for invalid client attributes
2024-03-23 10:16:05 +01:00
0cf61932b6
feat: allow using client_attrs in authentication templates
2024-03-23 10:16:05 +01:00
8254b801ae
feat: support initialize client attribute from user property
2024-03-23 10:16:02 +01:00
2fd0a2cd4d
feat: support extracting initial client attrs from clientinfo
2024-03-23 10:16:02 +01:00
c75840306b
fix: restrict client_attr key and value string format
...
The keys and values are used to render templates for
authz rules, such as topic names, and SQL statements etc.
2024-03-23 10:16:02 +01:00
9ec99fef4a
feat: allow client_attr used in authz rules
2024-03-23 10:16:02 +01:00
e5816f5a13
refactor: rename attr to client_attr
...
client_attr is unique enough for all contexts
so the name can be unified from external responses
to internal template rendering, and rule-engine template rendering
2024-03-23 10:16:02 +01:00
cc4805b1ac
feat: extract attrs field from http and jwt auth response
2024-03-23 10:16:02 +01:00
f9d51484cc
feat: introuce attrs field in clientinfo
...
the 'attrs' field is to be used as extra attributes extracted
from various client properties
for example, HTTP auth result can return an extra client_attributes
field in the result which will get merged into the original
attrs field in clientinfo
2024-03-23 10:16:02 +01:00
ef0fee52f4
fix: query_by_clientid with v0.7 sql api
2024-03-22 17:09:23 -07:00
1fcd06f4bc
feat: update greptimedb client lib and ci version
2024-03-22 15:33:15 -07:00
303d0ed81b
test: attempt to fix flaky test
2024-03-22 16:29:00 -03:00
70378052dd
Merge pull request #12769 from SergeTupchiy/EMQX-12058-handle-client-shutdown-errors
...
fix:(emqx_mgmt): handle client shutdown errors in call_client
2024-03-22 17:43:57 +02:00
cbc84900b2
chore: update bpapi versions dump
2024-03-22 16:24:52 +01:00
9986de49d1
chore: make dyalizer happy
2024-03-22 16:16:49 +01:00
be1886fb91
chore: make dyalizer happy and update changelog
2024-03-22 15:56:44 +01:00
ed0b5ac939
fix(emqx_mgmt): handle client shutdown errors in call_client
...
The client process may be shutdown normally during/shortly before the call.
2024-03-22 16:47:02 +02:00
b81a11b790
chore(routing): mark function that is now an RPC target
2024-03-22 15:14:53 +01:00
ba276d807f
fix(routing): add heuristic for routing schema in empty cluster
...
When the cluster is empty but comprised of different versions of EMQX,
try to additionally ask the cluster nodes for the routing schema in use,
to be able to make more informed decisions.
2024-03-22 15:14:32 +01:00
bc5775a988
fix(kafka_producer): handle ancient v1 config when migrating to actions
...
Fixes https://emqx.atlassian.net/browse/EMQX-12064
2024-03-22 09:45:27 -03:00
d2a1a7f7cf
chore: rename `message_queue_too_long` error reason to `mailbox_overflow`
...
`mailbox_overflow` is consistent with the corresponding config parameter:
'force_shutdown.max_mailbox_size'
2024-03-22 12:20:20 +02:00
42faffc320
refactor: uniform shared_sub table macros
2024-03-22 18:03:13 +08:00
45eedbb9c6
fix: bpapi undef on old node
2024-03-22 17:02:09 +08:00
2b684c7bcc
fix(emqx_connector_info): refactoring based on thalesmg's comments
2024-03-22 09:29:26 +01:00
5d77e1b01e
fix: return not_implemented error for mqueue/inflight msgs queries
2024-03-22 10:02:31 +02:00
9b085b5797
fix(api_schema): removed metrics schema in api spec
...
- Followup [PR#6622](https://github.com/emqx/emqx/pull/6622 ).
2024-03-22 10:57:14 +08:00
23ad37f566
Merge pull request #12762 from thalesmg/ds-fix-sync-egress-reply-m-20240321
...
fix(ds): add caller to pending replies before flushing and reply failures to sync callers
2024-03-21 16:42:42 -03:00
cfd4a1297f
Merge pull request #12761 from thalesmg/test-fix-flaky-dash-mon-m-20240321
...
test: fix flaky test
2024-03-21 16:04:08 -03:00
e17f663fa5
Merge pull request #12749 from thalesmg/mv-followup1-m-20240320
...
Follow up features and fixes for message validation - part 1
2024-03-21 16:03:49 -03:00
796c04e7a8
test: fix flaky test
...
We should emit the trace event before replying to callers.
Example failure:
https://github.com/emqx/emqx/actions/runs/8378977952/job/22946318696#step:6:182
```
=CRITICAL REPORT==== 21-Mar-2024::17:45:37.676024 ===
"check stage" failed: error
{assertMatch,[{module,emqx_ds_storage_bitfield_lts_SUITE},
{line,270},
{expression,"? of_kind ( emqx_ds_replication_layer_egress_flush , Trace )"},
{pattern,"[ # { batch := [ _ , _ , _ ] } ]"},
{value,[]}]}
Stacktrace: [{emqx_ds_storage_bitfield_lts_SUITE,
'-t_atomic_store_batch/1-fun-1-',1,
[{file,
"/__w/emqx/emqx/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl"},
{line,270}]},
{emqx_ds_storage_bitfield_lts_SUITE,t_atomic_store_batch,1,
[{file,
"/__w/emqx/emqx/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl"},
{line,249}]}]
```
2024-03-21 15:47:29 -03:00
68af211130
fix(ds): reply sync callers after raft store failure
2024-03-21 15:40:21 -03:00
70737a437a
fix(ds): add caller to pending replies before flushing
2024-03-21 14:39:21 -03:00
93c80bc912
chore: 5.6.0-rc.3
2024-03-21 18:10:59 +01:00
fa7ec231e3
test: fix flaky test
2024-03-21 14:07:09 -03:00
e2a2295c99
Merge pull request #12759 from SergeTupchiy/EMQX-11808-remove-uploaded-invalid-backups
...
fix(emqx_mgmt_data_backup): remove invalid uploaded backup files
2024-03-21 18:58:48 +02:00
00aa7b5621
feat: create new `message.validation_failed` hookpoint and rule engine event
2024-03-21 13:46:27 -03:00
7d7c6685d4
Merge pull request #12753 from thalesmg/test-flaky-retry-m-20240321
...
test: attempt to stabilize flaky test
2024-03-21 13:43:51 -03:00
40eccb10d6
fix(emqx_mgmt_data_backup): remove an uploaded backup file if it's not valid
2024-03-21 17:36:39 +02:00
4bfc51c052
refactor(emqx_connector_info): improve documentation and type specs
2024-03-21 16:08:31 +01:00
7d6fde5960
fix(topics-api): respond with correct totals to paged queries
2024-03-21 16:02:16 +01:00
e837791f94
fix(kafka-based bridges): avoid trying to get raw config for replayq dir
...
Fixes https://emqx.atlassian.net/browse/EMQX-12049
2024-03-21 11:49:00 -03:00
64399b6861
test: attempt to stabilize flaky test
2024-03-21 11:04:14 -03:00
89befa580e
refactor(emqx_connector_info): remove code that is no longer needed
2024-03-21 11:58:54 +01:00
cada944350
feat(sessds): Expose relevant durable session info in the REST API
2024-03-21 10:37:04 +01:00
a3b75eb39d
refactor(remaining connectors): to use emqx_connector_info
...
This commit refactors the sqlserver, iotdb, elasticsearch, opents,
greptimedb, pulsar, tdengine, rabbitmq and s3 connectors to use the
`emqx_connector_info` behavior.
2024-03-21 09:47:57 +01:00
62030e8942
feat(message validation): forbid repeated schema checks
...
Fixes https://emqx.atlassian.net/browse/EMQX-12054
2024-03-20 15:26:16 -03:00
b8cd1c9020
feat(message validation api): add enable/disable HTTP API
2024-03-20 14:59:32 -03:00
4944cc080e
feat(message_validation): add `ignore` failure action
2024-03-20 14:31:43 -03:00
8753e3583f
feat(message_validation): add `none` log level
2024-03-20 14:31:26 -03:00
e767f01e0a
fix(message_validation): take `enable` into account
2024-03-20 13:57:12 -03:00
7aa287c6c1
fix: add message validation schema to `emqx_enterprise_schema`
2024-03-20 13:37:01 -03:00
49e0488498
chore: 5.6.0-rc.2
2024-03-20 17:11:37 +01:00
bede5a5b85
Merge pull request #12746 from zmstone/0320-add-throttle-log-for-auth-failure
...
feat: add authentication_failure throttled log
2024-03-20 17:06:36 +01:00
27c1f1a4e4
Merge pull request #12732 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API-updates
...
EMQX-11861 client mqueue inflight api updates
2024-03-20 16:27:44 +02:00
7257fe526b
fix(ci): add `ra` to emqx app dependencies as well
2024-03-20 14:46:53 +01:00
90fce21adc
docs: add change log for #12746
...
also updated #12520 to include newly added throttled log id
2024-03-20 14:43:28 +01:00
38e30622f3
chore(license): add LICENSE tag to logs
2024-03-20 14:42:01 +01:00
2c81b56377
feat: add authentication_failure throttled log
...
also included changes:
1. add 'username' to log metadata
2. add 'tag' to authz warning logs
2024-03-20 14:42:01 +01:00
e5f9e7631d
Merge pull request #12736 from zmstone/0319-json-schema-draft-06-as-default-version
...
chore: upgrade to jesse 1.8.0 for draft-06 as default version
2024-03-20 14:39:37 +01:00
27d109ce6d
refactor(syskeeper connector): to use emqx_connector_info
...
This commit refactors the syskeeper connector to use the
`emqx_connector_info` behavior.
2024-03-20 14:29:02 +01:00
cb5fdb3c79
fix: rework In-flight / Mqueue API
...
- use timestamp 'position' to find the next chunk of data
- add 'start' position to response meta
- sort In-flight messages by insertion time
- sort Mqueue messages by priority
2024-03-20 15:25:33 +02:00
188ab71673
refactor(rocketmq connector): to use emqx_connector_info
...
This commit refactors the rocketmq connector to use the
`emqx_connector_info` behavior.
2024-03-20 14:02:18 +01:00
aa368bb9d0
refactor(redis connector): to use emqx_connector_info
...
This commit refactors the redis connector to use the
`emqx_connector_info` behavior.
2024-03-20 13:49:28 +01:00
12717baa0b
refactor(mqtt connector): to use emqx_connector_info
...
This commit refactors the mqtt connector to use the
`emqx_connector_info` behavior.
2024-03-20 13:33:13 +01:00
e55e1dd1b2
chore: whitelist `ra` to make RPCs w/o BPAPIs
2024-03-20 13:20:25 +01:00
a8baff61ec
docs(dsrepl): describe briefly what `n_sites` is for
2024-03-20 13:20:25 +01:00
efac5c6197
test(ds): stabilize `t_message_gc` testcase
2024-03-20 13:20:25 +01:00
fe50a1711b
fix(ds-egress): drop pending batch on failures
...
Before this commit, messages in the current batch will be retried as
part of next batch. This could have led to message duplication which is
probably not what the user wants by default.
2024-03-20 13:20:25 +01:00
a1f5de3f5b
fix(dsrepl): turn memoize into a safer function
2024-03-20 13:20:24 +01:00
d39ca41070
chore(dsrepl): mark per-node `add_generation` RPC target obsolete
...
Also annotate internal exports with comments according with their
intended use.
2024-03-20 13:20:24 +01:00
35b18f9125
fix(dsrepl): properly handle error conditions in generation mgmt
...
Also update few outdated typespecs. Also make error reasons easier
to comprehend.
2024-03-20 13:20:24 +01:00
f2268aa69a
fix(dsrepl): use correct base dir for ra system stuff
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-20 13:20:24 +01:00
404e919494
refactor(dsrepl): make shard allocator more robust and consistent
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-20 13:20:24 +01:00
0e18bd6e80
fix(dsrepl): increase replication site id bitsize back
...
In order to minimize chances of site id collision to practically zero.
2024-03-20 13:20:24 +01:00
ac9700dd28
fix(dsrepl): split shard allocator into a separate module
2024-03-20 13:20:23 +01:00
1b647035d0
chore(dsrepl): make dialyzer a bit happier
2024-03-20 13:20:23 +01:00
611b3f0e07
feat(dsrepl): use more straightforward way to drop ra shards
2024-03-20 13:20:23 +01:00
74881e8706
feat(dsrepl): make storage layer unaware of granularity of time
...
Storage also becomes a bit more pure, depending on the upper layer to
provide the timestamps, which also makes it possible to handle more
operations idempotently.
2024-03-20 13:20:23 +01:00
783f769a98
fix(mgmt): Fix bulk kickout of durable sessions
2024-03-20 12:52:44 +01:00
223bbc05fe
refactor(mysql connector): to use emqx_connector_info
...
This commit refactors the mysql connector to use the
`emqx_connector_info` behavior.
2024-03-20 11:12:38 +01:00
c6d7c57bbd
refactor(clickhouse connector): to use emqx_connector_info
...
This commit refactors the clickhouse connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:59:58 +01:00
afa3da6d0d
refactor(cassandra connector): to use emqx_connector_info
...
This commit refactors the cassandra connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:50:44 +01:00
159ab81904
refactor(influxdb connector): to use emqx_connector_info
...
This commit refactors the influxdb connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:39:16 +01:00
c1e6092013
refactor(oracle connector): to use emqx_connector_info
...
This commit refactors the oracle connector to use the
`emqx_connector_info` behavior.
2024-03-20 10:12:55 +01:00
808efe0f64
chore: rm outdated comment
2024-03-20 16:49:36 +08:00
b8b4c5a34a
fix(otel): cpu usage/idle metrics for opentelemetry
2024-03-20 16:48:21 +08:00
0547b32727
feat(sessds): Add zone overrides for session durability settings
2024-03-19 22:51:06 +01:00
611e4f6710
Merge pull request #12740 from ieQu1/dev/kick-durable-sessions
...
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 21:30:40 +01:00
e2db038e7c
Merge pull request #12734 from keynslug/fix/EMQX-12030/topics-api
...
fix(api-topics): expose persistent session topics in APIs
2024-03-19 21:08:23 +01:00
3cb36a5619
feat(ds-lts): extract timestamp from storage key itself
...
1. This avoids the need to deserialize the message to get the timestamp.
2. It also makes possible to decouple the storage key timestamp from the
message timestamp, which might be useful for replication purposes.
2024-03-19 20:21:56 +01:00
5cc0246351
feat(dsrepl): allow to tune select ra options
2024-03-19 20:21:55 +01:00
54b5adf868
feat(dsrepl): allocate shards predictably
...
To ensure strictly optimal and fair shard allocation across
cluster. Before this commit it was quite easy to end up with
an allocation significantly skewed towards some node, because
of the nature of randomness and relatively small number of
shards.
2024-03-19 20:21:55 +01:00
d30c99512a
feat(utils-stream): add a few more stream combinators
2024-03-19 20:21:54 +01:00
887e151be5
fix(dsrepl): handle errors gracefully in shard egress process
...
Also add cooldown on timeout / unavailability.
2024-03-19 20:21:53 +01:00
e16aee99b4
chore(dsrepl): clarify how to perform leadership transfer in runtime
2024-03-19 20:21:18 +01:00
00d509f27b
feat(dsrepl): prefer local replica in read path
...
To optimize out any unnecessary RPCs. Given the load should be
smoothed evenly across the cluster, choosing non-leader node is
not a priority.
2024-03-19 20:11:42 +01:00
19305c223c
fix(dsrepl): require majority for replication-related tables
2024-03-19 20:11:42 +01:00
f89909f60c
fix(dsrepl): tolerate trigger election timeouts for existing servers
2024-03-19 20:11:42 +01:00
6c6ea50e42
fix(boot): add `emqx_durable_storage` to the list of restarted apps
2024-03-19 20:11:42 +01:00
3b59cf2ebf
feat(dsrepl): move shard allocation to separate process
...
That starts shard and egress processes only when shards are fully
allocated.
2024-03-19 20:11:41 +01:00
4dafbf21f6
fix(dsrepl): make db + shard part of machine state
...
It doesn't feel right, but right now is the easiest way to have it
in the scope of `apply/3`, because `init/1` doesn't actually invoked
for ra machines recovered from the existing log / snapshot.
2024-03-19 20:11:41 +01:00
d19128ed65
feat(dsrepl): cache shard metadata in persistent terms
2024-03-19 20:11:41 +01:00
e6c2c2fb07
feat(dsrepl): manage generations / db config through ra machine
2024-03-19 20:11:39 +01:00
5e94bdb932
feat(dsrepl): allocate shards once predefined number of sites online
...
Before this commit the most likely shard allocation outcome was
that all shard are allocated to just one node.
2024-03-19 20:11:03 +01:00
be793e4735
fix(dsrepl): reassign timestamp at the time of submission
...
This is needed to ensure total message order for a shard, and
guarantee that no messages will be written "in the past". which
may break replay consistency.
2024-03-19 20:11:01 +01:00
146f082fdc
feat(dsrepl): implement raft-based replication
...
Still very rough but mostly working.
2024-03-19 20:09:44 +01:00
923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
...
sync release-56
2024-03-19 19:09:41 +01:00
f453bdf055
chore(topic-api): describe `emqx_persistent_session_ds_router:stream/1`
2024-03-19 18:51:06 +01:00
027b237f07
test(ds): stabilize `t_session_replay_retry` testcase
2024-03-19 18:33:38 +01:00
236685f714
fix(sessds): Channel must destroy sessions that are kicked
2024-03-19 17:49:04 +01:00
df8c7152fc
refactor(mongodb connector): to use emqx_connector_info
...
This commit refactors the mongodb connector to use the
`emqx_connector_info` behavior.
2024-03-19 16:09:30 +01:00
c3d7d68cfc
refactor(kinesis,pgsql,timescale,matrix connectors): emqx_connector_info
...
This commit refactors the kinesis, pgsql, timescale and matrix
connectors to use the `emqx_connector_info` behavior.
2024-03-19 15:57:23 +01:00
f2dc940436
Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56
2024-03-19 15:20:08 +01:00
a689ae72e3
Merge pull request #12711 from thalesmg/data-validation-m-20240311
...
feat: implement message validation
2024-03-19 11:12:28 -03:00
10ea367030
test(client-api): attempt to get rid of inter-testcase side effects
2024-03-19 14:56:50 +01:00
3129a8fa42
test(client-api): fix snabbkaffe crash noise in the CT logs
2024-03-19 14:55:28 +01:00
b8505ebd54
chore: upgrade to jesse 1.8.0 for draft-06 as default version
2024-03-19 14:47:11 +01:00
a3e631cda2
refactor(kafka connectors): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_kafka` to use the
`emqx_connector_info` behavior. The `emqx_bridge_kafka` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
b8ef357fef
refactor(http connector): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_http` to use the
`emqx_connector_info` behavior. The `emqx_bridge_http` related
information can thus be removed from `emqx_connector_chema` and
`emqx_connector_resource`.
2024-03-19 14:41:23 +01:00
dc08fab0e8
refactor(hstreamdb connector): to use emqx_connector_info
...
This commit refactors the `emqx_bridge_hstreamdb` to use the
`emqx_connector_info` behavior. The `emqx_bridge_hstreamdb` related
information can thus be removed from `emqx_connector_ee_schema` and
`emqx_connector_schema`.
2024-03-19 14:41:20 +01:00
d0bebe8b93
refactor(gcp_pubsub connectors): to use emqx_connector_info
...
This commit refactors the emqx_bridge_gcp_pubsub app to use the
emqx_connector_info behavior. The emqx_bridge_gcp_pubsub
related information can thus be removed from emqx_connector_ee_schema
and emqx_connector_schema.
2024-03-19 14:40:33 +01:00
76d32b6e97
refactor(confluent connector): to use emqx_connector_info
...
This commit refactors the emqx_bridge_confluent to use the
emqx_connector_info behavior. The emqx_bridge_confluent related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:43 +01:00
795b668c4f
refactor(azure_event_hub connector): to use emqx_connector_info
...
This commit refactors the emqx_bridge_azure_event_hub to use the
emqx_connector_info behavior. The emqx_bridge_azure_event_hub related
information can thus be removed from emqx_connector_ee_schema and
emqx_connector_schema.
2024-03-19 14:39:17 +01:00
c478e375f6
refactor: change failure log configuration schema
2024-03-19 10:04:18 -03:00
c7db157ecd
Merge pull request #12735 from zhongwencool/fix-ban-crash
...
fix: don't crash when ban ip error
2024-03-19 20:57:54 +08:00
2eac54bf54
chore: rm `move` API
...
Only `reorder` API will be supported, after discussions with product and frontend teams.
2024-03-19 09:55:55 -03:00
92fd0e453a
Merge pull request #12719 from SergeTupchiy/EMQX-11900-multiple-clientid-username-clients-API
...
Support multiple clientid/username Qs params in clients API
2024-03-19 14:44:39 +02:00
e1b3263177
fix(sessds): make dialyzer a tad happier
2024-03-19 13:11:32 +01:00
24c04e715d
Merge pull request #12715 from zhongwencool/fix-source-replace-error
...
fix: cant replace source conf
2024-03-19 19:32:47 +08:00
111cbbbe3c
fix: don't crash when ban ip error
2024-03-19 19:10:36 +08:00
d7c01a4fe0
fix: cant replace source conf
2024-03-19 18:53:51 +08:00
985f1c4879
fix(api-topics): expose persistent session topics in APIs
2024-03-19 11:01:26 +01:00
9a44c6712a
fix: otel metrics create view crash
2024-03-19 12:20:49 +08:00
e8a8416e44
feat(message_validation): impose restrictions on validation name
2024-03-18 15:57:39 -03:00
74c03377f2
feat(message_validation_api): implement `reorder` API
2024-03-18 14:30:52 -03:00
bcb7fe96d5
refactor: remove ambiguity from api
...
In preparation for the "reorder" API
2024-03-18 13:20:05 -03:00
96ffc9b174
refactor: export `schema_check` fn from schema registry
2024-03-18 13:11:39 -03:00
cde87bce66
chore: add missing traces
2024-03-18 13:11:39 -03:00
f84a996671
feat: implement message validation
...
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
8f8b023429
refactor: simplify match spec holder construction
2024-03-18 18:00:26 +02:00
b3d44125f6
feat: support multiple clientid / username Qs params in "/clients" API
...
Additionally, add an option to specify which Client info fields to return in the response.
2024-03-18 18:00:26 +02:00
af3a604354
refactor: add emqx_connector_info behavior
...
This commit adds the behavior `emqx_connector_info`. The
`emqx_connector_info` behavior should be implement when creating a new
connector to provide information about the connector (such as connector
schema etc) to the `emqx_connetor` application.
The connector in the `emqx_bridge_dynamo` application has also been
refactored to use the new behavior (as a test to see that the behavior
is working as intended).
Fixes:
https://emqx.atlassian.net/browse/EMQX-11427
2024-03-18 13:50:23 +01:00
2a611e4508
chore: fix the error api desc
2024-03-18 18:27:28 +08:00
485d8dbbdb
feat: add the `get /source_types` endpoint to return all supported source
2024-03-18 18:11:06 +08:00
889eb48368
fix(shared-sub): update stats when unsub shared topic filter
2024-03-16 20:43:05 +08:00
59e280dc54
test: cluster consistented metrics and label for nodes running/stopped
...
- `emqx_cluster_sessions_count`
- `emqx_cluster_sessions_max`
- `emqx_subscriptions_shared_count`
- `emqx_subscriptions_shared_max`
- `emqx_cluster_nodes_running`
- `emqx_cluster_nodes_stopped`
2024-03-16 20:43:04 +08:00
55a8488a4f
test: fields in `/monitor_current`
...
- `retained_msg_count`
- `shared_subscriptions`
2024-03-16 20:43:03 +08:00
02b3292025
fix: cluster nodes running/stopped with node name as label
2024-03-16 20:39:34 +08:00
0e25eb287b
fix(prom): cluster session count/max is cluster consistented
2024-03-16 20:39:33 +08:00
62f8f889e1
refactor: func rename to fix typo
2024-03-16 20:39:32 +08:00
e56bee618b
fix: `shared_subscriptions` is cluster consistented
2024-03-16 20:39:31 +08:00
3f93780d8c
Merge pull request #12717 from id/0315-prep-5.6.0-rc.1
...
chore: 5.6.0-rc.1
2024-03-15 21:28:31 +01:00
e28992d586
chore: 5.6.0-rc.1
2024-03-15 17:08:00 +01:00
ea76dac760
chore: fix xref
2024-03-15 17:08:00 +01:00
1fecd01405
Merge pull request #12708 from kjellwinblad/kjell/dynamo_bridge/update_error/EMQX-11984
...
fix: DynamoDB connector status check takes too long
2024-03-15 16:20:19 +01:00
c22735b3f5
Merge pull request #12707 from ieQu1/dev/store-ds-session-ip
...
Store peer name in the durable session metadata.
2024-03-15 16:15:29 +01:00
a93f747afa
fix(sessds): Return peername of the disconnected client in the REST
2024-03-15 13:48:53 +01:00
b6cc9177a6
feat(sessds): Store peername in the persistent session state
2024-03-15 13:48:53 +01:00
e35e8847b9
Merge pull request #12685 from SergeTupchiy/refactor-emqx-mgmt-call-client
...
Refactor emqx mgmt call client
2024-03-15 10:01:49 +02:00
11ae04d810
test(ds): rm unused var warning
2024-03-14 17:16:24 -03:00
d8bdb3c9aa
fix(api/clients): drop expired sessions from durable storage
2024-03-14 20:57:53 +01:00
4e9f2c8f5d
fix: DynamoDB connector status check takes too long
...
The DynamoDB connector status checks takes very long when the server is
unavailable which makes the resource manager blocked for a long time.
This causes calls to update the Bridge config with the Bridge V1 API fail
due to a timeout when it calls the resource manager to remove the
channel.
A better fix would be to change the resource manager so that the status
check cannot block it for a long time. However, this is more complicated
so it needs to be done in a later commit. A new ticket has been created
for this task https://emqx.atlassian.net/browse/EMQX-12015 .
Fixes:
https://emqx.atlassian.net/browse/EMQX-11984
2024-03-14 17:21:55 +01:00
2ebc8dcc55
fix(ds): use `infinity` timeout when storing batches
2024-03-14 10:17:18 -03:00
e340ab7f1b
Merge pull request #12696 from thalesmg/fix-default-enable-r56-20240313
...
fix(bridges): fix default value for `enable` when attempting operations
2024-03-14 09:50:40 -03:00
69a9431471
Merge pull request #12665 from ieQu1/dev/sessds-hot-conf-disable
...
fix(sessds): Prevent hot update of session_persistence.enable config
2024-03-14 13:20:52 +01:00
d11949ac51
Merge pull request #12694 from thalesmg/fix-kconsu-tm-validation-r56-20240313
...
fix(kafka_consumer): validate topic mapping in v2 schema
2024-03-14 09:19:41 -03:00
93903eb04b
Merge pull request #12683 from kjellwinblad/kjell/kinesis2/fix/parameter_restrictions/EMQX-11983
...
fix(Amazon Kinesis Action): batch size restriction
2024-03-14 10:55:44 +01:00
cb77dea1e9
Merge pull request #12699 from zmstone/0313-fix-session-count-on-replicant-node
...
fix: do not crash on replicant node
2024-03-14 09:42:27 +01:00
afb7075f82
Merge pull request #12695 from lafirest/fix/opents
...
perf(opents): Improve the message processing efficiency of opentsdb
2024-03-14 09:32:42 +08:00
45fe0787ca
Merge pull request #12689 from lafirest/fix/ld_log
...
fix(ldap): lower the log level of LDAB to `debug`
2024-03-14 09:21:44 +08:00
4f97743652
Merge pull request #12687 from lafirest/fix/iotdb
...
fix(iotdb): Fix function_clause when the convert value is null
2024-03-14 08:52:18 +08:00
35fb6ee656
Merge pull request #12688 from lafirest/fix/tdengine
...
fix(tdengine): enhanced health check result, make it more sense
2024-03-14 08:50:04 +08:00
685f29d3f2
chore: upgrade hocon to 0.42.1
...
hocon 0.42.1 allows to use "b" or "B" (byte) unit in bytesize fields.
2024-03-13 19:30:28 +02:00
4ac6d0716a
Merge pull request #12698 from thalesmg/test-flaky-conf-r56-20240313
...
test(conf): fix flaky config sync testcases
2024-03-13 13:47:23 -03:00
ccd973d13e
fix: do not crash on replicant node
2024-03-13 16:42:39 +01:00
9d21372ff0
Merge pull request #12678 from kjellwinblad/kjell/dynamo_bridge/fix/error_log/EMQX-11934
...
fix: return error reason from dynamo connector status check
2024-03-13 15:27:08 +01:00
92088a5905
test(conf): fix flaky config sync testcases
...
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-13 11:26:49 -03:00
b587ebac0c
fix(bridges): fix default value for `enable` when attempting operations
...
Fixes https://emqx.atlassian.net/browse/EMQX-11999
Although the frontend doesn't send `enable` when creating a bridge/action/connector, the
default value in the schema is `true`, so when attempting to fetch it from the raw config
we should adhere to that default.
2024-03-13 11:25:22 -03:00
a58ee801b2
fix(tdengine): enhanced health check result, make it more sense
2024-03-13 21:30:30 +08:00
b156e55430
perf(opents): Improve the message processing efficiency of opentsdb
2024-03-13 21:06:52 +08:00
a852695950
fix(kafka_consumer): validate topic mapping in v2 schema
...
Fixes https://emqx.atlassian.net/browse/EMQX-12008
2024-03-13 09:57:46 -03:00
e514c4a2ac
fix(emqx_mgmt_api_clients): check that max_payload_bytes is higher than 0
2024-03-13 14:00:30 +02:00
e9febf231f
fix(ldap): lower the log level of LDAB to `debug`
2024-03-13 19:20:40 +08:00
7908e2d591
Merge pull request #12682 from kjellwinblad/kjell/hstream_bridge/fix/parameter_restrictions/EMQX-11939
...
fix(HStreamDB bridge/action): restrict configuration parameters
2024-03-13 11:40:53 +01:00
9d999920be
Merge pull request #12681 from kjellwinblad/kjell/rocketmq_conn/remove_secret_from_logs/EMQX-11987
...
fix: redact secrets from RocketMQ debug log
2024-03-13 11:39:47 +01:00
d723a5edf7
Merge pull request #12670 from JimMoen/feat-shared-sub-count-monitor-current
...
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
2024-03-13 17:09:30 +08:00
65be76aa06
refactor(emqx_mgmt): add call_client timeout and improve RPC error handling
2024-03-13 10:56:02 +02:00
4b3c6f8330
fix(iotdb): Fix function_clause when the convert value is null
2024-03-13 11:28:07 +08:00
8be02327b2
refactor(emqx_mgmt): avoid call_client RPC to all runing nodes if global cm_registry is enabled
2024-03-12 20:58:53 +02:00
4cd434dbe3
docs(kinesis action): add description for action resource_opts
2024-03-12 18:38:39 +01:00
1edf284fed
test: trigger hocon validators when checking schema
2024-03-12 14:32:16 -03:00
c90f4f5794
fix(DynamoDB connector): fix error in status check when no workers
2024-03-12 17:22:22 +01:00
7e73b79b75
fix(Amazon Kinesis Action): batch size restriction
...
Make sure that the Amazon Kinesis action has the same batch size
restriction as the Amazon Kinesis bridge.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11983
2024-03-12 12:52:54 +01:00
da87abd9c8
fix(HStreamDB bridge/action): restrict configuration parameters
...
This commit restricts a few HStreamDB bridge/action parameters from
being any integer to being integers greater than 0. Lower values than 1
for these parameters resulted in runtime errors.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11939
2024-03-12 11:01:15 +01:00
fc4603c078
fix: redact secrets from RocketMQ debug log
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11987
2024-03-12 10:32:25 +01:00
91514f3ace
refactor: clean up over complicated code
...
This commit cleans up overly complicated code and handles
the case when the worker pool is empty.
Thank you @thalesmg for suggesting this change.
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-12 10:22:29 +01:00
1d1cdc1009
fix: return error reason from dynamo connector status check
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-11934
2024-03-12 10:22:08 +01:00
0edeff4786
test: fix flaky
2024-03-12 15:21:17 +08:00
bc9084db79
test: cpu utilization value
2024-03-12 14:14:46 +08:00
0297c7b83b
refactor: conditional enclosed in parentheses
2024-03-12 14:14:45 +08:00
207f38c42a
fix(vm): cpu usage/idle handled by single worker
2024-03-12 14:14:44 +08:00
f24a76e770
Merge pull request #12668 from emqx/leap-year-bug-sql-fun
...
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-12 10:48:31 +08:00
552968e5fe
test(conf): fix flaky config sync testcases
...
Before the recent changes these testcases relied on the fact that
peer nodes that are part of the test cluster are started one after
the other. This is now not the case in general with `emqx_cth_cluster`,
so now we need to additionally find the "longest running node" in the
cluster before running test logic.
2024-03-11 13:03:13 +01:00
d725206bcb
Merge pull request #12624 from keynslug/fix/EMQX-11901/ds-error-class
...
feat(sessds): handle recoverable errors during replay
2024-03-11 12:41:45 +01:00
b673ad3e5c
Merge pull request #12673 from zmstone/0308-fix-base64-compatibility-on-otp25
...
fix(mgmt): avoid using base64:decode/2 and encode/2
2024-03-11 13:23:37 +02:00
a82d4e1f7a
fix: `retained_msg_count` and `shared_subscriptions` api schema
2024-03-11 17:44:11 +08:00
e7574a70fb
feat(api): field `shared_subscriptions` in endpoint `/monitor_current`
2024-03-11 17:44:10 +08:00
980a4c3a4d
fix(emqx_mgmt_api): use emqx_utils:bin_to_hexstr/2
2024-03-11 10:40:19 +02:00
f376aa8072
refactor(emqx_mgmt_api_clients): remove unnecessary function
2024-03-11 10:39:09 +02:00
68682c4231
Merge pull request #12671 from kjellwinblad/kjell/rule_engine/fix/unescape_rules_from_dashboard/EMQX-11847
...
feat(rule engine SQL): add an `unescape` function
2024-03-08 17:57:31 +01:00
2cc1c563df
fix(mgmt): avoid using base64:decode/2 and encode/2
...
they are not available in otp 25
2024-03-08 16:46:54 +01:00
d92fd3e42d
Merge pull request #12667 from thalesmg/ds-delete-m-20240307
...
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
2024-03-08 10:57:51 -03:00
79d7821222
Merge pull request #12655 from thalesmg/fix-kconsu-status-r56-20240306
...
fix(kafka_consumer): check client connectivity
2024-03-08 10:53:44 -03:00
6af01b916e
feat(ds): implement `get_delete_streams`, `make_delete_iterator` and `delete_next` callbacks for builtin storage
...
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-08 09:56:46 -03:00
5dfd864c1a
Merge pull request #12666 from thalesmg/test-flaky-machine-m-20240307
...
test(machine): attempt to stabilize flaky test
2024-03-08 08:57:35 -03:00
5a6f96212d
feat(rule engine SQL): add an `unescape` function
...
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:
- Standard C escape sequences:
- `\n` for newline (LF)
- `\t` for horizontal tab (HT)
- `\r` for carriage return (CR)
- `\b` for backspace (BS)
- `\f` for formfeed (FF)
- `\v` for vertical tab (VT)
- `\'` for single quote (')
- `\"` for double quote (")
- `\\` for backslash (\)
- `\?` for question mark (?)
- `\a` for alert (bell, BEL)
- Hexadecimal escape codes:
- `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
a-f), allowing for the encoding of arbitrary utf32 characters.
If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.
Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943
...
fix(rule_engine): support non-byte sized input to bin2hexstr
2024-03-08 09:41:03 +01:00
6054499607
Merge pull request #12657 from kjellwinblad/kjell/rule_engine/fix/fun_calls_as_array_items/EMQX-11953
...
fix(rule SQL): allow expressions as array items
2024-03-08 09:39:46 +01:00
29111a2192
fix: dialyzer problems
2024-03-08 14:25:19 +08:00
163d095dca
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-08 10:48:08 +08:00
963e0de0c3
fix(kafka_consumer): check client connectivity
...
Fixes https://emqx.atlassian.net/browse/EMQX-11945
2024-03-07 14:22:03 -03:00
f7e3afde16
test(ds): avoid introducing new macros
2024-03-07 16:49:20 +01:00
b52c9c0062
test(machine): attempt to stabilize flaky test
2024-03-07 10:23:02 -03:00
69427dc42d
test(ds): add tests for error mapping and replay recovery
2024-03-07 12:59:58 +01:00
e7e8771277
fix(sessds): set replay retry timer if initial `replay/3` fails
2024-03-07 12:59:58 +01:00
09905d78cd
chore(ds): make error handling slightly simpler
...
Co-Authored-By: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-03-07 12:59:57 +01:00
b604c3dbd4
refactor(sessds): make replay error handling a bit more clear
...
Also leave a forgotten TODO.
2024-03-07 12:59:57 +01:00
3f3e33b2cb
fix(sessds): untangle pull and replay retry timers
...
And restore the convention that timer handler always manages only
its own timers.
2024-03-07 12:59:57 +01:00
b39c710ec2
fix(ds): tidy up few typespecs
2024-03-07 12:59:57 +01:00
1cf672e78d
feat(sessds): handle recoverable errors during replay
2024-03-07 12:59:57 +01:00
2146d9e1fe
feat(ds): introduce error classes in critical API functions
...
For now, only recoverable / unrecoverable errors are introduced.
2024-03-07 12:59:57 +01:00
1f38813cb9
Merge pull request #12561 from SergeTupchiy/EMQX-11861-client-mqueue-inflight-API
...
feat: add client mqueue/inflight messages API
2024-03-07 13:20:31 +02:00
c62776edaf
fix(sessds): Prevent hot update of session_persistence.enable config
2024-03-07 12:19:53 +01:00
a2e761681e
feat: add client mqueue/inflight messages API
2024-03-07 12:06:49 +02:00
6a2731f2da
Merge remote-tracking branch 'origin/master' into release-56
2024-03-07 09:36:37 +01:00
98034cb4dd
Merge pull request #12660 from zmstone/sync-5.5.1
...
Sync 5.5.1
2024-03-07 09:36:08 +01:00
c62dd56a4a
Merge pull request #12661 from thalesmg/ds-atomic-store-m-20240305
...
feat(ds): add atomic store API
2024-03-06 17:48:28 -03:00
f57f2fa1b7
chore: bump app version numbers
2024-03-06 19:37:06 +01:00
df3ebc5c56
Merge pull request #12659 from thalesmg/test-flaky-pulsar-r56-20240306
...
test(pulsar): fix flaky test
2024-03-06 15:29:01 -03:00
b421363661
Merge pull request #12656 from thalesmg/fix-gprodu-status-r56-20240306
...
fix(gcp_pubsub_producer): check for topic existence when creating action
2024-03-06 15:28:51 -03:00
5d87d400f4
feat(ds): add atomic store API
...
Part of https://emqx.atlassian.net/browse/EMQX-11841
2024-03-06 15:24:14 -03:00
Thales Macedo Garitezi