c3da792323
feat(prom_di): cluster aggregated/unaggregated metrics
2024-01-21 00:14:53 +08:00
9627124d67
fix(prom_di): metric `emqx_rule_enable`
2024-01-21 00:14:52 +08:00
5534d5e9de
fix(bpapi): make static_check happy
2024-01-21 00:14:51 +08:00
5914eb5ca5
fix(prom_cert): rm cacertfile expiry epoch
2024-01-21 00:14:50 +08:00
4fb1ff2f9d
fix(prom_api): format-mode example value
2024-01-21 00:14:49 +08:00
57f3efde63
feat(prom_auth): cluster metrics with different format-mode
2024-01-21 00:14:48 +08:00
fb330f77e6
feat(prometheus): api `format_mode` parameter support
...
- node(default):
The only supported format_mode for PushGateway.
Return the current node's metrics.
- Without label `node_name` with content-type: `text/plain`.
- Without key `node_name` with content-type: `application/json`
- nodes_aggregated:
Return all nodes metrics Arithmetic-Sum or Logical-Sum.
See details in callback modules.
- Logical-Sum for metrics named with `xxx_enable` or `xxx_status`.
- Arithmetic-Sum for other metrics.
`node_name` field:
- Without label `node_name` with content-type: `text/plain`.
- Without key `node_name` with content-type: `application/json`
- nodes_unaggregated:
Return all nodes metrics without aggregated.
`node_name` field:
- _With_ label `node_name` with content-type: `text/plain`.
- _With_ key `node_name` with content-type: `application/json`
2024-01-21 00:14:47 +08:00
94032aafb2
fix(prom_data_integration): fix metric type
2024-01-21 00:14:46 +08:00
c3e9533260
fix: prometheus auth metrics fields and type
...
- rm rate fields
- fix few fields type to counter
2024-01-21 00:14:45 +08:00
a18c4d193a
refactor: abstract function call
2024-01-21 00:14:44 +08:00
8f7964f435
feat(prom): data integration metrics in josn format
2024-01-21 00:14:42 +08:00
36f009b0c2
fix(prom): connectors specific data
2024-01-21 00:14:41 +08:00
76d9ace582
fix: connector_count contains bridge_v1 and bridge_v2
2024-01-21 00:14:40 +08:00
f457def201
fix: use `id` uniformly as the label key for rules and actions
2024-01-21 00:14:39 +08:00
4b23930fce
fix(prometheus): license expiry and schema_registry only for ee
2024-01-21 00:14:38 +08:00
38a90bd273
fix(prom_auth): authn && authz data response by josn
2024-01-21 00:14:37 +08:00
0dca9905dd
test(prometheus): `{Registry, Collector}` tuple
2024-01-21 00:14:36 +08:00
7832bbc0a4
fix(prom): schema registry not in ce edition
2024-01-21 00:14:35 +08:00
5158395bcf
feat(prometheus): data integration prom data
2024-01-21 00:14:34 +08:00
bf2e4d134a
refactor(prometheus): generic api response funcs
2024-01-21 00:14:33 +08:00
e0feb580b6
feat(prometheus): auth metrics with text/plain
2024-01-21 00:14:32 +08:00
092159b071
feat(prometheus): cert expiry epoch in endpoint `/prometheus/stats`
2024-01-21 00:14:31 +08:00
558c471392
feat(prometheus): license expiry at epoch as gauge
2024-01-21 00:14:30 +08:00
fae0cea17f
fix(license): make dialyzer happy
2024-01-21 00:14:29 +08:00
81ba166b49
feat(license): expiry epoch api
2024-01-21 00:14:28 +08:00
89128958ed
fix(dashboard): licence quota not provided to ce edition
2024-01-21 00:14:27 +08:00
ecd0da9fde
feat(dashboard): `/monitor_current/nodes/{node}` provide `node_uptime`
2024-01-21 00:14:26 +08:00
7bc3a5090d
fix(dashboard): meck `emqx_retainer` in SUITE
2024-01-21 00:14:25 +08:00
3d6b65aced
chore: emqx_dashboard_monitor:current_rate/0 not exported anymore
2024-01-21 00:14:24 +08:00
a35698009c
feat(dashboard): endpoint `/monitor_current` provides mor fields
...
- `retained_msg_count`
Current retained_msg_count on each node and should be same on all nodes.
- `license_quota`
Only for enterprise edition, provides the max limited connections num.
2024-01-21 00:14:23 +08:00
dc15d37dcc
perf(emqx_bridge/connector): load and unload bridges/connectors in parallel
...
This should reduce app start/stop time, when a large number of bridges/connectors are not healthy.
2024-01-19 21:44:31 +02:00
d3a6870097
feat(emqx_utils): add pforeach/2,3
2024-01-19 21:44:31 +02:00
a6568dec75
perf(emqx_bridge/connector): apply post config bridge/connector changes in parallel
...
This can greatly improve the performance when many bridges/connectors are being changed,
e.g. when a backup file is being imported.
Fixes: EMQX-11751
2024-01-19 21:44:31 +02:00
54457b7093
feat(emqx_utils): allow `infinity` timeout in `pmap/3`
2024-01-19 19:34:04 +02:00
a40a13b786
Merge pull request #12357 from thalesmg/fix-bridge-bpapi-interval-r55-20240119
...
fix(bridge_api): explicitly state bpapi version ranges
2024-01-19 13:08:44 -03:00
d22092e3da
fix(mqtt_source): don't attempt local publish when legacy config is absent
...
Fixes https://emqx.atlassian.net/browse/EMQX-11765
2024-01-19 10:54:05 -03:00
e369c1b971
docs(mqtt_source): fix example qos
2024-01-19 10:54:05 -03:00
4068258c78
fix(bridge_api): explicitly state bpapi version ranges
...
Fixes https://emqx.atlassian.net/browse/EMQX-11767
The currently supported version for a node may not be available at the time it's probed.
2024-01-19 10:20:57 -03:00
e219c3eda1
Merge pull request #12166 from kjellwinblad/kjell/mqtt_conn_action_3/EMQX-11489
...
feat: refactor MQTT bridge to source, action, and connector
2024-01-18 09:28:14 -03:00
6b462a9100
Merge pull request #12349 from zmstone/0118-fix-authz-cache-excludes-type
...
0118 fix authz cache excludes type
2024-01-18 10:32:38 +00:00
1b432eadd8
Merge pull request #12348 from emqx/elasticsearch-e550
...
Elasticsearch e550
2024-01-18 18:06:48 +08:00
dae835635c
fix: don't crash in http SUITE
2024-01-18 16:45:46 +08:00
59797cfea7
feat: es's update support doc_as_upsert
2024-01-18 16:44:56 +08:00
91368a57ff
test: add es docker CI test
2024-01-18 16:44:56 +08:00
7f5fe91905
fix: es's action is atom not binary
2024-01-18 16:44:56 +08:00
7802d6e018
chore: fix typos
2024-01-18 16:35:57 +08:00
556092b7d0
feat(authz/prometheus): add authz cache_miss counter
2024-01-18 09:01:17 +01:00
85b6a3454c
fix(authz): use binary() type instead of string() for cache.excludes
2024-01-18 09:01:17 +01:00
6d6242c27a
Merge pull request #12335 from lafirest/fix/type_clause
...
fix(iotdb): enhances type checking when converting value
2024-01-18 11:28:40 +08:00
b2af7fdd70
fix(sysk): fix a update issue for the Syskeeper forwarder
2024-01-17 12:51:10 +08:00
238ecc68cf
fix(iotdb): enhances type checking when converting value
2024-01-17 12:00:15 +08:00
4fa906731d
Merge pull request #12333 from zmstone/0116-add-tags-for-action-and-connectors
...
feat: add 'tags' field for action and connector
2024-01-16 18:56:57 +00:00
01d52e37c4
fix: resource tag type should be binary string
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2024-01-16 17:35:22 +01:00
2a41cad54f
fix(sources): remote irrelevant `resource_opts` fields for sources
...
Since they don't use buffer workers, they shouldn't have buffer-related sub-fields.
2024-01-16 11:29:35 -03:00
a8f9e5676f
docs(mqtt_bridge): add API examples
2024-01-16 10:42:52 -03:00
60fab6ee45
refactor: attempt to improve function names
2024-01-16 10:42:52 -03:00
c6cd3adccb
refactor: fix type upgrade calls and move compatiblitly logic
...
Some bridge V1 to V2 calls were wrong but did not seem to cause issues
(perhaps due to locking test coverage). This commit also move
compatibility logic from the API module to the emqx_bridge_v2 module
where most of the compatibility logic exists.
2024-01-16 10:42:52 -03:00
440a543a85
docs: fix typo
2024-01-16 10:42:52 -03:00
938429f351
chore(mqtt_bridge): change schema to remote `remote` sub-fields and hide `local`
...
`local` is still needed for backwards compatibility
2024-01-16 10:42:52 -03:00
007af20a30
test(bridge_v2_api): adapt more tests to sources
2024-01-16 10:42:52 -03:00
fc88a1ed1e
test(sources_api): add some tests to cover `/sources` HTTP API
...
Also fixes a bug with `DELETE /sources/:id`
2024-01-16 10:42:52 -03:00
8f304d3456
test(bridge_v2_api): refactor suite to use CT matrix
2024-01-16 10:42:52 -03:00
cc24fe6e93
feat(mqtt_consumer): add support for rule engine `FROM`
2024-01-16 10:42:52 -03:00
28de7c89c7
feat: add `/sources*` HTTP APIs
2024-01-16 10:42:52 -03:00
e6ccfa5b39
fix(mqtt_bridge): fixes after rebasing onto current `master`
...
Rebased on top of 7f57ec47d5
2024-01-16 10:42:52 -03:00
6511693b2e
refactor(action_api): prepare for `/sources` HTTP API
2024-01-16 10:42:52 -03:00
12dc9fbeb9
test(mqtt_bridge): fix assertion
2024-01-16 10:42:52 -03:00
862283ff7c
test: fix expected connector name after name convention generation changed
2024-01-16 10:42:52 -03:00
7fc069da46
test: fix another broken test
2024-01-16 10:42:52 -03:00
1ad3100cad
chore: add i18n
2024-01-16 10:42:52 -03:00
3597ee7c93
fix(mqtt_action): fix resource_opts schema
2024-01-16 10:42:52 -03:00
ab1b0dda67
refactor: fix typo
2024-01-16 10:42:52 -03:00
697c8f5ee1
test: fix broken tests
2024-01-16 10:42:52 -03:00
cc34660ab9
fix(actions): use backward-compatible ids
2024-01-16 10:42:52 -03:00
14b99737e9
fix(mqtt_bridge): add missing fields to POST api spec; fix test
2024-01-16 10:42:52 -03:00
7befe898d0
fix(mqtt_bridge): fix schema
2024-01-16 10:42:52 -03:00
139da6d720
fix: don't double-write the transformed config; return a triplet in all cases
2024-01-16 10:42:52 -03:00
2ecc775fb7
style: remove commented out code and fix copyright headers
2024-01-16 10:42:52 -03:00
886ed55374
fix: don't call non-existing function
2024-01-16 10:42:52 -03:00
145ed2e632
fix: elvis style error
2024-01-16 10:42:52 -03:00
f199a0f24a
feat: refactor MQTT bridge to source, action, and connector
...
This commit:
* refactors the MQTT V1 bridge into connector, source and action
* Extends the compatibility layer so it works for sources
* Fixes the MQTT bridge test suite so that all test cases passes
We still need to add a HTTP API handling sources. Also, we still need to
add HTTP API example schemes and examples for the MQTT
connector/action/source.
We should also make sure that we handle the corner cases of the MQTT V1
bridge automatic upgrade downgrade in a sufficiently good way:
* An error is currently thrown when converting an MQTT V1 bridge without
egress or ingress config.
* If there is a source and action with the same name we will currently
throw an error in the compatibility layer.
* We will also throw an error when converting an MQTT V1 bridge with
both ingress and egress.
The above is probably the right thing to do but we have to make sure
that we return a reasonable error to the user when this happens.
(partly)
Fixes:
https://emqx.atlassian.net/browse/EMQX-11489
2024-01-16 10:42:52 -03:00
996a851cf6
chore: format username as string in log formatter
2024-01-16 14:42:10 +01:00
541525c50f
fix(authz): fix authz result logs
...
prior to this fix, it's always the default authz result logged at
warning level
2024-01-16 14:40:53 +01:00
1fe1a62fe2
test: fix already exported function warning
2024-01-16 14:00:34 +01:00
ab66986f16
feat: add 'tags' field for action and connector
2024-01-16 13:57:48 +01:00
b6d0365027
Merge pull request #12267 from HJianBo/new-timeout-param-for-invite
...
feat(cluster): supports inviting nodes to join the cluster in an asynchronous manner
2024-01-16 10:11:20 +08:00
58a54adbb4
Merge remote-tracking branch 'origin/master' into release-55
2024-01-15 17:58:48 +01:00
04d3e0335b
Merge pull request #12328 from lafirest/fix/iotdb
...
fix(iotdb): ensure the `data` field is `required`
2024-01-15 23:26:36 +08:00
50ce2384d9
fix(iotdb): ensure the `data` field is `required`
2024-01-15 20:29:58 +08:00
d2d3ddb72a
test(quic): listener port updates
2024-01-15 11:42:45 +01:00
c77837a9ea
feat(quic): support reload with new binding port
2024-01-15 11:33:25 +01:00
90c2a5445c
Merge pull request #12295 from zhongwencool/elasticsearch
...
feat: support elasticsearch bridge
2024-01-15 14:13:38 +08:00
ace443fc18
refactor: refactor es's action
2024-01-15 11:31:13 +08:00
4c40e754f4
chore: forward the async invite to leader node
2024-01-15 11:24:05 +08:00
e49d3ca50c
feat: support elasticsearch bridge
2024-01-15 09:33:44 +08:00
e3fee93d9f
fix: write quoted strings to influxdb failed
2024-01-15 09:29:09 +08:00
1668e9ac7d
fix: cannot write literal numbers to influxdb
2024-01-15 09:29:09 +08:00
93ef6766ef
chore: apply suggestions from code review
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-13 15:29:58 +08:00
d2991dae03
chore: make dialyzer happy
2024-01-13 15:29:58 +08:00
81209e0ded
chore: update changes
2024-01-13 15:29:58 +08:00
944137ad45
chore: format error message
2024-01-13 15:29:58 +08:00
6ff4c560e4
feat: support invite node in async mananer
2024-01-13 15:29:58 +08:00
a08f56db42
feat(cluster): expose the timeout parameter to invite node
2024-01-13 15:29:58 +08:00
4736012d0a
Merge pull request #12317 from thalesmg/fix-mongo-batch-schema-m-20240112
...
fix(mongodb_action): make `batch_size` hidden and fixed in the API
2024-01-12 14:51:38 -03:00
eca2f972bc
Merge pull request #12237 from keynslug/fix/EMQX-11483/broker
...
feat(routing): add route sync process pool
2024-01-12 14:44:31 +01:00
a5e7db793b
fix(mongodb_action): make `batch_size` hidden and fixed in the API
...
Fixes https://emqx.atlassian.net/browse/ED-1171
The old bridge schema already had this override, but it had not been ported to actions.
2024-01-12 10:04:32 -03:00
1c0ca72877
Merge pull request #12316 from lafirest/fix/iotdb
...
fix(iotdb): move the iot_version into IoTDB connector
2024-01-12 20:56:10 +08:00
61417f26d4
chore: bump versions to 5.5.x
2024-01-12 13:54:35 +01:00
b4eac25678
chore(broker): fix code style issue
...
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 13:54:23 +01:00
d1ae6f5195
Merge pull request #12306 from thalesmg/fix-password-deobfuscate-m-20240111
...
fix(bridge_api): correctly deobfuscate secrets during dry run
2024-01-12 09:13:20 -03:00
b15106c753
fix(iotdb): robustify type verification
...
1. let the type is not case sensitive
2. return error if type is invalid
2024-01-12 19:22:06 +08:00
2f98f1faaf
fix(route-sync): ensure batch sync preserve idemopotency
2024-01-12 11:57:26 +01:00
e21a3497c7
chore(route-sync): turn TODO into comment
2024-01-12 11:57:26 +01:00
8f4758d9d4
feat(route-sync): use the smallest possible min sync delay
2024-01-12 11:57:26 +01:00
2ac6cddf19
fix(route-sync): handle batch sync errors gracefully
2024-01-12 11:57:25 +01:00
a28fc7bfa8
feat(route-sync): do not run empty batches
2024-01-12 11:57:25 +01:00
2dffd44985
feat(route-sync): allow to enable syncer pool only on cores/replicants
...
So we would able to roll it out and test more gradually.
2024-01-12 11:57:25 +01:00
884f784c1c
refactor(router): don't `emqx_broker_helper:monitor/1` in batches
...
As per out current understanding, this doesn't changes much in terms of
observability, since other nodes call `ekka:monitor(membership)` anyway,
so they will observe nodedowns without explicitly writing an entry into
the `?ROUTING_NODE` table.
2024-01-12 11:57:25 +01:00
d6f731c4fc
fix(route-sync): use public function as mria activity target
2024-01-12 11:57:25 +01:00
7d037cfe91
chore(route-sync): clarify why wait indefinitely for reply
2024-01-12 11:57:24 +01:00
498b7a922d
chore(syncer): fix code style issues
...
Co-Authored-By: Zaiming (Stone) Shi <zmstone@gmail.com>
2024-01-12 11:57:24 +01:00
0b3f5f7c37
feat(syncer): allow to turn syncer pool on/off through config
2024-01-12 11:57:24 +01:00
38e13f2337
fix(syncer): start syncer pool before broker pool
...
As the latter depends on the former.
2024-01-12 11:57:24 +01:00
a1ccf85c66
test(routesync): verify that syncer preserves consistency
...
Under a highly concurrent load. Be aware that this testcase is not
deterministic.
2024-01-12 11:57:24 +01:00
5aeff20f8b
fix(routesync): ensure causal relationships are preserved
...
At the cost of strict FIFO semantics though.
2024-01-12 11:57:23 +01:00
f92b5b3f32
feat(stream): add simple stream over process message queue
2024-01-12 11:57:23 +01:00
54f8b47455
feat(routing): add route sync process pool
...
Dedicated to synchronizing local state updates with the global view
of the routing state.
2024-01-12 11:57:18 +01:00
0f1aaa65bc
fix(iotdb): move the `iot_version` into IoTDB connector
2024-01-12 17:52:31 +08:00
9e85b53c39
Merge pull request #12296 from HJianBo/fix-delayed-message-http-api-500
...
fix(delayed): fix http 500 error
2024-01-12 09:37:41 +08:00
79a4a041e4
fix(bridge_api): correctly deobfuscate secrets during dry run
...
Fixes https://emqx.atlassian.net/browse/EMQX-11733
2024-01-11 15:42:14 -03:00
ef0850c71f
Merge pull request #12305 from savonarola/1101-fix-channel-info-update
...
fix(conn): avoid storing incomplete channel info
2024-01-11 18:48:53 +02:00
4c66a1135b
Merge pull request #12303 from savonarola/0111-fix-indexing
...
fix(retainer): fix topic search by index
2024-01-11 18:44:56 +02:00
7b0b2a0527
fix(conn): avoid storing incomplete channel info
2024-01-11 18:08:13 +03:00
09d524144b
fix(retainer): fix topic search by index
2024-01-11 16:00:58 +03:00
6ddbfd7b43
fix(gcp_pubsub_producer): mark connector resource opts fields as deprecated
...
Fixes https://emqx.atlassian.net/browse/EMQX-11703
2024-01-11 09:53:36 -03:00
c026d3f18a
Merge pull request #12300 from lafirest/fix/clients_api
...
fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT
2024-01-11 19:10:54 +08:00
f042462d53
Merge pull request #12290 from lafirest/fix/retain_match
...
fix(retain): add `hasnext` into the meta data for the `GET /retain/messages`
2024-01-11 18:09:29 +08:00
aca9942fd4
fix(api): fix `api/v5/clients ` may crash when a client authenticate with JWT
2024-01-11 16:53:02 +08:00
f896cf2f4f
fix(retain): add `hasnext` into the meta data for the `GET /retain/messages`
2024-01-11 16:14:40 +08:00
596607d549
chore: update bpapi.versions
2024-01-11 14:56:02 +08:00
c4dd083fd9
fix(delayed): delete delayed messge on all nodes
2024-01-11 14:12:30 +08:00
eea0ec135f
fix(delayed): fix http 500 error
2024-01-11 10:36:55 +08:00
b16920e796
Merge pull request #12285 from HJianBo/shorten-coap-params
...
feat(coap): support short param names
2024-01-11 09:09:09 +08:00
8bea0711ac
Merge pull request #12196 from SergeTupchiy/EMQX-10891-route-cleanup-optimization
...
EMQX-10891 route cleanup optimization
2024-01-10 17:26:24 +02:00
8532e0cf59
Merge pull request #12292 from thalesmg/fix-syskeeper-v2-only-m-20240110
...
fix(syskeeper_bridge): mark syskeeper forwarder as a v2-only action
2024-01-10 11:42:24 -03:00
965ce5d446
perf: use mria:match_delete/2 to cleanup routes
...
This must be much more network efficient since both Mria and Mnesia
need only to replicate one op, e.g.: `{MatchPattern, clear_table}`
instead of replicating one per each key to be deleted.
2024-01-10 16:23:58 +02:00
febaaefc38
chore: upgrade ekka to 0.18.1 (mria 0.8.1)
...
Mria 0.8.1 release implements `mria:match_delete/2` function
2024-01-10 16:23:58 +02:00
b395924569
Merge pull request #12283 from thalesmg/fix-gcp-pubsub-produ-resopts-m-20240109
...
fix(gcp_pubsub_producer): fix connector `resource_opts` schema
2024-01-10 11:08:39 -03:00
7c8a36fc06
Merge pull request #12289 from zmstone/0108-support-acl-cache-masks
...
0108 support acl cache excludes
2024-01-10 13:40:29 +00:00
8d61f4665a
Merge pull request #12291 from keynslug/fix/bump-hocon-0.44.4
...
fix(conf): bump to hocon 0.40.4
2024-01-10 14:32:17 +01:00
JimMoen