8f8b023429
refactor: simplify match spec holder construction
2024-03-18 18:00:26 +02:00
5a6f96212d
feat(rule engine SQL): add an `unescape` function
...
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:
- Standard C escape sequences:
- `\n` for newline (LF)
- `\t` for horizontal tab (HT)
- `\r` for carriage return (CR)
- `\b` for backspace (BS)
- `\f` for formfeed (FF)
- `\v` for vertical tab (VT)
- `\'` for single quote (')
- `\"` for double quote (")
- `\\` for backslash (\)
- `\?` for question mark (?)
- `\a` for alert (bell, BEL)
- Hexadecimal escape codes:
- `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
a-f), allowing for the encoding of arbitrary utf32 characters.
If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.
Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943
...
fix(rule_engine): support non-byte sized input to bin2hexstr
2024-03-08 09:41:03 +01:00
6054499607
Merge pull request #12657 from kjellwinblad/kjell/rule_engine/fix/fun_calls_as_array_items/EMQX-11953
...
fix(rule SQL): allow expressions as array items
2024-03-08 09:39:46 +01:00
163d095dca
fix: port the changes for date_to_unix_ts SQL fun from 4.4
2024-03-08 10:48:08 +08:00
e99546e009
Merge remote-tracking branch 'origin/release-56' into sync-5.5.1
2024-03-06 17:27:54 +01:00
c206ee37e0
fix(rule SQL): allow expressions as array items
...
Fixes:
https://github.com/emqx/emqx/issues/12465
https://emqx.atlassian.net/browse/EMQX-11953
2024-03-06 15:38:19 +01:00
69114bc6c2
fix(rule_engine): support non-byte sized input to bin2hexstr
...
The rule engine subbits function can return a bitstring which size is
not divisible by 8. Therefore, it makes sense that the rule engine
function bin2hexstr can handle such bitstrings as well. This is fixed by
this commit.
Fixes:
https://github.com/emqx/emqx/issues/12586
https://emqx.atlassian.net/browse/EMQX-11943
2024-03-06 15:26:49 +01:00
365d054e01
fix: add subbits/4 and subits/5 rule_engine functions
...
The documentation for the family of subbits functions says that the
fifth and sixth parameters are optional (since they only make sense when
the forth parameter is 'integer'). However, before this commit
`subbits/4` and `subbits/5` did not exist.
Fixes:
https://emqx.atlassian.net/browse/EMQX-11942
https://github.com/emqx/emqx/issues/12587
2024-03-05 13:27:07 +01:00
9929025820
fix(rule_func): time zone shift at wrong precision
2024-03-04 21:25:49 +01:00
6d9b2a95c3
Merge remote-tracking branch 'origin/release-55'
2024-03-04 20:47:20 +08:00
0dd9990f24
Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304
2024-03-04 09:44:30 -03:00
0a33f9d027
fix(calendar): leap year time to unix timestamp
2024-03-04 15:37:49 +08:00
46877e979b
chore: update copyright-year
2024-02-23 08:21:06 +01:00
d469f4158e
chore: bump app vsns
2024-02-20 16:53:57 -03:00
e9c8446d57
feat: upgrade erlfmt to support maybe syntax
2024-01-31 20:06:49 +08:00
4afba8eb94
feat: port emqx/emqx-enterprise#1892 , add some SQL functions
2024-01-24 18:50:57 +08:00
cc24fe6e93
feat(mqtt_consumer): add support for rule engine `FROM`
2024-01-16 10:42:52 -03:00
23ded313ec
chore: update app versions
2023-12-22 15:29:22 +01:00
322b7bb7d2
chore: bump app vsn
2023-12-22 13:00:37 +01:00
18b69809da
Merge branch 'release-54' into sync-r54-m-20231221
2023-12-21 10:05:13 -03:00
b61ccf3373
fix(bridges_v1): don't list v2-only bridges in API
...
Fixes https://emqx.atlassian.net/browse/EMQX-11614
2023-12-20 10:46:35 -03:00
cf9331a95f
Merge branch 'release-54' into sync-r54-m-20231218
2023-12-18 17:21:08 -03:00
35504bd323
refactor: move ntoa (ip address formatting) code to emqx_utils
2023-12-17 21:03:16 +01:00
57a8f2351d
feat(rule_engine): also bump `unknown` counter for unrecoverable action errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-11494
2023-12-14 13:42:28 -03:00
99399c6e95
Merge pull request #12098 from thalesmg/revert-downgrade-bridge-id-m-20231205
...
chore: stop downgrading action id in rule API responses
2023-12-05 14:26:23 -03:00
e5a4be11f5
chore: stop downgrading action id in rule API responses
...
Fixes https://emqx.atlassian.net/browse/EMQX-11405
2023-12-05 11:48:46 -03:00
cea857cb8e
fix(rule_api): fix sql test when testing against events
...
Fixes https://emqx.atlassian.net/browse/EMQX-11449
2023-12-04 15:24:09 -03:00
7f4d91d490
Merge remote-tracking branch 'origin/release-54' into 1201-sync-release-54
2023-12-01 08:32:22 +01:00
9fd2fa95a8
chore: bump apps versions
2023-11-30 20:01:12 +01:00
ec10c51073
Merge remote-tracking branch 'upstream/release-53' into 1129-sync-r53
2023-11-30 19:51:12 +01:00
14644988e0
chore: change triple-quotes to single-quotes
2023-11-29 16:15:18 +01:00
0aec2f7605
feat: redis bridge v2
2023-11-29 16:06:46 +08:00
c8b5c51bbc
chore: fix failed test cases
2023-11-28 09:53:46 +08:00
dc99651690
test(bridge): ensure almost test cases passed
2023-11-28 09:53:46 +08:00
84ff7b0b38
feat(emqx_bridge): action_info with dynamic lookup
...
This allows a n:1 relation between v1 bridge_types to action/connector types as
it's the case with mongodb for instance, where we had `mongodb_single`
`mongodb_sharded` etc and the new implementation will just have `mongodb`.
2023-11-20 10:22:37 +01:00
7c0e345d3a
Merge remote-tracking branch 'upstream/release-54'
2023-11-14 19:38:21 +01:00
28a577ad09
chore: bump apps versions
2023-11-14 11:02:26 +01:00
1c57993c91
Merge remote-tracking branch 'upstream/release-53' into 1114-sync-r53
2023-11-14 10:44:35 +01:00
d26a1b9afb
fix(bridge_v1): no hard coded downgrade and upgrade type functions
2023-11-14 09:20:46 +01:00
86110824eb
feat: upgrade hocon to 0.40.0 which supports union type display name
2023-11-10 13:41:51 +01:00
d603de10e6
refactor(schema): add namespace/0 to all schema modules
2023-11-10 13:41:51 +01:00
0bf5deaed9
chore: merge master into release-54
2023-11-09 08:32:26 +01:00
0c91bec98d
chore: merge 'upstream/release-53'
2023-11-08 09:24:38 +01:00
f17b762596
chore: don't disable rule that references non-existent bridge
...
After feedback from QA, we decided to rollback enforcing the rule to be disabled.
2023-11-06 10:09:14 -03:00
0ee74cd8ae
fix(rules_api): downgrade bridge id
...
Fixes https://emqx.atlassian.net/browse/EMQX-11312
2023-11-03 15:57:13 -03:00
c84e4a4187
fix(rule_engine): don't enable a rule that references non-existent bridge
2023-11-03 09:24:53 -03:00
910e81bc41
Merge pull request #10442 from keynslug/ft/EMQX-9257/placeholder
...
feat(tpl): split `emqx_placeholder` into a couple of modules
2023-11-02 22:50:05 +07:00
729c6edff6
chore(repub): simplify error handling in MQTT props templating
2023-11-02 20:13:32 +07:00
02c1bd70b6
feat(tpl): factor out loose json concept into a separate module
...
Which is called `emqx_jsonish`. Also introduce an _access module_
abstraction to extract information from such data during rendering.
2023-11-02 20:13:31 +07:00
69cfa740ea
fix(ruleeng): ensure full backward compatibility
2023-11-02 17:11:12 +07:00
75cc663786
chore(ruleeng): streamline application dependencies
2023-11-02 17:11:12 +07:00
8e4585d64f
chore: move template modules to `emqx_utils`
...
Even though most of the time these modules will be used by
connectors, there are exceptions (namely, `emqx_rule_engine`).
Besides, they are general enough to land there, more so given
that `emqx_placeholder` is already there.
2023-11-02 17:11:12 +07:00
f689d6c233
fix(tpl): ensure backward compat in `emqx_rule_engine`
...
Missing bindings in string templates will be rendered as "undefined",
as before. Rendering still assumes that missing binding with implicit
default (`undefined`) is an error.
This will also restore complete backward compat in `emqx_prometheus`.
2023-11-02 17:11:11 +07:00
49f5325c67
feat(tpl): unify validations / errors var representations
2023-11-02 17:11:10 +07:00
b812f9af5a
feat(tpl): use `emqx_connector_template` in `emqx_rule_engine` app
2023-11-02 17:11:10 +07:00
b1ab213081
chore: merge 'upstream/release-53' into 1031-sync-r53
2023-10-31 11:06:25 +01:00
9dc3a169b3
feat: split bridges into a connector part and a bridge part
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
Co-authored-by: Stefan Strigler <stefan.strigler@emqx.io>
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
Several bridges should be able to share a connector pool defined by a
single connector. The connectors should be possible to enable and
disable similar to how one can disable and enable bridges. There should
also be an API for checking the status of a connector and for
add/edit/delete connectors similar to the current bridge API.
Issues:
https://emqx.atlassian.net/browse/EMQX-10805
2023-10-30 14:48:47 +01:00
d563121284
refactor: move ?REDISPATCH_TO macro to emqx_mqtt.hrl
2023-10-27 17:54:14 +08:00
b5411da770
refactor: subscribe process to fix shared-sub
2023-10-27 09:23:43 +08:00
8c4beec6f0
test(rule_engine): add test to exemplify `foreach` with json array payload
...
Relates to https://emqx.atlassian.net/browse/EMQX-11174
2023-10-19 18:19:42 -03:00
03d8e06ff7
chore(emqx_rule_funcs): regroup export functions
2023-10-09 20:03:21 +02:00
1eb75b43c4
chore(auth): split emqx_authn and emqx_authz apps
2023-10-05 13:41:50 +03:00
dc147fd310
fix(rule-engine): console action has no args field
2023-09-29 10:33:33 +02:00
dafd7c6085
chore: bump apps versions
2023-09-21 10:58:42 +02:00
105bebc250
chore: merge release-52 into master
2023-09-21 10:22:47 +02:00
8b6d5d4ca9
test: add test for build_console's args
2023-09-18 16:48:38 +08:00
5490807b20
chore: update apps/emqx_rule_engine/src/emqx_rule_engine_schema.erl
...
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-09-18 16:11:39 +08:00
ac41f497b4
feat: add bytesize rule sql function
2023-09-15 16:04:21 +02:00
32d838c85b
fix: format console action crash
2023-09-15 17:40:56 +08:00
d72348c8ed
fix: keep builtin_console actions not complain args
2023-09-15 16:38:04 +08:00
d0f1c75651
chore: bump apps vsns
2023-09-11 16:06:53 -03:00
e089fda260
Merge pull request #11568 from thalesmg/republish-props-m-20230905
...
feat(republish): allow templating mqtt properties
2023-09-07 16:43:00 -03:00
68293231b8
test: add cases for more complex placeholders
2023-09-07 13:40:03 -03:00
014bc64d3b
chore: ensure general prop is a binary
2023-09-07 13:40:03 -03:00
02b8bbf76a
chore: drop templating support for `Subscription-Identifier`
...
This is highly-dependent on the session state, as is `Topic-Alias`.
2023-09-07 13:40:03 -03:00
47bd19dee4
chore: lower error log messages down to debug
2023-09-07 13:40:03 -03:00
bb55b04d46
refactor: inline fn clauses, improve error logging
2023-09-06 13:54:40 -03:00
c9100c7591
refactor: improve logging information and descriptions
...
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-09-06 13:33:59 -03:00
78c5a779d7
feat(republish): allow templating mqtt properties
...
Fixes https://emqx.atlassian.net/browse/EMQX-10912
2023-09-06 09:54:02 -03:00
18c6bfec97
chore: bump app vsns
2023-09-06 11:12:56 +02:00
e794143ae1
Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master
2023-09-06 09:08:22 +02:00
b1defa29d7
feat(ruleeng): avoid storing whole rules in topic index
...
Because it doesn't really give any benefit, but wastes memory by
duplication.
2023-08-30 20:07:56 +04:00
166375a000
fix(topicidx): make `get_record/2` simpler to use in concurrent env
...
The mechanic of `emqx_topic_index` cannot really guarantee atomicity
of reading records associated with index matches, so instead it's
probably better to make the user aware of that lack of this guarantee.
2023-08-28 13:52:59 +04:00
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
...
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
66dec69d09
fix(rule_engine): capture function_clause errors
...
Fixes https://emqx.atlassian.net/browse/EMQX-10798
2023-08-18 13:51:19 -03:00
75ed6aa8e7
Merge pull request #11396 from keynslug/ft/EMQX-10712/ruleeng-topic-index-vol-2
...
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-17 20:43:09 +04:00
040d28c35e
fix(cli): fix two typos in the time unit
2023-08-16 18:46:17 +08:00
8cd21da94b
chore: update apps version
2023-08-16 10:55:54 +08:00
b08102269a
refactor(calendar): refactor datetime-related code and remove redundant
2023-08-16 10:55:46 +08:00
fe9477f92e
chore: bump applications versions
...
* emqx_rule_engine 5.0.23
2023-08-14 15:36:58 +04:00
47dfba4341
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-08-14 15:36:58 +04:00
bef8eddad3
Merge pull request #11405 from lafirest/fix/kal_kialo
...
fix(calendar): make date parse error reason more sense
2023-08-09 14:42:08 +08:00
b38461e50a
fix: mongo_date/2 shall give user friendly value in the test environment
2023-08-08 14:47:07 +02:00
9f38f5f26a
fix(calendar): make date parse error reason more sense
2023-08-08 14:28:38 +08:00
3ed031db70
fix: rule SQL mongo_date function should return a string in test mode
...
The rule SQL mongo_date function should return a string with the format
ISODate(*), where * is an ISO date string when running the rule in test
mode.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10727
2023-08-07 16:10:30 +02:00
63adeabf72
chore: bump app versions
2023-07-27 15:29:03 +02:00
951a96457b
Revert "feat(index): add topic index facility "
2023-07-27 13:42:43 +08:00
5e4855334e
Revert "Fix(topicidx): allow to return matches unique by record id"
2023-07-27 13:39:37 +08:00
2c193f26e6
Merge pull request #11332 from HJianBo/fix-topic-index-unique-bugs
2023-07-25 19:07:07 +08:00
d05a5cfe0f
fix(rule): fix the `matches/2` for some edge cases
2023-07-25 14:35:18 +08:00
e630331de1
fix(rule): fix a quering problem when 'a/b' and 'a/b/#' exist at the same time.
...
When using `ets:next` to query the next level of topic words, we should prioritize the next level
of '#', '+'.
2023-07-24 23:04:53 +08:00
dcf4819c04
test(rule): add tests to ensure the rules ordering
2023-07-24 19:30:34 +08:00
0496038361
fix(ruleeng): ensure topic index matched rules evalauted once
2023-07-24 17:44:12 +08:00
6432c9c8fc
fix(topicidx): allow to return matches unique by record id
2023-07-24 17:43:11 +08:00
af6405fa25
fix(nested_put): fix a data loss bug introduced by #11172
2023-07-24 16:39:37 +08:00
c393c2e091
test: ets table cleanup after cases
2023-07-24 09:33:28 +08:00
4e4b1ac115
refactor: module move to app emqx_rule_engine
...
- Rename to emqx_rule_index.erl
- Remove test funcs from src -> test dir
2023-07-24 09:33:28 +08:00
b821bdee00
perf(ruleeng): employ `emqx_topic_index` to speed up topic matching
2023-07-24 09:27:31 +08:00
2531c3e7d1
Merge pull request #11306 from thalesmg/rule-actions-reply-dropped-r51-20230719
...
fix(rule_metrics): notify rule metrics of late replies and expired requests
2023-07-21 16:51:14 -03:00
a1a7c4fab6
fix(emqx_rule_engine): fix typo in `behaviour` module attribute
...
The type resulted in missing `rule_engine` config after importing data from a backup file.
Fixes: EMQX-10590
2023-07-21 10:33:56 +03:00
eb41b77de4
fix(rule_metrics): notify rule metrics of late replies and expired requests
...
Fixes https://emqx.atlassian.net/browse/EMQX-10600
2023-07-19 11:39:28 -03:00
ab5fd1e5c3
Merge pull request #11260 from thalesmg/fix-rule-maps-nested-put-r51
...
fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
2023-07-14 15:02:24 -03:00
f9452241bd
test(rules): add a few more tests to assert our rule evaluation behavior
2023-07-14 10:23:20 -03:00
4034bcbb26
fix(rule_runtime): avoid rewriting select clause sources (by @keynslug)
...
Thanks to @keynslug for the insight and the patch.
With this, we avoid rewriting the parsed select clause sources, and
attempt to reuse the result context when reading values during
evaluation.
2023-07-13 14:46:08 -03:00
bffef386c1
fix(rule_maps): avoid losing data when using `emqx_rule_maps:nested_put`
...
Fixes https://emqx.atlassian.net/browse/EMQX-10541
2023-07-12 14:28:18 -03:00
541d03a0ba
refactor(schema_registry): refactor schema registry app and modules
...
Fixes https://emqx.atlassian.net/browse/EMQX-10361
- Moves `lib-ee/emqx_ee_schema_registry` to `apps/emqx_schema_registry`.
- Removes the `_ee_` segment from module names.
- Exceptions are the table names which are kept to avoid backwards incompatibilities.
2023-07-12 09:24:15 -03:00
19f9fc5089
feat(authz): bump app versions
2023-07-07 19:41:34 +03:00
7de26a1776
feat(authz): use extensible map format for actions in authz rules
...
* support authorization on retain, qos fields
* refactored authz tests heavily
2023-07-07 19:40:34 +03:00
bf5167a8bf
Merge pull request #11172 from lafirest/fix/dup_payload
...
fix(ruleengine): fix duplicate `payload` problems
2023-07-05 14:20:26 +08:00
33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
...
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
902603780b
test(ruleengine): add test for the fix of duplicate payload
2023-07-03 18:22:35 +08:00
bd4f207365
fix(ruleengine): fix duplicate `payload` problems
2023-07-03 14:21:56 +08:00
07cf250093
Merge pull request #11126 from sstrigler/EMQX-8842-fix-rule-metrics
...
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-30 20:07:23 +02:00
714363bd01
fix: decouple emqx_rule_engine application from emqx_ee_schema_registry
...
This commit decouples the emqx_rule_engine application from the
emqx_ee_schema_registry application by making it possible to register a
callback module that defines extra rule engine SQL functions instead of
calling a module in emqx_ee_schema_registry directly from the
emqx_rule_engine application.
2023-06-30 16:56:16 +02:00
039e27a153
feat: add Sparkplug encode and decode functions to the rule engine
...
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
872698d19e
feat(ruleengine): port random && uuid_v4 functions from v4
2023-06-30 14:38:27 +08:00
14da61194e
fix: handle resource errors
2023-06-29 16:09:45 +02:00
321fd53132
fix: use ReplyTo in QUERY for async
2023-06-29 16:09:45 +02:00
d71d70c889
fix: allow to pass in Opts rather than just a ReplyTo
2023-06-29 16:09:45 +02:00
422c79583a
test(emqx_rule_engine): fix simple sync
2023-06-29 16:09:45 +02:00
837acd4234
test(emqx_rule_engine): test rule metrics
2023-06-29 16:09:45 +02:00
ae636a52d7
fix(emqx_rule_engine): set inc_action_metrics as async_reply_fun
2023-06-29 16:09:45 +02:00
7cf8a6c892
chore: bump app vsns
2023-06-21 16:36:51 +02:00
59ed8798c3
Merge pull request #11026 from kjellwinblad/kjell/fix/rule_engine/div_mod_strange/EMQX-10216
...
fix: rule engine different behavior for div and mod
2023-06-17 07:20:37 +02:00
3331af4757
feat: make gateway and slow_subs low level and hide rule_engine
2023-06-13 11:47:38 +08:00
cf31b65076
fix: rule engine different behavior for div and mod
...
Previously, the div operation could only be used as an infix operation
while mod could only be used as a function call. After this commit, one
can use both div and mod using function call syntax and infix syntax.
Fixes: https://emqx.atlassian.net/browse/EMQX-10216
2023-06-12 17:20:52 +02:00
97850de524
Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master
2023-06-10 12:23:55 +02:00
a51baaa206
refactor(pluglib): move conversion utils to `emqx_utils_conv`
2023-06-09 14:44:37 +03:00
7d0abb6146
feat(emqx): add `emqx_topic:match_any/2` utility
2023-06-09 14:44:37 +03:00
d6c1ee183f
refactor(pluglib): move `emqx_placeholder` to utils app
...
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
e4d09d4ad4
feat: implement configuration and user data export/import CLI
...
Closes: EMQX-9203
2023-06-09 14:11:47 +03:00
711cedc282
fix: missing 404 response in rule engine api schema
2023-06-08 11:46:16 +02:00
79d430cf03
refactor(pluglib): move `emqx_plugin_libs_proto_v1` into emqx app
...
And rename it to `emqx_metrics_proto_v1` in the process.
2023-06-07 18:30:12 +03:00
46393343e2
chore: use `timeout_duration` types for timer fields
...
Fixes https://emqx.atlassian.net/browse/EMQX-10020
2023-06-05 11:46:38 -03:00
367a0e312a
chore: remove config_path() function
2023-06-02 07:09:08 +08:00
2e5401f3cb
fix: delete rule_engine failed
2023-06-02 00:30:07 +08:00
c27d844244
feat: improve authz/bridge/rule_engine/schema_registry config update
2023-06-01 23:20:56 +08:00
29fe201676
fix(ruleeng): parse bridge id on create / update
...
Instead of parsing it every time in the `send_message/2` hot path.
2023-05-31 17:08:42 +03:00
57aacb471c
fix(rule_engine_api): don't crash when formatting empty metrics
...
Fixes https://emqx.atlassian.net/browse/EMQX-10073
Fixes https://github.com/emqx/emqx/issues/10714#issuecomment-1567987664
Similar issue to https://github.com/emqx/emqx/pull/10743 , but on the
rule engine API.
2023-05-30 11:54:27 -03:00
Serge Tupchii