Commit Graph

57 Commits

Author SHA1 Message Date
zmstone 9479c8d33b fix(redis): json decode if arg for map_to_redis_hset_args is string 2024-06-17 20:19:53 +02:00
zmstone aa7ce1f641 fix(bridge/redis): add test case for map_to_redis_hset_args 2024-06-03 23:05:43 +02:00
zmstone e3ed7b59dd feat(redis): add a rule function to help formatting redis args
The new function named 'map_to_redis_hset_args' can be used to format
a map's key-value pairs into redis HSET (or HMSET) arg list.

This new function is dedicated for redis to avoid abuse for other
data integrations.
2024-06-03 14:09:15 +02:00
Kjell Winblad 916168c755 fix: make rule engine unescape convert \a to the terminal alarm char
The rule engine unescape function should convert the escape sequence \a
to the alarm bell symbol (ASCII 7). This bug was created as it was
assumed that Erlang convert $\a to 7 but Erlang does not handle the \a
escape sequence and instead convert it to 97 (ASCII code for a).

Fixes:
https://emqx.atlassian.net/browse/EMQX-12313
2024-05-08 15:18:13 +02:00
Ivan Dyachkov 923fd0a2a6
Merge pull request #12737 from id/0319-sync-release56
sync release-56
2024-03-19 19:09:41 +01:00
Ivan Dyachkov f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Thales Macedo Garitezi f84a996671 feat: implement message validation
Fixes https://emqx.atlassian.net/browse/EMQX-11980
2024-03-18 13:11:39 -03:00
Kjell Winblad 5a6f96212d feat(rule engine SQL): add an `unescape` function
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:

- Standard C escape sequences:
  - `\n` for newline (LF)
  - `\t` for horizontal tab (HT)
  - `\r` for carriage return (CR)
  - `\b` for backspace (BS)
  - `\f` for formfeed (FF)
  - `\v` for vertical tab (VT)
  - `\'` for single quote (')
  - `\"` for double quote (")
  - `\\` for backslash (\)
  - `\?` for question mark (?)
  - `\a` for alert (bell, BEL)

- Hexadecimal escape codes:
  - `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
    a-f), allowing for the encoding of arbitrary utf32 characters.

If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.

Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
Kjell Winblad 060e02c4c4
Merge pull request #12653 from kjellwinblad/kjell/rule_engine/fix/subbits/support_non_byte_sizes/EMQX-11943
fix(rule_engine): support non-byte sized input to bin2hexstr
2024-03-08 09:41:03 +01:00
Shawn 163d095dca fix: port the changes for date_to_unix_ts SQL fun from 4.4 2024-03-08 10:48:08 +08:00
zmstone e99546e009 Merge remote-tracking branch 'origin/release-56' into sync-5.5.1 2024-03-06 17:27:54 +01:00
Kjell Winblad 69114bc6c2 fix(rule_engine): support non-byte sized input to bin2hexstr
The rule engine subbits function can return a bitstring which size is
not divisible by 8. Therefore, it makes sense that the rule engine
function bin2hexstr can handle such bitstrings as well. This is fixed by
this commit.

Fixes:
https://github.com/emqx/emqx/issues/12586
https://emqx.atlassian.net/browse/EMQX-11943
2024-03-06 15:26:49 +01:00
Kjell Winblad 365d054e01 fix: add subbits/4 and subits/5 rule_engine functions
The documentation for the family of subbits functions says that the
fifth and sixth parameters are optional (since they only make sense when
the forth parameter is 'integer'). However, before this commit
`subbits/4` and `subbits/5` did not exist.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11942
https://github.com/emqx/emqx/issues/12587
2024-03-05 13:27:07 +01:00
zmstone 9929025820 fix(rule_func): time zone shift at wrong precision 2024-03-04 21:25:49 +01:00
firest 6d9b2a95c3 Merge remote-tracking branch 'origin/release-55' 2024-03-04 20:47:20 +08:00
Thales Macedo Garitezi 0dd9990f24 Merge remote-tracking branch 'origin/release-55' into sync-r55-r56-20240304 2024-03-04 09:44:30 -03:00
JimMoen 0a33f9d027
fix(calendar): leap year time to unix timestamp 2024-03-04 15:37:49 +08:00
Zaiming (Stone) Shi 46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Shawn 4afba8eb94 feat: port emqx/emqx-enterprise#1892, add some SQL functions 2024-01-24 18:50:57 +08:00
Zaiming (Stone) Shi ac41f497b4 feat: add bytesize rule sql function 2023-09-15 16:04:21 +02:00
firest 9f38f5f26a fix(calendar): make date parse error reason more sense 2023-08-08 14:28:38 +08:00
Andrew Mayorov a51baaa206
refactor(pluglib): move conversion utils to `emqx_utils_conv` 2023-06-09 14:44:37 +03:00
Kjell Winblad 354603da01 test: helper function was interpreted as a test function 2023-05-29 17:20:37 +02:00
Kjell Winblad 995025b572 feat: add timezone_to_second/1 function to rule engine
This commit adds the functions timezone_to_offset_seconds and its alias
timezone_to_second to the rule engine. As the names suggests, these
functions convert a timzone offset string such as "+02:00", "Z", "local"
to an integer representing how many seconds that the given timezone
differs from UTC. timezone_to_offset_seconds is the one of the two
functions that should be advertised while timezone_to_second in kept for
backwards compatibility with 4.X.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10058
2023-05-29 14:30:43 +02:00
firest 56a6b699ac fix: port the `emqx_calendar` from v4.4 2023-05-22 09:55:31 +08:00
Kjell Winblad 0c727fc9b2
Merge pull request #10392 from kjellwinblad/kjell/rule_engine/add_missing_datetime_function/EMQX-9245
freat: add date_to_unix_ts/3 function to the rule engine
2023-04-17 18:09:44 +02:00
Zaiming (Stone) Shi 18974a8e11 refactor: make schema dump and swagger spec work with split desc files 2023-04-17 13:58:11 +02:00
Kjell Winblad 37f42a486c test: improve proper test so it generates more cases 2023-04-17 12:03:13 +02:00
Zhongwen Deng 8facd130f6 chore: update cluster_hocon_file/0 function 2023-04-14 23:19:38 +08:00
Kjell Winblad d3ccd8a65d feat: add date_to_unix_ts/3 function to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-9245
2023-04-13 16:14:03 +02:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Zaiming (Stone) Shi f5e9d3eab2 test: use valid config for jq function timeout 2023-03-28 14:29:24 +02:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
JimMoen 4906da9796
test(rule-funcs): test compression funcs 2022-12-19 15:43:50 +08:00
Shawn 6685a3c5a8 fix: remove the 'headers' field from the rule events 2022-06-17 15:57:10 +08:00
Xinyu Liu 866810cea6
Merge pull request #8006 from emqx/copy-of-kjell/jq/timeout
feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
2022-05-23 11:58:40 +08:00
EMQ-YangM 005fce63d1 feat: add rule engine function float2str/2 2022-05-19 16:16:05 +08:00
EMQ-YangM 66c62f432b fix: zero padding should be to the MSB side 2022-05-18 15:28:01 +08:00
Kjell Winblad 05032467bd feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
This commit adds a default timeout of 10 seconds to the rule engine's
`jq/2` function, and adds a new function `jq/3` (where the last parameter is
a timeout value). The default timeout can be configured with the setting
"rule_engine.jq_function_default_timeout".

Having a timeout when executing jq code in the rule engine is important
as jq code can potentially run forever. Also, the Erlang jq library
limits the number of jq programs that can execute concurrently so a jq
program that loops forever could potentially also prevent a "non-buggy"
jq program from ever starting.
2022-05-17 19:36:32 +02:00
EMQ-YangM d8da8f1686 fix: hexstr_to_bin support half byte 2022-05-17 16:47:59 +08:00
Kjell Winblad 74c33cd4e5 feat(rule_engine): add jq function to the rule engine
This commit adds a function to the rule engine that alows users
to transform text or JSON objects using [jq filter programs][1].

[jq][1] is a command line tool that can be used to transform
and filter JSON text using jq's built-in language. The rule engine
function that is added with this commit uses the
[Erlang jq NIF library][2] that wraps the jq C library in an
Erlang NIF function.

[1]: https://stedolan.github.io/jq/
[2]: https://github.com/emqx/jq
2022-05-04 17:09:16 +02:00
Zaiming (Stone) Shi 02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
EMQ-YangM e4b62f3a5f feat: add rule-engine functions 2022-04-15 18:02:24 +08:00
Kian-Meng Ang fc92e4c8bf docs: fix typos 2022-01-27 12:29:38 +01:00
Zaiming (Stone) Shi 63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
lafirest 3524942766 fix: change all ~s in io format to ~ts
we should use unicode to replace latin1
2021-10-12 15:09:24 +08:00
Zaiming Shi 6354e75626 chore: update copyrights 2021-04-23 20:43:00 +02:00
Shawn 19a9bab3a4 feat(rule_engine): add rule sql functions for datetime rfc3339 2021-04-22 10:32:44 +08:00
Shawn 196fef0255 chore(rule): migrate rule-engine from enterprise version 2021-02-22 13:53:51 +08:00
Shawn c2e1bc039b
feat(rule): add more sql functions #4143 (#4144) 2021-02-20 17:10:10 +08:00