Commit Graph

74 Commits

Author SHA1 Message Date
zmstone
c8d6976b14 feat: add conditions to variform expressions
- refactored `coalesce` function to allow lazy evaluation
- added `iif(Cond, IfExpr, EleseExpr)` to allow simple conditions
2024-04-30 07:28:27 +02:00
zmstone
f80d078de3 feat(variform): Add more functions 2024-04-15 16:56:52 +02:00
zmstone
b76b6fbe63 feat(variform): initialize client_attrs with variform
Moved regular expression extraction as a variform function.
2024-04-14 10:13:24 +02:00
zmstone
5f26e4ed5e feat(variform): implement variform engine 2024-03-28 19:34:57 +01:00
zmstone
ad95473aae refactor: move string functions to emqx_variform 2024-03-28 18:03:37 +01:00
Ivan Dyachkov
f2dc940436 Merge remote-tracking branch 'upstream/release-56' into 0319-sync-release56 2024-03-19 15:20:08 +01:00
Kjell Winblad
5a6f96212d feat(rule engine SQL): add an unescape function
The added `unescape` function unescapes escape sequences, transforming
them back to their represented characters. The following escape
sequences are supported:

- Standard C escape sequences:
  - `\n` for newline (LF)
  - `\t` for horizontal tab (HT)
  - `\r` for carriage return (CR)
  - `\b` for backspace (BS)
  - `\f` for formfeed (FF)
  - `\v` for vertical tab (VT)
  - `\'` for single quote (')
  - `\"` for double quote (")
  - `\\` for backslash (\)
  - `\?` for question mark (?)
  - `\a` for alert (bell, BEL)

- Hexadecimal escape codes:
  - `\xH...` where `H...` is one or more hexadecimal digits (0-9, A-F,
    a-f), allowing for the encoding of arbitrary utf32 characters.

If an escape sequence is not recognized, or if the hexadecimal escape
does not form a valid Unicode character, the function generates an
exception.

Fixes:
https://github.com/emqx/emqx/issues/12460
https://emqx.atlassian.net/browse/EMQX-11847
2024-03-08 12:05:06 +01:00
Shawn
163d095dca fix: port the changes for date_to_unix_ts SQL fun from 4.4 2024-03-08 10:48:08 +08:00
Kjell Winblad
69114bc6c2 fix(rule_engine): support non-byte sized input to bin2hexstr
The rule engine subbits function can return a bitstring which size is
not divisible by 8. Therefore, it makes sense that the rule engine
function bin2hexstr can handle such bitstrings as well. This is fixed by
this commit.

Fixes:
https://github.com/emqx/emqx/issues/12586
https://emqx.atlassian.net/browse/EMQX-11943
2024-03-06 15:26:49 +01:00
Kjell Winblad
365d054e01 fix: add subbits/4 and subits/5 rule_engine functions
The documentation for the family of subbits functions says that the
fifth and sixth parameters are optional (since they only make sense when
the forth parameter is 'integer'). However, before this commit
`subbits/4` and `subbits/5` did not exist.

Fixes:
https://emqx.atlassian.net/browse/EMQX-11942
https://github.com/emqx/emqx/issues/12587
2024-03-05 13:27:07 +01:00
Zaiming (Stone) Shi
46877e979b chore: update copyright-year 2024-02-23 08:21:06 +01:00
Shawn
4afba8eb94 feat: port emqx/emqx-enterprise#1892, add some SQL functions 2024-01-24 18:50:57 +08:00
Zaiming (Stone) Shi
03d8e06ff7 chore(emqx_rule_funcs): regroup export functions 2023-10-09 20:03:21 +02:00
Zaiming (Stone) Shi
ac41f497b4 feat: add bytesize rule sql function 2023-09-15 16:04:21 +02:00
Ivan Dyachkov
999988cab4
Merge pull request #11401 from kjellwinblad/kjell/fix/mongo_date_format/EMQX-10727
fix: rule SQL mongo_date function should return a string in test mode
2023-08-21 16:20:59 +02:00
firest
b08102269a refactor(calendar): refactor datetime-related code and remove redundant 2023-08-16 10:55:46 +08:00
Kjell Winblad
b38461e50a fix: mongo_date/2 shall give user friendly value in the test environment 2023-08-08 14:47:07 +02:00
Kjell Winblad
3ed031db70 fix: rule SQL mongo_date function should return a string in test mode
The rule SQL mongo_date function should return a string with the format
ISODate(*), where * is an ISO date string when running the rule in test
mode.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10727
2023-08-07 16:10:30 +02:00
Thales Macedo Garitezi
541d03a0ba refactor(schema_registry): refactor schema registry app and modules
Fixes https://emqx.atlassian.net/browse/EMQX-10361

- Moves `lib-ee/emqx_ee_schema_registry` to `apps/emqx_schema_registry`.
- Removes the `_ee_` segment from module names.
  - Exceptions are the table names which are kept to avoid backwards incompatibilities.
2023-07-12 09:24:15 -03:00
lafirest
bf5167a8bf
Merge pull request #11172 from lafirest/fix/dup_payload
fix(ruleengine): fix duplicate `payload` problems
2023-07-05 14:20:26 +08:00
Kjell Winblad
33cb29efdb
Merge pull request #11169 from kjellwinblad/kjell/improved_sparkplugb/EMQX-10351
feat: add Sparkplug encode and decode functions to the rule engine
2023-07-04 07:24:06 +02:00
firest
902603780b test(ruleengine): add test for the fix of duplicate payload 2023-07-03 18:22:35 +08:00
Kjell Winblad
714363bd01 fix: decouple emqx_rule_engine application from emqx_ee_schema_registry
This commit decouples the emqx_rule_engine application from the
emqx_ee_schema_registry application by making it possible to register a
callback module that defines extra rule engine SQL functions instead of
calling a module in emqx_ee_schema_registry directly from the
emqx_rule_engine application.
2023-06-30 16:56:16 +02:00
Kjell Winblad
039e27a153 feat: add Sparkplug encode and decode functions to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-10429
2023-06-30 10:06:56 +02:00
firest
872698d19e feat(ruleengine): port random && uuid_v4 functions from v4 2023-06-30 14:38:27 +08:00
Andrew Mayorov
a51baaa206
refactor(pluglib): move conversion utils to emqx_utils_conv 2023-06-09 14:44:37 +03:00
Kjell Winblad
995025b572 feat: add timezone_to_second/1 function to rule engine
This commit adds the functions timezone_to_offset_seconds and its alias
timezone_to_second to the rule engine. As the names suggests, these
functions convert a timzone offset string such as "+02:00", "Z", "local"
to an integer representing how many seconds that the given timezone
differs from UTC. timezone_to_offset_seconds is the one of the two
functions that should be advertised while timezone_to_second in kept for
backwards compatibility with 4.X.

Fixes:
https://emqx.atlassian.net/browse/EMQX-10058
2023-05-29 14:30:43 +02:00
Thales Macedo Garitezi
6c414ab991 fix(schema_registry): ensure schema_encode output in rule engine is a binary (r5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9981
2023-05-24 09:13:38 -03:00
firest
56a6b699ac fix: port the emqx_calendar from v4.4 2023-05-22 09:55:31 +08:00
Kjell Winblad
7d3367467a
Merge pull request #10408 from kjellwinblad/kjell/rule_engine/add_missing_mongo_date_functions/EMQX-9244
feat: add mongo_date functions to the rule engine
2023-04-20 09:46:45 +02:00
Kjell Winblad
29584ca721 feat: add mongo_date functions to the rule engine
This commit adds mong_date built-in functions to the rule engine
SQL-like language. Corresponding functions already existed in EMQX 4.4
and this commit makes sure that EMQX 5.X also has these functions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9244
2023-04-18 10:01:35 +02:00
Kjell Winblad
0c727fc9b2
Merge pull request #10392 from kjellwinblad/kjell/rule_engine/add_missing_datetime_function/EMQX-9245
freat: add date_to_unix_ts/3 function to the rule engine
2023-04-17 18:09:44 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
Kjell Winblad
a66d01d6f0
style: remove code duplication
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-04-14 12:48:03 +02:00
Kjell Winblad
d3ccd8a65d feat: add date_to_unix_ts/3 function to the rule engine
Fixes:
https://emqx.atlassian.net/browse/EMQX-9245
2023-04-13 16:14:03 +02:00
Thales Macedo Garitezi
33100ecca6 feat: implement schema registry for 5.0 (avro)
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
2023-04-06 16:28:45 -03:00
Zaiming (Stone) Shi
dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
JimMoen
5151242583
feat(rule-engine): zip, gzip, zip_compress funcs in rule-sql 2022-12-19 15:43:50 +08:00
Xinyu Liu
866810cea6
Merge pull request #8006 from emqx/copy-of-kjell/jq/timeout
feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
2022-05-23 11:58:40 +08:00
EMQ-YangM
005fce63d1 feat: add rule engine function float2str/2 2022-05-19 16:16:05 +08:00
Kjell Winblad
05032467bd feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
This commit adds a default timeout of 10 seconds to the rule engine's
`jq/2` function, and adds a new function `jq/3` (where the last parameter is
a timeout value). The default timeout can be configured with the setting
"rule_engine.jq_function_default_timeout".

Having a timeout when executing jq code in the rule engine is important
as jq code can potentially run forever. Also, the Erlang jq library
limits the number of jq programs that can execute concurrently so a jq
program that loops forever could potentially also prevent a "non-buggy"
jq program from ever starting.
2022-05-17 19:36:32 +02:00
Shawn
82da4d82be chore: update jq to v0.2.0 2022-05-06 22:20:35 +08:00
Kjell Winblad
74c33cd4e5 feat(rule_engine): add jq function to the rule engine
This commit adds a function to the rule engine that alows users
to transform text or JSON objects using [jq filter programs][1].

[jq][1] is a command line tool that can be used to transform
and filter JSON text using jq's built-in language. The rule engine
function that is added with this commit uses the
[Erlang jq NIF library][2] that wraps the jq C library in an
Erlang NIF function.

[1]: https://stedolan.github.io/jq/
[2]: https://github.com/emqx/jq
2022-05-04 17:09:16 +02:00
Zaiming (Stone) Shi
6b44d96c0d refactor: rename functions to please elvis 2022-04-29 10:52:08 +02:00
Zaiming (Stone) Shi
02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
EMQ-YangM
e4b62f3a5f feat: add rule-engine functions 2022-04-15 18:02:24 +08:00
Kian-Meng Ang
fc92e4c8bf docs: fix typos 2022-01-27 12:29:38 +01:00
Zaiming (Stone) Shi
63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
Zaiming Shi
6edd862dd5 refactor: no more EMQX_ENTERPRISE compile flag
The compile flag was introduced in EQM X 4.3 series
where CE and EE code was diverged large enough which made
non-practicle to determin edition at runtime.

such approach made testing quite challenging as we'll have to
build with different compile flags inorder to run per-edition
test cases

In this commit, we try to retrieve edition info from EMQX's
description text, (put to PT for fast access) at runtime
so we can test ALL editions from a super-set edition (EE).
2021-11-20 20:41:06 +01:00