Commit Graph

260 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 1824e7efcc fix(rule_engine): count referenced bridges in `from` clauses as dependencies (rv5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-9325

Currently, ingress bridges referenced in the `FROM` clause of rules
are not being accounted as dependencies.

When we try to delete an ingress bridge that's referenced in a rule
like `select * from "$bridges/mqtt:ingress"`, that bridge does not
trigger an UI warning about dependent actions.
2023-03-28 10:31:10 -03:00
Zaiming (Stone) Shi f5e9d3eab2 test: use valid config for jq function timeout 2023-03-28 14:29:24 +02:00
Stefan Strigler 6ebd3dc747 feat(emqx_rule_engine): decompose error tuples
sqltester for instance returns sth like {"...sytnax error...", OrigSql}
2023-03-03 09:59:24 +01:00
Stefan Strigler 64b5e9585e feat(emqx_rule_engine): API format errors in a human readable way 2023-03-02 16:49:11 +01:00
Thales Macedo Garitezi eb7dca3691 chore(rule_engine): reset metrics when disabling a rule
https://emqx.atlassian.net/browse/EMQX-8502

When a bridge is disabled, its metrics are reset.  With this change,
we make rule actions behave like that: disabling a rule will reset its
metrics.
2023-01-04 09:30:47 -03:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
JimMoen 4906da9796
test(rule-funcs): test compression funcs 2022-12-19 15:43:50 +08:00
Zaiming (Stone) Shi bd65e8aad6 Merge remote-tracking branch 'origin/release-50' into 1206-chore-merge-ee50-to-release-50 2022-12-06 16:42:50 +01:00
Zaiming (Stone) Shi 1c2fc4b6c3 Merge remote-tracking branch 'origin/release-50' into 1206-chore-merge-ee50-to-release-50 2022-12-06 16:35:56 +01:00
Stefan Strigler 0b324da7cb refactor: move metrics out of /rules(/:id) to /rules/:id/metrics 2022-12-06 15:23:28 +01:00
Zaiming (Stone) Shi cca3421308 refactor: use static function references 2022-12-06 09:40:03 +01:00
Zaiming (Stone) Shi 7ee53e5319 Merge tag 'v5.0.11' into dev/ee5.0 2022-11-28 21:02:21 +01:00
JianBo He 8a0c468b01 test: refine tests for lots of List HTTP API 2022-11-24 20:14:33 +01:00
Zaiming (Stone) Shi f3df2c80d8 feat: add user_properties arg for republish action 2022-11-22 20:09:59 +01:00
kraftwerk28 00c57de4c3 feat: do not drop MQTTv5 properties in rule/bridge 2022-11-22 20:09:59 +01:00
Zaiming (Stone) Shi 09455edae8 Merge tag 'v5.0.10' into dev/ee5.0 2022-11-16 16:20:30 +01:00
JimMoen 4f00206693
test: different resons for disconnect event 2022-11-08 23:22:58 +08:00
Zaiming (Stone) Shi d704de4a84 fix(emqx_rule_events): drop tuple-value from message headers
the message headers are fed to a JSON ecnoder which
may crash if the header value is a tuple
2022-10-27 20:31:33 +02:00
Shawn 60a90858f8 feat: check dependent actions before removing the bridges 2022-09-06 15:34:23 +08:00
Zhongwen Deng 56417a3130 feat: list rules support for pagination and fuzzy filtering 2022-07-25 09:36:08 +08:00
Shawn 6685a3c5a8 fix: remove the 'headers' field from the rule events 2022-06-17 15:57:10 +08:00
Shawn 55fce33477 fix: add telemetry testcases in all/0 2022-05-25 21:55:28 +08:00
Shawn a157539710 refactor(rule): rename http_bridge -> webhook 2022-05-25 18:56:12 +08:00
Shawn 71de9616d3 refactor(rule): rename outputs -> actions 2022-05-25 17:02:37 +08:00
Xinyu Liu 866810cea6
Merge pull request #8006 from emqx/copy-of-kjell/jq/timeout
feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
2022-05-23 11:58:40 +08:00
EMQ-YangM 005fce63d1 feat: add rule engine function float2str/2 2022-05-19 16:16:05 +08:00
EMQ-YangM 66c62f432b fix: zero padding should be to the MSB side 2022-05-18 15:28:01 +08:00
Kjell Winblad 05032467bd feat(rule_engine): default timeout for jq/2 and jq/3 with timeout
This commit adds a default timeout of 10 seconds to the rule engine's
`jq/2` function, and adds a new function `jq/3` (where the last parameter is
a timeout value). The default timeout can be configured with the setting
"rule_engine.jq_function_default_timeout".

Having a timeout when executing jq code in the rule engine is important
as jq code can potentially run forever. Also, the Erlang jq library
limits the number of jq programs that can execute concurrently so a jq
program that loops forever could potentially also prevent a "non-buggy"
jq program from ever starting.
2022-05-17 19:36:32 +02:00
EMQ-YangM d8da8f1686 fix: hexstr_to_bin support half byte 2022-05-17 16:47:59 +08:00
DDDHuang 5210cd6e8d fix(test): rule engine api SUITE , bad rule name 2022-05-12 18:01:37 +08:00
EMQ-YangM 685b06eeed fix: avoid 'rule_test' api crash 2022-05-05 17:28:59 +08:00
Kjell Winblad 74c33cd4e5 feat(rule_engine): add jq function to the rule engine
This commit adds a function to the rule engine that alows users
to transform text or JSON objects using [jq filter programs][1].

[jq][1] is a command line tool that can be used to transform
and filter JSON text using jq's built-in language. The rule engine
function that is added with this commit uses the
[Erlang jq NIF library][2] that wraps the jq C library in an
Erlang NIF function.

[1]: https://stedolan.github.io/jq/
[2]: https://github.com/emqx/jq
2022-05-04 17:09:16 +02:00
Zaiming (Stone) Shi 02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
EMQ-YangM 1fac70d2bb fix: remove error field 2022-04-27 09:48:57 +08:00
EMQ-YangM e4b62f3a5f feat: add rule-engine functions 2022-04-15 18:02:24 +08:00
EMQ-YangM ee6ac90d47 fix: behavior of rule-engine when selection field does not exist 2022-04-13 18:19:42 +08:00
EMQ-YangM fa0c4d17ee feat: impl rule_engine reset_metrics api 2022-04-11 10:25:48 +08:00
Thales Macedo Garitezi 1aea6b3ea7
Merge pull request #7522 from thalesmg/telemetry-revamp-part5
feat: add basic rule engine and bridge usage info to telemetry
2022-04-07 10:19:55 -03:00
Thales Macedo Garitezi 8354095e4b
feat(rules): export basic usage info for telemetry 2022-04-06 10:20:23 -03:00
EMQ-YangM a31891e8e6 feat: enhanced rule engine error handling when json parsing error 2022-04-06 09:56:55 +08:00
Thales Macedo Garitezi ed7035ec41
style(bridge): please elvis checks 2022-04-05 15:59:50 -03:00
EMQ-YangM ccc134d061 fix(emqx_rule_api_schema): add rule events schema 2022-03-29 14:03:15 +08:00
EMQ-YangM 4791c64b73 feat: add rule event 'client.check_authz_complete' 2022-03-29 11:24:36 +08:00
EMQ-YangM 65164fb046 feat: add rule event 'client.connack' 2022-03-28 11:32:07 +08:00
Shawn b69dca4f08 feat(rule): add testcase for '/delivery_dropped' 2022-02-08 20:37:40 +08:00
Kian-Meng Ang fc92e4c8bf docs: fix typos 2022-01-27 12:29:38 +01:00
Zaiming (Stone) Shi 2105d1f22b test: call ct helpers to load config 2022-01-25 11:50:18 +01:00
Shawn 8cee9a9e47 fix(rule): compare to null variables should return false 2022-01-17 14:17:49 +08:00
Zaiming (Stone) Shi 63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
Shawn affe69afd6 fix(rules): update the test cases for rule APIs 2021-12-07 09:48:47 +08:00
Ilya Averyanov ea0fa24ce5 chore(ct): remove verbose output during tests 2021-11-25 18:30:50 +03:00
Shawn 4c149f92c1 feat(bridge): support metrics for bridges 2021-11-23 09:36:10 +08:00
JimMoen dcc5ae0642 test: API return code section 2021-11-15 16:30:44 +08:00
zhongwencool d784e63b9f
emqx_conf (#5939)
* feat(emqx_conf): move conf manager for emqx_machine to emqx_conf

* chore(emqx_conf): change emqx:get_config/2 to emqx_conf:get/2

* fix: common test failed

* fix: badmatch by typo wrong key

* fix(emqx_conf): get the wrong core nodes

* fix(emqx_conf): get core node's tnx_id not latest tnx_id

* fix: add ro_transation when copy conf file

* fix: delete debug info

* fix: change ekka_rlog to mria_rlog

* fix: remove cluster_rpc from emqx_machine.

* fix: don't call ekka:start/0 explicitly

* fix: ekka should be start in emqx_machine
2021-10-21 18:08:51 +08:00
Tobias Lindahl aa19c2a908 chore: Remove emqx_ct_helpers as dependency 2021-10-15 10:40:57 +02:00
Shawn 0e7a3f89a9 feat(rules): support output functions in <<"Mod:Func">> format 2021-10-14 15:09:37 +08:00
Shawn 9c7eef5295 feat(rules): update rule_engine configs from APIs 2021-10-14 15:09:37 +08:00
Shawn 1ffae5d1b0 refactor(rule_engine): merge code for emqx_rule_registry and emqx_rule_engine 2021-10-14 15:09:37 +08:00
Shawn b063b6f253 feat(rules): support configure rules in config file 2021-10-14 15:09:37 +08:00
lafirest 3524942766 fix: change all ~s in io format to ~ts
we should use unicode to replace latin1
2021-10-12 15:09:24 +08:00
Shawn 0d26e50e87 fix(rules): parse outputs failed 2021-09-26 23:00:08 +08:00
Shawn 420ccf0f51 refactor(rules): change republish as an output 2021-09-26 23:00:08 +08:00
Shawn f33e28af6d fix(rules): update test cases for emqx_rule_engine_SUITE 2021-09-26 23:00:08 +08:00
Shawn 7bc69b129a fix(rules): update test cases for rule metrics 2021-09-26 23:00:08 +08:00
Shawn af295a9b71 refactor(rules): remove resources and actions 2021-09-26 23:00:08 +08:00
Shawn b055464f6b refactor(rule): remove CLIs for rules
We will have new CLIs based on HTTP API in the future.
2021-09-26 23:00:08 +08:00
Shawn d46241fe2f feat(bridges): avoid clientid competition between bridges on different nodes 2021-09-10 14:21:17 +08:00
Shawn 1dae970cd3 refactor(rule): move emqx_rule_utils.erl -> emqx_plugin_libs_rule.erl 2021-09-10 11:43:03 +08:00
zhongwencool 73238ed81f feat: emqx_resource support cluster_call 2021-08-26 17:23:39 +08:00
zhongwencool e5129ead6d fix(CI): Error detected: 'Invalid reference to group api in emqx_rule_engine_SUITE:all/0' 2021-08-26 17:17:47 +08:00
DDDHuang c11a8c6db6 refactor: clients api; status api; adapter minirest v1
The serious influence:

authn:
    api
authz:
    api; api test suit
dashboard:
    all closed
lwm2m:
    api;
modules:
    api(api_topic_metrics, modules_api); test suit(emqx_modules_SUITE)
prometheus:
    api
retainer:
    api; api test suit
rule_engine:
    api: api test suit
telemetry:
    api
2021-07-09 18:09:07 +08:00
Turtle 54aeacee14 feat(rule-engine): update the configuration file to hocon 2021-06-28 14:51:15 +08:00
Zaiming Shi 4ee6c82714 test(emqx_rule_engine_SUITE): delegate app start to ct-helper (2.0.0) 2021-06-07 09:09:28 +02:00
z8674558 a9efdaeacd fix(emqx_rule_engine_SUITE): use emqx_ct_helpers:read_schema_configs 2021-06-01 17:06:09 +09:00
Zaiming Shi d904a1048b Merge remote-tracking branch 'origin/master' into merge-master-to-dev/v5.0 2021-04-30 15:09:31 +02:00
Zaiming Shi 6354e75626 chore: update copyrights 2021-04-23 20:43:00 +02:00
Shawn 19a9bab3a4 feat(rule_engine): add rule sql functions for datetime rfc3339 2021-04-22 10:32:44 +08:00
z8674558 971e6ca90e Merge branch 'master' of github.com:emqx/emqx into merge-master-to-v5 2021-04-07 20:29:08 +09:00
wwhai f7b34cb098 feat(rule-engine): list rule support order by timestrap 2021-03-12 17:35:13 +08:00
Zaiming Shi 23269c13bc
Merge pull request #4229 from emqx/master
Auto-pull-request-on-2021-02-23
2021-02-24 09:57:51 +01:00
Shawn 6d83bc3e9b
fix(rule): destroy the actions when disabling the rule (#4232)
Merge the following PR from enterprise version:
https://github.com/emqx/emqx-rule-engine/pull/231
2021-02-24 13:24:23 +08:00
Zaiming Shi 13e8093583 Merge remote-tracking branch 'origin/dev/v4.3.0' into merge-dev-v4.3.0 2021-02-22 21:08:08 +01:00
Zaiming Shi 505ae24ded
Merge branch 'dev/v5.0' into resolve-dev/v4.3.0-to-dev/v5.0-conflict 2021-02-22 15:20:34 +01:00
Shawn 196fef0255 chore(rule): migrate rule-engine from enterprise version 2021-02-22 13:53:51 +08:00
Shawn 7778cd8623
Add hooks only when creating the rules (#4160)
* refactor(rules): add hook only when creating rules

* fix(rule): update hooks after application restarted

* fix(rule): remove the extra guard
2021-02-22 11:16:47 +08:00
Shawn 1e047e84c2
feat(resource): keep restart disconnected resources after emqx bootup (#4125)
* feat(resource): keep restart disconnected resources after emqx bootup

* feat(resource): improve the restart monitor

* fix(test): improve emqx_rule_monitor_SUITE

* fix(resource): refresh resource should be only applied on local node

* fix(test): improve the test case for restart_resource

* fix(resource): rename some functions
2021-02-20 17:10:50 +08:00
Shawn c2e1bc039b
feat(rule): add more sql functions #4143 (#4144) 2021-02-20 17:10:10 +08:00
Zaiming Shi c2cd2fd231 Merge remote-tracking branch 'origin/dev/v4.3.0' into dev/v5.0 2021-02-19 21:13:33 +01:00
Zaiming Shi 4a877a3a69 chore(apps): relocate some apps
emqx_telemetry is for opensource onlyh
emqx_rule_engiune is common
2021-02-19 11:24:41 +01:00
Zaiming Shi 8f477e8411 refactor(proj): Move opensource apps to lib-opensource
One step closer for better code abstraction for opensorce vs enterprise
So far this relocation is to make opensource to enterprise merges
conflict free.
2021-02-11 16:45:05 +01:00
Shawn 70a490114d fix(rulesql): proc cassandra cql with apostrophes failed 2021-02-05 15:30:16 +08:00
Shawn a2fa84255f
fix(rulesql): proc_sql_param_str incorrect with apostrophes #4135 (#4137) 2021-02-05 10:41:54 +08:00
wwhai a3206f5e0b fix(rule-engine): reformat code style 2021-01-29 19:32:05 +08:00
bignullnull 67bfb4f10a
fix(rule_engine): resource update problem and add cli (#3991) 2021-01-15 09:12:24 +08:00
Shawn d86b5ef210 fix(rule_engine): compile warnings on the macro RAISE 2020-12-31 17:03:39 +08:00
bignullnull efb32e9903
test(rule_engine): add update resource unit test case (#3964)
* test(rule_engine): add update resource unit test case

* feat(rule_engine): add update resource support

Co-authored-by: wwhai <cnwwhai@gmail.com>
2020-12-28 14:59:28 +08:00
JianBo He 9e22f492c7 refactor(rule-engine): add proc_sql_param_str/2 to handle sql params 2020-12-28 10:51:13 +08:00
Shawn 6b5878cdd0 fix(test): solve verification issue when verify the resource numbers 2020-12-23 16:39:40 +08:00
zhanghongtong 3ef08f9ce3 test(meck): add meck unload after meck load 2020-12-22 13:39:49 +08:00
Shawn e15c0c325e fix(rule_engine): nested_get value from binary was supported in 4.1 2020-12-22 11:18:16 +08:00
Shawn 86d20b060c fix(metrics): counters not created before incr 'rule.matched' 2020-12-22 11:18:16 +08:00
Shawn c3159420bb feat(funcs): add bitsize/1 and subbits/2,3,6 functions 2020-12-22 11:18:16 +08:00
Shawn e72cbd22c4 fix(rule_validator): fill default values when validating the params 2020-12-18 11:08:17 +08:00
Shawn 0a44270932
* fix(dialyzer): remove the dialyzer errors 2020-12-17 10:04:49 +08:00
Shawn 10dbb5bc96
Fix testcases warns (#3906) 2020-12-11 18:54:48 +08:00
Zaiming Shi 3a91ff8a7b chore(proj): resync apps 2020-12-11 08:19:26 +01:00
Shawn 15d048d8ec feat(funcs): add bitsize/1 and subbits/2,3,6 functions
Given following Rule SQL:

```
select
  subbits(payload, 1, 8) as firstByte,
  case
    when firstByte = 123 then 't/{'
    when firstByte = 125 then 't/}'
    else 't/' + str(firstByte)
  end as tp
from "t/#"
```

Input Payload: `{x}`
Output: `{"tp": "t/{", "firstByte": 123}`

Input Payload: `}x`
Output: `{"tp": "t/}", "firstByte": 125}`

Input Payload: `a`
Output: `{"tp": "t/97", "firstByte": 97}`
2020-12-10 11:26:07 +08:00
Shawn 573a4b2df8 chore(rule_engine): sync the code from rule-engine/dev/v4.3.0 2020-12-09 17:58:26 +08:00
Zaiming Shi 2fda0a1565 chore(apps): Sync again all apps 2020-12-07 21:30:21 +01:00
Zaiming Shi 686c006d6e refactor(proj): Add apps 2020-12-04 22:11:53 +01:00