yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,782 Commits 393 Branches 684 Tags 74 MiB
Commit Graph

141 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 969fa03ecc feat: implement kafka consumer 2023-03-16 13:43:01 -03:00
Andrew Mayorov 53bc27e0f4
refactor(bridge): avoid unnecessary `maps:to_list/1` when listing 2023-03-13 14:49:38 +03:00
firest b8b1a78025 feat(bridges): integrate the DynamoDB into data bridges 2023-03-08 11:13:38 +08:00
Kjell Winblad 67acdf0888 feat: add clickhouse database bridge 
This commit adds a Clickhouse bridge to EMQX 5. The bridge is similar to
the Clickhouse bridge in the 4.4, but adds the possibility to use
different formats (such as JSON) for values to be inserted.
 2023-03-02 12:22:11 +01:00
Zaiming (Stone) Shi 7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi c47be57c59 fix(bridge): ensure all bridge resources are stopped before app stop 2023-01-27 12:39:05 +01:00
Erik Timan 8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi 5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Thales Macedo Garitezi 47f796dd12 refactor: rename `emqx_resource_worker` -> `emqx_resource_buffer_worker` 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Zaiming (Stone) Shi d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
Thales Macedo Garitezi fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
firest c3d5c25c26 fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12 2023-01-17 17:42:10 +08:00
Shawn 13b2f45405 fix: function_clause when sending messages to bridges 2023-01-06 09:24:38 +08:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 42c58e2a91 Merge remote-tracking branch 'origin/release-50' into 1214-sync-master-upstreams 2022-12-14 15:29:13 +01:00
Thales Macedo Garitezi b9bc82f87a feat(gcp_pubsub): add `local_topic` config 
Given the implicit convention that an egress bridge containing the
`local_topic` config will forward messages without the need for a rule
action, this was added to avoid needing a rule action.
 2022-12-12 17:18:19 -03:00
Thales Macedo Garitezi c69022f3c4 test(ci): clean config after test 
Trying to fix this error for profile `emqx`:

```
=CRASH REPORT==== 30-Nov-2022::13:25:46.763989 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.9682.1>
    registered_name: []
    exception exit: {bad_return,
                        {{emqx_conf_app,start,[normal,[]]},
                         {emqx_conf_schema,
                             [#{kind => validation_error,path => "bridges",
                                reason => unknown_fields,
                                unknown => <<"influxdb_api_v1">>,
                                unmatched => <<"mqtt,webhook">>}]}}}
      in function  application_master:init/4 (application_master.erl, line 142)
    ancestors: [<0.9681.1>]
    message_queue_len: 1
    messages: [{'EXIT',<0.9683.1>,normal}]
    links: [<0.9681.1>,<0.44.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 29
    reductions: 195
  neighbours:

=INFO REPORT==== 30-Nov-2022::13:25:46.777895 ===
    application: emqx_conf
    exited: {bad_return,
                {{emqx_conf_app,start,[normal,[]]},
                 {emqx_conf_schema,
                     [#{kind => validation_error,path => "bridges",
                        reason => unknown_fields,
                        unknown => <<"influxdb_api_v1">>,
                        unmatched => <<"mqtt,webhook">>}]}}}
    type: temporary

%%% emqx_plugins_SUITE ==> init_per_suite: FAILED
%%% emqx_plugins_SUITE ==> {{failed_to_start_app,emqx_conf,
     {emqx_conf,
         {bad_return,
             {{emqx_conf_app,start,[normal,[]]},
              {emqx_conf_schema,
                  [#{kind => validation_error,path => "bridges",
                     reason => unknown_fields,
                     unknown => <<"influxdb_api_v1">>,
                     unmatched => <<"mqtt,webhook">>}]}}}}},
 [{emqx_common_test_helpers,start_app,4,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
       {line,227}]},
  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
  {emqx_plugins_SUITE,init_per_suite,1,
      [{file,
           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
       {line,34}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1380}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

%%% emqx_plugins_SUITE ==> t_bad_info_json: SKIPPED
%%% emqx_plugins_SUITE ==> {tc_auto_skip,
    {failed,
        {emqx_plugins_SUITE,init_per_suite,
            {'EXIT',
                {{failed_to_start_app,emqx_conf,
                     {emqx_conf,
                         {bad_return,
                             {{emqx_conf_app,start,[normal,[]]},
                              {emqx_conf_schema,
                                  [#{kind => validation_error,
                                     path => "bridges",
                                     reason => unknown_fields,
                                     unknown => <<"influxdb_api_v1">>,
                                     unmatched => <<"mqtt,webhook">>}]}}}}},
                 [{emqx_common_test_helpers,start_app,4,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                       {line,227}]},
                  {lists,foreach,2,[{file,"lists.erl"},{line,1342}]},
                  {emqx_plugins_SUITE,init_per_suite,1,
                      [{file,
                           "/__w/emqx/emqx/source/apps/emqx_plugins/test/emqx_plugins_SUITE.erl"},
                       {line,34}]},
                  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
                  {test_server,run_test_case_eval1,6,
                      [{file,"test_server.erl"},{line,1380}]},
                  {test_server,run_test_case_eval,9,
                      [{file,"test_server.erl"},{line,1224}]}]}}}}}
```
 2022-12-12 17:18:13 -03:00
William Yang 6b6bb09d4a fix: ingress only bridge causes egress bridge traffic stop 2022-12-09 15:36:20 +01:00
Ilya Averyanov 6692b0c895 feat(bridge): add Redis bridge 2022-12-06 23:15:42 +03:00
Zaiming (Stone) Shi 9a6901987f feat: add mqtt bridge config upgrade converter 2022-11-30 21:01:59 +01:00
Shawn d24441b204 fix(bridges): some issues found in code review 2022-09-23 18:32:39 +08:00
JimMoen 6788881642 fix: hide influxdb udp config 2022-09-20 14:46:59 +08:00
Shawn 60a90858f8 feat: check dependent actions before removing the bridges 2022-09-06 15:34:23 +08:00
JimMoen ca6533395c feat: influxdb bridge structure fits new style 2022-08-24 09:59:22 +08:00
Shawn aea8c77b49 refactor: new config structure for mqtt bridge 2022-08-22 18:24:59 +08:00
JimMoen 83746daad5 fix: update bridge config badmap error 2022-08-11 23:15:53 +08:00
JimMoen ed796acb95 fix: fetch resource options after reboot 2022-08-11 22:11:28 +08:00
Shawn 2872f0b668 fix(bridges): support create resources with options 2022-08-11 19:11:44 +08:00
Shawn 6685a3c5a8 fix: remove the 'headers' field from the rule events 2022-06-17 15:57:10 +08:00
Shawn 39b1b20506 feat: fix the hook priorities 2022-06-15 19:03:40 +08:00
Shawn dd099137f0 fix: catch errors when load bridges 2022-06-13 22:31:58 +08:00
Shawn 69fba6958b refactor: remove the emqx_bridge_monitor module 2022-06-01 15:31:48 +08:00
Shawn a157539710 refactor(rule): rename http_bridge -> webhook 2022-05-25 18:56:12 +08:00
Shawn d89d692d3e refactor: move some APIs to new module emqx_bridge_resource 2022-05-18 15:12:34 +08:00
Chris Hicks 841acb7828
Merge pull request #7783 from emqx/EMQX-4199-introduce-a-new-emqx-resource-manager-module 
feat: isolate resource manager processes
 2022-05-09 18:33:01 +02:00
Chris 0b3e30e813 feat: isolate resource manager processes 2022-05-09 13:24:34 +02:00
Shawn 1ce8a8c886 fix: testing connector removes ssl cert files that are in use 2022-05-05 14:55:40 +08:00
Shawn e18c2a6de2 fix: convert ssl certfiles for dry-run creating bridges 2022-04-28 16:17:55 +08:00
Zaiming (Stone) Shi 02c3f87b31 style: reformat all remaining apps 2022-04-27 15:51:18 +02:00
Thales Macedo Garitezi ada3b03f6e
fix(telemetry): do not crash if bridge app is not ready yet 2022-04-25 11:23:13 -03:00
EMQ-YangM f00854aa9d fix(emqx_bridge): add required fields 2022-04-22 14:22:22 +08:00
Shawn 5f74db6d88 fix: treat all returns except {error, Reason} as ok 2022-04-18 22:24:39 +08:00
EMQ-YangM 5aeb3c2dc2 feat: impl bridge reset_metrics api 2022-04-11 10:25:48 +08:00
Thales Macedo Garitezi 911e02f626
feat(bridge): export basic usage info for telemetry 2022-04-06 10:20:23 -03:00
Shawn 1d023b541f refactor(connnector): rename waiting_connect_complete -> wait_for_resource_ready 
Rename the option to wait_for_resource_ready and defaults to 5s.
 2022-03-10 10:46:57 +08:00
Shawn 36e068d00d feat(bridge): add APIs for restart/stop bridges on one node 2022-03-09 18:42:19 +08:00
EMQ-YangM 583624fb8d fix(emqx_authn): fix test suite 2022-03-09 13:53:05 +08:00
EMQ-YangM f29877bb6a fix(emqx_resource): remove create_opts async_create 2022-03-08 14:09:39 +08:00
Shawn 8a0565a53b chore(elvis): please the elvis 2022-03-08 11:11:29 +08:00
Shawn 2897af9650 fix: update testcases for emqx_connector 2022-03-07 18:39:23 +08:00
Shawn 9a9c92ae88 fix: update testcases for emqx_bridge 2022-03-07 16:48:36 +08:00
EMQ-YangM f70fc1a3b0 fix(emqx_bridge, emqx_retainer): hard code group name 2022-02-16 14:02:13 +08:00
EMQ-YangM d8819559f7 fix(emqx_resource): fix emqx_bridge create 2022-02-11 18:36:55 +08:00
Zaiming (Stone) Shi 63167cea70 chore: update copyright 2022-01-05 20:55:00 +01:00
Shawn e95445728c fix(test): wait until the bridge ready 2022-01-02 23:59:56 +08:00
Shawn 11736dc1d7 fix(bridge): check health immediately after updated 2022-01-02 22:45:32 +08:00
Shawn e1ab331a30 refactor(resource): support async create mode 2022-01-02 20:20:39 +08:00
Shawn 808646c2a1 fix(bridge): prohibit deleting connectors that are in use 2022-01-01 22:10:13 +08:00
Shawn 925d46fe86 fix(connector): add testcase for binding egress mqtt bridge to rules 2022-01-01 04:12:20 +08:00
Shawn 9a7452e1c5 fix(connector): add testcase for binding ingress mqtt bridge to rules 2022-01-01 03:07:31 +08:00
Shawn 658f819aab fix(bridges): keep multiple bridges from affecting each other on crash 2021-12-31 21:28:32 +08:00
Shawn 657ecef67b fix(resource): don't crash on resource stopped 2021-12-31 20:57:34 +08:00
Shawn e299d8d138 fix(rule): rules not triggered after the ingress mqtt bridge received some msg 2021-12-31 15:47:03 +08:00
Shawn d11cf6ad64 fix(bridges): store connector name and bridge name to config files 2021-12-30 23:48:41 +08:00
zhongwencool 121d906992 chore(log): add SLOG/3 to add meta info 2021-12-29 14:48:48 +08:00
Shawn a879ec0f3a feat(resource): add option 'force_create' to emqx_resource:create/4 2021-12-20 10:26:27 +08:00
Shawn 05e24b457a fix(bridge): update emqx_bridge.conf 2021-12-18 20:53:14 +08:00
Shawn 11e8e0db69 fix(bridge): stop http failed due to econnrefused 2021-12-18 15:33:25 +08:00
Shawn f08f37ec9c fix(bridge): create bridge failed but the config was saved 2021-12-18 11:19:22 +08:00
Shawn 0699682f38 refactor(bridges): rename some config entries for MQTT bridge 2021-12-17 21:22:04 +08:00
Shawn d76f82d3d2 fix(bridges): add new option 'enable' for bridge configs 2021-12-08 18:16:38 +08:00
Shawn 12fbcd8181 feat(connector): add num_of_bridges to the response of GET /connectors 2021-12-07 15:09:08 +08:00
Shawn 29ad6d215e feat(resource): add metrics to emqx_resource 2021-11-23 10:04:31 +08:00
Shawn 9c93ea0338 feat(connector): add API for /connectors_test 2021-11-22 17:35:33 +08:00
Shawn 4dac90f4a7 refactor(bridges): add POST /bridges for creating 2021-11-22 17:35:32 +08:00
Shawn 7d64013edd refactor(connector): add connector configs 2021-11-22 17:35:32 +08:00
Shawn e41736f5cb fix(bridge): update testcases for bridge APIs 2021-11-22 17:35:32 +08:00
Shawn 7996d8f709 refactor(bridge): rename the APIs for emqx_bridge 2021-11-22 17:35:32 +08:00
Shawn 1e6884ee7d fix(bridge): convert confs for http bridge when creating 2021-11-22 17:35:32 +08:00
Shawn f01f9632c1 fix(bridge): cannot start mqtt bridge from config 2021-11-22 17:35:32 +08:00
Shawn d046f9c6e7 fix(bridge): improve the schema of connector_http 2021-11-22 17:35:32 +08:00
Shawn dd9e2c4b24 refactor(bridge): the configs for http bridges 2021-11-22 17:35:32 +08:00
zhongwencool d784e63b9f
emqx_conf (#5939) 
* feat(emqx_conf): move conf manager for emqx_machine to emqx_conf

* chore(emqx_conf): change emqx:get_config/2 to emqx_conf:get/2

* fix: common test failed

* fix: badmatch by typo wrong key

* fix(emqx_conf): get the wrong core nodes

* fix(emqx_conf): get core node's tnx_id not latest tnx_id

* fix: add ro_transation when copy conf file

* fix: delete debug info

* fix: change ekka_rlog to mria_rlog

* fix: remove cluster_rpc from emqx_machine.

* fix: don't call ekka:start/0 explicitly

* fix: ekka should be start in emqx_machine
 2021-10-21 18:08:51 +08:00
Shawn a9185f964e
fix(rules): improve specs and logs (#5821) 
Co-authored-by: Zaiming Shi <zmstone@gmail.com>
 2021-09-27 21:10:48 +02:00
Shawn e2721c144c feat(bridge): support http bridge 2021-09-27 22:01:18 +08:00
Shawn b1999b2205 fix(bridge): update bridge with new conf failed 2021-09-15 17:46:42 +08:00
Shawn 7058b83760 refactor(bridge): rename message_in/out to ingress/egress_channels 2021-09-15 17:46:42 +08:00
Shawn c1ff8778e1 fix(bridges): add logs for creating/removing bridges 2021-09-15 17:46:42 +08:00
Shawn cb8dabe579 feat(bridges): add CRUD HTTP APIs for bridges 2021-09-15 17:46:42 +08:00
Shawn 39bb1b8d9d fix(bridges): start mqtt bridge failed 2021-09-10 11:43:03 +08:00
Shawn bfb2df37ce refactor(bridge): rename emqx_data_bridge to emqx_bridge 2021-09-10 11:43:03 +08:00