Commit Graph

15826 Commits

Author SHA1 Message Date
Thales Macedo Garitezi
0790c88aaf
refactor: use default's type as first union member
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-06-02 09:08:11 -03:00
Thales Macedo Garitezi
3e4790edd4 test(pulsar_producer): fix flaky test 2023-06-01 13:01:58 -03:00
Thales Macedo Garitezi
99796224d8 refactor(resource): rename request_timeout -> request_ttl
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
Thales Macedo Garitezi
f42ccb6262 feat(resource): increase default request timeout to 45 s
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 11:20:06 -03:00
Thales Macedo Garitezi
10425eb925 feat(resource): deprecate auto_restart_interval in favor of health_check_interval
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Andrew Mayorov
b5f24c4f88
Merge pull request #10900 from keynslug/fix/EEC-856/parsed-bridge-id
fix(ruleeng): parse bridge id on create / update
2023-06-01 10:21:26 +03:00
JianBo He
bcc5e30f22
Merge pull request #10895 from lafirest/fix/refactor_on_stop
feat: refactored some bridges to avoid leaking resources during crashes at creation
2023-06-01 15:17:37 +08:00
firest
cb31032e6d chore: make dialyzer && test happy 2023-06-01 11:15:51 +08:00
JianBo He
a402acdc43
Merge pull request #10897 from Kinplemelon/kinple/upgrade-dashboard
chore: upgrade dashboard to e1.0.8-beta.1 for ee
2023-06-01 09:46:01 +08:00
Ivan Dyachkov
239f71cecf
Merge pull request #10899 from id/0530-update-perf-test
0530 update perf test
2023-05-31 23:17:10 +02:00
Thales Macedo Garitezi
b7c72e6cfb
Merge pull request #10890 from thalesmg/test-flakiness-20230530-e
more attempts to fix more test flakiness
2023-05-31 16:07:18 -03:00
Andrew Mayorov
aa98740280
Merge pull request #10903 from fix/ci-flaky-dashboard
test(dashboard): fix inter-suite test flakiness
2023-05-31 20:28:39 +03:00
Andrew Mayorov
f90ce9bbfe
test(dashboard): fix inter-suite test flakiness 2023-05-31 19:09:37 +03:00
Thales Macedo Garitezi
189e2c87be test(channel): fix flaky test
```
%%% emqx_channel_SUITE ==> t_handle_kicked_publish_will_msg: FAILED
%%% emqx_channel_SUITE ==> will_message_not_published
```

The problem was that the whole generated message was compared for
equality when receiving the will message, but the timestamp may be
different at publishing time...
2023-05-31 12:42:24 -03:00
JianBo He
a3a6480f00
Merge pull request #10880 from HJianBo/batch-kickout-clients
feat: support kickout clients in batch
2023-05-31 22:36:35 +08:00
JianBo He
8ce0132569
Merge pull request #10871 from HJianBo/fix-coap-bugs
fix(coap): to better handle coap requests in connection mode
2023-05-31 22:36:15 +08:00
JianBo He
c67135dd4b
Merge pull request #10598 from HJianBo/refactor_exproto_stream
refactor(exproto): support unary handler
2023-05-31 22:36:01 +08:00
Kjell Winblad
036f180c27
Merge pull request #10878 from kjellwinblad/kjell/bridge/rabbitmq_password_log_leak/EMQX-9976
fix: password log leak in RabbitMQ bridge
2023-05-31 16:17:19 +02:00
Andrew Mayorov
29fe201676
fix(ruleeng): parse bridge id on create / update
Instead of parsing it every time in the `send_message/2` hot path.
2023-05-31 17:08:42 +03:00
Thales Macedo Garitezi
77c9eda036 test: rm stray clear_screen calls 2023-05-31 11:07:44 -03:00
Thales Macedo Garitezi
dc4ae1a424 ci: make emqx standalone test output more readable 2023-05-31 10:56:06 -03:00
Ivan Dyachkov
ac94a00c5b ci: adjust perf test parameters 2023-05-31 15:46:48 +02:00
Thales Macedo Garitezi
870cb00ac4
Merge pull request #10842 from thalesmg/dev-elixir-script-v50
ci: add elixir support to `dev` / `make quickrun` scripts
2023-05-31 10:27:26 -03:00
Thales Macedo Garitezi
83a7c800cc test(kafka_consumer): reduce flakiness
The crash reason can also be `{error, not_found}` depending on
timing...

```
=ERROR REPORT==== 31-May-2023::12:17:31.461490 ===
    exception: error
    key_path: [bridges,kafka_consumer,
               't_resource_manager_crash_before_subscriber_started-576460752303419630']
    module: emqx_conf_schema
    msg: change_config_crashed
    reason: {badmatch,{error,not_found}}
    stacktrace: [{emqx_resource_manager,create_and_return_data,5,
                     [{file,
                          "/emqx/apps/emqx_resource/src/emqx_resource_manager.erl"},
                      {line,119}]},
                 {emqx_bridge_resource,create,4,
                     [{file,
                          "/emqx/apps/emqx_bridge/src/emqx_bridge_resource.erl"},
                      {line,171}]},
                 {emqx_bridge,'-perform_bridge_changes/2-fun-1-',5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,367}]},
                 {maps,fold_1,3,[{file,"maps.erl"},{line,411}]},
                 {emqx_bridge,perform_bridge_changes,2,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,354}]},
                 {emqx_bridge,post_config_update,5,
                     [{file,"/emqx/apps/emqx_bridge/src/emqx_bridge.erl"},
                      {line,229}]},
                 {emqx_config_handler,call_post_config_update,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,398}]},
                 {emqx_config_handler,do_post_config_update,8,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,336}]},
                 {emqx_config_handler,check_and_save_configs,7,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,281}]},
                 {emqx_config_handler,handle_update_request,4,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,195}]},
                 {emqx_config_handler,handle_call,3,
                     [{file,"/emqx/apps/emqx/src/emqx_config_handler.erl"},
                      {line,124}]},
                 {gen_server,try_handle_call,4,
                     [{file,"gen_server.erl"},{line,1149}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1178}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,240}]}]
    update_req: {{update,#{<<"authentication">> => <<"none">>,
                           <<"bootstrap_hosts">> =>
                               <<"toxiproxy.emqx.net:9292">>,
                           <<"connect_timeout">> => <<"5s">>,
                           <<"enable">> => true,
                           <<"kafka">> =>
                               #{<<"max_batch_bytes">> => <<"896KB">>,
                                 <<"max_rejoin_attempts">> => 5,
                                 <<"offset_commit_interval_seconds">> => 3,
                                 <<"offset_reset_policy">> => <<"latest">>},
                           <<"key_encoding_mode">> => <<"none">>,
                           <<"metadata_request_timeout">> => <<"5s">>,
                           <<"min_metadata_refresh_interval">> => <<"3s">>,
                           <<"ssl">> =>
                               #{<<"enable">> => false,
                                 <<"server_name_indication">> => <<"auto">>,
                                 <<"verify">> => <<"verify_none">>},
                           <<"topic_mapping">> =>
                               [#{<<"kafka_topic">> =>
                                      <<"t_resource_manager_crash_before_subscriber_started-576460752303419662">>,
                                  <<"mqtt_topic">> =>
                                      <<"mqtt/topic/-576460752303422911">>,
                                  <<"payload_template">> => <<"${.}">>,
                                  <<"qos">> => 0}],
                           <<"value_encoding_mode">> => <<"none">>}},
                 #{override_to => cluster}}
```
2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
e7b35011e6 test(oracle): attempt to fix flaky test
https://github.com/emqx/emqx/actions/runs/5126014459/jobs/9220155908#step:7:754
```
=CRITICAL REPORT==== 30-May-2023::20:50:26.267103 ===
Run stage failed: error:{assertMatch,
                         [{module,emqx_bridge_oracle_SUITE},
                          {line,394},
                          {expression,
                           "emqx_resource : simple_sync_query ( ResourceId , { query , \"SELECT COUNT(*) FROM mqtt_test\" } )"},
                          {pattern,
                           "{ ok , [ { result_set , _ , _ , [ [ { 3 } ] ] } ] }"},
                          {value,
                           {ok,[{result_set,[<<"COUNT(*)">>],[],[[{0}]]}]}}]}
Stacktrace: [{emqx_bridge_oracle_SUITE,'-t_batch_sync_query/1-fun-5-',1,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,394}]},
             {emqx_bridge_oracle_SUITE,'-t_batch_sync_query/1-fun-6-',5,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,390}]},
             {emqx_bridge_oracle_SUITE,t_batch_sync_query,1,
                 [{file,
                      "/emqx/apps/emqx_bridge_oracle/test/emqx_bridge_oracle_SUITE.erl"},
                  {line,364}]}]
```
2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
0d539e91d1 test(pulsar_producer): attempt to stabilize flaky test
https://github.com/emqx/emqx/actions/runs/5125166433/jobs/9218613872?pr=10886#step:7:679
```
  =CRITICAL REPORT==== 30-May-2023::19:38:58.003170 ===
  Run stage failed: error:{badmatch,
                              {timeout,
                                  [#{msg => pulsar_producer_bridge_started,
                                     '~meta' =>
                                         #{gl => <97891.472.0>,
                                           location =>
                                               #Fun<emqx_bridge_pulsar_impl_producer.11.109752493>,
                                           node => 'autocluster_node1@127.0.0.1',
                                           pid => <97891.787.0>,
                                           time => -576460611692219}}]}}
  Stacktrace: [{emqx_bridge_pulsar_impl_producer_SUITE,'-t_cluster/1-fun-10-',
                   6,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1073}]},
               {emqx_bridge_pulsar_impl_producer_SUITE,t_cluster,1,
                   [{file,
                        "/emqx/apps/emqx_bridge_pulsar/test/emqx_bridge_pulsar_impl_producer_SUITE.erl"},
                    {line,1064}]}]
```
2023-05-31 10:19:55 -03:00
Thales Macedo Garitezi
1e25ebb64c test(gcp_pubsub): attempt to fix flakiness
https://github.com/emqx/emqx/actions/runs/5125118728/jobs/9218520994?pr=10887#step:8:309
```
  =CRITICAL REPORT==== 30-May-2023::19:19:34.887082 ===
  "check stage" failed: error
  {assertMatch,[{module,emqx_bridge_gcp_pubsub_SUITE},
                {line,1066},
                {expression,"? of_kind ( gcp_pubsub_request_failed , Trace )"},
                {pattern,"[ # { reason := Error , connector := ResourceId } | _ ]"},
                {value,[#{connector =>
                              <<"bridge:gcp_pubsub:emqx_bridge_gcp_pubsub_SUITE0005FCEE15534E9CD4CD02004CF10000">>,
                          msg => gcp_pubsub_request_failed,query_mode => async,
                          reason => {closed,"The connection was lost."},
                          recoverable_error => true,
                          '~meta' =>
                              #{gl => <0.17903.2>,
                                location =>
                                    #Fun<emqx_bridge_gcp_pubsub_connector.19.19548918>,
                                node => 'test@127.0.0.1',pid => <0.19724.2>,
                                time => -576460610660164}}]}]}
  Stacktrace: [{emqx_bridge_gcp_pubsub_SUITE,
                   '-do_econnrefused_or_timeout_test/2-fun-2-',3,
                   [{file,
                        "/__w/emqx/emqx/source/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_SUITE.erl"},
                    {line,1066}]},
               {emqx_bridge_gcp_pubsub_SUITE,do_econnrefused_or_timeout_test,2,
                   [{file,
                        "/__w/emqx/emqx/source/apps/emqx_bridge_gcp_pubsub/test/emqx_bridge_gcp_pubsub_SUITE.erl"},
                    {line,1022}]}]
```
2023-05-31 10:19:55 -03:00
Ivan Dyachkov
2078c0a5c7 ci: add 1m conns in perf test 2023-05-31 13:28:55 +02:00
lafirest
502b24bbe2
Merge pull request #10877 from lafirest/fix/rocketmq_sensitive_data
fix(rocketmq): bump RocketMQ driver to enhance security for sensitive data
2023-05-31 18:54:44 +08:00
Kinplemelon
d39c9fed46 chore: upgrade dashboard to e1.0.8-beta.1 for ee 2023-05-31 18:30:35 +08:00
zhongwencool
26c2ab0bef
Merge pull request #10894 from zhongwencool/crl-cache-conf-hot-update
feat: update crl_cache conf at runtime
2023-05-31 17:59:09 +08:00
firest
ce32526c96 chore: bump version && update changes 2023-05-31 17:31:56 +08:00
William Yang
2250ed7052
Merge pull request #10823 from qzhuyan/dev/william/meck-olp
test: use meck in olp tests
2023-05-31 11:01:56 +02:00
firest
6081ce8b00 feat: refactored some bridges to avoid leaking resources during crashes at creation 2023-05-31 16:48:13 +08:00
Ivan Dyachkov
78ed78240f ci: update perf test 2023-05-31 10:41:33 +02:00
Zhongwen Deng
38e57f511c feat: update crl_cache conf at runtime 2023-05-31 16:18:03 +08:00
JianBo He
d0a9dca10b chore: update changes 2023-05-31 13:44:51 +08:00
JianBo He
3f83e52efa chore: update grpc-erl to 0.6.8 2023-05-31 13:44:51 +08:00
JianBo He
837dfeb46f feat(exproto): add raw_publish function 2023-05-31 13:44:51 +08:00
JianBo He
5256d8092c fix: keep the real closed_reason 2023-05-31 13:44:51 +08:00
JianBo He
2e4ec4888d fix(exproto): ensure the on_socket_closed event delivering correctly 2023-05-31 13:44:51 +08:00
JianBo He
9dff026f50 test: fix failed test cases 2023-05-31 13:44:51 +08:00
JianBo He
b6bc3cd921 refactor(exproto): support unary handler 2023-05-31 13:44:51 +08:00
JianBo He
afcd9fcb5e chore: update bpapi.versions 2023-05-31 11:29:55 +08:00
zhongwencool
e6049df724
Merge pull request #10849 from zhongwencool/licence-conf-update
feat: support emqx_conf:update([license],Conf)
2023-05-31 11:08:47 +08:00
lafirest
5968c71d0c
Merge pull request #10782 from lafirest/feat/retainer_flow_control
feat(retainer): add a field `deliver_rate` to limit the maximum delivery rate
2023-05-31 11:01:10 +08:00
zhongwencool
6202cefd5a
Merge pull request #10870 from zhongwencool/stop-dashboard-listener-waiting
fix: flaky test when stopping dashboard listener
2023-05-31 10:54:12 +08:00
zhongwencool
ec580f0454
Merge pull request #10866 from zhongwencool/add-delayed-test
test: add test for delayed update
2023-05-31 10:53:55 +08:00
firest
a725ff0440 chore: bump version && update changes 2023-05-31 10:18:00 +08:00
lafirest
592a87d99f
Merge pull request #10800 from lafirest/pref/default_behaviour
fix(limiter): improve the default configuration
2023-05-31 10:10:17 +08:00