Commit Graph

24 Commits

Author SHA1 Message Date
firest c28346d67b chore: bump ecql for obfuscate sensitive data to avoid leakage 2023-06-15 10:35:18 +08:00
Andrew Mayorov d6c1ee183f
refactor(pluglib): move `emqx_placeholder` to utils app
Also make user that existing code calls it directly.
2023-06-09 14:44:36 +03:00
JianBo He dbc0cdce67 chore: dont allocate resource for simple connectiors 2023-06-05 10:59:00 +08:00
JianBo He 3739230435 feat: refactor connectors on_stop function to avoid resources leaking
Supplement to https://github.com/emqx/emqx/pull/10895
2023-06-05 10:19:46 +08:00
Thales Macedo Garitezi 33aa879ad4
Merge pull request #10910 from thalesmg/unify-restart-interval-v50
feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
2023-06-02 16:20:36 -03:00
Thales Macedo Garitezi 0072f4a598 fix(ecpool,bridge): treat `{error, ecpool_empty}` as a retriable error
Example from flaky CI test:

```
/emqx/apps/emqx_resource/src/emqx_resource_buffer_worker.erl:716 2023-06-01T19:30:37.119862+00:00 [buffer_worker_flush_ack] #{batch_or_query => [{query,undefined,{send_message,#{payload => <<"-576460752303420927">>,timestamp => 1668602148000,topic => <<"t_write_failure">>}},false,-576460614954536809}],queue_count => 0,result => {async_return,{error,{unrecoverable_error,ecpool_empty}}},'~meta' => #{gl => <0.3883.5>,node => 'test@127.0.0.1',pid => <0.4724.5>}}.
```
2023-06-01 17:13:06 -03:00
Thales Macedo Garitezi 99796224d8 refactor(resource): rename `request_timeout` -> `request_ttl`
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
Thales Macedo Garitezi 8c565abc84 test(cassandra): fix flaky test 2023-05-30 15:42:53 -03:00
Thales Macedo Garitezi bae00c909f test(cassandra): attempt to fix flaky test
```
=ERROR REPORT==== 29-May-2023::20:35:20.457796 ===
    id: <<"bridge:cassandra:emqx_bridge_cassandra_SUITE">>
    msg: unrecoverable_error
    reason: {8704,<<"Expected 8 or 0 byte long for date (4)">>}
=CRITICAL REPORT==== 29-May-2023::20:35:20.458850 ===
"check stage" failed: error
{assertMatch,[{module,emqx_bridge_cassandra_SUITE},
              {line,619},
              {expression,"Trace"},
              {pattern,"[ # { result := { ok , _Pid } } ]"},
              {value,[]}]}
Stacktrace: [{emqx_bridge_cassandra_SUITE,'-t_missing_data/1-fun-3-',1,
                 [{file,
                      "/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                  {line,619}]},
             {emqx_bridge_cassandra_SUITE,t_missing_data,1,
                 [{file,
                      "/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                  {line,607}]}]

=CRITICAL REPORT==== 29-May-2023::20:35:20.471872 ===
Check stage failed. Trace dump: "/emqx/_build/test/logs/ct_run.test@127.0.0.1.2023-05-29_20.34.32/snabbkaffe/1685392520460040358.log"

%%% emqx_bridge_cassandra_SUITE ==> tcp.sync.with_batch.t_missing_data: FAILED
%%% emqx_bridge_cassandra_SUITE ==> {{panic,#{msg => "Unexpected result",result => {error,check_stage_failed}}},
 [{emqx_bridge_cassandra_SUITE,t_missing_data,1,
                               [{file,"/emqx/apps/emqx_bridge_cassandra/test/emqx_bridge_cassandra_SUITE.erl"},
                                {line,624}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1291}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1223}]}]}
```
2023-05-29 18:19:26 -03:00
Thales Macedo Garitezi 658160f09a test(cassandra): attempt to reduce flakiness 2023-05-24 16:57:51 -03:00
JianBo He 383fec9dfc docs: fix invalid links 2023-05-15 11:18:09 +08:00
Zhongwen Deng 4f396a36a9 Merge remote-tracking branch 'upstream/master' into release-50 2023-05-08 14:58:03 +08:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
zhongwencool c78004001c
Merge pull request #10481 from HJianBo/hidden-bad-link
chore: hide bad links in README files
2023-04-26 15:22:30 +08:00
Thales Macedo Garitezi 377b143325 refactor: split `parse_server` into smaller functions, improve return type to use map 2023-04-24 14:17:29 -03:00
JianBo He 6e1d6f1991 chore: hide bad links in README files 2023-04-23 11:27:59 +08:00
JianBo He b270623c46 chore: rename cassandra_impl to cassandra_connector 2023-04-21 16:49:25 +08:00
JianBo He 5cc28a7b45 chore: fix mix.exs checking 2023-04-21 14:46:09 +08:00
JianBo He bdce32e713 refactor(cassandra): move cassandra bridge into its own app 2023-04-21 14:45:38 +08:00
JianBo He c53ccfea61 chore: fix typos 2023-04-18 13:48:55 +08:00
JianBo He 8cb9389d21 chore: apply suggestions from code review
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-04-18 09:36:42 +08:00
JianBo He ec8d8b805f chore: add ee bridge apps 2023-04-18 09:36:41 +08:00
JianBo He 7eacbffae9 chore: create dirs and README template for all ee bridges 2023-04-18 09:36:41 +08:00