yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,227 Commits 393 Branches 669 Tags 74 MiB
244188982b
Commit Graph

7330 Commits

Author SHA1 Message Date
Ilya Averyanov
cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
firest
7a96a9772d fix(limiter): update change && fix deprecated version 2023-05-05 16:26:06 +08:00
William Yang
e4f501417b
Merge pull request #10525 from qzhuyan/perf/william/avoid-new-map-when-get-mqtt-caps 
perf(config): avoid build new map in emqx_mgmt_caps:get_caps
 2023-05-05 10:22:08 +02:00
firest
4f47e65b7b perf(limiter): simplify the memory represent of limiter configuration 2023-05-05 16:21:21 +08:00
Zaiming (Stone) Shi
14e055f18b
Merge pull request #10578 from zmstone/0502-refactor-delete-stale-config-check-opt 
0502 refactor delete stale config check opt
 2023-05-04 16:32:49 +02:00
firest
277deee616 fix: add new sensitive keywords to redact checklist 2023-05-04 16:32:32 +08:00
zhongwencool
2bc1ddbd19
Merge pull request #10588 from zhongwencool/fix-array-default 
chore: make sure topic_metrics/rewrite's default is []
 2023-05-04 16:03:21 +08:00
Zaiming (Stone) Shi
06e25f5c72 test: fix delayed-pubish test case flakyness 2023-05-04 09:48:28 +02:00
Zaiming (Stone) Shi
856129984b refactor: remove raw_with_default config load option 
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
 2023-05-04 09:48:28 +02:00
Zhongwen Deng
06960853e0 feat: increase the time precision of trace logs to microseconds 2023-05-04 14:41:02 +08:00
Zhongwen Deng
b09ee540d0 chore: make sure topic_metrics/rewrite's default is [] 2023-05-04 14:29:54 +08:00
Paulo Zulato
f965fa7b9a feat: add shutdown count 
Fixes https://emqx.atlassian.net/browse/EMQX-9008
Fixes https://emqx.atlassian.net/browse/EMQX-9219
 2023-05-03 20:00:38 -03:00
William Yang
c38cec77a7
Merge pull request #10417 from qzhuyan/perf/william/no-make-ref-in-get-config 
perf(config): eliminate make_ref() calls in config get calls
 2023-05-03 16:56:44 +02:00
William Yang
2ded8e3836
Merge pull request #10577 from qzhuyan/dev/william/importance-mark-quic-listener 
chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN
 2023-05-03 13:23:32 +02:00
Andrew Mayorov
754045173e
chore: bump snabbkaffe to 1.0.8 2023-05-03 11:16:51 +03:00
Zaiming (Stone) Shi
66112fceed
Merge pull request #10572 from zmstone/0502-merge-release-50-back-to-master 
0502 merge release 50 back to master
 2023-05-03 09:01:35 +02:00
Ilya Averyanov
0272cf4473
Merge pull request #10571 from savonarola/0502-do-not-pullute-logs-on-metrics-stop 
Do not complain on nonexisting ets while `emqx_topic_metrics` stops
 2023-05-03 09:44:25 +03:00
William Yang
b418cc766d test: update emqx_gateway_conf_SUITE 
for error return messages
 2023-05-02 22:59:24 +02:00
William Yang
a4a5599636 chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN 2023-05-02 22:20:39 +02:00
Thales Macedo Garitezi
8aa7c014e7 perf(buffer_worker): avoid calling ets:info/2 
(Almost?) fixes https://emqx.atlassian.net/browse/EMQX-9637

During the course of performance tests comparing the performance of
e5.0.3 and e4.4.16 regarding the webhook bridge in sync mode, we
observed that the throughput in e5.0.3 (sync) was much lower than in
e4.4.16: ~ 9 k msgs / s vs. ~ 50 k msgs / s, respectively.

Analyzing `observer_cli` output, we noticed that a lot of the time
both buffer workers and ehttpc processes was spent in `ets:info/2`.
That function was called to check the size of the inflight table when
updating metrics and checking if the inflight table was full.  Other
uses of `ets:info/2` were contained inside the arguments to some
`?tp/2` macro usages (https://github.com/kafka4beam/snabbkaffe/pull/60).

By using a specific record to track the size of the table, we managed
to improve the bridge performance to ~ 45 k msgs / s in sync mode.
 2023-05-02 17:05:32 -03:00
William Yang
c7af43bd72 chore: bump app vsn emqx 5.0.25 2023-05-02 21:51:43 +02:00
William Yang
e5884e0e87 chore: remove unused mqtt cap 'subscription_identifiers' 2023-05-02 21:50:36 +02:00
William Yang
be6cd5705f fix(mqtt-cap): use global config as defaults 
Don't use default values in module, use global mqtt config as default
 2023-05-02 21:50:36 +02:00
William Yang
df0911be6e Revert "perf: keep defaults of zone.mqtt" 
This reverts commit 6184bc02ee30aa2d022793762e626893a8a32599.
 2023-05-02 21:50:36 +02:00
William Yang
77405d9cd7 perf: keep defaults of zone.mqtt 2023-05-02 21:50:36 +02:00
William Yang
265c9ea668 chore: bump emqx app vsn to 5.0.25 2023-05-02 21:50:36 +02:00
William Yang
3d7201502b perf(config): avoid build new map in emqx_mqtt_caps:get_caps 2023-05-02 21:50:36 +02:00
William Yang
77f67a9d07 perf(config): get_raw with 'binary' root key 2023-05-02 21:49:55 +02:00
William Yang
ce6343a4f3 perf(config): ensure root keys of 'conf' config is atom 2023-05-02 21:49:55 +02:00
Zaiming (Stone) Shi
a4bc15cfa1 Merge remote-tracking branch 'origin/master' into 0502-merge-release-50-back-to-master 2023-05-02 18:59:31 +02:00
Thales Macedo Garitezi
f6da18dc1b fix(webhook): consider {shutdown, closed} return as recoverable 2023-05-02 13:12:03 -03:00
Zaiming (Stone) Shi
d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Andrew Mayorov
90cf1ade74
chore: bump application versions 
* emqx_connector 0.1.22
* emqx_bridge_gcp_pubsub 0.1.1
* emqx_bridge_kafka 0.1.2
* emqx_bridge_pulsar 0.1.1
* emqx_ee_connector 0.1.12
 2023-05-02 17:29:22 +03:00
Andrew Mayorov
670709f746
feat(resource): ensure uniqueness through gproc 
Also use it instead of a custom ETS table for simplicity and
better consistency. This has drawbacks though: expect slightly
increased load on gproc gen_server due to how `gproc:set_value/2`
works.
 2023-05-02 17:29:22 +03:00
Andrew Mayorov
4575167607
feat(resource): drop manager_id() type 2023-05-02 17:29:20 +03:00
Andrew Mayorov
aaef95b1da
feat(resman): stop adding uniqueness to manager ids 
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
 2023-05-02 17:28:26 +03:00
Thales Macedo Garitezi
2c4fd98ce5
Merge pull request #10559 from thalesmg/pulsar-more-tests-v50 
test(pulsar): add more test cases for Pulsar Producer bridge
 2023-05-02 10:16:17 -03:00
Thales Macedo Garitezi
654d274039
Merge pull request #10562 from thalesmg/fix-webhook-async-shutdown-normal-reply-r50 
fix(webhook): treat `{shutdown, normal}` and `{closed, _}` async replies as retriable
 2023-05-02 10:08:18 -03:00
Zaiming (Stone) Shi
0fd5fee4ca
Merge pull request #10547 from zmstone/0427-default-listeners-in-schema 
0427 default configs in schema
 2023-05-02 15:03:49 +02:00
Ilya Averyanov
791010a65c chore(topic_metrics): do not complain on nonexisting ets while topic metrics stop 2023-05-02 15:59:50 +03:00
Zaiming (Stone) Shi
4d60b0da1d
Merge pull request #10459 from zmstone/0420-delete-old-code 
0420 delete old code
 2023-05-02 13:27:51 +02:00
Zaiming (Stone) Shi
cad4144773
Merge pull request #10536 from zmstone/0118-for-better-test-coverage 
0118 for better test coverage
 2023-05-02 13:26:22 +02:00
Zaiming (Stone) Shi
2dd9191718 refactor: use different terms for config tombstone 
there are 3 different kind of Erlang terms for tombstone related configs
1. the schema type (must be an atom)
2. the config value (must be a binary)
3. the config change comamnd (request) which is only used
   in the code, but never persisted
 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
b1dfbf7984 refactor: move shared macros to header file 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
674f837f36 refactor(emqx_listeners): better variable names 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
29c9edeb4c test(emqx_delayed_SUITE): fix flaky test 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
37bf12c29e test(emqx_telemetry_SUITE): fix flakyness 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
c825102bed fix(authz): ensure acl.conf path template rendered 2023-05-02 09:10:04 +02:00
Zaiming (Stone) Shi
a3b1664c06 test: allow inter-test case config dependency for emqx_exhook_SUITE 2023-05-01 20:15:47 +02:00
Zaiming (Stone) Shi
b65a71b498 test: allow emqx_ws_connection_SUITE to run without erasing configs 2023-05-01 19:37:12 +02:00
Zaiming (Stone) Shi
95cb262067 test: fix authn test cases 2023-05-01 18:42:45 +02:00
Zaiming (Stone) Shi
a03f2dd64b test: allow pre-load configs before emqx_conf app 2023-05-01 15:35:33 +02:00
Zaiming (Stone) Shi
43c80ba635 chore: always init_load config wiht defaults populated 
this effectively eliminates the need for raw_with_default
because it's now always set to true everywhere.
will remove it in a followup.
 2023-04-30 21:24:46 +02:00
Zaiming (Stone) Shi
475dee32ee test(emqx_dashboard): refine spec error 2023-04-30 17:28:41 +02:00
Zaiming (Stone) Shi
57cc854a4a test(bridge): fix bridge map type filed converters 
now the converters on map type fields only work at the wrapping map
level but not the values
 2023-04-30 10:45:11 +02:00
Zaiming (Stone) Shi
03ae61569f test(authn): fix test case after authentication default value added 2023-04-30 10:18:18 +02:00
Zhongwen Deng
2ab0e30489 chore: seperate avail and default tls version 2023-04-30 10:48:54 +08:00
Zhongwen Deng
d8c4c6637b feat: mark ssl_options.password as low level 2023-04-30 10:48:54 +08:00
Zhongwen Deng
cc2d529562 feat: remove tlsv1.1,tlsv1,dtlsv1 from default ssl version 2023-04-30 10:48:54 +08:00
Zhongwen Deng
b0eca5bc00 feat: aliases etcd.ssl to etcd.ssl_options 2023-04-30 10:48:54 +08:00
Zaiming (Stone) Shi
2e9dca280c refactor(listener-schema): use a tombstone for deleted listeners 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
c13a972bf0 test(emqx_management): add test group for listener API 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
b3c0abf494 test(emqx_management): fix listeners api test cases 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
dbf9bae7dc test(statsd): fix raw config default value 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
4d705817d8 refactor(log): move default values to schema 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
41f13330ba refactor: export EMQX_LOG_DIR 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
a1551213c8 test: EMQX_ETC_DIR for test is app's etc dir 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
e0dc5645ab fix(config): always fill defautls for all roots 
prior to this commit, if a root existed in config files
it skips populating default values in raw config,
this made impossible to add default values for authz sources.
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
5acf0e281e refactor: delete default authz config from emqx.conf 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
7c5a9e0e20 refactor: move the env interpolation function to emqx_schema 
also added test cases
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
b0f3a654ee refactor: delete default listeners from default config 
The new config overriding rule is very much confusing for
people who wants to persist listener config changes made from
dashboard

This commit moves the default values from default config file
to schema source code.
In order to support build-time cert path at runtime, there
is also a naive environment variable interplation feature added.
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
c58ffce75f fix(hocon): pin 0.38.2 with the map type value converter fixed 2023-04-29 22:10:19 +02:00
Thales Macedo Garitezi
a19621e533 fix(webhook): treat {shutdown, normal} and {closed, _} async reply as retriable 
Apparently, the async reply returned by ehttpc can be `{shutdown,
normal}` or `{closed, "The connection was lost."}`, in which case the
request should be retried.

```
Apr 28 17:40:41 emqx-0.int.thales bash[48880]: 17:40:41.803 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:shutdown, :normal}]
Apr 28 18:36:37 emqx-0.int.thales bash[53368]: 18:36:37.605 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:closed, 'The connection was lost.'}]
```
 2023-04-28 18:07:32 -03:00
Zaiming (Stone) Shi
7a81b96be0 fix(emqx_conf_app): print init_load failure to standard_error 
logger may not get the chance to spit out the logs before the vm
dies, no matter how long sleep is added before init:stop(1)
 2023-04-28 22:23:30 +02:00
Zaiming (Stone) Shi
d3a26b45be docs: update config note 2023-04-28 22:23:30 +02:00
Thales Macedo Garitezi
633eacad3b test(pulsar): add more test cases for Pulsar Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-8400
 2023-04-28 10:47:03 -03:00
Andrew Mayorov
bd9f129bb8
test(ft-api): also run testcases under cluster setup 2023-04-28 13:50:23 +03:00
Andrew Mayorov
a9866fede4
feat(ft-api): support paging in S3 storage exporter 2023-04-28 13:49:17 +03:00
Andrew Mayorov
75cceffa06
fix(ft-test): rename testcases for consistency 2023-04-28 13:49:17 +03:00
Andrew Mayorov
ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks 
In order to support paging over filesystem contents, to serve REST
APIs effectively.
 2023-04-28 13:47:50 +03:00
Andrew Mayorov
a79cf681f7
fix(s3): drop default from acl config field 2023-04-28 10:54:21 +03:00
Andrew Mayorov
811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Zaiming (Stone) Shi
6e36139a17 chore: bump to e5.0.3-alpha.5 2023-04-27 22:56:34 +02:00
Thales Macedo Garitezi
270fa5d19d
Merge pull request #10550 from thalesmg/fix-ocsp-disabled-r50 
fix(ocsp): disable periodic refresh when listener or stapling are disabled
 2023-04-27 17:22:44 -03:00
Thales Macedo Garitezi
928ca7d565
Merge pull request #10538 from thalesmg/rel-e503-alpha4 
chore: bump release version to `e5.0.3-alpha.4`
 2023-04-27 16:37:08 -03:00
Paulo Zulato
5f835627f9
Merge pull request #10498 from paulozulato/master 
feat(oracle): Oracle Database integration
 2023-04-27 16:05:23 -03:00
Thales Macedo Garitezi
2bb40787f7
Merge pull request #10552 from thalesmg/fix-flaky-banned-test-v50 
test(banned): attempt to fix flaky test
 2023-04-27 15:47:37 -03:00
Thales Macedo Garitezi
0bd4beae6e
Merge pull request #10549 from thalesmg/tests-is-ci-var-v50 
ci: set `IS_CI=yes` when running tests
 2023-04-27 14:12:31 -03:00
Thales Macedo Garitezi
d845c4807d fix(ocsp): disable periodic refresh when listener or stapling are disabled 
Fixes https://emqx.atlassian.net/browse/EMQX-9773
 2023-04-27 14:02:55 -03:00
Thales Macedo Garitezi
d0c4c70f74 test(banned): attempt to fix flaky test 
Example failure:
https://github.com/emqx/emqx/actions/runs/4821105856/jobs/8587006829#step:8:4495

```
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
emqx_common_test_helpers:wait_for_down failed on line 434
Reason: {{t_session_taken,178,timeout},[{emqx_common_test_helpers,...},{...}|...]}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Testing lib.emqx.emqx_banned_SUITE: *** FAILED test case 4 of 5 ***
%%% emqx_banned_SUITE ==> t_session_taken: FAILED
%%% emqx_banned_SUITE ==> {{t_session_taken,178,timeout},
 [{emqx_common_test_helpers,wait_for_down,6,
                            [{file,"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                             {line,434}]},
  {emqx_banned_SUITE,t_session_taken,1,
                     [{file,"/__w/emqx/emqx/source/apps/emqx/test/emqx_banned_SUITE.erl"},
                      {line,176}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1291}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1223}]}]}
```
 2023-04-27 13:19:26 -03:00
Paulo Zulato
43bb6f00ca fix(oracle): drop support for async queries 
jamdb_oracle does not provide interface for performing async queries and
ecpool does not monitor the worker which calls jamdb_oracle, so it's
safer to keep support for sync queries only.
 2023-04-27 13:07:20 -03:00
Paulo Zulato
dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi
c984449bad
Merge pull request #10530 from thalesmg/fix-wait-ssl-crl-test-v50 
test(crl): ensure ssl_manager is ready to avoid flakiness
 2023-04-27 11:59:34 -03:00
Thales Macedo Garitezi
521b549049 test(peer): define cookie when using ct_slave module 2023-04-27 11:58:40 -03:00
Thales Macedo Garitezi
7853a4c36e chore: bump app vsns 2023-04-27 11:58:28 -03:00
Thales Macedo Garitezi
567413389c
Merge pull request #10519 from thalesmg/fix-flaky-res-test-v50 
test(resource): fix flaky test
 2023-04-27 09:33:40 -03:00
Thales Macedo Garitezi
99448151e9 test(crl): ensure ssl_manager is ready to avoid flakiness 
Example failure: https://github.com/emqx/emqx/actions/runs/4806430125/jobs/8555021522?pr=10524#step:8:49138
 2023-04-27 09:12:40 -03:00
Zhongwen Deng
e789e57b65 chore: change node.data_dir from hidden to low 2023-04-27 14:41:23 +08:00
firest
ce2f2217ee chore: bump versions 2023-04-27 10:36:50 +08:00