yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,806 Commits 393 Branches 669 Tags 74 MiB
760de7becf
Commit Graph

7749 Commits

Author SHA1 Message Date
Zaiming (Stone) Shi
f2a223c1e9 chore: prepare for e5.0.3 release 2023-05-08 10:59:04 +02:00
Stefan Strigler
d920f415cd fix(emqx_logger): fix spec for set_log_level 2023-05-08 09:57:26 +02:00
Stefan Strigler
9e324e8523 style: fix typo 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-05-08 09:56:43 +02:00
Stefan Strigler
b639353cf4 fix(emqx_connector_http): wrap and unwrap auth headers 
This so we don't have sensitive information in a process' state
 2023-05-08 09:56:43 +02:00
Zhongwen Deng
14ecfce69f fix: systemd stop don't stop port in the desired order 2023-05-08 15:44:11 +08:00
William Yang
b6c7e55348
Merge pull request #10528 from qzhuyan/perf/william/force-config-atom-path 
perf(config): ensure root keys of 'conf' config is atom
 2023-05-08 09:37:18 +02:00
Zhongwen Deng
4f396a36a9 Merge remote-tracking branch 'upstream/master' into release-50 2023-05-08 14:58:03 +08:00
firest
d914d1ee1d refactor(limiter): simplify limiter configuration 2023-05-08 11:45:49 +08:00
Zaiming (Stone) Shi
aaf3bf27c4 fix(kafka): ensure brod_gssapi sasl_auth are include in release 2023-05-07 09:43:58 +02:00
zhongwencool
d7b10fc329
Merge pull request #10489 from zhongwencool/warning-config-unknown-key 
feat: warning unknown config root key
 2023-05-06 14:28:56 +08:00
JimMoen
4000fb6692
docs: refine link text after review 2023-05-06 10:52:16 +08:00
Zhongwen Deng
f45efbd12d feat: warning unknown config root key 2023-05-06 10:43:52 +08:00
JimMoen
d5dce771bd
refactor(sqlserver): move sqlserver bridge into its own app 2023-05-06 10:22:05 +08:00
William Yang
48cf089870 fix(mqtt): drop all local messages in session deliver 2023-05-05 18:38:18 +02:00
William Yang
08d67aac9f test: fix emqx_mqtt_caps_SUITE 2023-05-05 17:45:11 +02:00
Ilya Averyanov
dd3471bc22 Merge branch 'master' into file-transfer 
* master: (194 commits)
  fix(limiter): update change && fix deprecated version
  chore: update changes
  perf(limiter): simplify the memory represent of limiter configuration
  ci(perf test): update tf variable name and set job timeout
  ci: fix artifact name in scheduled packages workflow
  fix: build_packages_cron.yaml workflow
  ci: move scheduled builds to a separate workflow
  build: check mnesia compatibility when generating mria config
  docs: fix a typo in api doc description
  feat(./dev): use command style and added 'ctl' command
  test: fix delayed-pubish test case flakyness
  refactor: remove raw_with_default config load option
  chore: add changelog for trace timestrap
  feat: increase the time precision of trace logs to microseconds
  chore: make sure topic_metrics/rewrite's default is []
  docs: Update changes/ce/perf-10417.en.md
  chore: bump `snabbkaffe` to 1.0.8
  ci: run static checks in separate jobs
  chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN
  chore: remove unused mqtt cap 'subscription_identifiers'
  ...
 2023-05-05 16:50:18 +03:00
zhongwencool
fb3c0c1fe9
Merge pull request #10546 from zhongwencool/04-27-ssl_options 
feat: organize the ssl_options
 2023-05-05 20:51:19 +08:00
Zaiming (Stone) Shi
8ce2122516 chore: prepare to cut e5.0.3-rc.1 2023-05-05 12:56:34 +02:00
lafirest
335d948bce
Merge pull request #10591 from lafirest/fix/simplify_limiter_client_cfg 
perf(limiter): simplify the memory represent of limiter configuration
 2023-05-05 16:59:04 +08:00
Ilya Averyanov
cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
firest
7a96a9772d fix(limiter): update change && fix deprecated version 2023-05-05 16:26:06 +08:00
William Yang
e4f501417b
Merge pull request #10525 from qzhuyan/perf/william/avoid-new-map-when-get-mqtt-caps 
perf(config): avoid build new map in emqx_mgmt_caps:get_caps
 2023-05-05 10:22:08 +02:00
firest
4f47e65b7b perf(limiter): simplify the memory represent of limiter configuration 2023-05-05 16:21:21 +08:00
Zaiming (Stone) Shi
14e055f18b
Merge pull request #10578 from zmstone/0502-refactor-delete-stale-config-check-opt 
0502 refactor delete stale config check opt
 2023-05-04 16:32:49 +02:00
firest
277deee616 fix: add new sensitive keywords to redact checklist 2023-05-04 16:32:32 +08:00
zhongwencool
2bc1ddbd19
Merge pull request #10588 from zhongwencool/fix-array-default 
chore: make sure topic_metrics/rewrite's default is []
 2023-05-04 16:03:21 +08:00
Zaiming (Stone) Shi
06e25f5c72 test: fix delayed-pubish test case flakyness 2023-05-04 09:48:28 +02:00
Zaiming (Stone) Shi
856129984b refactor: remove raw_with_default config load option 
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
 2023-05-04 09:48:28 +02:00
Zhongwen Deng
06960853e0 feat: increase the time precision of trace logs to microseconds 2023-05-04 14:41:02 +08:00
Zhongwen Deng
b09ee540d0 chore: make sure topic_metrics/rewrite's default is [] 2023-05-04 14:29:54 +08:00
Paulo Zulato
f965fa7b9a feat: add shutdown count 
Fixes https://emqx.atlassian.net/browse/EMQX-9008
Fixes https://emqx.atlassian.net/browse/EMQX-9219
 2023-05-03 20:00:38 -03:00
William Yang
c38cec77a7
Merge pull request #10417 from qzhuyan/perf/william/no-make-ref-in-get-config 
perf(config): eliminate make_ref() calls in config get calls
 2023-05-03 16:56:44 +02:00
William Yang
2ded8e3836
Merge pull request #10577 from qzhuyan/dev/william/importance-mark-quic-listener 
chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN
 2023-05-03 13:23:32 +02:00
Andrew Mayorov
754045173e
chore: bump snabbkaffe to 1.0.8 2023-05-03 11:16:51 +03:00
Zaiming (Stone) Shi
66112fceed
Merge pull request #10572 from zmstone/0502-merge-release-50-back-to-master 
0502 merge release 50 back to master
 2023-05-03 09:01:35 +02:00
Ilya Averyanov
0272cf4473
Merge pull request #10571 from savonarola/0502-do-not-pullute-logs-on-metrics-stop 
Do not complain on nonexisting ets while `emqx_topic_metrics` stops
 2023-05-03 09:44:25 +03:00
William Yang
b418cc766d test: update emqx_gateway_conf_SUITE 
for error return messages
 2023-05-02 22:59:24 +02:00
William Yang
a4a5599636 chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN 2023-05-02 22:20:39 +02:00
Thales Macedo Garitezi
8aa7c014e7 perf(buffer_worker): avoid calling ets:info/2 
(Almost?) fixes https://emqx.atlassian.net/browse/EMQX-9637

During the course of performance tests comparing the performance of
e5.0.3 and e4.4.16 regarding the webhook bridge in sync mode, we
observed that the throughput in e5.0.3 (sync) was much lower than in
e4.4.16: ~ 9 k msgs / s vs. ~ 50 k msgs / s, respectively.

Analyzing `observer_cli` output, we noticed that a lot of the time
both buffer workers and ehttpc processes was spent in `ets:info/2`.
That function was called to check the size of the inflight table when
updating metrics and checking if the inflight table was full.  Other
uses of `ets:info/2` were contained inside the arguments to some
`?tp/2` macro usages (https://github.com/kafka4beam/snabbkaffe/pull/60).

By using a specific record to track the size of the table, we managed
to improve the bridge performance to ~ 45 k msgs / s in sync mode.
 2023-05-02 17:05:32 -03:00
William Yang
c7af43bd72 chore: bump app vsn emqx 5.0.25 2023-05-02 21:51:43 +02:00
William Yang
e5884e0e87 chore: remove unused mqtt cap 'subscription_identifiers' 2023-05-02 21:50:36 +02:00
William Yang
be6cd5705f fix(mqtt-cap): use global config as defaults 
Don't use default values in module, use global mqtt config as default
 2023-05-02 21:50:36 +02:00
William Yang
df0911be6e Revert "perf: keep defaults of zone.mqtt" 
This reverts commit 6184bc02ee30aa2d022793762e626893a8a32599.
 2023-05-02 21:50:36 +02:00
William Yang
77405d9cd7 perf: keep defaults of zone.mqtt 2023-05-02 21:50:36 +02:00
William Yang
265c9ea668 chore: bump emqx app vsn to 5.0.25 2023-05-02 21:50:36 +02:00
William Yang
3d7201502b perf(config): avoid build new map in emqx_mqtt_caps:get_caps 2023-05-02 21:50:36 +02:00
William Yang
77f67a9d07 perf(config): get_raw with 'binary' root key 2023-05-02 21:49:55 +02:00
William Yang
ce6343a4f3 perf(config): ensure root keys of 'conf' config is atom 2023-05-02 21:49:55 +02:00
Zaiming (Stone) Shi
a4bc15cfa1 Merge remote-tracking branch 'origin/master' into 0502-merge-release-50-back-to-master 2023-05-02 18:59:31 +02:00
Thales Macedo Garitezi
f6da18dc1b fix(webhook): consider {shutdown, closed} return as recoverable 2023-05-02 13:12:03 -03:00
Zaiming (Stone) Shi
d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Andrew Mayorov
90cf1ade74
chore: bump application versions 
* emqx_connector 0.1.22
* emqx_bridge_gcp_pubsub 0.1.1
* emqx_bridge_kafka 0.1.2
* emqx_bridge_pulsar 0.1.1
* emqx_ee_connector 0.1.12
 2023-05-02 17:29:22 +03:00
Andrew Mayorov
670709f746
feat(resource): ensure uniqueness through gproc 
Also use it instead of a custom ETS table for simplicity and
better consistency. This has drawbacks though: expect slightly
increased load on gproc gen_server due to how `gproc:set_value/2`
works.
 2023-05-02 17:29:22 +03:00
Andrew Mayorov
4575167607
feat(resource): drop manager_id() type 2023-05-02 17:29:20 +03:00
Andrew Mayorov
aaef95b1da
feat(resman): stop adding uniqueness to manager ids 
Before this change, a separate `manager_id` / `instance_id` was used
as resource manager id, which made connector interface somewhat
inconsistent: part of function calls to connector implementation
used instance id as first argument while the rest used resource id
itself.
 2023-05-02 17:28:26 +03:00
Thales Macedo Garitezi
2c4fd98ce5
Merge pull request #10559 from thalesmg/pulsar-more-tests-v50 
test(pulsar): add more test cases for Pulsar Producer bridge
 2023-05-02 10:16:17 -03:00
Thales Macedo Garitezi
654d274039
Merge pull request #10562 from thalesmg/fix-webhook-async-shutdown-normal-reply-r50 
fix(webhook): treat `{shutdown, normal}` and `{closed, _}` async replies as retriable
 2023-05-02 10:08:18 -03:00
Zaiming (Stone) Shi
0fd5fee4ca
Merge pull request #10547 from zmstone/0427-default-listeners-in-schema 
0427 default configs in schema
 2023-05-02 15:03:49 +02:00
Ilya Averyanov
791010a65c chore(topic_metrics): do not complain on nonexisting ets while topic metrics stop 2023-05-02 15:59:50 +03:00
Zaiming (Stone) Shi
4d60b0da1d
Merge pull request #10459 from zmstone/0420-delete-old-code 
0420 delete old code
 2023-05-02 13:27:51 +02:00
Zaiming (Stone) Shi
cad4144773
Merge pull request #10536 from zmstone/0118-for-better-test-coverage 
0118 for better test coverage
 2023-05-02 13:26:22 +02:00
Zaiming (Stone) Shi
2dd9191718 refactor: use different terms for config tombstone 
there are 3 different kind of Erlang terms for tombstone related configs
1. the schema type (must be an atom)
2. the config value (must be a binary)
3. the config change comamnd (request) which is only used
   in the code, but never persisted
 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
b1dfbf7984 refactor: move shared macros to header file 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
674f837f36 refactor(emqx_listeners): better variable names 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
29c9edeb4c test(emqx_delayed_SUITE): fix flaky test 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
37bf12c29e test(emqx_telemetry_SUITE): fix flakyness 2023-05-02 13:24:59 +02:00
Zaiming (Stone) Shi
c825102bed fix(authz): ensure acl.conf path template rendered 2023-05-02 09:10:04 +02:00
Zaiming (Stone) Shi
a3b1664c06 test: allow inter-test case config dependency for emqx_exhook_SUITE 2023-05-01 20:15:47 +02:00
Zaiming (Stone) Shi
b65a71b498 test: allow emqx_ws_connection_SUITE to run without erasing configs 2023-05-01 19:37:12 +02:00
Zaiming (Stone) Shi
95cb262067 test: fix authn test cases 2023-05-01 18:42:45 +02:00
Zaiming (Stone) Shi
a03f2dd64b test: allow pre-load configs before emqx_conf app 2023-05-01 15:35:33 +02:00
Zaiming (Stone) Shi
43c80ba635 chore: always init_load config wiht defaults populated 
this effectively eliminates the need for raw_with_default
because it's now always set to true everywhere.
will remove it in a followup.
 2023-04-30 21:24:46 +02:00
Zaiming (Stone) Shi
475dee32ee test(emqx_dashboard): refine spec error 2023-04-30 17:28:41 +02:00
Zaiming (Stone) Shi
57cc854a4a test(bridge): fix bridge map type filed converters 
now the converters on map type fields only work at the wrapping map
level but not the values
 2023-04-30 10:45:11 +02:00
Zaiming (Stone) Shi
03ae61569f test(authn): fix test case after authentication default value added 2023-04-30 10:18:18 +02:00
Zhongwen Deng
2ab0e30489 chore: seperate avail and default tls version 2023-04-30 10:48:54 +08:00
Zhongwen Deng
d8c4c6637b feat: mark ssl_options.password as low level 2023-04-30 10:48:54 +08:00
Zhongwen Deng
cc2d529562 feat: remove tlsv1.1,tlsv1,dtlsv1 from default ssl version 2023-04-30 10:48:54 +08:00
Zhongwen Deng
b0eca5bc00 feat: aliases etcd.ssl to etcd.ssl_options 2023-04-30 10:48:54 +08:00
Zaiming (Stone) Shi
2e9dca280c refactor(listener-schema): use a tombstone for deleted listeners 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
c13a972bf0 test(emqx_management): add test group for listener API 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
b3c0abf494 test(emqx_management): fix listeners api test cases 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
dbf9bae7dc test(statsd): fix raw config default value 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
4d705817d8 refactor(log): move default values to schema 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
41f13330ba refactor: export EMQX_LOG_DIR 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
a1551213c8 test: EMQX_ETC_DIR for test is app's etc dir 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
e0dc5645ab fix(config): always fill defautls for all roots 
prior to this commit, if a root existed in config files
it skips populating default values in raw config,
this made impossible to add default values for authz sources.
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
5acf0e281e refactor: delete default authz config from emqx.conf 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
7c5a9e0e20 refactor: move the env interpolation function to emqx_schema 
also added test cases
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
b0f3a654ee refactor: delete default listeners from default config 
The new config overriding rule is very much confusing for
people who wants to persist listener config changes made from
dashboard

This commit moves the default values from default config file
to schema source code.
In order to support build-time cert path at runtime, there
is also a naive environment variable interplation feature added.
 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi
c58ffce75f fix(hocon): pin 0.38.2 with the map type value converter fixed 2023-04-29 22:10:19 +02:00
Thales Macedo Garitezi
a19621e533 fix(webhook): treat {shutdown, normal} and {closed, _} async reply as retriable 
Apparently, the async reply returned by ehttpc can be `{shutdown,
normal}` or `{closed, "The connection was lost."}`, in which case the
request should be retried.

```
Apr 28 17:40:41 emqx-0.int.thales bash[48880]: 17:40:41.803 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:shutdown, :normal}]
Apr 28 18:36:37 emqx-0.int.thales bash[53368]: 18:36:37.605 [error] [id: "bridge:webhook:webhook", msg: :unrecoverable_error, reason: {:closed, 'The connection was lost.'}]
```
 2023-04-28 18:07:32 -03:00
Zaiming (Stone) Shi
7a81b96be0 fix(emqx_conf_app): print init_load failure to standard_error 
logger may not get the chance to spit out the logs before the vm
dies, no matter how long sleep is added before init:stop(1)
 2023-04-28 22:23:30 +02:00
Zaiming (Stone) Shi
d3a26b45be docs: update config note 2023-04-28 22:23:30 +02:00
Thales Macedo Garitezi
633eacad3b test(pulsar): add more test cases for Pulsar Producer bridge 
Fixes https://emqx.atlassian.net/browse/EMQX-8400
 2023-04-28 10:47:03 -03:00
Andrew Mayorov
bd9f129bb8
test(ft-api): also run testcases under cluster setup 2023-04-28 13:50:23 +03:00
Andrew Mayorov
a9866fede4
feat(ft-api): support paging in S3 storage exporter 2023-04-28 13:49:17 +03:00
Andrew Mayorov
75cceffa06
fix(ft-test): rename testcases for consistency 2023-04-28 13:49:17 +03:00
Andrew Mayorov
ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov
573bb22ada
feat(ft-fs): introduce fs iterators concept + forward seeks 
In order to support paging over filesystem contents, to serve REST
APIs effectively.
 2023-04-28 13:47:50 +03:00
Andrew Mayorov
a79cf681f7
fix(s3): drop default from acl config field 2023-04-28 10:54:21 +03:00
Andrew Mayorov
811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Zaiming (Stone) Shi
6e36139a17 chore: bump to e5.0.3-alpha.5 2023-04-27 22:56:34 +02:00
Thales Macedo Garitezi
270fa5d19d
Merge pull request #10550 from thalesmg/fix-ocsp-disabled-r50 
fix(ocsp): disable periodic refresh when listener or stapling are disabled
 2023-04-27 17:22:44 -03:00
Thales Macedo Garitezi
928ca7d565
Merge pull request #10538 from thalesmg/rel-e503-alpha4 
chore: bump release version to `e5.0.3-alpha.4`
 2023-04-27 16:37:08 -03:00
Paulo Zulato
5f835627f9
Merge pull request #10498 from paulozulato/master 
feat(oracle): Oracle Database integration
 2023-04-27 16:05:23 -03:00
Thales Macedo Garitezi
2bb40787f7
Merge pull request #10552 from thalesmg/fix-flaky-banned-test-v50 
test(banned): attempt to fix flaky test
 2023-04-27 15:47:37 -03:00
Thales Macedo Garitezi
0bd4beae6e
Merge pull request #10549 from thalesmg/tests-is-ci-var-v50 
ci: set `IS_CI=yes` when running tests
 2023-04-27 14:12:31 -03:00
Thales Macedo Garitezi
d845c4807d fix(ocsp): disable periodic refresh when listener or stapling are disabled 
Fixes https://emqx.atlassian.net/browse/EMQX-9773
 2023-04-27 14:02:55 -03:00
Thales Macedo Garitezi
d0c4c70f74 test(banned): attempt to fix flaky test 
Example failure:
https://github.com/emqx/emqx/actions/runs/4821105856/jobs/8587006829#step:8:4495

```
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
emqx_common_test_helpers:wait_for_down failed on line 434
Reason: {{t_session_taken,178,timeout},[{emqx_common_test_helpers,...},{...}|...]}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Testing lib.emqx.emqx_banned_SUITE: *** FAILED test case 4 of 5 ***
%%% emqx_banned_SUITE ==> t_session_taken: FAILED
%%% emqx_banned_SUITE ==> {{t_session_taken,178,timeout},
 [{emqx_common_test_helpers,wait_for_down,6,
                            [{file,"/__w/emqx/emqx/source/apps/emqx/test/emqx_common_test_helpers.erl"},
                             {line,434}]},
  {emqx_banned_SUITE,t_session_taken,1,
                     [{file,"/__w/emqx/emqx/source/apps/emqx/test/emqx_banned_SUITE.erl"},
                      {line,176}]},
  {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
  {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1291}]},
  {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1223}]}]}
```
 2023-04-27 13:19:26 -03:00
Paulo Zulato
43bb6f00ca fix(oracle): drop support for async queries 
jamdb_oracle does not provide interface for performing async queries and
ecpool does not monitor the worker which calls jamdb_oracle, so it's
safer to keep support for sync queries only.
 2023-04-27 13:07:20 -03:00
Paulo Zulato
dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi
c984449bad
Merge pull request #10530 from thalesmg/fix-wait-ssl-crl-test-v50 
test(crl): ensure ssl_manager is ready to avoid flakiness
 2023-04-27 11:59:34 -03:00
Thales Macedo Garitezi
521b549049 test(peer): define cookie when using ct_slave module 2023-04-27 11:58:40 -03:00
Thales Macedo Garitezi
7853a4c36e chore: bump app vsns 2023-04-27 11:58:28 -03:00
Thales Macedo Garitezi
567413389c
Merge pull request #10519 from thalesmg/fix-flaky-res-test-v50 
test(resource): fix flaky test
 2023-04-27 09:33:40 -03:00
Thales Macedo Garitezi
99448151e9 test(crl): ensure ssl_manager is ready to avoid flakiness 
Example failure: https://github.com/emqx/emqx/actions/runs/4806430125/jobs/8555021522?pr=10524#step:8:49138
 2023-04-27 09:12:40 -03:00
Zhongwen Deng
e789e57b65 chore: change node.data_dir from hidden to low 2023-04-27 14:41:23 +08:00
firest
ce2f2217ee chore: bump versions 2023-04-27 10:36:50 +08:00
firest
262f0e8dd9 chore: update README 2023-04-27 10:31:19 +08:00
Thales Macedo Garitezi
7e24b35bb3 chore: bump release version to e5.0.3-alpha.4 2023-04-26 17:09:39 -03:00
Zaiming (Stone) Shi
0c284ce5fe chore: bump app versions 2023-04-26 20:50:06 +02:00
Zaiming (Stone) Shi
d89975c6ee test: add a test case for emqx_machine:node_status 2023-04-26 20:48:53 +02:00
Zaiming (Stone) Shi
51cd83e70f refactor: delete dead code 2023-04-26 20:48:53 +02:00
Ivan Dyachkov
31e54d0598
Merge pull request #10532 from id/0426-v5.0.24 
0426 v5.0.24
 2023-04-26 19:21:22 +02:00
Zaiming (Stone) Shi
75294a4f73
Merge pull request #10513 from zmstone/0424-EMQX-9689-stop-providing-desc-and-label-in-schemas-api 
0424 emqx 9689 stop providing desc and label in schemas api (part 1)
 2023-04-26 19:04:33 +02:00
Thales Macedo Garitezi
c53741a08c fix(buffer_worker): avoid sending late reply messages to callers 
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
 2023-04-26 13:18:28 -03:00
Ivan Dyachkov
35c48ef009 chore: v5.0.24 2023-04-26 18:02:44 +02:00
Zaiming (Stone) Shi
ed7a8659d2 feat: add a json format support for the /status API 2023-04-26 14:41:33 +02:00
Zaiming (Stone) Shi
9260b5ec6c test(emqx_dashboard): add test case for api/v5/schemas API 2023-04-26 14:41:33 +02:00
Zaiming (Stone) Shi
55c488fa95 refactor: stop generating static hot-conf and bridges schema files 2023-04-26 14:41:33 +02:00
Zaiming (Stone) Shi
28a68a0ec7 refactor: stop i18n support in hotconf and bridges 
frontend team has decided to deal with translations all by themselves
 2023-04-26 14:41:33 +02:00
Thales Macedo Garitezi
d78312e10e test(resource): fix flaky test 2023-04-26 09:25:33 -03:00
William Yang
15fe445c66
Merge pull request #10527 from qzhuyan/test/william/test-use-os-selected-port 
fix(test): avoid port collision
 2023-04-26 13:39:36 +02:00
William Yang
abf150518c fix(test): avoid port collision 
Use OS selected free port to avoid port collision among the test runs.
 2023-04-26 12:16:16 +02:00
William Yang
5ed3c3a92c perf(config): eliminate make_ref() calls in config get calls 2023-04-26 10:58:08 +02:00
SergeTupchiy
d398276852
Merge pull request #10518 from SergeTupchiy/upgrade-ekka-to-0.15.1-mria-0.5.2 
chore: bump ekka to 0.15.1
 2023-04-26 11:03:23 +03:00
zhongwencool
c78004001c
Merge pull request #10481 from HJianBo/hidden-bad-link 
chore: hide bad links in README files
 2023-04-26 15:22:30 +08:00
zhongwencool
62e1dbdb4e
Merge pull request #10521 from zhongwencool/sync-release-50-to-master 
Sync release 50 to master
 2023-04-26 14:27:51 +08:00
zhongwencool
713e050a79
Merge pull request #10512 from zhongwencool/hocon-0.39.3 
feat: improved the storage format of Unicode characters in data files
 2023-04-26 13:59:51 +08:00
zhongwencool
9d893b49eb
Merge branch 'master' into sync-release-50-to-master 2023-04-26 10:54:46 +08:00
Thales Macedo Garitezi
79cf5cad19
Merge pull request #10378 from thalesmg/pulsar-producer-e50 
feat: implement Pulsar Producer bridge (e5.0)
 2023-04-25 18:01:43 -03:00
Serge Tupchii
19b5ebff81 chore: bump ekka to 0.15.1 
ekka 0.15.1 uses mria 0.5.2, which includes the following changes:
  - fix(mria_membership): call `mria_rlog:role/1` safely
  - feat: add extra field to ?rlog_sync table (for future use)
 2023-04-25 23:21:07 +03:00
Thales Macedo Garitezi
b56a158a54 fix(pulsar): fix function return typespec 2023-04-25 14:29:40 -03:00
Thales Macedo Garitezi
f69ebdcd1a test(pulsar): teardown tls group 2023-04-25 14:27:24 -03:00
Thales Macedo Garitezi
4cc4c4ffaa style: format rebar.config file 2023-04-25 14:26:19 -03:00
Thales Macedo Garitezi
a703707803 chore: tag e5.0.3-alpha.3 2023-04-25 10:51:34 -03:00
Thales Macedo Garitezi
fe1c3cd1a7
Merge pull request #10503 from thalesmg/fix-kconsumer-flaky-test-v50 
test: attempt to fix flaky test
 2023-04-25 09:33:58 -03:00
Thales Macedo Garitezi
3138e2b3a1 chore: un-hide ocsp stapling config 
Undoing https://github.com/emqx/emqx/pull/10160
 2023-04-25 09:33:24 -03:00
Zhongwen Deng
308056f0fc feat: improved the storage format of Unicode characters in data files 2023-04-25 18:08:34 +08:00
Zhongwen Deng
d6208d8847 test: add test for depreated config file 2023-04-25 14:47:05 +08:00
JianBo He
699bd4b85b
Merge pull request #10451 from HJianBo/update-conf-plugin-readme 
chore: update README files
 2023-04-25 14:43:55 +08:00
Zhongwen Deng
f84fc6f8b9 fix: can't update authentication when cluster-override.conf 2023-04-25 11:59:06 +08:00
zhongwencool
79a235f8b5
Merge pull request #10491 from zhongwencool/rename-etcd-ssl-to-ssl_options 
feat: rename etcd.ssl to etcd.ssl_options
 2023-04-25 10:49:09 +08:00
Zhongwen Deng
3f689d0fdf feat: don't do rpc call to check deprecated file 2023-04-25 10:10:56 +08:00
zhongwencool
996d5eee45
Merge pull request #10493 from zhongwencool/remove-dashboard-default-username-conf 
chore: remove dashboard's default username from emqx.conf
 2023-04-25 09:45:30 +08:00
lafirest
29d3afd692
Merge pull request #10487 from lafirest/fix/limiter_instance 
fix(limiter): optimize the instance of limiter
 2023-04-25 09:43:04 +08:00
Andrew Mayorov
a420c92d28
fix(ft): fix typing issue 2023-04-24 21:51:54 +03:00
Andrew Mayorov
5efd590ca4
feat(ft): properly propagate config updates 
Ensure that:
* Storage config might be removed.
* Local FS GC process is set up when Local FS storage is configured.
* Local FS GC process gets its timer reset on config updates.
* Storage / exporter gets chosen based on `type` only.
* Exporter config updates propagated as before.

Also employ `emqx_ft_schema:translate/1` instead of duplicating
defaults where applicable.
 2023-04-24 21:51:54 +03:00
Thales Macedo Garitezi
e9fde12913 test: attempt to fix flaky test 
Example failure: https://github.com/emqx/emqx/actions/runs/4789177314/jobs/8517116154#step:7:503
 2023-04-24 15:19:33 -03:00
Serge Tupchii
99e892b5c4 chore: bump ekka to 0.15.0 
ekka 0.15.0 uses mria 0.5.0, which adds several fixes, enhancements and features:
  - protect `mria:join/1,2` with a global lock
  - implement new function `mria:sync_transaction/4,3,2`, which waits for
    a transaction replication to be ready on the local node
    (if the local node is a replicant)
  - optimize `mria:running_nodes/0`
  - optimize `mria:ro_transaction/2` when called on a replicant node.

Fixes: EMQX-9588 (#10380), EMQX-9102, EMQX-9152, EMQX-9213
 2023-04-24 20:52:20 +03:00
Thales Macedo Garitezi
377b143325 refactor: split parse_server into smaller functions, improve return type to use map 2023-04-24 14:17:29 -03:00
lafirest
2845469c48
Merge pull request #10490 from lafirest/fix/rmv_def_conn_limit 
fix(limiter): remove the default limit of connect rate
 2023-04-24 22:04:29 +08:00
Thales Macedo Garitezi
4f2262129b refactor: rename {first_,}key_dispatch partition strategy option 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
4af6e3eb6e refactor: rm unused modules 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
120d3e70ea chore: bump app vsns 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
1e8dd70a11 chore: fix error message and rename variable 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
180b6acd9e docs: remove auto-generated comment 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
f4a3affd6f docs: change phrasing after review 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-8398
 2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi
f6da118ebd test: fix flaky test 2023-04-24 09:28:36 -03:00
Thales Macedo Garitezi
dc48032309 feat(schema): add support for schemes in server parser/validator 2023-04-24 09:28:36 -03:00
Thales Macedo Garitezi
4bcfbea056 refactor(kafka_consumer): follow up refactoring requested from previous pull request 
Follow up from https://github.com/emqx/emqx/pull/10273
 2023-04-24 09:28:36 -03:00
Thales Macedo Garitezi
26883eec02 test(kafka): fix innocuous test assertion 2023-04-24 09:28:36 -03:00
Thales Macedo Garitezi
8cfb24b5b4 docs(kafka_bridge): minor fixes to license and readme 
Fixes https://emqx.atlassian.net/browse/EMQX-9481
 2023-04-24 09:28:36 -03:00
lafirest
0e96b5d01e
Merge pull request #10483 from lafirest/fix/resource_aovid_crash 
fix(resource): make sure resource will not crash when stopping
 2023-04-24 20:16:01 +08:00
firest
6110aad23f chore: bump version && update changes 2023-04-24 18:17:23 +08:00
firest
feeb3df994 fix(api): add limiter API back which deleted by mistake 2023-04-24 18:09:17 +08:00
zhongwencool
22d6a2eaee
Merge pull request #10484 from zhongwencool/fix-copy-conf 
fix: copy cluster-override.conf from old version
 2023-04-24 17:06:07 +08:00
Zhongwen Deng
275967a49f chore: remove dashboard's default username from emqx.conf 2023-04-24 16:13:42 +08:00
Zaiming (Stone) Shi
366aa53c80
Merge pull request #10462 from zmstone/0430-EMQX-8434-shared-dispatch-ack-deprecated 
chore: Hide config shared_dispatch_ack_enabled
 2023-04-24 09:42:09 +02:00
Zhongwen Deng
db0c951e30 feat: don't do rpc call to check deprecated file 2023-04-24 15:31:02 +08:00
Ivan Dyachkov
988e4ec1aa
Merge pull request #10476 from id/e5.0.3-alpha.2 
chore: e5.0.3-alpha.2
 2023-04-24 09:08:29 +02:00
Zhongwen Deng
0b1a2dd193 feat: rename etcd.ssl to etcd.ssl_options 2023-04-24 14:40:30 +08:00
firest
7b51a49f84 fix(limiter): remove the default limit of connect rate 2023-04-24 14:09:23 +08:00
firest
c2e35a42b0 fix(limiter): optimize the instance of limiter 
We can reduce a limiter container with all types are `infinity` to just a `infinity` atom
 2023-04-24 10:43:55 +08:00
某文
8bfee90322 chore: make static_check happy 2023-04-23 20:06:51 +08:00
某文
6dcecfed40 chore: make static_check happy 2023-04-23 19:47:57 +08:00
某文
b4c16d37c7 chore: make static_check happy 2023-04-23 18:09:51 +08:00
某文
f96c1630e1 chore: pin emqx_conf to 0.17.0 2023-04-23 17:30:07 +08:00
某文
e0fd861863 chore: make static_check happy 2023-04-23 17:24:34 +08:00
某文
5593e38ed3 fix: copy cluster-override.conf from old version 2023-04-23 17:24:24 +08:00
某文
7af9c18caa fix: copy cluster-override.conf from old version 2023-04-23 15:43:18 +08:00
firest
7d2c336ab7 fix(resource): make sure resource will not crash when stopping 2023-04-23 15:31:08 +08:00
firest
5ad5d7ee8d fix(opents): adjust code structure 2023-04-23 14:28:14 +08:00
JianBo He
6e1d6f1991 chore: hide bad links in README files 2023-04-23 11:27:59 +08:00
firest
540518eac3 chore: add README for OpenTSDB bridge 2023-04-23 11:22:48 +08:00
firest
0b46acda87 test(opents): add test cases for OpenTSDB 2023-04-23 11:03:20 +08:00
firest
5074825075 feat(opents): OpenTSDB integration 2023-04-23 09:56:24 +08:00
Ilya Averyanov
69c4ba2a62 feat(ft): use new utils application 2023-04-22 22:50:19 +03:00
Ivan Dyachkov
6beb9e00b6 chore: e5.0.3-alpha.2 2023-04-22 20:09:40 +02:00
Zaiming (Stone) Shi
ceafc52ad6 refactor: use emqx_utils_ets for ets table creation 2023-04-22 09:07:15 +02:00
Ilya Averyanov
ebb75b275e feat(ft): update app versions 2023-04-21 17:52:13 +03:00
Ilya Averyanov
0211bcf030 Merge branch 'master' into file-transfer 
* master: (279 commits)
  chore: shorten ct/run.sh script
  chore: rename cassandra_impl to cassandra_connector
  chore: fix mix.exs checking
  refactor(cassandra): move cassandra bridge into its own app
  chore: apply review suggestions
  chore: update changes/ce/fix-10449.en.md
  test: add a test for authn {}
  chore: add changlog for authn_http validation
  fix: always check authn_http's header and ssl_option
  chore: apply suggestions from code review
  fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
  fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager
  perf(emqx_alarm): use dirty Mnesia operations to activate an alarm
  ci: simplify find-apps.sh for ee apps
  perf(emqx_resource): don't reactivate alarms on reoccurring errors
  ci: check if Elixir files are formatted in pre-commit hook
  fix(dynamo): fix field name errors
  chore: remove *_collector for prometheus api's example
  chore: make plugins config to low level
  chore: re-split dynamo i18n file
  ...
 2023-04-21 17:37:17 +03:00
某文
df31a8a342 test: conf_schema_tests failed 2023-04-21 20:29:14 +08:00
Zaiming (Stone) Shi
701a1f65f9 chore: Hide config shared_dispatch_ack_enabled 2023-04-21 13:59:55 +02:00
JianBo He
b270623c46 chore: rename cassandra_impl to cassandra_connector 2023-04-21 16:49:25 +08:00
JianBo He
cea0502160
chore: update apps/emqx_conf/README.md 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
 2023-04-21 16:23:47 +08:00
Zhongwen Deng
1db38de71f chore: apply review suggestions 2023-04-21 15:32:16 +08:00
Zhongwen Deng
dc92b4f63f test: add a test for authn {} 2023-04-21 15:30:17 +08:00
Zhongwen Deng
397e28f5a4 chore: add changlog for authn_http validation 2023-04-21 15:30:05 +08:00
Zhongwen Deng
c1000ccaed fix: always check authn_http's header and ssl_option 2023-04-21 15:29:51 +08:00
JianBo He
5cc28a7b45 chore: fix mix.exs checking 2023-04-21 14:46:09 +08:00
JianBo He
bdce32e713 refactor(cassandra): move cassandra bridge into its own app 2023-04-21 14:45:38 +08:00
JianBo He
6532cfb337
Merge pull request #10452 from thalesmg/refactor-gcp-pubsub-bridge-v50 
refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
 2023-04-21 14:42:00 +08:00
Zhongwen Deng
fdf9b2a383 chore: apply review suggestions 2023-04-21 12:19:38 +08:00
Zhongwen Deng
1e54d23d31 test: add a test for authn {} 2023-04-21 11:54:45 +08:00
Zhongwen Deng
2aef9ca215 chore: add changlog for authn_http validation 2023-04-21 11:54:45 +08:00
Zhongwen Deng
8e8ba6ce7e fix: always check authn_http's header and ssl_option 2023-04-21 11:54:45 +08:00
SergeTupchiy
0b105dcb8d
Merge pull request #10463 from SergeTupchiy/EMQX-9310-webhook-port-validation 
fix(emqx_bridge): validate Webhook bad URL
 2023-04-20 20:48:12 +03:00
Ilya Averyanov
8c4b32580e
Merge pull request #10420 from savonarola/0417-authn-authz-fix-path-quoting 
Fix path joining and quoting in authn/authz
 2023-04-20 18:03:04 +03:00
SergeTupchiy
b38ae7f78f
Merge pull request #10407 from SergeTupchiy/EMQX-9529-resource-manager-crash-on-alarm-timeout 
fix(emqx_resource): call emqx_alarm safely and don't reactivate alarm on reoccurring errors
 2023-04-20 17:53:56 +03:00
Zaiming (Stone) Shi
82e6ce53be
Merge pull request #10460 from zmstone/0420-sync-release-50-to-masteer 
0420 sync release 50 to masteer
 2023-04-20 16:41:50 +02:00
JianBo He
de8da7dc37
Merge pull request #10434 from HJianBo/improve-modules-promethues-readme 
Improve modules promethues readme
 2023-04-20 22:37:54 +08:00
JianBo He
5a7685a341
chore: apply suggestions from code review 
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-20 22:26:29 +08:00
Serge Tupchii
45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid 
Fixes: EMQX-9310
 2023-04-20 17:21:43 +03:00
Serge Tupchii
423a30fbb3 fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager 
Don't let 'emqx_resource_manager' crash because of emqx_alarm timeouts.

Fixes: EMQX-9529/#10357
 2023-04-20 17:15:13 +03:00
Serge Tupchii
b960d2ecb3 perf(emqx_alarm): use dirty Mnesia operations to activate an alarm 
Alarms are stored in a local content shard and all 'activate' operations
are serialized as they are called by one process ('emqx_alarm' gen_server), so
using dirty operations gives performance gain without sacrificing consistency.

Fixes: EMQX-9529/#10357
 2023-04-20 17:15:13 +03:00
Serge Tupchii
b5eda9f0d1 perf(emqx_resource): don't reactivate alarms on reoccurring errors 
Avoid unnecessary calls to activate an alarm if it has been already activated.

Fixes: EMQX-9529/#10357
 2023-04-20 16:37:33 +03:00
Thales Macedo Garitezi
3f18c5e2e3
Merge pull request #10455 from thalesmg/fix-late-gen-server-replies-buf-worker-v50 
fix(buffer_worker): avoid sending late reply messages to callers
 2023-04-20 10:19:06 -03:00
Zhongwen Deng
4464a31fab chore: remove *_collector for prometheus api's example 2023-04-20 18:01:27 +08:00
Zhongwen Deng
bcc8f4313b chore: make plugins config to low level 2023-04-20 18:00:19 +08:00
zhongwencool
72685beb72
Merge pull request #10457 from HJianBo/hide-statsd 
chore: deprecate statsd
 2023-04-20 17:19:33 +08:00
Zaiming (Stone) Shi
415830a0a2 Merge remote-tracking branch 'origin/release-50' into 0420-sync-release-50-to-masteer 2023-04-20 11:03:39 +02:00
Kjell Winblad
7d3367467a
Merge pull request #10408 from kjellwinblad/kjell/rule_engine/add_missing_mongo_date_functions/EMQX-9244 
feat: add mongo_date functions to the rule engine
 2023-04-20 09:46:45 +02:00
JianBo He
794ddd8d73 chore: update changes 2023-04-20 14:23:55 +08:00
JianBo He
712f7b7454 chore: deprecate statsd 2023-04-20 14:15:09 +08:00
Thales Macedo Garitezi
cb995e2033 fix(buffer_worker): avoid sending late reply messages to callers 
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
 2023-04-19 18:27:10 -03:00
Thales Macedo Garitezi
a9bd91fcff refactor(gcp_pubsub): move GCP PubSub Bridge to its own app 
Fixes https://emqx.atlassian.net/browse/EMQX-9536

Note: since GCP PubSub is not shared by any authn/authz backend,
there's no need to separate its connector into another app.
 2023-04-19 13:24:32 -03:00
Andrew Mayorov
f06300cbed
fix(s3): mark S3 secrets as sensitive in schema 2023-04-19 19:05:12 +03:00
Andrew Mayorov
04523b3f81
fix(ft): restrict max filename length in transfers 
Reject transfers with too long filenames right away, during `init`
handling, to avoid having to deal with filesystem errors later.
 2023-04-19 18:48:40 +03:00
lafirest
8ccfbe9e16
Merge pull request #10448 from lafirest/fix/limiter_compatibility 
fix(limiter): fix compatibility problem of configuration
 2023-04-19 23:19:19 +08:00
JianBo He
eda2f0819d
chore: apply suggestions from code review 2023-04-19 21:22:01 +08:00
JianBo He
bd935e34e6
chore: update apps/emqx_modules/README.md 2023-04-19 21:12:24 +08:00
JianBo He
13f50b2ba9 chore: update README files 2023-04-19 21:05:10 +08:00
Thales Macedo Garitezi
cb3ef65775
Merge pull request #10439 from thalesmg/unhide-ocsp-config-v50 
chore: un-hide ocsp stapling config
 2023-04-19 09:21:10 -03:00
firest
4f0c891aa6 chore: bump version && update changes 2023-04-19 15:45:31 +08:00
firest
5455500647 fix(limiter): fix compatibility problem of configuration 2023-04-19 15:12:55 +08:00
lafirest
4591020ac3
Merge pull request #10435 from lafirest/fix/slow_subs_readme 
chore: add README for slow subscriptions
 2023-04-19 10:06:03 +08:00
JianBo He
d947b66327
chore: update apps/emqx_slow_subs/README.md 2023-04-19 09:51:19 +08:00
JianBo He
628a5e6eaf
Merge pull request #10402 from HJianBo/prepared-bridges-apps 
chore: create dirs and README template for all ee bridges
 2023-04-19 09:37:53 +08:00
Thales Macedo Garitezi
6a1ef5e68a
Merge pull request #10422 from thalesmg/fix-plugin-sync-single-node-v50 
fix(plugins): attempt to extract plugin from current node on startup
 2023-04-18 14:39:02 -03:00
Thales Macedo Garitezi
199cbc60d9 chore: un-hide ocsp stapling config 
Undoing https://github.com/emqx/emqx/pull/10160
 2023-04-18 14:38:03 -03:00
Zaiming (Stone) Shi
f361870ca7 refactor: hide sysmon.top config 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
12e549a3a2 refactor: change exhook config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
be70c7d385 refactor: change rule_engine config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
30fd9b10f7 refactor: change bridges config importance level from hidden to low 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
d51cc750de refactor: hide psk user_lookup_fun 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
4d67312bab refactor: set authz config at 'high' importance level 
and authorization.sources at 'low' level
prior to this commit, the root was set to 'hidden'
which is not ideal because some may still want to configure
the sources from files
 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
6bc33e86be refactor: set rpc config at 'low' importance level instead of 'hidden' 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
a30e08d06a refactor: set authn config at 'low' prio instead of 'hidden' 2023-04-18 19:17:03 +02:00
Zaiming (Stone) Shi
9efbb71ae8
Merge pull request #10441 from zmstone/0418-fix-flakyness 
test: pick random port number for gcp pubsub mock server
 2023-04-18 19:11:30 +02:00
Ivan Dyachkov
c858fba9d4 chore: v5.0.23 2023-04-18 17:34:28 +02:00
Ivan Dyachkov
dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
JianBo He
7de6c5b392 chore: update prometheus README 2023-04-18 22:47:26 +08:00
JianBo He
acea64790b chore: add README for emqx_modoules 2023-04-18 22:47:26 +08:00
JianBo He
46bbcf93f4
Merge pull request #10426 from zhongwencool/conf-changelog 
chore: update config's changelog and emqx_conf.template
 2023-04-18 22:16:45 +08:00
Thales Macedo Garitezi
89cd6cfede fix(plugins): attempt to extract plugin from current node on startup 
Fixes https://emqx.atlassian.net/browse/EMQX-9605

Fixes https://github.com/emqx/emqx-elixir-plugin/issues/25

If an user happens to configure a plugin in a lone-node cluster via
environment variables, it would fail to start up as there are no other
nodes to copy the plugin from.  Here, we attempt to check if the
package is present in the current node but not yet extracted.
 2023-04-18 11:10:23 -03:00
Andrew Mayorov
75813683be
Merge pull request #10371 from ft/EMQX-9257/resource-pool-switch 
feat(respool): switch to `emqx_resource_pool`
 2023-04-18 16:58:17 +03:00
Zaiming (Stone) Shi
97b8f00508 test: pick random port number for gcp pubsub mock server 2023-04-18 15:07:57 +02:00
zhongwencool
59182ee0fc
chore: update apps/emqx_conf/etc/emqx_conf.conf 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-18 21:07:16 +08:00
zhongwencool
9ca09383ba
chore: remove desc from emqx.conf 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-18 21:07:02 +08:00
某文
444196922c chore: update emqx.conf's note 2023-04-18 20:18:46 +08:00
某文
fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
Zhongwen Deng
22a1d05d7b feat: hide ssl_options.user_lookup_fun 2023-04-18 20:13:44 +08:00
Zhongwen Deng
e3d6fa1f21 chore: update config's changelog and emqx_conf.template 2023-04-18 20:13:44 +08:00
Zaiming (Stone) Shi
d0a7e7c406
Merge pull request #10421 from zmstone/0417-refine-authn-schema-namespace 
0417 refine authn schema namespace
 2023-04-18 14:04:20 +02:00
ieQu1
c0d8e9c402
Merge pull request #10369 from ieQu1/fix-stats-api 
fix(emqx_management): Ignore results from the nodes that are down
 2023-04-18 12:36:55 +02:00
Andrew Mayorov
21e19a33ce
feat(respool): switch to emqx_resource_pool 
Which was previously known as `emqx_plugin_libs_pool`. This is part
of the effort to get rid of `emqx_plugin_libs` application.
 2023-04-18 12:51:14 +03:00
zhongwencool
4240720753
Merge pull request #10431 from zhongwencool/delete-trace-limiter-api 
feat: delete trace/limiter config's hot update api
 2023-04-18 17:40:39 +08:00
firest
69e334a77d chore: reorganize the README to follow template 2023-04-18 17:25:11 +08:00
firest
f3614b2c65 chore: add README for slow subscriptions 2023-04-18 17:13:06 +08:00
ieQu1
a6688ed07a refactor(emqx_management): Use emqx:running_node function 2023-04-18 10:24:39 +02:00
Zhongwen Deng
d0f30d2f7c feat: delete trace/limiter config's hot update api 2023-04-18 16:13:17 +08:00
Kjell Winblad
29584ca721 feat: add mongo_date functions to the rule engine 
This commit adds mong_date built-in functions to the rule engine
SQL-like language. Corresponding functions already existed in EMQX 4.4
and this commit makes sure that EMQX 5.X also has these functions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9244
 2023-04-18 10:01:35 +02:00
Zaiming (Stone) Shi
cd1197925f test: fix unit tests after type names changed 2023-04-18 09:48:28 +02:00
Zaiming (Stone) Shi
6dd7befaab refactor: unify authn authz type names 2023-04-18 09:48:28 +02:00
firest
721125a9f6 fix(retainer): mark flow-control as non-importance field 2023-04-18 15:34:42 +08:00
ieQu1
282bfee8ff feat(emqx): Add an API that returns the list of running nodes 2023-04-18 09:22:58 +02:00
ieQu1
a947df1ea3 fix(emqx_management): Ignore results from the nodes that are down 2023-04-18 09:22:56 +02:00
JianBo He
c53ccfea61 chore: fix typos 2023-04-18 13:48:55 +08:00
JianBo He
8cb9389d21 chore: apply suggestions from code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-18 09:36:42 +08:00
JianBo He
ec8d8b805f chore: add ee bridge apps 2023-04-18 09:36:41 +08:00
JianBo He
7eacbffae9 chore: create dirs and README template for all ee bridges 2023-04-18 09:36:41 +08:00
zhongwencool
ad3e529994 chore: update changes/ce/feat-10389.en.md 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-18 09:34:01 +08:00
某文
933e6727ba feat: node array support array(atom()) and comma_separated_atoms 2023-04-18 09:28:36 +08:00
Ilya Averyanov
88ca94b417 fix(auth): fix uri path handling 
Fix uri path handling `emqx_connector_http`,
 HTTP authentication and authorization backends.
 2023-04-17 23:57:56 +03:00
Kjell Winblad
0c727fc9b2
Merge pull request #10392 from kjellwinblad/kjell/rule_engine/add_missing_datetime_function/EMQX-9245 
freat: add date_to_unix_ts/3 function to the rule engine
 2023-04-17 18:09:44 +02:00
Ivan Dyachkov
c03a94bae9 chore: e5.0.3-alpha.1 2023-04-17 16:48:07 +02:00
Zaiming (Stone) Shi
466a28daf2 test: fix test cases to work with new exctption 2023-04-17 13:58:14 +02:00
Zaiming (Stone) Shi
373e7b33f9 test: allow tests to run without desc cache 2023-04-17 13:58:14 +02:00
Zaiming (Stone) Shi
6969c2a670 refactor: not leagal -> invalid 2023-04-17 13:58:14 +02:00
Zaiming (Stone) Shi
1aa5b528e9 feat: generate hotconf and bridge schema on the fly 2023-04-17 13:58:14 +02:00
Zaiming (Stone) Shi
56b9667436 refactor(emqx_conf): prepare for dynamic api schema generation 2023-04-17 13:58:14 +02:00
Zaiming (Stone) Shi
18974a8e11 refactor: make schema dump and swagger spec work with split desc files 2023-04-17 13:58:11 +02:00
JianBo He
f2fae16d3b fix(gw): load emqx applications before hocon configs checking 2023-04-17 13:56:39 +02:00
Ivan Dyachkov
9712aad7a8
Merge pull request #10413 from id/0416-e5.0.3-code-freeze 
0416 e5.0.3 code freeze
 2023-04-17 13:51:58 +02:00
zhongwencool
35c36e8872
Merge pull request #10415 from zhongwencool/authz-rule-and-or-improve 
chore: short-circuit expressions on authz‘s and/or rules.
 2023-04-17 18:21:04 +08:00
Kjell Winblad
37f42a486c test: improve proper test so it generates more cases 2023-04-17 12:03:13 +02:00
JianBo He
19eda3bc5f
Merge pull request #10410 from HJianBo/fix-load-gw-from-confs 
fix(gw): load emqx applications before hocon configs checking
 2023-04-17 18:01:31 +08:00
某文
e9e0ae7f0a chore: When matching authz's and/or rules, check the simple ones first to improve efficiency 2023-04-17 17:23:39 +08:00
Ivan Dyachkov
9fc8a498f8 chore: bump apps versions 2023-04-17 09:09:08 +02:00
firest
02f8d073f8 test(limiter): fix test errors and make spellcheck happy 2023-04-17 10:06:43 +08:00
firest
55376144ce fix(limiter): simplify the configuration of the limiter 2023-04-17 10:06:36 +08:00
zhongwencool
69d1a35c90
Merge pull request #10156 from zhongwencool/conf-refactor 
feat: configuration priority ENV > emqx.conf > API
 2023-04-16 13:45:35 +08:00
zhongwencool
bcbeae1da5
Merge pull request #10405 from zhongwencool/hide-dashboard-conf 
feat: hide dashboard's default_username/default_password conf
 2023-04-16 13:35:34 +08:00
Zhongwen Deng
98e8287260 fix: hidden password in dashboard https's shema 2023-04-16 13:30:12 +08:00
Zhongwen Deng
ad65fefac2 fix: rename emqx_misc to emqx_utils 2023-04-16 10:27:23 +08:00
Zhongwen Deng
782649020b Merge remote-tracking branch 'upstream/master' into deprecated-stats-conf 2023-04-16 10:17:38 +08:00
zhongwencool
3510ac63b9
Merge pull request #10406 from zhongwencool/hide-node 
feat: hide node advance config
 2023-04-16 09:57:42 +08:00
JianBo He
3a511c6229 fix(gw): load emqx applications before hocon configs checking 2023-04-15 19:41:07 +08:00
Zhongwen Deng
8461551f51 test: rename module from emqx_map_lib to emqx_utils_maps 2023-04-15 06:56:33 +08:00
zhongwencool
a00daa4d97
Merge branch 'master' into deprecated-stats-conf 2023-04-15 06:47:28 +08:00
zhongwencool
494ffe86ad
Merge branch 'master' into conf-refactor 2023-04-15 06:40:38 +08:00
Stefan Strigler
7df0493312
Merge pull request #10390 from sstrigler/EMQX-9549-new-emqx-utils-app-to-collect-utility-modules 
New emqx_utils app to collect utility modules
 2023-04-14 20:33:11 +02:00
Thales Macedo Garitezi
0e3a6d7f22
Merge pull request #10404 from thalesmg/buffer-mem-only-v50 
feat(buffer_worker): set default queue mode to `memory_only`
 2023-04-14 15:18:12 -03:00
Stefan Strigler
17d84fb5e0 Merge branch 'EMQX-9549-new-emqx-utils-app-to-collect-utility-modules' of github.com:sstrigler/emqx into EMQX-9549-new-emqx-utils-app-to-collect-utility-modules 2023-04-14 18:53:47 +02:00
Stefan Strigler
9ccfa643ce
style: fix wording 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-14 18:51:51 +02:00
Stefan Strigler
a20797160e style: remove unnecessary ifdef(TEST) 2023-04-14 18:45:25 +02:00
Zhongwen Deng
97a936ad4f test: fix failed test 2023-04-14 23:46:05 +08:00
Zhongwen Deng
8948be49e7 feat: hide node advance config 2023-04-14 23:29:41 +08:00
Stefan Strigler
16c49b2cc1 fix: undo wrong thinking about returned types from decode 2023-04-14 17:21:38 +02:00
Zhongwen Deng
8facd130f6 chore: update cluster_hocon_file/0 function 2023-04-14 23:19:38 +08:00
Stefan Strigler
24df1045de fix: test not updated after rebase 2023-04-14 16:40:25 +02:00
Thales Macedo Garitezi
e073bc90bc refactor(buffer_worker): rename s/queue/buffer/g 2023-04-14 11:37:19 -03:00
Thales Macedo Garitezi
14ed4a7ada feat(buffer_worker): set default queue mode to memory_only 
Fixes https://emqx.atlassian.net/browse/EMQX-9367

For better user experience and performance for the average bridge, we
should change the default queue mode to `memory_only`, as was the
behavior of most bridges in e4.x.  This leads to better performance
when message rate is high enough and the remote resource is not
keeping up with EMQX.

Also, we set the default segment size to equal max queue bytes.
 2023-04-14 11:37:19 -03:00
zhongwencool
7c5dead03a
chore: update cluster_hocon_file doc 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-14 22:30:35 +08:00
Stefan Strigler
53871e3a2c fix: stale ref to emqx_json after rebase 2023-04-14 16:30:27 +02:00
zhongwencool
bcce989906
chore: update has_deprecated_file doc 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-14 22:30:11 +08:00
Stefan Strigler
a9976287b5 fix: force proper calling decode to return proplist 2023-04-14 16:26:32 +02:00
Zhongwen Deng
99d6c5e179 feat: hide dashboard's default_username/default_password conf 2023-04-14 21:57:09 +08:00
Zhongwen Deng
3789ca2622 chore: revert stats desc 2023-04-14 21:31:34 +08:00
Zhongwen Deng
7934a1cea1 feat: hide overload_protection,conn_congestion,flapping_detect 2023-04-14 21:29:27 +08:00
某文
e5b85916b6 feat: hidden stats config 2023-04-14 21:29:27 +08:00
Zhongwen Deng
c0e6e79bcd feat: don't remove default value when save config 2023-04-14 21:17:34 +08:00
Zhongwen Deng
c21744c260 fix: hocon_pp crash when atom_key 2023-04-14 20:51:58 +08:00
Zhongwen Deng
fa753cf333 fix: failed ct test 2023-04-14 20:51:58 +08:00
Zhongwen Deng
7f870257b2 test: fix failed ct 2023-04-14 20:51:58 +08:00
Zhongwen Deng
a271ba5ff9 feat: don't check_permission on local.conf 2023-04-14 20:51:58 +08:00
Zhongwen Deng
5f4ea3b6d8 feat: deprecated cluster-override.conf 2023-04-14 20:51:58 +08:00
Zhongwen Deng
180f571765 feat: configuration priority ENV > emqx.conf > API 2023-04-14 20:51:58 +08:00
zhongwencool
57e8e2dae2
Merge pull request #10401 from zhongwencool/hide-conf-04-14 
feat: hide shared_subscription_group,rpc,slow_subs
 2023-04-14 20:36:10 +08:00
Thales Macedo Garitezi
4de13d2800 feat(buffer_worker): change default max queue bytes to 256 MB 2023-04-14 09:31:33 -03:00
Stefan Strigler
a295d0f134 fix: add rebar3_path_deps plugin 2023-04-14 14:29:25 +02:00
Stefan Strigler
1bad6ca67d fix: jiffy pretty print is a bit different from jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler
92ca2f66f5 style: add original copyright header 2023-04-14 13:41:34 +02:00
Stefan Strigler
0f162fb50a test: add tests for emqx_utils_binary from original site 2023-04-14 13:41:34 +02:00
Stefan Strigler
badf962800 fix: stale call to emqx_misc 2023-04-14 13:41:34 +02:00
Stefan Strigler
d98f7222ff style: add comment to binary_string/1 2023-04-14 13:41:34 +02:00
Stefan Strigler
90520a5382 docs: add an actual README 2023-04-14 13:41:34 +02:00
Stefan Strigler
4f80690162 fix: byebye jsx 2023-04-14 13:41:34 +02:00
Stefan Strigler
19981757ae fix: add is_json 2023-04-14 13:41:34 +02:00
Stefan Strigler
062ce5f819 refactor: rename emqx_map_lib to emqx_utils_maps 2023-04-14 13:41:34 +02:00
Stefan Strigler
6e8665365b refactor: rename emqx_tables to emqx_utils_ets 2023-04-14 13:41:33 +02:00
Stefan Strigler
d0df086c80 refactor: rename emqx_api_lib to emqx_utils_api 2023-04-14 13:41:33 +02:00
Stefan Strigler
9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler
1880da0a2e refactor: move binary_util to emqx_utils_binary 2023-04-14 13:35:15 +02:00
Stefan Strigler
f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
JianBo He
a0cfac3a1a
Merge pull request #10386 from HJianBo/refactor-directoires 
refactor(gw): rename all gateway application name
 2023-04-14 19:26:53 +08:00
Kjell Winblad
a66d01d6f0
style: remove code duplication 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-14 12:48:03 +02:00
JimMoen
790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge 
feat: implement Microsoft SQL Server bridge (e5.0)
 2023-04-14 17:55:48 +08:00
Andrew Mayorov
5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics 
feat(resource): ask for metrics only when needed
 2023-04-14 12:28:52 +03:00
Zhongwen Deng
be35ae2132 feat: hide shared_subscription_group,rpc,slow_subs 2023-04-14 17:14:11 +08:00
zhongwencool
1e07f37ab0
Merge pull request #10354 from zhongwencool/straightforward-log-info 
feat: more straightforward log for force_shutdown reason
 2023-04-14 10:15:32 +08:00
JimMoen
5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
JimMoen
8ceeafb0de
chore: fix file license in lib-ee 2023-04-14 09:56:59 +08:00
zhongwencool
0553facfca
Merge pull request #10391 from zhongwencool/04-13-hide-conf 
feat: hide ex_hook/rewrite/topic_metric/persistent_session_store
 2023-04-14 09:39:29 +08:00
zhongwencool
626e814e9a
Merge pull request #10385 from zhongwencool/hide-conf 
feat: hide data items from configuration files and documentation.
 2023-04-14 09:38:34 +08:00
zhongwencool
835f66a8d2
Merge pull request #10375 from zhongwencool/trace-conf 
feat: deprecated trace config
 2023-04-14 09:22:12 +08:00
Ilya Averyanov
92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
ieQu1
c5ad3c8f41 docs(emqx_machine): Improve README 2023-04-13 22:57:30 +02:00
Ilya Averyanov
e22c1c01ec feat(s3): make acl optional 2023-04-13 19:47:12 +03:00
JianBo He
9f99a72594 chore: refine tests 2023-04-14 00:45:22 +08:00
Thales Macedo Garitezi
9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50 
refactor(kafka_bridge): move kafka bridge into its own app
 2023-04-13 13:26:36 -03:00
JianBo He
1759dd1157
Merge pull request #10368 from HJianBo/improve-gateway-readme 
docs: update gateway README
 2023-04-14 00:14:01 +08:00
Kjell Winblad
d3ccd8a65d feat: add date_to_unix_ts/3 function to the rule engine 
Fixes:
https://emqx.atlassian.net/browse/EMQX-9245
 2023-04-13 16:14:03 +02:00
zhongwencool
2334917e35
chore: apply suggestions from code review 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-13 21:59:26 +08:00
某文
58e31d5efd feat: hide ex_hook/rewrite/topic_metric/persistent_session_store/overload_protection 2023-04-13 21:52:35 +08:00
某文
c1163d1952 chore: share emqx_trace.hrl between code and test 2023-04-13 20:51:14 +08:00
某文
dd7dcfe373 fix: fix dialyzer warning 2023-04-13 20:04:03 +08:00
JianBo He
c7054886b1 chore: refine test cases 2023-04-13 17:50:58 +08:00
JianBo He
4dd0080e82 chore: fix unreachable links 2023-04-13 17:19:47 +08:00
JianBo He
40e4419a62
chore: update apps/emqx_gateway/README.md 
Co-authored-by: Stefan Strigler <stefan@strigler.de>
 2023-04-13 16:18:46 +08:00
JianBo He
aa9292e37e
chore: update apps/emqx_gateway/README.md 
Co-authored-by: Stefan Strigler <stefan@strigler.de>
 2023-04-13 16:18:34 +08:00
JianBo He
0a62d6c556 refactor(gw): rename all gateway application name 2023-04-13 14:53:47 +08:00
zhongwencool
cf2f00c937
Merge pull request #10381 from zhongwencool/hiden-auto-subscribe-conf 
feat: hiden auto_subscribe conf
 2023-04-13 14:42:59 +08:00
Zhongwen Deng
cc2beda37a feat: hiden rule_engine/bridge/authz/authn from doc/example 2023-04-13 14:42:21 +08:00
Zhongwen Deng
4ba4c6bf62 chore: only hiden root keys 2023-04-13 14:23:25 +08:00
Zhongwen Deng
a70930fed0 feat: hiden auto_subscribe conf 2023-04-13 09:26:16 +08:00
Thales Macedo Garitezi
871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app 
Fixes https://emqx.atlassian.net/browse/EMQX-9481
 2023-04-12 13:54:45 -03:00
Ivan Dyachkov
1e33600190 chore: bump version to v5.0.22 2023-04-12 17:33:53 +02:00
Ivan Dyachkov
f01e2f358b
Merge pull request #10367 from id/0411-sync-release-50-back-to-master 
0411 sync release 50 back to master
 2023-04-12 17:23:17 +02:00
Ivan Dyachkov
bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Andrew Mayorov
9c9f39d0f7
feat(resman): also move out metrics collection for debugging 
Now `emqx_resource:list_instances_verbose/0` will populate the metrics
for each instance, for the sake of simplicity.
 2023-04-12 16:14:42 +03:00
JianBo He
d16b5c40d6
Merge pull request #10356 from HJianBo/async-batch-cassa 
feat: support async and batch callback for cassandra connector
 2023-04-12 20:59:36 +08:00
JianBo He
9774ad2cdd
chore: update apps/emqx_gateway/README.md 2023-04-12 20:54:08 +08:00
Zhongwen Deng
45254c5936 chore: add changelog for create trace 2023-04-12 17:59:14 +08:00
Stefan Strigler
f668ad7b9e
Merge pull request #10336 from sstrigler/EMQX-8507-rule-engine-need-a-new-api-to-crud-rule-engines-setting-configs 
feat: add `/rule_engine` API endpoint
 2023-04-12 11:44:42 +02:00
Zhongwen Deng
f161399bcc feat: deprecated trace config 2023-04-12 17:07:45 +08:00
lafirest
a49eea5838
Merge pull request #10355 from lafirest/fix/auto_sub_doc 
chore: update README for auto subscribe
 2023-04-12 15:47:58 +08:00
JianBo He
80f964bdbe
chore: update apps/emqx_gateway/README.md 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-12 10:44:37 +08:00
Thales Macedo Garitezi
914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50 
feat: implement schema registry for 5.0 (avro)
 2023-04-11 16:46:27 -03:00
Stefan Strigler
e6f8682c47 fix: ensure we don't return 'rules' in rule_engine 2023-04-11 15:42:56 +02:00
ieQu1
bc463a1b1c
Merge pull request #10343 from ieQu1/emqx-machine-readme 
doc(emqx_machine): Add readme
 2023-04-11 15:15:32 +02:00
Stefan Strigler
8ef36f29ce fix: add 'rule_engine' as possible tag() value for spec 2023-04-11 12:03:42 +02:00
Stefan Strigler
b48fb17f4a fix: CHECK_PARAMS macro defines unused var 2023-04-11 12:03:42 +02:00
JianBo He
945c6bc757 docs: update gateway README 2023-04-11 17:26:33 +08:00
Andrew Mayorov
e70deae1c3
feat(resource): ask for metrics only when needed 2023-04-11 12:00:19 +03:00
JianBo He
aee85dc328 chore: fix flaky tests 2023-04-11 14:49:07 +08:00
JianBo He
d501c1ee9c chore: update changes 2023-04-11 14:04:41 +08:00
Ivan Dyachkov
a711ce6b93 chore: bump version to e5.0.2 2023-04-10 18:10:31 +02:00
Andrew Mayorov
37a520d797
fix(i18n): relocate i18n files to rel/i18n 2023-04-10 16:02:34 +03:00
firest
5a58dfc3a4 chore: update README for auto subscribe 2023-04-10 14:39:13 +08:00
Zhongwen Deng
c68909767a feat: more straightforward log for force_shutdown reason 2023-04-10 11:21:07 +08:00
ieQu1
c2ca9089ca docs(emqx_machine): Add readme 2023-04-08 13:22:52 +02:00
JianBo He
e186477531
Merge pull request #10278 from HJianBo/refactor-gw-dir 
Refactor gateway application dirs
 2023-04-08 09:42:14 +08:00
Andrew Mayorov
8daa38ef06
feat(ft-s3): store metadata in ASCII-safe format 
Also ensure consistent encoding and decoding filenames throughout
the `emqx_ft` application.
 2023-04-07 22:20:20 +03:00
Andrew Mayorov
0d86ef8d3a
fix(ft-s3): use AWS4 signed urls for S3 export URIs 
This will ensure that the S3 export URIs are valid in all AWS regions.
 2023-04-07 22:15:01 +03:00
Ilya Averyanov
b2fd2dcbc1 fix(ft): make configs user friendlier 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7eeba32619 fix(s3): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
52f3189779 fix(ft): fix default ft config 2023-04-07 17:36:47 +03:00
Ilya Averyanov
eae2478678 fix(ft): add descriptions for missing ft schema fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
bd7250cb13 fix(s3): fix hash pool type 2023-04-07 17:36:47 +03:00
Ilya Averyanov
4a144044b7 fix(ft): set default ft config in tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
918bdcae7d fix(ft): fix schema descriptions 2023-04-07 17:36:47 +03:00
Ilya Averyanov
8b13d249eb fix(ft): fix detection of ee apps in mix 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7bcb60a84a fix(ft): fix config update tests 2023-04-07 17:36:47 +03:00
Ilya Averyanov
6e50b168a0 fix(ft): add empty strings for zh translations 2023-04-07 17:36:47 +03:00
Ilya Averyanov
f6461fe287 fix(ft): fix typings 2023-04-07 17:36:47 +03:00
Ilya Averyanov
2f1970adbc fix(ft): make ee schema aware of ee fields 2023-04-07 17:36:47 +03:00
Ilya Averyanov
18aa39f892 fix(ft): add emqx_ft_schema to ee schemas 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d333187c5a docs(s3): update readme 2023-04-07 17:36:47 +03:00
Ilya Averyanov
820e06d756 fix(ft-s3): make controllable use of max_retries and request_timeout arguments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
be99242e32 fix(ft-s3): fix review comments 2023-04-07 17:36:47 +03:00
Ilya Averyanov
d7a85242de fix(ft-s3): fix s3 listing 2023-04-07 17:36:47 +03:00
Ilya Averyanov
7d13862da5 fix(ft-s3): use uploader specific headers only for object creation 2023-04-07 17:36:47 +03:00
Ilya Averyanov
c7865e5eae fix(ft-s3): fix dependency dependencies 2023-04-07 17:36:45 +03:00
Ilya Averyanov
4fbabe5a76 fix(ft-s3): fix atom and variable naming 2023-04-07 17:27:01 +03:00
Ilya Averyanov
07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov
3ffa01e160 feat(ft-s3): fix logging level 2023-04-07 17:25:23 +03:00
Ilya Averyanov
ca22f11161 feat(ft-s3): fix logging 2023-04-07 17:25:23 +03:00
Ilya Averyanov
9bb72ee020 feat(ft-s3): fix behaviour specification 2023-04-07 17:25:23 +03:00
Ilya Averyanov
43f9737420 feat(ft-s3): integrate list API 2023-04-07 17:25:23 +03:00
Ilya Averyanov
5ac3543a76 feat(ft-s3): integrate exporter configs 2023-04-07 17:25:22 +03:00
Ilya Averyanov
8361223648 feat(ft-s3): extract checksum verification 2023-04-07 17:25:22 +03:00
Ilya Averyanov
818a5cacf2 feat(ft-s3): add initial integration 2023-04-07 17:25:22 +03:00
Ilya Averyanov
31b441a46e feat(s3): add S3 client application 2023-04-07 17:25:22 +03:00
Andrew Mayorov
28d87ca62d fix(ft-fs): move default config to the backend impl 2023-04-07 17:25:22 +03:00
Andrew Mayorov
258fabbf8b fix(fs-exp): reply with error when listing failed everywhere 2023-04-07 17:25:22 +03:00
Andrew Mayorov
11edfc1c6a fix(ft): comment out some clauses as "unreachable" 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bef5cc9c0f fix(fs-fold): avoid folding through symlinked directories 
Also a testsuite that verifies multilevel fold behaviour.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
6ad7ce55d2 fix(ft-conf): separate local segments storage settings 
To make things less ambiguous.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45e3b62dc4 refactor(ft): prefer plain files over exports in generic APIs 
So there's no more `exports` magically becomings `files` in the
REST API which was slightly confusing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
c24c7eca34 fix(ft-test): unbreak testcase by inhibiting local fs storage GC 2023-04-07 17:25:22 +03:00
Andrew Mayorov
23cd78b8d6 refactor(ft-asm): turn state data record into a map 
Which should be more future-proof.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b6b044f429 feat(ft): move out exporter concept into dedicated modules 2023-04-07 17:25:22 +03:00
Andrew Mayorov
54e54cc63d feat(api): provide utility to qualify API URIs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
64f15f1fdb fix(ft-gc): ensure directories of complete transfers are GCed 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2707b4500f feat(ft-fs): ensure filsystem-safe handling of client input 
Also restrict the filenames clients may specify in a filemeta to some
safe subset, as a future proofing measure.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
9aec01e7a3 fix(ft-asm): use regular map for meta fragments 
There's actually no need to use an ordered data structure at all.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4132f5a5fb feat(ft): introduce exporter concept in local storage backend 
The exporter is responsible for keeping fully transferred and
successfully assembled files. This was on the local storage itself
before. This abstraction is needed to give us an ability to
support S3 destinations more easily, just by swapping the storage
exporter.

Also implement local filesystem exporter and reimplement parts of
the `emqx_ft` API on top of it.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2880c8f4eb fix(ft): unwrap error details when reader fails to start 2023-04-07 17:25:22 +03:00
Andrew Mayorov
45f00e14a9 feat(ft): ensure that clientid is always binary 
For the sake of simplicity (e.g. transfer ids are now easier to
compare).
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
4f2600b9f1 feat(ft-gc): treat all transfer as incomplete 
Since the concept of _complete transfers_ is being split out into
the _export_ concept, we lose knowledge of completeness in the GC.
Instead of asking exporters for transfer statuses we just treat all
transfer as incomplete when GCing.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0d39546080 feat(wdgraph): add fold/3 which folds over graph edges 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c821cd3bd test(ft): inject configs through hocon subsystem 
So that relevant parts of config would be initialized with defaults.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e1dc48fa2b feat(fs-gc): wire gc up with emqx config 2023-04-07 17:25:22 +03:00
Andrew Mayorov
50c6eef2bc fix(fs-gc): do not hide enoents 
Also use `file:read_link_info/2`, it actually fetches any file info
while also not following symlinks automatically, which is better for
GC usecases.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
bcd2099ce1 fix(fs-gc): make deletion empty transfer directories safer 2023-04-07 17:25:22 +03:00
Andrew Mayorov
344799f100 fix(fs-gc): add testcase on incomplete transfers GC 
And fix logic that made GC delete incomplete segments prematurely.
 2023-04-07 17:25:22 +03:00
Ilya Averyanov
15dc7c3e84 fix(test): fix node helper usage 2023-04-07 17:25:22 +03:00
Andrew Mayorov
715816e67b feat(ft): add GC logic and process for the FS storage backend 2023-04-07 17:25:22 +03:00
Andrew Mayorov
e9f98adca2 test: make content_gen emit meta with each chunk 
Which will also contain total content size, to increase this tool's
utility.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
788e76ed2d test: make assembly proptests waste less memory 2023-04-07 17:25:22 +03:00
Andrew Mayorov
8e6f960c09 refactor(ft): employ emqx_wdgraph for coverage computation 
Also describe how coverage problem maps to shortest path problem.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
b189ee463c feat: add weighted directional graph ADT with shortest path 
Basically, separate what abstraction was in `emqx_ft_assembly` into
dedicated module with a compact interface and a basic testsuite.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
130601376a perf(ft-asm): express coverage through shortest path on graph 
Coverage becomes the shortest path problem on graph where nodes are
offsets and edges are nodes' segments. Implement a simple Dijkstra
algorithm to find one.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0c84fc28b0 perf(asm-ft): tradeoff optimality for computational complexity 
Through squashing segments table into consecutive "runs".
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
97cfdf8eef test(ft-asm): add property tests for file assembly 2023-04-07 17:25:22 +03:00
Andrew Mayorov
0af7e2a002 perf(ft-asm): optimize away remote segments identical to local 2023-04-07 17:25:22 +03:00
Andrew Mayorov
15d967459c feat(ft): add segment checksum validation 
Also downgrade validation errors to mere info messages.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5998961f9f fix(ft): log errors where they might get lost 2023-04-07 17:25:22 +03:00
Andrew Mayorov
5014cc15f4 test(ft): do not rely on fs backend implementation details 2023-04-07 17:25:22 +03:00
Andrew Mayorov
93865a79e9 fix(ft): disallow empty fileids 2023-04-07 17:25:22 +03:00
Andrew Mayorov
75070102ec fix(ft): improve typespecs 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2b925aa60b fix(ft): drop unrelated TODO 2023-04-07 17:25:22 +03:00
Andrew Mayorov
7ff253e80d test(ft): attempt to avoid gen_rpc config reset hacks 2023-04-07 17:25:22 +03:00
Andrew Mayorov
a3f47641f3 test(ft): describe the complex test case with comments 
In order to make the test client model and behavior clearer.
 2023-04-07 17:25:22 +03:00
Andrew Mayorov
10d4c4305a feat(maybe): add basic tests for the new module 2023-04-07 17:25:22 +03:00
Andrew Mayorov
2cf2a2d952 fix(ft): make fin packet parser stricter + safer 2023-04-07 17:25:22 +03:00