Commit Graph

266 Commits

Author SHA1 Message Date
JianBo He dd9938114c chore(audit): add from field 2023-09-28 10:25:37 +08:00
Zaiming (Stone) Shi 34367fc4ec fix(audit_log): pretty print shell args 2023-09-27 23:19:39 +02:00
Thales Macedo Garitezi 0498e59c45 test: fix flaky test 2023-09-25 13:35:17 -03:00
Thales Macedo Garitezi 806017ef90 chore: check ekka proto dist module type when resolving node address
Follow up to https://github.com/emqx/emqx/pull/11637#discussion_r1334462917

Fixes https://emqx.atlassian.net/browse/EMQX-10944
2023-09-25 10:31:50 -03:00
Thales Macedo Garitezi d6935b6a67 feat: add port scan diagnostics to mria waiting for tables checks
Fixes https://emqx.atlassian.net/browse/EMQX-10944

Also updates ekka -> 0.15.15, mria -> 0.6.4

How to test
===========

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop them in order.
3. Start only the first node that was stopped in the previous step.
4. Wait until the log is printed.

Or, more easily:

1. Start 2 or more EMQX nodes and merge them in a cluster.
2. Stop all but one.
3. Run `mria_mnesia:diagnosis([]).` on that node.

Example output
==============

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status =>
                                                                 bad_ports},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```

After one node is back:

```
   Check check_open_ports should get ok but got #{msg =>
                                                     "some ports are unreachable",
                                                 results =>
                                                     #{'emqx@172.100.239.4' =>
                                                           #{ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   4}],
                                                             status => ok},
                                                       'emqx@172.100.239.5' =>
                                                           #{open_ports =>
                                                                 #{4370 => false,
                                                                   5370 =>
                                                                       false},
                                                             ports_to_check =>
                                                                 [4370,5370],
                                                             resolved_ips =>
                                                                 [{172,100,239,
                                                                   5}],
                                                             status =>
                                                                 bad_ports}}}
```
2023-09-21 14:29:01 -03:00
zhongwencool beea1be9f0
Merge pull request #11599 from zhongwencool/audit-log 2023-09-21 18:39:23 +08:00
Ivan Dyachkov dafd7c6085 chore: bump apps versions 2023-09-21 10:58:42 +02:00
zhongwencool 21bb209fb1 feat: added support for auditing API and CLI activity in logs 2023-09-21 16:35:46 +08:00
firest 2cddce5479 feat(dashboard): add SSO feature and integrate with LDAP 2023-09-19 14:10:29 +08:00
firest 45ccc66474 fix(dashbaord): fix test cases to support RBAC 2023-09-15 15:49:37 +08:00
Zaiming (Stone) Shi 18c6bfec97 chore: bump app vsns 2023-09-06 11:12:56 +02:00
Zaiming (Stone) Shi e794143ae1 Merge remote-tracking branch 'origin/release-52' into 0906-sync-release-52-to-master 2023-09-06 09:08:22 +02:00
zhongwencool 8ab6eb5638 test: start emqx_opentelemetry in emqx_machine SUITE 2023-08-25 10:09:00 +08:00
zhongwencool c215fe3736 fix: stop otel deps appication before reboot 2023-08-24 22:14:14 +08:00
Ilya Averyanov ed67303281
Merge pull request #11382 from savonarola/0801-auth-refactor
Auth refactor stage 1
2023-08-21 00:24:15 +03:00
ieQu1 7466024bca
Merge pull request #11458 from ieQu1/fix-excluded-apps-230916
fix(emqx_machine_boot): Fix excluded_apps
2023-08-17 11:36:23 +02:00
Ilya Averyanov 128aa19d47 chore(auth): improve emqx_config_handler to replace adhoc config updates 2023-08-16 20:41:05 +03:00
ieQu1 7b44caeae4 fix(emqx_machine_boot): Fix excluded_apps 2023-08-16 16:04:35 +02:00
zhongwencool b817e03c08 fix: start os_mon application temporary 2023-08-16 20:49:27 +08:00
zhongwencool 848eb7e3c3 fix: remove os_mon application in Windows release 2023-08-16 20:49:27 +08:00
JimMoen 060074e04a
fix: api total accumulate ignore `undefined` by down nodes result 2023-08-11 11:27:55 +08:00
Thales Macedo Garitezi 65aee8870b
Merge pull request #11427 from id/0810-cut-v5.1.5-build.3
v5.1.5-build.3
2023-08-10 09:52:58 -03:00
Paulo Zulato 9ca9c65af2
Merge pull request #11367 from paulozulato/feat-gcp-devices
feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
2023-08-10 09:44:55 -03:00
Ivan Dyachkov 801c93dd46 chore: v5.1.5-build.3 2023-08-10 12:56:28 +02:00
Paulo Zulato a85c948e23 feat(gcp-iot): port GCP IoT Core compatibility layer from e4.4
Fixes https://emqx.atlassian.net/browse/EMQX-10341
2023-08-09 10:48:25 -03:00
zhongwencool 3933227636 feat: support opentelemetry metrics 2023-08-09 20:23:40 +08:00
firest efff585b82 feat(ldap-authn): add test suite for the LDAP authenticator 2023-08-03 18:56:41 +08:00
Thales Macedo Garitezi 5abe4bed88 feat: add Azure Event Hub Producer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10338
2023-07-27 09:22:39 -03:00
firest 8439ce0e84 chore: update app version && reboot_lists 2023-07-24 19:08:41 +08:00
Ivan Dyachkov 3a3b843f4a fix(machine): add emqx_bridge_kinesis to reboot_lists.eterm 2023-07-21 13:20:50 +02:00
Thales Macedo Garitezi 6cd503865b fix(machine_boot): ensure `emqx_bridge` starts after its companion apps
We need to reverse the dependency of `emqx_bridge` and `emqx_bridge_*`, because the former
loads and starts bridges during its application startup.  If the individual bridge
application being loaded has not started with its dependencies, the supervision tree will
not be ready for that.
2023-07-20 13:11:44 -03:00
Thales Macedo Garitezi 77e2d852e5 fix: add emqx_durable_storage to app list 2023-07-20 09:55:05 -03:00
Thales Macedo Garitezi 89ea40a8b7 refactor: apply review remarks 2023-07-20 09:31:13 -03:00
Thales Macedo Garitezi b9b11d8f4d fix(machine_boot): use shared list of reboot apps and add bridges to reboot list 2023-07-19 20:15:42 -03:00
Ivan Dyachkov 8ebe099653
Merge pull request #11253 from id/kjell/refactor/http_bridge/EMQX-9568
refactor: HTTP bridge into the emqx_bridge_http application
2023-07-12 15:58:02 +02:00
Thales Macedo Garitezi c94dc2e63f fix: add emqx_dashboard as a dependency of emqx_machine
```
===> Compiling emqx_machine
===> Compiling src/user_default.erl failed
src/user_default.erl:{24,14}: can't find include lib "emqx_dashboard/include/emqx_dashboard.hrl"; Make sure emqx_dashboard is in your app file's 'applications' list

==> emqx_mix
** (Mix) Could not compile dependency :emqx_machine, "/github/home/.mix/elixir/1-14/rebar3 bare compile --paths /__w/emqx/emqx/_build/emqx/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile emqx_machine", update it with "mix deps.update emqx_machine" or clean it with "mix deps.clean emqx_machine"
make: *** [Makefile:276: emqx-elixir] Error 1
```
2023-07-12 14:47:28 +02:00
Thales Macedo Garitezi 541d03a0ba refactor(schema_registry): refactor schema registry app and modules
Fixes https://emqx.atlassian.net/browse/EMQX-10361

- Moves `lib-ee/emqx_ee_schema_registry` to `apps/emqx_schema_registry`.
- Removes the `_ee_` segment from module names.
  - Exceptions are the table names which are kept to avoid backwards incompatibilities.
2023-07-12 09:24:15 -03:00
Serge Tupchii cacfb51280 fix(emqx_machine): add emqx_ee_schema_registry to the reboot apps list
As emqx_ee_schema_registry uses Mria tables (schema_registry_shard),
a node joining a cluster needs to restart this application in order to
restart relevant Mria shard processes.
2023-07-12 09:23:28 -03:00
zhongwencool 802a50601a chore: add comment for init_loader_done 2023-07-10 12:03:36 +08:00
zhongwencool c4ba558ee3 fix: unset config_loader in emqx's env when stop emqx app 2023-07-10 11:06:08 +08:00
Serge Tupchii 8e877caf47 fix(emqx_psk): wait for Mria table on app start
This ensures emqx_psk table is synced to a replicant even if it doesn't have init psk file.
2023-06-27 20:22:40 +03:00
Andrew Mayorov c8360f86b7
chore: bump application versions 2023-06-09 14:44:38 +03:00
Andrew Mayorov 830ba54721
refactor(pluglib): remove `emqx_plugin_libs` application 2023-06-09 14:44:37 +03:00
JimMoen 9526d080af
fix: override `emqx_license` is unnecessary
* continue #10820
2023-05-29 19:10:31 +08:00
JimMoen 6f90369429
fix: make dialyzer happy 2023-05-29 13:48:00 +08:00
JimMoen cb05406a00
fix: reboot `emqx_license` app for join cluster 2023-05-29 13:47:46 +08:00
Zaiming (Stone) Shi c2450d230d
Merge pull request #10605 from zmstone/0503-no-telemetry-app-for-ee
0503 refactoring: no telemetry app for ee
2023-05-27 10:57:15 +02:00
Zaiming (Stone) Shi 6e2cde8224 refactor: delete emqx_statsd all together 2023-05-26 15:17:42 +02:00
Zaiming (Stone) Shi b8b1779627 Merge remote-tracking branch 'origin/master' into 0503-no-telemetry-app-for-ee 2023-05-23 21:42:07 +02:00
Zaiming (Stone) Shi e78ffb2dd4 chore: bump apps version 2023-05-23 13:04:28 +02:00
Zaiming (Stone) Shi 3bb1f7ab2b Merge remote-tracking branch 'origin/master' into 0503-no-telemetry-app-for-ee 2023-05-23 13:03:55 +02:00
Zhongwen Deng cb5a596c57 fix: make sure swagger.json is fully generated 2023-05-22 16:58:13 +08:00
Ilya Averyanov b71955e368 fix(ft): bump application versions 2023-05-17 00:29:15 +03:00
Ilya Averyanov 8d9b785bd7 Merge branch 'release-50' into file-transfer
* release-50:
  chore(rebalance): fix app metadata
  chore(rebalance): move apps from lib-ee, add READMEs
  docs: refine zh tr
  docs: delete APL header from ee file
  docs: delete zh changelog
  chore(rebalance): review fixes
  chore(rebalance): rebase and review fixes
  feat(rebalance): port apps from 4.x
2023-05-10 11:55:23 +05:00
Ilya Averyanov 7fa166f034 Merge branch 'release-50' into file-transfer
* release-50: (73 commits)
  feat: add RabbitMQ bridge
  docs: improve rule engine labels and descriptions
  chore: bump version && update changes
  refactor(rocketmq): move rocketmq bridge into its own app
  test: dashboard_listener_test crash
  chore: bump chart versions
  chore: bump ee version to e5.0.4-alpha.1
  test: fix inter-suite flakiness
  build: compatibility to make 4.4+
  feat: add IotDB bridge
  ci: ensure git safe dir in build_packages
  ci: ensure git safe dir
  test: check_oom's max_mailbox_size
  feat: rename max_message_queue_len to max_mailbox_size
  fix(buffer_worker): fix inflight count when updating inflight item
  chore: prepare for v5.0.25-rc.1 release
  docs: add change log entry
  fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
  chore: `MQTT X` -> `MQTTX`
  chore: make sure brod_gssapi app is included in relese package
  ...
2023-05-09 23:30:22 +05:00
Ilya Averyanov 609f7bd8fd feat(rebalance): port apps from 4.x 2023-05-09 20:51:22 +05:00
Zaiming (Stone) Shi 4a2e583e3f refactor: move telemetry to its own app 2023-05-08 15:16:27 +02:00
Zhongwen Deng 14ecfce69f fix: systemd stop don't stop port in the desired order 2023-05-08 15:44:11 +08:00
Ilya Averyanov dd3471bc22 Merge branch 'master' into file-transfer
* master: (194 commits)
  fix(limiter): update change && fix deprecated version
  chore: update changes
  perf(limiter): simplify the memory represent of limiter configuration
  ci(perf test): update tf variable name and set job timeout
  ci: fix artifact name in scheduled packages workflow
  fix: build_packages_cron.yaml workflow
  ci: move scheduled builds to a separate workflow
  build: check mnesia compatibility when generating mria config
  docs: fix a typo in api doc description
  feat(./dev): use command style and added 'ctl' command
  test: fix delayed-pubish test case flakyness
  refactor: remove raw_with_default config load option
  chore: add changelog for trace timestrap
  feat: increase the time precision of trace logs to microseconds
  chore: make sure topic_metrics/rewrite's default is []
  docs: Update changes/ce/perf-10417.en.md
  chore: bump `snabbkaffe` to 1.0.8
  ci: run static checks in separate jobs
  chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN
  chore: remove unused mqtt cap 'subscription_identifiers'
  ...
2023-05-05 16:50:18 +03:00
Zaiming (Stone) Shi d89975c6ee test: add a test case for emqx_machine:node_status 2023-04-26 20:48:53 +02:00
Zaiming (Stone) Shi 51cd83e70f refactor: delete dead code 2023-04-26 20:48:53 +02:00
Ilya Averyanov ebb75b275e feat(ft): update app versions 2023-04-21 17:52:13 +03:00
Ilya Averyanov 0211bcf030 Merge branch 'master' into file-transfer
* master: (279 commits)
  chore: shorten ct/run.sh script
  chore: rename cassandra_impl to cassandra_connector
  chore: fix mix.exs checking
  refactor(cassandra): move cassandra bridge into its own app
  chore: apply review suggestions
  chore: update changes/ce/fix-10449.en.md
  test: add a test for authn {}
  chore: add changlog for authn_http validation
  fix: always check authn_http's header and ssl_option
  chore: apply suggestions from code review
  fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
  fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager
  perf(emqx_alarm): use dirty Mnesia operations to activate an alarm
  ci: simplify find-apps.sh for ee apps
  perf(emqx_resource): don't reactivate alarms on reoccurring errors
  ci: check if Elixir files are formatted in pre-commit hook
  fix(dynamo): fix field name errors
  chore: remove *_collector for prometheus api's example
  chore: make plugins config to low level
  chore: re-split dynamo i18n file
  ...
2023-04-21 17:37:17 +03:00
JianBo He c53ccfea61 chore: fix typos 2023-04-18 13:48:55 +08:00
Stefan Strigler 9c11bfce80 refactor: rename emqx_misc to emqx_utils 2023-04-14 13:41:27 +02:00
Stefan Strigler f8e9e54393 refactor: move emqx_json to emqx_utils_json 2023-04-14 13:31:27 +02:00
ieQu1 c5ad3c8f41 docs(emqx_machine): Improve README 2023-04-13 22:57:30 +02:00
ieQu1 c2ca9089ca docs(emqx_machine): Add readme 2023-04-08 13:22:52 +02:00
Ilya Averyanov 07ae50a54b fix(ft-s3): fix release settings 2023-04-07 17:25:23 +03:00
Ilya Averyanov 31b441a46e feat(s3): add S3 client application 2023-04-07 17:25:22 +03:00
Zhongwen Deng 0738444da9 fix: stop port apps when emqx_machine_terminator shutdown 2023-03-14 15:39:50 +08:00
Zaiming (Stone) Shi fe27604010 Merge remote-tracking branch 'origin/release-50' into 0308-merge-release-50-back-to-master 2023-03-08 16:46:45 +01:00
Zaiming (Stone) Shi 4eb9a69df0 feat(bin/emqx): diagnose boot failure
Prior to this change, when EMQX daemon mode failed to start
it's not quite easy for users to understand what went wrong.
All the know is the node did not start in time
and then instructed to boot the node in 'console' mode wishing
for some logs.
However, the node might actuay be running, causing 'console' mode
to fail with a different reason.

With this change, after a filure of daemon mode boot,
we issue a diagnosis.
1. if node can not be found from ps -ef, instruct the user
   to find information in erlang.log.N
2. if the node is found running, but not responding to pings
   instruct the user to check if the node name is
   resolvable and reachable
3. if the node is responding to pings but emqx app is not
   running, then it's likely a bug. so the user is advised
   to report a github issue.
2023-03-02 14:46:09 +01:00
Zaiming (Stone) Shi 3458aee296
Merge branch 'master' into 0213-port-blackbox-cover-report-from-v4 2023-02-22 22:21:44 +01:00
ieQu1 794bedef9f fix(emqx_ctl): Start CLI before ekka 2023-02-17 09:50:50 +01:00
Zaiming (Stone) Shi ba65cf48c3 feat: add new module emqx_cover.erl 2023-02-13 22:31:18 +01:00
Zhongwen Deng f98786c91f fix: bad crash_dump_file default value in rpm 2023-01-17 20:47:54 +08:00
Zhongwen Deng cf99089f84 chore: add some trace func to user_default 2023-01-17 20:33:01 +08:00
Zaiming (Stone) Shi dbc10c2eed chore: update copyright year 2023 2023-01-02 09:22:27 +01:00
Zaiming (Stone) Shi 0ce1ca89b7 refactor: use string type for server and servers 2022-12-30 14:20:23 +01:00
Zaiming (Stone) Shi cac7e0c5f0 refactor: move version prefix string detection to a static fun
this is to make dialyzer happy
2022-12-30 14:19:03 +01:00
Zaiming (Stone) Shi 0412826dd5 refactor: change ce | ee to v | prefix for shell prompt
reason: 'ce' (Community Edition) is only for internal use,
when it comes to user/customer facing descriptions,
we should use Opensource edition and Enterprise edition.

Similary, for user/customer facing shell prompt,
use `v` for Opensource edition and `e` for Enterprise
2022-12-14 13:05:44 +01:00
Zaiming (Stone) Shi 1c2fc4b6c3 Merge remote-tracking branch 'origin/release-50' into 1206-chore-merge-ee50-to-release-50 2022-12-06 16:35:56 +01:00
Zhongwen Deng 59ac5320b1 chore: update emqx_conf/emqx_machine version 2022-11-25 07:33:08 +08:00
Zhongwen Deng 70b6397ada feat: support node.global_gc_interval=disabled 2022-11-24 20:35:41 +08:00
Zhongwen Deng 57cc880977 chore: change community to opensource 2022-09-05 14:27:47 +08:00
Zhongwen Deng d447659365 feat: add edition info to /nodes api 2022-09-05 11:29:49 +08:00
Shawn 45352206a3 fix(auth): remove emqx_connector from testcases of authz/authn 2022-08-22 20:20:45 +08:00
Zhongwen Deng cd91b4b582 fix: dialyzer never match warning 2022-06-07 10:47:40 +08:00
Zhongwen Deng eb2a4df4af feat: change node.applications default to [] 2022-06-07 10:19:26 +08:00
ieQu1 27c922dbf2 fix(emqx_machine): Start essential applications as permanent 2022-05-24 12:42:30 +02:00
Georgy Sychev 0522361604 feat(shared_sub): Per-group shared subscription and local strategy 2022-05-13 14:23:06 +02:00
zhongwencool cee4607472
Merge pull request #7738 from zhongwencool/obfuscate_sensitive_config
feat: obfuscate sensitive values default_password
2022-04-24 14:05:23 +08:00
Zhongwen Deng 26f82fe2af chore: don't display arg(too long) 2022-04-24 11:15:53 +08:00
Zaiming (Stone) Shi b445182335 style: reformat emqx_machine, emqx_plugin_libs, and emqx_statsd 2022-04-23 09:55:50 +02:00
Thales Macedo Garitezi 5748b9b492
feat: allow customizing per-shard transaction log transport 2022-03-30 11:11:30 -03:00
Thales Macedo Garitezi 2a48f143bd
style: please elvis checks 2022-03-30 11:11:30 -03:00
Zaiming (Stone) Shi 64db085163 chore: EMQ X -> EMQX in source code 2022-02-16 22:08:12 +01:00
Zaiming (Stone) Shi 0528262705 chore(app.src): EMQ X -> EMQX in .app.src files 2022-02-16 17:51:45 +01:00
Shawn 8207090419 fix: format the code to remove extra spaces 2022-02-09 18:46:49 +08:00