Commit Graph

678 Commits

Author SHA1 Message Date
Ilya Averyanov 58a83739b8
Merge pull request #10986 from savonarola/ft/EMQX-9523/config-api
feat(ft-api): provide configuration API
2023-07-10 15:32:30 +03:00
Ilya Averyanov 12e237c36d
Merge pull request #11132 from savonarola/0531-authz-qos-retain
feat(authz): use extensible map format for actions in authz rules
2023-07-10 15:32:02 +03:00
firest a3de04ebd2 chore: add example for DynamoDB template 2023-07-09 16:23:27 +08:00
Ilya Averyanov 7de26a1776 feat(authz): use extensible map format for actions in authz rules
* support authorization on retain, qos fields
* refactored authz tests heavily
2023-07-07 19:40:34 +03:00
Andrew Mayorov a2b03716be feat(ft-api): provide configuration API
To configure `emqx_ft` during the runtime.
2023-07-07 19:15:42 +03:00
JimMoen 1587f038a5
feat: hstreamdb bridge with batch query 2023-07-07 12:25:36 +08:00
JimMoen d31a305bcb
refactor(hstreamdb): HStreamDB bridge to its own application 2023-07-07 12:25:36 +08:00
zhongwencool d509c47344 feat: add conf load api 2023-07-04 14:17:11 +08:00
zhongwencool 8b679cf358
Merge pull request #11137 from zhongwencool/dashboard-https-ssl-options
feat: refactor dashboard https ssl_options
2023-06-30 09:57:07 +08:00
firest 6fedd0328f fix(retainer): Fix missing changes in configuration 2023-06-29 14:06:17 +08:00
zhongwencool 82ea9e5a04 test: dashbaord api ct failed 2023-06-29 10:49:39 +08:00
zhongwencool e26c30ff0c feat: refactor dashboard https ssl_options 2023-06-29 10:49:39 +08:00
Kjell Winblad f9ea924cd0
Merge pull request #11151 from kjellwinblad/kjell/refactor/mysql_bridge/EMQX-9533
refactor: MySQL bridge and connector to separate applications
2023-06-28 15:15:55 +02:00
Kjell Winblad 3d1712c00c refactor: MySQL bridge to its own application
This commit refactor the MySQL bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9533
2023-06-27 17:13:26 +02:00
Kjell Winblad 85a130f68b refactor: move MySQL open source connector to its own app 2023-06-27 16:15:18 +02:00
Zaiming (Stone) Shi 4ed422cb9a
Merge pull request #11127 from zmstone/0622-refine-examples
0622 refine examples
2023-06-27 15:30:30 +02:00
Zaiming (Stone) Shi 92ed7d7639 docs: move some prometheus docs from schema desc to code comments 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 5425cae650 chore: delete ipv6_probe from s3 example 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 43705a54be docs: cosmetic changes in example config files 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 0428d91aa4 fix(config-examples): fix bad configs
prior to this commit, the provided configs are not type-checked
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 03d05825db docs: hide core_nodes config doc
core_nodes config is no longer required.
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 9aa83cb0a2 refactor(config-examples): do not include ee examples in ce 2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi e0d05a02ff refactor: refine examples
Also turns off vm_dist_collector metrics collector by default
2023-06-27 14:23:18 +02:00
Zaiming (Stone) Shi 49c4ea9ae9 refactor: mvoe config examples to 'rel/config' 2023-06-27 14:23:18 +02:00
Kjell Winblad e76daa0f3b refactor: redis bridge to its own application
This commit refactor the redis bridge to its own application.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9534
2023-06-27 10:24:44 +02:00
Kjell Winblad dd6aa96c99 refactor: move Redis open source connector to its own app 2023-06-27 10:24:40 +02:00
Kjell Winblad 8281b74695 docs: remove translations that should go to https://github.com/emqx/emqx-i18n 2023-06-26 14:32:59 +02:00
Kjell Winblad fa2c2a300a refactor: rename emqx_connector_mongo to emqx_mongodb
This renaming is consistent with the naming in the only other connector
only application emqx_oracle.
2023-06-26 14:32:59 +02:00
Kjell Winblad e3199ef813 refactor: MongoDB bridge to use the new app structure 2023-06-26 14:32:53 +02:00
firest 6bb8bf58ea fix(retainer): rename deliver_rate to delivery_rate 2023-06-23 16:51:19 +00:00
Paulo Zulato f4f068faaa
Merge pull request #11079 from paulozulato/feat-kafka-headers
feat(kafka): add support for kafka headers and ext headers
2023-06-22 12:04:32 -03:00
Thales Macedo Garitezi 7ef03d9e1f
Merge pull request #11090 from thalesmg/gcp-pubsub-consumer
feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
2023-06-22 09:17:45 -03:00
Paulo Zulato f47cc2a458 feat(kafka): add support for kafka headers and ext headers
Fixes https://emqx.atlassian.net/browse/EMQX-9175
2023-06-21 18:27:39 -03:00
firest 3f0d9db674 chore: update examples to new version and fix errors 2023-06-20 15:38:30 +08:00
firest c4222557ba chore: add examples of how to configure 2023-06-20 14:53:14 +08:00
Thales Macedo Garitezi b442910ff1 feat(gcp_pubsub_consumer): implement GCP PubSub Consumer bridge
Fixes https://emqx.atlassian.net/browse/EMQX-10281
2023-06-19 16:04:12 -03:00
JimMoen 611fc6c53c
chore: rm unused hocon file 2023-06-15 16:10:16 +08:00
Zaiming (Stone) Shi e8ccdb8d0f
Merge pull request #10998 from zmstone/0609-no-batch-for-mongodb
fix(mongodb): hide batch_size for mongodb resource
2023-06-11 21:26:12 +02:00
Zaiming (Stone) Shi ddef751527 fix(mongodb): hide batch_size for mongodb resource
MongoDB connector currently does not support batching
so the batch_size option has no effect.
However we cannot remove the field, so we choose to hide it from
schema
2023-06-11 11:08:58 +02:00
Zaiming (Stone) Shi 97850de524 Merge remote-tracking branch 'origin/release-51' into 0610-merge-release-51-to-master 2023-06-10 12:23:55 +02:00
Serge Tupchii 48ac942807 feat(emqx_bridge_kafka): add tcp keepalive option
Closes: EMQX-8725
2023-06-09 22:10:17 +03:00
Thales Macedo Garitezi 40675843a1 fix(iotdb): remove root `request_timeout` option
Fixes https://emqx.atlassian.net/browse/EMQX-10175

Since IoTDB bridge is not used as an authn/authz source and reuses the
webhook connector, we should have only one request timeout field (the
`resource_opts.request_ttl` one) like the webhook bridge does, to
avoid confusion and potentially bad configuration combinations.
2023-06-09 10:39:01 -03:00
Kjell Winblad 4215da12f0
Merge pull request #10970 from kjellwinblad/kjell/feat/kafka_add_async_param/EMQX-8631
feat: add sync/async option to the Kafka producer bridge
2023-06-09 12:51:23 +02:00
Ilya Averyanov c2635e938b
Merge pull request #10974 from keynslug/ft/EMQX-9523/backend-enable-flag
fix(ft): add `enable` flag to every backend
2023-06-09 11:44:37 +03:00
Zaiming (Stone) Shi 9f6beee6b3
Merge pull request #10936 from zmstone/0603-imporove-desc
docs: refine some desc
2023-06-09 08:44:03 +02:00
Zaiming (Stone) Shi af5c6720de Merge remote-tracking branch 'origin/master' into release-51 2023-06-09 08:37:29 +02:00
Zaiming (Stone) Shi 47f826f99a chore: fix typo 2023-06-09 09:31:32 +08:00
Kjell Winblad d524f8c805 refactor: rename config parameter 2023-06-08 16:27:04 +02:00
Kjell Winblad 6f2271e9f0 test: add Kafka producer bridge test case for query_mode parameter 2023-06-08 16:27:04 +02:00
Ivan Dyachkov 0434e6c6c3 docs: update description of TLS options incompatible with v1.3 2023-06-08 15:26:15 +02:00
Zaiming (Stone) Shi 7d2ea3dc76 docs: refine some desc 2023-06-08 13:58:31 +02:00
Andrew Mayorov ae97d516b8
fix(ft): add `enable` flag to every backend
And tolerate multiple configured backends, as long as only one of
them is enabled.
2023-06-08 14:54:13 +03:00
Kjell Winblad 47fa17b3c1 feat: add sync/async option to the Kafka producer bridge
This commit makes it possible to configure if a Kafka bridge should work
in query mode sync or async by setting the resource_opts.query_mode
configuration option.

Fixes:
https://emqx.atlassian.net/browse/EMQX-8631
2023-06-08 13:16:06 +02:00
Zaiming (Stone) Shi c1cf2365c2 Merge remote-tracking branch 'origin/master' into release-51 2023-06-08 12:30:02 +02:00
Ilya Averyanov 7f2de66dab fix(rebalance): add wait_health_check timeout to node evacuation
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
2023-06-07 21:37:01 +03:00
JimMoen 7d073c0d3d
style: make static_checks happy 2023-06-07 23:06:47 +08:00
Paulo Zulato a4407764f3 feat(connection): configurable TCP keepalive
Fixes https://emqx.atlassian.net/browse/EMQX-9852
2023-06-07 10:23:38 -03:00
Zaiming (Stone) Shi bdcc069aac chore: start deprecating mcast cluster discovery 2023-06-05 14:58:53 +02:00
Thales Macedo Garitezi 99796224d8 refactor(resource): rename `request_timeout` -> `request_ttl`
See
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options
2023-06-01 13:01:53 -03:00
Thales Macedo Garitezi 10425eb925 feat(resource): deprecate `auto_restart_interval` in favor of `health_check_interval`
See:
https://emqx.atlassian.net/wiki/spaces/P/pages/612368639/open+e5.1+remove+auto+restart+interval+from+buffer+worker+resource+options

Current problem:

In 5.0.x, we have two timer options that control the state changing of buffer worker
resources: auto_restart_interval and health_check_interval.

- auto_restart_interval controls how often the resource attempts to transition from
disconnected to connected.

- health_check_interval controls how often the resource is checked and potentially moved
from connected to disconnected or connecting.

The existence of two independent timers for very similar purposes is confusing to users,
QA and even developers.  Also, an intimately related configuration is request_timeout,
which can interact badly with auto_restart_interval if the latter is poorly configured:
requests may always expire if request_timeout < auto_restart_interval and if the resource
enters the disconnected state.  For health_check_interval, we attempt to derive a sane
default that gives requests a chance to retry (if request timeout is finite, then the
resource retries requests with a period of min(health_check_interval, request_timeout /
3).

Another problem with the separate auto_restart_interval is that its default value (60 s)
is too high when compared to the default request timeout and health check, leading to the
problems described above if not tuned.

Proposed solution:

We propose to drop auto_restart_interval in favor of health_check_interval, which will be
used for both disconnected -> connected and connected -> {disconnected, connecting}
transition checks.  With that, the resource will attempt to reconnect at the same interval
as the health check, which currently is 15 s.

Also, as two smaller changes to accompany this one:

- Increase the default request_timeout from 15 s to 45 s.
- Rename request_timeout to request_ttl.
2023-06-01 11:20:06 -03:00
JianBo He a3a6480f00
Merge pull request #10880 from HJianBo/batch-kickout-clients
feat: support kickout clients in batch
2023-05-31 22:36:35 +08:00
JianBo He c67135dd4b
Merge pull request #10598 from HJianBo/refactor_exproto_stream
refactor(exproto): support unary handler
2023-05-31 22:36:01 +08:00
JianBo He 5256d8092c fix: keep the real closed_reason 2023-05-31 13:44:51 +08:00
JianBo He b6bc3cd921 refactor(exproto): support unary handler 2023-05-31 13:44:51 +08:00
firest 03c7b84e89 feat(retainer): add a field `deliver_rate` to limit the maximum delivery rate 2023-05-31 09:44:25 +08:00
Andrew Mayorov 26819a647c
fix(mqttbridge): clarify schema descriptions + log messages
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-05-30 22:21:22 +03:00
Andrew Mayorov 7e7b50c5ba
refactor(mqttbridge): move into separate application 2023-05-30 17:56:34 +03:00
Andrew Mayorov b9021dfa30
chore: drop orphaned i18n file 2023-05-30 17:21:45 +03:00
Andrew Mayorov a5fc26736d
refactor(mqttconn): split ingress/egress into 2 separate pools
Each with a more refined set of responsibilities, at the cost of slight
code duplication. Also provide two different config fields for each pool
size.
2023-05-30 17:21:44 +03:00
JianBo He 1e7872c319 feat: support kickout clients in batch 2023-05-30 19:58:40 +08:00
Zaiming (Stone) Shi 7032881790 docs: delete i18n/zh from this repo 2023-05-29 14:22:23 +02:00
Zaiming (Stone) Shi 25319c3151
Merge pull request #10839 from zmstone/0526-merge-release-50-to-master
0526 merge release 50 to master
2023-05-29 08:56:20 +02:00
Zaiming (Stone) Shi c2450d230d
Merge pull request #10605 from zmstone/0503-no-telemetry-app-for-ee
0503 refactoring: no telemetry app for ee
2023-05-27 10:57:15 +02:00
Zaiming (Stone) Shi cc5b4d3748 Merge remote-tracking branch 'origin/release-50' into 0526-ci-delete-otp-24-from-standalone-app-test 2023-05-26 15:58:16 +02:00
Zaiming (Stone) Shi 6e2cde8224 refactor: delete emqx_statsd all together 2023-05-26 15:17:42 +02:00
Zhongwen Deng df4a0afdf5 fix: zones api default value 2023-05-26 18:59:38 +08:00
Zaiming (Stone) Shi ee122155f1 docs: fix spellcheck for telemetry app 2023-05-26 08:44:19 +02:00
JimMoen 7eb445707f
chore: fix schema rename 2023-05-25 17:10:40 +08:00
Paulo Zulato 8d3d3e523b fix(oracle): replace 'database' field by 'service name'
Fixes https://emqx.atlassian.net/browse/EMQX-9980
2023-05-24 15:40:56 -03:00
JimMoen 28015597ee
Merge remote-tracking branch 'emqx/release-50' into merge-release-50 2023-05-24 19:34:12 +08:00
Zaiming (Stone) Shi 2de2a979f5
Merge pull request #10779 from zmstone/0522-remove-log-file-paht-validator
chore: allow special chars in log dir
2023-05-23 22:42:12 +02:00
Zaiming (Stone) Shi 2fa5b511bf chore: hide stale config 2023-05-23 14:20:49 +02:00
Thales Macedo Garitezi 29e0e41ec7
docs: improve descriptions 2023-05-23 09:08:55 -03:00
Zaiming (Stone) Shi 732a7be187 Merge remote-tracking branch 'origin/release-50' 2023-05-22 17:46:54 +02:00
Thales Macedo Garitezi 65f973044f feat(pulsar): improve authn error check time and add connect timeout
Fixes https://emqx.atlassian.net/browse/EMQX-9910
2023-05-22 11:33:16 -03:00
Zhongwen Deng e797f93a53 fix: can't get file_list_transfer desc cause 500 error on /swagger.json api 2023-05-22 15:17:50 +08:00
Andrew Mayorov 9aaa0b6fe1
Merge pull request #9927 from emqx/file-transfer
feat: implement file transfer over mqtt
2023-05-19 18:35:13 +03:00
Kjell Winblad 28dda39e39
Merge pull request #10724 from kjellwinblad/kjell/docs/missing_http_api_endpoint_descriptions/EMQX-9767
docs: make sure there is a summary for all API endpoints
2023-05-19 12:05:41 +02:00
zhongwencool ba43a0e30f
Merge pull request #10734 from thalesmg/merge-r50-into-v50-a
merge `release-50` into `master`
2023-05-18 14:24:53 +08:00
zhongwencool bf5ee41009
Merge pull request #10702 from zhongwencool/keepalive-backoff-rename
feat: deprecated keepalive_backoff, introduce keepalive_multiplier
2023-05-18 10:00:17 +08:00
Thales Macedo Garitezi 447b76464b Merge branch 'release-50' into merge-r50-into-v50-a 2023-05-17 14:50:18 -03:00
Zhongwen Deng e26ce5816e feat: hide keepalive_backoff, introduce keepalive_multiplier 2023-05-17 21:25:10 +08:00
Kjell Winblad 397dce0d78
Merge pull request #10654 from kjellwinblad/kjell/refactor/clickhouse_bridge/EMQX-9538
refactor: the clickhouse bridge to use the new bridge app structure
2023-05-17 11:19:33 +02:00
Kjell Winblad bce43978a2 refactor: the clickhouse bridge to use the new bridge app structure
Fixes:
https://emqx.atlassian.net/browse/EMQX-9538
2023-05-17 10:12:43 +02:00
Kjell Winblad ab8b8ccaad docs: make sure there is a summary for all API endpoints
Many HTTP API endpoints did not have a summary when looking at the API
docs at "http://emqx_host_name:18083/api-docs". This has been fixed by
making sure there is a summary corresponding to all endpoints.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9767
2023-05-17 09:58:56 +02:00
Ilya Averyanov 1a8cf0e392 Merge branch 'release-50' into file-transfer
* release-50:
  fix(pulsar): use a binary duration as default `health_check_interval`
  docs: add changelog entry
  docs: clarify description of bridge username and password
  chore: bump to v5.0.25
  fix(limiter): adjust type for compatibility
  fix(limiter): fix that update node-level limiter config will not working
  chore: upgrade dashboard to v1.2.4-1 for ce
  chore: upgarde rulesql to 0.1.6 to fix invaid utf8 input
  chore: add changelog for 10659
  fix: crash when sysmon.os.mem_check_interval = disabled
  chore: bump influxdb version && update changes
  refactor(influxdb): move influxdb bridge into its own app
  chore: add listener default changelog
  fix: ocsp cache SUITE failed
  fix: ensure atom key for emqx_config:get
  fix: only fill cerf_file default in server side
  fix: authn init is empty
  fix: bad listeners default ssl_options
2023-05-17 00:27:55 +03:00
Zaiming (Stone) Shi 9b9731b943
Merge pull request #10473 from zmstone/0421-refactor-delete-unused-schema-field
0421 refactor delete unused schema field
2023-05-16 22:22:03 +02:00
Andrew Mayorov 91ce1c2a90
fix(ft-schema): make description more natural
Co-authored-by: ieQu1 <99872536+ieQu1@users.noreply.github.com>
2023-05-16 16:27:33 +03:00
Kjell Winblad 46f05056de docs: clarify description of bridge username and password
Fixes:
https://emqx.atlassian.net/browse/EMQX-9613
2023-05-15 16:29:52 +02:00
zhongwencool a953b951fe
Merge branch 'master' into sync-release-50-to-master 2023-05-12 18:01:58 +08:00
firest 6497bcae46 refactor(influxdb): move influxdb bridge into its own app 2023-05-12 11:48:49 +08:00
Ilya Averyanov 5b5d7ceac5 Merge branch 'release-50' into file-transfer
* release-50:
  fix(limiter): fix an error when setting `max_conn_rate` in a listener
  chore: bump erlcloud dependencies vsns
  chore: rename dynamo template files
  refactor(dynamo): move dynamo bridge into its own app
  chore: update changes && bump app versions
  fix: issues with the RabbitMQ config
  refactor(pgsql): move pgsql && matrix && timescale bridges into their own app
  fix: the iotdb password field so it has the password format
  chore: update changes
  refactor(tdengine): move tdengine bridge into its own app
  feat: deprecate listeners's authn http api
2023-05-11 16:15:38 +05:00
Zaiming (Stone) Shi 7566c25199 refactor: delete unused connector field 'base_url' 2023-05-11 09:04:55 +02:00
Zaiming (Stone) Shi b3e35dac1c
Merge pull request #10584 from paulozulato/debug-ssl-handshake
feat: add log level configuration to ssl communication
2023-05-11 08:59:13 +02:00
JimMoen fbd516d740
Merge pull request #10622 from JimMoen/refactor-dynamo-bridge
refactor(dynamo): move dynamo bridge into its own app
2023-05-11 11:00:50 +08:00
JimMoen 8914e006c2
refactor(dynamo): move dynamo bridge into its own app 2023-05-10 18:49:21 +08:00
firest 066ed5c6ec refactor(pgsql): move pgsql && matrix && timescale bridges into their own app 2023-05-10 16:51:33 +08:00
lafirest 594ceaf9e6
Merge pull request #10650 from lafirest/refactor/tdengine_dir
refactor(tdengine): move tdengine bridge into its own app
2023-05-10 16:42:35 +08:00
Ilya Averyanov 8d9b785bd7 Merge branch 'release-50' into file-transfer
* release-50:
  chore(rebalance): fix app metadata
  chore(rebalance): move apps from lib-ee, add READMEs
  docs: refine zh tr
  docs: delete APL header from ee file
  docs: delete zh changelog
  chore(rebalance): review fixes
  chore(rebalance): rebase and review fixes
  feat(rebalance): port apps from 4.x
2023-05-10 11:55:23 +05:00
firest 0f979b3d24 refactor(tdengine): move tdengine bridge into its own app 2023-05-10 14:12:57 +08:00
Andrew Mayorov 5e5f854ce1 feat(ft-conf): simplify schema of storage / exporter backends
Assumption is this changes will make `emqx_ft` config schema
user-friendlier and also more future-proof.
2023-05-09 23:32:16 +05:00
Ilya Averyanov 7fa166f034 Merge branch 'release-50' into file-transfer
* release-50: (73 commits)
  feat: add RabbitMQ bridge
  docs: improve rule engine labels and descriptions
  chore: bump version && update changes
  refactor(rocketmq): move rocketmq bridge into its own app
  test: dashboard_listener_test crash
  chore: bump chart versions
  chore: bump ee version to e5.0.4-alpha.1
  test: fix inter-suite flakiness
  build: compatibility to make 4.4+
  feat: add IotDB bridge
  ci: ensure git safe dir in build_packages
  ci: ensure git safe dir
  test: check_oom's max_mailbox_size
  feat: rename max_message_queue_len to max_mailbox_size
  fix(buffer_worker): fix inflight count when updating inflight item
  chore: prepare for v5.0.25-rc.1 release
  docs: add change log entry
  fix: non_neg_integer() translated to minimum = 1 in bridge-api-en.json
  chore: `MQTT X` -> `MQTTX`
  chore: make sure brod_gssapi app is included in relese package
  ...
2023-05-09 23:30:22 +05:00
Zaiming (Stone) Shi 3ff04d51bc docs: refine zh tr 2023-05-09 20:51:24 +05:00
Ilya Averyanov ffed8a132b chore(rebalance): review fixes
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-05-09 20:51:24 +05:00
Ilya Averyanov e683d28973 chore(rebalance): rebase and review fixes 2023-05-09 20:51:24 +05:00
Kjell Winblad 70cf1533db feat: add RabbitMQ bridge 2023-05-09 14:32:26 +02:00
Kjell Winblad 4d1499b747
Merge pull request #10582 from kjellwinblad/kjell/fix/max_overflow_desc/EMQX-9714
docs: better max_overflow field label and description for MongoDB bridge
2023-05-09 12:26:26 +02:00
Kjell Winblad b0c647dd44
Merge pull request #10649 from kjellwinblad/kjell/fix/unknown_desc/EMQX-9645
docs: improve rule engine labels and descriptions
2023-05-09 12:25:01 +02:00
Kjell Winblad 9af8375773 docs: improve rule engine labels and descriptions
Fixes:
https://emqx.atlassian.net/browse/EMQX-9645
2023-05-09 11:44:30 +02:00
firest a119100a0e refactor(rocketmq): move rocketmq bridge into its own app 2023-05-09 15:57:56 +08:00
lafirest b94290db58
Merge pull request #10625 from lafirest/refactor/limiter_cfg
refactor(limiter): simplify limiter configuration
2023-05-09 11:40:05 +08:00
Stefan Strigler 1ade7ce9a3
Merge pull request #10560 from sstrigler/EMQX-8390-5-0-x-io-tdb-bridge-port-conf-and-api-to-5-0
IoTDB Bridge
2023-05-08 20:27:01 +02:00
Paulo Zulato 77176787ca feat: add log level configuration to ssl communication
Fixes https://emqx.atlassian.net/browse/EMQX-9781
2023-05-08 15:11:49 -03:00
Stefan Strigler d231e708cb feat: add IotDB bridge 2023-05-08 16:19:01 +02:00
Zhongwen Deng eaa129d0d7 feat: rename max_message_queue_len to max_mailbox_size 2023-05-08 20:27:52 +08:00
firest d914d1ee1d refactor(limiter): simplify limiter configuration 2023-05-08 11:45:49 +08:00
JimMoen d5dce771bd
refactor(sqlserver): move sqlserver bridge into its own app 2023-05-06 10:22:05 +08:00
Ilya Averyanov dd3471bc22 Merge branch 'master' into file-transfer
* master: (194 commits)
  fix(limiter): update change && fix deprecated version
  chore: update changes
  perf(limiter): simplify the memory represent of limiter configuration
  ci(perf test): update tf variable name and set job timeout
  ci: fix artifact name in scheduled packages workflow
  fix: build_packages_cron.yaml workflow
  ci: move scheduled builds to a separate workflow
  build: check mnesia compatibility when generating mria config
  docs: fix a typo in api doc description
  feat(./dev): use command style and added 'ctl' command
  test: fix delayed-pubish test case flakyness
  refactor: remove raw_with_default config load option
  chore: add changelog for trace timestrap
  feat: increase the time precision of trace logs to microseconds
  chore: make sure topic_metrics/rewrite's default is []
  docs: Update changes/ce/perf-10417.en.md
  chore: bump `snabbkaffe` to 1.0.8
  ci: run static checks in separate jobs
  chore(schema): mark deprecated quic listener fields ?IMPORTANCE_HIDDEN
  chore: remove unused mqtt cap 'subscription_identifiers'
  ...
2023-05-05 16:50:18 +03:00
Ilya Averyanov cff7788b2e chore(ft): add s3 exporter tests 2023-05-05 11:55:09 +03:00
Zaiming (Stone) Shi 232c5926f2 docs: fix a typo in api doc description 2023-05-04 10:50:14 +02:00
Zaiming (Stone) Shi 856129984b refactor: remove raw_with_default config load option
This option was previously only in tests to avoid
emqx_conf app overwriting previously set configs with default values.

After a03f2dd64b, the issue for
test cases had been resolved.
This commit is to get rid of the option all together
2023-05-04 09:48:28 +02:00
Kjell Winblad b7c7d400e2 docs: better max_overflow field label and description for MongoDB bridge
The new label and description for the max_overflow field clarifies the
purpose of the field, which is to manage the maximum number of
additional workers created when all workers in the pool are busy,
allowing more concurrent connections to the MongoDB server during
workload spikes.

Fixes part of:
https://emqx.atlassian.net/browse/EMQX-9714
2023-05-03 17:49:08 +02:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Zaiming (Stone) Shi 4d705817d8 refactor(log): move default values to schema 2023-04-29 22:10:19 +02:00
Zaiming (Stone) Shi 41f13330ba refactor: export EMQX_LOG_DIR 2023-04-29 22:10:19 +02:00
Andrew Mayorov ed3756ea09
feat(ft-api): add paging support through cursors 2023-04-28 13:49:15 +03:00
Andrew Mayorov 811e449357
feat(ft-conf): provide global killswitch 2023-04-28 10:54:21 +03:00
Paulo Zulato dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Zaiming (Stone) Shi ed7a8659d2 feat: add a json format support for the /status API 2023-04-26 14:41:33 +02:00
Zhongwen Deng f0cd5c98c7 chore: split i18n with script 2023-04-26 13:57:15 +08:00
zhongwencool 9d893b49eb
Merge branch 'master' into sync-release-50-to-master 2023-04-26 10:54:46 +08:00
firest 3bb50a5751 fix(rocketmq): fix that the update of ACL info not working 2023-04-25 16:55:03 +08:00
Thales Macedo Garitezi cb149ac345 docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi 631863d843 docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-04-24 10:28:26 -03:00
Thales Macedo Garitezi ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0)
Fixes https://emqx.atlassian.net/browse/EMQX-8398
2023-04-24 10:28:26 -03:00
Zaiming (Stone) Shi 366aa53c80
Merge pull request #10462 from zmstone/0430-EMQX-8434-shared-dispatch-ack-deprecated
chore: Hide config shared_dispatch_ack_enabled
2023-04-24 09:42:09 +02:00
lafirest ac54fa0ba5
Merge pull request #10482 from lafirest/chore/rocketmq_template
chore: add examples for RocketMQ template
2023-04-23 18:10:35 +08:00
firest 5ab0887687 chore: add examples for RocketMQ template 2023-04-23 15:17:51 +08:00
firest 5ad5d7ee8d fix(opents): adjust code structure 2023-04-23 14:28:14 +08:00
firest 932a327952 chore: make spellcheck and xref happy 2023-04-23 11:22:56 +08:00
firest 0b46acda87 test(opents): add test cases for OpenTSDB 2023-04-23 11:03:20 +08:00
firest 5074825075 feat(opents): OpenTSDB integration 2023-04-23 09:56:24 +08:00
Ilya Averyanov b951de4c6e feat(ft): convert hocon files to the new format 2023-04-21 19:14:24 +03:00
Ilya Averyanov 0211bcf030 Merge branch 'master' into file-transfer
* master: (279 commits)
  chore: shorten ct/run.sh script
  chore: rename cassandra_impl to cassandra_connector
  chore: fix mix.exs checking
  refactor(cassandra): move cassandra bridge into its own app
  chore: apply review suggestions
  chore: update changes/ce/fix-10449.en.md
  test: add a test for authn {}
  chore: add changlog for authn_http validation
  fix: always check authn_http's header and ssl_option
  chore: apply suggestions from code review
  fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid
  fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager
  perf(emqx_alarm): use dirty Mnesia operations to activate an alarm
  ci: simplify find-apps.sh for ee apps
  perf(emqx_resource): don't reactivate alarms on reoccurring errors
  ci: check if Elixir files are formatted in pre-commit hook
  fix(dynamo): fix field name errors
  chore: remove *_collector for prometheus api's example
  chore: make plugins config to low level
  chore: re-split dynamo i18n file
  ...
2023-04-21 17:37:17 +03:00
Zaiming (Stone) Shi 701a1f65f9 chore: Hide config shared_dispatch_ack_enabled 2023-04-21 13:59:55 +02:00
JianBo He b270623c46 chore: rename cassandra_impl to cassandra_connector 2023-04-21 16:49:25 +08:00
firest 7704995279 fix(rocketmq): expose the driver parameter `sync_timeout` into the RocketMQ bridge configuration 2023-04-21 15:10:25 +08:00
JianBo He 5cc28a7b45 chore: fix mix.exs checking 2023-04-21 14:46:09 +08:00
JianBo He bdce32e713 refactor(cassandra): move cassandra bridge into its own app 2023-04-21 14:45:38 +08:00
JianBo He 6532cfb337
Merge pull request #10452 from thalesmg/refactor-gcp-pubsub-bridge-v50
refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
2023-04-21 14:42:00 +08:00
Zaiming (Stone) Shi adc422d063 chore: re-split dynamo i18n file 2023-04-20 11:11:35 +02:00
Zaiming (Stone) Shi 415830a0a2 Merge remote-tracking branch 'origin/release-50' into 0420-sync-release-50-to-masteer 2023-04-20 11:03:39 +02:00
Zaiming (Stone) Shi c2385f85d6
Merge pull request #10444 from zmstone/0418-cherry-pick-fixes-for-e503
0418 cherry pick fixes for e503
2023-04-20 10:58:57 +02:00
Thales Macedo Garitezi a9bd91fcff refactor(gcp_pubsub): move GCP PubSub Bridge to its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9536

Note: since GCP PubSub is not shared by any authn/authz backend,
there's no need to separate its connector into another app.
2023-04-19 13:24:32 -03:00
firest e89f4d4565 fix(dynamo): fix terminology erros
- Changed `username` to `aws_access_key_id`
- Changed `password` to `aws_secret_access_key`
2023-04-19 16:26:42 +08:00
firest bc353b0a06 fix(dynamo): change `database` to `table` in the schema of the DynamoDB bridge
there is no term like `database` in DynamoDB, the correct concept should be `table`
2023-04-19 15:56:29 +08:00
Zaiming (Stone) Shi 6c352fd525 docs: update config docs 2023-04-18 19:18:27 +02:00
Zaiming (Stone) Shi 9a4af6bd76 docs: update config docs 2023-04-18 19:15:40 +02:00
某文 3cd9ca7067 chore: change config seq 2023-04-18 21:10:40 +08:00
Zhongwen Deng e3d6fa1f21 chore: update config's changelog and emqx_conf.template 2023-04-18 20:13:44 +08:00
Zaiming (Stone) Shi 48d8381a6d docs: update doc refs 2023-04-18 09:48:28 +02:00
Thales Macedo Garitezi 9d15247dd5 feat(schema_registry): add support for protobuf schemas
Fixes https://emqx.atlassian.net/browse/EMQX-9470
2023-04-17 13:04:54 -03:00
Zaiming (Stone) Shi b9a43ead39 refactor: Split i18n files 2023-04-17 13:35:33 +02:00
firest 02f8d073f8 test(limiter): fix test errors and make spellcheck happy 2023-04-17 10:06:43 +08:00
firest 55376144ce fix(limiter): simplify the configuration of the limiter 2023-04-17 10:06:36 +08:00
Thales Macedo Garitezi 3373a63137
docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-04-14 13:21:30 -03:00
Thales Macedo Garitezi e073bc90bc refactor(buffer_worker): rename `s/queue/buffer/g` 2023-04-14 11:37:19 -03:00
Thales Macedo Garitezi dd38122797 docs: improve descriptions
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2023-04-14 11:37:19 -03:00
Thales Macedo Garitezi 14ed4a7ada feat(buffer_worker): set default queue mode to `memory_only`
Fixes https://emqx.atlassian.net/browse/EMQX-9367

For better user experience and performance for the average bridge, we
should change the default queue mode to `memory_only`, as was the
behavior of most bridges in e4.x.  This leads to better performance
when message rate is high enough and the remote resource is not
keeping up with EMQX.

Also, we set the default segment size to equal max queue bytes.
2023-04-14 11:37:19 -03:00
JimMoen 06b1ec862e
fix: hocon label field 2023-04-14 12:43:29 +08:00
JimMoen 35ab1c0190
style: make hocon style check and spellcheck happy
- with more human readable i18n style check format
2023-04-14 10:02:47 +08:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
Ilya Averyanov 92ff2b7d6e feat(ft): make timeouts configurable 2023-04-14 01:58:14 +03:00
Thales Macedo Garitezi 9acfe00498
Merge pull request #10347 from thalesmg/refactor-kafka-bridge-dirs-v50
refactor(kafka_bridge): move kafka bridge into its own app
2023-04-13 13:26:36 -03:00
JianBo He 993be31998
chore: update rel/i18n/emqx_conf_schema.hocon
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-13 18:12:17 +08:00
JianBo He b57b964083 chore(i18n): improve config desc 2023-04-13 11:15:36 +08:00
Thales Macedo Garitezi 871ee90b3e refactor(kafka_bridge): move kafka bridge into its own app
Fixes https://emqx.atlassian.net/browse/EMQX-9481
2023-04-12 13:54:45 -03:00
Ivan Dyachkov bdffa925db chore: merge upstream/master release-50 2023-04-12 15:30:20 +02:00
Stefan Strigler f668ad7b9e
Merge pull request #10336 from sstrigler/EMQX-8507-rule-engine-need-a-new-api-to-crud-rule-engines-setting-configs
feat: add `/rule_engine` API endpoint
2023-04-12 11:44:42 +02:00
Thales Macedo Garitezi 914184697e
Merge pull request #10337 from thalesmg/schema-registry-v50
feat: implement schema registry for 5.0 (avro)
2023-04-11 16:46:27 -03:00
Stefan Strigler e282d361f8
style: fix Chinese translations
Co-authored-by: LenaLenaPan <120552185+LenaLenaPan@users.noreply.github.com>
2023-04-11 09:34:51 +02:00
Andrew Mayorov 37a520d797
fix(i18n): relocate i18n files to rel/i18n 2023-04-10 16:02:34 +03:00
Thales Macedo Garitezi 33100ecca6 feat: implement schema registry for 5.0 (avro)
Part of https://emqx.atlassian.net/browse/EMQX-9251

This ports part of the Schema Registry app from 4.x to 5.0.  Here,
only support for Avro is added.  Subsequent PRs will follow to add
support for other formats.
2023-04-06 16:28:45 -03:00
JianBo He 702ecbcb6e Merge remote-tracking branch 'upstream/master' into refactor-gw-dir 2023-04-06 10:44:59 +08:00
Stefan Strigler 576d1524bb style: fix wording of i18n files 2023-04-05 16:29:17 +02:00
Stefan Strigler aea870f319 feat: add `/rule_engine` API endpoint 2023-04-05 15:19:31 +02:00
Zaiming (Stone) Shi a9bf633e03
Merge pull request #10320 from zmstone/0403-sync-release-50-back-to-master
0403 sync release 50 back to master
2023-04-04 23:31:24 +02:00
Zaiming (Stone) Shi 429b3d9efd Merge remote-tracking branch 'origin/master' into 0403-sync-release-50-back-to-master 2023-04-03 20:32:33 +02:00
Thales Macedo Garitezi f3ffc02bff feat(bridges): enable async query mode for all bridges with buffer workers
Fixes https://emqx.atlassian.net/browse/EMQX-9130

Since buffer workers always support async calls ("outer calls"), we
should decouple those two call modes (inner and outer), and avoid
exposing the inner call configuration to user to avoid complexity.

For bridges that currently only allow sync query modes, we should
allow them to be configured with async.  That means basically all
bridge types except Kafka Producer.
2023-04-03 14:49:51 -03:00
Zaiming (Stone) Shi 5f6d318cf0 fix(i18n): fix missing docs for gateway configs 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi c17de6c415 docs: fix i18n desc style 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi 36000abf51 refactor: relocate i18n files for apps/emqx 2023-04-03 13:12:24 +02:00
Zaiming (Stone) Shi 4d28731176 docs: delete local-override.conf from config template 2023-03-24 08:37:00 +01:00
firest 6f3e952edf fix(psk): add new supported ciphers to the conf template 2022-12-20 22:40:49 +08:00
Zaiming (Stone) Shi 4cadb86b18 docs: fix typos 2022-10-27 20:09:31 +02:00
Zaiming (Stone) Shi c157392452 docs: fix self-closing html tag, change </br> to <br/> 2022-10-27 13:57:18 +02:00
Zaiming (Stone) Shi a314950be9 docs: fix zh punctuations 2022-10-26 16:19:38 +02:00
Zaiming (Stone) Shi cf76267155 docs: add a string value quote example 2022-09-27 09:02:49 +02:00
Zaiming (Stone) Shi 9bd9e2ecd8 feat: add EMQX_LOG_DIR env 2022-09-16 09:49:55 +08:00
Zaiming (Stone) Shi ac840e318e refactor(cookie): Warning message when boot with default Erlang cookie 2022-09-06 18:58:35 +02:00
Zhongwen Deng 29f9258595 chore: update RUNNER_LOG_DIR 2022-09-05 17:36:35 +08:00
zhongwencool 1819b7e224
chore: update rel/emqx_vars
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
2022-09-05 17:34:29 +08:00
Zhongwen Deng e35957af0b chore: add some warning to emqx_vars 2022-09-05 16:12:40 +08:00
Zhongwen Deng 63281e5491 feat: don't override env RUNNER_LOG_DIR in emqx_vars 2022-09-05 16:10:51 +08:00
Zaiming (Stone) Shi 8717535d32 refactor: populate ciphers list at runtime
Populating ciphers list when checking schema makes the
config file example and the schmea documents quite bloated
2022-08-24 21:02:21 +02:00
Zaiming (Stone) Shi 22f5b62531 docs: add default ciphers to document config 2022-08-24 21:02:21 +02:00
Zaiming Shi a548b82bc9 docs: add zh version config doc template 2022-06-28 22:34:24 +02:00
Zaiming (Stone) Shi 4fdd793071 refactor: move config template file to rel dir 2022-06-12 19:57:08 +02:00
Zaiming (Stone) Shi 3c03c0cfc1 feat(bin/emqx): make possible to boot Erlang node from Elixir build 2022-04-13 21:55:05 +02:00
Zaiming (Stone) Shi 72405f977c refactor(bin/emqx): remove unnecessary intermediate env variables 2022-04-13 13:50:56 +02:00
Zaiming (Stone) Shi c8242816b0 refactor(bin/emqx): no need to template-render RUNNER_ROOT_DIR 2022-04-13 13:50:25 +02:00
Zaiming (Stone) Shi 32694e601a refactor: rename RUNNER_ETC_DIR to EMQX_ETC_DIR 2022-04-05 23:08:45 +02:00
Zaiming (Stone) Shi 4e64010a3b build: more structured build info 2022-03-02 21:22:03 +01:00
Zaiming (Stone) Shi e22104476a fix(data_dir): get data_dir config in the right way
now data_dir config is respected by boot script as well as
EMQX application after booted
2022-02-28 09:59:56 +01:00
Zaiming (Stone) Shi a8aa312825 fix(boot): allow user defined data_dir 2022-02-25 12:08:05 +01:00
Ilya Averyanov 64c59b5469 feat(license): check license before node start 2022-02-11 11:49:29 +03:00
Ilya Averyanov 17599432d1 feat(license): add license application 2022-02-07 19:05:47 +03:00
Zaiming (Stone) Shi a04c834d05 chore: relocate some release files from data/ to rel/
data dir at the root is easily confused with the data
dir for EMQ X at run time
2022-01-14 16:41:22 +01:00
Thales Macedo Garitezi 67cca5d3a1
chore(mix): use the same script in mix release as in rebar release
Surprisingly enough, by doing small cirurgical changes in the existing
EMQX control scripts, we are able to get it running with Elixir and
with existing functionalities (`console`, `remote_console`, `start`,
`stop`, `ctl`, `foreground`, `eval`).
2021-12-23 17:20:04 -03:00
Thales Macedo Garitezi 014b6d3cfd
chore(shellcheck): fix shellcheck linting warnings 2021-12-22 15:43:43 -03:00
Thales Macedo Garitezi 62024854e5
chore(mix): wrap produced script and set required variables
We create a shim script that calls the one produced by Mix Release,
creating and setting the required configurations.
2021-12-22 15:15:19 -03:00
Thales Macedo Garitezi 917575de5a
chore(mix): minimal elixir mix release build
This commit enables a minimal working build of EMQX release using
Mix.  However, to properly start the release, several configuration
steps are still missing.  A `mix_release.sh` script does a few hacks
to get the release built with Mix to start properly, by first assuming
that `make emqx` has been run prior to the release, ran once to
generate the `app.*.config` files, and then it copies that and some
other files to the expected places.

Also, `emqx_telemetry` hangs the start procedure because it thinks
it's in an official release and tries to make a request.  We disable
it temporarily via config just to get a working build for now.
2021-12-21 13:51:22 -03:00
Feng Lee c84598dae9 2.0 - move rel/ to emqttd-relx project 2016-07-21 18:29:04 +08:00
Feng 710890dd22 relx 2016-07-21 09:31:14 +08:00
Feng 2e200f253f 2.0 - erlang.mk to replace rebar 2016-07-19 17:46:05 +08:00
Feng 03d6710b60 gen_conf to improve the configuration of emqttd broker 2016-07-19 16:49:30 +08:00
Feng 61b2ab48ad merge emq10 2016-07-16 21:55:02 +08:00
Feng Lee 223c06fd68 fix ldap sslopts 2016-07-12 17:36:23 +08:00
Feng Lee 842df63605 merge master 2016-06-28 11:27:52 +08:00
Feng Lee 834e6f7269 Add MQTT specs. etc/vm.args: -env ERL_CRASH_DUMP log/emqttd_crash.dump 2016-06-27 09:11:09 +08:00
Feng Lee fe3becbf9b Fix issue#599 - Reject empty clientId with non-clean session 2016-06-21 11:29:59 +08:00
Feng eaf254e919 Merge branch 'master' into plus 2016-06-05 12:51:35 +08:00
Feng Lee 791c50282f Merge pull request #566 from emqtt/issue#557
load 'observer' application and fix issue #557
2016-05-25 14:20:39 +08:00
Feng Lee b0541ac18c load 'observer' application and fix issue #557 2016-05-25 14:19:25 +08:00
Feng Lee 9e9b074d84 error 2016-05-07 13:07:02 +08:00
Feng Lee c7ecc0ace0 merge emq10 2016-05-07 10:30:22 +08:00
Feng Lee 76851e58fb atom_to_list(Username) 2016-05-03 18:31:58 +08:00
Feng Lee 631609f035 ensure gen_logger started 2016-05-02 15:15:05 +08:00
Feng 0d84c7474a rm plugins 2016-04-30 10:21:53 +08:00
Feng 16fcb2997e issue#516 - The erlang '-name' requires the fully qualified host name 2016-04-19 16:14:07 +08:00