yuanbiao
/
emqx
1
0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,158 Commits 393 Branches 684 Tags 74 MiB
Commit Graph

701 Commits

Author SHA1 Message Date
Thales Macedo Garitezi 8aa7c014e7 perf(buffer_worker): avoid calling `ets:info/2` 
(Almost?) fixes https://emqx.atlassian.net/browse/EMQX-9637

During the course of performance tests comparing the performance of
e5.0.3 and e4.4.16 regarding the webhook bridge in sync mode, we
observed that the throughput in e5.0.3 (sync) was much lower than in
e4.4.16: ~ 9 k msgs / s vs. ~ 50 k msgs / s, respectively.

Analyzing `observer_cli` output, we noticed that a lot of the time
both buffer workers and ehttpc processes was spent in `ets:info/2`.
That function was called to check the size of the inflight table when
updating metrics and checking if the inflight table was full.  Other
uses of `ets:info/2` were contained inside the arguments to some
`?tp/2` macro usages (https://github.com/kafka4beam/snabbkaffe/pull/60).

By using a specific record to track the size of the table, we managed
to improve the bridge performance to ~ 45 k msgs / s in sync mode.
 2023-05-02 17:05:32 -03:00
William Yang 3d7201502b perf(config): avoid build new map in emqx_mqtt_caps:get_caps 2023-05-02 21:50:36 +02:00
Zaiming (Stone) Shi d5f5f35787 Merge remote-tracking branch 'origin/release-50' into 0502-merge-release-50-back-to-master 2023-05-02 18:08:04 +02:00
Ilya Averyanov 791010a65c chore(topic_metrics): do not complain on nonexisting ets while topic metrics stop 2023-05-02 15:59:50 +03:00
Thales Macedo Garitezi 77f5e461a3 chore: bump ehttpc -> 0.4.8 
Fixes https://emqx.atlassian.net/browse/EMQX-9656

See also https://github.com/emqx/ehttpc/pull/45

This fixes a race condition where the remote server would close the
connection before or during requests, and, depending on timing, an
`{error, normal}` response would be returned.  In those cases, we
should just retry the request without using up "retry credits".
 2023-04-27 14:10:37 -03:00
Paulo Zulato dd90b2f498 feat(oracle): Oracle Database integration 2023-04-27 13:07:20 -03:00
Thales Macedo Garitezi c53741a08c fix(buffer_worker): avoid sending late reply messages to callers 
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
 2023-04-26 13:18:28 -03:00
Ivan Dyachkov 50504a4cbf docs: Generate changelog for v5.0.24 2023-04-26 18:03:00 +02:00
William Yang 5ed3c3a92c perf(config): eliminate make_ref() calls in config get calls 2023-04-26 10:58:08 +02:00
SergeTupchiy d398276852
Merge pull request #10518 from SergeTupchiy/upgrade-ekka-to-0.15.1-mria-0.5.2 
chore: bump ekka to 0.15.1
 2023-04-26 11:03:23 +03:00
zhongwencool 341024f0bc
Merge pull request #10497 from HJianBo/improve-chages-sentence 
chore: improve changes
 2023-04-26 14:56:29 +08:00
zhongwencool 713e050a79
Merge pull request #10512 from zhongwencool/hocon-0.39.3 
feat: improved the storage format of Unicode characters in data files
 2023-04-26 13:59:51 +08:00
Thales Macedo Garitezi 79cf5cad19
Merge pull request #10378 from thalesmg/pulsar-producer-e50 
feat: implement Pulsar Producer bridge (e5.0)
 2023-04-25 18:01:43 -03:00
Serge Tupchii 19b5ebff81 chore: bump ekka to 0.15.1 
ekka 0.15.1 uses mria 0.5.2, which includes the following changes:
  - fix(mria_membership): call `mria_rlog:role/1` safely
  - feat: add extra field to ?rlog_sync table (for future use)
 2023-04-25 23:21:07 +03:00
Zhongwen Deng 308056f0fc feat: improved the storage format of Unicode characters in data files 2023-04-25 18:08:34 +08:00
zhongwencool 79a235f8b5
Merge pull request #10491 from zhongwencool/rename-etcd-ssl-to-ssl_options 
feat: rename etcd.ssl to etcd.ssl_options
 2023-04-25 10:49:09 +08:00
JianBo He 09b17000c8
chore: update changes/ce/feat-10457.en.md 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-25 09:56:40 +08:00
lafirest 29d3afd692
Merge pull request #10487 from lafirest/fix/limiter_instance 
fix(limiter): optimize the instance of limiter
 2023-04-25 09:43:04 +08:00
Serge Tupchii 99e892b5c4 chore: bump ekka to 0.15.0 
ekka 0.15.0 uses mria 0.5.0, which adds several fixes, enhancements and features:
  - protect `mria:join/1,2` with a global lock
  - implement new function `mria:sync_transaction/4,3,2`, which waits for
    a transaction replication to be ready on the local node
    (if the local node is a replicant)
  - optimize `mria:running_nodes/0`
  - optimize `mria:ro_transaction/2` when called on a replicant node.

Fixes: EMQX-9588 (#10380), EMQX-9102, EMQX-9152, EMQX-9213
 2023-04-24 20:52:20 +03:00
lafirest 2845469c48
Merge pull request #10490 from lafirest/fix/rmv_def_conn_limit 
fix(limiter): remove the default limit of connect rate
 2023-04-24 22:04:29 +08:00
JianBo He 4aad5c7454 chore: improve changes 2023-04-24 21:44:21 +08:00
Thales Macedo Garitezi ad4be08bb2 feat: implement Pulsar Producer bridge (e5.0) 
Fixes https://emqx.atlassian.net/browse/EMQX-8398
 2023-04-24 10:28:26 -03:00
firest 6110aad23f chore: bump version && update changes 2023-04-24 18:17:23 +08:00
zhongwencool 22d6a2eaee
Merge pull request #10484 from zhongwencool/fix-copy-conf 
fix: copy cluster-override.conf from old version
 2023-04-24 17:06:07 +08:00
Zaiming (Stone) Shi 366aa53c80
Merge pull request #10462 from zmstone/0430-EMQX-8434-shared-dispatch-ack-deprecated 
chore: Hide config shared_dispatch_ack_enabled
 2023-04-24 09:42:09 +02:00
Zhongwen Deng dacf92c4ab chore: rename etcd.ssl changelog 2023-04-24 14:46:06 +08:00
firest 24cecae1f8 chore: update changes 2023-04-24 14:15:45 +08:00
firest 2f2f32ac7b chore: update changes 2023-04-24 10:52:30 +08:00
某文 38cebf2fdc chore: add changelog for 10484 2023-04-23 15:53:17 +08:00
firest 6631fb7457 chore: update changes 2023-04-23 11:22:56 +08:00
Zaiming (Stone) Shi 81340edbca docs: add changelog 2023-04-22 09:08:07 +02:00
zhongwencool f831a0b827 chore: update changes/ce/fix-10449.en.md 
Co-authored-by: JianBo He <heeejianbo@163.com>
 2023-04-21 15:30:27 +08:00
Zhongwen Deng 397e28f5a4 chore: add changlog for authn_http validation 2023-04-21 15:30:05 +08:00
zhongwencool ad6090a778 chore: update changes/ce/fix-10449.en.md 
Co-authored-by: JianBo He <heeejianbo@163.com>
 2023-04-21 11:54:45 +08:00
Zhongwen Deng 2aef9ca215 chore: add changlog for authn_http validation 2023-04-21 11:54:45 +08:00
SergeTupchiy 0b105dcb8d
Merge pull request #10463 from SergeTupchiy/EMQX-9310-webhook-port-validation 
fix(emqx_bridge): validate Webhook bad URL
 2023-04-20 20:48:12 +03:00
Ilya Averyanov 8c4b32580e
Merge pull request #10420 from savonarola/0417-authn-authz-fix-path-quoting 
Fix path joining and quoting in authn/authz
 2023-04-20 18:03:04 +03:00
SergeTupchiy b38ae7f78f
Merge pull request #10407 from SergeTupchiy/EMQX-9529-resource-manager-crash-on-alarm-timeout 
fix(emqx_resource): call emqx_alarm safely and don't reactivate alarm on reoccurring errors
 2023-04-20 17:53:56 +03:00
Zaiming (Stone) Shi 82e6ce53be
Merge pull request #10460 from zmstone/0420-sync-release-50-to-masteer 
0420 sync release 50 to masteer
 2023-04-20 16:41:50 +02:00
Serge Tupchii 45254d7d85 fix(emqx_bridge): validate Webhook bad URL and return 'BAD_REQUEST' if it's invalid 
Fixes: EMQX-9310
 2023-04-20 17:21:43 +03:00
Serge Tupchii 423a30fbb3 fix(emqx_alarm): add safe call API to activate/deactivate alarms and use it in resource_manager 
Don't let 'emqx_resource_manager' crash because of emqx_alarm timeouts.

Fixes: EMQX-9529/#10357
 2023-04-20 17:15:13 +03:00
Thales Macedo Garitezi 3f18c5e2e3
Merge pull request #10455 from thalesmg/fix-late-gen-server-replies-buf-worker-v50 
fix(buffer_worker): avoid sending late reply messages to callers
 2023-04-20 10:19:06 -03:00
Zhongwen Deng bcc8f4313b chore: make plugins config to low level 2023-04-20 18:00:19 +08:00
zhongwencool 72685beb72
Merge pull request #10457 from HJianBo/hide-statsd 
chore: deprecate statsd
 2023-04-20 17:19:33 +08:00
Zaiming (Stone) Shi 415830a0a2 Merge remote-tracking branch 'origin/release-50' into 0420-sync-release-50-to-masteer 2023-04-20 11:03:39 +02:00
Zaiming (Stone) Shi c2385f85d6
Merge pull request #10444 from zmstone/0418-cherry-pick-fixes-for-e503 
0418 cherry pick fixes for e503
 2023-04-20 10:58:57 +02:00
Kjell Winblad 7d3367467a
Merge pull request #10408 from kjellwinblad/kjell/rule_engine/add_missing_mongo_date_functions/EMQX-9244 
feat: add mongo_date functions to the rule engine
 2023-04-20 09:46:45 +02:00
JianBo He 794ddd8d73 chore: update changes 2023-04-20 14:23:55 +08:00
Thales Macedo Garitezi cb995e2033 fix(buffer_worker): avoid sending late reply messages to callers 
Fixes https://emqx.atlassian.net/browse/EMQX-9635

During a sync call from process `A` to a buffer worker `B`, its call
to the underlying resource `C` can be very slow.  In those cases, `A`
will receive a timeout response and expect no more messages from `B`
nor `C`.  However, prior to this fix, if `B` is stuck in a long sync
call to `C` and then gets its response after `A` timed out, `B` would
still send the late response to `A`, polluting its mailbox.
 2023-04-19 18:27:10 -03:00
firest 8219af2fd4 chore: improve changelog 2023-04-19 16:45:53 +08:00
firest e89f4d4565 fix(dynamo): fix terminology erros 
- Changed `username` to `aws_access_key_id`
- Changed `password` to `aws_secret_access_key`
 2023-04-19 16:26:42 +08:00
firest b112f544a9 chore: update changes 2023-04-19 15:56:35 +08:00
firest 4f0c891aa6 chore: bump version && update changes 2023-04-19 15:45:31 +08:00
Thales Macedo Garitezi 6a1ef5e68a
Merge pull request #10422 from thalesmg/fix-plugin-sync-single-node-v50 
fix(plugins): attempt to extract plugin from current node on startup
 2023-04-18 14:39:02 -03:00
Zaiming (Stone) Shi 3e72d6c4a3 docs: sync changelogs 2023-04-18 19:17:03 +02:00
Ivan Dyachkov f8ff2d6bc9 docs: Generate changelog for v5.0.23 2023-04-18 18:09:24 +02:00
Ivan Dyachkov dc78ecb41c chore: merge upstream/master 2023-04-18 17:33:32 +02:00
Thales Macedo Garitezi 89cd6cfede fix(plugins): attempt to extract plugin from current node on startup 
Fixes https://emqx.atlassian.net/browse/EMQX-9605

Fixes https://github.com/emqx/emqx-elixir-plugin/issues/25

If an user happens to configure a plugin in a lone-node cluster via
environment variables, it would fail to start up as there are no other
nodes to copy the plugin from.  Here, we attempt to check if the
package is present in the current node but not yet extracted.
 2023-04-18 11:10:23 -03:00
某文 fbadfc06e4 feat: change exhook, rule_engine, bridge to low importance level instead of hidden 2023-04-18 20:15:19 +08:00
JianBo He e3a84e0010 chore: improve the changes for configuration 2023-04-18 20:13:44 +08:00
Zhongwen Deng 22a1d05d7b feat: hide ssl_options.user_lookup_fun 2023-04-18 20:13:44 +08:00
Zhongwen Deng e3d6fa1f21 chore: update config's changelog and emqx_conf.template 2023-04-18 20:13:44 +08:00
ieQu1 c0d8e9c402
Merge pull request #10369 from ieQu1/fix-stats-api 
fix(emqx_management): Ignore results from the nodes that are down
 2023-04-18 12:36:55 +02:00
Kjell Winblad 8836011310 docs: better changelog entry for mongo_date functions 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-18 10:01:35 +02:00
Kjell Winblad 29584ca721 feat: add mongo_date functions to the rule engine 
This commit adds mong_date built-in functions to the rule engine
SQL-like language. Corresponding functions already existed in EMQX 4.4
and this commit makes sure that EMQX 5.X also has these functions.

Fixes:
https://emqx.atlassian.net/browse/EMQX-9244
 2023-04-18 10:01:35 +02:00
firest 61f3e62ba8 chore: update changes 2023-04-18 15:49:34 +08:00
ieQu1 282bfee8ff feat(emqx): Add an API that returns the list of running nodes 2023-04-18 09:22:58 +02:00
ieQu1 a947df1ea3 fix(emqx_management): Ignore results from the nodes that are down 2023-04-18 09:22:56 +02:00
JianBo He 77d300482c
chore: update changes/ce/feat-10389.en.md 2023-04-18 10:49:51 +08:00
zhongwencool acb30c7957
chore: update changes/ce/feat-10389.en.md 
Co-authored-by: JianBo He <heeejianbo@163.com>
 2023-04-18 10:47:07 +08:00
zhongwencool ad3e529994 chore: update changes/ce/feat-10389.en.md 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-04-18 09:34:01 +08:00
某文 1ef74f1ada chore: upgrade hocon to 0.39.1 2023-04-18 09:28:36 +08:00
Ilya Averyanov 88ca94b417 fix(auth): fix uri path handling 
Fix uri path handling `emqx_connector_http`,
 HTTP authentication and authorization backends.
 2023-04-17 23:57:56 +03:00
Thales Macedo Garitezi 586cd54aa2
Merge pull request #10409 from thalesmg/protobuf-schema-v50 
feat(schema_registry): add support for protobuf schemas
 2023-04-17 14:51:51 -03:00
Kjell Winblad 0c727fc9b2
Merge pull request #10392 from kjellwinblad/kjell/rule_engine/add_missing_datetime_function/EMQX-9245 
freat: add date_to_unix_ts/3 function to the rule engine
 2023-04-17 18:09:44 +02:00
Thales Macedo Garitezi 9d15247dd5 feat(schema_registry): add support for protobuf schemas 
Fixes https://emqx.atlassian.net/browse/EMQX-9470
 2023-04-17 13:04:54 -03:00
JianBo He 3bf642cdff chore: update changes/ce/fix-10410.en.md 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-17 13:57:02 +02:00
JianBo He 069afd042a chore: update changes 2023-04-17 13:56:50 +02:00
Ivan Dyachkov 9712aad7a8
Merge pull request #10413 from id/0416-e5.0.3-code-freeze 
0416 e5.0.3 code freeze
 2023-04-17 13:51:58 +02:00
JianBo He 19eda3bc5f
Merge pull request #10410 from HJianBo/fix-load-gw-from-confs 
fix(gw): load emqx applications before hocon configs checking
 2023-04-17 18:01:31 +08:00
JianBo He 549283ae75
chore: update changes/ce/fix-10410.en.md 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-04-17 17:35:50 +08:00
firest 2a54d93c7e chore: update changes 2023-04-17 10:06:43 +08:00
zhongwencool 69d1a35c90
Merge pull request #10156 from zhongwencool/conf-refactor 
feat: configuration priority ENV > emqx.conf > API
 2023-04-16 13:45:35 +08:00
Zhongwen Deng ad65fefac2 fix: rename emqx_misc to emqx_utils 2023-04-16 10:27:23 +08:00
zhongwencool 454d244535
chore: update changes/ce/feat-10156.en.md 
Co-authored-by: JianBo He <heeejianbo@163.com>
 2023-04-16 09:58:22 +08:00
JianBo He 73c15d9668 chore: update changes 2023-04-15 21:47:38 +08:00
zhongwencool e1faae451a
chore: Update changes/ce/feat-10156.en.md 
Co-authored-by: JianBo He <heeejianbo@163.com>
 2023-04-15 18:17:28 +08:00
zhongwencool 494ffe86ad
Merge branch 'master' into conf-refactor 2023-04-15 06:40:38 +08:00
Zhongwen Deng 8facd130f6 chore: update cluster_hocon_file/0 function 2023-04-14 23:19:38 +08:00
Thales Macedo Garitezi 14ed4a7ada feat(buffer_worker): set default queue mode to `memory_only` 
Fixes https://emqx.atlassian.net/browse/EMQX-9367

For better user experience and performance for the average bridge, we
should change the default queue mode to `memory_only`, as was the
behavior of most bridges in e4.x.  This leads to better performance
when message rate is high enough and the remote resource is not
keeping up with EMQX.

Also, we set the default segment size to equal max queue bytes.
 2023-04-14 11:37:19 -03:00
JimMoen 790d841697
Merge pull request #10363 from JimMoen/feat-mssqlserver-birdge 
feat: implement Microsoft SQL Server bridge (e5.0)
 2023-04-14 17:55:48 +08:00
Andrew Mayorov 5e92ba6fa9
Merge pull request #10359 from ft/EMQX-9136/no-ask-metrics 
feat(resource): ask for metrics only when needed
 2023-04-14 12:28:52 +03:00
zhongwencool 1e07f37ab0
Merge pull request #10354 from zhongwencool/straightforward-log-info 
feat: more straightforward log for force_shutdown reason
 2023-04-14 10:15:32 +08:00
JimMoen 35ab1c0190
style: make hocon style check and spellcheck happy 
- with more human readable i18n style check format
 2023-04-14 10:02:47 +08:00
JimMoen 5841969877
feat: implement Microsoft SQL Server bridge (e5.0) 2023-04-14 10:02:45 +08:00
zhongwencool 0553facfca
Merge pull request #10391 from zhongwencool/04-13-hide-conf 
feat: hide ex_hook/rewrite/topic_metric/persistent_session_store
 2023-04-14 09:39:29 +08:00
zhongwencool 626e814e9a
Merge pull request #10385 from zhongwencool/hide-conf 
feat: hide data items from configuration files and documentation.
 2023-04-14 09:38:34 +08:00
zhongwencool 835f66a8d2
Merge pull request #10375 from zhongwencool/trace-conf 
feat: deprecated trace config
 2023-04-14 09:22:12 +08:00
Kjell Winblad 601564fa0a
Merge pull request #10324 from kjellwinblad/kjell/fix/EMQX-9374 
fix: check Clickhouse connection after creation
 2023-04-13 16:31:35 +02:00
Kjell Winblad d3ccd8a65d feat: add date_to_unix_ts/3 function to the rule engine 
Fixes:
https://emqx.atlassian.net/browse/EMQX-9245
 2023-04-13 16:14:03 +02:00