yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,286 Commits 393 Branches 684 Tags 74 MiB
c6a78cbfda
Commit Graph

6197 Commits

Author SHA1 Message Date
Ilya Averyanov
9d91ebe266
Merge pull request #9842 from savonarola/fix-redis-cluster-recover 
fix: fix redis cluster resource recovery
 2023-02-01 10:38:52 +02:00
Zaiming (Stone) Shi
f4381d90ca refactor: http cliet should not need to retry HTTP requests for bridge 2023-02-01 09:30:30 +01:00
Zaiming (Stone) Shi
6ca816f7a4 docs: remove impl details from user doc 2023-02-01 09:30:10 +01:00
Zhongwen Deng
3c4d09a752 fix: get_memory_cache return {ok, Mem} | stale 2023-02-01 16:00:55 +08:00
Zhongwen Deng
ced55719ef chore: only run t_sys_mem_check ct in linux 2023-02-01 14:47:41 +08:00
Zhongwen Deng
56b9238645 fix: only cache sys_memory in linux 2023-02-01 12:21:34 +08:00
Zhongwen Deng
95ef1e806c chore: don't crash when add_handler failed 2023-02-01 11:24:27 +08:00
Zhongwen Deng
9cbad5297c fix: primary logger level not update 2023-02-01 11:16:30 +08:00
Zhongwen Deng
71f00f2962 test: ctl suite failed 2023-02-01 10:07:22 +08:00
Zhongwen Deng
7a9f477267 feat: use emqx_mgmt_cache to cache sys_memory 2023-02-01 10:07:22 +08:00
Zhongwen Deng
c2bdb9faa7 test: multiple_nodes case failed 2023-02-01 10:07:22 +08:00
Zhongwen Deng
b6e6315b50 feat: change loads from string to float 2023-02-01 10:07:22 +08:00
Zhongwen Deng
0b19be074c feat: cache OTP_VERSION in persistent_term 2023-02-01 10:07:22 +08:00
Zhongwen Deng
6162f90610 fix: don't crash when OTP_VERSION file is missing 2023-02-01 10:07:22 +08:00
Zhongwen Deng
5783127c30 test: cpu_sup:load mock test 2023-02-01 10:07:22 +08:00
Zhongwen Deng
2d67bb3fb6 fix: /api/nodes is timeout if emqx in high load 2023-02-01 10:07:22 +08:00
Zaiming (Stone) Shi
7e8253e3af chore: bump version to e5.0.0-rc.3 2023-01-31 20:45:28 +01:00
Zaiming (Stone) Shi
78028a7fcf
Merge pull request #9876 from zmstone/0131-avoid-persisting-logging-configs-if-no-change 
0131 avoid persisting logging configs if no change
 2023-01-31 20:43:32 +01:00
Zaiming (Stone) Shi
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback 
feat(mqtt-bridge): avoid middleman process
 2023-01-31 19:12:18 +01:00
Zaiming (Stone) Shi
ddfbc0a19f refactor: refresh logger config before starting listeners 2023-01-31 18:50:16 +01:00
Zaiming (Stone) Shi
605d9972e4 fix(config): avoid calling emqx:update_config/2 for logger refresh 
Prior to this fix, whatever configured for loggers are persisted to
cluster-override.conf.
This may cause troulbe for users who changes boot mode.

For example if the node is once started in console mode,
the logging config is persisted, with console enabled, but file
disabled.
Then if the user decide to start in daemon mode, all the logs
will silently go to erlang.log.N and emqx.log.N will be empty.

After this fix, only changes really made into cluter-override.conf
will take effect.
 2023-01-31 18:50:16 +01:00
Zaiming (Stone) Shi
b0ac924ca9 refactor: less copy-paste 2023-01-31 18:40:55 +01:00
lafirest
3d59be6ab6
Merge pull request #9871 from lafirest/fix/authz_rules 
feat(authz): allow the placeholder to be anywhere in the topic for authz rules
 2023-02-01 00:12:06 +08:00
Andrew Mayorov
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter 2023-01-31 18:30:42 +03:00
Andrew Mayorov
c76311c9c3
fix(buffer): count inflight batches properly 2023-01-31 18:30:42 +03:00
firest
9f4c36ecbc chore: bump version && update changes 2023-01-31 23:30:22 +08:00
firest
c034cbf6de feat(authz): allow the placeholder to be anywhere in the topic for authz rules 2023-01-31 23:30:12 +08:00
Andrew Mayorov
d0c10b59aa
feat(mqtt-bridge): avoid middleman process 
Instead, supervise `emqtt` client process directly.
 2023-01-31 17:59:03 +03:00
Ilya Averyanov
fce1e74c3d fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00
Zaiming (Stone) Shi
f6dafc20ea feat: export API to read cluster and local overrides 2023-01-31 14:20:34 +01:00
Stefan Strigler
ade5316419
Merge pull request #9802 from sstrigler/EMQX-8469-enable-proxy-protocol-support-in-emqx-dashboard-schema 
feat: support HAProxy protocol for dashboard API
 2023-01-31 14:19:56 +01:00
Erik Timan
3786bb8086
Merge pull request #9824 from olcai/internal-error-api-get-topic 
fix(emqx_management): handle multiple routes in topics/{topic} API
 2023-01-31 13:50:42 +01:00
Andrew Mayorov
4d146c521b
fix(mqtt-bridge): ensure proper feedback on async forwards 
So that buffer worker would notice a connection loss in time, and
recycle inflight messages subsequently.
 2023-01-31 11:09:39 +03:00
Andrew Mayorov
35c429ef1d
refactor: drop a couple of unused macros / includes 2023-01-31 11:09:39 +03:00
Zaiming (Stone) Shi
b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback 
0127 refactor buffer worker no need to keep request for reply callback
 2023-01-31 08:44:01 +01:00
Erik Timan
5b3a77e3c7 test(emqx_management): fix flaky route handling in get topic test 
This reworks a test case to use a second slave node. This ensures that
an added route is permanently in the routing table. The old version
reverted the routing table quickly since the node name given wasn't a
real node.
 2023-01-31 08:28:55 +01:00
Erik Timan
03cabf6b26 chore: bump app VSN 2023-01-31 08:27:54 +01:00
Erik Timan
33e011aff5 fix(emqx_management): handle multiple routes in topics/{topic} API 
The topics/{topic} API endpoint would return 500 - Internal Error if a
topic had multiple routes. This is now fixed by returning a list of
routes.
 2023-01-31 08:27:33 +01:00
Ilya Averyanov
1a1c3afa81
Merge pull request #9852 from savonarola/fix-redis-desc 
Fix Redis configuration field description
 2023-01-30 23:26:37 +02:00
Stefan Strigler
3ce6cbcd6a chore: bump vsn 2023-01-30 18:09:10 +01:00
Stefan Strigler
0b1483040a test: add actual test for haproxy 2023-01-30 18:07:06 +01:00
lafirest
e288424924
Merge pull request #9864 from lafirest/fix/exclusive-unsub-5.0 
fix: the exclusive topics aren't removed when the session has already been cleaned
 2023-01-31 00:33:45 +08:00
Kjell Winblad
0c4134c423 test: add unit test case for redact function in http connector 2023-01-30 11:51:28 +01:00
Stefan Strigler
27881064dc fix: increase dropped.queue_full by number of messages 2023-01-30 11:37:35 +01:00
lafirest
3e6cc40525
Merge pull request #9843 from lafirest/fix/speed_up_test 
fix: speed up banned test cases
 2023-01-30 18:17:27 +08:00
firest
3bdffca488 fix: the exclusive topics aren't removed when the session has already been cleaned 2023-01-30 18:17:00 +08:00
Zhongwen Deng
bb636394e1 chore: add debug log for raw data 2023-01-30 16:35:06 +08:00
Zhongwen Deng
b73d11675e chore: log the bad mqtt packet(frame error) 2023-01-30 15:05:49 +08:00
Zhongwen Deng
3d07271ea5 fix: crash when mfa not found 2023-01-30 14:30:20 +08:00
Zhongwen Deng
25090563af chore: use brackets to wrap the mqtt packet when logging 2023-01-30 12:29:00 +08:00
Zhongwen Deng
ce32ea7334 feat: Make the log output format order fixed 2023-01-30 12:18:22 +08:00
Zhongwen Deng
6cbad047cd fix: don't log CONNECT twice when debug 2023-01-30 12:16:17 +08:00
Zaiming (Stone) Shi
d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
fc38ea9571 refactor(buffer_worker): do not keep request body in reply context 
the request body can be potentially very large
the reply context is sent to the async call handler and kept
in its memory until the async reply is received from bridge
target service.

this commit tries to minimize the size of the reply context
by replacing the request body with `[]`.
 2023-01-27 17:12:55 +01:00
Ilya Averyanov
72f39b9b72 fix(docs): correct Redis conf field description 2023-01-27 17:39:16 +02:00
Zaiming (Stone) Shi
578271ea3d refactor: use lists:map instead of lc for safty 2023-01-27 15:15:46 +01:00
Zaiming (Stone) Shi
f793807bc1 refactor(buffer_worker): rename function 
batch_reply_after_query to handle_async_batch_reply
 2023-01-27 15:04:28 +01:00
Zaiming (Stone) Shi
262c3a2869 refactor(buffer_worker): rename function 
from reply_after_query to handle_async_reply
 2023-01-27 15:03:18 +01:00
Zaiming (Stone) Shi
6a58bafcb0 chore: bump release version to e5.0.0-rc.2 2023-01-27 14:38:21 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi
514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak 
0127 fix influxdb bridge atom leak
 2023-01-27 14:30:20 +01:00
Zaiming (Stone) Shi
96ed725a55
Merge pull request #9849 from zmstone/0127-refactor-buffer-worker-simplify-caller-reply 
0127 refactor buffer worker simplify caller reply
 2023-01-27 14:06:56 +01:00
Zaiming (Stone) Shi
c47be57c59 fix(bridge): ensure all bridge resources are stopped before app stop 2023-01-27 12:39:05 +01:00
Zaiming (Stone) Shi
d53106145f fix: stop resource when resource manager terminates 2023-01-27 12:39:05 +01:00
Andrew Mayorov
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching 
feat(redis): disable batching in redis_cluster bridges
 2023-01-27 15:27:57 +04:00
Stefan Strigler
7005b71ddf style: fix typo in comment 2023-01-27 11:43:51 +01:00
Stefan Strigler
2d62de5188 test: fix expected result from timeout error 2023-01-27 11:43:48 +01:00
Stefan Strigler
a180bd9aa5 fix: catch error, not exit 2023-01-27 11:40:06 +01:00
Stefan Strigler
b7e3f9d5a6 fix: try-case-of rather than try-of 
try-of catches only what happens within but not after
 2023-01-27 11:40:06 +01:00
Zaiming (Stone) Shi
db2f631a8a refactor(buffer_worker): simplify caller reply 2023-01-27 11:33:45 +01:00
Zaiming (Stone) Shi
965236c888
Merge pull request #9845 from zmstone/0126-reply-caller-for-buffer-overflow-queue-items 
0126 reply caller for buffer overflow queue items
 2023-01-27 11:32:59 +01:00
Zaiming (Stone) Shi
d4fab92b72 refactor(buffer_worker): no need to keep request for REPLY macro 2023-01-27 10:41:30 +01:00
Andrew Mayorov
71f996b9d5
refactor(mqtt-bridge): unwrap single statem actions 
So that the code would be easier to follow and harder to break.
Also drop a couple of unused macrodefs.
 2023-01-26 21:13:18 +03:00
Andrew Mayorov
c95f979413
fix(mqtt_bridge): use correct gen_statem reply action 2023-01-26 21:12:05 +03:00
Zaiming (Stone) Shi
1f799dfd59 fix: reply with {error, buffer_overflow} when discarded 2023-01-26 17:15:36 +01:00
Zaiming (Stone) Shi
ed28789164 refactor(buffer_worker): no need to return after collect into buf queue 2023-01-26 14:50:40 +01:00
Zaiming (Stone) Shi
25b4821adc refactor: move the the per-message overflow log from error to info level 2023-01-26 14:48:43 +01:00
Zaiming (Stone) Shi
bb26632c8a fix(buffer_worker): fix a wrong assertion 
the assertion is to ensure queue items are not binary
but should not assert the queue itself
 2023-01-26 14:33:16 +01:00
Zaiming (Stone) Shi
f6b3b930b0 chore: improve a error log 2023-01-26 14:21:27 +01:00
Kjell Winblad
86cfbfb43c fix: Authorization header leak in log entries for webhook 
There might be another possibility for leakage. If the resource mangager
for the webhook resource crashes, OTP might log the spec for the
resource manager which contains the Config and thus the Authorization
header. This is probably an issue for other resources as well and should
be fixed in another commit. The following issue has been created for
that:

https://emqx.atlassian.net/browse/EMQX-8794

Fixes:
https://emqx.atlassian.net/browse/EMQX-8791
 2023-01-26 12:44:16 +01:00
firest
908ac8d948 fix: speed up banned test cases 2023-01-26 13:04:55 +08:00
Andrew Mayorov
2ee00b75a7
fix(redis): unwrap pipeline queries against redis cluster 
This is an additional safety measure in addition to the disabled
batching on the bridge level.
 2023-01-25 17:28:11 +03:00
Erik Timan
805d08e823 fix: reduce log level from error to warning in several places 
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
 2023-01-25 14:49:50 +01:00
Erik Timan
8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Zaiming (Stone) Shi
6175076f6f
Merge pull request #9835 from olcai/add-influxdb-test-files 
fix: add influxdb test files and fixes
 2023-01-24 17:02:23 +01:00
Zaiming (Stone) Shi
e5b65087af
Merge pull request #9834 from zmstone/0123-fix-idle_timeout-infinity 
fix(emqx_connection): crash when idle_timeout is set to infinity
 2023-01-24 16:05:09 +01:00
Erik Timan
9d20431257 fix(emqx_resource): fix crash while flushing queue 
We used next_event for flushing the queue in emqx_resource, but this
leads to a crash. We now call flush_worker/1 instead.
 2023-01-24 14:13:35 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Zaiming (Stone) Shi
7575120ea6 test: use snabbkaffe retry macro 2023-01-24 10:54:20 +01:00
Zaiming (Stone) Shi
140cda2f13 fix(emqx_connection): crash when idle_timeout is set to infinity 2023-01-24 10:14:35 +01:00
Zaiming (Stone) Shi
a0100c0a4e chore: bump app versions 2023-01-23 16:20:21 +01:00
Zaiming (Stone) Shi
a83ed2fe94 Merge remote-tracking branch 'origin/master' into 0123-chore-sync-release-50-to-master 2023-01-23 16:20:01 +01:00
Zaiming (Stone) Shi
727100e094 chore: prepare for v5.0.15 release 2023-01-20 16:42:01 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi
57607ca0ce chore: prepare for v5.0.15 release 2023-01-20 11:20:34 +01:00
Zaiming (Stone) Shi
1c3e055b13
Merge pull request #9822 from JimMoen/fix-schema-typo 
chore: i18n typo fix
 2023-01-20 11:11:18 +01:00