yuanbiao/emqx
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,514 Commits 393 Branches 669 Tags 74 MiB
a6d88c3caa
Commit Graph

6338 Commits

Author SHA1 Message Date
Zhongwen Deng
7a9f477267 feat: use emqx_mgmt_cache to cache sys_memory 2023-02-01 10:07:22 +08:00
Zhongwen Deng
c2bdb9faa7 test: multiple_nodes case failed 2023-02-01 10:07:22 +08:00
Zhongwen Deng
b6e6315b50 feat: change loads from string to float 2023-02-01 10:07:22 +08:00
Zhongwen Deng
0b19be074c feat: cache OTP_VERSION in persistent_term 2023-02-01 10:07:22 +08:00
Zhongwen Deng
6162f90610 fix: don't crash when OTP_VERSION file is missing 2023-02-01 10:07:22 +08:00
Zhongwen Deng
5783127c30 test: cpu_sup:load mock test 2023-02-01 10:07:22 +08:00
Zhongwen Deng
2d67bb3fb6 fix: /api/nodes is timeout if emqx in high load 2023-02-01 10:07:22 +08:00
Zaiming (Stone) Shi
7e8253e3af chore: bump version to e5.0.0-rc.3 2023-01-31 20:45:28 +01:00
Zaiming (Stone) Shi
78028a7fcf
Merge pull request #9876 from zmstone/0131-avoid-persisting-logging-configs-if-no-change 
0131 avoid persisting logging configs if no change
 2023-01-31 20:43:32 +01:00
Zaiming (Stone) Shi
b3ad9e97d2
Merge pull request #9870 from keynslug/fix/mqtt-connection-loss-feedback 
feat(mqtt-bridge): avoid middleman process
 2023-01-31 19:12:18 +01:00
Zaiming (Stone) Shi
ddfbc0a19f refactor: refresh logger config before starting listeners 2023-01-31 18:50:16 +01:00
Zaiming (Stone) Shi
605d9972e4 fix(config): avoid calling emqx:update_config/2 for logger refresh 
Prior to this fix, whatever configured for loggers are persisted to
cluster-override.conf.
This may cause troulbe for users who changes boot mode.

For example if the node is once started in console mode,
the logging config is persisted, with console enabled, but file
disabled.
Then if the user decide to start in daemon mode, all the logs
will silently go to erlang.log.N and emqx.log.N will be empty.

After this fix, only changes really made into cluter-override.conf
will take effect.
 2023-01-31 18:50:16 +01:00
Zaiming (Stone) Shi
b0ac924ca9 refactor: less copy-paste 2023-01-31 18:40:55 +01:00
lafirest
3d59be6ab6
Merge pull request #9871 from lafirest/fix/authz_rules 
feat(authz): allow the placeholder to be anywhere in the topic for authz rules
 2023-02-01 00:12:06 +08:00
Andrew Mayorov
c5a7cd5acd
fix(mqtt-bridge): drop unused configuration parameter 2023-01-31 18:30:42 +03:00
Andrew Mayorov
c76311c9c3
fix(buffer): count inflight batches properly 2023-01-31 18:30:42 +03:00
firest
9f4c36ecbc chore: bump version && update changes 2023-01-31 23:30:22 +08:00
firest
c034cbf6de feat(authz): allow the placeholder to be anywhere in the topic for authz rules 2023-01-31 23:30:12 +08:00
Andrew Mayorov
d0c10b59aa
feat(mqtt-bridge): avoid middleman process 
Instead, supervise `emqtt` client process directly.
 2023-01-31 17:59:03 +03:00
Ilya Averyanov
fce1e74c3d fix(connector): fix redis cluster resource recovery 2023-01-31 16:55:05 +02:00
Zaiming (Stone) Shi
f6dafc20ea feat: export API to read cluster and local overrides 2023-01-31 14:20:34 +01:00
Stefan Strigler
ade5316419
Merge pull request #9802 from sstrigler/EMQX-8469-enable-proxy-protocol-support-in-emqx-dashboard-schema 
feat: support HAProxy protocol for dashboard API
 2023-01-31 14:19:56 +01:00
Erik Timan
3786bb8086
Merge pull request #9824 from olcai/internal-error-api-get-topic 
fix(emqx_management): handle multiple routes in topics/{topic} API
 2023-01-31 13:50:42 +01:00
Andrew Mayorov
4d146c521b
fix(mqtt-bridge): ensure proper feedback on async forwards 
So that buffer worker would notice a connection loss in time, and
recycle inflight messages subsequently.
 2023-01-31 11:09:39 +03:00
Andrew Mayorov
35c429ef1d
refactor: drop a couple of unused macros / includes 2023-01-31 11:09:39 +03:00
Zaiming (Stone) Shi
b3e486041b
Merge pull request #9853 from zmstone/0127-refactor-buffer-worker-no-need-to-keep-request-for-reply-callback 
0127 refactor buffer worker no need to keep request for reply callback
 2023-01-31 08:44:01 +01:00
Erik Timan
5b3a77e3c7 test(emqx_management): fix flaky route handling in get topic test 
This reworks a test case to use a second slave node. This ensures that
an added route is permanently in the routing table. The old version
reverted the routing table quickly since the node name given wasn't a
real node.
 2023-01-31 08:28:55 +01:00
Erik Timan
03cabf6b26 chore: bump app VSN 2023-01-31 08:27:54 +01:00
Erik Timan
33e011aff5 fix(emqx_management): handle multiple routes in topics/{topic} API 
The topics/{topic} API endpoint would return 500 - Internal Error if a
topic had multiple routes. This is now fixed by returning a list of
routes.
 2023-01-31 08:27:33 +01:00
Ilya Averyanov
1a1c3afa81
Merge pull request #9852 from savonarola/fix-redis-desc 
Fix Redis configuration field description
 2023-01-30 23:26:37 +02:00
Stefan Strigler
3ce6cbcd6a chore: bump vsn 2023-01-30 18:09:10 +01:00
Stefan Strigler
0b1483040a test: add actual test for haproxy 2023-01-30 18:07:06 +01:00
lafirest
e288424924
Merge pull request #9864 from lafirest/fix/exclusive-unsub-5.0 
fix: the exclusive topics aren't removed when the session has already been cleaned
 2023-01-31 00:33:45 +08:00
Kjell Winblad
0c4134c423 test: add unit test case for redact function in http connector 2023-01-30 11:51:28 +01:00
Stefan Strigler
27881064dc fix: increase dropped.queue_full by number of messages 2023-01-30 11:37:35 +01:00
lafirest
3e6cc40525
Merge pull request #9843 from lafirest/fix/speed_up_test 
fix: speed up banned test cases
 2023-01-30 18:17:27 +08:00
firest
3bdffca488 fix: the exclusive topics aren't removed when the session has already been cleaned 2023-01-30 18:17:00 +08:00
Zhongwen Deng
bb636394e1 chore: add debug log for raw data 2023-01-30 16:35:06 +08:00
Zhongwen Deng
b73d11675e chore: log the bad mqtt packet(frame error) 2023-01-30 15:05:49 +08:00
Zhongwen Deng
3d07271ea5 fix: crash when mfa not found 2023-01-30 14:30:20 +08:00
Zhongwen Deng
25090563af chore: use brackets to wrap the mqtt packet when logging 2023-01-30 12:29:00 +08:00
Zhongwen Deng
ce32ea7334 feat: Make the log output format order fixed 2023-01-30 12:18:22 +08:00
Zhongwen Deng
6cbad047cd fix: don't log CONNECT twice when debug 2023-01-30 12:16:17 +08:00
Zaiming (Stone) Shi
d47941601d refactor(buffer_worker): rename trace points 2023-01-28 11:52:11 +01:00
Zaiming (Stone) Shi
7f66c6a9e2
Merge pull request #9840 from olcai/redact-influxdb-tokens 
fix: redact influxdb tokens in logs and reduce log level
 2023-01-28 11:47:36 +01:00
Zaiming (Stone) Shi
fc38ea9571 refactor(buffer_worker): do not keep request body in reply context 
the request body can be potentially very large
the reply context is sent to the async call handler and kept
in its memory until the async reply is received from bridge
target service.

this commit tries to minimize the size of the reply context
by replacing the request body with `[]`.
 2023-01-27 17:12:55 +01:00
Ilya Averyanov
72f39b9b72 fix(docs): correct Redis conf field description 2023-01-27 17:39:16 +02:00
Zaiming (Stone) Shi
578271ea3d refactor: use lists:map instead of lc for safty 2023-01-27 15:15:46 +01:00
Zaiming (Stone) Shi
f793807bc1 refactor(buffer_worker): rename function 
batch_reply_after_query to handle_async_batch_reply
 2023-01-27 15:04:28 +01:00
Zaiming (Stone) Shi
262c3a2869 refactor(buffer_worker): rename function 
from reply_after_query to handle_async_reply
 2023-01-27 15:03:18 +01:00
Zaiming (Stone) Shi
6a58bafcb0 chore: bump release version to e5.0.0-rc.2 2023-01-27 14:38:21 +01:00
Zaiming (Stone) Shi
52b75ada04
Merge pull request #9832 from sstrigler/EMQX-8774-failure-to-handle-timeout-error-in-resource-worker 
EMQX 8774 failure to handle timeout error in resource worker
 2023-01-27 14:36:44 +01:00
Zaiming (Stone) Shi
514609bcf7
Merge pull request #9850 from zmstone/0127-fix-influxdb-bridge-atom-leak 
0127 fix influxdb bridge atom leak
 2023-01-27 14:30:20 +01:00
Zaiming (Stone) Shi
96ed725a55
Merge pull request #9849 from zmstone/0127-refactor-buffer-worker-simplify-caller-reply 
0127 refactor buffer worker simplify caller reply
 2023-01-27 14:06:56 +01:00
Zaiming (Stone) Shi
c47be57c59 fix(bridge): ensure all bridge resources are stopped before app stop 2023-01-27 12:39:05 +01:00
Zaiming (Stone) Shi
d53106145f fix: stop resource when resource manager terminates 2023-01-27 12:39:05 +01:00
Andrew Mayorov
d35e46b2d5
Merge pull request #9838 from keynslug/fix/redis-cluster-batching 
feat(redis): disable batching in redis_cluster bridges
 2023-01-27 15:27:57 +04:00
Stefan Strigler
7005b71ddf style: fix typo in comment 2023-01-27 11:43:51 +01:00
Stefan Strigler
2d62de5188 test: fix expected result from timeout error 2023-01-27 11:43:48 +01:00
Stefan Strigler
a180bd9aa5 fix: catch error, not exit 2023-01-27 11:40:06 +01:00
Stefan Strigler
b7e3f9d5a6 fix: try-case-of rather than try-of 
try-of catches only what happens within but not after
 2023-01-27 11:40:06 +01:00
Zaiming (Stone) Shi
db2f631a8a refactor(buffer_worker): simplify caller reply 2023-01-27 11:33:45 +01:00
Zaiming (Stone) Shi
965236c888
Merge pull request #9845 from zmstone/0126-reply-caller-for-buffer-overflow-queue-items 
0126 reply caller for buffer overflow queue items
 2023-01-27 11:32:59 +01:00
Zaiming (Stone) Shi
d4fab92b72 refactor(buffer_worker): no need to keep request for REPLY macro 2023-01-27 10:41:30 +01:00
Andrew Mayorov
71f996b9d5
refactor(mqtt-bridge): unwrap single statem actions 
So that the code would be easier to follow and harder to break.
Also drop a couple of unused macrodefs.
 2023-01-26 21:13:18 +03:00
Andrew Mayorov
c95f979413
fix(mqtt_bridge): use correct gen_statem reply action 2023-01-26 21:12:05 +03:00
Zaiming (Stone) Shi
1f799dfd59 fix: reply with {error, buffer_overflow} when discarded 2023-01-26 17:15:36 +01:00
Zaiming (Stone) Shi
ed28789164 refactor(buffer_worker): no need to return after collect into buf queue 2023-01-26 14:50:40 +01:00
Zaiming (Stone) Shi
25b4821adc refactor: move the the per-message overflow log from error to info level 2023-01-26 14:48:43 +01:00
Zaiming (Stone) Shi
bb26632c8a fix(buffer_worker): fix a wrong assertion 
the assertion is to ensure queue items are not binary
but should not assert the queue itself
 2023-01-26 14:33:16 +01:00
Zaiming (Stone) Shi
f6b3b930b0 chore: improve a error log 2023-01-26 14:21:27 +01:00
Kjell Winblad
86cfbfb43c fix: Authorization header leak in log entries for webhook 
There might be another possibility for leakage. If the resource mangager
for the webhook resource crashes, OTP might log the spec for the
resource manager which contains the Config and thus the Authorization
header. This is probably an issue for other resources as well and should
be fixed in another commit. The following issue has been created for
that:

https://emqx.atlassian.net/browse/EMQX-8794

Fixes:
https://emqx.atlassian.net/browse/EMQX-8791
 2023-01-26 12:44:16 +01:00
firest
908ac8d948 fix: speed up banned test cases 2023-01-26 13:04:55 +08:00
Andrew Mayorov
2ee00b75a7
fix(redis): unwrap pipeline queries against redis cluster 
This is an additional safety measure in addition to the disabled
batching on the bridge level.
 2023-01-25 17:28:11 +03:00
Erik Timan
805d08e823 fix: reduce log level from error to warning in several places 
This reduces the log level from error to warning in places that are
connected to the influxdb bridge. Transient errors for external
resources should not render an error log.
 2023-01-25 14:49:50 +01:00
Erik Timan
8836494542 fix: redact influxdb tokens in a few logs 2023-01-25 14:48:32 +01:00
Zaiming (Stone) Shi
5fdf7fd24c fix(kafka): use async callback to bump success counters 
some telemetry events from wolff are discarded:

* dropped:
    this is double counted in wolff,
    we now only subscribe to the dropped_queue_full event
* retried_failed:
    it has different meanings in wolff,
    in wolff, it means it's the 2nd (or onward) produce attempt
    in EMQX, it means it's eventually failed after some retries

* retried_success
    since we are going to handle the success counters in callbac
    this having this reported from wolff will only make things
    harder to understand

* failed
    wolff never fails (unelss drop which is a different counter)
 2023-01-24 21:12:36 +01:00
Zaiming (Stone) Shi
6175076f6f
Merge pull request #9835 from olcai/add-influxdb-test-files 
fix: add influxdb test files and fixes
 2023-01-24 17:02:23 +01:00
Zaiming (Stone) Shi
e5b65087af
Merge pull request #9834 from zmstone/0123-fix-idle_timeout-infinity 
fix(emqx_connection): crash when idle_timeout is set to infinity
 2023-01-24 16:05:09 +01:00
Erik Timan
9d20431257 fix(emqx_resource): fix crash while flushing queue 
We used next_event for flushing the queue in emqx_resource, but this
leads to a crash. We now call flush_worker/1 instead.
 2023-01-24 14:13:35 +01:00
Erik Timan
28718edbfd chore: bump application VSNs 2023-01-24 14:12:34 +01:00
Zaiming (Stone) Shi
8fde169abb
Merge pull request #9821 from thalesmg/buffer-worker-expiry-v50 
feat(buffer_worker): add expiration time to requests
 2023-01-24 13:54:04 +01:00
Zaiming (Stone) Shi
7575120ea6 test: use snabbkaffe retry macro 2023-01-24 10:54:20 +01:00
Zaiming (Stone) Shi
140cda2f13 fix(emqx_connection): crash when idle_timeout is set to infinity 2023-01-24 10:14:35 +01:00
Zaiming (Stone) Shi
a0100c0a4e chore: bump app versions 2023-01-23 16:20:21 +01:00
Zaiming (Stone) Shi
a83ed2fe94 Merge remote-tracking branch 'origin/master' into 0123-chore-sync-release-50-to-master 2023-01-23 16:20:01 +01:00
Zaiming (Stone) Shi
727100e094 chore: prepare for v5.0.15 release 2023-01-20 16:42:01 +01:00
Thales Macedo Garitezi
ca4a262b75 refactor: re-organize dealing with unrecoverable errors 2023-01-20 12:00:17 -03:00
Thales Macedo Garitezi
6fa6c679bb feat(buffer_worker): add expiration time to requests 
With this, we avoid performing work or replying to callers that are no
longer waiting on a result.

Also introduces two new counters:

- `dropped.expired` :: happens when a request expires before being
  sent downstream
- `late_reply` :: when a response is receive from downstream, but the
  caller is no longer for a reply because the request has expired, and
  the caller might even have retried it.
 2023-01-20 11:36:52 -03:00
Zaiming (Stone) Shi
57607ca0ce chore: prepare for v5.0.15 release 2023-01-20 11:20:34 +01:00
Zaiming (Stone) Shi
1c3e055b13
Merge pull request #9822 from JimMoen/fix-schema-typo 
chore: i18n typo fix
 2023-01-20 11:11:18 +01:00
JimMoen
16f45a60fd
chore: i18n typo fix 2023-01-20 11:50:01 +08:00
Thales Macedo Garitezi
a6ad97e6a5
Merge pull request #9807 from thalesmg/buffer-worker-rename-v50 
buffer work refactoring follow up (part 1)
 2023-01-19 14:17:10 -03:00
Zaiming (Stone) Shi
abe7a69696 Merge remote-tracking branch 'origin/master' into release-50 2023-01-19 17:51:38 +01:00
Stefan Strigler
6bde1173af chore: add translations 2023-01-19 13:26:36 +01:00
Thales Macedo Garitezi
d755b43c77 fix(jwt_worker): handle exceptions when decoding jwk from pem 
Returns a more controlled error if users attempt to use the Service
Account JSON from the GCP PubSub example from swagger, which is
redacted.
 2023-01-19 09:24:45 -03:00
Zaiming (Stone) Shi
63748aba3c
Merge pull request #9804 from emqx/release-50 
Merge release-50 (candidate of e5.0.0-rc.1) back to master.
 2023-01-19 08:48:41 +01:00
Thales Macedo Garitezi
47f796dd12 refactor: rename emqx_resource_worker -> emqx_resource_buffer_worker 
To make it more clear that it's purpose is serve as a buffering layer.
 2023-01-18 16:15:34 -03:00
Ilya Averyanov
f9843de7ae
Merge pull request #9628 from savonarola/fix-flaky-redis-bridge-test 
chore(ee bridge): fix Redis bridge test flakyness
 2023-01-18 20:56:13 +02:00
Zaiming (Stone) Shi
1716a5da99 chore: bump version to e5.0.0-rc.1 2023-01-18 17:22:05 +01:00
Stefan Strigler
fb763ecebd feat: support HAProxy protocol for dashboard API 2023-01-18 15:49:57 +01:00
Zaiming (Stone) Shi
7abba17b25
Merge pull request #9765 from zmstone/0115-add-password-converter 
fix(schema): add password converter to ensure its binary() type
 2023-01-18 15:09:05 +01:00
Andrew Mayorov
33b3c4fa9a
Merge pull request #9753 from feat/EMQX-8738/convert-ordered-sets 
feat: turn tables queried with search APIs into ordered sets
 2023-01-18 18:05:57 +04:00
Ilya Averyanov
44a6e5ed15 chore(resources): add missing parameters to emqx_resource schema 2023-01-18 14:33:45 +02:00
Zaiming (Stone) Shi
7e8381f4c7
Merge pull request #9785 from savonarola/fix-authn-handling 
fix(authn): stop authn handling when emqx_authentication provides a result
 2023-01-18 13:24:22 +01:00
ieQu1
c46d7f3404
Merge pull request #9801 from ieQu1/ekka-0.13.9 
chore(ekka): Bump version to 0.13.9
 2023-01-18 13:23:59 +01:00
Thales Macedo Garitezi
167b623497
Merge pull request #9699 from thalesmg/fix-buffer-clear-replayq-on-delete-v50 
fix(buffer): fix `replayq` usages in buffer workers (5.0)
 2023-01-18 09:08:50 -03:00
Erik Timan
46fc69cd48
Merge pull request #9781 from olcai/delete-zip-file-from-trace-log-download 
fix(emqx_management): delete files after trace log download
 2023-01-18 13:05:39 +01:00
ieQu1
d7242739e0 chore(ekka): Bump version to 0.13.9 2023-01-18 12:01:03 +01:00
Zaiming (Stone) Shi
d4f3b4c8c2 Merge remote-tracking branch 'origin/master' into fix-buffer-clear-replayq-on-delete-v50 2023-01-18 11:39:47 +01:00
zhongwencool
8e1475addb
Merge pull request #9798 from zhongwencool/dashboard-document 
chore: improve the dashboard's configuration
 2023-01-18 18:20:00 +08:00
Erik Timan
42182279b7 fix(emqx_management): ensure trace file dir is deleted on zip exception 2023-01-18 10:20:41 +01:00
Zhongwen Deng
fb84d5b817 chore: make spellcheck happy 2023-01-18 17:06:46 +08:00
Ivan Dyachkov
430b0a03d4
Merge pull request #9780 from id/fix-ensure-no-colon-in-filenames 
fix: ensure no colon in filenames
 2023-01-18 09:36:16 +01:00
Zhongwen Deng
0d852d9122 docs: improve the dashboard's document 2023-01-18 16:28:35 +08:00
Zaiming (Stone) Shi
faf5916ed6 test: relax recoverable/unrecoverable error check 
for now, treat all other errors unrecoverable
 2023-01-18 07:52:28 +01:00
zhongwencool
bc9d97ea53
Merge pull request #9791 from zhongwencool/crash-dump-doc 
chore: more detail about crash dump config
 2023-01-18 09:56:21 +08:00
Thales Macedo Garitezi
5c2ac0ac81 chore: don't cancel inflight items upon worker death; retry them 2023-01-17 19:50:30 -03:00
Thales Macedo Garitezi
087b667263 fix(buffer_worker): allow signalling unrecoverable errors 2023-01-17 19:50:30 -03:00
Stefan Strigler
f899284e3a
Merge pull request #9789 from sstrigler/EMQX-8754-test-function-return-500-of-data-integration-google-pubsub 
EMQX 8754 test function return 500 of data integration google pubsub
 2023-01-17 22:49:28 +01:00
lafirest
dea0c8230e
Merge pull request #9787 from lafirest/fix/webhook_bridge_cfg_upgrade 
fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12
 2023-01-18 04:47:08 +08:00
Thales Macedo Garitezi
4ed7bff33f chore: fix dialyzer warnings 2023-01-17 16:49:16 -03:00
Thales Macedo Garitezi
fa01deb3eb chore: retry as much as possible, don't reply to caller too soon 2023-01-17 16:49:15 -03:00
Thales Macedo Garitezi
b82009bc29 refactor: use monotonic times as refs and store initial times when creating ets 
with this, we may measure latencies in the future.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
3ba65c4377 feat: poke the buffer workers when inflight is no longer full 
if max inflight = 1, then we only make progress based on the state
timer, since the callbacks were not poking the buffer workers.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
b5aaef084c refactor: enter running state directly 
now that we don't have the possibility of dirty disk queues (we always
use volatile replayq), we will never resume old work.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd0e2a74ba refactor: rename inflight_name field to inflight_tid 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
006b4bda97 feat(buffer_worker): monitor async workers and cancel their inflight requests upon death 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
731ac6567a fix(buffer_worker): don't retry all kinds of inflight requests 
Some requests should not be retried during the blocked state.  For
example, if some async requests are just taking some time to process,
we should avoid retrying them periodically, lest risk overloading the
downstream further.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5425f3d88e refactor: rm unused fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
5dd24a64c3 refactor(buffer_worker): check if inflight is full before flushing 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
344eeebe63 fix: always ack async replies 
The caller should decide if it should retry in that case, to avoid
overwhelming the resource with retries.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
bd95a95409 refactor: remove redundant BlockWorker arg, change boolean to ack/nack 
`BlockWorker` was always false (ack).  Also, changed the return to
something more semantic than a boolean to avoid [boolean
blindness](https://runtimeverification.com/blog/code-smell-boolean-blindness/)
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
478fcc6ffd test: fix flaky test 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
30a227bd38 refactor: rename resume state timeout to unblock 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
7401d6f0ce refactor: rename ack fn 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
196bf1c5ba feat: mass collect calls from mailbox also when blocked 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
d4724d6ce9 refactor: remove redundant function 
`retry_queue` does basically what the running state does, now that we
refactored the buffer workers to always use the queue.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
d6a9d0aa48 fix: set queuing to 0 after buffer worker termination 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
81fc561ed5 fix(buffer_worker): check for overflow after enqueuing new requests 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4cb83d0c9a fix: fix some expressions after refactoring 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
fecdbac9a8 refactor: rename a few functions 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
cdd8de11b0 chore: fix a typo in function name 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
618b97870b refactor: call local function queue_count everywhere 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
249c4c1c79 refactor: use 'bufs' for resource worker replayq dir 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
af6807e863 refactor: cancel flush timer sooner 
Avoids the cancellation being delayed.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
477c55d8ef fix: sanitizy replayq dir filepath 
Colons (`:`) are not allowed in Windows.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
4c04a01370 refactor(buffer_worker): remove ?Q_ITEM wrapping and use lightweight size estimate 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
32a9e60313 feat(buffer_worker): also use the inflight table for sync requests 
Related: https://emqx.atlassian.net/browse/EMQX-8692

This should also correctly account for `retried.*` metrics for sync
requests.

Also fixes cases where race conditions for retrying async requests
could potentially lead to inconsistent metrics.

Fixes more cases where a stale reference to `replayq` was being held
accidentally after a `pop`.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
ff23d25e8b chore(replayq): update replayq -> 0.3.6 and use clean_start for buffer workers 
So we can truly avoid resuming work after a node restart.
 2023-01-17 16:48:48 -03:00
Thales Macedo Garitezi
c383558467 fix(buffer): fix replayq usages in buffer workers (5.0) 
https://emqx.atlassian.net/browse/EMQX-8700

Fixes a few errors in the usage of `replayq` queues.

- Close `replayq` when `emqx_resource_worker` terminates.
- Do not keep old references to `replayq` after any `pop`s.
- Clear `replayq`'s data directories when removing a resource.
 2023-01-17 16:48:48 -03:00
Zaiming (Stone) Shi
fc992f28bc test: add test coverage 2023-01-17 20:13:01 +01:00
Zaiming (Stone) Shi
fff6bf921f refactor(authz): call emqx_resource:simple_sync_query 
there is no need to route the request through the buffer workers
 2023-01-17 20:01:45 +01:00
Zaiming (Stone) Shi
7a471faf67 refactor(authn): use simple sync query for authn 
there is no need to route the authn queries through the buffer worker
 2023-01-17 19:57:14 +01:00
Ilya Averyanov
1d7364dab1 fix(authn): fix authn hook chain evaluation 
Stop authentication handling when
`emqx_authentication` provides a definitive result.
 2023-01-17 20:06:20 +02:00
Stefan Strigler
d164e5bc0f fix: don't require ssl conf 2023-01-17 15:29:19 +01:00
Stefan Strigler
e54f2f83b3 test: use same default timeout as elsewhere 2023-01-17 15:29:19 +01:00
Erik Timan
30a5cfaa83 fix(emqx_management): remove trace files after zip download 
We only deleted the resulting zip after a trace file download, not the
actual trace files. This adds a deletion of the uncompressed trace files
as well. It also creates unique directories when collecting trace files
so that concurrent downloads doesn't overwrite files in transit.
 2023-01-17 14:24:10 +01:00
Erik Timan
94296258b4 test(emqx_management): refactor test suite to use common test utils for API calls 2023-01-17 14:24:10 +01:00
Zhongwen Deng
f98786c91f fix: bad crash_dump_file default value in rpm 2023-01-17 20:47:54 +08:00
Zhongwen Deng
cf99089f84 chore: add some trace func to user_default 2023-01-17 20:33:01 +08:00
Zhongwen Deng
5a56013193 docs: more detail about crashdump config 2023-01-17 20:33:01 +08:00
zhongwencool
9f80eb2497
Merge pull request #9777 from zmstone/0116-refactor-config_files-is-readonly 
refactor: config_files is a readonly config
 2023-01-17 20:32:39 +08:00
Zaiming (Stone) Shi
263deae1f3 refactor: add a more generic name for password_converter 2023-01-17 11:12:17 +01:00
Zaiming (Stone) Shi
2afbf6a406 test: unlink process which is getting shutdown 2023-01-17 11:12:17 +01:00
Zaiming (Stone) Shi
4a7e74f5d6 fix(schema): add password converter to ensure its binary() type 2023-01-17 11:12:17 +01:00
Zaiming (Stone) Shi
8b3ec48406 refactor: node.config_files should be hidden as it's not configurable 
this config schema exists only for app-env mapping.
users can not change it because the bootstraping config
has to be etc/emqx.conf
 2023-01-17 11:08:40 +01:00
Stefan Strigler
74ae7c4264
Merge pull request #9782 from sstrigler/EMQX-8361-fix-ct-suite-for-plugins-api 
test: get plugin template as artifact
 2023-01-17 11:08:07 +01:00
firest
c3d5c25c26 fix(bridges): fix a compatible problem for old webhook bridge config which created before the v5.0.12 2023-01-17 17:42:10 +08:00
zhongwencool
cb5314bb9e
Merge pull request #9774 from zhongwencool/dashboard-password-validate-v50 
feat: add dashboard password validation
 2023-01-17 17:02:51 +08:00
Stefan Strigler
bc69a00329 test: get plugin template as artifact 2023-01-17 09:50:17 +01:00
Shawn
44f635300d chore: update mysql-otp to 1.7.2 2023-01-17 13:36:26 +08:00
Zhongwen Deng
3dfdad9cc9 test: dashboard default password 2023-01-17 11:30:42 +08:00
Zaiming (Stone) Shi
b9911e281e docs: Update apps/emqx_dashboard/src/emqx_dashboard_admin.erl 2023-01-17 11:30:42 +08:00
zhongwencool
c549379b7c chore: apply suggestions from code review 
Co-authored-by: Zaiming (Stone) Shi <zmstone@gmail.com>
 2023-01-17 11:30:42 +08:00
Zhongwen Deng
82af854c4a test: dashbashboard new password failed 2023-01-17 11:30:42 +08:00
Zhongwen Deng
a523fa2fa2 feat: add dashboard password validate 2023-01-17 11:30:42 +08:00
Zaiming (Stone) Shi
33be7672ad test: recover boot_modules app env after test 2023-01-17 00:43:25 +01:00
Ivan Dyachkov
676f017ec0 fix: ensure no colon in filenames 2023-01-16 21:27:01 +01:00
Andrew Mayorov
5fd157fb1d
Merge pull request #9776 from keynslug/chore/bump-epgsql-otp-25 
chore: update epgsql to 4.7.0.1 with full OTP-25 compat
 2023-01-16 23:51:09 +04:00
Zaiming (Stone) Shi
e668044ebc fix: version number previs in restricted shell 2023-01-16 13:47:00 +01:00
Andrew Mayorov
34571c779d
feat: make suboption table ordering more natural 2023-01-16 15:39:10 +03:00
Andrew Mayorov
ce2dba15b4
feat: turn tables queried with search APIs into ordered sets 
This is needed to ensure more or less consistent client experience
for the new planned cursor-based search APIs.
 2023-01-16 15:39:10 +03:00
Zaiming (Stone) Shi
a7fc5e8fe1
Merge pull request #9761 from zmstone/0114-fix-kafka-value-template-and-docs 
feat: introduce 'this' concept for placeholder, and use it in Kafka bridge
 2023-01-16 13:37:29 +01:00
Andrew Mayorov
54cea3b3f2
chore: update epgsql to 4.7.0.1 with full OTP-25 compat 2023-01-16 15:35:51 +03:00
Stefan Strigler
67909f0b40 fix: testing metrics for emqx_bridge_mqtt_SUITE 2023-01-16 12:10:06 +01:00
Stefan Strigler
a19226326c fix: don't test for strict values as that makes test flaky 2023-01-16 12:10:06 +01:00
Stefan Strigler
9a53410efb fix: don't add empty list element to options proplist 2023-01-16 12:00:03 +01:00
Stefan Strigler
edfcea7f43 fix: add log for timeout 2023-01-16 12:00:03 +01:00
Zaiming (Stone) Shi
47414e0d53 docs: improve kafka key and value field description 2023-01-16 11:32:09 +01:00
Zaiming (Stone) Shi
376dab7cd8
Merge pull request #9743 from zmstone/0112-docs-fix-some-descriptions 
0112 docs fix some descriptions
 2023-01-16 09:56:56 +01:00
JimMoen
54ebc27d24
Merge pull request #9672 from JimMoen/0103-fix-mqtt-bridge 
Fix the problem that the bridge is not available when the Payload template is empty in the MQTT bridge.
 2023-01-16 09:57:20 +08:00
Zaiming (Stone) Shi
089389100e
Merge pull request #9763 from zmstone/0114-fix-crash-when-password-undefined 
0114 fix crash when password undefined
 2023-01-15 12:41:02 +01:00
Zaiming (Stone) Shi
df202c69ad
Merge pull request #9746 from zmstone/0113-fix-mqtt-connect-packet-format-when-password-is-empty 
fix(emqx_packet): log empty string if password value is missing
 2023-01-14 19:42:07 +01:00
Zaiming (Stone) Shi
0f2f5fbbe0 fix(authn): no exception when password is 'undefined' 2023-01-14 17:58:55 +01:00
Zaiming (Stone) Shi
7073c62dd9 feat: add a 'this' for rule engine put env 
now a dot is allowed in front of a var (path)
e.g. ${.clientid} is equivalent to ${clientid}

${.} however, means everything.
the parsed var is interally represented as $_THIS_
to make it easier to read/search.
 2023-01-14 11:03:58 +01:00
Zaiming (Stone) Shi
002f5dbd89
Merge pull request #9750 from zhongwencool/bootstrap-app-env 
fix: save app's env when bootstrapping and reloading
 2023-01-14 07:33:13 +01:00
Zhongwen Deng
b07befbf56 test: conf_app ct failed 2023-01-14 01:03:57 +08:00
Stefan Strigler
6fe09447ed fix: stale test using old resource paths after merge 2023-01-13 17:23:25 +01:00
Zhongwen Deng
09160383f5 chore: bump emqx_resource to 0.1.5 2023-01-13 23:42:41 +08:00
Zhongwen Deng
588140b842 chore: why we should ingore save kernel app env when init 2023-01-13 23:35:45 +08:00
Zhongwen Deng
1fe0061123 fix: node.config_files type is hocon:array(string())) 2023-01-13 23:35:45 +08:00
Zhongwen Deng
c9f0355f5a fix: save app env when bootstrap and reload 2023-01-13 23:35:45 +08:00
Zhongwen Deng
1085492171 chore: don't refresh when CT is running 2023-01-13 23:35:45 +08:00
Zhongwen Deng
91d38b410d fix: refresh logger handler when starting 2023-01-13 23:35:45 +08:00
Stefan Strigler
8f5881d1a5 fix: remove stale request/3 from merge error 2023-01-13 16:19:35 +01:00
Stefan Strigler
e08c1d2229 Merge remote-tracking branch 'olcai/refactor-bridges-api' into dev/api-refactor 2023-01-13 15:49:52 +01:00
Stefan Strigler
1690a6dcfc
Merge branch 'master' into dev/api-refactor 2023-01-13 15:34:13 +01:00
Erik Timan
61e98900be chore: bump app vsn of emqx_resource 2023-01-13 15:13:35 +01:00
Erik Timan
c3133fb6a2 fix(emqx_bridge): small fixes from review 2023-01-13 14:54:54 +01:00
Zaiming (Stone) Shi
a77d33b97d
Merge pull request #9751 from lafirest/fix/un_cleared_ca 
fix that obsoleted cert file will not be deleted after the listener is updated/deleted
 2023-01-13 14:38:55 +01:00
Kjell Winblad
1ac03ab208
Merge pull request #9730 from kjellwinblad/kjell/fix/resource_atom_leak/EMQX-8583 
fix: remove atom leaks
 2023-01-13 14:38:28 +01:00
Zaiming (Stone) Shi
bb3dceb456
Merge pull request #9749 from keynslug/fix/count-respect-matchspec 
fix(paging): respect matchspec even if qs is empty when counting
 2023-01-13 14:29:03 +01:00
Erik Timan
8dd52e5a6e chore: add translations to schemas 2023-01-13 14:19:23 +01:00
Erik Timan
f1c58c34ed test(emqx_bridge): fix fetching of metrics in emqx_bridge_mqtt_SUITE 2023-01-13 14:19:23 +01:00
Zaiming (Stone) Shi
2974269f2a
Merge pull request #9723 from emqx/release-50 
chore: sync release-50 to master
 2023-01-13 14:13:12 +01:00
firest
0a965879dd fix: fix that obsoleted cert file will not be deleted after the listener is updated/deleted 2023-01-13 18:42:27 +08:00
Zhongwen Deng
f15b29b1ef chore: upgrade app version 2023-01-13 18:22:29 +08:00
JimMoen
b7259d9a20
test(mqtt-bridge): use empty payload template for ingress/egress mqtt bridge 2023-01-13 18:12:53 +08:00
Kjell Winblad
734e6b9c96 chore: fix flaky test cases, log labels and review comments 
Co-authored-by: Thales Macedo Garitezi <thalesmg@gmail.com>
 2023-01-13 11:05:02 +01:00
Andrew Mayorov
b3e62bd8f8
fix(paging): respect matchspec even if qs is empty when counting 2023-01-13 12:56:29 +03:00
zhongwencool
383a6812d1
Merge pull request #9748 from zhongwencool/listener-max-connections-default-value 
fix: default max_connections in raw config should be binary not atom
 2023-01-13 17:55:22 +08:00
JimMoen
59ac0b1424
fix(mqtt-bridge): transmit raw msg payload with empty template 2023-01-13 17:23:04 +08:00
zhongwencool
0049b4a294
Merge branch 'master' into release-50 2023-01-13 16:39:35 +08:00
Zaiming (Stone) Shi
e7883e0ee2
Merge pull request #9741 from id/fix-description-for-duration-fields 
fix: remove time unit from duration fields description
 2023-01-13 09:31:48 +01:00
Zhongwen Deng
eacdaffe3c fix: default max_connections in raw config should be binary not atom 2023-01-13 16:01:57 +08:00
Andrew Mayorov
33d88c95de
Merge pull request #9745 from keynslug/fix/EMQX-8702/changelog 
chore: add changelog for the fuzzy search API fix
 2023-01-13 11:59:37 +04:00
Zaiming (Stone) Shi
2bc19c3a62 fix(emqx_packet): log empty string if password value is missing 
to keep it a consistent behaviour comparing when logging username
 2023-01-13 08:21:09 +01:00
Zaiming (Stone) Shi
153d64068a refactor: rename root level cache type to authz_cache 2023-01-13 07:47:54 +01:00
Zaiming (Stone) Shi
c7ac55520d docs: update pool_size doc 2023-01-13 07:46:23 +01:00
Zaiming (Stone) Shi
b40ce0fc2d
Merge pull request #9626 from id/fix-enable-authz-cache-by-default 
fix: enable authorization cache by default
 2023-01-13 07:39:38 +01:00
Xinyu Liu
6b75077c47
Merge pull request #9734 from terry-xiaoyu/influxdb-default-value-of-write-syntax 
refactor: remove default value of timestamp field of influxdb
 2023-01-13 10:14:14 +08:00
Erik Timan
42f42de4d9 feat(emqx_bridge): add separate endpoint for enable/disable of bridge 
In order to improve the consistency with other API endpoints, we move
the enable/disable operations to a separate endpoint
/bridges/{id}/enable/[true,false].
 2023-01-12 19:39:43 +01:00
Erik Timan
860e21d40f feat(emqx_bridge): move metrics to own endpoint, rename reset_metrics 
In order for the /bridges APIs to be consistent with other APIs, we move
out metrics from GET /bridges/{id} to its own endpoint,
/bridges/{id}/metrics. We also rename /bridges/reset_metrics to
/bridges/metrics/reset.
 2023-01-12 19:39:41 +01:00
Erik Timan
c2fd1a4482 feat(emqx_bridge): shorten operation endpoint URLs 
This shortens and simplifies URLs for performing bridge operations so
that the API looks more congruent.
 2023-01-12 19:36:22 +01:00
Andrew Mayorov
90e837783f
fix(paging): rename partial to overflow for clarity 2023-01-12 19:25:40 +03:00
lafirest
a26c05f4f6
Merge pull request #9586 from lafirest/fix/disable_basic_auth_api 
fix: disable basic auth for HTTP API
 2023-01-12 23:02:29 +08:00
Zaiming (Stone) Shi
370b6b0d2f
Merge pull request #9735 from kjellwinblad/kjell/fix/password_log_leak/EMQX-8718 
fix: remove password from info logs
 2023-01-12 15:34:16 +01:00
Shawn
1711823487 refactor: remove default value of timestamp field of influxdb 2023-01-12 22:25:44 +08:00
firest
c5f557e315 fix: disable basic auth for HTTP API 2023-01-12 21:35:49 +08:00
Stefan Strigler
2a81fa1522 fix: remove redundant log message 2023-01-12 14:29:11 +01:00
Stefan Strigler
04f46f5227 feat: make http connector report errors 2023-01-12 14:26:36 +01:00
Stefan Strigler
77f043dedf fix: don't require username and password 
not enforced by schema so we shouldn't either
 2023-01-12 14:26:36 +01:00
Ivan Dyachkov
b5d3e9d8b8 fix: remove time unit from duration fields description 2023-01-12 14:18:55 +01:00
Ivan Dyachkov
f90c41f769 fix: set default value in schema 2023-01-12 13:49:52 +01:00
Ivan Dyachkov
1f57e7b538 fix: enable authorization cache by default 2023-01-12 12:36:31 +01:00
Andrew Mayorov
9e9d97b4d8
Merge pull request #9726 from keynslug/fix/EMQX-8702/fuzzy-search-paging 
fix(api): augment paged search responses with `hasnext` flag
 2023-01-12 13:35:59 +04:00
Kjell Winblad
82e80e5610 fix: remove password from info logs 
Information log messages for several connectors contained sensitive
password information. This commit fixes this by hiding the sensitive
information from the log messages.

Fixes: https://emqx.atlassian.net/browse/EMQX-8718
 2023-01-12 09:28:01 +01:00
JianBo He
13fdbd695e
Merge pull request #9722 from HJianBo/more-confs-for-prometheus 
feat: more confs for prometheus pushing
 2023-01-12 16:06:21 +08:00
Andrew Mayorov
50c9321b9e
fix(api): anticipate empty set of nodes handling cluster query 2023-01-12 10:57:50 +03:00