Move github wiki to redthedocs

This commit is contained in:
Feng 2016-01-14 17:20:01 +08:00
parent 0d52082b2e
commit b46c8d2f0f
14 changed files with 827 additions and 8 deletions

View File

@ -3,6 +3,7 @@ Applications
============
.. toctree::
:maxdepth: 2
push
chat

View File

@ -0,0 +1,167 @@
0.13.0 Benchmark (2015-11-08)
=============================
Benchmark with the [emqtt_benchmark](https://github.com/emqtt/emqtt_benchmark) tool.
Server: 8 Core, 32G Memory CentOS6.
etc/vm.args
-----------
::
## max process numbers
+P 1000000
## Sets the maximum number of simultaneously existing ports for this system
+Q 409600
## max atom number
## +t
## Set the distribution buffer busy limit (dist_buf_busy_limit) in kilobytes.
## Valid range is 1-2097151. Default is 1024.
## +zdbbl 8192
##-------------------------------------------------------------------------
## Env
##-------------------------------------------------------------------------
## Increase number of concurrent ports/sockets, deprecated in R17
-env ERL_MAX_PORTS 409600
-env ERTS_MAX_PORTS 409600
250K Subscribers on five nodes
------------------------------
Create 50K connections on each node and subscribe test/%i topics::
./emqtt_bench_sub -h server -p 1883 -c 50000 -i 5 -t test/%i -q 1
Publish 4K msg/sec on server
------------------------------
Publish qos1 messages at the rate of 4K(msg/sec) on the server::
./emqtt_bench_pub -h localhost -c 40 -i 10 -I 10 -t test/%i -q 1
Each node will receive 4k msg/sec.
Benchmark Result
----------------
250K Connections,
50K Topics,
250K Subscribers,
4K Qos1 Messages/Sec In,
20K Qos1 Messages/Sec Out,
12M+(bps) In, 56M+(bps) Out Traffic
Consumed about 3.1G memory and 400+ CPU.
./bin/emqttd_ctl broker
-----------------------
sysdescr : Erlang MQTT Broker
version : 0.13.0
uptime : 20 minutes, 25 seconds
datetime : 2015-11-08 10:38:30
./bin/emqttd_ctl broker stats
-----------------------
```
clients/count : 250040
clients/max : 250040
queues/count : 0
queues/max : 0
retained/count : 2
retained/max : 2
sessions/count : 0
sessions/max : 0
subscribers/count : 250000
subscribers/max : 250000
topics/count : 50050
topics/max : 50050
```
./bin/emqttd_ctl listeners
-----------------------
```
listener on http:8083
acceptors : 4
max_clients : 64
current_clients : 0
shutdown_count : []
listener on mqtts:8883
acceptors : 4
max_clients : 512
current_clients : 0
shutdown_count : []
listener on mqtt:1883
acceptors : 64
max_clients : 1000000
current_clients : 250040
shutdown_count : []
listener on http:18083
acceptors : 4
max_clients : 512
current_clients : 0
shutdown_count : []
```
./bin/emqttd_ctl vm
-----------------------
```
cpu/load1 : 13.44
cpu/load5 : 10.43
cpu/load15 : 5.98
memory/total : 2710277048
memory/processes : 1420519328
memory/processes_used : 1419564424
memory/system : 1289757720
memory/atom : 512601
memory/atom_used : 486464
memory/binary : 380872488
memory/code : 13077799
memory/ets : 408483440
process/limit : 1048576
process/count : 500353
io/max_fds : 500000
io/active_fds : 75
```
./bin/emqttd_ctl recon node_stats
---------------------------------
```
{[{process_count,500353},
{run_queue,41},
{error_logger_queue_len,0},
{memory_total,2698242896},
{memory_procs,1408734784},
{memory_atoms,486706},
{memory_bin,380825008},
{memory_ets,408483456}],
[{bytes_in,256851},
{bytes_out,1202095},
{gc_count,24423},
{gc_words_reclaimed,4092612},
{reductions,1850034},
{scheduler_usage,[{1,0.46640942781698586},
{2,0.5293768498814092},
{3,0.441425019999723},
{4,0.45550895378436373},
{5,0.45318168320081786},
{6,0.4627325387117833},
{7,0.5144161001107628},
{8,0.46406643808409137}]}]}
```

View File

@ -1,2 +1,32 @@
etc/app.config
==============
etc/app.config for C1000K
=========================
::
{mqtt, 1883, [
%% Size of acceptor pool
{acceptors, 64},
%% Maximum number of concurrent clients
{max_clients, 1000000},
%% Socket Access Control
{access, [{allow, all}]},
%% Connection Options
{connopts, [
%% Rate Limit. Format is 'burst, rate', Unit is KB/Sec
%% {rate_limit, "100,10"} %% 100K burst, 10K rate
]},
%% Socket Options
{sockopts, [
%Tune buffer if hight thoughtput
%{recbuf, 4096},
%{sndbuf, 4096},
%{buffer, 4096},
%{nodelay, true},
{backlog, 1024}
]}
]},

View File

@ -1,4 +1,26 @@
etc/vm.args
==============
etc/vm.args for C1000K
======================
::
## 12 threads/core.
+A 64
## Max process numbers > connections * 2
## 2M
+P 2097152
## Sets the maximum number of simultaneously existing ports for this system
## 1M
+Q 1048576
## Increase number of concurrent ports/sockets, deprecated in R17
-env ERL_MAX_PORTS 1048576
-env ERTS_MAX_PORTS 1048576
## Mnesia and SSL will create temporary ets tables
## 16K
-env ERL_MAX_ETS_TABLES 16384

View File

@ -1,9 +1,12 @@
Tune Guide
----------
.. toctree::
:maxdepth: 1
linux_kernel_tuning
etc_vm_args
etc_app_config
benchmark_0.13.0

View File

@ -0,0 +1,175 @@
Linux Kernel Tuning for C1000K concurrent connections::
# system-wide limit on max opened files for all processes
# sysctl -n fs.nr_open
# 1M
sysctl -w fs.file-max=1048576
sysctl -w fs.nr_open=1048576
ulimit -n 1048576
# Increase number of incoming connections
net.core.somaxconn = 65536
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 2048000
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.ip_local_port_range = 8000 65535
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
vm.min_free_kbytes = 65536
vm.swappiness = 0
vm.overcommit_memory = 1
/etc/sysctl.conf
----------------
::
fs.file-max = 1048576
/etc/security/limits.conf
-------------------------
::
* soft nofile 1048576
* hard nofile 1048576
## Kernel/Network Tunings from @galvezlj
Server side:
```
fs.file-max = 1000000
net.core.somaxconn = 65536
net.ipv4.ip_local_port_range = 500 65535
net.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
```
Client side:
```
sysctl -w net.ipv4.ip_local_port_range="500 65535"
echo 1000000 > /proc/sys/fs/nr_open
```
## Kernel/Network Tunings from my benchmark server
```
fs.file-max = 1000000
net.core.somaxconn = 65536
net.core.wmem_max = 124928
net.core.rmem_max = 124928
net.core.wmem_default = 124928
net.core.rmem_default = 124928
net.core.dev_weight = 64
net.core.netdev_max_backlog = 1000
net.core.message_cost = 5
net.core.message_burst = 10
net.core.optmem_max = 20480
net.core.busy_poll = 0
net.core.busy_read = 0
net.core.netdev_budget = 300
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.ip_default_ttl = 64
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_tw_buckets = 262144
net.ipv4.ip_dynaddr = 0
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.inet_peer_threshold = 65664
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_gc_mintime = 10
net.ipv4.inet_peer_gc_maxtime = 120
net.ipv4.tcp_mem = 3080640 4107520 6161280
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_abc = 0
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.udp_mem = 3080640 4107520 6161280
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.tag = 0
vm.min_free_kbytes = 67584
vm.swappiness = 60
vm.overcommit_memory = 0
```

View File

@ -0,0 +1,61 @@
Clustering
==========
Suppose we cluster two nodes on hosts:
Node | Host | IpAddress
-----|--------|-------------
node1(disc_copy)| host1 | 192.168.0.10
node2(ram_copy) | host2 | 192.168.0.20
Configure and start 'node1'
---------------------------
configure 'etc/vm.args'::
-name emqttd@192.168.0.10
If host1, host2 added to /etc/hosts of OS::
-name emqttd@host1
Start node1::
./bin/emqttd start
.. NOTE:: Notice that data/mnesia/* should be removed before you start the broker with different node name.
Configure and start 'node2'
---------------------------
Configure 'etc/vm.args'::
-name emqttd@192.168.0.20
or::
-name emqttd@host2
Then start node2::
./bin/emqttd start
Cluster two nodes
---------------------------
Run './bin/emqttd_ctl cluster' on host2::
./bin/emqttd_ctl cluster emqttd@192.168.0.10
Check cluster status
---------------------------
And then check clustered status on any host::
./bin/emqttd_ctl cluster

View File

@ -0,0 +1,74 @@
/etc/init.d/emqttd
==================
::
#!/bin/sh
#
# emqttd Startup script for emqttd.
#
# chkconfig: 2345 90 10
# description: emqttd is mqtt broker.
# source function library
. /etc/rc.d/init.d/functions
# export HOME=/root
start() {
echo "starting emqttd..."
cd /opt/emqttd && ./bin/emqttd start
}
stop() {
echo "stopping emqttd..."
cd /opt/emqttd && ./bin/emqttd stop
}
restart() {
stop
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop}"
RETVAL=2
esac
chkconfig
---------
::
chmod +x /etc/init.d/emqttd
chkconfig --add emqttd
chkconfig --list
boot test on ubuntu
-------------------
::
service emqttd start
erlexec: HOME must be set
-------------------
uncomment '# export HOME=/root' if "HOME must be set" error.
Referrence
---------
.. `How to make unix service see environment variables?`_: http://unix.stackexchange.com/questions/44370/how-to-make-unix-service-see-environment-variables/44378#44378

View File

@ -2,4 +2,32 @@
HTTP API
========
emqttd support HTTP API to publish message from your application server to MQTT clients. For example::
curl -v --basic -u user:passwd -d "qos=1&retain=0&topic=/a/b/c&message=hello from http..." -k http://localhost:8083/mqtt/publish
HTTP API URL
-----------
::
HTTP POST http://host:8083/mqtt/publish
HTTP Parameters
---------------
+---------+-----------------+
| Name | Description |
+-------------------+-------+
| client | ClientId |
+-------------------+-------+
| qos | QoS(0, 1, 2) |
+-------------------+-------+
| retain | Retain(0, 1) |
+-------------------+-------+
| topic | Topic |
+-------------------+-------+
| message | Message |
+-------------------+-------+

View File

@ -1,8 +1,8 @@
User Guide
----------
.. toctree::
:maxdepth: 2
install
configuration
@ -11,5 +11,7 @@ User Guide
http
websocket
admin
systopics
logs
trace

View File

@ -0,0 +1,205 @@
$SYS Topics
===========
$SYS Topics of broker is started with::
$SYS/brokers/${node}
${node} is erlang node name of clustered brokers. For example::
$SYS/brokers/emqttd@host1/version
$SYS/brokers/emqttd@host2/version
Broker $SYS Topics
------------------
Topic | Description
-------------------------------|------------
$SYS/brokers | Broker nodes
$SYS/brokers/${node}/version | Broker Version
$SYS/brokers/${node}/uptime | Broker Uptime
$SYS/brokers/${node}/datetime | Broker DateTime
$SYS/brokers/${node}/sysdescr | Broker Description
 
Client $SYS Topics
------------------
Start with: $SYS/brokers/${node}/clients/
Topic | Payload(json) | Description
----------------------|---------------------|---------------
${clientid}/connected | {ipaddress: "127.0.0.1", username: "test", session: false, version: 3, connack: 0, ts: 1432648482} | Publish when client connected
${clientid}/disconnected | {reason: "normal" | "keepalive_timeout" | "conn_closed"}
Parameters of 'connected' Payload::
ipaddress: "127.0.0.1",
username: "test",
session: false,
protocol: 3,
connack: 0,
ts: 1432648482
Parameters of 'disconnected' Payload::
reason: normal,
ts: 1432648486
Statistics $SYS Topics
----------------------
Start with '$SYS/brokers/${node}/stats/'
Client Stats
----------------------
Topic | Description
-------------------------------------|------------
clients/count | count of current connected clients
clients/max | max connected clients in the same time
Session Stats
----------------------
Topic | Description
-----------------|------------
sessions/count | count of current sessions
sessions/max | max number of sessions
Subscriber Stats
----------------------
Topic | Description
------------------|------------
subscriptions/count | count of current subscriptions
subscriptions/max | max number of subscriptions
Topic Stats
----------------------
Topic | Description
------------------|------------
topics/count | count of current topics
topics/max | max number of topics
Queue Stats
----------------------
Topic | Description
------------------|------------
queues/count | count of current queues
queues/max | max number of queues
Metrics $SYS Topics
----------------------
Start with '$SYS/brokers/${node}/metrics/'
Bytes sent and received
----------------------
Topic | Description
------------------------------------|------------
bytes/received | MQTT Bytes Received since broker started
bytes/sent | MQTT Bytes Sent since the broker started
Packets sent and received
-------------------------
Topic | Description
-------------------------|------------
packets/received | MQTT Packets received
packets/sent | MQTT Packets sent
packets/connect | MQTT CONNECT Packet received
packets/connack | MQTT CONNACK Packet sent
packets/publish/received | MQTT PUBLISH packets received
packets/publish/sent | MQTT PUBLISH packets sent
packets/subscribe | MQTT SUBSCRIBE Packets received
packets/suback | MQTT SUBACK packets sent
packets/unsubscribe | MQTT UNSUBSCRIBE Packets received
packets/unsuback | MQTT UNSUBACK Packets sent
packets/pingreq | MQTT PINGREQ packets received
packets/pingresp | MQTT PINGRESP Packets sent
packets/disconnect | MQTT DISCONNECT Packets received
Messages sent and received
---------------------------
Topic | Description
---------------------------------------|-------------------
messages/received | Messages Received
messages/sent | Messages Sent
messages/retained | Messages Retained
messages/stored | TODO: Messages Stored
messages/dropped | Messages Dropped
Alarm Topics
---------------------------
Start with '$SYS/brokers/${node}/alarms/'
Topic | Description
-----------------|-------------------
${alarmId}/alert | New Alarm
${alarmId}/clear | Clear Alarm
Logs
---------------------------
'$SYS/brokers/${node}/logs/${severity}'
Severity | Description
-----------|-------------------
debug | Debug Log
info | Info Log
notice | Notice Log
warning | Warning Log
error | Error Log
critical | Critical Log
Sysmon
---------------------------
Start with '$SYS/brokers/${node}/sysmon/'
Topic | Description
-----------------|-------------------
long_gc | Long GC Warning
long_schedule | Long Schedule
large_heap | Large Heap Warning
busy_port | Busy Port Warning
busy_dist_port | Busy Dist Port
Logs(TODO)
---------------------------
'$SYS/brokers/${node}/log/${severity}'
Severity | Description
------------|-------------------
debug | Debug
info | Info Log
notice | Notice Log
warning | Warning Log
error | Error Log
critical | Critical Log
alert | Alert Log
VM Load Topics(TODO)
---------------------------
Start with '$SYS/brokers/${node}/vm/'
Topic | Description
-----------------|-------------------
memory/* | TODO
cpu/* | TODO
processes/* | TODO
Sys Interval
---------------------------
sys_interval: 1 minute default

View File

@ -0,0 +1,37 @@
emqttd could trace packets received/sent from/to specific client, or trace publish/subscribe to specific topic.
emqttd use lager:trace_file api and write trace log to file.
TODO: NOTICE
Trace client
-------------
```
./bin/emqttd_ctl trace client "ClientId" "trace_clientid.log"
```
Trace topic
-------------
```
./bin/emqttd_ctl trace topic "Topic" "trace_topic.log"
```
Stop Trace
-------------
```
./bin/emqttd_ctl trace client "ClientId" off
./bin/emqttd_ctl trace topic "Topic" off
```
Lookup Traces
-------------
```
./bin/emqttd_ctl trace list
```

View File

@ -1,5 +1,20 @@
MQTT Over WebSocket
===================
WebSocket
=========
emqttd supports MQTT over WebSocket(SSL).
URI
----
::
ws(s)://host:8083/mqtt
Sec-WebSocket-Protocol
----------------------
::
'mqttv3.1', 'mqttv3.1.1'

View File

@ -13,4 +13,3 @@
{mod, {emqttd_app, []}},
{env, []}
]}.