From 8b8ceaac2ff060f9869d36b89d107e97a2ecceab Mon Sep 17 00:00:00 2001 From: Feng Date: Sat, 27 Feb 2016 15:24:46 +0800 Subject: [PATCH] Tuning --- docs/source/design.rst | 327 +++++++++++++++++++++++++++++++++++++++++ docs/source/faq.rst | 77 ++++++++++ docs/source/guide.rst | 275 ++++++++++++++++++++++++++++++++++ docs/source/tune.rst | 6 +- 4 files changed, 682 insertions(+), 3 deletions(-) create mode 100644 docs/source/faq.rst diff --git a/docs/source/design.rst b/docs/source/design.rst index 2b8f7e884..729f82dbc 100644 --- a/docs/source/design.rst +++ b/docs/source/design.rst @@ -1,3 +1,330 @@ ============== Design Guide ============== + +--------------- +Pubsub Sequence +--------------- + +## PubSub Sequence + +### Clean Session = 1 + +``` + +title PubSub Sequence(Clean Session = 1) + +ClientA-->PubSub: Publish Message +PubSub-->ClientB: Dispatch Message +``` + +![PubSub_CleanSess_1](http://emqtt.io/static/img/design/PubSub_CleanSess_1.png) + +### Clean Session = 0 + +``` +title PubSub Sequence(Clean Session = 0) + +ClientA-->SessionA: Publish Message +SessionA-->PubSub: Publish Message +PubSub-->SessionB: Dispatch Message +SessionB-->ClientB: Dispatch Message + +``` +![PubSub_CleanSess_0](http://emqtt.io/static/img/design/PubSub_CleanSess_0.png) + + +## Qos + +PubQos | SubQos | In Message | Out Message +-------|--------|------------|------------- +0 | 0 | 0 | 0 +0 | 1 | 0 | 0 +0 | 2 | 0 | 0 +1 | 0 | 1 | 0 +1 | 1 | 1 | 1 +1 | 2 | 1 | 1 +2 | 0 | 2 | 0 +2 | 1 | 2 | 1 +2 | 2 | 2 | 2 + + +## Topic Functions Benchmark + +Mac Air(11): + +Function | Time(microseconds) +-------------|-------------------- +match | 6.25086 +triples | 13.86881 +words | 3.41177 +binary:split | 3.03776 + +iMac: + +Function | Time(microseconds) +-------------|-------------------- +match | 3.2348 +triples | 6.93524 +words | 1.89616 +binary:split | 1.65243 + + +-------------- +Cluster Design +-------------- + +## Cluster Design + +1. One 'disc_copies' node and many 'ram_copies' nodes. + + 2. Topic trie tree will be copied to every clusterd node. + + 3. Subscribers to topic will be stored in each node and will not be copied. + + ## Cluster Strategy + + TODO:... + + 1. A message only gets forwarded to other cluster nodes if a cluster node is interested in it. this reduces the network traffic tremendously, because it prevents nodes from forwarding unnecessary messages. + + 2. As soon as a client on a node subscribes to a topic it becomes known within the cluster. If one of the clients somewhere in the cluster is publishing to this topic, the message will be delivered to its subscriber no matter to which cluster node it is connected. + + .... + +## Cluster Architecture + +![Cluster Design](http://emqtt.io/static/img/Cluster.png) +## Cluster Command + +```sh +./bin/emqttd_ctl cluster DiscNode +``` + +## Mnesia Example + +``` +(emqttd3@127.0.0.1)3> mnesia:info(). +---> Processes holding locks <--- +---> Processes waiting for locks <--- +---> Participant transactions <--- +---> Coordinator transactions <--- +---> Uncertain transactions <--- +---> Active tables <--- +mqtt_retained : with 6 records occupying 221 words of mem +topic_subscriber: with 0 records occupying 305 words of mem +topic_trie_node: with 129 records occupying 3195 words of mem +topic_trie : with 128 records occupying 3986 words of mem +topic : with 93 records occupying 1797 words of mem +schema : with 6 records occupying 1081 words of mem +===> System info in version "4.12.4", debug level = none <=== +opt_disc. Directory "/Users/erylee/Projects/emqttd/rel/emqttd3/data/mnesia" is NOT used. +use fallback at restart = false +running db nodes = ['emqttd2@127.0.0.1','emqttd@127.0.0.1','emqttd3@127.0.0.1'] +stopped db nodes = [] +master node tables = [] +remote = [] +ram_copies = [mqtt_retained,schema,topic,topic_subscriber,topic_trie, +topic_trie_node] +disc_copies = [] +disc_only_copies = [] +[{'emqttd2@127.0.0.1',ram_copies}, +{'emqttd3@127.0.0.1',ram_copies}, +{'emqttd@127.0.0.1',disc_copies}] = [schema] +[{'emqttd2@127.0.0.1',ram_copies}, +{'emqttd3@127.0.0.1',ram_copies}, +{'emqttd@127.0.0.1',ram_copies}] = [topic,topic_trie,topic_trie_node, +mqtt_retained] +[{'emqttd3@127.0.0.1',ram_copies}] = [topic_subscriber] +44 transactions committed, 5 aborted, 0 restarted, 0 logged to disc + 0 held locks, 0 in queue; 0 local transactions, 0 remote + 0 transactions waits for other nodes: [] + ``` + + ## Cluster vs Bridge + + Cluster will copy topic trie tree between nodes, Bridge will not. + + + +------------- +Hooks Design +------------- + +## Overview + +emqttd supported a simple hooks mechanism in 0.8.0 release to extend the broker. The designed is improved in 0.9.0 release. + +## API + +emqttd_broker Hook API: + +``` +-export([hook/3, unhook/2, foreach_hooks/2, foldl_hooks/3]). +``` + +### Hook + +``` +-spec hook(Hook :: atom(), Name :: any(), MFA :: mfa()) -> ok | {error, any()}. +hook(Hook, Name, MFA) -> + ... + ``` + + ### Unhook + + ``` + -spec unhook(Hook :: atom(), Name :: any()) -> ok | {error, any()}. + unhook(Hook, Name) -> + ... + ``` + + ### Foreach Hooks + + ``` + -spec foreach_hooks(Hook :: atom(), Args :: list()) -> any(). + foreach_hooks(Hook, Args) -> + ... + ``` + + ### Foldl Hooks + + ``` + -spec foldl_hooks(Hook :: atom(), Args :: list(), Acc0 :: any()) -> any(). + foldl_hooks(Hook, Args, Acc0) -> + ... + ``` + + ## Hooks + + Name | Type | Description + --------------- | ----------| -------------- + client.connected | foreach | Run when client connected successfully + client.subscribe | foldl | Run before client subscribe topics + client.subscribe.after | foreach | Run After client subscribe topics + client.unsubscribe | foldl | Run when client unsubscribe topics + message.publish | foldl | Run when message is published + message.acked | foreach | Run when message is acked + client.disconnected | foreach | Run when client is disconnnected + + ## End-to-End Message Pub/Ack + + Could use 'message.publish', 'message.acked' hooks to implement end-to-end message pub/ack: + + ``` + PktId <-- --> MsgId <-- --> MsgId <-- --> PktId + |<--- Qos --->|<---PubSub--->|<-- Qos -->| + ``` +## Limit + +The design is experimental. + + +-------------- +Plugin Design +-------------- + +## Overview + +**Notice that 0.11.0 release use rebar to manage plugin's deps.** + +A plugin is just an erlang application that extends emqttd broker. + +The plugin application should be put in "emqttd/plugins/" folder to build. + + +## Plugin Project + +You could create a standalone plugin project outside emqttd, and then add it to "emqttd/plugins/" folder by "git submodule". + +Git submodule to compile emqttd_dashboard plugin with the broker, For example: + +``` +git submodule add https://github.com/emqtt/emqttd_dashboard.git plugins/emqttd_dashboard +make && make dist +``` + +## plugin.config + +**Each plugin should have a 'etc/plugin.config' file** + +For example, project structure of emqttd_dashboard plugin: + +``` +LICENSE +README.md +ebin +etc +priv +rebar.config +src +``` + +etc/plugin.config for emqttd_dashboard plugin: + +``` +[ +{emqttd_dashboard, [ +{listener, +{emqttd_dashboard, 18083, [ +{acceptors, 4}, +{max_clients, 512}]}} +]} +]. +``` + +## rebar.config + +**Plugin should use 'rebar.config' to manage depencies** + +emqttd_plugin_pgsql plugin's rebar.config, for example: + +``` +%% -*- erlang -*- + +{deps, [ +{epgsql, ".*",{git, "https://github.com/epgsql/epgsql.git", {branch, "master"}}} +]}. +``` + +## Build emqttd with plugins + +Put all the plugins you required in 'plugins/' folder of emqttd project, and then: + +``` +make && make dist +``` + +## Load Plugin + +'./bin/emqttd_ctl' to load/unload plugin, when emqttd broker started. + +``` +./bin/emqttd_ctl plugins load emqttd_plugin_demo + +./bin/emqttd_ctl plugins unload emqttd_plugin_demo +``` + +## List Plugins + +``` +./bin/emqttd_ctl plugins list +``` + +## API + +``` +%% Load all active plugins after broker started +emqttd_plugins:load() + +%% Load new plugin +emqttd_plugins:load(Name) + +%% Unload all active plugins before broker stopped +emqttd_plugins:unload() + +%% Unload a plugin +emqttd_plugins:unload(Name) +``` + + diff --git a/docs/source/faq.rst b/docs/source/faq.rst new file mode 100644 index 000000000..e8933b2b9 --- /dev/null +++ b/docs/source/faq.rst @@ -0,0 +1,77 @@ +=== +FAQ +=== + + +##### Q1. Is port 4369 and another random port secure? + +``` +HI, when start emqttd , I found the port 4369 and another random port(63703) is open, are this security ? + +Example: +tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 13736/epmd +tcp 0 0 0.0.0.0:8083 0.0.0.0:* LISTEN 16745/beam.smp +tcp 0 0 0.0.0.0:8883 0.0.0.0:* LISTEN 16745/beam.smp +tcp 0 0 0.0.0.0:63703 0.0.0.0:* LISTEN 16745/beam.smp +tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN 16745/beam.smp + +1883: mqtt no ssl +8883: mqtt with ssl +8083: websocket +``` + +4369 and some random ports are opened by erlang node for internal communication. Configure your firewall to allow 1883, 8883, 8083 ports to be accessed from outside for security. + +Access control of emqttd broker has two layers: + +eSockd TCP Acceptor, Ipaddress based Access Control, Example: + +``` +{access, [{deny, "192.168.1.1"}, +{allow, "192.168.1.0/24"}, +{deny, all}]} +``` + +MQTT Subscribe/Publish Access Control by etc/acl.config, Example: + +``` +{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}. + +{deny, all, subscribe, ["$SYS/#", {eq, "#"}]}. + +{allow, all}. +``` + +##### Q2. cannot compile emqttd under Chinese folder? + +It seems that rebar cannot support Chinese folder name. + +##### Q3. emqttd is ready for production ? + +The core features are solid and scalable. A small full-time team with many contributors are developing this project. You could submit issues if any feature requests or bug reports. + +##### Q4. Benchmark and performance issue + +Wiki: https://github.com/emqtt/emqttd/wiki/One-Million-Connections + +##### Q5. 'session' identified by clientID？when session expired what will happen？All queued messages will be deleted and subscribed topics will be deleted too？when reconnected， need redo subscription？(#150) + +When a client connected to broker with 'clean session' flag 0, a session identified by clientId will be created. The session will expire after 48 hours(configured in etc/emqttd.config) if no client connections bind with it, and all queued messages and subscriptions will be dropped. + +##### Q6. "{max_queued_messages, 100}" in 0.8 release or "{queue, {max_length, 1000},..." means queue for one session or one topic？If it stands for session，while one topic has lots of offline messages(100), the user's other topic offline messages will be flushed? (#150) + +For session. Topic just dispatch messages to clients or sessions that matched the subscriptions. Will Flood. + +##### Q7. About the retained message, how to config one topic only keep the latest retained message, the older retained messages will be auto deleted?(#150) + +By default, the broker only keep the latest retained message of one topic. + +##### Q8. When the persistent client with 'clean session' flag 0 is offline but not expired, will the broker put session's subscribed topic new messages to session's queue?(#150) + +Yes + +##### Q9. If max_length of queue is 100, when the session subscribed topic1 and topic2, what will happen when topic1 fill 70 messages, then topic2 fill 80 messages? After the reconnection, will the session lose first 50 message?(#150) + +Lose the oldest 50 messages. + + diff --git a/docs/source/guide.rst b/docs/source/guide.rst index d3e7a893e..039e531f0 100644 --- a/docs/source/guide.rst +++ b/docs/source/guide.rst @@ -29,8 +29,283 @@ Will Message HTTP Publish API ---------------- +emqttd support HTTP API to publish message from your application server to MQTT clients. For example:: + +curl -v --basic -u user:passwd -d "qos=1&retain=0&topic=/a/b/c&message=hello from http..." -k http://localhost:8083/mqtt/publish + +HTTP API URL +------------ + +HTTP POST http://host:8083/mqtt/publish + +HTTP Parameters +--------------- + +Name | Description +--------|--------------- +client | ClientId +qos | QoS(0, 1, 2) +retain | Retain(0, 1) +topic | Topic +message | Message + + ------------------- MQTT Over WebSocket ------------------- +## Overview + +emqttd 0.7.0 release support MQTT over WebSocket(SSL). + +## URI + +``` +ws(s)://host:8083/mqtt +``` + +## Sec-WebSocket-Protocol + +'mqttv3.1', 'mqttv3.1.1' + + +----------------------- +Trace Topic or Client +----------------------- + +## Overview + +emqttd could trace packets received/sent from/to specific client, or trace publish/subscribe to specific topic. + +emqttd use lager:trace_file api and write trace log to file. + + +## Trace Commands + +### Trace client + +``` +./bin/emqttd_ctl trace client "ClientId" "trace_clientid.log" +``` + +### Trace topic + +``` +./bin/emqttd_ctl trace topic "Topic" "trace_topic.log" +``` + +### Stop Trace + +``` +./bin/emqttd_ctl trace client "ClientId" off +./bin/emqttd_ctl trace topic "Topic" off +``` + +### Lookup Traces + +``` +./bin/emqttd_ctl trace list +``` + + + +------------------- +$SYS Topics +------------------- + +NOTICE: This is the design of 0.9.0 release + +## Overview + +For emqttd is clustered, $SYS Topics of broker is started with: + +``` +$SYS/brokers/${node} +``` + +${node} is erlang node of clustered brokers. For example: + +``` +$SYS/brokers/emqttd@host1/version +$SYS/brokers/emqttd@host2/version +``` + +## Broker $SYS Topics + +Topic | Description +-------------------------------|------------ +$SYS/brokers | Broker nodes +$SYS/brokers/${node}/version | Broker Version +$SYS/brokers/${node}/uptime | Broker Uptime +$SYS/brokers/${node}/datetime | Broker DateTime +$SYS/brokers/${node}/sysdescr | Broker Description +  +## Client $SYS Topics + +Start with: $SYS/brokers/${node}/clients/ + +Topic | Payload(json) | Description +----------------------|---------------------|--------------- +${clientid}/connected | {ipaddress: "127.0.0.1", username: "test", session: false, version: 3, connack: 0, ts: 1432648482} | Publish when client connected +${clientid}/disconnected | {reason: "normal" | "keepalive_timeout" | "conn_closed"} + +Parameters of 'connected' Payload: + +``` +ipaddress: "127.0.0.1", +username: "test", +session: false, +protocol: 3, +connack: 0, +ts: 1432648482 +``` + +Parameters of 'disconnected' Payload: + +``` +reason: normal, +ts: 1432648486 +``` + +## Statistics $SYS Topics + +Start with '$SYS/brokers/${node}/stats/' + +### Client Stats + +Topic | Description +-------------------------------------|------------ +clients/count | count of current connected clients +clients/max | max connected clients in the same time + +### Session Stats + +Topic | Description +-----------------|------------ +sessions/count | count of current sessions +sessions/max | max number of sessions + +### Subscriber Stats + +Topic | Description +------------------|------------ +subscriptions/count | count of current subscriptions +subscriptions/max | max number of subscriptions + +### Topic Stats + +Topic | Description +------------------|------------ +topics/count | count of current topics +topics/max | max number of topics + +### Queue Stats + +Topic | Description +------------------|------------ +queues/count | count of current queues +queues/max | max number of queues + + +## Metrics $SYS Topics + +Start with '$SYS/brokers/${node}/metrics/' + +### Bytes sent and received + +Topic | Description +------------------------------------|------------ +bytes/received | MQTT Bytes Received since broker started +bytes/sent | MQTT Bytes Sent since the broker started + +### Packets sent and received + +Topic | Description +-------------------------|------------ +packets/received | MQTT Packets received +packets/sent | MQTT Packets sent +packets/connect | MQTT CONNECT Packet received +packets/connack | MQTT CONNACK Packet sent +packets/publish/received | MQTT PUBLISH packets received +packets/publish/sent | MQTT PUBLISH packets sent +packets/subscribe | MQTT SUBSCRIBE Packets received +packets/suback | MQTT SUBACK packets sent +packets/unsubscribe | MQTT UNSUBSCRIBE Packets received +packets/unsuback | MQTT UNSUBACK Packets sent +packets/pingreq | MQTT PINGREQ packets received +packets/pingresp | MQTT PINGRESP Packets sent +packets/disconnect | MQTT DISCONNECT Packets received + +### Messages sent and received + +Topic | Description +---------------------------------------|------------------- +messages/received | Messages Received +messages/sent | Messages Sent +messages/retained | Messages Retained +messages/stored | TODO: Messages Stored +messages/dropped | Messages Dropped + +## Alarm Topics + +Start with '$SYS/brokers/${node}/alarms/' + +Topic | Description +-----------------|------------------- +${alarmId}/alert | New Alarm +${alarmId}/clear | Clear Alarm + +## Log + +'$SYS/brokers/${node}/logs/${severity}' + +Severity | Description +-----------|------------------- +debug | Debug Log +info | Info Log +notice | Notice Log +warning | Warning Log +error | Error Log +critical | Critical Log + +## Sysmon + +Start with '$SYS/brokers/${node}/sysmon/' + +Topic | Description +-----------------|------------------- +long_gc | Long GC Warning +long_schedule | Long Schedule +large_heap | Large Heap Warning +busy_port | Busy Port Warning +busy_dist_port | Busy Dist Port + +## Log + +'$SYS/brokers/${node}/log/${severity}' + +Severity | Description +------------|------------------- +debug | Debug +info | Info Log +notice | Notice Log +warning | Warning Log +error | Error Log +critical | Critical Log +alert | Alert Log + +## VM Load Topics + +Start with '$SYS/brokers/${node}/vm/' + +Topic | Description +-----------------|------------------- +memory/* | TODO +cpu/* | TODO +processes/* | TODO + +## Sys Interval + +sys_interval: 1 minute default + diff --git a/docs/source/tune.rst b/docs/source/tune.rst index 8e5f1bba1..c4152a532 100644 --- a/docs/source/tune.rst +++ b/docs/source/tune.rst @@ -79,9 +79,9 @@ Timeout for FIN-WAIT-2 sockets:: net.ipv4.tcp_fin_timeout = 15 ---------- -Erlang VM ---------- +---------------- +Erlang VM Tuning +---------------- Tuning and optimize the Erlang VM in etc/vm.args file::