diff --git a/docs/README b/docs/README index d7f4fb801..d5c46fb45 100644 --- a/docs/README +++ b/docs/README @@ -5,3 +5,7 @@ or http://emqttd-docs.rtfd.org +or + +http://emqttd.io/docs + diff --git a/docs/source/changes.rst b/docs/source/changes.rst index ab334485b..4e48e1ba9 100644 --- a/docs/source/changes.rst +++ b/docs/source/changes.rst @@ -5,6 +5,107 @@ Changes ======= +.. _release_1.1: + +----------- +Version 1.1 +----------- + +*Release Date: 2016-06-01* + +Highlights +---------- + +Upgrade eSockd library to 4.0 and Support IPv6 + +Support to listen on specific IP Address:: + + {mqtt, {"192.168.1.20", 1883}, [ + ... + ]}, + +Add MongoDB, HTTP Authentication/ACL Plugins + +Upgrade MySQL, PostgreSQL, Redis Plugins to support superuser authentication and avoid SQL Injection + +Enhancements +------------ + +Allow human-friendly IP addresses (PR#395) + +File operation error: emfile (#445) + +emqttd_plugin_mongo not found in emqttd (#489) + +emqttd_plugin_mongo Error While Loading in emqttd (#505) + +Feature request: HTTP Authentication (#541) + +Compatible with the Qos0 PUBREL packet (#575) + +Bugfix +------ + +Bugfix: function_clause exception occurs when registering a duplicated authentication module (#542) + +Bugfix: ./emqttd_top msg_q result: {"init terminating in do_boot",{undef,[{etop,start,[],[]},{init,start_it,1,[]},{init,start_em,1,[]}]}} (#557) + +Tests +----- + +111 common test cases. + +Dashboard Plugin +---------------- + +WebSocket Page: Support 'Clean Session', Qos, Retained parameters (emqttd_dashboard#52) + +Upgrade eSockd library to 4.0, Show OTP Release on Overview Page (emqttd_dashboard#61) + +Changing dashboard credentials for username authentication (emqttd_dashboard#56) + +Add './bin/emqttd_ctl admins' CLI，support to add/delete admins + +HTTP Auth Plugin +---------------- + +Authentication/ACL by HTTP API: https://github.com/emqtt/emqttd_auth_http + +MongoDB Plugin +-------------- + +Upgrade Erlang MongoDB driver to v1.0.0 + +Support superuser authentication + +Support ACL (emqttd_plugin_mongo#3) + +MySQL Plugin +------------ + +Support superuser authentication + +Use parameterized query to avoid SQL Injection + +Postgre Plugin +-------------- + +Support superuser authentication + +Use parameterized query to avoid SQL Injection + +Redis Plugin +------------ + +Support superuser authentication + +Support ClientId authentication by '%c' variable + +Reloader Plugin +--------------- + +Reload modified modules during development automatically. + .. _release_1.0.3: ------------- diff --git a/docs/source/commands.rst b/docs/source/commands.rst index 3aff69d5b..7b4243c96 100644 --- a/docs/source/commands.rst +++ b/docs/source/commands.rst @@ -20,7 +20,7 @@ Show running status of the broker:: $ ./bin/emqttd_ctl status Node 'emqttd@127.0.0.1' is started - emqttd 0.16.0 is running + emqttd 1.1 is running .. _command_broker:: @@ -756,3 +756,41 @@ mnesia Show system_info of mnesia database. +------ +admins +------ + +The 'admins' CLI is used to add/del admin account, which is registered by the dashboard plugin. + ++------------------------------------+-----------------------------+ +| admins add | Add admin account | ++------------------------------------+-----------------------------+ +| admins passwd | Reset admin password | ++------------------------------------+-----------------------------+ +| admins del | Delete admin account | ++------------------------------------+-----------------------------+ + +admins add +---------- + +Add admin account:: + + $ ./bin/emqttd_ctl admins add root public + ok + +admins passwd +------------- + +Reset password:: + + $ ./bin/emqttd_ctl admins passwd root private + ok + +admins del +---------- + +Delete admin account:: + + $ ./bin/emqttd_ctl admins del root + ok + diff --git a/docs/source/getstarted.rst b/docs/source/getstarted.rst index e22f853d0..55b11164f 100644 --- a/docs/source/getstarted.rst +++ b/docs/source/getstarted.rst @@ -41,7 +41,7 @@ Features * IpAddress Authentication * Username and Password Authentication * Access control based on IpAddress, ClientID, Username -* Authentication with LDAP, Redis, MySQL, PostgreSQL +* Authentication with LDAP, Redis, MySQL, PostgreSQL and HTTP API * Cluster brokers on several servers * Bridge brokers locally or remotely * mosquitto, RSMB bridge @@ -63,7 +63,7 @@ Installing on Mac, for example: .. code-block:: bash - unzip emqttd-macosx-0.16.0-beta-20160216.zip && cd emqttd + unzip emqttd-macosx-1.1-beta-20160601.zip && cd emqttd # Start emqttd ./bin/emqttd start @@ -77,7 +77,7 @@ Installing on Mac, for example: Installing from Source ---------------------- -.. NOTE:: emqttd broker requires Erlang R17+ to build. +.. NOTE:: emqttd broker requires Erlang R18+ to build since 1.1 release. .. code-block:: bash @@ -163,6 +163,8 @@ A plugin is an Erlang application to extend the emqttd broker. +----------------------------+-----------------------------------+ | `emqttd_dashboard`_ | Web Dashboard | +----------------------------+-----------------------------------+ +| `emqttd_auth_http`_ | Authentication/ACL with HTTP API | ++----------------------------+-----------------------------------+ | `emqttd_plugin_mysql`_ | Authentication with MySQL | +----------------------------+-----------------------------------+ | `emqttd_plugin_pgsql`_ | Authentication with PostgreSQL | @@ -289,6 +291,7 @@ GitHub: https://github.com/emqtt .. _emqttd_plugin_template: https://github.com/emqtt/emqttd_plugin_template .. _emqttd_dashboard: https://github.com/emqtt/emqttd_dashboard +.. _emqttd_auth_http: https://github.com/emqtt/emqttd_auth_http .. _emqttd_plugin_mysql: https://github.com/emqtt/emqttd_plugin_mysql .. _emqttd_plugin_pgsql: https://github.com/emqtt/emqttd_plugin_pgsql .. _emqttd_plugin_redis: https://github.com/emqtt/emqttd_plugin_redis diff --git a/docs/source/install.rst b/docs/source/install.rst index 9939d0c07..044d5e85a 100644 --- a/docs/source/install.rst +++ b/docs/source/install.rst @@ -35,7 +35,7 @@ Download binary packages from: http://emqtt.io/downloads The package name consists of platform, version and release time. -For example: emqttd-centos64-0.16.0-beta-20160216.zip +For example: emqttd-centos64-1.1-beta-20160601.zip .. _install_on_linux: @@ -47,7 +47,7 @@ Download CentOS Package from: http://emqtt.io/downloads/centos, and then unzip: .. code-block:: bash - unzip emqttd-centos64-0.16.0-beta-20160216.zip + unzip emqttd-centos64-1.1-beta-20160601.zip Start the broker in console mode: @@ -80,7 +80,7 @@ If the broker is started successfully, console will print: mqtt listen on 0.0.0.0:1883 with 16 acceptors. mqtts listen on 0.0.0.0:8883 with 4 acceptors. http listen on 0.0.0.0:8083 with 4 acceptors. - Erlang MQTT Broker 0.16.0 is running now + Erlang MQTT Broker 1.1 is running now Eshell V6.4 (abort with ^G) (emqttd@127.0.0.1)1> @@ -100,7 +100,7 @@ Check the running status of the broker: $ ./bin/emqttd_ctl status Node 'emqttd@127.0.0.1' is started - emqttd 0.16.0 is running + emqttd 1.1 is running Or check the status by URL:: diff --git a/docs/source/plugins.rst b/docs/source/plugins.rst index 7d42a9514..6c4fa552a 100644 --- a/docs/source/plugins.rst +++ b/docs/source/plugins.rst @@ -16,18 +16,24 @@ The plugins that emqtt project released: +---------------------------+---------------------------+ | `emqttd_dashboard`_ | Web Dashboard | +---------------------------+---------------------------+ +| `emqttd_auth_http`_ | HTTP Auth/ACL Plugin | ++---------------------------+---------------------------+ | `emqttd_plugin_mysql`_ | MySQL Auth/ACL Plugin | +---------------------------+---------------------------+ | `emqttd_plugin_pgsql`_ | PostgreSQL Auth/ACL Plugin| +---------------------------+---------------------------+ | `emqttd_plugin_redis`_ | Redis Auth/ACL Plugin | +---------------------------+---------------------------+ +| `emqttd_plugin_mongo`_ | MongoDB Auth/ACL Plugin | ++---------------------------+---------------------------+ | `emqttd_stomp`_ | STOMP Protocol Plugin | +---------------------------+---------------------------+ | `emqttd_sockjs`_ | STOMP over SockJS Plugin | +---------------------------+---------------------------+ | `emqttd_recon`_ | Recon Plugin | +---------------------------+---------------------------+ +| `emqttd_reloader`_ | Reloader Plugin | ++---------------------------+---------------------------+ ---------------------------------------- emqttd_plugin_template - Template Plugin @@ -81,10 +87,6 @@ emqttd_dashboard/etc/plugin.config: [ {emqttd_dashboard, [ - {default_admin, [ - {login, "admin"}, - {password, "public"} - ]}, {listener, {emqttd_dashboard, 18083, [ {acceptors, 4}, @@ -93,6 +95,75 @@ emqttd_dashboard/etc/plugin.config: ]} ]. +--------------------------------------- +emqttd_auth_http - HTTP Auth/ACL Plugin +--------------------------------------- + +MQTT Authentication/ACL with HTTP API: https://github.com/emqtt/emqttd_auth_http + +.. NOTE:: Supported in 1.1 release + +Configure emqttd_auth_http/etc/plugin.config +-------------------------------------------- + +.. code:: erlang + + [ + + {emqttd_auth_http, [ + + %% Variables: %u = username, %c = clientid, %a = ipaddress, %t = topic + + {super_req, [ + {method, post}, + {url, "http://localhost:8080/mqtt/superuser"}, + {params, [ + {username, "%u"}, + {clientid, "%c"} + ]} + ]}, + + {auth_req, [ + {method, post}, + {url, "http://localhost:8080/mqtt/auth"}, + {params, [ + {clientid, "%c"}, + {username, "%u"}, + {password, "%P"} + ]} + ]}, + + %% 'access' parameter: sub = 1, pub = 2 + + {acl_req, [ + {method, post}, + {url, "http://localhost:8080/mqtt/acl"}, + {params, [ + {access, "%A"}, + {username, "%u"}, + {clientid, "%c"}, + {ipaddr, "%a"}, + {topic, "%t"} + ]} + ]} + ]} + + ]. + +HTTP API +-------- + +Return 200 if ok + +Return 4xx if unauthorized + +Load emqttd_auth_http plugin +---------------------------- + +.. code:: bash + + ./bin/emqttd_ctl plugins load emqttd_auth_http + ------------------------------------------- emqttd_plugin_mysql - MySQL Auth/ACL Plugin ------------------------------------------- @@ -109,6 +180,7 @@ MQTT User Table `username` varchar(100) DEFAULT NULL, `password` varchar(100) DEFAULT NULL, `salt` varchar(20) DEFAULT NULL, + `is_superuser` tinyint(1) DEFAULT 0, `created` datetime DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `mqtt_username` (`username`) @@ -130,6 +202,14 @@ MQTT ACL Table PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; + INSERT INTO `mqtt_acl` (`id`, `allow`, `ipaddr`, `username`, `clientid`, `access`, `topic`) + VALUES + (1,1,NULL,'$all',NULL,2,'#'), + (2,0,NULL,'$all',NULL,1,'$SYS/#'), + (3,0,NULL,'$all',NULL,1,'eq #'), + (5,1,'127.0.0.1',NULL,NULL,2,'$SYS/#'), + (6,1,'127.0.0.1',NULL,NULL,2,'#'), + (7,1,NULL,'dashboard',NULL,1,'$SYS/#'); Configure emqttd_plugin_mysql/etc/plugin.config ----------------------------------------------- @@ -140,11 +220,11 @@ Configure MySQL host, username, password and database: [ - {emqttd_plugin_mysql, [ + {emqttd_plugin_mysql, [ {mysql_pool, [ %% ecpool options - {pool_size, 4}, + {pool_size, 8}, {auto_reconnect, 3}, %% mysql options @@ -156,10 +236,15 @@ Configure MySQL host, username, password and database: {encoding, utf8} ]}, - %% select password only + %% Variables: %u = username, %c = clientid, %a = ipaddress + + %% Superuser Query + {superquery, "select is_superuser from mqtt_user where username = '%u' limit 1"}, + + %% Authentication Query: select password only {authquery, "select password from mqtt_user where username = '%u' limit 1"}, - %% hash algorithm: md5, sha, sha256, pbkdf2? + %% hash algorithm: plain, md5, sha, sha256, pbkdf2? {password_hash, sha256}, %% select password with salt @@ -171,12 +256,15 @@ Configure MySQL host, username, password and database: %% sha256 with salt suffix %% {password_hash, {sha256, salt}}, - %% comment this query, the acl will be disabled - {aclquery, "select * from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'"}, + %% '%a' = ipaddress, '%u' = username, '%c' = clientid + %% Comment this query, the acl will be disabled + {aclquery, "select allow, ipaddr, username, clientid, access, topic from mqtt_acl where ipaddr = '%a' or username = '%u' or username = '$all' or clientid = '%c'"}, - %% If no rules matched, return... + %% If no ACL rules matched, return... {acl_nomatch, allow} - ]} + + ]} + ]. Load emqttd_plugin_mysql plugin @@ -199,6 +287,7 @@ MQTT User Table CREATE TABLE mqtt_user ( id SERIAL primary key, + is_superuser boolean, username character varying(100), password character varying(100), salt character varying(40) @@ -240,23 +329,29 @@ Configure host, username, password and database of PostgreSQL: {emqttd_plugin_pgsql, [ {pgsql_pool, [ - %% ecpool options - {pool_size, 4}, - {auto_reconnect, 3}, + %% ecpool options + {pool_size, 8}, + {auto_reconnect, 3}, - %% pgsql options - {host, "localhost"}, - {port, 5432}, - {username, "feng"}, - {password, ""}, - {database, "mqtt"}, - {encoding, utf8} + %% pgsql options + {host, "localhost"}, + {port, 5432}, + {ssl, false}, + {username, "feng"}, + {password, ""}, + {database, "mqtt"}, + {encoding, utf8} ]}, - %% select password only + %% Variables: %u = username, %c = clientid, %a = ipaddress + + %% Superuser Query + {superquery, "select is_superuser from mqtt_user where username = '%u' limit 1"}, + + %% Authentication Query: select password only {authquery, "select password from mqtt_user where username = '%u' limit 1"}, - %% hash algorithm: md5, sha, sha256, pbkdf2? + %% hash algorithm: plain, md5, sha, sha256, pbkdf2? {password_hash, sha256}, %% select password with salt @@ -288,7 +383,7 @@ Load emqttd_plugin_pgsql Plugin emqttd_plugin_redis - Redis Auth/ACL Plugin ------------------------------------------- -MQTT Authentication, ACL with Redis. +MQTT Authentication, ACL with Redis: https://github.com/emqtt/emqttd_plugin_redis Configure emqttd_plugin_redis/etc/plugin.config ----------------------------------------------- @@ -310,6 +405,11 @@ Configure emqttd_plugin_redis/etc/plugin.config {password, ""} ]}, + %% Variables: %u = username, %c = clientid + + %% HMGET mqtt_user:%u is_superuser + {supercmd, ["HGET", "mqtt_user:%u", "is_superuser"]}, + %% HMGET mqtt_user:%u password {authcmd, ["HGET", "mqtt_user:%u", "password"]}, @@ -322,18 +422,37 @@ Configure emqttd_plugin_redis/etc/plugin.config %% If no rules matched, return... {acl_nomatch, deny}, - %% Store subscriptions to redis when SUBSCRIBE packets received. - {subcmd, ["HMSET", "mqtt_subs:%u"]}, - %% Load Subscriptions form Redis when client connected. - {loadsub, ["HGETALL", "mqtt_subs:%u"]}, - - %% Remove subscriptions from redis when UNSUBSCRIBE packets received. - {unsubcmd, ["HDEL", "mqtt_subs:%u"]} - + {subcmd, ["HGETALL", "mqtt_subs:%u"]} ]} ]. +User HASH +--------- + +Set a 'user' hash with 'password' field, for example:: + + HSET mqtt_user: is_superuser 1 + HSET mqtt_user: password "passwd" + +ACL Rule SET +------------ + +The plugin uses a redis SET to store ACL rules:: + + SADD mqtt_acl: "publish topic1" + SADD mqtt_acl: "subscribe topic2" + SADD mqtt_acl: "pubsub topic3" + +Subscription HASH +----------------- + +The plugin can store static subscriptions in a redis Hash:: + + HSET mqtt_subs: topic1 0 + HSET mqtt_subs: topic2 1 + HSET mqtt_subs: topic3 2 + Load emqttd_plugin_redis Plugin ------------------------------- @@ -341,6 +460,114 @@ Load emqttd_plugin_redis Plugin ./bin/emqttd_ctl plugins load emqttd_plugin_redis +--------------------------------------------- +emqttd_plugin_mongo - MongoDB Auth/ACL Plugin +--------------------------------------------- + +MQTT Authentication, ACL with MongoDB: https://github.com/emqtt/emqttd_plugin_mongo + +Configure emqttd_plugin_mongo/etc/plugin.config +----------------------------------------------- + +.. code-block:: erlang + + [ + {emqttd_plugin_mongo, [ + + {mongo_pool, [ + {pool_size, 8}, + {auto_reconnect, 3}, + + %% Mongodb Driver Opts + {host, "localhost"}, + {port, 27017}, + %% {login, ""}, + %% {password, ""}, + {database, "mqtt"} + ]}, + + %% Variables: %u = username, %c = clientid + + %% Superuser Query + {superquery, [ + {collection, "mqtt_user"}, + {super_field, "is_superuser"}, + {selector, {"username", "%u"}} + ]}, + + %% Authentication Query + {authquery, [ + {collection, "mqtt_user"}, + {password_field, "password"}, + %% Hash Algorithm: plain, md5, sha, sha256, pbkdf2? + {password_hash, sha256}, + {selector, {"username", "%u"}} + ]}, + + %% ACL Query: "%u" = username, "%c" = clientid + {aclquery, [ + {collection, "mqtt_acl"}, + {selector, {"username", "%u"}} + ]}, + + %% If no ACL rules matched, return... + {acl_nomatch, deny} + + ]} + ]. + +MongoDB Database +---------------- + +.. code-block:: + + use mqtt + db.createCollection("mqtt_user") + db.createCollection("mqtt_acl") + db.mqtt_user.ensureIndex({"username":1}) + +User Collection +--------------- + +.. code-block:: json + + { + username: "user", + password: "password hash", + is_superuser: boolean (true, false), + created: "datetime" + } + +For example:: + + db.mqtt_user.insert({username: "test", password: "password hash", is_superuser: false}) + db.mqtt_user:insert({username: "root", is_superuser: true}) + +ACL Collection +-------------- + +.. code-block:: json + + { + username: "username", + clientid: "clientid", + publish: ["topic1", "topic2", ...], + subscribe: ["subtop1", "subtop2", ...], + pubsub: ["topic/#", "topic1", ...] + } + +For example:: + + db.mqtt_acl.insert({username: "test", publish: ["t/1", "t/2"], subscribe: ["user/%u", "client/%c"]}) + db.mqtt_acl.insert({username: "admin", pubsub: ["#"]}) + +Load emqttd_plugin_mongo Plugin +------------------------------- + +.. code-block:: bash + + ./bin/emqttd_ctl plugins load emqttd_plugin_mongo + ----------------------------- emqttd_stomp - STOMP Protocol ----------------------------- @@ -454,6 +681,29 @@ Recon CLI recon node_stats #recon:node_stats(10, 1000) recon remote_load Mod #recon:remote_load(Mod) +--------------------------------- +emqttd_reloader - Reloader Plugin +--------------------------------- + +Erlang Module Reloader for Development + +.. NOTE:: Don't load the plugin in production! + +Load emqttd_reloader Plugin +--------------------------- + +.. code-block:: bash + + ./bin/emqttd_ctl plugins load emqttd_reloader + +reload CLI +---------- + +.. code-block:: bash + + ./bin/emqttd_ctl reload + + reload # Reload a Module ------------------------ Plugin Development Guide @@ -602,11 +852,15 @@ There will be a new CLI after the plugin loaded:: .. _emqttd_dashboard: https://github.com/emqtt/emqttd_dashboard +.. _emqttd_auth_http: https://github.com/emqtt/emqttd_auth_http .. _emqttd_plugin_mysql: https://github.com/emqtt/emqttd_plugin_mysql .. _emqttd_plugin_pgsql: https://github.com/emqtt/emqttd_plugin_pgsql .. _emqttd_plugin_redis: https://github.com/emqtt/emqttd_plugin_redis +.. _emqttd_plugin_mongo: https://github.com/emqtt/emqttd_plugin_mongo .. _emqttd_stomp: https://github.com/emqtt/emqttd_stomp .. _emqttd_sockjs: https://github.com/emqtt/emqttd_sockjs .. _emqttd_recon: https://github.com/emqtt/emqttd_recon +.. _emqttd_reloader: https://github.com/emqtt/emqttd_reloader .. _emqttd_plugin_template: https://github.com/emqtt/emqttd_plugin_template .. _recon: http://ferd.github.io/recon/ + diff --git a/plugins/emqttd_dashboard b/plugins/emqttd_dashboard index f841ab9b4..26c727be2 160000 --- a/plugins/emqttd_dashboard +++ b/plugins/emqttd_dashboard @@ -1 +1 @@ -Subproject commit f841ab9b4ca40f47575b5f98c6707c6598ef0f91 +Subproject commit 26c727be2b31180780d582e460e20fe418af1909