MQTT: Get started with IoT protocols

 

MQTT is a machine-to-machine, Internet of Things connectivity protocol. It is an extremely lightweight publish-subscribe communication model, useful for connections in remote locations where a small code footprint is the order of the day.

MQTT is a telemetry protocol based on on on the publish-subscribe communication model. It was initially developed by IBM and is of OASIS standard now, with the latest release of v3.1.1 in 2013. Formerly, the acronym MQTT stood for MQ Telemetry Transport. This lightweight, open, simple and easy-to-implement protocol is suitable for constrained environments and applicable in M2M and IoT contexts. MQTT tools have a minimal code footprint and allow optimal bandwidth usage. It runs on TCP by default, but can be run on any ordered, loss-less, bi-directional protocol.

MQTT nodes communicate in a one-to-many mapping model, where a message sent by one client (the publisher) is delivered to many clients (subscribers) through topic names. Messages are exchanged via a central node known as the broker.

The MQTT packet structure
A MQTT packet consists of a fixed header with a minimum size of 2 bytes and a maximum size of 5 bytes, which makes it very lightweight compared to other protocols like HTTP, XMPP, etc. After the fixed header, variable header elements and optional payload may follow. The first byte of the header comprises a 4 bit packet type and a few flags known as header flags. The second byte onwards represents the remaining length covering the optional header and payload, excluding the fixed header.

MQTT uses a variable length scheme to fit the remaining length with a minimum of one byte and up to four bytes encoded, as follows. The most significant bit of each byte is considered as the continuation bit, where 0 means the last byte of length and 1 means that the length follows in the next byte. The continuation bit of the fourth byte must be zero. Values up to 127 are encoded in a single byte with the continuation bit as zero.

For example, a value of 390 is encoded as 0x86 (cbit=1, LSB 7 bits=6) in byte1 and 0x03 (cbit=0, LSB 7 bits=3) in byte2, where the remaining length is computed as [6*1280+6*1281].

If a, b, c and d are magnitudes of the least significant 7 bits of each byte in the 0-127 range, length is computed as a*1280 + b*1281 + c*1282 + d*1283. This allows a maximum payload up to 256MB, but one should consider TCP limits to avoid fragmentation at low levels and optimal transfers.

Control codes for various packet types are shown in Figure 3.
Let’s have a quick look at the features of MQTT, some of its tools and libraries before going into the details of header flags, variable header components and payload elements.

Topics
MQTT packets sent by publishers are delivered to all subscribers registered for the same topic or matching topic filter. The topic name can be hierarchical with ‘/’ as the delimiter in between to avoid collisions; e.g., a topic name can be in the form of a country/state/city/temperature, city/area/street/light, hospital/ward/patient/pulse, etc. Most cloud platforms suggest user-specific patterns for starting levels of a topic.

Two wild cards are allowed in topic names, where ‘+’ matches a single level in the topic name and ‘#’ matches multiple levels; e.g., let’s consider four published topics: (1) a/b/c/d, (2) a/b/x/y, (3) a/b/c/y, and (4) a/b/x/d. Now the topic filter a/b/+/d matches (1) and (4); a/b/c/+ matches (1) and (3); and a/b/# matches all.

QoS levels
MQTT supports three levels of QoS, specified by each published message and while subscribers are connecting.

QoS 0 (Maximum once): This is also known as ‘fire and forget’; no acknowledgement is sent by the receiver.

QoS 1 (At least once): Each published message will be acknowledged using PUBACK; the sender retransmits a message if no acknowledgement is received within a time-out by setting the DUP flag.
QoS 2 (Exactly once): Sender and receiver exchange PUBLISH, PUBREC, PUBREL, PUBCOMP to ensure assured delivery of messages without duplicates.

Level 0 is faster but less reliable, whereas Level 2 is the most reliable, yet slow.
Published messages will be delivered to subscribers with the 0x3 packet type, and QoS level will be downgraded to the specified value of the subscriber. For example, if the publisher QoS is 2 and the subscriber QoS is 1, the messages will be delivered to the subscriber using QoS 1 only.
Retain flag: If the retain flag part of header flags is set, subscribers can get the last message published on this topic, which is retained by the broker.

Clean session: By default, this flag is set in ‘connect flags’ by subscribers, and published messages will be stored and delivered when they disconnect and reconnect again. We may disable this flag to discard messages published in between.

Last will and testament (LWT): Clients can specify a will topic, will message or will QoS and set the ‘will flag’ to notify other clients when they terminate abnormally without the DISCONNECT message.

Keep alive timer: When there is no message flow for a particular duration specified while connecting, the client sends a PINGREQ and the broker replies by sending a PINGRESP indicating the ‘liveness’ of the connection.

MQTT brokers
MQTT clients exchange messages via the broker node. The broker is not identical to a typical server, as apart from message reception and delivery, it has little functionality. For additional functionality like logging, message persistence, visualisation, analytics, Web integration, etc, one should consider additional subscribers or develop plugins for the broker.

Mosquitto is an Eclipse IOT project, lightweight broker implementation written in C and it supports MQTT protocol versions 3.1 and 3.1.1. For building Mosquitto, install the dependency libcares-devel.

Download Mosquitto-1.4.9.tar.gz (or any recent stable version) from https://mosquitto.org/download/, extract it and switch into…

tar -zxvf mosquitto-1.4.9.tar.gz
cd mosquitto-1.4.9
mkdir build && cd build
cmake .. #install any other missing dependencies
make
make install

You can run the broker using the command mosquitto located in /usr/local/sbin, which is copied from build/src during installation. It runs on TCP Port 1883, by default.

Other open source implementations of the MQTT broker are:

• Emqttd, Vernemq written in Erlang
• Mosca in Nodejs (npm package)
• Surgemq in Go language
• Moquette, Vertx-mqtt-broker in Java
• HBMQTT in Python

Apache ActiveMQ, ApacheApollo and RabitMQ also support the protocol through their plugins.
A majority of these brokers target the latest version 3.1.1, but some still support v3.1 (IBM version) for backward compatibility.

MQTT clients
There are many implementations of MQTT clients, but we’ll only discuss the most popular. One is the Eclipse Paho client library with support for many languages and the other is from the Mosquitto library.
Paho provides APIs for two types of clients for operations like connect, publish, subscribe, unsubscribe, etc.
1. Synchronous API: The above operations are blocked until they are completed and run in a single thread.
2. Asynchronous API: Call backs are specified using threads, to notify clients when the above operations are completed using the concerned acknowledgements.

Building a Paho C library
First, type the following commands:

git clone https://github.com/eclipse/paho.mqtt.c
cd paho.mqtt.c
cmake . #may go for a clean directory
make 
make install

This builds and installs four types of libraries in /usr/local/lib:

• libpaho-mqtt3c.* for synchronous clients over TCP
• libpaho-mqtt3cs.* for synchronous clients over SSL
• libpaho-mqtt3a.* for asynchronous clients over TCP
• libpaho-mqtt3as.* for asynchronous clients over SSL

You can also find various reference examples and tools installed in /usr/local/bin. paho_c_pub, paho_c_sub are generic synchronous clients, which take the topic name as argument; they take options like –host, –port, –qos, –keepalive, –clientid, –username, –password for both the clients and –retained for the publisher.

Mosquitto clients
You can also obtain clients by building the Mosquitto source as mentioned above. These are available as mosquitto_pub, mosquitto_sub and installed to /usr/local/bin, which are copied from the client sub-directory during installation. Here is how these tools are used:

mosquitto_pub -h hostname -p portno -t topicname -m payload
mosquitto_sub -h hostname -p portno -t topicfilter

Other important options of these tools are listed below.
-i: client ID
-u: username
-p: password
-k: keep alive duration
-q: QoS level
-r: retain message
-l: payload from stdin(single line)
-s: payload from stdin(multi line)
–will-topic, –will-message, –will-retain: LWT support
-c: disabling clean session
By default, Mosquitto clients use v3.1. Specify -V mqttv311 to enforce v3.1.1.
A few more client implementations are:

• Paho client libraries in many other languages
• Paho UI tool developed using Java Swing
• mqtt.js in Nodejs through npm also provides a simple command line tool
• NodeRED nodes for MQTT
• mqtt-spy in Java
• MQTTLens, a Chrome plugin

Android apps
There are many apps for talking to MQTT brokers. Among these, a few come with the source code. The sample app part of Eclipse Paho Android service is a good one to start I feel, and can be customised for your needs. For this, check out the code from http://www.eclipse.org/paho/clients/android/sample/ and build using Android Studio or the Gradle build system, and deploy to the device.

Open the installed app and click on the ‘+’ symbol to provide broker details and hit Connect. Once connected, use the UI features on the publish, subscribe tabs and play around. Delivered messages based on subscriptions are notified using toast messages and listed in the history tab. You may use the public broker available at test.mosquitto.org for quick testing.

MQTT broker on the cloud
Many cloud platforms support connectivity through MQTT with certain topic patterns and payloads in plain text or JSON format. A few such platforms are listed below. Apart from hosting brokers, they offer features like dashboards with graph plotting, Web integration and interoperability with other protocols, etc.

• IBM Bluemix IOT App
• Opensensors.io
• ThingMQ
• Carriots
• CloudMQTT
• AWS IOT
  …and many more. Please explore these platforms and their connectivity options.

MQTT over Websockets
MQTT can run over Websockets also, which enables browser based clients. Mosquitto, Surgemq, etc, support Websockets through a custom port along with the default TCP port. Paho provides JavaScript clients, which can be embedded in a Web page and use the Websocket transport layer. Security over Websockets is still under evaluation with various brokers.

Using Wireshark for a better understanding of the protocol
Wireshark comes with MQTT support from version 1.12 onwards. Start Wireshark with suitable privileges, type mqtt in the filter box and hit Enter. Let’s observe the packet flow and anatomy of packets using any broker and Mosquitto or Paho clients with a given syntax.

1. For each packet, observe the message type (packet type) in the fixed header and Msg Len (remaining length).
2. For each connected client, you can find the packet flow of CONNECT, CONNACK and SUBSCRIBE, SUBACK for each subscription.

mosquitto_sub -t hello
mosquitto_pub -t hello -l #payload from stdin,line wise

3. In the CONNECT packet, find out the protocol name, version, connect flags, keep alive value and client ID as part of the variable header. Observe the protocol name and version for both v3.1 and v3.1.1 by specifying –V mqttv31 or -V mqttv311 for Mosquitto clients.
4. Identify connect flags (user name present, password present, will retain, will QoS, will flag, clean session, etc) in the variable header of the CONNECT packet.
5. For each publisher, you can find the flow of CONNECT, CONNACK, PUBLISH and DISCONNECT when the client terminates. You can also identify QoS level in header flags and there will be no acknowledgment for QoS 0:

mosquitto_pub -t hello -m abcd
mosquitto_pub -t hello -l #hit ctrl+D to quit

6. For QoS 1, observe the flow of PUBLISH, PUBACK. Also match the message identifier in PUBLISH, PUBACK packets.

mosquitto_pub -t hello -m abcd -q 1

7. For QoS 2, observe the flow of PUBLISH, PUBREC, PUBREL, PUBCOMP with the same message identifier.

mosquitto_pub -t hello -m abcd -q 2

8. Try various combinations of QoS levels between publish, subscribe and observe the downgrading of QoS levels, as well as the packet flow between broker and subscribers (better to force v3.1.1 using -V mqtt311 or use Paho clients for this testing).

mosquitto_sub -t hello --qos 2 
mosquitto_pub -t hello -m abcd --qos 1

9. Observe the retain value in the header flags of the PUBLISH message:

mosquitto_pub -t hello -m abcd -r

10. Observe the clean session in connect flags of the CONNECT packet of the subscriber, message ID as sub.1:

mosquitto_sub -t hello -c -i sub.1

11. Observe the flow of PINGREQ, PINGRESP for every 10 seconds if there is no message flow between client and broker for the keep alive duration.

mosquitto_sub -t hello -k 10

12. Identify the LWT fields with the following commands:

mosquitto_sub -t hello --will-topic “death” --will-payload “I am leaving...”
mosquitto_pub -t hello -l --will-topic “death” --will-payload “I am leaving...”

Any other client subscribing to the ‘death’ topic receives the message ‘I am leaving…’ when the above client terminates abnormally without DISCONNECT. You can try this out by terminating the client with Ctrl+c.

You can also try the options –will-qos, –will-retain and repeat the same steps.
13. Use the –u and -P options for authentication and observe the connect flags, username and password in the payload:

mosquitto_pub -u admin -P ***** -t hello -m abcd

The broker is assumed on the localhost in the above examples. Use the -h option to connect the broker on the other node.

The link https://www.wireshark.org/docs/dfref/m/mqtt.html summarises Wireshark support for MQTT.

This is a gist of the protocol and some implementation examples. Do refer to the given links and explore more about the protocol, its security aspects, message persistence and detailed usage of specified client libraries, tools and cloud platforms, based on these initial pointers.