Introduction

MQTT is evolving into the de facto protocol in the IoT world due to its topic-based pub/sub messaging pattern, which is quite different from the req/res model in the RESTful world. It requires us to think differently when it comes to designing topic-based APIs, that said, designing in a topic-centralized way.

This document is created with the intention to facilitate MQTT development in terms of message subscribing and publishing, topics design, testing, or even just for fun.

Audience

• Developers who are developing MQTT features.
• Those who are interested in and want to get their hands dirty with MQTT basics.

Goals

1. Set up a local MQTT broker and clients for message subscriptions and publications.
2. Visualize topics as well as corresponding payloads on the go.
3. Inspire topics design or design topics in a standardized way.

Setup MQTT Broker

While there are a bunch of brokers when it comes to Choosing an MQTT broker for your IoT project, the lightweight and open-sourced Mosquitto broker is demonstrated here due to its simplicity and support for MQTT v5 and ACL in its most recent v2 release.

1. Install mosquitto on the host machine(Mac)
   Download and install mosquitto as per the guidance from the official homepage.

brew install mosquitto

2. Install and run Docker-based Eclipse Mosquitto Broker.

• Pull cedalo/installer docker image

docker pull cedalo/installer:2-macos

• Run from the docker image

docker run -it -v ~/cedalo_platform:/cedalo cedalo/installer:2-macos

• Brings the broker up by running the start script

cd ~/cedalo_platform/
./start.sh

Once started, the following components should be working in the specific ports now

• Eclipse Mosquitto: mqtt://localhost:1883
• Eclipse Streamsheets Web UI: http://localhost:8081
• Mosquitto proxy(Management Center Web UI): http://localhost:8088

Navigate to the Management Web UI to take a look quick at what the dashboard looks like.

Screen Shot 2021-04-14 at 2.46.19 PM.png

Verify message pub & sub

1. Makes sure MQTT broker has started

docker ps

The outputs should print the running containers that look like

CONTAINER ID   IMAGE                             COMMAND                  CREATED        STATUS        PORTS                                                                       NAMES
bf19c83f9d46   cedalo/installer:2-macos          "./docker-entrypoint…"   22 hours ago   Up 22 hours                                                                               loving_kare
e98c90d8deea   cedalo/streamsheets:2-milestone   "docker-entrypoint.s…"   43 hours ago   Up 22 hours   1883/tcp, 6379/tcp, 8080/tcp, 8083/tcp, 27017/tcp, 0.0.0.0:8081->8081/tcp   cedalo_platform_streamsheets_1
2722f7f34f1c   cedalo/management-center:2        "docker-entrypoint.s…"   43 hours ago   Up 22 hours   0.0.0.0:8088->8088/tcp                                                      cedalo_platform_management-center_1
3479c5c781c7   eclipse-mosquitto:2-openssl       "/docker-entrypoint.…"   43 hours ago   Up 22 hours   0.0.0.0:1883->1883/tcp                                                      cedalo_platform_mosquitto_1

If the containers were down, try to bring them up

docker run -it -v ~/cedalo_platform:/cedalo cedalo/installer:2-macos

2. Creates MQTT client(s) with unique username(s) to connect to the broker.
   This is optional if anonymous authentication is allowed, which is not enabled by default as of Mosquitto v2 with the Dynamic security model enabled by default. To enable anonymous authentication, one needs to config allow_anonymousin the configuration file and restart the broker.

Enables allow_anonymous config

# ~cedalo_platform/mosquitto/config/mosquitto.conf
allow_anonymous true 

and restarts the broker to apply the new configuration (if it’s up now)

# ~cedalo_platform
./start.sh

Note: If anonymous authentication is disabled, then both publisher and subscriber should at least hold a client role, from the ACL(Access Control Lists)'s perspective, in order to connect to the broker, otherwise, subscription/publication requests will get denied and end up with an “authentication failed” error.

Here is a screenshot with two clients newly created.

Screen Shot 2021-04-14 at 5.16.47 PM.png

3. Subscribes to a test topic

mosquitto_sub -d -k 10 -v -h localhost -u test -P 123456 -t 'test/topic'

Sample outputs:

➜ mosquitto_sub -d -k 10 -v -h localhost -u test -P 123456 -t 'test/topic'
Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending SUBSCRIBE (Mid: 1, Topic: test/topic, QoS: 0, Options: 0x00)
Client (null) received SUBACK
Subscribed (mid: 1): 0
Client (null) received PUBLISH (d0, q0, r1, m0, 'test/topic', ... (26 bytes))
test/topic {"greeting": "Cool MQTT2"}
Client (null) sending PINGREQ
Client (null) received PINGRESP
Client (null) sending PINGREQ
Client (null) received PINGRESP
...

Notes:

• -d indicates to work in debug mode, printing verbose debug info. For example, the PINGREQ and PINGRESP pairs show an alive session.
• -k changes the keep-alive interval value from the default 60 to the specified.
• -v prints the topic name of the published messages.
• -h hostname (or IP address) of the machine the broker is running on. Here, as it’s running in the container on the local machine, so localhost is used to accept pub/sub requests from terminal clients on the same machine. Other clients running on e.g. a real iPhone/Android device, or a simulator/emulator should use the IP address of the dev machine so as to be able to connect to the broker.
• -p network port to connect to. Defaults to 1883 for plain MQTT and 8883 for MQTT over TLS.
• -u -P user name and password of the client to authenticate with. Optional if anonymous authentication is allowed supported.
• -t the target topic to subscribe to.

Other useful options that can be used

• -C stops subscribing and automatically disconnects once the specified amount of messages were received.
• -q changes the QoS level in the pub/sub requests.
• -c -x 60 -i cid disables clean session and shares session states for the same client ID in the new connection. which means the session will never expire after the client disconnects (prior to MQTT v5) or will last at least x seconds (60 here) before it expires (for MQTT V5+), starting from the time of the most recent connection.

4. Publishes test message to a test topic

mosquitto_pub -d -r             \
-h 192.168.7.123                \
-u test -P 123456               \
-t 'test/topic'                 \
-m "{\"greeting\": \"Cool MQTT2\"}"

Sample outputs:

Client (null) sending CONNECT
Client (null) received CONNACK (0)
Client (null) sending PUBLISH (d0, q0, r1, m1, 'test/topic', ... (26 bytes))
Client (null) sending DISCONNECT

Notes:

1. -r indicates the published message will be retained in the broker so that subsequent clients that subscribe to the same topic are able to receive the published message immediately once connected.
2. -m specified the message payload to be published to the target topic 'test/topic'. Note that JSON-formatted payload should be escaped. Freeformatter can be used for this purpose.

Here is a screenshot that depicts the activities coming from the broker, terminal publisher, terminal subscriber, Android publisher/subscriber.

Screen Shot 2021-04-15 at 3.03.14 PM.png

Screen Shot 2021-04-15 at 3.14.13 PM.png

1. The running broker dynamically prints the sub/sub events from clients.
2. The Android client log dumped from

adb logcat -v -t time | grep mqttkotlinsample

3. A terminal client subscribing to updates from the test topic.
4. A terminal client publishing message to the test topic.

Take also a look at the dynamic topics inside the broker from the Topic Inspector

Screen Shot 2021-04-15 at 3.19.51 PM.png

MQTT Topic Naming Scheme

As we know, topics are organized in a hierarchical way, to keep the hierarchical topic tree flexible, it is important to design the topic tree very carefully and leave room for future use cases.

1. MQTT Topic and Payload Design Notes

• Topic-hierarchy-centralized way
  Put as much information in the topic fields as possible
• Payload-centralized way
  Put as much information in the payload as possible
  Some principles for topics naming scheme should follow
• Human readability
• Traffic minimization
• Granular access control

Some Best Practice suggested by HiveMQ team:

• Never use a leading forward slash.
• Never use spaces in a topic.
• Keep the topic short and concise.
• Use only ASCII characters, avoid non-printable characters.

Last Will And Testament Message (LWT)

LWT is used to notify subscribers (through the broker) that the publisher has lost connectivity somehow, for example, suffered from an unexpected shutdown.

1. Starts a publisher with configured LWT message

mosquitto_pub -d -r -q 2     \
--repeat 3 --repeat-delay 30 \
-h 192.168.7.123 -p 1883     \
-u test -P 123456            \
-t 'test/topic' -m "{\"greeting\": \"Last will and testament test\"}" \
--will-topic 'test/lwt' --will-qos 2 --will-payload "publisher has lost connectivity"

--repeat 3 --repeat-delay 30 sending repeatable messages to simulate the publisher is working
--will-topic, --will-qos, --will-payload sends LWT message to the specified topic.

2. Starts a subscriber monitoring the LWT message on the will topic test/lwt

mosquitto_sub -d -k 10 -v \
-h localhost              \
-u test -P 123456         \
-t 'test/lwt'

3. Kills the publisher and observes if the LWT message was received.

kill -9 `ps aux | grep mosquitto_pub | awk '{print $2}'` 

Observed LWT message

...
Client (null) sending PINGREQ
Client (null) received PINGRESP
Client (null) received PUBLISH (d0, q0, r0, m0, 'test/lwt', ... (31 bytes))
test/lwt publisher has lost connectivity
Client (null) sending PINGREQ
Client (null) received PINGRESP
...

QoS Management

• QoS 0, At most once - the message is sent only once and the client and broker take no additional steps to acknowledge delivery (fire and forget).
• QoS 1, the message is re-tried by the sender multiple times until acknowledgment is received (acknowledged delivery).
• QoS 2, the sender and receiver engage in a two-level handshake to ensure only one copy of the message is received (assured delivery).

The two-level and four-step handshake flow ensure the exact delivery in the case of QoS 2.

Screen Shot 2021-04-16 at 2.30.22 PM.png

Note that the QoS value can be applied to either a sub or a pub request, however, the QoS level that’s eventually used to deliver the message from the broker to the subscriber is determined by the lower of the two, for example:

Screen Shot 2021-04-16 at 6.25.06 PM.png

1. subscribe message with qos = x to topic.
2. publish a message with qos = y to topic.
3. deliver the message with qos = min(x, y).

Security Concerns

Authentication

The methodology and procedure to validate that users are who they claim to be before they are allowed to log into a secure system and eventually get access to the system resources. It corresponds to the AUTH flow in the MQTT context.

Authorization

The process of giving the user permission to access a specific resource or function in a secure system, which corresponds to the ACL (Access Control Lists) rules for topics in the MQTT context.

• Mosquitto introduces the Dynamic Security plugin which features role-based topic authorization.
• EMQ X provides client pub/sub ACLs for Permission Management.

User name and password-based authentication

Certificates (TLS) based authentication

The goal here is to use a client certificate rather than plain username password text for authentication.

Prons:

• Encrypt and transport credentials and MQTT payload in a secure way.
• Generate and manage certificates on a per-client basis.

Cons:

MQTT TLS Security in Mosquitto

1. Generate CA certificate (for self-signed certificates purpose)

openssl req -new -x509 -days 365 -extensions v3_ca -keyout ca.key -out ca.crt

Note: Just in case of “Error Loading extension section v3_ca” error on MacOS, append the following config to the end of OpenSSL config file.

# sudo vi /etc/ssl/openssl.cnf
[ v3_ca ]
basicConstraints = critical, CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always, issuer:always

To inspect details about the generated cert file

openssl x509 -inform pem -in ca.crt -text

2. Prepare server certificate (to be signed by trusted CA), which is required by the MQTT broker running in TLS mode.

openssl req -out server.csr -key server.key -new

Create self-signed server certificate

openssl x509 -req                        \
-in server.csr                           \ 
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt                          \
-days 365

Note: The CN(Common Name, eg, fully qualified host name) field provided when filling up the certificate details is very important. So be careful to provide it.

Here is an example to explore the details in the generated server.crt using openssl command

Screen Shot 2021-04-20 at 12.22.53 PM.png

3. Prepare client certificate (to be signed by trusted CA) which is required to connect to the MQTT broker.

openssl req -out client.csr -key client.key -new

Create a self-signed client certificate

openssl x509 -req                           \
-in client.csr                              \
-CA ca.crt -CAkey ca.key -CAcreateserial    \
-out client.crt                             \
-days 365

Screen Shot 2021-04-20 at 12.33.09 PM.png

As of self-signed certificates no longer working in Android Q, the self-signed *.pem certificate should be exported to *.p12 format so as to be installed on Android 10+ devices.

openssl pkcs12 -export -in server.crt -inkey server.key -out server.p12

4. Config server certificates

• Move certs to config folder so that they can be accessed thru the shared folder between the broker (running in the docker) and the host machine.

➜ pwd && ll .
/Users/nling/cedalo_platform/mosquitto/config/cert
total 72
-rw-r--r--@ 1 nling  staff   1.4K Apr 20 00:19 ca.crt
-rw-r--r--@ 1 nling  staff   1.8K Apr 20 00:19 ca.key
-rw-r--r--@ 1 nling  staff    17B Apr 20 01:13 ca.srl
-rw-r--r--@ 1 nling  staff   1.1K Apr 20 01:11 client.crt
-rw-r--r--@ 1 nling  staff   1.0K Apr 20 01:11 client.csr
-rw-r--r--@ 1 nling  staff   1.6K Apr 20 01:10 client.key
-rw-r--r--@ 1 nling  staff   1.1K Apr 20 01:13 server.crt
-rw-r--r--@ 1 nling  staff   1.0K Apr 20 01:13 server.csr
-rw-r--r--@ 1 nling  staff   1.6K Apr 20 01:12 server.key
-rw-r--r--@ 1 nling  staff   2.4K Apr 20 15:20 server.p12

• Modify Mostiqutto config to apply the server configs.
  (➜ cert vim ../mosquitto.conf)

listener 1883

#allow_anonymous true

persistence true
persistence_location /mosquitto/data/

plugin /usr/lib/mosquitto_dynamic_security.so
plugin_opt_config_file /mosquitto/data/dynamic-security.json

listener                  8883
cafile                    mosquitto/config/cert/ca.crt
certfile                  mosquitto/config/cert/server.crt
keyfile                   mosquitto/config/cert/server.key
tls_version               tlsv1.2
require_certificate       true
use_identity_as_username  true

log_type all

Note: It’s possible to listen on both 1883 and 8883 ports.

• Export the 8883 port in the docker config so that the broker is accessed from outside the docker, failing to do so will lead to a “Connection Refused” error.

# ➜  cert vim ../../../docker-compose.yml
services:
  mosquitto:
    image: eclipse-mosquitto:2-openssl
    ports:
      - 1883:1883
      - 8883:8883
    networks:
      - cedalo-platform
    volumes:
      - ./mosquitto/config:/mosquitto/config
      - ./mosquitto/data:/mosquitto/data

• Restart the broker to bring it up again, listening on 8883 port.

5. Initiate a test subscribe request from the terminal, using the client certificate.

 mosquitto_sub -d -k 10 -v                          \
 -i DSN000001                                       \
 -h liot.com -p 8883                                \
 --cafile ca.crt --cert client.crt --key client.key \
 -t test/topic

Congrats if you got the output look like

Client DSN000001 sending CONNECT
Client DSN000001 received CONNACK (0)
Client DSN000001 sending SUBSCRIBE (Mid: 1, Topic: test/topic, QoS: 0, Options: 0x00)
Client DSN000001 received SUBACK
Subscribed (mid: 1): 128
Client DSN000001 sending DISCONNECT
All subscription requests were denied.

Outputs from the docker console

mosquitto_1          | 1618914506: New client connected from 172.20.0.1:63288 as DSN000001 (p2, c1, k10, u'client.liot.com').
mosquitto_1          | 1618914506: Client DSN000001 disconnected.

Note 1: As the hostname of the broker is configured to be liot.com in the CN field, it’s not reachable unless a new entry is configured in the hosts file.

# ➜  cert sudo vim /private/etc/hosts
...
192.168.7.123   liot.com  # change to the IP address of the host machine
...

Note 2: Pub/Sub requests from a certificate-authenticated client will be denied due to restrictions introduced in Dynamic Security, so a fake user that stands for clients of this kind should be created and assigned client role, so that they are able to pub and sub. Note that the user name is from the CN (Common Name) field defined in the client certificate.

Screen Shot 2021-04-21 at 11.40.28 AM.png

Comparison Between MQTT and TLS Based Traffics

MQTT by default transports payload in plain text mode, whereas MQTT over TLS will encrypt and decrypt the payload in a secure way.

1. Publish test message thru MQTT

mosquitto_pub -r -d \
-u test -P 123456 \
-h 192.168.7.123 -p 1883 \
-t 'test/topic' -m "hello there"

Screen Shot 2021-04-21 at 6.20.57 PM.png

2. Publish test message thru MQTT over TLS

mosquitto_pub -d \
--cafile ca.crt --cert client.crt --key client.key \
-h server.liot.com -p 8883 \
-t 'test/topic' -m "hello there, this is secure message"

Screen Shot 2021-04-21 at 6.21.29 PM.png

Here are some references about MQTT over TLS

• Choosing an MQTT broker for your IoT project
• Mosquitto SSL Configuration -MQTT TLS Security
• Creating and Using Client Certificates with MQTT and Mosquitto
• Generating a TLS certificate for mosquitto
• Enable SSL/TLS for EMQ X MQTT broker
• <u>Mosquitto TLS Manual Page.</u>
• SSL and SSL Certificates Explained For Beginners
• How To: DER vs CRT vs CER vs PEM Certificates and How to Convert Them
• HiveMQ. MQTT & MQTT 5 Essentials. 2020, 04/27/2021.
• HiveMQ. X509 Client Certificate Authentication - MQTT Security Fundamentals. 05/25/2015, 04/27/2021.

Others

How to open multiple terminals in docker?

docker exec -it <container> bash

docker ps
docker exec 3479c5c781c7 cat /mosquitto/data/dynamic-security.json

References

1. Steve Cope. Using The Mosquitto_pub and Mosquitto_sub MQTT Client Tools- Examples. 02/21/2021, 04/16/2021.
2. Homieiot. An MQTT Convention for IoT/M2M.
3. OwnTracks. Topics
4. MQTT SmartHome. Smart home automation with MQTT as the central message bus.
5. Steve Cope. MQTT Last Will and Testament Examples. 02/19/2021, 04/16/2021.
6. Mqtt-smarthome.
7. MQTT Wiki Page. 04/19/2021.
8. OASIS. MQTT Version 5.0 Spec. 03/07/2019, 04/19/2021.