An MIT-licensed project combines ESP32-S3 firmware and open-source software to let vision models control an iPhone through its screen and USB input.
ESP32 iPhone Agent is an open-source project that allows AI models to interact with a real iPhone using an inexpensive ESP32-S3 development board. Published under the MIT licence, its GitHub repository includes microcontroller firmware, a Python-based agent server, a browser interface and setup scripts. The project provides an alternative to conventional iPhone automation tools that require a Mac, Xcode, an Apple developer account or an application installed on the phone.
The system separates screen capture, physical input and AI decision-making into three parts. It uses AirPlay screen mirroring to display the iPhone screen on a computer running UxPlay, while an ESP32-S3 connected to the phone’s USB port acts as a standard mouse and keyboard through USB Human Interface Device (HID) functionality. A Python server processes screenshots and sends commands to the board over Wi-Fi. This allows the system to interact with apps through their visible interfaces rather than depending on application-specific APIs.
The repository provides firmware written using the Arduino ESP32 framework, alongside a Python server built with FastAPI. Developers can configure an OpenAI-compatible vision model through environment variables, using either a cloud API or a locally hosted model. A planning component breaks a user request into steps, while an execution component examines screenshots and chooses actions such as tapping, swiping, typing or drawing. After each action, the software checks whether the screen has changed and can retry missed interactions.
The project also includes a browser-based control panel with live screen mirroring, manual input controls, calibration tools and an action history showing screenshots alongside the corresponding interactions. Developers can flash the ESP32-S3 firmware using the supplied scripts and install the server dependencies on macOS or Linux. The hardware is based on a Seeed Studio XIAO ESP32S3 board, which costs approximately US$7 according to the project documentation. iPhones with USB-C connect directly, while Lightning models require a compatible USB camera adapter.
By publishing the complete implementation under MIT, the developer allows others to inspect, modify, redistribute and integrate the software under the licence’s conditions. The repository welcomes issues and pull requests, making it possible to contribute improvements to the firmware, automation logic and interface. However, the project still requires a computer to run the server and capture the mirrored screen, and its current limitations include portrait-only operation and ASCII-only keyboard input. It is intended for authorised automation of devices and accounts, with users able to monitor actions and stop the agent when necessary.
For more information, click here.
