Tensorflow ESP32
Introducing ESP32: The Wi-Fi MCU
We are glad to announce TensorFlow Lite Micro support for the ESP32 chipset.The ESP32 is a Wi-Fi/BT/BLE enabled MCU (micro-controller) that is widely used by hobbyists and makers to build cool and interesting projects that sense or modify real world data/object, and also commonly deployed in smart home appliances like light bulbs, switches, refrigerators, and air conditioners to provide connectivity.
ESP32 is already being used in a number of smart-home/connected-device projects with a variety of sensors and actuators connected to the microcontroller to sense the environment and act accordingly. With TensorFlow Lite for Microcontrollers executing on ESP32, this opens up scenarios for all kinds of use-cases that are triggered by local inference. ESP32 has 2 CPU cores and a bunch of optimizations, making it easier to run heavy TF Micro workfloads. The Wi-Fi backhaul helps to raise remote events and trigger actions based on the inferences made.
Person Detection or a Door-Bell Camera?
As an example, we have modified the person_detection example that you all might be familiar with to make it a smart door-bell camera. We use the ESP-EYE developer kit for this demonstration. Note that this example uses person detection (it detects when a face is in front of the camera), and not person identification (identifying who the person is).The ESP-EYE dev-kit includes the ESP32 Wi-Fi/BT MCU coupled with a 2MP camera.
In Action
In our example, we will use this camera to observe and send out an email notification if we detect a person in the vicinity.
Building it for yourself
- Order the ESP-EYE: You can get the ESP-EYE Development Kit from your favourite distributor, or from here. You will need a USB to micro-USB cable for connecting this to your Windows/Linux/macOS host.
- Clone the repository: https://github.com/espressif/tensorflow/
- Setup your development host: Setup your development host with toolchains and utilities required to cross-build for ESP32. Follow the instructions of the ESP-IDF get started guide to set up the toolchain and the ESP-IDF itself.
- Generate the example: The example project can be generated with the following command:
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=esp generate_doorbell_camera_esp_project
- Build the example:
a. Go to the example project directorycd tensorflow/lite/micro/tools/make/gen/esp_xtensa-esp32/prj/doorbell_camera/esp-idf
b. Clone the esp32-camera component with following command:$ git clone https://github.com/espressif/esp32-camera components/esp32-camera
c. Configure the camera and the email address:idf.py menuconfig
d. Enter the Camera Pins configuration and SMTP Configuration menus to select the camera details, and also the email details.
e. Build the example:idf.py build
- Flash and Run the program: Use the following command to flash and run the program:
idf.py --port /dev/ttyUSB0 flash monitor
- Now, whenever a person’s face is detected, the program will send out an email to the configured email address.
What Next?
Now that you have tried the door bell camera example, you may try the other applications that are part of the TF Micro repository: hello_world and micro_speech.ESP32 is pretty powerful for a microcontroller. Clocked at 240MHz, with just a single core it can do the detection well under 1 second (roughly ~700ms; additional optimizations are on the way to reduce this even further). This leaves the second core free for other tasks from your application.
The TinyML book is an excellent resource for a thorough understanding of TensorFlow Lite for Microcontrollers.
Comments