Porting MicroPython to pyBox

The standard ESP32 MicroPython port gets built for a generic ESP32 board. Compared to a generic board, pyBox has more to offer, most of which is not supported out of the box.

Generic ESP32 pyBox
2GB Flash Memory 16GB Flash Memory
16 MHz CPU freq. 24 MHz CPU freq.
16×2 Display
SDCard Drive
Battery Fuel Gage
2 Touch Buttons
Blue LED
One Neopixel

Some of these additional features like the blue LED (GPIO 13), are easily accessible. Others, like the display need support libraries that would ideally be part of MicroPython instead of being loaded from the filesystem at runtime.

'frozen' into the firmware
When MicroPython loads Python code from the filesystem, it first has to parse the file into a temporary in-memory representation and then generate bytecode for execution, both of which are stored in RAM. This can lead to significant amounts of memory being used. Alternatively, MicroPython’s cross compiler can generate a .mpy file (bytecode) for a given Python module, which can be “frozen” into the firmware. This bytecode will then be executed from ROM, significantly saving memory and reducing heap fragmentation.

 

Porting MicroPython to pyBox

To port MicroPython to pyBox requires the following steps:

  1. Downloading and building the Espressif IoT Development Framework, which contains
    • idf.py, the ESP-IDF CLI build tool:
      ~/mp_pybox_port/esp-idf/tools/idf.py
    • esptool.py, to communicate with the ROM bootloader in Espressif chips
      ~/mp_pybox_port/esp-idf/components/esptool_py/esptool/esptool.py
  2. Downloading Micropython and building the cross compiler, which can compile python code into python byte code (.mpy files).
    • ~/mp_pybox_port/micropython/mpy-cross/build/mpy-cross
  3. Building the Python Standard Library and MicroPython-specific modules
    • ~/mp_pybox_port/micropython/lib/micropython-lib
    • ~/mp_pybox_port/micropython/lib/berkeley-db-1.xx
  4. Building the firmware for the pyBox board, which contains:
    • ~/mp_pybox_port/micropython/ports/esp32/build-PYBOX/firmware.bin
      • ~/mp_pybox_port/micropython/ports/esp32/build-PYBOX/bootloader/bootloader.bin
      • ~/mp_pybox_port/micropython/ports/esp32/build-PYBOX/partition_table/partition-table.bin
      • ~/mp_pybox_port/micropython/ports/esp32/build-PYBOX/micropython.bin

1. Espressif IoT Development Framework

Following the instructions here will setup the software development environment for ESP32 chips. Please keep in mind that building MicroPython will require a Python version 3.10 or lower. I.e., the build process is not yet compatible with Python 3.11. For the same reason, esp-idf version of 4.4 or lower will be used.

brew install [email protected] cmake ninja dfu-util

mkdir mp_pybox_port
cd mp_pybox_port

git clone -b release/v4.4 --recursive https://github.com/espressif/esp-idf.git
cd esp-idf
./install.sh esp32

Here is what you should see happening …

Detecting the Python interpreter
Checking "python" ...
/Users/wolf/mp_pybox_port/esp-idf/tools/detect_python.sh: line 16: python: command not found
Checking "python3" ...
Python 3.10.8
"python3" has been detected
Installing ESP-IDF tools
Current system platform: macos

...

All done! You can now run:
  . ./export.sh

So it’s time to run

source export.sh

The next step involves cloning the MicroPython sources and building the cross compiler. I have submitted a pull request and hopefully you won’t need to clone my forked repository much longer and instead use the official MicroPython repo. For now however, https://github.com/wolfpaulus/micropython will have to do.

2. MicroPython Cross Compiler

cd ..
git clone https://github.com/wolfpaulus/micropython.git
cd micropython
make -C mpy-cross

Here is what you should see happening …

Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
mkdir -p build/genhdr
GEN build/genhdr/mpversion.h

....

CC main.c
CC gccollect.c
CC ../shared/runtime/gchelper_generic.c
LINK build/mpy-cross
__TEXT	__DATA	__OBJC	others	dec	hex
327680  16384   0       4295000304      4295344368      10005c0f0

3. Building the Python Standard Library

This step will build the Python Standard Library, MicroPython-specific modules, and Berkeley DB

cd ports/esp32
make submodules

Here is what you should see happening …

/Library/Developer/CommandLineTools/usr/bin/make -f ../../py/mkrules.mk GIT_SUBMODULES="lib/berkeley-db-1.xx lib/micropython-lib" submodules
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
Updating submodules: lib/berkeley-db-1.xx lib/micropython-lib
Submodule 'lib/berkeley-db-1.xx' (https://github.com/pfalcon/berkeley-db-1.xx) registered for path '../../lib/berkeley-db-1.xx'
Submodule 'lib/micropython-lib' (https://github.com/micropython/micropython-lib.git) registered for path '../../lib/micropython-lib'
Cloning into '/Users/wolf/mp_pybox_port/micropython/lib/berkeley-db-1.xx'...
Cloning into '/Users/wolf/mp_pybox_port/micropython/lib/micropython-lib'...
Submodule path '../../lib/berkeley-db-1.xx': checked out '35aaec4418ad78628a3b935885dd189d41ce779b'
Submodule path '../../lib/micropython-lib': checked out '038b4ac6572d0d5b4c18148dc7d7fdc026369fa4'

4. Building the pyBox firmware

All parts are now in place and the pyBox port can finally be build.

make BOARD=PYBOX

Here is what you should see happening …

idf.py -D MICROPY_BOARD=PYBOX -B build-PYBOX  build || (echo -e "See \033[1;31mhttps://github.com/micropython/micropython/wiki/Build-Troubleshooting\033[0m"; false)
Executing action: all (aliases: build)
Running cmake in directory /Users/wolf/mp_pybox_port/micropython/ports/esp32/build-PYBOX

...

Successfully created esp32 image.
Generated /Users/wolf/mp_pybox_port/micropython/ports/esp32/build-PYBOX/micropython.bin
[1399/1399] cd /Users/wolf/mp_pybox_po...orts/esp32/build-PYBOX/micropython.bin
micropython.bin binary size 0x16ed70 bytes. Smallest app partition is 0x1f0000 bytes. 0x81290 bytes (26%) free.

Project build complete. To flash, run this command:
/Users/wolf/.espressif/python_env/idf4.4_py3.10_env/bin/python ../../../esp-idf/components/esptool_py/esptool/esptool.py -p (PORT) -b 460800 --before default_reset --after no_reset --chip esp32  write_flash --flash_mode dio --flash_size detect --flash_freq 80m 0x1000 build-PYBOX/bootloader/bootloader.bin 0x8000 build-PYBOX/partition_table/partition-table.bin 0x10000 build-PYBOX/micropython.bin
or run 'idf.py -p (PORT) flash'
bootloader  @0x001000    23296  (    5376 remaining)
partitions  @0x008000     3072  (    1024 remaining)
application @0x010000  1502912  (  528704 remaining)
total                  1568448

Flashing the pyBox firmware

The firmware image can now be found here: ~/mp_pybox_port/micropython/ports/esp32/build-PYBOX/firmware.img.
Before flashing it onto the device, I’d always recommend erasing what’s currently there.

make BOARD=PYBOX PORT=/dev/tty.wchusbserial141440 erase

Here is what you should see happening …

...
Chip is ESP32-D0WD-V3 (revision 3)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Erasing flash (this may take a while)...
Chip erase completed successfully in 49.9s
Staying in bootloader.
Done

Now we can flash the custom pyBox firmware, just like we did with the generic ESP32 firmware:

make BOARD=PYBOX PORT=/dev/tty.wchusbserial141440 deploy

And one more time, here is what you should see happening …

Connecting...................
Chip is ESP32-D0WD-V3 (revision 3)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00001000 to 0x00006fff...
Flash will be erased from 0x00010000 to 0x0017efff...
Flash will be erased from 0x00008000 to 0x00008fff...
Compressed 23296 bytes to 14916...
Writing at 0x00001000... (100 %)
Wrote 23296 bytes (14916 compressed) at 0x00001000 in 0.7 seconds (effective 283.6 kbit/s)...
Hash of data verified.
Compressed 1502912 bytes to 1020833...
Writing at 0x00010000... (1 %)
Writing at 0x00018ff4... (3 %)
...
Writing at 0x0017c90e... (100 %)
Wrote 1502912 bytes (1020833 compressed) at 0x00010000 in 23.4 seconds (effective 514.5 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 117...
Writing at 0x00008000... (100 %)
Wrote 3072 bytes (117 compressed) at 0x00008000 in 0.1 seconds (effective 464.1 kbit/s)...
Hash of data verified.

Leaving...
Staying in bootloader.
Done

Getting to the Python prompt on pyBox

screen /dev/tty.wchusbserial141440 115200
MicroPython 939e99957 on 2022-12-25; pyBox with ESP32
>>> import pybox
>>> dir()
['uos', '__name__', 'gc', 'bdev', 'pybox']
>>> dir(pybox)
['__class__', '__name__', 'const', '__dict__', '__file__', 'OrderedDict', 'Pin', 'RTC', 'SDCard', 'STA_IF', 'Signal', 'WLAN', 'connect', 'freq', 'hostname', 'mount', 'sleep', 'NeoPixel', 'is_connected', 'Button', 'Neo', 'turbo', 'mount_sd', 'set_system_time', 'rgb_color', 'neo', 'palette', 'blue_led', 'btn_up', 'btn_dn', 'ip_addr']
>>> pybox.blue_led.on()
>>> 

Leave a Reply