Storage Management

This Component provides support for using storage devices in a structured way by partitioning them into areas for specific uses. Partitions may can contain information such as:

Application (firmware) images
Filesystem(s)
Configuration/calibration/parameter data
Custom flash storage areas

A single partition table is located on the main flash device, Storage::spiFlash, and defines all partitions with a unique name and associated Storage::Partition::Type / Storage::Partition::SubType.

Hardware configuration

Each project has an associated Hardware configuration, specified by the HWCONFIG setting: this is a JSON file with a .hw extension.

For user convenience, the configuration file may contain comments however these are stripped before processing.

The build system locates the file by searching, in order:

{PROJECT_DIR} the root project directory
{SMING_HOME}/Arch/{SMING_ARCH}
{SMING_HOME}

Each architecture provides a standard configuration which defines such things as the partition table location and standard system partitions. Other configurations inherit from this by providing a base_config value.

You can list the available configs like this:

make hwconfig-list

This also shows the file path should you wish to view or edit it.

To select and view the resulting configuration, do this:

make hwconfig HWCONFIG=spiffs

or, to show the partition map:

make map HWCONFIG=spiffs

Note

You can set HWCONFIG in your project’s component.mk file, however as with other configuration variables it will be overridden by the cached value set on the command line.

For example, if you want to change from standard to standard-4m for your project, first add this line to your component.mk file:

HWCONFIG := standard-4m

Then either run make HWCONFIG=standard-4m or make config-clean.

Hardware configuration options

Commonly used settings can be stored in an option library for easier use. The library files are named options.json and located in the place as .hw files.

For example, we can do this:

make HWCONFIG=standard HWCONFIG_OPTS=4m,spiffs

This loads the ‘standard’ profile then merges the fragments found in the option library with the given names. This is how the standard-4m profile is constructed.

If using this approach, remember to updated your project’s component.mk with the desired settings, and verify the layout is correct using make map.

OTA updates

When planning OTA updates please check that the displayed partition map corresponds to your project. For example, the partition table requires a free sector so must not overlap other partitions.

Your OTA update process must include a step to write the partition table to the correct location. See Partition table migration.

It is not necessary to update the bootloader. See rBoot for further information.

Custom configurations

To customise the hardware configuration for a project, for example ‘my_project’:

Create a new configuration file in your project root, such as my_project.hw:
```
{
   "name": "My project config",
   "base_config": "spiffs",
   "options": ["vdd"]
}
```
You can use any available configuration as the base_config. Option fragments can be pulled in as shown. See Hardware configuration options.

If required, modify any inherited settings:

{
   "name": "My config",
   "base_config": "standard",
   "devices": {
      "spiFlash": {
         "speed": 80,
         "mode": "qio",
         "size": "2M"
      }
   },
   "partitions": {
      "rom0": {
         "address": "0x10000",
         "size": "0x80000"
      },
      "factory": {
         "size": 0
      }
   }
}

This will adjust flash parameters (previously via SPI_SPEED, SPI_MODE and SPI_SIZE), and the location/size of the primary application partition. The factory partition is also dropped by setting its size to 0. This is ignored if no such partition is defined.

Add any additional partitions:

{
   "name": "My config",
   "base_config": "standard-4m",
   "partitions": {
      "rom0": {
         "address": "0x10000",
         "size": "0x80000"
      },
      "spiffs1": {
            "address": "0x00280000",
            "size": "256K",
            "type": "data",
            "subtype": "spiffs",
            "filename": "$(FW_BASE)/spiffs1_rom.bin",
            "build": {
               "target": "spiffsgen",
               "files": "files/spiffs1"
            }
      }
   }
}

This adds a second SPIFFS partition, and instructs the build system to generate an image file for it using the files in the project’s files/spiffs1 directory.

Select the new configuration and re-build the project:
```
make HWCONFIG=my_project
```
You should also add this to your project’s component.mk file:
```
HWCONFIG := my_project
```
Program your device:
```
make flash
```
This will flash everything: bootloader, partition table and all defined partitions (those with a filename entry).

Note

The build system isn’t smart enough to track dependencies for partition build targets.

To rebuild these manually type:

make buildpart

These will be removed when make clean is run, but you can also clean them separately thus:

make part-clean

Partition maps

This is a concise view of your flash partitions. Display it like this:

make map

For the Basic Storage sample application, we get this:

Basic_Storage: Invoking 'map' for Esp8266 (debug) architecture
Partition map:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x00001fff          8K                      Boot Sector
spiFlash          0x00002000  0x00002fff          4K                      Partition Table
spiFlash          0x00003000  0x00003fff          4K  data      phy       phy_init          $(FLASH_INIT_DATA)
spiFlash          0x00004000  0x00007fff         16K  data      sysparam  sys_param
spiFlash          0x00008000  0x000fffff        992K  app       factory   rom0              $(RBOOT_ROM_0_BIN)
spiFlash          0x00100000  0x001effff        960K                      (unused)
spiFlash          0x001f0000  0x001f3fff         16K  user      0         user0             user0.bin
spiFlash          0x001f4000  0x001f7fff         16K  user      1         user1
spiFlash          0x001f8000  0x001fffff         32K                      (unused)
spiFlash          0x00200000  0x0027ffff        512K  data      spiffs    spiffs0           $(SPIFF_BIN_OUT)
spiFlash          0x00280000  0x002bffff        256K  data      spiffs    spiffs1           $(FW_BASE)/spiffs1_rom.bin
spiFlash          0x002c0000  0x002fffff        256K  data      spiffs    spiffs2           $(FW_BASE)/spiffs2_rom.bin
spiFlash          0x00300000  0x003fffff          1M                      (unused)

For comparison, here’s the output for Esp32:

Basic_Storage: Invoking 'map' for Esp32 (debug) architecture
Partition map:
Device            Start       End         Size        Type      SubType   Name              Filename
----------------  ----------  ----------  ----------  --------  --------  ----------------  ------------
spiFlash          0x00000000  0x00007fff         32K                      Boot Sector
spiFlash          0x00008000  0x00008fff          4K                      Partition Table
spiFlash          0x00009000  0x0000efff         24K  data      nvs       nvs
spiFlash          0x0000f000  0x0000ffff          4K  data      phy       phy_init
spiFlash          0x00010000  0x001fffff       1984K  app       factory   factory           $(TARGET_BIN)
spiFlash          0x001f0000  0x001f3fff         16K  user      0         user0             user0.bin
spiFlash          0x001f4000  0x001f7fff         16K  user      1         user1
spiFlash          0x001f8000  0x001fffff         32K                      (unused)
spiFlash          0x00200000  0x0027ffff        512K  data      spiffs    spiffs0           $(SPIFF_BIN_OUT)
spiFlash          0x00280000  0x002bffff        256K  data      spiffs    spiffs1           $(FW_BASE)/spiffs1_rom.bin
spiFlash          0x002c0000  0x002fffff        256K  data      spiffs    spiffs2           $(FW_BASE)/spiffs2_rom.bin
spiFlash          0x00300000  0x003fffff          1M                      (unused)

To compare this with the partition map programmed into a device, do this:

make readmap map

JSON validation

When the binary partition table is built or updated, the configuration is first validated against a schema Sming/Components/Storage/schema.json.

This complements the checks performed by the hwconfig tool.

You can run the validation manually like this:

make hwconfig-validate

See JSON Schema for details about JSON schemas.

Configuration

HWCONFIG

default: standard

Set this to the hardware configuration to use for your project.

Default configurations:

standard

Base profile with 1MB flash size which should work on all device variants. Located in the Sming/Arch/{SMING_ARCH} directory.

standard-4m

Overrides standard to set 4Mbyte flash size

spiffs: Adds a single SPIFFS partition. See SPIFFS IFS Library.

Other configurations may be available, depending on architecture. You can see these by running make hwconfig-list.

For example, to select spiffs add the following line to your project:

HWCONFIG := spiffs

You will also need to run make HWCONFIG=spiffs to change the cached value (or make config-clean to reset everything).

HWCONFIG_OPTS: Set this to adjust the hardware profile using option fragments. See Hardware configuration options.

ENABLE_STORAGE_SIZE64

Build with ENABLE_STORAGE_SIZE64=1 to enable support for storage devices of more than 4GB capacity.

Device and partition addresses and sizes use the storage_size_t type, which by default is uint32_t. Setting this value changes it to uint64_t.

When enabling this setting, care must be taken in code especially with printf style format strings such as in debug statements. The safest way to handle both cases is like this:

debug_i("Partition size: %llu", uint64_t(part.size()));

Binary partition table

Sming uses the same binary partition table structure as ESP-IDF, located immediately after the boot sector. However, it is organised slightly differently to allow partitions to be registered for multiple storage devices.

Entries are fixed 32-byte structures, Storage::esp_partition_info_t, organised as follows:

The first entry is always a storage type defining the main spiFlash device.
This is followed by regular partition entries sorted in ascending address order. There may be gaps between the partitions.
The partition table md5sum entry is inserted as normal
If any external devices are defined: - A SMING_EXTENSION entry, which the esp32 bootloader interprets as the end of the partition table. - The next entry is a storage type for the external device. - This is followed by regular partition entries as before. - A second md5sum entry is inserted for the entire partition table thus far
The end of the partition table is identified by an empty sector (i.e. all bytes 0xFF).

Partition API

This is a C++ interface. Some examples:

Storage::Partition part = Storage::findPartition("spiffs0"); // Find by name
if(part) {
  Serial << part << endl;
} else {
  Serial << "spiffs0 partition NOT Found" << endl;
}

// Enumerate all partitions
for(auto part: Storage::findPartition()) {
  Serial << part << endl;
}

// Enumerate all SPIFFS partitions
for(auto part: Storage::findPartition(Storage::Partition::SubType::Data::spiffs)) {
  Serial << part << endl;
}

A Storage::Partition object is just a wrapper and can be freely copied around. It defines methods which should be used to read/write/erase the partition contents.

Each partition has an associated Storage::Device. This is usually Storage::spiFlash for the main flash device.

Other devices must be registered via Storage::PartitionTable::registerStorageDevice().

You can query partition entries from a Storage object directly, for example:

#include <Storage/SpiFlash.h>

for(auto part: Storage::spiFlash->partitions()) {
  Serial << part << endl;
}

External Storage

If your design has additional fixed storage devices, such as SPI RAM, flash or EEPROM, you can take advantage of the partition API to manage them as follows:

Implement a class to manage the storage, inheriting from Storage::Device.
Create a custom hardware configuration for your project and add a devices entry describing your storage device, plus partition entries: the device field identifies which device these entries relate to.
Create an instance of your custom device and make a call to Storage::registerDevice() in your init() function (or elsewhere if more appropriate).

See Disk Storage for how devices such as SD flash cards are managed.

API

Core Functions

void Storage::initialize(): Called early in the startup phase.

const Device::List Storage::getDevices(): Get read-only reference to device list.

bool Storage::registerDevice(Device *device)

Register a storage device.

Return values:: bool – true on success, false if another device already registered with same name

bool Storage::unRegisterDevice(Device *device)

Unregister a storage device.

Use extreme care: behaviour is unpredictable if partitions are in use

Device *Storage::findDevice(const String &name): Find a registered device.

Partition Storage::findPartition(const String &name): Find the first partition matching the given name.

inline Iterator Storage::findPartition(Partition::Type type = Partition::Type::any, uint8_t subType = Partition::SubType::any): Find partitions of the given type.

SpiFlash *Storage::spiFlash: The main SPI flash device. Where possible, access via partition API. If lower-level access is required, use this. It should not be necessary to access low-level flash access routines directly.

Main classes

class Device : public LinkedObjectTemplate<Device>

Represents a storage device (e.g. flash memory)

Subclassed by Storage::Disk::BlockDevice, Storage::FileDevice, Storage::ProgMem, Storage::SpiFlash, Storage::StreamDevice, Storage::SysMem

Public Types

enum class Type : uint8_t

Storage type.

Values:

enumerator partitionType

enumerator XX

Public Functions

inline const PartitionTable &partitions() const: Provide read-only access to partition table.

inline PartitionTable &editablePartitions(): Provide full access to partition table.

inline bool loadPartitions(uint32_t tableOffset)

Load partition table entries @tableOffset Location of partition table to read.

Return values:: bool – true on success, false on failure

bool loadPartitions(Device &source, uint32_t tableOffset)

Load partition table entries from another table.

Parameters:: source – Device to load entries from @tableOffset Location of partition table to read
Return values:: bool – true on success, false on failure

virtual String getName() const = 0: Obtain unique device name.

inline virtual uint32_t getId() const

Obtain device ID.

Return values:: uint32_t – typically flash chip ID

virtual size_t getBlockSize() const = 0: Obtain smallest allocation unit for erase operations.

virtual storage_size_t getSize() const = 0

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

virtual Type getType() const = 0: Obtain device type.

virtual bool read(storage_size_t address, void *dst, size_t size) = 0

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) = 0

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) = 0

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline virtual uint16_t getSectorSize() const

Get sector size, the unit of allocation for block-access devices.

Override this method only if the device does not support standard 512-byte sector access. For example, ‘Advanced-Format’ drives use 4096-byte sectors.

inline virtual storage_size_t getSectorCount() const: Obtain total number of sectors on this device.

inline virtual bool sync()

Flush any pending writes to the physical media.

Devices with intermediate buffering should implement this method.

Return values:: bool – Return false if sync operation failed.

class SpiFlash : public Storage::Device 

Main flash storage device.

Public Functions

virtual String getName() const override: Obtain unique device name.

virtual size_t getBlockSize() const override: Obtain smallest allocation unit for erase operations.

virtual storage_size_t getSize() const override

Obtain addressable size of this device.

Return values:: storage_size_t – Must be at least as large as the value declared in the hardware configuration

inline virtual Type getType() const override: Obtain device type.

virtual uint32_t getId() const override

Obtain device ID.

Return values:: uint32_t – typically flash chip ID

virtual bool read(storage_size_t address, void *dst, size_t size) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

virtual bool write(storage_size_t address, const void *src, size_t size) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

virtual bool erase_range(storage_size_t address, storage_size_t size) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

class Partition

Represents a flash partition.

Confirm partition is of the expected type

bool verify(Type type, uint8_t subtype) const

Strong C++ type value.

Parameters:

type – Expected partition type
subtype – Expected partition sub-type

Return values:

bool – true if type is OK, false if not. Logs debug messages on failure.

inline bool verify(uint8_t type, uint8_t subtype) const: Weak ‘type’ value.

template<typename T> inline bool verify(T subType) const: Derive type from subtype, expressed as strong C++ enum.

Public Functions

bool read(storage_size_t offset, void *dst, size_t size)

Read data from the partition.

Parameters:

offset – Where to start reading, relative to start of partition
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

bool write(storage_size_t offset, const void *src, size_t size)

Write data to the partition.

Note

Flash region must be erased first

Parameters:

offset – Where to start writing, relative to start of partition
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

bool erase_range(storage_size_t offset, storage_size_t size)

Erase part of the partition.

Note

Both offset and size must be aligned to flash sector size (4Kbytes)

Parameters:

offset – Where to start erasing, relative to start of partition
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

inline Partition::Type type() const: Obtain partition type.

inline uint8_t subType() const: Obtain partition sub-type.

inline FullType fullType() const: Obtain both type and subtype.

inline storage_size_t address() const

Obtain partition starting address.

Return values:: storage_size_t – Device address

inline storage_size_t lastAddress() const

Obtain address of last byte in this this partition.

Return values:: storage_size_t – Device address

inline storage_size_t size() const

Obtain partition size.

Return values:: storage_size_t – Size in bytes

inline String name() const: Get partition name.

inline Flags flags() const: Get partition flags.

inline bool isEncrypted() const: Check state of partition encrypted flag.

inline bool isReadOnly() const: Check state of partition readOnly flag.

bool getDeviceAddress(storage_size_t &address, storage_size_t size) const

Get corresponding storage device address for a given partition offset.

Parameters:

address – IN: Zero-based offset within partition, OUT: Device address
size – Size of data to be accessed

Return values:

bool – true on success, false on failure Fails if the given offset/size combination is out of range, or the partition is undefined.

String getDeviceName() const

Get name of storage device for this partition.

Return values:: String –

inline bool contains(storage_size_t addr) const: Determine if given address contained within this partition.

size_t getBlockSize() const: Obtain smallest allocation unit for erase operations.

uint16_t getSectorSize() const

Get sector size for block-addressable devices.

Streaming

enum class Storage::Mode

Values:

enumerator ReadOnly

enumerator Write: Write but do not erase, region should be pre-erased.

enumerator BlockErase: Erase blocks as required before writing.

class PartitionStream : public ReadWriteStream 

Stream operating directory on a Storage partition.

To support write operations, the target region must be erased first.

Public Functions

inline PartitionStream(Partition partition, storage_size_t offset, size_t size, bool blockErase)

Access part of a partition using a stream.

If blockErase is false then region must be pre-erased before writing.

Deprecated:: Use mode parameter instead of blockErase

Parameters:

partition –
offset – Limit access to this starting offset
size – Limit access to this number of bytes from starting offset
blockErase – Set to true to erase blocks before writing

inline PartitionStream(Partition partition, bool blockErase)

Access entire partition using stream.

If blockErase is false then partition must be pre-erased before writing.

Deprecated:: Use mode parameter instead of blockErase

Parameters:

partition –
blockErase – Set to true to erase blocks before writing

inline PartitionStream(Partition partition, storage_size_t offset, size_t size, Mode mode = Mode::ReadOnly)

Access part of a partition using a stream.

Note

When writing in Mode::BlockErase, block erasure is only performed at the start of each block. Therefore if offset is not a block boundary then the corresponding block will not be erased first.

Parameters:

partition –
offset – Limit access to this starting offset
size – Limit access to this number of bytes from starting offset
mode –

inline PartitionStream(Partition partition, Mode mode = Mode::ReadOnly)

Access entire partition using stream.

Parameters:

partition –
mode – If blockErase is false then partition must be pre-erased before writing.

inline virtual int available() override

Return the total length of the stream.

Return values:: int – -1 is returned when the size cannot be determined

virtual uint16_t readMemoryBlock(char *data, int bufSize) override

Read a block of memory.

Todo:: Should IDataSourceStream::readMemoryBlock return same data type as its bufSize param?

Parameters:

data – Pointer to the data to be read
bufSize – Quantity of chars to read

Return values:

uint16_t – Quantity of chars read

virtual int seekFrom(int offset, SeekOrigin origin) override

Change position in stream.

Note

This method is implemented by streams which support random seeking, such as files and memory streams.

Parameters:

offset –
origin –

Return values:

New – position, < 0 on error

virtual size_t write(const uint8_t *buffer, size_t size) override

Write chars to stream.

Note

Although this is defined in the Print class, ReadWriteStream uses this as the core output method so descendants are required to implement it

Parameters:

buffer – Pointer to buffer to write to the stream
size – Quantity of chars to write

Return values:

size_t – Quantity of chars written to stream

inline virtual bool isFinished() override

Check if all data has been read.

Return values:: bool – True on success.

class StreamDevice : public Storage::Device 

Read-only partition on a stream object.

Note

Writes not possible as streams always append data, cannot do random writes

Public Functions

inline StreamDevice(IDataSourceStream *stream, size_t size)

Create a Device object using data from a stream with restricted size.

Parameters:

stream – Backing data source for this device
size – Size of device in bytes

inline StreamDevice(IDataSourceStream *stream)

Create a Device object using data from a stream with all available data.

Parameters:: stream – Backing data source for this device

inline virtual Type getType() const override: Obtain device type.

inline virtual bool read(storage_size_t address, void *buffer, size_t len) override

Read data from the storage device.

Parameters:

address – Where to start reading
dst – Buffer to store data
size – Size of data to be read, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool write(storage_size_t address, const void *data, size_t len) override

Write data to the storage device.

Parameters:

address – Where to start writing
src – Data to write
size – Size of data to be written, in bytes.

Return values:

bool – true on success, false on error

inline virtual bool erase_range(storage_size_t address, storage_size_t len) override

Erase a region of storage in preparation for writing.

Parameters:

address – Where to start erasing
size – Size of region to erase, in bytes

Return values:

bool – true on success, false on error

Debugging

namespace Debug

Functions

void listPartitions(Print &out)

void listPartitions(Print &out, const Device &device)

void listDevices(Print &out, bool fullPartitionInfo = true)

ProgMem Storage::progMem

class ProgMem : public Storage::Device : Storage device to access PROGMEM using flash API.

SysMem Storage::sysMem

class SysMem : public Storage::Device : Storage device to access system memory, e.g. RAM.

References

Source Code

Used by

Sming (main) Component
Installable File System Component
Disk Storage Library

Environment Variables

ENABLE_STORAGE_SIZE64
HWCONFIG
HWCONFIG_OPTS
PARTITION_TABLE_OFFSET

SoC support

esp32
esp32c2
esp32c3
esp32s2
esp32s3
esp8266
host
rp2040
rp2350