How OXFileManager Works

What this page is

The API reference tells you what each method does. This page explains how the class is built and why — the moving parts, the data flow, and the design choices behind them. Read it when you need to extend the class, add a new storage backend, or debug behavior on real hardware.

Wiring diagram (external SD card)

The external storage is a microSD adapter on the SPI bus. Wire it to the ESP32-S3 Super Mini as below — these are the validated default pins (beginExternal()). Internal storage (LittleFS) needs no wiring; it lives in the chip's flash.

The SD backend mounts in DEDICATED_SPI mode at 1 MHz with a fallback ladder; keep the SPI wires short for reliable multi-sector writes.

The core problem

OXFileManager has to serve two storage devices whose libraries share no common API:

• Internal flash uses Arduino LittleFS — an fs::FS object returning fs::File (an Arduino Stream). Paths, modes, and capacity calls all follow the Arduino filesystem conventions.
• SD card uses SdFat — a global SdFs volume returning FsFile, with its own open() flags (O_WRONLY | O_CREAT), its own sync/close semantics, and capacity expressed in clusters.

We also want the class to be unit-testable on a laptop, where neither LittleFS nor SdFat exists. So the design needs a seam that hides both real filesystems behind a common shape — one the production code and a test fake can both satisfy.

The architecture: one interface, three backends

Everything hangs off a single pure-virtual interface, IStorageBackend (in include/OXFileManager.h). OXFileManager talks only to this interface; it never includes a filesystem header.

OXFileManager  (src/OXFileManager.cpp — portable, no FS headers)
   │  holds two raw pointers, selected by the Storage enum
   │      IStorageBackend* internal_
   │      IStorageBackend* external_
   ▼
IStorageBackend  (pure virtual — the seam)
   ├─ LittleFsBackend   src/backends/LittleFsBackend.cpp   → LittleFS
   ├─ SdFatBackend      src/backends/SdFatBackend.cpp      → SdFs/FsFile
   └─ RamBackend        test/test_oxfilemanager/RamBackend.h → std::map (tests)

This is the same dependency-injection idea used by OXLog's pluggable sink. Because the manager depends on an abstraction rather than a concrete filesystem, the real backends compile only for the device and an in-memory fake drives the host tests — with the exact same manager code under both.

Why an enum instead of two objects

Rather than expose separate internalFs and sdCard objects, every method takes a Storage argument and routes internally:

IStorageBackend *OXFileManager::backendFor(Storage where) const {
  return where == Storage::Internal ? internal_ : external_;
}

Every public call begins with the same guard — resolve the backend, and bail out safely if it is missing or unmounted:

IStorageBackend *b = backendFor(where);
if (b == nullptr || !b->isMounted()) {
  OXLog::error("writeFile: %s storage not mounted", storageName(where));
  return false;
}

The result is that the choice of filesystem is explicit at every call site (writeFile(Storage::External, …)), and adding a third storage later is a matter of extending the enum and the router.

The flash quota

Internal flash is tiny and shared with the firmware image, so the manager enforces a byte budget that writes can never exceed. beginInternal(maxBytes) records the limit, clamped to the real partition size (0 means "use the whole partition"):

size_t partition = g_internalBackend.totalBytes();
internalQuota_ = (maxBytes == 0 || maxBytes > partition) ? partition : maxBytes;

Before each internal write, the manager computes how much room is left and rejects the write if it would not fit. The subtlety is a truncating overwrite: replacing an existing file first frees that file's bytes, so they count as reclaimable headroom.

size_t existing = b->exists(path) ? b->fileSize(path) : 0;
size_t reclaimable = internalQuotaFree() + existing;
if (len > reclaimable) {
  OXLog::warn("writeFile: internal quota exceeded (need %u, free %u)", …);
  return false;
}

internalQuotaFree() is just limit − usedBytes() (floored at zero). The external SD card has no such check — it is bounded only by the card's own capacity. This is also why totalBytes(Storage::Internal) and freeBytes(Storage::Internal) report against the quota, not the raw partition: the numbers reflect what the firmware is actually allowed to use.

Recursive mkdir lives in the core

Neither backend is asked to create parent directories. Instead the portable core walks the path once, creating each prefix that does not yet exist, and calls the backend's single-level mkdir for each missing segment:

// "/a/b/c" → create "/a", then "/a/b", then "/a/b/c"
for (size_t i = 1; i <= n; ++i) {
  if (buf[i] == '/' || buf[i] == '\0') {
    char saved = buf[i];
    buf[i] = '\0';                      // temporarily terminate at the separator
    if (!b->exists(buf) && !b->mkdir(buf)) return false;
    buf[i] = saved;                      // restore and continue
  }
}

Keeping this logic in one place means both backends stay simple — they only ever create a single directory — and the recursive behavior is identical and tested once.

Streaming writes: opaque handles + move-only ownership

Audio capture and other large outputs cannot be buffered whole in RAM, so openWrite() returns an OXFileStream you feed in chunks. The challenge: LittleFS and SdFat return different file types (fs::File vs FsFile), but OXFileStream must be one concrete type the manager can return by value.

The solution is an opaque handle. Each backend allocates its native file on the heap and hands back a void*; the stream methods pass that pointer straight back to the backend, which casts it to the right type. OXFileStream never knows what is behind the pointer.

// SdFatBackend: open returns a heap FsFile as an opaque handle
void *SdFatBackend::openStream(const char *path, bool append) {
  FsFile *f = new FsFile(g_sd.open(path, append ? (O_WRONLY|O_CREAT|O_APPEND)
                                          : (O_WRONLY|O_CREAT|O_TRUNC)));
  if (!*f) { delete f; return nullptr; }
  return f;
}

Because the stream owns an open file (a resource that must be closed exactly once), OXFileStream is move-only: the copy constructor and copy assignment are deleted, and the move operations transfer the handle and null out the source. The destructor closes whatever it still holds, so a stream cleans up even if you forget to call close():

OXFileStream::~OXFileStream() { close(); }

void OXFileStream::close() {
  if (backend_ && handle_) backend_->streamClose(handle_);  // frees the heap file
  backend_ = nullptr;
  handle_ = nullptr;                                       // idempotent: safe to call twice
}

Quota also applies to streams

A streamed internal write could otherwise sneak past the quota one chunk at a time. To prevent that, the stream carries a quotaRemaining_ counter, seeded by the manager at open time — the current free space for internal flash, or SIZE_MAX (effectively unlimited) for the SD card:

size_t quota = (where == Storage::Internal) ? internalQuotaFree() : SIZE_MAX;
return OXFileStream(b, handle, quota);

Each write() trims its length to what remains and decrements the counter, so an internal stream stops accepting bytes exactly at the quota boundary (and returns a short count to tell the caller).

One-shot writeFile/appendFile vs. a stream — and why it matters on real SD

The one-shot calls and the streaming API look interchangeable for "write these bytes," but they have very different costs. Every writeFile() / appendFile() is self-contained: it opens the file, writes, sync()s the bytes to media, and closes — a full round trip to the filesystem each call.

// SdFatBackend::writeFile — note the open + sync + close on every call
FsFile f = g_sd.open(path, append ? (O_WRONLY|O_CREAT|O_APPEND)
                              : (O_WRONLY|O_CREAT|O_TRUNC));
size_t written = f.write(data, len);
bool ok = written == len && !f.getWriteError() && f.sync();
f.close();

That is exactly what you want for a config blob or a status file — one call, durably on disk, no handle left open. It is exactly what you do not want inside a loop. Call appendFile() once per chunk of a large transfer and every chunk pays for a fresh open, a metadata-updating sync(), and a close. Over SPI — the bus the validated SD adapter uses, clocked low for reliability — those FAT round trips dominate, and on real hardware a write partway through a large file simply fails. (This bit a streaming download: appending each ~4 KB network chunk meant ~20 open/sync/close cycles for an 80 KB file, and the card gave out around 35 KB in.)

The stream is the cure: openWrite() opens the file once, each write() just appends to the already-open handle, and you flush() on your own schedule — periodically (say every 64 KB) rather than after every chunk — with a single close() at the end. One open, a handful of syncs, one close, regardless of how many chunks arrive.

Device vs. host: how the same code builds both ways

The portable core (src/OXFileManager.cpp) declares but does not define beginInternal()/beginExternal(). Those live in src/backends/OXFileManagerDevice.cpp, which instantiates the real LittleFsBackend and SdFatBackend. Everything that touches a real filesystem sits under src/backends/.

The native test environment then excludes that whole directory in platformio.ini:

[env:native]
build_src_filter = +<*> -<main.cpp> -<backends/>

So on the host, only the portable core compiles; the tests inject a RamBackend (a std::map-backed IStorageBackend) through setBackend() and verify routing, quota math, recursive mkdir, listing, and streaming — all with no hardware. On the device, the full set compiles and beginInternal()/beginExternal() wire in the real backends. This mirrors how the OXLog tests use a fake Arduino Print sink.

The SD mount sequence

SdFatBackend reuses the configuration proven in the INMP441-ESP32-S3 recorder so the SD card mounts reliably on the same hardware. Three pieces matter:

• SdFs + FsFile — the SdFat types that support FAT16/FAT32 and exFAT, so >32 GB SDXC cards work. Selected by the build flag SDFAT_FILE_TYPE=3; a #error guards against building without it.
• USE_SD_CRC=1 — the validated adapter otherwise reports a CRC error during card init; the firmware sends valid SPI command CRCs to match it.
• Explicit SPI bring-up — idle CS high, start SPI on the ESP32-S3 pins, then begin SdFat in SHARED_SPI | USER_SPI_BEGIN mode at the configured clock:

pinMode(csPin_, OUTPUT);
digitalWrite(csPin_, HIGH);
SPI.begin(sckPin_, misoPin_, mosiPin_, csPin_);

SdSpiConfig config(csPin_, SHARED_SPI | USER_SPI_BEGIN, spiHz_, &SPI);
mounted_ = g_sd.begin(config);

Capacity is computed from the volume geometry — clusterCount() × bytesPerCluster() for the total, and the same times freeClusterCount() for what is used. All audio/I2S code from the reference project was deliberately left out; this backend is pure storage.

Note: mount() makes a single attempt at the configured clock. The reference project also had a fallback ladder (dedicated SPI, stepping down to 100 kHz) that is not ported here yet — see the API reference page for the current parameters.

Request lifecycle, end to end

A typical call — fm.writeFile(Storage::Internal, "/cfg", data, len) — flows like this:

1. backendFor(Storage::Internal) resolves internal_.
2. The mount guard rejects the call (logging via OXLog) if the backend is missing or unmounted.
3. The quota check computes reclaimable space and rejects an oversized write before any bytes are touched.
4. The call delegates to LittleFsBackend::writeFile(), which opens the LittleFS file, writes, and closes.
5. A bool success propagates back to the caller.

For streaming, steps 1–2 are the same, then openStream() returns an opaque handle wrapped in an OXFileStream carrying the remaining quota; subsequent write()/flush()/close() calls forward to the backend through that handle.

Extending it: adding a backend

1. Implement IStorageBackend for the new medium (e.g. a USB mass-storage or network backend), following LittleFsBackend as a template.
2. Add a value to the Storage enum and extend backendFor() to route to it.
3. Wire its begin…() in a device-only translation unit under src/backends/ so the host build stays clean.
4. Add native tests by pointing a RamBackend (or a new fake) at the new Storage slot.

Because the manager only knows the interface, none of the routing, quota, or mkdir logic needs to change.

File map