OPDS browsing: Expat buffer allocation failure on xteink devices

[gershalfred]

Summary

Two xteink/CrossInk devices reproduce this when browsing a Calibre-Web OPDS catalog:

couldn't allocate memory for buffer

The same server works from Kobo and the Calibre-Web website. The OPDS XML validates with curl/XML tooling, so this looks like a CrossInk OPDS parser heap-pressure issue rather than a malformed server feed.

I coded a small candidate fix and pushed it here:

• Branch: https://github.com/gershalfred/CrossInk/tree/fix/opds-catalog-memory-buffer
• Commit: gershalfred@1156394f

I attempted to open this as a PR, but GitHub returned:

gershalfred does not have the correct permissions to execute CreatePullRequest

So I’m posting the full diagnosis here for maintainers to pull/cherry-pick or recreate.

User-visible symptom

While browsing an OPDS catalog on xteink/CrossInk:

• root/server access succeeds
• browsing some feeds may work briefly
• selecting Authors / Books / an author can fail with:

couldn't allocate memory for buffer

This was verified on two xteink devices.

Why this appears firmware-side

The exact message maps to the OPDS parser path in:

lib/OpdsParser/OpdsParser.cpp

The failure path is:

void* const buf = XML_GetBuffer(parser, chunkSize);
if (!buf) {
  errorOccured = true;
  errorReason = OpdsParserError::BUFFER_MEMORY;
  LOG_DBG("OPDS", "Couldn't allocate memory for buffer");
  destroyXmlParser(parser);
  parser = nullptr;
  return length;
}

So the message means Expat could not allocate its parse buffer on-device.

Why Calibre-Web is probably not the root cause

The same Calibre-Web server:

• works from Kobo
• works from the Calibre-Web website
• returns valid OPDS XML via curl/XML validation
• returns normal sized feeds from server-side tests

A temporary server-side compatibility shim with tiny feeds reduces the issue, which further points to device-side memory pressure during OPDS parsing.

Suspected regression class

This may be related to the broader OPDS/networking changes where HttpDownloader moved to esp_http_client and buffering was tuned for download reliability/speed.

Those changes are useful for actual EPUB downloads, but catalog browsing has a different memory shape:

• Wi-Fi/TLS / esp_http_client buffers are live
• the fetch body buffer is live
• Expat asks for a parse buffer
• parsed OPDS entries and their std::string fields are retained

On ESP32-C3/no-PSRAM hardware, this can leave insufficient contiguous heap for XML_GetBuffer(...) even when the XML is valid.

Candidate fix coded

The candidate fix separates catalog fetches from file downloads:

1. Keep book/file downloads at the current larger buffer:

DEFAULT_DOWNLOAD_BUFFER_SIZE = 2048

2. Add a smaller default for catalog/text fetches:

DEFAULT_FETCH_BUFFER_SIZE = 512

3. Make HttpDownloader::fetchUrl(...) use the smaller fetch buffer.

4. Reduce Expat parse chunk size:

chunkSize = 1024 -> 256

That lowers transient contiguous heap requirements while browsing catalogs, without intentionally slowing actual EPUB downloads.

Patch

diff --git a/lib/OpdsParser/OpdsParser.cpp b/lib/OpdsParser/OpdsParser.cpp
index 2a935947..e47c0200 100644
--- a/lib/OpdsParser/OpdsParser.cpp
+++ b/lib/OpdsParser/OpdsParser.cpp
@@ -34,7 +34,9 @@ size_t OpdsParser::write(const uint8_t* xmlData, const size_t length) {
 
   const char* currentPos = reinterpret_cast<const char*>(xmlData);
   size_t remaining = length;
-  constexpr size_t chunkSize = 1024;
+  // Keep Expat's transient parse buffer small; OPDS parsing happens while the
+  // HTTP/TLS stack and entry strings are also consuming fragmented heap.
+  constexpr size_t chunkSize = 256;
 
   while (remaining > 0) {
     const size_t toRead = remaining < chunkSize ? remaining : chunkSize;
diff --git a/src/network/HttpDownloader.cpp b/src/network/HttpDownloader.cpp
index ed9ccdf7..a9e9c389 100644
--- a/src/network/HttpDownloader.cpp
+++ b/src/network/HttpDownloader.cpp
@@ -25,6 +25,9 @@ constexpr int HTTP_TX_BUF = 1024;
 constexpr int HTTP_TIMEOUT_MS = 60000;
 constexpr int HTTP_READ_POLL_TIMEOUT_MS = 5000;
 constexpr uint32_t DOWNLOAD_IDLE_TIMEOUT_MS = 30000;
+// Catalog fetches stream into parsers while WiFi/TLS buffers are live. Keep
+// their body buffer smaller than file downloads to preserve contiguous heap.
+constexpr size_t DEFAULT_FETCH_BUFFER_SIZE = 512;
 constexpr size_t DEFAULT_DOWNLOAD_BUFFER_SIZE = 2048;
 constexpr uint8_t MAX_REDIRECTS = 5;
 
@@ -415,7 +418,7 @@ bool HttpDownloader::fetchUrl(const std::string& url, const DataCallback& onData
 
   Sink sink;
   sink.write = onData;
-  return runGet(url, username, password, sink, DEFAULT_DOWNLOAD_BUFFER_SIZE) == OK;
+  return runGet(url, username, password, sink, DEFAULT_FETCH_BUFFER_SIZE) == OK;
 }

Local verification

This passed:

git diff --check

I attempted:

pio run -e default

but my local checkout fails before compiling because the SDK/library symlink target is missing:

PackageException: Can not create a symbolic link for `freeink-sdk/libs/hardware/BatteryMonitor`, not a directory

So this needs CI or a maintainer machine with the SDK checkout available for compile validation.

Hardware verification request

On a device that reproduces the issue:

1. Configure a Calibre-Web OPDS server.
2. Browse root → Authors → letter → author.
3. Browse root → Books.
4. Confirm the device no longer shows:

couldn't allocate memory for buffer

5. Confirm selecting/downloading an EPUB still works and download speed is not materially regressed.

Why this should be safe

This does not remove the faster file download path. It only makes catalog/text fetches and XML parsing less demanding on contiguous heap while browsing.

[gershalfred]

Additional live evidence from the same user/server after opening this issue:

Slow OPDS book download evidence

The same xteink/CrossInk device that had catalog parser failures is also downloading a small OPDS EPUB extremely slowly.

User-reported live case:

• Book: Dungeon Crawler Carl - Matt Dinniman.epub
• Calibre book id: 404
• File size on server: 1,112,569 bytes (~1.1 MB)
• xteink result: still downloading after 10+ minutes, but progressing slowly

Server-side benchmarks from the same Calibre-Web endpoint:

https://gersh.tplinkdns.com/opds/download/404/EPUB/
HTTP=200 bytes=1112569 speed=1898154 B/s time=0.586s type=application/epub+zip

Same path exposed over LAN/plain HTTP for testing:

http://192.168.0.103:8095/opds/download/404/EPUB/
HTTP=200 bytes=1112569 speed=9091695 B/s time=0.122s type=application/epub+zip

The server also showed an active connection from the user's home IP with data queued in the server send buffer, which suggests the server was ready to send and the device/client was reading very slowly.

Relevant server headers for the download route:

HTTP/2 206
accept-ranges: bytes
content-range: bytes 0-0/1112569
content-type: application/epub+zip
content-length: 1

So the download issue is not file size or Calibre-Web throughput. It appears to be the device-side HTTP/TLS/read loop under CrossInk.

This reinforces the earlier diagnosis: OPDS catalog browsing and OPDS downloads should have separate memory/transport tuning. Catalog fetches need smaller transient buffers to preserve heap for Expat; downloads need the most stable read-loop behavior for ESP32-C3 Wi-Fi/TLS.

Temporary workaround being tested: a LAN-only HTTP OPDS endpoint to avoid TLS overhead on-device:

http://192.168.0.103:8095/xteink/opds

If that materially improves download speed, it would strongly implicate the HTTPS/TLS + esp_http_client_read(...) path rather than the OPDS server.

[alpire]

btw, I ran into the same issue with a Grimmory OPDS catalog, and created a separate solution downloading the feed to the SD card, in case that's helpful: 6a45458

[gershalfred]

Updated candidate fix: spool OPDS feed to SD before parsing

After reviewing the suggested fix from Alex Rebert, I replaced the earlier buffer-only candidate on my fork with the stronger phase-separation approach:

• Branch: https://github.com/gershalfred/CrossInk/tree/fix/opds-catalog-memory-buffer
• Commit: gershalfred@5948ffe
• Reference implementation: 6a45458

The updated implementation:

1. Downloads the OPDS feed to /.crosspoint/opds-feed.tmp.
2. Closes the HTTP/TLS connection.
3. Opens the temporary file from SD.
4. Constructs/parses the Expat feed after the TLS buffers are gone.
5. Removes the temporary file on success, read failure, or allocation failure.
6. Supports Back cancellation during the feed download.

This directly addresses the suspected heap collision between the live HTTPS/TLS session and Expat's parser/string allocations. It is more robust than the earlier 512-byte/2048-byte buffer adjustment, which I have replaced rather than presenting as the final fix.

The source diff passes git diff --check. I attempted a PlatformIO default build, but this machine's environment stopped before source compilation on an existing dependency-link error:

PackageException: Can not create a symbolic link for
freeink-sdk/libs/hardware/BatteryMonitor`, not a directory

No PR is being requested from this fork at this point; this is posted as a candidate for maintainer review.

[Drudoo]

Having the same issue using BookOrbit with the endpoint /api/v1/opds.