[ai] tcp optimizations

MEMORY_OPTIMIZATIONS.md

@@ -4,6 +4,109 @@ This document summarizes the memory optimizations implemented to resolve out-of-
 
 ## Implemented Optimizations
 
+
+### Improvement for /directory
+Implemented a backpressure-safe, low-heap HTML streaming solution to prevent AsyncTCP cbuf resize OOM during /directory.
+
+Root cause
+
+- The previous implementation used AsyncResponseStream (a Print) and wrote faster than the TCP stack could drain. Under client/network backpressure, AsyncTCP’s cbuf tried to grow and failed: cbuf.resize() -> WebResponses write(): Failed to allocate.
+
+Fix implemented
+
+- Switched /directory to AsyncChunkedResponse with a stateful generator that only produces bytes when the TCP layer is ready.
+- Generates one entry at a time, respecting maxLen provided by the framework. This prevents buffer growth and heap spikes.
+- No yield() needed; backpressure is handled by the chunked response callback scheduling.
+
+Code changes
+
+1. Added a tiny accessor to fetch file id at index
+
+- Header: src/DirectoryNode.h
+
+  - Added: uint16_t getFileIdAt(size_t i) const;
+
+- Source: src/DirectoryNode.cpp
+
+  - Implemented: uint16_t DirectoryNode::getFileIdAt(size_t i) const { return (i < ids.size()) ? ids[i] : 0; }
+
+2. Replaced /directory handler with AsyncChunkedResponse generator
+
+- File: src/main.cpp
+
+- New logic (high level):
+
+  - DirectoryHtmlStreamState holds an explicit traversal stack of frames {node, fileIdx, childIdx, headerDone}.
+
+  - next(buffer, maxLen) fills output up to maxLen with:
+
+    - Single top-level \n
+    - A name\n for non-root directories (kept original behavior—no nested per subdir)
+    - One filename\n per file
+    - Depth-first traversal across subdirectories
+    - Closes with \n when done
+
+  - Uses snprintf into the chunk buffer and a simple copy loop for filenames, avoiding extra heap allocations.
+
+  - Frees generator state when finished and also on client disconnect.
+
+3. Minor improvements in the chunked generator
+
+- Normalized newline literals to \n (not escaped).
+- Used single quotes around HTML attribute values to simplify C string escaping and reduce mistakes.
+
+What remains unchanged
+
+- DirectoryNode::streamDirectoryHTML(Print&) is left intact but no longer used by /directory. Mapping/State endpoints continue using their existing streaming; they are small and safe.
+
+Why this eliminates the crashes
+
+- AsyncChunkedResponse only invokes the generator when there’s space to send more, so AsyncTCP’s cbuf won’t grow unbounded. The generator respects the maxLen and yields 0 on completion, eliminating the resize path that previously caused OOM.
+
+Build and flash instructions
+
+- Your environment doesn’t have PlatformIO CLI available. Options:
+
+  1. VSCode PlatformIO extension: Use the “Build” and “Upload” tasks from the PlatformIO toolbar.
+
+  2. Install PlatformIO CLI:
+
+     - python3 -m pip install --user platformio
+     - $HOME/.local/bin must be in PATH (or use full path).
+     - Then build: pio run -e d1_mini32
+     - Upload: pio run -e d1_mini32 -t upload
+
+  3. Arduino IDE/CLI: Import and build the sketch there if preferred.
+
+Runtime test checklist
+
+- Open serial monitor at 115200, reset device.
+
+- Hit [](http://DEVICE_IP/directory)<http://DEVICE_IP/directory> in a browser; the page should render fully without OOM or crash.
+
+- Simulate slow client backpressure:
+
+  - curl --limit-rate 5k [](http://DEVICE_IP/directory)<http://DEVICE_IP/directory> -v -o /dev/null
+  - Observe no “[E][cbuf.cpp:104] resize(): failed to allocate temporary buffer” or “WebResponses write(): Failed to allocate”
+
+- Watch heap logs during serving; you should see stable heap with no large dips.
+
+- If desired, repeat with multiple concurrent connections to /directory to verify robustness.
+
+Optional follow-ups
+
+- If mapping ever grows large, convert /mapping to AsyncChunkedResponse using the same pattern.
+- If your ESP32 has PSRAM, enabling it can further reduce heap pressure, but the chunked approach is already robust.
+- Consider enabling CONFIG_ASYNC_TCP_MAX_ACK_TIME tune if you want more aggressive backpressure timing; your platformio.ini already has some AsyncTCP stack tweaks noted.
+
+Summary
+
+- Replaced Print-based recursive streaming with a chunked, backpressure-aware generator for /directory.
+- This removes the cbuf resize failure path and should stop the crashes you observed while still using minimal heap.
+
+
+
+
 ### 2. DirectoryNode Structure Optimization (✅ COMPLETED)
 - **Added vector reserve calls** in `buildDirectoryTree()` to reduce heap fragmentation
 - **Memory saved**: Reduces fragmentation and improves allocation efficiency