# [2] Data Engineering: Why and How to Convert LeRobot (Parquet/MP4) to HDF5 In my previous post, I explained why the **JEPA** architecture is such a promising lead for robotics. But between Yann LeCun’s theory and the first `$loss.backward()$`, there is a massive wall: **the data.** For my POC on the **Koch arm (SO-ARM101)**, I’m using the **LeRobot** ecosystem. It’s a goldmine of data, but its default storage format isn't built for the intensive training cycles required by World Models. Here is why I had to build a technical "bridge" to the **HDF5** format. 🔗 [Bridge repository](https://github.com/tanguy-pauwels/lerobot-dataset-to-HDF5) ## 1\. The Format War: Storage vs. Training

Feature	LeRobot Format (Parquet + MP4)	HDF5 Format (LeWM Optimized)
Ideal Use	Lightweight distribution and archiving.	Intensive Training (GPU Datasets).
Pros	Highly compressed, easy to visualize, HF standard.	Ultra-fast Random Access to frames and actions.
Cons	Heavy CPU video decoding for every batch, potential desync.	Large file size, less "standard" for sharing.

**The Problem**: Training a World Model (JEPA) requires sampling random time windows across thousands of episodes. Attempting a `seek` in an MP4 file for every single frame in a batch of 256 is performance suicide. HDF5 allows us to treat the dataset as one massive tensor living on the disk. * * * ## 2\. The "Little Mac" Challenge: Optimize or Die Fetching datasets via the `lerobot` library is a breeze. The real challenge began during conversion. My Mac, with its limited resources, suffered several **Kernel Panics** before I got it right. To succeed without saturating the RAM, I had to implement a **"Lean & Mean"** pipeline: ### The "Low-Memory" Conversion Strategy * **Linear Pipeline**: I abandoned aggressive parallelism. We process one episode at a time, one camera at a time. It’s slower, but it’s **predictable**. * **Micro-Batching (64 frames)**: Instead of loading an entire episode, we decode and write to the HDF5 in small chunks. * **Video Streaming**: Using iterators (PyAV/OpenCV) to ensure a full video never materializes in the RAM. * **LZF Compression**: The perfect compromise. It's ultra-fast, CPU-light, and significantly reduces the final file weight. ### Safety and Reliability (Diagnostic Mode) Because a 2-hour conversion crashing at 99% is unacceptable, I integrated several safeguards: * **Pre-validation**: We check the integrity of metadata (episodes, frames, flags) *before* touching the video files. * **Watchdog & Heartbeat**: If the script shows no progress for 120 seconds, it "fails-fast" rather than wasting heat. * **RAM Estimation**: The script calculates and displays the estimated memory footprint of the batch before starting. ## 3\. Alignment for LeWM (World Model) The **LeWM** model is demanding. Conversion isn't enough; we need adaptation: * **Resize 224x224**: The standard for modern vision backbones. Resizing is done on-the-fly during conversion. * **Key Normalization**: LeRobot names columns one way, while LeWM expects another (e.g., `pixels`, `action`, `state`, `done`). My bridge handles the translation automatically. * * * ## 4\. The Golden Rule: No "Dirty" Data In low-cost robotics, datasets are often imperfect—truncated episodes or missing done flags are common. > My Policy: dirty\_episode\_policy=fail On small datasets, the model is extremely sensitive to overfitting. Introducing inconsistent trajectories or ill-defined episode endings condemns the model to learn nonsense. I would rather have a converter that refuses to work than one that produces toxic data. * * * ### Tanguy's Advice > If you attempt this: **watch your HDF5 chunks.** A mismatch between your chunk size and your conversion micro-batches can turn your hard drive into a massive bottleneck. Yes, I learned this the hard way—remember, I only have a little Mac! **Next Step**: We launch the training on an **RTX 4090** runpod instance and see if our latent space survives physical reality.