# Data Formats

Working with diverse data formats: binary, text, structured, and custom

- Repository: https://github.com/vstorm-co/pydantic-deepagents (1111 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add vstorm-co/pydantic-deepagents --skill data-formats -g -a claude-code -y`
Codex: `npx skills add vstorm-co/pydantic-deepagents --skill data-formats -g -a codex -y`
Cursor: `npx skills add vstorm-co/pydantic-deepagents --skill data-formats -g -a cursor -y`
Gemini CLI: `npx skills add vstorm-co/pydantic-deepagents --skill data-formats -g -a gemini-cli -y`

## Description

Working with diverse data formats: binary, text, structured, and custom

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data Formats

How to work with diverse and unknown data formats.

## Format Detection

Always inspect before parsing:

```bash
file <filename>                    # MIME type detection
xxd <filename> | head -5           # hex dump (first bytes)
head -3 <filename>                 # text preview
python3 -c "
with open('<filename>', 'rb') as f:
    h = f.read(16)
    print(h, h.hex())
"
```

## Common Formats

### Binary
- **Magic bytes**: Most binary formats start with a signature (ELF: `\x7fELF`, PNG: `\x89PNG`)
- **Endianness**: Check if little-endian or big-endian (`struct.unpack('<I', ...)` vs `'>I'`)
- **Alignment**: Fields are often aligned to 4 or 8 bytes
- **Offsets**: Binary headers often contain offsets to other sections

### Structured text
- **CSV/TSV**: Check delimiter (comma, tab, pipe), quoting, header row
- **JSON**: `python3 -c "import json; json.load(open('f'))"`
- **YAML**: Check indentation, anchors/aliases
- **TOML**: `python3 -c "import tomllib; ..."`
- **XML**: Check encoding declaration, namespaces

### Checkpoints / Model files
- **PyTorch**: `.pt`, `.pth` → `torch.load(f, map_location='cpu')`
- **TensorFlow**: `.ckpt` → index + data files, use `tf.train.load_checkpoint()`
- **NumPy**: `.npy`, `.npz` → `numpy.load()`
- **HuggingFace**: `config.json` + `model.safetensors`
- **ONNX**: `onnx.load()`

### Database files
- **SQLite**: `file` says "SQLite 3.x database" → `sqlite3 <file> ".tables"`
- **WAL files**: SQLite write-ahead log, recover with `sqlite3` PRAGMA
- **CSV dumps**: Often need schema inference

## Parsing Strategies

### Unknown binary format
1. Hex dump first 256 bytes: `xxd file | head -16`
2. Look for magic bytes, version numbers, string tables
3. Check file size, does it suggest a pattern? (e.g., N * record_size)
4. Look for documentation of the format online
5. Write a minimal parser, test on known values

The rest: https://github.com/vstorm-co/pydantic-deepagents/blob/HEAD/apps/cli/skills/data-formats/SKILL.md