# Nlp Pretraining

Best practices for language model pretraining and fine-tuning.

- Repository: https://github.com/aiming-lab/AutoResearchClaw (14798 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-08-19
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add aiming-lab/AutoResearchClaw --skill nlp-pretraining -g -a claude-code -y`
Codex: `npx skills add aiming-lab/AutoResearchClaw --skill nlp-pretraining -g -a codex -y`
Cursor: `npx skills add aiming-lab/AutoResearchClaw --skill nlp-pretraining -g -a cursor -y`
Gemini CLI: `npx skills add aiming-lab/AutoResearchClaw --skill nlp-pretraining -g -a gemini-cli -y`

## Description

Best practices for language model pretraining and fine-tuning. Use when generating or reviewing NLP training code.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

## NLP Pretraining/Fine-tuning Best Practice
Fine-tuning recipe:
- Use pre-trained checkpoints (HuggingFace hub)
- AdamW optimizer, lr=2e-5 to 5e-5
- Linear warmup (6% of total steps) + linear decay
- Batch size: 16-32 (use gradient accumulation for larger effective batch)
- 3-5 epochs for classification, 1-2 for generation
- Weight decay: 0.01

Parameter-efficient methods:
- LoRA: r=8-64, alpha=16-128, apply to q/v projections
- Prefix tuning: 10-20 prefix tokens
- Adapters: bottleneck dimension 64-256

Evaluation:
- Classification: accuracy, F1 (macro for imbalanced)
- Generation: perplexity, BLEU/ROUGE, human evaluation
- Use multiple seeds and report mean +/- std