# Data Analysis

Scientific data analysis including data cleaning, exploratory data analysis (EDA)...

- Repository: https://github.com/beita6969/ScienceClaw (1024 stars)
- Category: Documents
- License: MIT
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add beita6969/ScienceClaw --skill data-analysis -g -a claude-code -y`
Codex: `npx skills add beita6969/ScienceClaw --skill data-analysis -g -a codex -y`
Cursor: `npx skills add beita6969/ScienceClaw --skill data-analysis -g -a cursor -y`
Gemini CLI: `npx skills add beita6969/ScienceClaw --skill data-analysis -g -a gemini-cli -y`

## Description

Scientific data analysis including data cleaning, exploratory data analysis (EDA), statistical testing, regression, and reporting. Uses Python with pandas, scipy, statsmodels, scikit-learn. Use when user asks to analyze data, clean a dataset, run statistics, do EDA, fit a model, or process CSV/Excel files. Triggers on "analyze this data", "clean my dataset", "run regression", "EDA", "descriptive statistics", "data processing", "correlation analysis".

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Data Analysis

Scientific data analysis with Python. All scripts use the venv at `/Users/zhangmingda/clawd/.venv`.

## Setup

```bash
source /Users/zhangmingda/clawd/.venv/bin/activate
```

## Workflow

### 1. Data Loading

```python
import pandas as pd
import numpy as np

# CSV
df = pd.read_csv('data.csv')
# Excel
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')
# JSON
df = pd.read_json('data.json')
# Clipboard (from user paste)
# Save user's data to a temp file first, then read

# Quick inspection
print(f"Shape: {df.shape}")
print(f"Columns: {list(df.columns)}")
print(df.dtypes)
print(df.describe())
print(f"Missing values:\n{df.isnull().sum()}")
```

### 2. Data Cleaning

```python
# Missing values
df.dropna(subset=['critical_column'])
df['col'].fillna(df['col'].median(), inplace=True)

# Duplicates
df.drop_duplicates(inplace=True)

# Outliers (IQR method)
Q1, Q3 = df['col'].quantile([0.25, 0.75])
IQR = Q3 - Q1
mask = (df['col'] >= Q1 - 1.5*IQR) & (df['col'] <= Q3 + 1.5*IQR)
df_clean = df[mask]

# Type conversion
df['date'] = pd.to_datetime(df['date'])
df['category'] = df['category'].astype('category')
```

### 3. Exploratory Data Analysis

```python
import matplotlib.pyplot as plt
import seaborn as sns

# Distribution
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
for i, col in enumerate(numeric_cols[:4]):
    ax = axes[i//2, i%2]
    sns.histplot(df[col], kde=True, ax=ax)
    ax.set_title(col)
plt.tight_layout()
plt.savefig('distributions.png', dpi=150)

# Correlation matrix
corr = df[numeric_cols].corr()
sns.heatmap(corr, annot=True, cmap='RdBu_r', center=0, fmt='.2f')
plt.savefig('correlation.png', dpi=150)

# Pairplot for key variables
sns.pairplot(df[key_cols], hue='group')
plt.savefig('pairplot.png', dpi=150)
```

### 4. Statistical Tests

Choose test based on:
- **Data type**: continuous vs categorical
- **Distribution**: normal vs non-normal (Shapiro-Wilk test)
- **Groups**: 2 vs 3+ groups
- **Pairing**: independent vs paired/repeated

The rest: https://github.com/beita6969/ScienceClaw/blob/HEAD/skills/data-analysis/SKILL.md