Hierarchical Taxonomy Clustering

Create a unified multi-level taxonomy from hierarchical category paths by clustering similar paths and automatically generating meaningful category names.

Problem

Given category paths from multiple sources (e.g., "electronics -> computers -> laptops"), create a unified taxonomy that groups similar paths across sources, generates meaningful category names, and produces a clean N-level hierarchy (typically 5 levels). The unified category taxonomy could be used to do analysis or metric tracking on products from different platform.

Methodology

1. Hierarchical Weighting: Convert paths to embeddings with exponentially decaying weights (Level i gets weight 0.6^(i-1)) to signify the importance of category granularity
2. Recursive Clustering: Hierarchically cluster at each level (10-20 clusters at L1, 3-20 at L2-L5) using cosine distance
3. Intelligent Naming: Generate category names via weighted word frequency + lemmatization + bundle word logic
4. Quality Control: Exclude all ancestor words (parent, grandparent, etc.), avoid ancestor path duplicates, clean special characters

Output

DataFrame with added columns:

• unified_level_1: Top-level category (e.g., "electronic | device")
• unified_level_2: Second-level category (e.g., "computer | laptop")
• unified_level_3 through unified_level_N: Deeper levels

Category names use | separator, max 5 words, covering 70%+ of records in each cluster.

Installation

pip install pandas numpy scipy sentence-transformers nltk tqdm
python -c "import nltk; nltk.download('wordnet'); nltk.download('omw-1.4')"

4-Step Pipeline