A GPT-2 that learned to speak conversational Hindi
This project fine-tunes a Hindi GPT-2 checkpoint on everyday, spoken-register Hindi so it completes sentences the way people actually talk, not the way a newspaper is written. It was built for Stanford’s CS224N.
Base model
surajp/gpt2-hindi — a GPT-2 architecture (about 88M parameters) pretrained from scratch on Hindi text, loaded via Hugging Face Transformers.
Corpus
Hindi lines pulled from the OPUS OpenSubtitles en–hi bitext, keeping only the Hindi side. Lines are cleaned (length filtered, bracketed stage directions and leading dashes stripped) and deduplicated to build a conversational, spoken-register corpus rather than formal prose.
Tokenizing & chunking
The same tokenizer as the base model is reused (retraining it would break the pretrained embeddings' meaning). Lines are joined with the tokenizer's EOS token and cut into fixed 128-token blocks, the standard packing strategy for causal-LM fine-tuning.
Fine-tuning
Hugging Face Trainer, 3 epochs, batch size 16, learning rate 5e-5, 100 warmup steps, mixed precision when a GPU is available. Eval loss is tracked each epoch and converted to perplexity to catch overfitting early.
Whole-word decoding
A single forward pass gives next-token probabilities, but GPT-2 tokens are subwords. The predictor walks forward token-by-token per candidate until the next token would start a new word (a leading space or punctuation), so predictions surface as real words, not fragments.
Sentence completion
Generation stops after a target number of sentence-ending characters (., ?, !, and the Devanagari danda ।) rather than a fixed token count, so short prompts don't get padded out and longer ones don't cut off mid-thought.
What this live demo actually calls
The fine-tuned checkpoint produced by the notebook is saved locally to the training session and was never pushed to the Hugging Face Hub, so a deployed website can’t call it directly. This playground calls the public base model, surajp/gpt2-hindi, through Hugging Face’s hosted Inference API instead — same decoding logic from the notebook, base weights. Point HF_MODEL_ID at your own published fine-tuned repo to serve the real thing. See the docs for setup.
Source material
The full training notebook and a walkthrough recording are in the project repository.