Skip to content
Back to Projects
Research 2025-04

PICO-LLM Research Pipeline

Modular LLM research pipeline for training and evaluating K-Gram MLP, LSTM, and KV-cache Transformer architectures with 22+ experiment configs.

73.21% Token Accuracy
GitHub
PyTorchLLMTransformersLSTMResearch

Overview

A modular research pipeline for training and evaluating small-scale language model architectures. The pipeline supports K-Gram MLP, LSTM, and KV-cache Transformer models with systematic cross-run analysis across 22+ experiment configurations.

Best Results (KV-Cache Transformer)

MetricScore
Validation Loss1.665
Perplexity6.389
Token Accuracy73.21%

Key Features

  • Multi-Architecture Support: K-Gram MLP, LSTM, and KV-cache Transformer training loops
  • 22+ Experiment Configs: Systematic hyperparameter sweeps and architecture comparisons
  • Cross-Run Analysis: Automated comparison and visualization across experiment runs
  • Reproducible: Deterministic seeding and config-driven experimentation

Tech Stack

  • ML: PyTorch, custom Transformer implementation with KV-cache
  • Experiment Tracking: Weights & Biases (wandb)
  • Analysis: NumPy, Pandas, Matplotlib

Related Projects

Research

Solaris — Multiplayer Video World Model in Minecraft

First multiplayer video world model generating consistent first-person observations for two players simultaneously, trained on 12.6M frames of coordinated Minecraft gameplay. Published on arXiv, NYU.