Little Language Model

Live experiment · by Grady Davis

A language model, built from scratch, learning to write in your browser.

The same kind of network behind modern AI assistants, just tiny: 62,000 parameters instead of billions. Every layer and every derivative is written by hand in plain JavaScript, and it’s training right now on your CPU cores. It starts out knowing nothing. Watch the text.

It’s writing

waiting 

    

      
0steps

      
—loss

      
—letters it’s torn between

      
0characters read

      
—characters / sec

      
—CPU cores training

    

  









  

    
Learning curve
Loss is how surprised it is by the real next character. Lower is better.

    
    
training texttext it has never seenpure guessing

  


  

    
How it sees letters
Each character’s learned vector, flattened to 2D. Letters that behave alike drift together.

    
    
vowelsconsonantscapitalspunctuationspace & newline

  






  
Talk to it
Give it a start. It continues one character at a time, rolling weighted dice from its own odds.

  

    
    
    
    
  

  


  

    

      
What it thinks comes next

      
Its probability for each possible next character after your prompt, updated as it trains.

      
    
    
    
    
    
      
    Where it’s looking
    
      
    Attention: for the next character, how much each head weighs every earlier character. Brighter means more.
    
      
    
    
    
  
    

    


    
  
    Time-lapse
    Same prompt, same dice, at each milestone. This is the whole story of training.
    
  
      

      


      
  
      Under the hood
      
  
      How it works
      
  
      About 1,017 lines of JavaScript and CSS, no libraries, no server, covered by 10 automated tests. AI does most of the typing; the engineering is mine.
      

  
        
    
      1. Characters inEach character becomes a token; up to 48 at a time.
        2. 
    
      2. EmbeddingsA learned 48-number vector per character, plus one per position.
        3. 
    
      3. Transformer block × 2Layer norm → 4-head causal self-attention → residual → layer norm → MLP (192 wide, GELU) → residual.
        4. 
    
      4. Final norm + outputScores for every character, using the embedding table again (tied weights).
        5. 
    
      5. SoftmaxScores become odds for the next character.
        6. 
  
      

  
      
    
      
      
      Backprop, by hand
      
      
      No autograd. The backward pass for layer norm, attention, GELU, the linear layers and the tied output head is derived on paper and written out. Every tensor’s gradient is checked against finite differences to better than one part in 100,000 before anything ships.
      
    
      
    
      
      
      Your cores as GPUs
      
      
      Training is synchronous data parallel, the way big models train across GPUs: each Web Worker gets the weights, samples its own batch, and sends back gradients. The page averages them, clips them, and takes an AdamW step with warm-up and cosine decay.
      
    
      
    
      
      
      Fast enough in plain JavaScript
      
      
      Weights are transposed so every matrix multiply is a dot product with its running sum in a register. GELU uses its one-exp sigmoid form. A separate worker writes the samples and draws what it’s looking at, so the page never stalls.
      
    
      
    
      
      
      Honest about size
      
      
      This is the architecture idea, not the scale. Real assistants have billions of parameters, read trillions of words and train for weeks on thousands of GPUs. This one reads a few hundred thousand characters for a few minutes. Shakespeare text is public domain; nothing you paste leaves the tab.