LLM from scratch

Resources

• Build a Large Language Model
• Writing an LLM from scratch, part 28
• nanochat

TODO:

• chat cli, evaluate each epoch

• better arch (read nanochat)

• count tokens

• download more data (code, full fineweb)

• better train progress bar

• Notes

• TrainTestIterator

  • total length
  • deterministic shuffle
  • prepare in parallel
  • refactor new() into builder
  • small texts (<|bos|>?)

• Training

  • multi-device training
  • model parameters in file