LLM360 Pre-Training Guide

The LLM360 Pre-Training Guide provides complete details and considerations for anyone leading a pre-training project. The Guide is divided into three categories: Project Planning, Initial Training Decision, Ensuring Success.Learning and procedural steps were gathered from the team while training Amber, Crystal, and [65B] models. All training artifacts are made available to be reviewed and built off of.

  • Goal and Budgets
  • Data Preparation
  • Model Architecture Choices
  • Hyperparameter study
  • Training Curriculum Planning
  • Preparing Runtime
  • The Training Job
  • Training Wrap Up
  • What is LLM360 open source
  • LLM360 Case Studies

We find these resources valuable:

The Foundation Model Development Cheatsheet

LMSYS Chatbot Arena: Live and Community-Driven LLM Evaluation

OLMo: Open Language Model