LLM360 Pre-Training Guide
The LLM360 Pre-Training Guide provides complete details and considerations for anyone leading a pre-training project. The Guide is divided into three categories: Project Planning, Initial Training Decision, Ensuring Success.Learning and procedural steps were gathered from the team while training Amber, Crystal, and [65B] models. All training artifacts are made available to be reviewed and built off of.
Content:
- Goal and Budgets
- Data Preparation
- Model Architecture Choices
- Hyperparameter study
- Training Curriculum Planning
- Preparing Runtime
- The Training Job
- Training Wrap Up
- What is LLM360 open source
- LLM360 Case Studies
