数据与研究

distributed-llm-pretraining-torchtitan

Name: distributed-llm-pretraining-torchtitan
Author: orchestra-research

orchestra-research/ai-research-skills

Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

970 次安装
1.4万星标
收录于 2026.02.03

安装命令

$ npx skills add https://github.com/orchestra-research/ai-research-skills --skill distributed-llm-pretraining-torchtitan

SKILL.md

TorchTitan - PyTorch Native Distributed LLM Pretraining

Quick start

TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.

Installation:

# From PyPI (stable)
pip install torchtitan

# From source (latest features, requires PyTorch nightly)
git clone https://github.com/pytorch/torchtitan
cd torchtitan
pip install -r requirements.txt

SKILL.md

TorchTitan - PyTorch Native Distributed LLM Pretraining

Quick start

相关 Skills