Research Scientist - Pre-training
AI
Pluralis Research is seeking a Research Scientist - Pre-training to build distributed training systems for Protocol Learning, scaling from an 8B model to frontier scale on heterogeneous consumer hardware over the internet. The role requires hands-on distributed training experience in PyTorch with FSDP, DeepSpeed, or Megatron, strong Python engineering, and a belief in decentralized, trustless AI.
What you'll do
- Implement and optimize model-parallel training for large models on heterogeneous GPUs under low-bandwidth, high-latency links.
- Implement techniques that reduce communication overhead while maintaining model convergence in challenging network environments.
- Make runs survive node churn with robust checkpointing, state synchronization, and recovery.
- Build monitoring that shows throughput, bottlenecks, and model quality across hundreds of devices.
What you need
Must-have
PyTorchFSDPDeepSpeedMegatronData parallelismTensor parallelismPipeline parallelismPython
Nice-to-have
Large language modelsNemotronQwenOLMoP2P networkingNAT traversalPost-trainingReinforcement Learning
Soft skills
Mission alignmentComfortable working across timezonesProfessional-level English proficiency
Research ScientistPre-trainingDistributed TrainingProtocol LearningDecentralized AIPyTorchLarge Language ModelsRemoteHybridFull-time