Research Engineer - Pre-training
AI
Pluralis Research is seeking a Research Engineer - Pre-training to build distributed training systems for Protocol Learning, training large models on consumer-grade devices over the internet. The role involves implementing model-parallel training, performance optimization, fault tolerance, and run instrumentation. Requires hands-on distributed training experience with PyTorch and strong Python engineering skills.
What you'll do
- Implement and optimize model-parallel training for large models on heterogeneous GPUs under low-bandwidth, high-latency links.
- Implement techniques that reduce communication overhead while maintaining model convergence in challenging network environments.
- Make runs survive node churn with robust checkpointing, state synchronization, and recovery.
- Build monitoring that shows throughput, bottlenecks, and model quality across hundreds of devices.
What you need
Must-have
PyTorchFSDPDeepSpeedMegatronData parallelismTensor parallelismPipeline parallelismPython
Nice-to-have
Large language modelsNemotronQwenOLMoP2P networkingNAT traversalPost-trainingReinforcement Learning
Soft skills
Mission alignmentComfortable working across timezonesProfessional-level English proficiency
Research EngineerPre-trainingDistributed TrainingProtocol LearningPyTorchMachine LearningRemoteFull-timeAI ResearchDecentralized AI