Forward Deployed Engineer - Language Models
AI
Artificial Analysis, the leading independent AI benchmarking company, is hiring a Forward Deployed Engineer for Language Models. The role involves operating and maintaining the Python-based benchmarking pipeline, debugging issues, and serving as the primary technical contact for AI lab customers. Requires 3+ years in a client-facing technical role, strong Python proficiency, and experience with AI/ML model APIs.
What you'll do
- Operate and maintain Python-based language model benchmarking pipeline end-to-end: onboard new models, configure evaluations, execute benchmark runs, and validate results
- Debug issues across the stack, from API endpoint timeouts and errors to unexpected benchmark outputs, and resolve them quickly
- Serve as the primary technical contact for AI lab customers: communicate benchmarking results clearly, explain methodology, field technical questions, and troubleshoot integration issues via Slack and video conferencing
What you need
Must-have
PythonAI/ML model APIsDebuggingSlackVideo conferencing
Nice-to-have
AI evaluationBenchmarkingTesting methodologiesLLM inference infrastructureTokenizationLatency measurementThroughput metricsB2B SaaSDeveloper tools
Soft skills
CommunicationResponsivenessReliabilityOwnershipAttention to detailCalm under pressure
Forward Deployed EngineerLanguage ModelsAI benchmarkingPythonClient-facingAI/ML APIsFull-timeOn-siteMelbourneTechnology