Member of Technical Staff - Research Scientist
Job Description
Job Description\n Tech stack -Django, AWS, React, TSX, Python, PyTorch, TensorFlow, Flask, Git, CI/CD\n\n\n Why candidates should join\n\n\n Engineers get early access to all of the newest LLMs and will be working closely with researchers and engineers at the AI labs (OpenAI, Anthropic, Cohere, etc.).\n\n\n Raised $5M Seed. Likely going to be closing a large Series A this month (DO NOT DISCLOSE)\n\n\n Other sell points\n\n
- \n
- Stanford NLP research origins, 300+ citations and hundreds more with their newest benchmarks
- Perks lunch/dinner provided, relocation support, health/dental, 401K, unlimited PTO
\n
\n
\n\n Work experience:\n\n
- \n
- Research experience in AI/ML, particularly in benchmarking, evaluation methodologies, or language models
- Experience working in teams. This includes working in development sprints, knowledge of best practices in working with Git, reviewing pull requests.
- Strong Python programming skills with ability to build clean, maintainable code
\n
\n
\n
\n\n Experience with deep learning frameworks (PyTorch, TensorFlow) and diffusion models, language modeling, and other relevant AI/ML techniques.\n\n\n
\n 0 - 3 years of experience in applied AI roles, with emphasis on benchmarking, evaluation methodologies, or language models. Experience building new benchmarks and evaluation methodologies is highly valued.\n\n\n Previous experience at AI startups, research labs like Anthropic or DeepMind, or founding/early employee at your own company is highly valued.\n\n\n True research experience with published papers (especially in NLP or benchmarking) is valued but not a hard requirement. We prioritize applied work over purely academic publications.\n\n\n Interest and familiarity with LLM infrastructure. Experience with Django or other Python-based HTTP servers (e.g., Flask) is a plus.\n\n\n Strong communication skills. You can provide input to others and equally receive/integrate feedback.\n\n\n
\n Willing to work in person in SF. We will support your relocation or transportation as needed.\n\n\n
\n Experience building new benchmarks and evaluation methodologies\n\n\n NLP research experience with papers published in reputable journals.\n\n\n Education\n\n
- \n
- Academic background in Computer Science, Machine Learning, or related field (Masters or PhD preferred). Candidates with a bachelor's degree and 1-3 years of experience could also be considered.
\n
\n\n Ideally from top tech/AI programs\n\n\n About this role\n\n\n In this role, you'll be at the forefront of developing benchmarks and evaluation methodologies for large language models. You'll help shape how the most advanced AI systems are tested and validated before being deployed in enterprise environments.\n\n\n
\n\n\n Key Responsibilities\n\n
- \n
- Evaluate new AI models as they're released (such as DeepSeek, Gemini, etc.)
- Create new benchmarks from scratch, including hiring labelers, constructing datasets, and writing white papers
- Improve underlying methods for auto-evaluation of generated text
- Work closely with engineering to implement and scale evaluation methodologies
- Collaborate with top AI labs and enterprise customers to understand evaluation needs
\n
\n
\n
\n
\n
