Senior Research Engineer, RL Distributed Systems
AI research company organized as a public benefit corporation, building large language models and AI safety research
| Employer | AI research company organized as a public benefit corporation, building large language models and AI safety research (not named) |
|---|---|
| Location | San Francisco, CA (Bay Area) · Hybrid |
| Type & level | Full-time · Senior |
| Pay | USD 500k–850k base Source: pay range stated in the source posting |
| Visa path | O-1A fit (senior candidates) |
| Role family | Foundation Model Research Scientist |
About the role
This team keeps the distributed systems behind frontier reinforcement learning runs working. Training, sampling and environment execution run at the same time across a large accelerator fleet, and the quality of that system decides how much compute turns into learning. The role goes after whichever layer currently limits throughput or reliability.
What you'll do
- Design, build and operate the distributed systems that run reinforcement learning across training, sampling and environment execution.
- Find and remove current bottlenecks in scheduling, data movement, storage, networking and coordination.
- Build failure detection, isolation and recovery so long-running jobs keep progressing without manual intervention.
- Create resource management and autoscaling so compute follows the shifting needs of a run.
- Build observability and automated remediation that explain why a run slowed, stalled or behaved unexpectedly.
- Work with researchers and performance engineers so systems changes preserve training correctness and avoid subtle nondeterminism.
Requirements
- Strong software engineering in Python and at least one systems language such as Rust, C++ or Go.
- Experience designing, building and operating large-scale distributed systems in production.
- Solid grounding in consistency, coordination, consensus, failure modes and recovery.
- Ability to reason quantitatively about throughput, latency and cost across compute, memory, storage and network.
- Experience debugging complex multi-host failures that cannot be reproduced locally, plus clear design and incident writing.
- Bachelor's degree or equivalent combination of education, training and experience.
Nice to have
- Experience running ML training or inference infrastructure at scale.
- Experience building schedulers, autoscalers or resource management systems.
- Kubernetes and sandboxed or virtualized code execution at scale.
- High-performance networking, RDMA or collective communication libraries.
- Async Python frameworks such as Trio or asyncio, and familiarity with RL or LLM training workloads.
Pay & benefits
USD 500k–850k base. The posting lists a base salary range and mentions competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, and flexible working hours.
Pay source: pay range stated in the source posting. Market data, not an offer from O1DMatch.
Visa path
Engineers in this kind of role often support an O-1A case with evidence such as authored technical papers, open-source or systems contributions with wide adoption, patents, conference service or peer review, and high compensation relative to the field. The posting states no citizenship restriction and says the employer offers visa sponsorship for some roles and candidates, though not for every one. An employer like this or a U.S. agent can file an O-1A petition; approval depends on the individual's evidence.
How to get matched
- Take the free eligibility checkAn AI review of your background against the eight O-1 criteria, at check.o1dmatch.com.
- Create your profile with code WORKWITHUSIt unlocks the Starter plan at no cost. Tell us your field and the work you do today.
- Upload your evidenceEach document is sorted under the criterion it supports, and your criteria map shows what is still missing.
- Get matchedWe match strong profiles with U.S. employers hiring for roles like this one. Gigs you confirm with verified companies count toward your itinerary, and at a score of 75 we connect you with an immigration attorney.
This listing describes a real U.S. posting so you can see what employers ask for. O1DMatch does not represent that employer and does not hold this opening; applying matches you with employers hiring for this kind of role. No visa outcome is promised.