397,582 Jobs

Job Description

Job Description

We are looking for a senior software engineer to own and evolve the pipeline infrastructure that powers our production workflows. This role is focused on pipeline scheduling, maintenance, support, scaling, reliability, and operational quality. The work goes beyond writing one-off scripts or adding tasks to an existing queue: the goal is to build pipeline systems that can keep running as workload, product needs, and team complexity grow.

You will work across services, data movement, workflow orchestration, failure recovery, monitoring, developer experience, and production support. We care about strong engineering judgment, practical technical sense, clear communication, and the ability to turn recurring operational pain into maintainable systems.

What You Will Own

• Design, build, and evolve pipeline scheduling and orchestration systems for production workflows.

• Maintain and support critical pipelines, including monitoring, incident response, failure recovery, backfills, and operational tooling.

• Improve pipeline scalability, throughput, reliability, and cost efficiency as workload and product requirements grow.

• Define clear interfaces, ownership boundaries, retry semantics, idempotency, data contracts, and observability standards for pipeline execution.

• Collaborate with product, operations, data, and engineering partners to turn ambiguous workflow needs into robust delivery plans.

• Identify recurring manual work or fragile processes and replace them with simple, well-designed automation.

What We Look For

• Strong backend engineering fundamentals and experience owning production systems with real operational requirements.

• Practical experience with pipeline orchestration, schedulers, queues, async workers, data movement, or workflow engines.

• Good judgment around reliability, retries, idempotency, concurrency, state management, observability, and failure recovery.

• Ability to debug production issues across application logic, infrastructure, data dependencies, and external services.

• Clear technical and product sense: you can prioritize the right reliability and scaling problems without over-engineering.

• High standards for engineering quality, including tests, runbooks, refactoring, and long-term maintainability.

Tech Stack You May Work With

• Python and/or TypeScript

• Pipeline schedulers, workflow orchestration, queues, workers, and event-driven systems

• APIs, data stores, caches, object storage, and metadata systems

• Docker, CI/CD, deployment automation, monitoring, logging, and alerting

• Internal developer tools and AI-assisted engineering workflows

Bonus Points

• Experience building or operating production data pipelines, workflow platforms, job schedulers, or internal automation systems.

• Experience with large-scale backfills, dependency management, rate limits, leases, distributed locks, or multi-tenant job execution.

• Experience improving operational excellence through dashboards, alerts, runbooks, self-healing automation, or incident review processes.

• Experience migrating fragile scripts or cron-based workflows into a more reliable platform without disrupting users.

This Role May Not Be a Fit If

• You only want to build new features and do not enjoy operational ownership or production support.

• You treat pipeline failures as isolated bugs instead of signals about system design, observability, or ownership boundaries.

• You prefer local optimization and do not want to reason about end-to-end workflow reliability and scaling.

• You are uncomfortable working with ambiguous cross-functional requirements or coordinating across teams.

Why This Role Matters

Pipelines are the connective tissue of many production systems. When they are reliable, scalable, and easy to operate, the whole organization moves faster; when they are fragile, every team pays the cost. This role gives the right person a chance to build durable infrastructure, reduce operational drag, and directly improve the speed and reliability of our platform.

What We Would Like to See When You Apply

• Production pipelines, workflow systems, schedulers, or automation platforms you have owned.

• Examples of how you improved reliability, scalability, observability, or operational support for a critical system.

• Cases where you handled ambiguous workflow requirements, complex dependencies, failures, backfills, or migrations.

• Examples that show your technical judgment, communication, and ability to balance pragmatic delivery with long-term maintainability.