Job Description
Job DescriptionPay Rate Range: $70.00 - $75.35/hr. on W2
About the Role:
- We are looking for a Cloud & Data Engineer to help build, optimize, and operate our Voice of Customer (VoC) AI Platform.
- The platform ingests structured and unstructured data from multiple enterprise sources with varying degrees of signal-to-noise ratio and is primarily built on Microsoft Azure, with data processing and ETL/ELT workloads powered by Azure Databricks.
- This is a hands-on role spanning data engineering, cloud infrastructure, and platform reliability.
Key Responsibilities:
Data Engineering
- Build and maintain ETL/ELT pipelines for structured and unstructured data sources.
- Develop data-processing workloads using Azure Databricks, Python/PySpark, and SQL.
- Optimize Databricks jobs, compute, and data layouts for performance and cost.
- Work with Delta Lake and Unity Catalog for data storage, governance, and access control.
- Implement data quality checks, monitoring, error handling, and recovery mechanisms.
Cloud & Platform Engineering:
- Build, operate, and optimize Azure services including App Service, Functions, Blob/ADLS storage, Azure SQL DB for Postgres, Key Vault, Container Registry and Azure Monitor.
- Experience with cloud AI services including vector search and embedding pipelines.
- Improve platform performance, scalability, reliability, security, and cost efficiency.
- Implement secure identity, and access management using Azure managed identities, Key Vault, secrets managed via scopes.
- Support development, staging, and production environments.
Automation & Operations:
- Improve platform observability through logging, monitoring, dashboards, and alerting.
- Automate infrastructure and deployments using Infrastructure as Code (IaC) and CI/CD.
- Troubleshoot issues across cloud services, storage, databases, networking, and data pipelines.
- Support production incidents and root-cause analysis.
- Collaborate with application, AI/ML, and data engineering teams.
Skills:
Required Qualifications:
- 3–5 years of experience in data engineering, cloud engineering, or a related role.
- Hands-on experience with Microsoft Azure, Databricks, APIs, and object storage.
- Strong skills in Python/PySpark, SQL.
- Experience building and operating production data ETL/ELT pipelines.
- Experience with relational and semi-structured/unstructured data, database technologies.
- Experience with Spark, Delta Lake, and lakehouse architectures.
- Working knowledge of Databricks Unity Catalog and data-access controls.
- Understanding of cloud security concepts such as RBAC, managed identities, service principals, and Key Vault.
- Experience with Git, CI/CD, monitoring, troubleshooting, and performance optimization.
Preferred Qualifications:
- Experience with Terraform or ARM templates.
- Experience with Azure DevOps or GitHub Actions.
- Familiarity with Azure networking, including VNets and private endpoints.
- Experience with Azure Data Factory, Databricks Workflows, or Airflow.
- Familiarity with data platforms supporting AI/ML, NLP, analytics, or GenAI applications.
Education:
- No formal requirements if skills are met.