We are seeking an experienced Senior DataOps Engineer to implementation, and maintenance of our data infrastructure and pipelines. The ideal candidate will have a strong background in data engineering and DevOps principles, with a passion for automation, quality, and governance. You will act as a bridge between technical and business teams, ensuring our data platform is not only efficient and scalable but also reliable and compliant. This role is crucial for enabling data-driven decisions and accelerating the development lifecycle for all data initiatives.Key Responsibilities
Design & Implement Data Platforms: Design, develop, and maintain robust, scalable data pipelines and ETL processes, with a focus on automation and operational excellence.
Ensure Data Quality and Governance: Implement automated data validation, quality checks, and monitoring systems to ensure data accuracy, consistency, and reliability.
Manage CI/CD for Data: Develop, own and optimize the CI/CD pipelines for data engineering workflows, including automated testing and deployment of data transformations and schema changes.
Architect & Implement IaC: Use Infrastructure as Code (IaC) with Terraform to manage data infrastructure across various cloud platforms (Azure, AWS, GCP).
Performance & Optimization: Proactively monitor and optimize query performance, data storage, and resource utilization to manage costs and enhance efficiency.
Provide Technical Leadership: Mentor junior engineers, provide technical leadership, and champion best practices in data engineering and DataOps methodologies.
Collaborate with Stakeholders: Manage communication with technical and business teams to understand requirements, assess technical and business impact, and deliver effective data solutions.
Strategic Design: Possess the ability to see the big picture in architectural design, conduct thorough risk assessments, and plan for future scalability and growth.
Requirements
Experience: 3+ years of experience in data engineering, data warehousing, and ETL processes, with a significant portion of that time focused on DataOps or a similar operational role.
Platform Expertise: Strong experience with data platforms such as Databricks and exposure to multiple cloud environments (Azure, AWS, or GCP).
Data Processing: Extensive experience with Apache Spark for large-scale data processing.
Orchestration: Proficiency with data orchestration tools like Azure Data Factory (ADF), Apache Airflow, or similar.
CI/CD & Version Control: Expert-level knowledge of version control (Git) and experience with CI/CD pipelines (GitLab CI/CD, GitHub Actions).
IaC: Extensive hands-on experience with Terraform.
Programming: Strong programming skills in Python and advanced proficiency in SQL.
Data Modeling: Experience with data modeling tools and methodologies, specifically with dbt (data build tool).
AI & ML: Experience with AI-related technologies like Retrieval-Augmented Generation (RAG) and frameworks such as LangChain.
Data Observability: Hands-on experience with data quality and observability tools such as Great Expectations, Monte Carlo, or Soda Core.
Data Governance: Familiarity with data governance principles, compliance requirements, and data catalogs (e.g., Unity Catalog).
Streaming Technologies: Experience with stream processing technologies like Kafka or Flink.
BI Tools: Experience with data visualization tools (e.g. Power BI).
Containerization: Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
