senior data engineer

توضیحات آگهی

We re looking for a Senior Data Engineer to join the Snapp Market Data Engineering team and help build and operate scalable reliable data platforms that power analytics BI and data driven products We re looking for someone who takes end to end ownership can turn ambiguous requirements into reliable technical solutions thinks beyond individual pipelines and cares deeply about reliability performance data quality and maintainability You ll work closely with Data Engineers Data Scientists BI Analytics and Product teams taking ownership of critical data systems and contributing to architecture performance reliability and engineering standards What You ll Do Design build and operate scalable batch and real time data pipelines using Kafka Spark and Airflow Take end to end ownership of critical pipelines ensuring reliability scalability performance and maintainability Design and optimize ClickHouse schemas data models and queries for high volume analytical workloads Build and maintain ETL ELT and CDC pipelines using MySQL Debezium Kafka Connect and Schema Registry Work with distributed systems and event driven architectures to improve latency throughput reliability and data quality Build production grade data services and tooling using Python and or Java with MinIO or similar object storage technologies Operate data services on Docker Kubernetes and contribute to CI CD and GitOps workflows using Argo CD Argo Workflows Establish best practices around testing observability monitoring documentation and operational excellence using tools such as Prometheus and Grafana Troubleshoot complex production and data quality issues and continuously improve platform performance reliability and cost efficiency Mentor Data Engineers participate in technical reviews and contribute to architectural decisions and engineering standards Required Qualifications 4 years of professional experience in Data Engineering Software Engineering or a related field Strong programming skills in Python and or Java Strong SQL skills and experience with MySQL or similar relational databases Hands on experience with ClickHouse including data modeling schema design and query optimization Strong experience with Kafka and event driven architectures Hands on experience with Debezium Kafka Connect and Schema Registry with a solid understanding of CDC concepts Production experience with Airflow and Spark PySpark Experience with Docker and Kubernetes in production environments Experience with Prometheus Grafana and production monitoring observability Strong understanding of distributed systems including partitioning replication fault tolerance scalability and high availability Strong software engineering fundamentals including Git testing code review and system design Ability to take ownership troubleshoot complex production issues and make sound technical decisions independently Preferred Qualifications Experience building and operating large scale high throughput data platforms Experience with StarRocks Trino Iceberg or similar modern data technologies Experience with data quality data catalog and observability frameworks Experience with Golang Kubernetes operators or platform engineering Experience mentoring engineers and contributing to technical direction and engineering standards تهران تهران زعفرانیه تمام وقت اسنپ مارکت کارشناس ارشد جنسیت تفاوتی ندارد اینترنت تجارت الکترونیک خدمات آنلاین