Job description
- Develop and maintain scalable ETL/ELT pipelines using Python, Airflow, and dbt
- Build and optimize real-time streaming pipelines using Kafka, RabbitMQ, Spark and event-driven microservices
- Leverage AI coding agents (Claude Code, Cursor AI, GitHub Copilot) to accelerate data pipeline development, data transformation, and optimization
- Design and manage data storage solutions with PostgreSQL,ClickHouse and Elasticsearch
- Define and implement data modeling best practices for efficient and scalable data storage.
- Ensure high availability, performance, and security of data infrastructure on AWS cloud
- Design and implement BI-ready datasets in ClickHouse optimized for OLAP workloads and reporting use cases.
- Develop and manage denormalized schemas and data models to support analytics and amCharts-based reporting.
- Optimize query performance, partitioning, and indexing strategies in ClickHouse to handle large-scale analytical workloads.
- Implement best practices for data quality, validation, and observability
- Work on data ingestion using Airbyte or other connector frameworks
- Implement data transformation pipelines supporting analytical and ML workloads
- Collaborate with cloud and DevOps teams for containerized deployments (Docker, Kubernetes, AWS)
- Support integration of ML workflows and model APIs into production pipelines
Skills
Data EngineeringDistributed SystemsAI-assisted coding toolClickHouseOLAPSQLETL/ELTPower BIKafkaSparkPostgresMLOps
Nice to have
Problem SolvingCommunicationDebugging