Senior Data Engineer
We are looking for a Senior Data Engineer with strong experience in real-time data processing, large-scale data platforms, and cloud-based analytics solutions. The ideal candidate should have hands-on expertise in Spark Streaming, Python, BigQuery, and modern Data Lake architectures with a strong focus on Azure cloud technologies.
Required Skills
• Strong experience with *Apache Spark Streaming* and real-time data pipelines
. 5+ years of experience is mandatory
• Proficiency in *Python* for data engineering and automation
• Experience working with *Google BigQuery* for analytics and reporting
• Hands-on experience with *Azure Data Lake*, ADLS Gen2, and Azure cloud ecosystem
• Experience building scalable ETL/ELT pipelines
• Strong understanding of distributed data processing and large datasets
• Experience with Kafka/Event Hub or other streaming technologies
• Good knowledge of SQL and performance optimization
• Experience with CI/CD and data pipeline deployment processes
Preferred Skills
• Experience with Azure Databricks
• Exposure to Delta Lake or Lakehouse architecture
• Knowledge of Airflow, ADF, or orchestration tools
• Understanding of data governance and security best practices
• Experience supporting enterprise-scale analytics platforms
Responsibilities
• Design and develop scalable real-time and batch data pipelines
• Build and optimize streaming solutions using Spark Streaming
• Integrate data from multiple enterprise systems into Azure Data Lake
• Develop reusable Python-based data processing frameworks
• Work closely with analytics and business teams for reporting/data needs
• Ensure data quality, monitoring, and platform reliability
• Participate in architecture discussions and technical design reviews