ADF (Azure Data Factory)Databricks+Pyspark

You’ll lead the design and delivery of modern data engineering solutions that turn raw data into trusted, analytics-ready assets. Working at the intersection of orchestration, scalable processing, and cloud-native platforms, you’ll partner closely with product owners, analysts, and engineering teams to build reliable pipelines that power business decisions. This role is ideal for someone who enjoys owning end-to-end delivery—shaping architecture, guiding implementation, and mentoring engineers—while continuously improving performance, quality, and operational excellence. If you’re excited by solving complex data challenges, enabling self-serve analytics, and building a collaborative culture that values craftsmanship and learning, this is the place to make a meaningful impact. Technical Requirements: Technology->Big Data - Data Processing->PySpark Technology->Cloud Integration->Azure Data Factory (ADF) Technology->Data Engineering->Databricks Responsibilities: Key Responsibilities: Data Engineering & Delivery • Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing • Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability • Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing • Build reusable frameworks, templates, and standards for pipeline development and deployment Architecture & Performance • Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices • Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing) • Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration • Provide technical leadership, code reviews, and mentoring to ensure high engineering standards • Collaborate with stakeholders to translate business requirements into scalable data solutions • Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications: • BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field • 7–9 years of experience in data engineering with strong hands-on delivery ownership • Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns • Strong expertise in Databricks for building scalable data processing solutions • Hands-on proficiency with PySpark for building and optimizing distributed data transformations • Experience building production-grade pipelines with logging, error handling, and operational support readiness Preferred Skills: Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Data Engineering->Databricks,Technology->Big Data - Data Processing->PySpark