Lead Data Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer based in India. As a Lead Data Engineer, you’ll architect and own the end-to-end data sourcing and ETL infrastructure powering a large-scale B2B data platform. You’ll transform complex, unstructured web data into accurate, structured datasets across multiple international markets. The role combines advanced data engineering, web extraction, LLM infrastructure, agentic workflows, and production-grade orchestration. You’ll establish the technical standards and frameworks that enable other engineers to build reliable, scalable extraction systems. A significant part of the role is hands-on, tackling difficult problems such as anti-bot defenses, JavaScript-heavy sources, schema drift, and cost optimization. You’ll also shape how AI is evaluated and operated in production through versioning, observability, labelled datasets, and precision and recall measurement. Working within a lean, senior, AI-native environment, you’ll have substantial autonomy and direct ownership of architecture, quality, performance, and outcomes.