Job Summary:
We are looking for a Big Data Engineering Leader with 15+ years of experience, including strong expertise in Azure and Databricks ecosystems. The role involves leading data engineering team, advising clients on modern data platforms, designing lakehouse and data fabric solutions, defining technology roadmaps, and driving innovation through PoCs and IP development. Candidates should have expertise in Azure Databricks, PySpark, Microsoft Fabric, Azure Data Factory, Kafka, Hadoop, CI/CD, data governance, and emerging technologies such as GenAI and LLMs.
Job Description:
As a Big Data Engineering Leader, you will work with key leaders across the organization – tech consulting, analytics consulting, data science, data engineering, and business development to advance the company’s capabilities and positioning.
On a typical day you might,
· Provide client advisory on key engineering & architectural decisions across the full life cycle of big data engineering projects like Data Lake / Lakehouse implementation, DWH Modernization, Data Fabric implementation etc.
· Work with strategic clients to help define their data engineering roadmaps
· Identify new ways of leveraging technology to solve client engineering problems
· Track latest tech innovations and build relevant PoVs & PoCs to further our capabilities
· Identify opportunities to build IP. Lead the development efforts
· Lead capability building initiatives as part of our Innovation Lab and L&D Academy
· Participate in organization building, hiring, partnerships, conferences etc.
Profile Description:
· 15+ years of data engineering experience with 5+ years in the Big Data and Azure ecosystem.
· Experience of leading & mentoring multiple high-performing data engineering teams.
· Good experience of working on proposals, defining solutions, roadmaps & plan, doing estimation etc.
· Strong experience of working with following technologies:
· Azure ecosystem - Azure Databricks, Data Factory, Microsoft Fabric, Event Hub, Streaming Analytics, Cosmos DB, Unity Catalog and Purview.
· Must have: Strong Experience of working on Databricks ecosystem (PySpark, Notebooks, Unity Catalog, DLT, Workflows, SQL Warehouse etc.).
· Hadoop Ecosystem: Hadoop, Spark, Kafka, Hive, NoSQL – HBase / MongoDB / Cassandra, Impala, Sqoop etc.
· Programming experience either in Python or Scala/Java.
· CI/CD with Git, Jenkins / Azure DevOps.
· Exposure to NoSQL Database and Data Modelling
· Good exposure to some of the emerging technologies like Gen AI, LLM etc.
· Experience with:
· Data lake / Lakehouse / data fabric / data mesh design and implementation.
· Designing/building large cloud-based Data Platform solutions (on Azure).
· Experience of building solutions to support BI and Data Science teams in consuming the data in a secure and governed manner.
· B.Tech/B.E from reputed institute preferred.
