Data Engineering Specialist (Databricks)

Accenture helps the world’s leading enterprises reinvent by building their digital core and unleashing the power of AI to create value at speed for organizations across industries. Our strategy is to be the reinvention partner of choice for our clients and lead in the safe, widespread adoption of AI, and to be the most client-focused, AI-enabled, great place to work in the world. We bring together the talent of our approximately 799,000 people with proprietary assets and platforms, deep process and industry expertise, and leading ecosystem relationships to deliver end-to-end solutions and measurable outcomes at scale. Through our Reinvention Services, we offer broad expertise across Cybersecurity, Digital Core, Finance, Industry and Enterprise, Song, Supply Chain and Engineering, and Talent, with advanced capabilities in AI and Data, Industry and Process, and Technology. We serve approximately 9,000 clients and generated approximately $70 billion in FY25 revenue. Visit us at accenture.com.

Role overview 

Lead a delivery workstream within a client data engineering team migrating data from legacy warehouses to an Azure Databricks lakehouse. The role owns technical design, production-grade pipeline delivery, engineering standards, and coordination of a distributed engineering team. It also translates business requirements into technical solutions and communicates delivery trade-offs to client stakeholders. 

Key responsibilities 

• Lead a migration workstream from requirements and technical specifications through production deployment and stabilization. 

• Design and build scalable batch and streaming pipelines using PySpark, Databricks SQL, and Lakeflow Jobs and Declarative Pipelines, including cleansing, validation, monitoring, and logging. 

• Design Delta Lake medallion layers (bronze, silver, and gold) and dimensional models using fact and dimension tables. 

• Optimize Databricks workloads for performance and cost through compute sizing, partitioning, liquid clustering, and shuffle optimization. 

• Implement governance, security, access controls, and data quality standards using Unity Catalog. 

• Plan migrations from legacy warehouses and ETL tools, including data reconciliation and cutover validation. 

• Create and groom backlog items, estimate effort, track sprint delivery, and manage workstream risks, blockers, and dependencies. 

• Review code and establish reusable engineering patterns, CI/CD practices, and documentation standards. 

• Guide engineers across India, Canada, and Costa Rica and coordinate priorities and handoffs across time zones. 

• Present technical options, including cost, performance, and timeline implications, to client stakeholders. 

Required qualifications 

• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent relevant professional experience. 

• 5+ years of data engineering experience, including 3+ years of hands-on production experience with Databricks and Apache Spark. 

• Hands-on experience with Databricks workspace and compute management, Delta Lake, Unity Catalog, Lakeflow Jobs and Pipelines, Databricks SQL, notebooks, and Git folders. 

• Advanced PySpark, including the DataFrame API and performance tuning, plus advanced SQL. 

• Production experience designing dimensional models, including star schemas with fact and dimension tables. 

• Experience migrating data from legacy data warehouses to a cloud data platform. 

• Experience with Azure Data Factory and Azure Data Lake Storage Gen2. 

• Python experience for data processing and automation. 

• Working knowledge of data governance, data quality, data warehousing, and data architecture principles. 

• Demonstrated experience leading a technical workstream, reviewing code, coordinating delivery, and guiding engineers with minimal supervision. 

• Ability to explain technical trade-offs and summarize recommendations for client and senior stakeholders. 

Preferred qualifications 

• Databricks Certified Data Engineer Associate or Professional. 

• Microsoft Certified: Azure Databricks Data Engineer Associate. 

• Experience with IBM DataStage and SQL Server as legacy source systems. 

• Experience with CI/CD for data pipelines using Databricks Asset Bundles, Git, or Azure DevOps. 

• Experience with streaming ingestion using Structured Streaming or Auto Loader. 

• Exposure to AWS, Google Cloud, or PLX. 

• Client-facing consulting experience in digital analytics. 

What we offer

• Health and Life Insurance
• Accenture days, 3 additional vacation days
• On site doctor
• Birthday leave
• Internet reimbursement

About Accenture

Accenture is a leading global professional services company that helps the world’s leading businesses, governments and other organizations build their digital core, optimize their operations, accelerate revenue growth and enhance citizen services—creating tangible value at speed and scale. We are a talent- and innovation-led company with approximately 791,000 people serving clients in more than 120 countries. Technology is at the core of change today, and we are one of the world’s leaders in helping drive that change, with strong ecosystem relationships. We combine our strength in technology and leadership in cloud, data and AI with unmatched industry experience, functional expertise and global delivery capability. Our broad range of services, solutions and assets across Strategy & Consulting, Technology, Operations, Industry X and Song, together with our culture of shared success and commitment to creating 360° value, enable us to help our clients reinvent and build trusted, lasting relationships. We measure our success by the 360° value we create for our clients, each other, our shareholders, partners and communities.

 Visit us at www.accenture.com 

Equal Employment Opportunity Statement

We believe that no one should be discriminated against because of their differences. All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, military veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by applicable law. Our rich diversity makes us more innovative, more competitive, and more creative, which helps us better serve our clients and our communities.