iLink Digital · 4 hours ago
Lead Data Engineer- DataBricks
Milpitas, CA
Full-time
Onsite
Lead/StaffiLink Digital is seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud-based lakehouse platforms. The role combines hands-on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.
Embedded SoftwareSoftwareInformation TechnologyEmbedded Systems
Responsibilities
Lead the design and implementation of Databricks Lakehouse architectures
Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
Drive architectural decisions for batch and streaming data pipelines
Establish coding standards, best practices, and reusable frameworks
Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
Optimize Spark jobs for performance, reliability, and cost
Implement Delta Lake features (ACID, time travel, schema enforcement)
Develop and manage Databricks workflows, jobs, and clusters
Architect Databricks solutions on Azure (preferred) or AWS
Integrate Databricks with cloud storage and data services
Enable BI and analytics consumption (Power BI, Tableau)
Implement data governance using Unity Catalog
Define RBAC, data access controls, and security best practices
Enable CI/CD for Databricks using GitHub / Azure DevOps
Use Infrastructure-as-Code (Terraform) for environment management
Lead, mentor, and grow data engineering teams
Conduct design and code reviews
Collaborate with Data Architects, Product Owners, and stakeholders
Support production releases, monitoring, and incident resolution
Qualification
Required
Expert-level Databricks experience (Azure or AWS)
Strong Spark / PySpark / Spark SQL expertise
Delta Lake and Lakehouse architecture
Streaming (Structured Streaming) experience
Strong experience with Azure or AWS cloud platforms
Data orchestration tools (ADF, Airflow, or similar)
Strong SQL and data modeling skills
Git-based version control
CI/CD pipelines for data engineering workloads
Terraform or similar IaC tools
Bachelor's or Master's degree in Computer Science, Engineering, or related field
Preferred
Experience with MLflow and MLOps workflows
Exposure to Microsoft Fabric or Snowflake
Databricks certifications (Professional Data Engineer / Architect)
Experience working in Agile environments
Company
iLink Digital
Ilink Digital is a software company that develops custom applications, components, and frameworks on the Microsoft platform.
Founded in 2002
1001-5000 employees
H1B Sponsorship
iLink Digital has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (43)
2024 (35)
2023 (36)
2022 (46)
2021 (39)
2020 (60)
Funding
Current Stage
Late StageTotal Funding
$75.01MKey Investors
True North
2023-10-05Private Equity· $75M
2014-04-16Seed· $0.01M
Leadership Team
Sree Balaji
Group CEO & Co-founder
Ritesh Kapadia
Field Chief Technology Officer
Recent News
Company data provided by crunchbase