DataPattern · 5 hours ago
Senior Data Engineer
San Ramon, CA
Full-time
Onsite
Senior Level, Lead/Staff
8+ years expDataPattern is seeking a highly skilled Data Engineer with deep Databricks expertise to design, build, and optimize scalable, cloud-based data platforms and analytics solutions. The ideal candidate will have strong hands-on experience with distributed data processing, modern data architectures, and performance optimization across large-scale datasets.
Artificial Intelligence (AI)ConsultingDevOpsInformation TechnologyInternet of ThingsMachine LearningProduct Design
Responsibilities
Design, develop, and maintain end-to-end data pipelines using Databricks (Spark) for batch and streaming workloads
Build and optimize ETL/ELT frameworks leveraging Delta Lake, medallion architecture (Bronze/Silver/Gold), and scalable data models
Implement data ingestion from diverse sources including relational databases, APIs, event streams, and cloud storage
Develop robust data transformations using PySpark/Scala, ensuring performance, reliability, and scalability
Optimize Spark jobs through partitioning, caching, broadcast joins, and cluster tuning
Implement data quality checks, validations, and monitoring to ensure accuracy and consistency
Work closely with analytics, data science, and business teams to enable self-service analytics and reporting
Support data governance, security, and access control using cloud-native and Databricks capabilities
Troubleshoot pipeline failures, optimize costs, and improve overall platform efficiency
Qualification
Required
Databricks Certified Data Engineer (Mandatory)
Advanced SQL skills and experience in data modeling (dimensional & analytical models)
Hands-on experience building large-scale ETL/ELT pipelines
Deep experience with Databricks features: Delta Lake, Unity Catalog, Workflows, and Notebooks
Strong experience with cloud platforms – AWS, Azure, or GCP (storage, compute, networking)
Experience with data orchestration tools such as Airflow, Azure Data Factory, or similar
Experience in performance tuning, cost optimization, and data reliability engineering
Strong understanding of data security, governance, and compliance best practices
Excellent communication, analytical, and problem-solving skills
Preferred
Knowledge of streaming frameworks (Spark Structured Streaming, Kafka/Event Hubs – preferred)
Company
DataPattern
DataPattern is an IT consulting firm that offers data, application development, IoT, AI, product engineering, DevOps, and MLOps services.
51-200 employees
H1B Sponsorship
DataPattern has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (11)
2024 (8)
Funding
Current Stage
Growth StageLeadership Team
Stanley Moses
Founder & Managing Director
Company data provided by crunchbase