ALLDATA · 10 hours ago
Sr. Data Architect
Elk Grove, CA
Full-time
Hybrid
Senior Level, Lead/Staff
$119K/yr - $208K/yr
10+ years expALLDATA is the industry’s #1 choice for unedited original equipment manufacturer (OEM) automotive repair and collision information. The Senior Data Architect will lead a team of data engineers to build and optimize a robust data platform, focusing on the implementation of the Databricks Lakehouse platform and ensuring best practices in data engineering and governance.
AutomotiveInformation TechnologyLogisticsSoftware
Responsibilities
Lead, mentor, and manage a team of data engineers, providing technical guidance, code reviews, and foster a high-performing team
Own the Databricks platform architecture and implementation, ensuring the environment is secure, scalable, and optimized for the organization’s data processing needs. Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark
Implement and manage Databricks Unity Catalog for unified data governance. Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive data
Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards. Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine
Lead performance tuning initiatives. Profile data processing code to identify bottlenecks and refactor for improved throughput and lower latency. Implement best practices for incremental data processing with Delta Lake, and ensure compute cost efficiency (e.g., by optimizing cluster utilization and job scheduling)
Work closely with domain analysts, data scientists and product owners to understand requirements and translate them into robust data pipelines and solutions. Ensure that data architectures support analytics, reporting, and machine learning use cases effectively
Integrate Databricks workflows into the CI/CD pipeline using DevOps principles and Git. Develop automated deployment processes for notebooks and jobs to promote consistent releases. Manage source control for Databricks code (using GitLab) and collaborate with DevOps engineers to implement continuous integration and delivery for data projects
Collaborate with security and compliance teams to uphold data governance standards. Implement data masking, encryption, and audit logging as needed, leveraging Unity Catalog and GCP security features to protect sensitive data
Stay up to date with the latest Databricks features and industry’s best practices. Proactively recommend and implement improvements (such as new performance optimization techniques or cost-saving configurations) to continuously enhance the platform’s reliability and efficiency
Qualification
Required
10+ years of experience in data engineering, data architecture, or related roles, with a track record of designing and deploying data pipelines and platforms at scale
Significant hands-on experience with Databricks (preferably GCP) and the Apache Spark ecosystem. Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format
Strong experience working with cloud data platforms (GCP preferred, or AWS/Azure). Familiarity with GCP Storage principles
Strong skills in vector databases and embedding models to support scalable RAG systems. Proficient in optimizing retrieval and indexing for LLM integration
Strong experience in managing structured, semi structured and unstructured data in Databricks
Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in Databricks
Advanced SQL skills with the ability to write and optimize complex queries. Solid understanding of data warehousing concepts and performance tuning for SQL engines
Proven ability to optimize ETL jobs for performance and cost efficiency. Experience tuning cluster configurations, parallelism, and caching to improve job runtimes and resource utilization
Demonstrated experience implementing data security and governance measures. Comfortable configuring Unity Catalog or similar data catalog tools to manage schemas, tables, and fine-grained access controls. Able to ensure compliance with data security standards and manage user/group access to data assets
Experience leading and mentoring engineering teams. Excellent project leadership abilities to coordinate multiple projects and priorities. Strong communication skills to effectively collaborate with cross-functional teams and present architectural plans or results to stakeholders
Experience working in an Agile environment
Preferred
Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate
Exposure to related big data and streaming tools such as Apache Kafka, GCP Pub/Sub services, Apache Airflow and BI/analytics tools (e.g., Power BI, Looker Studio) is advantageous
Benefits
Competitive pay
Unrivaled company culture
Medical, dental and vision plans
Exclusive discounts and perks, including an AutoZone in-store discount
401(k) with company match and Stock Purchase Plan
AutoZoners Living Well Program for free mental health support
Opportunities for career growth
Paid time off
Life, and short- and long-term disability insurance options
Health Savings and Flexible Spending Accounts with wellness rewards
Tuition reimbursement
Company
ALLDATA
Alldata is the leading provider of OEM service and repair information to the professional automotive service and collision industries. It is a sub-organization of AutoZone.
201-500 employees
H1B Sponsorship
ALLDATA has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (4)
2024 (2)
2023 (5)
2022 (6)
2021 (6)
2020 (5)
Funding
Current Stage
Growth StageTotal Funding
unknown1996-02-07Acquired
Leadership Team
Satwinder Mangat
President, ALLDATA
Recent News
Globe Newswire
2024-03-01
Company data provided by crunchbase