Apply on Employer Site

INSPYR Solutions · 5 days ago

Sr. Data Architect, Elk Grove, CA (Hybrid – 3 days a week on-site)

Elk Grove, CA

Full-time

Hybrid

Senior Level, Lead/Staff

$160K/yr - $175K/yr

10+ years exp

INSPYR Solutions is a national expert in delivering flexible technology and talent solutions. They are seeking a Senior Data Architect to build and optimize a robust data platform in the automotive industry, leading a team of data engineers and owning the architecture and implementation of the Databricks Lakehouse platform.

Information TechnologyProfessional ServicesStaffing Agency

No H1B

Responsibilities

Lead, mentor, and manage a team of data engineers, providing technical guidance, code reviews, and foster a high-performing team

Own the Databricks platform architecture and implementation, ensuring the environment is secure, scalable, and optimized for the organization’s data processing needs. Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark

Implement and manage Databricks Unity Catalog for unified data governance. Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive data

Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards. Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine

Lead performance tuning initiatives. Profile data processing code to identify bottlenecks and refactor for improved throughput and lower latency. Implement best practices for incremental data processing with Delta Lake, and ensure compute cost efficiency (e.g., by optimizing cluster utilization and job scheduling)

Work closely with domain analysts, data scientists and product owners to understand requirements and translate them into robust data pipelines and solutions. Ensure that data architectures support analytics, reporting, and machine learning use cases effectively

Integrate Databricks workflows into the CI/CD pipeline using DevOps principles and Git. Develop automated deployment processes for notebooks and jobs to promote consistent releases. Manage source control for Databricks code (using GitLab) and collaborate with DevOps engineers to implement continuous integration and delivery for data projects

Collaborate with security and compliance teams to uphold data governance standards. Implement data masking, encryption, and audit logging as needed, leveraging Unity Catalog and GCP security features to protect sensitive data

Stay up to date with the latest Databricks features and industry’s best practices. Proactively recommend and implement improvements (such as new performance optimization techniques or cost-saving configurations) to continuously enhance the platform’s reliability and efficiency

Qualification

DatabricksApache SparkData Pipeline DesignGCPSQL OptimizationData GovernanceETL OptimizationAgile MethodologiesLeadershipCommunicationTeam CollaborationProject Management

Required

10+ years of experience in data engineering, data architecture, or related roles, with a track record of designing and deploying data pipelines and platforms at scale

Significant hands-on experience with Databricks (preferably GCP) and the Apache Spark ecosystem. Proficient in building data pipelines using PySpark/Scala and managing data in Delta Lake format

Strong experience working with cloud data platforms (GCP preferred, or AWS/Azure). Familiarity with GCP Storage principles

Strong skills in vector databases and embedding models to support scalable RAG systems. Proficient in optimizing retrieval and indexing for LLM integration

Strong experience in managing structured, semi structured and unstructured data in Databricks

Ability to inspect existing data pipelines, discern their purpose and functionality, and re-implement them efficiently in Databricks

Advanced SQL skills with the ability to write and optimize complex queries. Solid understanding of data warehousing concepts and performance tuning for SQL engines

Proven ability to optimize ETL jobs for performance and cost efficiency. Experience tuning cluster configurations, parallelism, and caching to improve job runtimes and resource utilization

Demonstrated experience implementing data security and governance measures. Comfortable configuring Unity Catalog or similar data catalog tools to manage schemas, tables, and fine-grained access controls. Able to ensure compliance with data security standards and manage user/group access to data assets

Experience leading and mentoring engineering teams. Excellent project leadership abilities to coordinate multiple projects and priorities. Strong communication skills to effectively collaborate with cross-functional teams and present architectural plans or results to stakeholders

Experience working in an Agile environment

US Citizen, GC Holders, or Authorized to Work in the U.S

Preferred

Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate

Exposure to related big data and streaming tools such as Apache Kafka, GCP Pub/Sub services, Apache Airflow and BI/analytics tools (e.g., Power BI, Looker Studio) is advantageous