Talent Groups · 5 hours ago
Senior Data Engineer
Talent Groups is seeking a Senior Data Engineer to join a growing Enterprise Data team and play a key role in building and scaling modern data platforms. This role focuses on designing and optimizing data pipelines that support analytics reporting and machine learning initiatives across the organization.
Responsibilities
Design develop and optimize scalable data pipelines using Azure Databricks
Build and manage data ingestion transformation and storage solutions using Azure Data Factory, Azure Data Lake, and Delta Lake
Develop data solutions that support analytics reporting and machine learning use cases
Implement CI CD pipelines using tools such as Azure DevOps, Git, and Terraform
Optimize Spark performance including partitioning orchestration and cost efficiency
Collaborate with analysts data scientists and business stakeholders to deliver trusted reusable datasets
Serve as the onsite face of the data engineering team and partner with leadership
Mentor junior and associate data engineers
Monitor troubleshoot and improve the reliability and scalability of existing pipelines
Qualification
Required
Five plus years of experience in data engineering with a strong focus on Azure
Hands on experience with Azure Databricks, Azure Data Lake, and Data Factory
Strong proficiency in PySpark, SQL, Python, and Apache Spark
Solid understanding of data warehousing, ETL, ELT, and data modeling best practices
Experience with version control CI/CD pipelines and infrastructure as code
Knowledge of Spark performance tuning and job orchestration
Strong problem solving skills and attention to detail
Excellent communication skills with both technical and non technical stakeholders
Ability to work independently and lead in a fast paced Agile environment
Passion for delivering clean maintainable high quality code
Preferred
Experience with Unity Catalog, Databricks Workflows, and Delta Live Tables
Familiarity with DevOps practices or Terraform for Azure provisioning
Understanding of data security, RBAC, and cloud compliance
Experience integrating Databricks with Power BI or similar analytics tools
Exposure to real time data processing using Kafka Event Hubs or Structured Streaming
Company
Talent Groups
Talent Groups delivers exceptional experiences by solving the toughest IT talent and technology challenges across all industries.
Funding
Current Stage
Late StageTotal Funding
unknown2018-07-01Acquired
Recent News
2024-05-20
Company data provided by crunchbase