Apply on Employer Site

Stanley David and Associates · 7 hours ago

Senior Data Engineer

United States

Full-time

Remote

Senior Level

$120K/yr - $150K/yr

Stanley David and Associates is seeking a Senior Data Engineer to lead the design and development of robust ETL/ELT data pipelines. The role involves architecting scalable data architectures and collaborating with stakeholders to deliver tailored data solutions that enable data-driven decision-making.

Human ResourcesInformation TechnologyService Industry

Growth Opportunities

Responsibilities

Lead the design and development of robust ETL/ELT data pipelines, ensuring efficient data ingestion, processing, and transformation from diverse sources into AWS data warehouses and data lakes. This includes designing and implementing solutions for batch and streaming data, handling various data formats like JSON, CSV, Parquet, and Avro

Architect, build, and optimize scalable data architectures, including data lakes (e.g., S3, Delta Lake, Iceberg) and data warehouses (e.g., Redshift, Snowflake) on AWS, ensuring optimal performance and data accessibility

Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements, design appropriate data models and schemas, and deliver tailored data solutions that enable data-driven decision-making

Implement advanced data quality and governance practices, ensuring data accuracy, consistency, and compliance with relevant regulations

Optimize data retrieval and develop dashboards and reports using various tools, leveraging deep understanding of data warehousing and analytics concepts

Mentor and guide junior data engineers, fostering a culture of technical excellence and continuous improvement within the team

Proactively identify and resolve operational issues, troubleshoot complex data pipeline failures, and implement evolutionary recommendations for system improvements

Qualification

AWS cloud servicesData pipeline developmentData modeling techniquesBig data technologiesPython programmingSQL proficiencyContainerization technologiesData security principlesDevOps practicesVersion control systemsCommunication skillsProblem-solving skills

Required

Experience as a Data Engineer, with a strong focus on AWS cloud services for data solutions

Expertise in designing, developing, optimizing, and troubleshooting complex data pipelines, leveraging AWS services like Glue, Lambda, EMR, S3, Redshift, Athena, Kinesis, Step Functions, DynamoDB, and Lake Formation

Proficiency in programming languages such as Python (including Py Spark) and SQL (including advanced SQL, PL/SQL, query tuning), with the ability to write and optimize complex queries and scripts for data manipulation and transformation

Experience with big data technologies like Apache Spark, Hadoop, Kafka, and potentially real-time streaming platforms like Kinesis or Flink

In-depth knowledge of data modeling techniques, including relational, dimensional (star schema, snowflake schema), and potentially graph databases, and schema design for data warehouses and data lakes

Experience with containerization technologies like Docker and Kubernetes for scalable deployments

Strong understanding of data security and compliance principles and best practices

Familiarity with DevOps practices for managing data infrastructure and CI/CD pipelines using tools like AWS CodePipeline, Jenkins, or others

Demonstrated experience in supporting ML/AI projects, including deploying pipelines for feature engineering and model inference

Excellent problem-solving, analytical, and communication skills, with the ability to collaborate effectively with cross-functional teams and stakeholders

Experience with version control systems like Git