PETADATA · 19 hours ago
Data Engineer
Albany, NY
Full-time
Onsite
Senior Level, Lead/Staff
10+ years expPETADATA is currently hiring for a Data Engineer for one of their clients. The role involves designing and maintaining scalable data pipelines, integrating data from multiple sources, and optimizing data architectures and processing frameworks.
Information TechnologyMobile AppsProject ManagementSoftware
Responsibilities
Design, build, and maintain scalable data pipelines for ingesting, processing, and transforming data
Develop batch and real-time data processing workflows
Ensure data pipelines are reliable, efficient, and fault-tolerant
Design and implement data architectures(data lakes, data warehouses, lakehouses)
Create and maintain data models, schemas, and metadata
Optimize data storage and retrieval strategies
Integrate data from multiple sources (databases, APIs, streaming platforms, third-party tools)
Handle structured, semi-structured, and unstructured data
Implement data validation and quality checks
Work with big data tools such as Apache Spark, Hadoop, Kafka, Flink
Build streaming and real-time data processing systems
Optimize large-scale data processing performance
Use cloud platforms such as AWS, Azure, or GCP
Manage cloud-based data infrastructure
Manage and optimize SQL and NoSQL databases
Implement indexing, partitioning, and performance tuning
Support data access for analytics, AI, and reporting teams
Implement data quality checks, monitoring, and alerts
Enforce data governance, lineage, and metadata management
Ensure data security, privacy, and compliance (GDPR, HIPAA, etc.)
Automate data workflows using tools like Airflow, Luigi, Prefect
Schedule, monitor, and troubleshoot data jobs
Reduce manual processes through automation
Collaborate with data scientists, analysts, and AI engineers
Support analytics, BI, and machine learning initiatives
Translate business requirements into data solutions
Monitor pipeline performance and data freshness
Identify and resolve data issues and bottlenecks
Optimize costs and resource usage
Define data engineering standards and best practices
Lead data platform design and modernization efforts
Mentor junior data engineers
Drive data strategy and roadmap planning
Qualification
Required
Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field
Proven experience as a Data Engineer or similar role
Strong programming skills in Python and SQL (Scala or Java preferred)
Hands-on experience with big data frameworks such as Spark, Kafka, Hadoop
Experience designing and maintaining data pipelines and ETL/ELT processes
Strong understanding of data modeling, data warehousing, and lakehouse architectures
Experience with cloud platforms: AWS, Azure, or GCP
Hands-on experience with SQL and NoSQL databases
Experience with workflow orchestration tools (Airflow, Prefect, Luigi)
Knowledge of data quality, governance, and security best practices
Strong analytical, troubleshooting, and problem-solving skills
Excellent communication and collaboration abilities
Preferred
Scala or Java programming skills
Experience with workflow orchestration tools (Airflow, Prefect, Luigi)
Define data engineering standards and best practices
Lead data platform design and modernization efforts
Mentor junior data engineers
Drive data strategy and roadmap planning
Company
PETADATA
PETADATA is an Information Technology & Services company that specializes in delivering technology solutions to meet the needs of our clients.
51-200 employees
H1B Sponsorship
PETADATA has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2024 (10)
2023 (6)
2022 (13)
2021 (5)
2020 (12)
Funding
Current Stage
Growth StageLeadership Team
R
Ravi Kumar
Founder
Company data provided by crunchbase