CGI · 15 hours ago
Lead Developer - Python/Pyspark/API
Pittsburgh, PA
Contract
Onsite
Senior Level, Lead/Staff
$71K/yr - $157K/yr
6+ years expCGI is a leading IT and business consulting services firm, and they are seeking a highly skilled developer to join their Data Products Capabilities team. The role involves owning back-end service development and integration, focusing on scalable micro-services and APIs to enhance customer experiences.
AnalyticsApplication Performance ManagementBusiness IntelligenceConsultingCyber SecurityFinanceInformation TechnologyTechnical Support
Responsibilities
Design and develop scalable ETL/ELT pipelines using PySpark and Python for batch and real-time processing
Build and optimized Spark Streaming applications for real-time ingestion, transformation, and event-driven processing using Kafka or other messaging systems
Develop distributed data-processing workflows on Apache Spark, ensuring efficient computation and fault tolerance
Work extensively with SQL for data transformation, aggregation, and performance-tuned querying across large datasets
Integrate pipelines with Hadoop ecosystem components (HDFS, Hive, Yarn) and modern data platforms
Implement data quality checks, validations, and reconciliation logic for both batch and streaming data
Tune Spark jobs using partitioning, caching, broadcast joins, and resource-optimization techniques
Build CI/CD workflows using Git, Jenkins, and Bitbucket for automated deployments and version control
Collaborate with cross-functional teams to troubleshoot, monitor, and improve data pipelines in production environments
Ensure compliance with data security, governance, and access control practices
Qualification
Required
6-8 years of hands-on experience with Python and PySpark development
Strong expertise in Spark DataFrames, RDDs, Spark SQL, and distributed data processing
Practical experience building Spark Streaming or Structured Streaming applications
Solid understanding of ETL/ELT pipeline development using PySpark
Strong proficiency with SQL and query optimization
Experience with the Hadoop ecosystem (HDFS, Hive, Yarn) or similar big-data platforms
Experience with containerization and orchestration (e.g., Docker, Kubernetes) is an advantage
Knowledge of CI/CD tools like Git, Jenkins, Bitbucket
Understanding of job monitoring, logging, and performance tuning for both batch and streaming workloads
Benefits
Competitive compensation
Comprehensive insurance options
Matching contributions through the 401(k) plan and the share purchase plan
Paid time off for vacation, holidays, and sick time
Paid parental leave
Learning opportunities and tuition assistance
Wellness and Well-being programs
Company
CGI
CGI is an IT and business consulting services firm that offers consulting, cyber security, cloud, and IT services.
10001+ employees
H1B Sponsorship
CGI has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (775)
2024 (762)
2023 (795)
2022 (940)
2021 (846)
2020 (582)
Funding
Current Stage
Public CompanyTotal Funding
$1.2B2025-03-12Post Ipo Debt· $650M
2024-09-03Post Ipo Debt· $550.87M
1998-10-06IPO
Leadership Team
François Boulanger
President and Chief Executive Officer at CGI
Raymond McMann
VP, Global Oil & Gas Industry
Recent News
2026-01-13
syncni.com
2026-01-09
Company data provided by crunchbase