Python with Spark Developer (5.1-7 years)-Chennai

MAKE AN IMPACT

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

#BEYOURSELFATWORK

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

MAKE AN IMPACT

We offer

• A work culture focused on innovation and creating lasting value for our clients and employees
• Ongoing learning opportunities to help you acquire new skills or deepen existing expertise
• A flat, non-hierarchical structure that will enable you to work with senior partners and directly with clients

Python Developer – PySpark | 5–7 Years | Chennai

Position Purpose

• Overall, 5-7 years of experience as a Python Developer with Spark. 

• The Python / PySpark Developer will design, develop, test, and maintain high‑performance data‑processing pipelines using strong Python, pyspark and SQL skills.  

• You will transform raw market, trade, and client data into trusted, consumable data‑sets that feed downstream analytics, reporting, and machine‑learning models used by the CEFS – Client Engagement and Financial Securities franchise. 

• Sound knowledge of Agile (Scrum/Kanban). 

Responsibilities 

Direct Responsibilities 

• Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL. 

• Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting. 

• Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications. 

• Unit‑test, integrate‑test, and review code. 

• Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs. 

• Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures. 

• Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint. 

• Keep up to date with latest technologies and trends and provide input, expertise and recommendations. 

Contributing Responsibilities 

• Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement.  

• Participate in Agile ceremonies (sprint planning, daily standups, retrospectives) and help groom the backlogs. 

• Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data‑engineering patterns. 

• Evaluate emerging technologies and deliver proof‑of‑concepts for CEFS. 

Technical & Behavioral Competencies 

• Resourceful to quickly understand complexities involved and provide the way forward. 

• Good experience in technical analysis of n-tier applications with multiple integrations using object oriented, APIs & Microservices approaches. 

• Strong knowledge about design patterns and development principles. 

• Inclination and prior experience of working across SQL, Python and ETL. 

• Strong Hands-on experience - Python (NumPy, pandas, Python Frameworks, Restful APIs, MS-SQL or Oracle.  

The rest of this description is on the employer’s own page.