Shubham Vats

Experience

• 

  Defence Research and Development Organisation (DRDO)

  May 2018 - Jul 2018

  Internship Trainee

  Data Analysis, gathering and management of data.

• 

  Capgemini

  Oct 2020 - Aug 2022

  Associate Consultant

  DBS Hyderabad- Specialist Data Engineer (Contract)- Took responsibility for development of module pushing data from HDFS to AWS S3 Bucket for over 100 TB of data and maintaining audit reports in MariaDB, reducing manual effort of 5 hours every week.• Worked on spark optimization for over 600 jobs including ingestions and computes and improved the performance of DAGs from 30 mins to 2 hours, consequently improving the cluster efficiency.• Designed and Implemented data extraction framework using shell scripting and spark from a cluster and ingesting into Hadoop and improved the efficiency from 50% to 90%.• Developed the email functionality for finding prohibited transactions from 2 million records and notifying the users on daily basis using python and shell reducing the manual workload by 60% over each week.• Translated business propositions into quantitative queries, restored and extracted over 100 million records for client requirements using Sqoop and HQL.• Root cause analysis for reporting issues using Teradata SQL backtracking from semantic layer to common data warehouse. Show less

• 

  Chubb

  Aug 2022 - Oct 2022

  Data Engineer
• 

  Amazon

  Nov 2022 - now

  Data Engineer II