Experience

• 

  Capgemini

  Nov 2014 - Jan 2018

  Hadoop developer

  • Creation of technical metadata in Hive using the mapping documents.• Developed the code to generate the DDL queries to create the tables in Hive based on the technical metadata.• Loading the files into Hive across different schemas on daily partition and performance optimizations in Hive.• Data validation in Hive with the source schema generated.• Provided the solution for reconciliation reports to ensure that no data loss from application source to Hive after loading.• Familiar with Spark framework and hands on experience in Scala using Eclipse with scala IDE & SBT.• Performed various spark core transformations and actions on top of data.• Involved in unit testing, regression testing and prepared the test cases.• Worked on falcon and error analysis using Oozie & logs from the Job history server. Show less

• 

  Renault nissan mitsubishi alliance, india

  Feb 2018 - Nov 2021

  Data engineer

  • Implemented End-To-End Big Data applications from scratch in Hadoop ecosystem• Responsible for building scalable distributed data solutions using Spark.• Worked with various files format like CSV, JSON, ORC, AVRO and Parquet• Optimized spark jobs using various optimization techniques• Responsible for developing custom UDFs and UDAFs in Hive

• 

  Innova solutions

  Sept 2021 - May 2022

  Senior data engineer

  • Worked on Healthcare product development in the AWS platform• Implemented end-to-end big data application (from Data Discovery to Reporting)• Integrated AWS S3, Glue, EMR, Lambda and Apache Druid to amplify data storage and analysis capabilities, yielding a significant 30% enhancement in data accessibility and processing efficiency.

• 

  Dubai technologies

  May 2022 - now

  Senior data engineer

  • Lead deep technical engagements with partners, including Briefings, Proof of Concepts• Worked on Azure POCs - ADLS, Delta Lake, Databricks, Data Factory• Implemented Data pipeline to ingest data in DataLake• Collaboration with stakeholders to define data requirements and data models• Optimized data ingestion applications for higher performance