Shivam Chauhan

Experience

• 

  Tata Consultancy Services

  May 2021 - Dec 2024

  System Engineer(Digital Cadre)

  - Built scalable and fault-tolerant systems and oversaw the complete ETL process- Experience in transferring data from GCP cloud to ADLS Gen2 using Azure Data Factory.- Gathering requirements from the user frequently to incorporate changes in the existing pipeline and help drive better business decisions and improve business outcomes.- Working closely with team members and solution architects to incorporate changes to the existing pipeline which are running on spark to deliver high performance for the product.- Built Azure Data Factory pipeline over Azure Databricks.- Worked on ADO (Azure Devops Organization) Stories, delivered and solved the most critical & highly prioritized Incidents with the minimum SLA cutting down the live impact on the business stakeholders.- Built a data lake for the Data Science team for their predictions and also built a DATA Warehouse for the BI team for their reporting and analysis. Utilized various activities of Azure Data Factory such as Copy, Lookup, GetMetadata, etc., along with various PySpark transformations to process data.- Used triggers to handle pipelines dependency and scheduling to achieve the end result.- Proficient in writing PySpark code for big data processing and analytics.- Experienced in creating and optimizing SQL queries for database management and data retrieval.Tech Stack: Spark, PySpark, SQL, Azure Databricks, Azure Data Factory, Python, Hive, CI/CD , HDFS, ETL, GIT, ADO(Azure Devops Organizations), Azure Data Lake, Delta Lake, Data Warehousing, ADLS GEN2 Show less

• 

  Quarks

  Dec 2024 - now

  Senior Data Engineer