Shivam Chauhan

Shivam Chauhan

System Engineer(Digital Cadre)

Followers of Shivam Chauhan2000 followers
location of Shivam ChauhanNoida, Uttar Pradesh, India

Connect with Shivam Chauhan to Send Message

Connect

Connect with Shivam Chauhan to Send Message

Connect
  • Timeline

  • About me

    Data Enthusiasts

  • Education

    • Noida Institute of Engineering & Technology

      2016 - 2020
      Bachelor of Technology - BTech Mechanical Engineering
  • Experience

    • Tata Consultancy Services

      May 2021 - Dec 2024
      System Engineer(Digital Cadre)

      - Built scalable and fault-tolerant systems and oversaw the complete ETL process- Experience in transferring data from GCP cloud to ADLS Gen2 using Azure Data Factory.- Gathering requirements from the user frequently to incorporate changes in the existing pipeline and help drive better business decisions and improve business outcomes.- Working closely with team members and solution architects to incorporate changes to the existing pipeline which are running on spark to deliver high performance for the product.- Built Azure Data Factory pipeline over Azure Databricks.- Worked on ADO (Azure Devops Organization) Stories, delivered and solved the most critical & highly prioritized Incidents with the minimum SLA cutting down the live impact on the business stakeholders.- Built a data lake for the Data Science team for their predictions and also built a DATA Warehouse for the BI team for their reporting and analysis. Utilized various activities of Azure Data Factory such as Copy, Lookup, GetMetadata, etc., along with various PySpark transformations to process data.- Used triggers to handle pipelines dependency and scheduling to achieve the end result.- Proficient in writing PySpark code for big data processing and analytics.- Experienced in creating and optimizing SQL queries for database management and data retrieval.Tech Stack: Spark, PySpark, SQL, Azure Databricks, Azure Data Factory, Python, Hive, CI/CD , HDFS, ETL, GIT, ADO(Azure Devops Organizations), Azure Data Lake, Delta Lake, Data Warehousing, ADLS GEN2 Show less

    • Quarks

      Dec 2024 - now
      Senior Data Engineer
  • Licenses & Certifications

  • Honors & Awards

    • Awarded to Shivam Chauhan
      Make a Ton - Secured 2nd position and won a cash prize of ₹15000/- at Galgotia University, for creating an innovative project