Prashant Chauhan

Experience

• 

  Infosys

  Jun 2021 - Sept 2022

  System Engineer

  1. Developed queries, scheduled queries and tables for data analysis2. Learnt about BigData concepts of HDFS, map reduce3. Created External tables and views4. Developed scripts to ingest from GCS storage bucket and load into BigQuery

• 

  News Corp

  Aug 2022 - now

  Associate Data Engieer

  1. Implemented airflow deferrable operators for significant resource savings and reduced operational costs by minimizing worker idle time and optimizing task scheduling2. Created multiple end-to-end ETL pipelines to seamlessly gather data fromvarious data sources by making an API call , ingesting in GCP Storage Bucketand load into GCP BigQuery DWH including Data Marts for enhanced dataanalysis and reporting capabilities3. Ingested data from diverse upstream sources such as GCS Storage, AWS S3bucket , BigQuery Data Lake, API Endpoints and Google Sheets4. Developed and maintained Airflow DAGs, enhancing ETL processes with varioustransformation logics for seamless data integration in our data warehouseestablishing deployment in both dev and prod environments5. Responsible for developing Python/ SQL scripts on regular basis for adhocrequirements from client like backfilling of data, business insights etc6. BigQuery Pricing Dashboard for granular project cost tracking, providingdetailed insights into storage costs ,processing cost of queries measured interms of slot hours and TBs processed, with a specific focus on model-specificanalysis across different regions using data transfer service as per variouspricing models offered by GCP in order to opt the best plan7. Cost reduction/ optimisation of pipeline by clustering and partition of tables inBigQuery projects, resulting in a substantial reduction in slot hours usage8. Lowered storage costs by migration of various tables from BigQuery to GCSstorage, implemented external tables for flexible retrieval, and utilized views aspipeline data sources to enhance efficiency.9. Implemented Metabase and Slack alerts for pipeline and table monitoring inBigQuery data warehouse to enhance operational visibility and efficiency.10. Produced comprehensive project documentation, elucidating key columnmeanings in each table, and crafted dimensional modelling along witharchitectural diagram Show less