Ashutosh Yadav

Experience

• 

  We - IT

  Jul 2021 - Aug 2021

  Software Developer

  Developed automation scripts for testing storage appliances using Python.Utilised frameworks such as Flask for persistence and application layers.Implemented security measures for the project by designing and executing an authentication and authorisation process.

• 

  Celebal Technologies

  Sept 2021 - Mar 2023

  Data Engineer

  Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats, resulting in uncovering insights into customer usage patterns by 80%.Extracted, transformed, and loaded data from various sources to Azure Data Storage services using a combination of Azure Data Factory and Spark SQL, resulting in a data ingestion rate of 90%. Participated in data quality checks and monitoring to ensure data integrity and accuracy by 100%.Created pipelines in ADF using Linked Services/Datasets/Pipeline to extract, transform, and load data from different sources like Blob Storage, Azure SQL Data Warehouse, write-back tool, and backward.Automated ELT processes, via email to destination using Azure Logic Apps and Data factory, resulting in translated data into actionable reports in Power BI.Generated and maintained scalable data pipelines and built new API integrations to support data volume and complexity, using Data Transformations on Azure Databricks.Collaborated with the analytics team to enhance business intelligence tools by 30%. Show less

• 

  Lentra

  Apr 2023 - Oct 2023

  Data Engineer

  Led the migration of legacy data infrastructure to Apache Pinot, enabling real-time data analytics and efficient querying capabilities.Utilised Bitbucket for version control, collaborating with the development team to manage code repositories and streamline the deployment process.Designed and implemented schema structures in Apache Pinot, ensuring optimal data storage, query performance, and scalability to meet analytical demands.

• 

  SpanIdea Systems

  Nov 2023 - now

  Senior Data Engineer

  Designed ETL/ELT pipelines using Azure Data Factory to ingest data from Workday into Azure data storage solutions such as Azure Data Lake and Azure SQL Database.Maintained Azure Data Lake Storage Gen2 architecture, enhancing data access speed by 30% for both structured and unstructured data.Implemented data integration workflows with Logic Apps, connecting different systems and automating business processes for seamless data flow.Build and maintained data models in Azure Synapse Analytics, creating fact and dimension tables to support HR metrics such as employee turnover, diversity, performance, and compensation analysis.Led the migration of HR analytics workloads, including Azure Synapse tables and stored procedures, to Databricks. Wrote PySpark code to replicate Synapse transformations and stored procedures within Databricks to take full advantage of the scalability and parallel processing power of the Databricks platform.Used Azure Databricks and PySpark to perform data cleaning, transformation, and aggregation of employee records, performance data, and HR survey results to generate insights and trends. Show less