Vikash Kushwaha

Experience

• 

  Celebal Technologies

  Feb 2023 - Dec 2023

  Consolidated data from various sources, including SFTP server and ADLS, into multiple files. Integrated and joined data from these sources to create a unified dataset, which was subsequently loaded into the Databricks staging layer.Implemented delta tables for optimizing data processing efficiency.Conducted comprehensive data quality checks on all tables using PySpark and Spark SQL to ensure the integrity of the data.In alignment with business requirements, reprocessed any corrupted records. Leveraging Azure Databricks, Azure Data Factory, and Azure Data Lake Storage, I integrated data from ADLS and SFTP into cohesive files. Employing Databricks, Spark SQL, and PySpark, I engineered a unified dataset with delta tables for efficient processing. Rigorous inspections guaranteed data accuracy, with damaged files reprocessed as per business needs. This practical experience addressed real-world data challenges, strengthening proficiency in ADF and Azure services. Analyzing a substantial dataset of over 270,000 NYC taxi records using Python, PySpark, SQL, and Azure Databricks, I addressed challenges like inconsistent data and missing information. Through effective SQL queries, I correlated factors such as distance, time, and traffic conditions, deriving valuable conclusions for optimizing ride predictions and service efficiency from real-world NYC taxi data.

  • Associate Data Engineer

    Feb 2023 - Dec 2023

  • Trainee

    Jun 2023 - Sept 2023

  • Intern

    Feb 2023 - May 2023
• 

  Fractal

  Mar 2024 - now

  Associate