Dhanush Shet KP

Experience

• 

  Hindustan Aeronautics Limited

  Jul 2018 - Jul 2018

  Intern at Hindustan aeronautics limited
• 

  Nineleaps

  Aug 2021 - now

  Client - Uber CloudLAKE Platform● Migrated data pipelines from PHX to PHXCLD, ensuring consistency and seamless integration.● Build and optimize data pipelines using Apache Spark, ensuring efficiency, scalability, and lowresource consumption.● Develop insightful and interactive dashboards using Tableau to provide business insights and trackkey metrics.● Ensure data quality by validating and troubleshooting issues within the pipeline, maintaining accuracyand integrity.● Debug and resolve pipeline errors, providing comprehensive solutions to prevent future occurrences.● Collaborate with cross-functional teams to gather requirements, translate them into technicalsolutions, and deliver robust data products. Show less Client- Uber FinanceSpark 3.0 Migration and Optimization:-Lead migration to Spark 3.0, focusing on making SQL operations faster.-Optimize pipelines to process data quicker and use resources better.-Ensure smooth transition from Hive to Spark, minimizing disruptions.Code Enhancement and Review:-Improve and clean up code to make it faster and easier to understand.-Review code carefully to ensure it follows the best practices.-Take advantage of Spark's latest features for better performance.Testing and Analysis:-Analyze data using Excel to find insights and report results.-Double-check data accuracy with both Excel and Spark queries for perfect results.Data Extraction and Processing:-Extract and process data from different sources efficiently.-Make sure all parts of the code are tested well for reliability.-Validate data accuracy using Excel and Spark's tools to ensure top quality. Show less Client - Uber Customer Obsession and CommOps Business Requirements● Worked on ETLs that run across the Petabyte scale. Created and managed 70+ tier 1 and tier 2 datasets which have 10B + events and and serving >1M queries to internal customers/services.● Optimize existing data pipelines to enhance efficiency and scalability, reducing runtime and resource utilization.● Utilized data pipeline and workflow management tools such as Airflow to orchestrate and optimize data workflows● Led initiatives to build and optimize 'big data' data pipelines, architectures, and datasets to improve efficiency and scalability● Implemented and managed stream-processing systems like Spark-Streaming for real-time data processing.● Track and analyze pipeline performance using scorecards and key metrics, providing insights and recommendations for optimization.● Document and maintain project artifacts, including technical specifications, design documents, and operational procedures. Show less ● Pipeline monitoring, finding the root cause and delivering updates based on the analysis and then fixing the issue.● Plugin development for data reliability, efficiency and quality.● Code optimisation, updating, modifying existing pipelines and plugins.● Use of the Python package for the testing, and Excel for the analysis.● Review code for quality and implement best practices● Writing testable code for optimal level of code coverage.● Responsible for optimizing the code wherever necessary and extracting the data from a wide variety of sources to ingest, evaluate, and perform different processes. Show less

  • Software Developer Engineer-1

    Apr 2024 - now

  • MTS-3

    Apr 2023 - Apr 2024

  • MTS-2

    Apr 2022 - Apr 2023

  • SDE-Intern

    Aug 2021 - Mar 2022