Experience

• 

  Hsbc

  Mar 2021 - Dec 2022

  Data engineer

  • Strong experience working in Agile development environments.• Designed and developed ETL workflows using Databricks and Azure Data Factory to process structured and semi-structured data.• Optimized Apache Spark jobs, reducing processing time by 25% and improving resource utilization.• Built real-time and batch processing pipelines, enabling low-latency analytics for business intelligence teams.• Implemented data governance best practices, ensuring compliance with industry regulations.• Automated data cleansing and quality checks, enhancing data reliability for downstream applications.• Conducted performance tuning for SQL-based transformations, reducing query execution time. Show less

• 

  Elevance health

  Jan 2024 - now

  Data engineer

  Developed and optimized ETL pipelines using Databricks and Azure Data Factory, efficiently processing large-scale datasets while implementing data transformation logic in PySpark, reducing pipeline execution time by 30%. Designed and maintained Delta Lake architecture to ensure data consistency and incremental processing, while automating data validation and integrity checks using SQL and PySpark, significantly improving data accuracy. Led a large-scale data migration project, seamlessly transferring terabytes of records with Databricks and Delta Lake. Designed optimized data partitioning strategies, enhancing query performance and storage efficiency, and built real-time data ingestion pipelines using Azure Data Factory and Databricks, reducing data latency for analytics teams. Optimized Apache Spark processing jobs for high throughput and low latency, while implementing data quality and consistency checks to ensure zero data loss during processing. Collaborated with cross-functional teams to enhance data pipeline performance and scalability, and implemented partitioning and indexing strategies in Azure Synapse Analytics, significantly improving query execution time. Passionate about data engineering, ETL automation, and performance optimization, I continuously strive to build scalable, efficient, and high-performing data solutions. Show less