Dolly Lipare

Experience

• 

  Quantiphi

  Jan 2020 - Sept 2023

  1.Developed robust kafka to bigquery streaming dataflow pipelines . Accommodated various data formats like text, JSON, and Avro. Performed different types of testing and conducted data validation.2.Designed and developed batch ETL pipelines to process data coming from adhoc files and AWS Redshift, utilizing GCS buckets for intermediate storage and executing SQL transformations before loading into BigQuery. Orchestrated workflows using composer =>Achieved a significant performance improvement, reducing [Data_Load+ETL+Build of cube] time from 18+ hours to 40-50 minutes.] 3. Worked in a Hive to BigQuery Migration and responsible for the following tasks:- i. Resolving key hive blockers-Identifying solution in BQ for hive-specific functionality ii. Creation of composer dags for establishing connection to Dataproc VMs and running hive queries. iii. Conducting extensive testing of the connector, including complex data types, file formats, and functions, with performance testing on queries processing approximately 250 GBs of data. Show less 1. Designed and implemented the ETL/ELT pipelines to extract, process, and analyse large-scale COVID and Non-COVID datasets, including Geospatial data obtained from REST APIs. Orchestrated the pipelines using Airflow, dynamically scheduling workflows.2.Led data engineering for a pharmaceutical project focused on phenotype entity mapping, collaborating closely with a Solution Architect. Performed parquet files ingestion and built scalable and schema variant data pipelines. Leveraged Elasticsearch for full-text search capabilities and GCP services like Firestore, cloud functions, cloud storage.. Achieved a 50% reduction in mapping time compared to traditional method.3.Successfully migrated a massive data warehouse, encompassing 60+ TB of data and 800+ tables, from Teradata to BigQuery. Conducted thorough data validation in BigQuery to ensure accuracy and consistency. . Implemented BigQuery best practices, including partitioning and clustering, resulting in an impressive 65% improvement in query performance.4.Designed and developed ETL pipelines for an energy domain project, handling data from APIs and Excels. Orchestrated Excel data pipelines using Composer workflows for seamless integration. Utilized Cloud SQL tables for audit data storage and leveraged Cloud Logging for efficient monitoring and troubleshooting. Show less

  • Senior Data Engineer

    Apr 2022 - Sept 2023

  • Data Engineer

    Sept 2020 - May 2022

  • Data Engineer - Intern

    Jan 2020 - Aug 2020
• 

  Priceline

  Oct 2023 - now

  Data Engineer