Shivani Bommidi

Experience

• 

  Capgemini

  Jul 2018 - May 2022

  • Developed Data Ingestion Airflow DAGs that extract data from the open-source systems and load them as parquet files into AWS S3 Data Lakes.• Designed Apache Airflow DAGs to efficiently extract real-time data in JSON format on an hourly basis, seamlessly loading it into AWS S3 Data Marts.• Analysed business mapping document of Consumer Banking Data and prepared metadata sheets for seamless integration into the data lineage tool Collibra.• Developed Unix scripts, leveraging AWS CLI commands, to automate the transfer of files from the staging server to AWS S3 by reducing 40% of the manual effort.• Enhanced dataset functionality by 60% within Hive by incorporating a new field with null data, utilizing PySpark commands.• Responsible for requirements gathering, analysis, review, and refinement of user stories that are tracked in Jira for business enhancement initiatives adhering to Agile Methodology. Show less

  • Consultant - Data Engineer

    Jul 2021 - May 2022

  • Associate Consultant

    Oct 2019 - Aug 2021

  • Senior Analyst

    Jul 2018 - Oct 2019
• 

  DBS Bank

  May 2022 - Jul 2023

  Data Engineer

  • Automated ETL pipelines in Airflow using complex Spark SQL queries and Spark optimization techniques with large runtimes improving performance by up to 70%.• Developed data pipelines utilizing SQL transformations on > 500 TB of structured data which transforms and loads data into Data marts.• Reduced manual effort by 60% by developing Unix shell scripts that transfer data files from AWS S3 to the User staging servers.• Efficiently deployed and automated Airflow jobs and Linux scripts using CI/CD deployment process.• Supervised and coached a team of 3 resources to ensure a comprehensive understanding of job responsibilities. Show less

• 

  University of Cincinnati

  Aug 2023 - Sept 2024

  Graduate Student

  RELEVANT COURSEWORK :Linear Regression, Big Data Integration, Optimization, Data Mining, Simulation Modelling, Data Visualization, Data Wrangling, Data Management, Data Visualization, Intelligent Data Analysis, Applied Statistical Methods, Graduate Case Studies in Business Analytics, Machine Learning DesignACADEMIC PROJECTS :Developed a Supervised Learning model on 1 million records of aircraft status data to predict the status of the aircraft. Performed Hyperparameter tuning and SHAP Analysis to tune the model.Skills Used: Python, Pandas, Scikit-learn, SHAP Analysis. Show less

• 

  ServiceNow

  Sept 2024 - now

  Senior Data Governance Engineer