Aman Agrawal

Experience

• 

  HashedIn

  Jul 2021 - Jun 2023

  At my current organization I am tasked with the following responsibilities:1. Client Infrastructure Modernization to AWS.2. Provisioning all AWS resources using Cloudformation templates using Bitbucket and Bamboo data pipeline.3. Converting Cobol DB2 batch scripts business logics into data transformation rules in Pyspark running on AWS Glue ETL.4. Created one time data migration and validation scripts in Pyspark to migrate DB2 legacy tables to AWS Aurora Cluster.5. Orchestration of Glue Jobs using AWS Glue workflow, Triggers, AWS Step functions, Control-M etc.6. Configure FTP on EC2 instances.7. Load real-time streams into Aurora using AWS Kinesis and Kafka.8. Develop and maintain data pipelines to extract data from a variety of sources(DB2, Kafka etc.) and populate target database.9. Analyze Data using SQL.10. Setup splunk logging and create dashboards.11. Performing code and data analysis in local using Jupyter Notebook.12. Creating serverless functions for several utilities using AWS Lambda.13. Implementing data quality checks and perform unit test of pyspark scripts using python libraries like boto3, moto, mock, awsglue utils etc. running over a docker container in python venv and generating sonar coverage report. Show less

  • Data Engineer

    Sept 2021 - Jun 2023

  • Software Engineer Intern

    Jul 2021 - Sept 2021
• 

  MSD

  Jul 2023 - Jul 2024

  Data Engineer

  Building data pipelines, data migration, streaming pipelinesthrough AWS Glue & EMR on top of Apache Spark with AWSRedshift as target. Develop DBT scripts for business rule transformations running over AWS Fargate. Using AWS lambda for handling real-time events.Writing Docker script for containerization of python code forrunning dbt-core in on AWS using ECS Fargate. Build images using Sagemaker Notebooks and storing in AWS ECR. Deploy images from AWS ECR over Fargate.Implementing Data Quality using Deequ framework that runs over glue job and generate audit results into s3 that can be queriedusing athena. Building alert and monitoring framework for data pipelines usingairflow. Show less

• 

  Fusemachines

  Jul 2024 - Feb 2025

  Data Engineer
• 

  Ameriprise Financial Services, LLC

  Feb 2025 - now

  Data Engineer