Experience

• 

  Educational Institution

  Jan 2022 - now

  Big data developer

  1- Build and manage systems that automatically move, change, and store data using tools like Spark, AWS Glue, and Redshift.2- Create and set up systems for storing and managing lots of data, aiming to make them scalable, reliable, and efficient to meet what the business needs.3- Set up a Data Catalog in AWS Glue to create metadata tables for datasets stored in S3.4- Use AWS Glue to create ETL (Extract, Transform, Load) jobs. These jobs define the steps to extract data from S3, apply transformations (such as cleaning or enriching the data), and load the transformed data back into S3 or another destination.5- Write transformation scripts using AWS Glue's built-in PySpark environment. These scripts implement the logic to clean, enrich, or transform the data according to requirements.6- Schedule the ETL jobs to run at regular intervals or in response to specific events using AWS Glue's scheduling capabilities. This ensures that the data stays up-to-date and the processing is automated.7. In AWS Glue, scripts are written to load processed data from S3 into Redshift tables. These scripts commonly utilize SQL COPY commands or Redshift-specific APIs for efficient data loading. Show less