Lucas Gomes

Experience

• 

  Thermon Mechanical Industry

  Aug 2019 - Feb 2020

  Data Engineer Intern

  Experience in network management, database management, IT resource maintenance, and VBA development for Microsoft Access applications with tools for environmental and quality management control. Additionally, I was responsible for generating reports using SQL queries on the company's database

• 

  Dti digital

  Mar 2021 - Jul 2022

  Participated in the architectural decisions and implementation of a comprehensive data lake infrastructure within AWS. By utilizing AWS Lambda, EMR with boto3, S3, Athena and Dremio. These resources were implemented using Terraform, ensuring consistent and automated provisioning. In this role, I worked mostly with two clouds: GCP and Azure. On GCP, I created pipelines with Apache Beam to process and clean data, feeding a Data Lake in Cloud Storage and BigQuery. I also developed data analysis and visualization using Data Studio. On Azure, I used Azure Resource Manager for IaC and CI/CD to create data pipelines with Synapse Dataflow and Apache Spark Pools. I actively contributed to dataOps processes.

  • Data Engineer

    Apr 2022 - Jul 2022

  • Data Engineer Trainee

    Mar 2021 - Apr 2022
• 

  A3Data

  Jul 2022 - Nov 2023

  Data Engineer

  Creation and maintenance of a Data Lake on Azure, consuming data from various sources such as Athena, SQL Server, and media platforms like Google ADS, Google Analytics, Facebook ADS, and TikTok ADS. This was achieved through the use of APIs, Data Factory integrations, and the Airbyte tool. The data was processed using Data Factory and clusters powered by PySpark provisioned through Kubernetes. All of this was orchestrated by Airflow.

• 

  Inter

  Nov 2023 - now

  Data Engineer