Debanjan Sarkar

Experience

• 

  Celebal Technologies

  Feb 2023 - now

  · ETL Migration Project- MASAN group of companies:- As an ETL developer, responsible for designing Full Load and Incremental load pipelines to migrate data of 12+ SKUs from GCS, Google Bigquery to Azure Data Lake Storage Gen2, on daily basis.- Designed process log table on Azure SQL Db instance to monitor the health and status of each pipeline run.- Implemented error handling mechanism to ensure for any skipped or failed pipeline run case, the data migration left incomplete is automatically handled at next pipeline run.- Setting up of pipeline failure email alerts using logic apps, and row count validation for daily migration.· Copilot-like Code generator based on AI agents – Internal Project:- Created a copilot-like code generator Azure OpenAI model, Microsoft Autogen framework and Langchain, that can take natural language prompt, or function prototype, and writes an entire working code.- Used multiple specialize agent architecture, with each performing task such as writing code, executing generated code, reading error messages and output to loopback to code writer agent. Show less · Analyzed customer data based on metrics related to YoY revenue growth, profit margin decrement due to discounts. Performed data enrichment by conversion of file formats, adding derived hash columns and perform deduplication of client's product sales bronze layer data using Data Flows transformation in ADF.· Designed Data Models for Data warehousing, that involved dimensional modelling, normalization of tables, and implemented SCD Type 1 and Type 2 on related dimensions. Implemented optimized logic for lineage table population that tracks the metrics of ETL full load and incremental load of silver layer data.· Written Stored Procedures with improved logic, that handles Full Load and Incremental load, with the Dimension loading done using Inner Join, that reduces the time taken by entire ETL loading process by 18%. Show less

  • Associate

    Feb 2024 - now

  • Junior Associate

    Jul 2023 - Jan 2024

  • Junior Associate

    Feb 2023 - Jul 2023