Sunit Shrivastava

Experience

• 

  Tech Mahindra

  Feb 2006 - Jul 2016

  Technical Archtect

  I am working as technical architect and handling the responsibility of interacting with client, dev team for requirement gathering, providing high level design solutions with AT&T client.

• 

  Amdocs

  Jun 2016 - now

  This project is in the domain of telecom to migrate on-prem oracle based data/processes to azure on-cloud system. It involved performing POCs to evaluate best possible technologies for the migration of oracle based data to cloud and then subsequently implementing the approved approach in productionIt involved latest cutting-edge technologies like oracle golden gate(CDC), Azure event hub, Spark structured streaming, Apache Databricks, Python, Apache NiFi and Apache Spark. Project architecture included designing and implementing real time/batch data pipeline to migrate on-prem oracle based data/processes to azure cloud. Oracle table transactions were captured through oracle golden gate CDC technology/replicats which moved data to azure event hubs. Afterwards, spark structured streaming jobs based on python were designed/scheduled in Databricks to ingest this data into staging area based on azure adlsgen2 storage. Finally, data was merged from this staging area to Databricks delta lake tables based on again adlsgen2 storage. Batch data based on file source was pulled through Apache Nifi. TWS jobs were used for scheduling of jobs. After getting all this data into Databricks delta tables, extracts were prepared and sent to downstream systems on daily/weekly/monthly basis as per the requirements.Responsibilities:• Collecting requirements• Performing POCs to select best tools/technologies for implementing pipeline• Requirement analysis and design• Designing/Implementing ingestion/Extraction pipeline • Unit/pre-prod testing• Fixing any issues reported in dev and prod env.Technologies: oracle golden gate(CDC), Azure event hub, Spark structured streaming, Apache Databricks, Python, Apache NiFi and Apache Spark Show less

  • Technical Architect

    Jan 2021 - now

  • Software Architect

    Jan 2021 - now

  • Technical Architect

    Jan 2021 - now

  • Technical Architect

    Jun 2016 - now
• 

  Syntel Inc

  Aug 2016 - Jun 2018

  Technical Architect

  I worked as technical architect for AMEX and Well Fargo. Main responsibilities included working as architect for data lake based migration for wells fargo application, performing evaluations of various technologies in data lake like no-sql databases, streaming solutions and OLTP, OLAP based analysis. I had exposure of working in MapR distribution in MapRDB and MapRFS to capture real time data in MapRDB and historic bulk data in MapRFS.

• 

  Tescra In

  Jun 2018 - Dec 2020

  Software Architect

  This project is in the domain of advertisement revenue related analytics for clients of AT&T/Xander.It involves egressing business data for the business info provided by clients. It involves latest cutting-edge technologies like amazon cloud, spark scala, snowflake, terraform, docker, spring boot etc.It involved creating a generic framework based on spark scala to parse and process json based request data and generate data for end users based on spark jobs. Users submit json requests to web part of the application based on spring boot which then forwards requests to AWS Lambda based scala framework. Lambda framework then prepares request for spark job and then submits to spark EMR cluster through Livy.It has provisions to send communications to the users based on email(AWS SES), notification(AWS SNS) and all success/error scenarios are notified to the users.It also stores the data in DynamoDB for tracking purpose.It included designing and implementing swagger based Restful API using spring boot and providing awsapplication load balancer based end point using target groups, autoscaling based ec2 instances. Clients send json request to this end point and then it comes to spring boot based restful API. Then, validation is performed and then awslambda based code is invoked which in turn submits spark jobs using apache livy. Spark job then reads data from another AWS account using cross account access and trust relationship. Then, spark job performs required logic and then writes output to s3.The project involved three inputs and three outputs –s3, presignedurl and snowflake. Show less