Nandan Reddy

Experience

• 

  Knoah Solutions

  Aug 2017 - Oct 2019

  Big Data Developer

  • Developed and deployed scalable distributed data solutions within the Hadoop ecosystem, ensuring optimized resource utilization.• Programmed and executed MapReduce streaming jobs using Python, integrating these processes with Hive and Pig for data manipulation and analysis.• Implemented optimization strategies for MapReduce jobs to reduce data storage requirements on the Hadoop Distributed File System (HDFS).• Established Continuous Integration/Continuous Deployment (CI/CD) pipelines to facilitate DevOps practices, including source code management with Git, extensive unit testing, and automated deployment processes.• Utilized a range of DevOps tools including Jenkins, and Autosys scheduler to enhance development efficiency and deployment reliability.• Crafted complex SQL queries, procedures, and triggers using Relational Database Management Systems (RDBMS) like Oracle, MySQL, and PostgreSQL aimed at improving data access and processing.• Addressed data consistency issues in distributed environments by leveraging Kafka for effective message ordering and delivery.• Executed data warehousing solutions within the Hadoop ecosystem, developing optimized data storage and retrieval systems for complex analytics.• Built CI/CD pipelines to streamline deployment, improving code quality and reducing downtime.• Exhibited strong problem-solving skills in diagnosing and resolving complex data issues within intricate software architectures.• Managed Hadoop cluster configurations and maintenance, employing distributions such as Apache Hadoop and Cloudera, to facilitate scalable big data processing.• Applied Agile development methodologies throughout the project lifecycle, including story grooming, sprint planning, and daily stand-up meetings, to increase project agility and team productivity. Show less

• 

  PepsiCo

  Jun 2019 - Nov 2022

  Data Engineer

  • Implemented AWS Glue to ingest data efficiently from various source systems, including both relational and non-relational databases, meeting the requirements of both functional and business stakeholders.• Created a sophisticated Data Lake architecture within AWS S3.• Set up AWS Glue and AWS EMR for adaptive resource scaling to manage fluctuating data volumes effectively during peak operational times.• Employed AWS Auto Scaling for the real-time adjustment of processing units/nodes according to the current data processing demands.• Managed EMR clusters, notebooks, jobs, and implemented autoscaling features to ensure smooth data processing operations.• Successfully integrated data from multiple sources into AWS S3 through AWS Glue and AWS Lambda, enhancing data consolidation.• Leveraged the parallel processing power of AWS Glue and AWS EMR to ingest data from various sources simultaneously.• Performed ETL tasks within AWS Glue, utilizing JDBC connectors for integration with multiple relational database systems.• Configured numerous EMR clusters dedicated to batch processing and continuous streaming analysis, achieving optimal computation times and cost savings.• Continuously monitored, automated, and refined data engineering workflows to maintain efficiency and performance.• Developed Lambda functions to facilitate data transfer from SFTP locations directly into AWS S3, streamlining the data ingestion process.• Applied AWS Auto Scaling in conjunction with AWS Glue and EMR for dynamic scaling of resources based on operational demands. Show less

• 

  Discover Financial Services

  Mar 2023 - now

  Data Engineer

  • Designed and developed scalable and cost-effective architecture in AWS Big Data services for the data life cycle of collection, ingestion, storage, processing, and visualization• Consolidated disparate data sources into a centralized Amazon S3 data lake, providing a single source of truth for business reporting.• Involved in creating an End-to-End data pipeline within a distributed environment using Big data tools, Spark framework, and Tableau for data visualization.• Experience in creating Python topology scripts to generate cloud formation templates for creating the EMR cluster in AWS.• Automated data cleaning and transformation using AWS Glue, cutting manual processing time.• Implemented automated data validation checks using AWS Glue Data Brew, ensuring 99% data accuracy in reports.• Migrated legacy systems to Amazon Redshift, improving query performance by 3x and accelerating time-to-insight for business users.• Empowered teams with Amazon Athena and Quick Sight, allowing them to run custom queries and build dashboards independently.• Set up real-time alerts using Amazon CloudWatch and SNS to detect and resolve data pipeline issues before impacting critical business reports.• Designed a pipeline that scaled automatically using AWS Auto Scaling and S3 Intelligent Tiering, adapting seamlessly to growing data volumes.• Used AWS Cost Explorer and Trusted Advisor to identify cost-saving opportunities, achieving a 20% reduction in cloud expenses.• Implemented robust data security measures using AWS IAM and KMS, ensuring compliance with industry regulations.• Built a real-time data ingestion system using Amazon Kinesis, enabling business teams to monitor key performance indicators (KPIs) as they happen.• Enabled marketing teams to optimize campaigns by analyzing real-time customer behavior.• Partnered with data scientists, analysts, and business leaders to align with organizational goals. Show less