Building a #serverless Data Pipeline with #aws S3 Lamba and DynamoDB
▻https://hackernoon.com/build-a-serverless-data-pipeline-with-aws-s3-lamba-and-dynamodb-5ecb8c3e
AWS Lambda plus Layers is one of the best solutions for managing a data pipeline and for implementing a serverless architecture. This post shows how to build a simple data pipeline using AWS Lambda Functions, S3 and DynamoDB.What this pipeline accomplishes?Every day an external datasource exports data to S3 and imports to AWS DynamoDB table.PrerequisitesServerless frameworkPython3.6PandasdockerHow this pipeline worksOn a daily basis, an external data source exports data of the pervious day in csv format to an S3 bucket. S3 event triggers an AWS Lambda Functions that do ETL process and save the data to DynamoDB.Install Serverless FrameworkBefore getting started, Install the Serverless Framework. Open up a terminal and type npm install -g serverless to install Serverless framework.Create (...)