SkillShare – Writing production: Ready ETL pipelines in Python using Pandas

SkillShare – Writing production-ready ETL pipelines in Python using Pandas-SkilledHares
English | Size: 3.15 GB
Category: Python

This course will show each step to write an ETL pipeline in Python from scratch to production using the necessary tools such as Python 3.9, Jupyter Notebook, Git and Github, Visual Studio Code, Docker and Docker Hub and the Python packages Pandas, boto3, pyyaml, awscli, jupyter, pylint, moto, coverage and the memory-profiler.

Two different approaches how to code in the Data Engineering field will be introduced and applied – functional and object oriented programming.

Best practices in developing Python code will be introduced and applied:

design principles

clean coding

virtual environments

project/folder setup

configuration

logging

exeption handling

linting

dependency management

performance tuning with profiling

unit testing

integration testing

dockerization

What is the goal of this course?

In the course we are going to use the Xetra dataset. Xetra stands for Exchange Electronic Trading and it is the trading platform of the Deutsche Börse Group. This dataset is derived near-time on a minute-by-minute basis from Deutsche Börse’s trading system and saved in an AWS S3 bucket available to the public for free.

The ETL Pipeline we are going to create will extract the Xetra dataset from the AWS S3 source bucket on a scheduled basis, create a report using transformations and load the transformed data to another AWS S3 target bucket.

The pipeline will be written in a way that it can be deployed easily to almost any production environment that can handle containerized applications. The production environment we are going to write the ETL pipeline for consists of a GitHub Code repository, a DockerHub Image Repository, an execution platform such as Kubernetes and an Orchestration tool such as the container-native Kubernetes workflow engine Argo Workflows or Apache Airflow.

So what can you expect in the course?

You will receive primarily practical interactive lessons where you have to code and implement the pipeline and theory lessons when needed. Furthermore you will get the python code for each lesson in the course material, the whole project on GitHub and the ready to use docker image with the application code on Docker Hub.

There will be power point slides for download for each theoretical lesson and useful links for each topic and step where you find more information and can even dive deeper.

Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares

Title: Writing production-ready ETL pipelines in Python using Pandas
Publisher: Skillshare
Category: Technology
Size: 3227M
Files: 10F
Date: 2021-07-16
Course #: 395128062
Published: Skillshare
Updated: N/A
Author: Jan Schwarzlose
Duration: 6h 47m
Exer/Code:

Installation:
Unpack that shit, run that shit

Description:
Insert generic comment here

NOTE: No subtitles were available at time of packing

NITROFLARE
nitro.download/view/391E21B98F316C3/Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares.part1.rar
nitro.download/view/64FBFE6646588E6/Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares.part2.rar
nitro.download/view/D06CB745934BC2A/Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares.part3.rar
nitro.download/view/B9F373EF3030332/Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares.part4.rar
nitro.download/view/1C135D1C5DDCEB1/Skillshare.Writing.production-ready.ETL.pipelines.in.Python.using.Pandas-SkilledHares.part5.rar