Python for Data Engineers

Job ads ask for Python. This is the part a data engineer uses every day.

Python is a large language, and a data engineer uses a small part of it on an ordinary working day. This book teaches that part, in the order a pipeline needs it, and keeps it short. Twenty-nine diagrams draw the ideas that have a shape: a record and a batch, a request and a retry, a transaction, an event loop. The text walks you through each one, a step at a time, and every term is defined the first time it appears.

One running example grows from the first script to the last chapter: a small bike-share scheme whose trip files land every night with a few bad rows in them. You read the files, check every row, set the bad ones aside with a reason, load the rest into a database without duplicates, test the pipeline with pytest and run it all as one nightly job. Every Python listing was run on Python 3.14.7, and wherever one prints something, its real output appears beneath it, so you can check your counts and totals against the page.

Inside, you will learn how to:

• set up a project with uv, run a script and read a traceback from the bottom up
• work with values, lists, dictionaries, loops and functions, using rows from the running example
• read CSV and JSON files whole or one row at a time
• handle errors, write a log and keep a rejects file with a reason for every bad row
• write classes, call a paginated API with retries and validate rows with pydantic
• summarise with pandas, write Apache Parquet files and load SQLite so a second run never doubles the data
• fetch faster with threads and asyncio, and test the pipeline with pytest
• build a capstone pipeline that runs every stage from end to end

The book ends with 45 practice tasks with worked answers and 40 interview questions with model answers, plus a glossary and a quick reference.

Written for the analyst or graduate who writes some SQL, has met Python in one tutorial or not at all, and wants a first data engineering job. No Python is assumed.

Python is a registered trademark of the Python Software Foundation. This book is not affiliated with or endorsed by the Foundation.