From the course: Data Cleaning in Python Essential Training

Unlock the full course today

Join today to access over 23,400 courses taught by industry experts.

Transactions

Transactions

- There are many ways why I like relational databases. They are an old and established technology. There's a lot of knowledge and tooling around them. They provide a schema and you can query data with SQL, which a lot of people know. But one of the most important features when talking about data pipelines is transactions. A transaction mean either all of your changes go in or none of it. Say you insert data and suddenly there's an invalid record and you get an exception. Without a transaction, you need to manually figure out which records made it to the database and which ones you need to try again. Also, if you have a downstream system that process the data for a given day, how will it know that the upstream process is done inserting the data? With transaction, the data is either there or it's not. Let's have a look at some introductions. So we have our ship data with the name of the ship, latitude and…

Contents