The objective of this scenario is to build an app that shows the relationship between Natural Gas prices in the U.S. and the temperature. The app uses VDK to ingest and process the following two CSV datasets:

• Natural Gas Prices from 1997 to 2020
• U.S. Climate Extremes Index from 1910 to 2020

This example provides the complete code to perform the following tasks:

• Data Ingestion from CSV
• Data Processing
• Data Publication in form of an innteractive App

The directory is organized as follows:

• energy - contains jobs
• dashboard.py - contains the code to generate the app

An American natural gas supplier has hired a team of data scientists to find out if there is a correlation between price trends and climatic conditions (minimum temperatures). Analysts found two datasets:

• Natural Gas Prices from 1997 to 2020
• U.S. Climate Extremes Index from 1910 to 2020

The Natural Gas Prices dataset contains 5,953 records, provided by the U.S. Energy Information Administration(EIA). Each record contains the daily price of natural gas. For each record, the following information is available:

• Date (provided on a daily basis)
• Price (in dollars)

The following table shows an excerpt of the dataset:

The U.S. Climate Extremes Index dataset contains 112 records. For each record, many parameters can be downloaded. Data Analysts select the following information:

• Date (provided on an annual basis)
• Percentage of minimum temperature much above normal
• Percentage of minimum temperature much below normal.

The following table shows an excerpt of the dataset:

To run this example, you need:

• Versatile Data Kit
• Trino DB
• Versatile Data Kit plugin for Trino
• Streamlit

For more details on how to install Versatile Data Kit, Trino DB, Streamlit and Versatile Data Kit plugin for Trino, please refer to this link.

This example also requires the following Python libraries, which are included in the requirement.txt file:

pandas

The following example uses the same configuration as the Life Expectancy scenario, with the only difference on the trino schema used by the Versatile Data Kit configuration file:

trino_schema = energy

Data Ingestion uploads in the database output of CSV files, defined in the Data Source section. Data Ingestion is performed through the following steps:

• delete the existing table (if any)
• create a new table
• ingest table values directly from the CSV.

To access the CSV file, this example requires an active Internet connection to work properly.

Jobs 01-03 are devoted to Data Ingestion of the Natural Gas Prices dataset. The output of the CSV file is imported in a tables, named nataural_gas_prices.

Jobs 04-06 are devoted to Data Ingestion of the U.S. Climate Extremes Index dataset. The output of the CSV file is imported in a tables, named climate_extremes_index.

Data Processing includes two steps:

• building an annual view of the natural_gas_prices with the average value of natural gas price
• merging the two tables, the previous one and climate_extremes_index. Values are normalized in the interval [0,1] thus making possible a comparison among them.

Jobs 07-08 are devoted to the first step. The produced output is stored in a table, called average_gas_price_by_year.

The following table shows an example of the produced table:

Jobs 09-10 are devoted to the second step. The produced output is stored in a table, called merged_tables.

The following table shows an example of the produced table:

The dashboard.py script contains an app on the Online Exhibition, showing the preview of all the artworks contained in the database. To run the app, simply run the following command:

streamlit run dashboard.py