Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Scenario.to_csv() method #300

Open
volker-krey opened this issue Mar 31, 2020 · 2 comments
Open

Add Scenario.to_csv() method #300

volker-krey opened this issue Mar 31, 2020 · 2 comments
Labels
enh New features & functionality help welcome

Comments

@volker-krey
Copy link

volker-krey commented Mar 31, 2020

The to_excel method of the message_ix.Scenario class suffers from size limitations of individual sheets in the xlsx format (maximum of 1048576 rows) when used with large models such as the global MESSAGEix-GLOBIOM model instance. To overcome this limitation… (this limitation, #292, was removed by #309 and #345)

It is proposed to add a to.csv() method that exports all sets and parameters to csv files and combines them in a zip archive. Below is a code snippet that exports all parameters of a model instance to csv files and can thus serve as a starting point for such a method.

# get list of parameters in model
parameter = scen.par_list()
for par in parameter:
    # read demand parameter into dataframe
    df = scen.par(par)
    # write demand dataframe to csv file
    df.to_csv('%s.csv' % par)
@khaeru khaeru transferred this issue from iiasa/message_ix Apr 1, 2020
@khaeru khaeru changed the title add to_csv() method to message_ix.Scenario class to overcome limitations of to_excel() Add to_csv() method to Scenario class to overcome limitations of to_excel() Apr 1, 2020
@khaeru
Copy link
Member

khaeru commented Apr 1, 2020

Related to #292.

@volker-krey
Copy link
Author

Here is more complete code that in principle does all that is needed from my perspective, but as I am not fluent in Python there are probably better ways of doing this.

 # import packages
import pandas as pd
import ixmp
import message_ix
import os
import zipfile
import shutil

# define model, scenario and zipfile name
model = 'SSP2_test'
scenario = 'baseline_DEFAULT_2020'
zipfilename = 'MESSAGEix-GLOBIOM_SSP2.zip'

# connect to platform and load scenario
mp = ixmp.Platform()
scen = message_ix.Scenario(mp, model=model, scenario=scenario)

# create temporary directory to write csv files into
tmppath = 'tmp'
if not os.path.exists(tmppath):
    os.makedirs(tmppath)
os.chdir(tmppath)
    
# check which sets exist
sets = scen.set_list()
# check which parameters exist
parameters = scen.par_list()    

# create zipfile and open for writing
message_zipfile = zipfile.ZipFile(zipfilename, mode='w', compression=zipfile.ZIP_DEFLATED)

# loop over sets, write to csv and to zip archive
for s in sets:
    # read set into dataframe
    df = scen.set(s)
    # write set dataframe to csv file 
    df.to_csv('set_%s.csv' % s, index = False)
    message_zipfile.write("set_%s.csv" % s)

# loop over parameters, write to csv and to zip archive
for p in parameters:
    # read parameter into dataframe
    df = scen.par(p)
    # write parameter dataframe to csv file 
    df.to_csv('par_%s.csv' % p, index = False)
    message_zipfile.write("par_%s.csv" % p)
    
# close zipfile
message_zipfile.close()

# move zipfile to parent directory 
os.rename(zipfilename, '../' + zipfilename)
os.chdir('..')

# delete temporary directory and files
shutil.rmtree(tmppath)

@khaeru khaeru added enh New features & functionality help welcome labels Apr 27, 2020
@khaeru khaeru changed the title Add to_csv() method to Scenario class to overcome limitations of to_excel() Add Scenario.to_csv() method Sep 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh New features & functionality help welcome
Projects
None yet
Development

No branches or pull requests

2 participants