Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Linopy to PyPSA-Eur #448

Closed
wants to merge 33 commits into from
Closed

Add Linopy to PyPSA-Eur #448

wants to merge 33 commits into from

Conversation

pz-max
Copy link
Collaborator

@pz-max pz-max commented Dec 7, 2022 •
edited
Loading

Changes proposed in this Pull Request

Provide new solve_networks option linopy=True, which uses Linopy as a backend. Many constraints had to be rewritten due to the xarray concept in Linopy. All constraints are tested against the previous implementation.

Test status per constraint for single country ["BE"] :

  • add_battery_constraints ✅ . Same objective as previous pypsa-eur implementation.
  • add_operational_reserve_margin ✅ . Tested and results in same objective as previous pypsa-eur implementation: 1.849977447e+09. The test can be reproduced by adding the following lines to the tutorial config.yaml:
 operational_reserve:
    activate: true
    epsilon_load: 0.02
    epsilon_vres: 0.02
    contingency: 400000
  • add_BAU_constraints ✅ . Tested and results in the same objective result as the previous implementation: 6.8079564e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:
scenario:
  opts: [Co2L-BAU-24H]
  
electricity:
  voltages: [220., 300., 380.]
  co2limit: 100.e+6
  BAU_mincapacities:
    solar: 10000
    onwind: 0
    OCGT: 100000
    offwind-ac: 0
    offwind-dc: 0
  • add_SAFE_constraints ✅ . Tested and results in the same objective result as the previous implementation: 3.4869299e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:
scenario:
  opts: [Co2L-SAFE-24H]
  
electricity:
  SAFE_reservemargin: 0.5
  • add_EQ_constraints ✅ . Tested and provides the same objective result as the previous implementation: 3.1334..e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:
scenario:
  opts: [Co2L-EQ0.7-24H]
  • add_CCL_constraints ✅ . Tested for one country. Provides the same objective result as the previous implementation: 3.1453661e+08. The test can be reproduced by adding the following lines to the tutorial config.yaml:
scenario:
  opts: [Co2L-CCL-24H]
electricity:
  agg_p_nom_limits: data/agg_p_nom_minmax.csv

and with the following data in agg_p_nom_minmax.csv

country,carrier,min,max
BE,onwind,10000,
BE,offwind-ac,,
BE,offwind-dc,,
BE,solar,20000,

Test status per constraint for single country ["BE", "DE"] :

  • add_CCL_constraints
  • add_battery_constraints
  • add_operational_reserve_margin
  • add_BAU_constraints
  • add_SAFE_constraints
  • add_EQ_constraints

Checklist

  • I tested my contribution locally and it seems to work fine.
  • Code and workflow changes are sufficiently documented.
  • A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

@pz-max pz-max changed the title Add Linopy to PyPSA-Eur Add Linopy to PyPSA-Eur with compat.py Dec 10, 2022
@pz-max
Copy link
Collaborator Author

pz-max commented Dec 10, 2022 •
edited
Loading

One observed issue is that PyPSA renames generators that are extendable to generator-ext and non-extendable generators as generator-fix. Xarray cannot easily deal with it when performing functions e.g. groupby along carriers. The reason PyPSA renames the arrays in that way is its performance. Not writing np.nan for expansion problems for non-extendable generators increases speed.

What can be investigated:

  • Does the compat.py work without any issues when e.g. -ext stuff is disregarded? (could help to make Linopy compat.py interface smoother). If the test demonstrates better usability it might make sense to reconsider some of the functions which use -fix and -ext syntax in pypsa/optimization
  • How much performance drop do we experience when disregarding this -ext stuff in PyPSA (Tip from Fabian H. to change this: "you can go in pypsa/descriptors.py and reset the index naming of get_extendables_i to the c, same with get_non_extendables_i and committables")
  • Does it help performance using instead of np.nan (float 64) -> pd.NaN (int 8)
  • Does it help usability to define extra_functionality constraints in PyPSA-Eur by Linopy directly vs compat.py?

Update:

  • I am ignoring most of the above
  • -ext is probably not an issue. Xarray just requires clear dimensions, coords and naming and data. RHS or LHS inputs sometimes require .rename() or redefining the axis .rename_axis()
  • Currently, I am disregarding the compat.py avoiding an extra interface layer

@pz-max pz-max changed the title Add Linopy to PyPSA-Eur with compat.py Add Linopy to PyPSA-Eur Dec 12, 2022
pre-commit-ci bot and others added 12 commits December 12, 2022 23:14
@pz-max
Copy link
Collaborator Author

pz-max commented Dec 18, 2022

FYI @FabianHofmann, I made all PyPSA-Eur functions run for a single country ["BE"].

I noted that the CCL constraint is a bit more tricky. Because there is a groupby with multi-index, which groupby_sum in Linopy does not support yet:

pypsa-eur/scripts/solve_network.py

Lines 167 to 178 in a22dc92

p_nom_per_cc = (
pd.DataFrame(
{
"p_nom": linexpr((1, get_var(n, "Generator", "p_nom"))),
"country": gen_country,
"carrier": n.generators.carrier,
}
)
.dropna(subset=["p_nom"])
.groupby(["country", "carrier"])
.p_nom.apply(join_exprs)
)
. The current code hacks around and makes it run for one country. It also seems that groupby in xarray with multi-index is not really supported/adequately addressed.

My code 100% breaks if it is applied to multiple countries e.g. ["BE", "DE"] for the CCL constraint.
I will test all other constraints to see if ["BE", "DE"] works before going back to the CCL stuff.

@pz-max
Copy link
Collaborator Author

pz-max commented Dec 19, 2022

Only CCL constraint breaks now for multiple countries. If the 'groupsum()' for the multiindex would work this could be solved with ease. As alternative,

@pz-max pz-max marked this pull request as ready for review December 20, 2022 18:06
@pz-max
Copy link
Collaborator Author

pz-max commented Dec 20, 2022

Thanks to @FabianHofmann for the little co-hacking session for the CCL constraint. Everything is now fixed and tested. As of now, Fabian H. will eventually review this PR in the next 1-2 weeks.

Open review questions:

  • Any code changes?
  • Deprecation status on Linopy==False implementation?
  • Any extra documentation necessary?

@pz-max pz-max mentioned this pull request Mar 9, 2023
Merged
@pz-max
Copy link
Collaborator Author

pz-max commented Mar 18, 2023

Closing because of reimplementation in #625

@pz-max pz-max closed this Mar 18, 2023
@pz-max pz-max deleted the linopy branch March 18, 2023 12:50
@fneum
Copy link
Member

fneum commented Mar 18, 2023

Thanks, @pz-max! @FabianHofmann used your branch as a starting point (preserving commits) and said your contribution made it easy to implement the switch to linopy!

@pz-max
Copy link
Collaborator Author

pz-max commented Mar 18, 2023

Yep @fneum, I was with Igor and FabianH when we moved the Linopy stuff to the merge branch :)
Congrats on the merge 🎉 🎉 🎉

@pz-max pz-max mentioned this pull request Jun 12, 2023
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FabianHofmann FabianHofmann Awaiting requested review from FabianHofmann

@fneum fneum Awaiting requested review from fneum

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pz-max @fneum