Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement correlated random number generation #1069

Merged
merged 40 commits into from
Oct 3, 2022
Merged

Implement correlated random number generation #1069

Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
eefce12
[gulpy] first implementation
mtazzari Jul 5, 2022
a8f89de
[gulpy] implementing correlated rng
mtazzari Jul 5, 2022
d8215d8
[wip] implementing correlated rng
mtazzari Jul 6, 2022
614dd3d
[wip]
mtazzari Jul 12, 2022
d407089
[gulpy] working implementation of the correlated random values
mtazzari Jul 15, 2022
579910b
Merge branch 'develop' into feature/correlated_rng
mtazzari Jul 15, 2022
5417f9b
minor cleanup
mtazzari Jul 15, 2022
5ad10f9
[gulpy] Update docstrings for random module functions
mtazzari Jul 22, 2022
4f564a1
Merge branch 'develop' into feature/correlated_rng
mtazzari Aug 1, 2022
2066aa2
[gulpy] remove unused generate_correlated_hash
mtazzari Aug 2, 2022
f0311c0
[gulpy] introduce --ignore-correlation flag
mtazzari Aug 2, 2022
1709cee
set hashed_group_id to True by default, cleanup
mtazzari Aug 3, 2022
2222be2
adding haahing patch
maxwellflitton Aug 8, 2022
2c0d5e3
adding haahing patch
maxwellflitton Aug 8, 2022
6621208
Merge branch 'develop' into hashing-investigation
mtazzari Aug 10, 2022
e8cf544
Merge branch 'develop' into feature/correlated_rng
mtazzari Aug 11, 2022
bb05858
[gulpy] minor cleanup files.py parameter on same line
mtazzari Aug 11, 2022
e593f0c
[gulpy] run correlation only if rho>0
mtazzari Aug 11, 2022
fbf1689
updating hashing
maxwellflitton Aug 11, 2022
d611869
[gulpy] improve flow depending on corr definitions
mtazzari Aug 12, 2022
82fb79c
Disable GroupID hashing for acceptance tests (#1094)
sambles Aug 12, 2022
fee427e
Update group_id_cols default in get_gul_input_items
mtazzari Aug 12, 2022
45f8779
Hashing investigation (#1096)
maxwellflitton Aug 12, 2022
461621f
[gul_inputs] bugfix don't modify inplace
mtazzari Aug 12, 2022
77245a3
Update test_summaries.py to not rely on "loc_id" as default for group…
sambles Aug 12, 2022
bf500be
Always create a correlations.bin, if missing model_settings file is b…
sambles Aug 23, 2022
fd210e2
Merge branch 'develop' of https://github.com/OasisLMF/OasisLMF into h…
maxwellflitton Aug 24, 2022
2b3d202
adding peril_correlation_group for valid_oasis_group_cols
maxwellflitton Sep 5, 2022
f7cb1ab
adding peril_correlation_group for valid_oasis_group_cols
maxwellflitton Sep 5, 2022
7d772fb
appending peril_correlation_group to columns if correlations group is…
maxwellflitton Sep 5, 2022
e6ac89e
adding peril_correlation_group column to hashing of group IDs if corr…
maxwellflitton Sep 7, 2022
823add8
updating hashing group ID
maxwellflitton Sep 15, 2022
ff9f568
updating to accomodate non-correlations
maxwellflitton Sep 20, 2022
32a5f66
fixxing run
maxwellflitton Sep 21, 2022
d60deb2
fixing empty correlations df write header if empty correlations
maxwellflitton Sep 21, 2022
1fc5651
Merge branch 'develop' into feature/correlated_rng
sambles Oct 3, 2022
e95dac6
Remove empty file
sambles Oct 3, 2022
62c805f
Add missing defaults to get_gul_input_items (backwards compatible)
sambles Oct 3, 2022
5f6933a
Fix Group_id valid column check
sambles Oct 3, 2022
44e43f2
Force retest
sambles Oct 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Hashing investigation (#1096) 
* adding haahing patch

* adding haahing patch

* updating hashing

* Update oasislmf/preparation/gul_inputs.py

Co-authored-by: Marco Tazzari <6020226+mtazzari@users.noreply.github.com>
  • Loading branch information
@maxwellflitton @mtazzari
maxwellflitton and mtazzari authored Aug 12, 2022
commit 45f8779f2b58ec38294f39006e19c18693582b77
4 changes: 2 additions & 2 deletions oasislmf/preparation/gul_inputs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ def get_gul_input_items(

# Concatenate chunks. Sort by index to preserve item_id order in generated outputs compared
# to original code.
gul_inputs_df = pd.concat(gul_inputs_reformatted_chunks).sort_index().reset_index()
gul_inputs_df = pd.concat(gul_inputs_reformatted_chunks).sort_index().reset_index(drop=True, inplace=True)
# Set default values and data types for BI coverage boolean, TIV, deductibles and limit
dtypes = {
**{t: 'uint8' for t in term_cols_ints + terms_ints},
Expand Down Expand Up @@ -333,7 +333,7 @@ def get_gul_input_items(

# this block gets fired if the hashed_group_id is True
else:
gul_inputs_df["group_id"] = (pd.util.hash_pandas_object(gul_inputs_df[group_id_cols]).to_numpy() >> 33)
gul_inputs_df["group_id"] = (pd.util.hash_pandas_object(gul_inputs_df[group_id_cols], index=False).to_numpy() >> 33)

gul_inputs_df['group_id'] = gul_inputs_df['group_id'].astype('uint32')

Expand Down