Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing reference model components to delay_group_lmpf and generated quantities #147

Merged
merged 60 commits into from
Sep 20, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b58ecfe
add required delay_group_lmpf changes for missing ref model
seabbs Jul 31, 2022
3cadc35
debug to compilation
seabbs Jul 31, 2022
a004d8e
add missing reference model to gq
seabbs Jul 31, 2022
5dd2dbc
use segment where possible
seabbs Aug 1, 2022
b235579
add first draft of missing reference look-up
seabbs Aug 1, 2022
48acfdc
correct delay structure
seabbs Aug 1, 2022
9a47a13
model fitting
seabbs Aug 1, 2022
f172d8a
debug allocation of missing reference effects
seabbs Aug 1, 2022
c1e8396
model fitting but not recovering simulated proportion
seabbs Aug 1, 2022
28c47bd
update snaps for enw_missing
seabbs Aug 1, 2022
77cee2d
add plot
seabbs Aug 2, 2022
1856b89
make a output processing
seabbs Aug 2, 2022
a9f8e01
use the correct likelihood you tool
seabbs Aug 2, 2022
28a0d58
make example multi-threaded
seabbs Aug 2, 2022
ddee268
use the correct helper function (log1m_exp not log1m)
seabbs Aug 2, 2022
85a4617
add enw_incidence_to_cumulativ and update enw_new_reports to match
seabbs Aug 10, 2022
0223a5c
update nowcast date for missing example and clean up code
seabbs Aug 10, 2022
7bbd646
reset to same date as used in all other examples
seabbs Aug 10, 2022
3da960e
fix merge issues and turn off warning
seabbs Aug 10, 2022
f726334
use a fixed proportion missing
seabbs Aug 10, 2022
d5dbd24
explore example
seabbs Aug 11, 2022
09e5536
add enw_incidence_to_cumulative
seabbs Aug 11, 2022
c009112
add enw_incidence_to_cumulative
seabbs Aug 11, 2022
ac9c2dd
local CRAN check
seabbs Aug 11, 2022
6a7aac5
solve merge conflicts
seabbs Aug 11, 2022
7a00cbe
add internal helper functions for missing reference lookup
seabbs Aug 11, 2022
4aa7a18
add new global variables
seabbs Aug 11, 2022
be5bbe8
update wordlist
seabbs Aug 11, 2022
d9e8f4f
Merge branch 'develop' into feature-missing-reference-function
seabbs Aug 12, 2022
8a09ec7
add missing refeerence model definition
seabbs Aug 12, 2022
2d1964e
Merge branch 'feature-missing-reference-function' of https://github.c…
seabbs Aug 12, 2022
0d45c7b
write tests for enw_reps_with_complete_refs
seabbs Aug 16, 2022
8e7c1de
fix indexing bug with enw_reference_by_report
seabbs Aug 16, 2022
e77c079
merge develop
seabbs Aug 16, 2022
74ad2df
fix issues from #151 causing spurious warnings
seabbs Aug 16, 2022
8029a18
fix failing tests due to sorting standardisation
seabbs Aug 16, 2022
0b64683
debug ordering changes
seabbs Aug 16, 2022
caef167
more test fixes
seabbs Aug 16, 2022
83ef418
fix enw_complete_dates tests
seabbs Aug 17, 2022
bd0a430
add enw_simulate_missing_reference to package
seabbs Aug 17, 2022
b6cc38f
exporte enw_simulate_missing_reference
seabbs Aug 17, 2022
5808134
make cmdstanr tests skip locally
seabbs Aug 17, 2022
6a2ab6d
complete missing model convergence check
seabbs Aug 17, 2022
79154a5
update news and contribnuting
seabbs Aug 17, 2022
38ae0d0
typo in contributing
seabbs Aug 18, 2022
40af056
Change temp variable name
adrian-lison Aug 30, 2022
b890f81
Refactor filt_obs_indexed
adrian-lison Aug 30, 2022
1a7cb2c
Fix filt_obs_indexes
adrian-lison Aug 30, 2022
5231dec
Improve in-model code doc
adrian-lison Aug 31, 2022
a68bf80
Streamline time wording in in-model doc
adrian-lison Aug 31, 2022
3cbe6b4
Refactor variable names in delay_group_lpmf
adrian-lison Sep 1, 2022
6f3ff55
add localisation changes from main
seabbs Sep 1, 2022
da55c32
Merge branch 'develop' into feature-missing-reference-function
seabbs Sep 1, 2022
0e6a1c7
add usage warning for the missing data MVP
seabbs Sep 1, 2022
11d647b
update news
seabbs Sep 1, 2022
2686216
spelling and global variables
seabbs Sep 1, 2022
2ec8058
add handling of group-wise missing reference observations and look-ups
seabbs Sep 1, 2022
c3494e5
use filtered missing reference obs in likelihood
seabbs Sep 1, 2022
aeb7eef
add missing lookup variables
seabbs Sep 1, 2022
8bdf530
update test snapshot
seabbs Sep 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Streamline time wording in in-model doc
  • Loading branch information
adrian-lison committed Aug 31, 2022
commit a68bf807a2f059cfa8b3b9099a3650a45b044f64
4 changes: 2 additions & 2 deletions inst/stan/chunks/debug.stan
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
if (model_refp > 1) {
print(refp_sd[i]);
}
print("Unique report day hazards");
print("Unique reporting time hazards");
print(srdlh);
if (model_obs) {
print("Overdispersion");
Expand All @@ -38,6 +38,6 @@
j += is_inf(abs(srdlh[k])) ? 1 : 0;
}
if (j) {
print("Hazard effects on report date");
print("Hazard effects on reporting time");
print(srdlh);
}
30 changes: 15 additions & 15 deletions inst/stan/epinowcast.stan
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ data {
// Expectation model
array[2] real eobs_lsd_p; // standard deviation for expected final obs

// Reference day model
// Reference time model
int model_refp;
int refp_fnrow;
array[s] int refp_findex;
Expand All @@ -51,7 +51,7 @@ data {
array[2] real refp_mean_beta_sd_p;
array[2] real refp_sd_beta_sd_p;

// Reporting day model
// Reporting time model
int model_rep;
int rep_t; // total number of reporting times (t + dmax - 1)
int rep_fnrow;
Expand All @@ -62,7 +62,7 @@ data {
matrix[rep_fncol, rep_rncol + 1] rep_rdesign;
array[2] real rep_beta_sd_p;

// Missing reference date model
// Missing reference time model
int model_miss;
int miss_obs;
int miss_fnindex;
Expand Down Expand Up @@ -100,7 +100,7 @@ transformed data{
real logdmax = 5*log(dmax); // scaled maxmimum delay to log for crude bounds
// prior mean of cases based on thoose observed
vector[g] eobs_init = log(to_vector(latest_obs[1, 1:g]) + 1);
// if no reporting day effects use native probability for reference day
// if no reporting time effects use native probability for reference time
// effects, i.e. do not convert to logit hazard
int ref_as_p = (model_rep > 0 || model_refp == 0) ? 0 : 1;
// Type of likelihood aggregation to use
Expand All @@ -125,7 +125,7 @@ parameters {
vector[rep_fncol] rep_beta;
vector<lower=0>[rep_rncol] rep_beta_sd;

// Missing reference date model
// Missing reference time model
array[model_miss] real miss_int;
vector[miss_fncol] miss_beta;
vector<lower=0>[miss_rncol] miss_beta_sd;
Expand All @@ -142,7 +142,7 @@ transformed parameters{
vector[refp_fnrow] refp_sd;
matrix[dmax, refp_fnrow] ref_lh; // sparse report logit hazards
// Report model
vector[rep_fnrow] srdlh; // sparse report day logit hazards
vector[rep_fnrow] srdlh; // sparse reporting time logit hazards
// Missing model
vector[miss_fnindex] miss_ref_lprop;

Expand All @@ -160,10 +160,10 @@ transformed parameters{
}

// Reference model
profile("transformed_delay_reference_date_total") {
profile("transformed_delay_reference_time_total") {
if (model_refp) {
// calculate sparse reference date effects
profile("transformed_delay_reference_date_effects") {
// calculate sparse reference time effects
profile("transformed_delay_reference_time_effects") {
refp_mean = combine_effects(refp_mean_int[1], refp_mean_beta, refp_fdesign,
refp_mean_beta_sd, refp_rdesign, 1);
if (model_refp > 1) {
Expand All @@ -172,8 +172,8 @@ transformed parameters{
refp_sd = exp(refp_sd);
}
}
// calculate reference date logit hazards (unless no reporting effects)
profile("transformed_delay_reference_date_hazards") {
// calculate reference time logit hazards (unless no reporting effects)
profile("transformed_delay_reference_time_hazards") {
for (i in 1:refp_fnrow) {
ref_lh[, i] = discretised_logit_hazard(
refp_mean[i], refp_sd[i], dmax, model_refp, 2, ref_as_p
Expand All @@ -184,7 +184,7 @@ transformed parameters{
}

// Report model
profile("transformed_delay_reporting_date_effects") {
profile("transformed_delay_reporting_time_effects") {
srdlh =
combine_effects(0, rep_beta, rep_fdesign, rep_beta_sd, rep_rdesign, 1);
}
Expand Down Expand Up @@ -234,10 +234,10 @@ model {
);
}
}
// priors and scaling for date of report effects
// priors and scaling for time of report effects
effect_priors_lp(rep_beta, rep_beta_sd, rep_beta_sd_p, rep_fncol, rep_rncol);

// priors for missing reference date effects
// priors for missing reference time effects
if (model_miss) {
miss_int ~ normal(miss_int_p[1], miss_int_p[2]);
effect_priors_lp(
Expand Down Expand Up @@ -351,7 +351,7 @@ generated quantities {
// from a temporary object to a permanent one
// drop predictions without linked observations
pp_obs = allocate_observed_obs(1, s, pp_obs_tmp, sl, csl, sdmax, csdmax);
// Posterior predictions for observations missing reference dates
// Posterior predictions for observations missing reference times
if (miss_obs) {
pp_miss_ref = obs_rng(log_exp_miss_ref, phi, model_obs);
}
Expand Down
4 changes: 2 additions & 2 deletions inst/stan/functions/combine_logit_hazards.stan
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ vector combine_logit_hazards(int i, array[,] int rdlurd, vector srdlh,
matrix ref_lh, array[] int dpmfs, int ref_p,
int rep_h, int g, int t, int l) {
vector[l] lh;
// allocate reference day effects
// allocate reference time effects
if (ref_p) {
lh = ref_lh[1:l, dpmfs[i]];
}else{
lh = rep_vector(0, l);
}
// allocate report day effects
// allocate reporting time effects
if (rep_h) {
vector[l] rlh = srdlh[rdlurd[g, t:(t + l - 1)]];
lh = lh + rlh;
Expand Down
8 changes: 4 additions & 4 deletions inst/stan/functions/delay_lpmf.stan
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ real delay_snap_lpmf(array[] int dummy, int start, int end, array[] int obs,
array[n[3]] int filt_obs = segment(obs, n[1], n[3]);
vector[n[3]] log_exp_obs;

// combine expected final obs and date effects to get expected obs
// combine expected final obs and time effects to get expected obs
log_exp_obs = expected_obs_from_snaps(
start, end, imp_obs, rdlurd, srdlh, ref_lh, dpmfs, ref_p, rep_h, ref_as_p,
sl, csl, sg, st, n[3]
Expand Down Expand Up @@ -40,8 +40,8 @@ real delay_group_lpmf(array[] int groups, int start, int end, array[] int obs,
vector[n[3]] log_exp_obs;
vector[model_miss ? miss_obs : 0] log_exp_miss_ref;

// Combine expected final obs and date effects to get expected obs
// If missing reference module in place calculated all expected obs vs
// Combine expected final obs and time effects to get expected obs
// If missing reference module in place calculate all expected obs vs
// just those observed and allocate if missing or not.
if (model_miss) {
array[3] int f = filt_obs_indexes(i_start, i_end, csdmax, sdmax);
Expand All @@ -60,7 +60,7 @@ real delay_group_lpmf(array[] int groups, int start, int end, array[] int obs,
i_start, i_end, log_exp_obs, sl, csl, log1m_exp(miss_ref_lprop)
);

// Allocate expected cases by report date
// Allocate expected cases by reporting time
if (miss_obs) {
log_exp_complete = apply_missing_reference_effects(
i_start, i_end, log_exp_complete, sdmax, csdmax, miss_ref_lprop
Expand Down
27 changes: 14 additions & 13 deletions inst/stan/functions/expected_obs.stan
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,19 @@
//
// Calculate expected observations (on the log scale) over time from a
// combination of final expected observations, the probability of reporting
// on a given day (or alternatively the logit hazard of this).
// at a given point in time (or alternatively the logit hazard of this).
//
// @param tar_obs The log of final expected observations that will be reported
// for a given date on the log scale.
// for a given time on the log scale.
//
// @param lh A vector of conditional log probabilities a report occurs on a
// given day. Optionally when ref_as_p = 0 this should be transformed first
// into the logit hazard.
// given point in time. Optionally when ref_as_p = 0 this should be transformed
// first into the logit hazard.
//
// @param ref_as_p Logical (0/1), should the reference date input be treatsd as
// a probability. Useful when no report date effects are present.
// @return A vector of expected observations for a given date by date of report
// @param ref_as_p Logical (0/1), should the reference time input be treated as
// a probability? Useful when no reporting time effects are present.
// @return A vector of expected observations for a given reference time, by
// reporting time
//
// @examples
// # compile function for use in R
Expand All @@ -27,17 +28,17 @@
// )
// rep_lh <- rep(0, 30)
//
// Example with no reporting day effect
// Example with no reporting time effect
// eobs <- exp(expected_obs(tar_obs, date_p + rep(0, 30), 1))
// all.equal(eobs, date_p)
//
// Example with hazard effect only on last day of report
// Example with hazard effect only on largest reporting delay
// ref_lh <- logit(hazard_to_log_prob(date_p))
// eobs <- exp(expected_obs(tar_obs, ref_lh, c(rep(0, 29), 0.1), 0))
// all.equal(eobs, date_p)
//
// Example with a single day of additional reporting hazard and
// no differential probability due to the reference date.
// Example with a single time point of additional reporting hazard and
// no differential probability due to the reference time.
// rep_lh <- rep(0, 30); rep_lh[7] = 2
// equal_lh <- logit(hazard_to_log_prob(rep(1/30, 30)))
// round(exp(expected_obs(tar_obs, equal_lh + rep_lh, 0)), 3)
Expand All @@ -46,7 +47,7 @@
// # 0.026 0.026 0.026 0.026 0.026 0.026
//
// Example combining multiple hazards and no differential prob due to reference
// date
// time
// rep_lh <- rep(0, 30)
// rep_lh[c(6, 12, 16)] = 2
// rep_lh[c(2, 20)] = -2
Expand All @@ -55,7 +56,7 @@
// # 0.021 0.021 0.021 0.106 0.014 0.014 0.014 0.002 0.016 0.016 0.016 0.016
// # 0.016 0.016 0.016 0.016 0.016 0.016
//
// Example combining both date of reference and date of report effects
// Example combining both reference time and reporting time effects
// eobs <- exp(expected_obs(tar_obs, ref_lh + rep_lh, 0))
// round(sum(eobs - 1e-4), 5) == 1
// round(eobs, 3)
Expand Down
4 changes: 2 additions & 2 deletions inst/stan/functions/expected_obs_from.stan
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ vector expected_obs_from_index(int i, array[] vector imp_obs,
i, rdlurd, srdlh, ref_lh, dpmfs, ref_p, rep_h, g, t, l
);
}
// combine expected final obs and date effects to get expected obs
// combine expected final obs and time effects to get expected obs
profile("model_likelihood_expected_obs") {
log_exp_obs = expected_obs(tar_obs, lh, ref_as_p);
}
Expand All @@ -42,7 +42,7 @@ vector expected_obs_from_snaps(int start, int end, array[] vector imp_obs,
t = st[i];
l = sl[i];
}
// combine expected final obs and date effects to get expected obs
// combine expected final obs and time effects to get expected obs
profile("expected_obs_from_index") {
esnap += l;
log_exp_obs[ssnap:esnap] = expected_obs_from_index(
Expand Down