Feature expectation reporting #154

seabbs · 2022-08-05T17:21:14Z

Rounding this week off with another Draft PR showcasing a prototype feature. Here we add latent reporting (i.e observations are based on some discrete scaled convolution of an underlying latent parameter) and a new formula interface for specifying observation level effects (i.e the growth rate is constant but reported observations have a day-of-the-week adjustment.

This in effect integrates EpiNow2 (in its most basic form with fixed delays) into the epinowcast framework with the addition of formula interface flexibility. No changes have been made to make nowcasting optional but this should be implemented in the future as it increases overlap with EpiNow2 making it easier to depreciate.

Note: As for other draft PRs this week no testing or bug checking has been done here and this is strictly a draft. I will add a list of to-dos next week/later.

# Load packages
library(epinowcast)
library(data.table)
library(ggplot2)

# Use 2 cores
options(mc.cores = 2)

# Load and filter germany hospitalisations
nat_germany_hosp <- germany_covid19_hosp[location == "DE"][age_group %in% "00+"]
nat_germany_hosp <- enw_filter_report_dates(
  nat_germany_hosp,
  latest_date = "2021-10-01"
)

# Make sure observations are complete
nat_germany_hosp <- enw_complete_dates(
  nat_germany_hosp,
  by = c("location", "age_group")
)
# Make a retrospective dataset
retro_nat_germany <- enw_filter_report_dates(
  nat_germany_hosp,
  remove_days = 40
)
retro_nat_germany <- enw_filter_reference_dates(
  retro_nat_germany,
  include_days = 40
)

# Get latest observations for the same time period
latest_obs <- enw_latest_data(nat_germany_hosp)
latest_obs <- enw_filter_reference_dates(
  latest_obs,
  remove_days = 40, include_days = 20
)

# Preprocess observations (note this maximum delay is likely too short)
pobs <- enw_preprocess_data(retro_nat_germany, max_delay = 20)

# Fit the default nowcast model and produce a nowcast
# Note that we have reduced samples for this example to reduce runtimes
nowcast <- epinowcast(pobs,
  expectation = enw_expectation(
    ~ rw(week), generation_time = c(0.2, 0.6, 0.2),
    observation = ~ (1 | day_of_week),
    latent_reporting_delay = c(0.3, 0.2, 0.1),
    data = pobs
  ),
  fit = enw_fit_opts(
    save_warmup = FALSE, pp = TRUE,
    chains = 2, iter_warmup = 500, iter_sampling = 500
  ),
  obs = enw_obs(family = "poisson", data = pobs)
)

The current example has some slight fitting complaints similar to those seen in #152 and is likely driven by bugs/non-optimal implementation choices.

seabbs · 2022-08-08T16:19:44Z

Added a new example for this functionality and returned the simple example to being simple (now with a fixed growth rate). Also made some basic plots and started thinking about if/what data modules need to carry for postprocessing

Including example plots below.

Interesting here that the expected reported cases look more convincing than the actual nowcast...

seabbs · 2022-09-01T11:12:05Z

I think it might be a good idea to merge this into develop in a very MVP state if possible to help with exploration and help make smaller PRs fleshing elements out easier.

I think the minimum that needs to be done is:

Merge in Add missing reference model components to delay_group_lmpf and generated quantities #147 and Flexible expectation model #152
Update this PR
Add a news item highlighting what has been done.

seabbs · 2022-09-01T16:55:09Z

Going to merge into #152 to make a single large PR for merging into develop.

seabbs

LGTM

* add control variables for flexible reporting * add reporting function * extended model compiles * add partial changes for R side observation model * add expectation model input code * add skipping for latent model * add new default enw_expectation args and standardise usage * add support for priors and correctly allow modifiers with no convolution * debug implementation * new example for Rt and prototype plots * update priors specification

* add required delay_group_lmpf changes for missing ref model * debug to compilation * use segment where possible * expectation model compiles * dont store local lobs * remove blocker for expectation model as now MVP * test simple example * add enw_incidence_to_cumulativ and update enw_new_reports to match * add enw_incidence_to_cumulative * add internal helper functions for missing reference lookup * fix indexing bug with enw_reference_by_report * Feature expectation reporting (#154) * add control variables for flexible reporting * add reporting function * extended model compiles * add partial changes for R side observation model * add expectation model input code * add skipping for latent model * add new default enw_expectation args and standardise usage * add support for priors and correctly allow modifiers with no convolution * debug implementation * new example for Rt and prototype plots * update priors specification * Streamline time wording in in-model doc * add usage warning for the missing data MVP * update news * add handling of group-wise missing reference observations and look-ups * use filtered missing reference obs in likelihood * fix merge issues * update test snapshot * update example data * Change formulation readme * Format stan code, improve inline doc * Change expo to expl * Further formatting and refactoring * Rename stan function files * Add log_dot_product * Further refactor latent vs obs in model block / functions * Fix remaining expo to expl * deal with duplicated news item * Add vector to reporting delay and generation time definitions. * refresh docs * fix priors as data list test * updat expectation test snaps * simulated example * Revert "simulated example" This reverts commit ad40667. * Feature optional intercepts (#170) * add tools for making the intercept optional both in model and in R * fix existing tests * add new test * add news item * update doc * update documentation with new defaults * update tests to new default expectation model * update README * spell check * more spell checks * update credit in news Co-authored-by: Adrian Lison <adrian.lison@bsse.ethz.ch> Co-authored-by: Sebastian Funk <sebastian.funk@lshtm.ac.uk>

seabbs added 9 commits August 4, 2022 11:44

add control variables for flexible reporting

d372572

add reporting function

ea3c926

extended model compiles

601affd

add partial changes for R side observation model

7bb0814

add expectation model input code

a2a99d4

add skipping for latent model

86dabd3

add new default enw_expectation args and standardise usage

d44d320

add support for priors and correctly allow modifiers with no convolution

47b6e4c

debug implementation

a87f63d

seabbs self-assigned this Aug 5, 2022

new example for Rt and prototype plots

a88b7e0

update priors specification

7901592

seabbs added the enhancement New feature or request label Aug 17, 2022

merge missingness changes

9776f58

seabbs marked this pull request as ready for review September 1, 2022 10:36

seabbs mentioned this pull request Sep 1, 2022

Add missing reference model components to delay_group_lmpf and generated quantities #147

Merged

6 tasks

seabbs commented Sep 1, 2022

View reviewed changes

seabbs merged commit bb614c0 into flexible-expectation Sep 1, 2022

seabbs deleted the feature-expectation-reporting branch September 1, 2022 16:56

This was referenced Sep 1, 2022

Flexible expectation model #152

Merged

Missing data proportion recovery #165

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature expectation reporting #154

Feature expectation reporting #154

seabbs commented Aug 5, 2022 •

edited

Loading

seabbs commented Aug 8, 2022 •

edited

Loading

seabbs commented Sep 1, 2022 •

edited

Loading

seabbs commented Sep 1, 2022

seabbs left a comment

Feature expectation reporting #154

Feature expectation reporting #154

Conversation

seabbs commented Aug 5, 2022 • edited Loading

seabbs commented Aug 8, 2022 • edited Loading

seabbs commented Sep 1, 2022 • edited Loading

seabbs commented Sep 1, 2022

seabbs left a comment

Choose a reason for hiding this comment

seabbs commented Aug 5, 2022 •

edited

Loading

seabbs commented Aug 8, 2022 •

edited

Loading

seabbs commented Sep 1, 2022 •

edited

Loading