Did a little more work on this and fixed a small bug in the ordering of data for the simulation that may have been causing some issues. The issue with not recovering missingness proportions sadly has not been resolved and appears to be present across multiple nowcast dates (though not always an underestimate),.

I have now:

• Checked on several other snapshots (for example see below).
• Added enw_incidence_to_cumulative() to the package and renamed enw_new_reports() to match
• Added unit tests for both these functions.

I still need to:

• Update the news item
• Add internal functions for the lookup code
• Add unit tests for the lookup code
• Add a regression test for epinowcast using the missing data model
• Add simulator to the package
• Add unit tests for simulator
• More tests of simulation recovery
• Update the model definition vignette
• Unit test internal missing stan functions

Example nowcast (with a daily random walk on the proportion missing)

Note: Not entirely clear what is driving the day of the week effect seen here)

Example nowcast (fixed proportion missing)

Example nowcast (fixed proportion missing for a different date)

So I was in a pedantic mood and made some changes to the in-code model documentation to make it a bit more intuitive (at least for me - hoping my brain is not too strange so others would profit as well 😆 )

Unfortunately, no solution yet found for the wrong missingsness proportions...

But it really looks like we are leaking observations with missing reference times in the beginning, see the plot of imp_obs below (dots are realized / ground truth observations, accounting for the missingness):

Corresponding missingness:

EDIT: not fully true, the above could also be a result of the lack of signal for the first dmax time points

This is still a riddle. Using print statements in stan, and copying the results into R (as a replacement for debugging), I have verified that no observations from imp_obs get lost further downstream:

• imp_obs get fully distributed across delays
• correct shares of cases with missing and non-missing reference allocated, sum up to overall expected obs
• obs with missing reference correctly allocated to reporting times

Since I am running out of possible implementation-level explanations for the observed behavior, I am starting to think about the model specification...

What seems to happen (not sure if this is the only problem) is that if specified flexibly, the share of cases with missing reference can compensate for a bad fit of imp_obs by producing too many cases with missing reference and distributing them over the delays. Since the likelihood for the cases with missing reference is less sharp (integrating over delays) the model can get away with that especially if case numbers are low, see the examples below.

My question is whether this is maybe an artefact of the parametric delay distribution not fitting super well to this real-world dataset. Perhaps we should do some tests with simulated data where everything is well-behaved.
I could also feed the Germany dataset with the simulated missingness into my model implementation (since it has a non-parametric reporting delay) and see what happens.

Example 1 (40% missingness)

(note that the dashed line is the realized missing proportion through binomial sampling in the simulation)

It even looks as if the share of cases with missing reference is also capturing some of the weekday effect... This would be quite weird, and I haven't seen something like that with my models.

Example 2 (10% missingness)

This basically illustrates that the problem does not go away even if missingness is small.

Unfortunately, no solution yet found for the wrong missingsness proportions...

I thought that this issue was maybe due to how we define the random walk (i.e it starts from t = 0 where data is sparse) vs the issue found elsewhere? As we discussed it's not totally clear to me what to do about this with the current implementation and it perhaps points to a flaw that should be corrected.

Since I am running out of possible implementation-level explanations for the observed behaviour, I am starting to think about the model specification...

The model being overly flexible in the wrong places is a sensible suggestion. If we moved testing to the implementations that also have a flexible expectation model that would perhaps help explore this?

the model can get away with that especially if case numbers are low, see the examples below.

Could this also be a tension between picking a more extreme growth rate and the right missingness proportion? The log-normal random walk may limit the amount of change over time required to fit this data well.

My question is whether this is maybe an artefact of the parametric delay distribution not fitting super well to this real-world dataset. Perhaps we should do some tests with simulated data where everything is well-behaved.
I could also feed the Germany dataset with the simulated missingness into my model implementation (since it has a non-parametric reporting delay) and see what happens.

This is a good suggestion and could well be the root cause. Both options for exploring are sensible. In general, we are crying out for a model simulator to ease testing and boost confidence.

Something that also concerns me is that perhaps this is highlighting a bug elsewhere. Perhaps with model specification or allocation of delays? I was fairly confident that was all working as intended but it hasn't recently been tested on clean simulated data (i.e not since initial development).

It even looks as if the share of cases with missing reference is also capturing some of the weekday effect... This would be quite weird, and I haven't seen something like that with my models.

This I found quite concerning and enforces my concern there is a bug elsewhere. Seems quite hard to explain/understand.

In terms of moving this forward, I wonder if perhaps we should merge this into develop with a big warning pointing back at this PR. Then work up #152 and #154 to MVP status and merge, and after that is done circle back to this before release. More eyes on this would likely help and it's starting to get quite weighty.

(this is a great investigation by the way, can you put the exploratory code for this somewhere so we can explore more in the future?)

Proposed tasks to get an MVP to merge:

• Fix missing_reference where multiple groups are present
• Add experimental warning
• Add Adrian to news items

Hey sam, thanks for your replies! I agree that adding this to develop may be a good idea. I think I can work on the open tasks as suggested above by you.

Can add my investigation code to the example script we already have.

For my own models, I have written a simulator that creates linelists based on a renewal process, which can then be used to create nowcasting datasets. I could maybe create a PR for this, although it would be more work to get it into the style of epinowcast (need to switch from dplyr to data.table e.g.).

So I think I have covered the minimum ground required to merge this as is.....

Can add my investigation code to the example script we already have.

I think this would be useful.

For my own models, I have written a simulator that creates linelists based on a renewal process, which can then be used to create nowcasting datasets. I could maybe create a PR for this, although it would be more work to get it into the style of epinowcast (need to switch from dplyr to data.table e.g.).

Yeah I think having something in the repository we can play with would be great - regardless of implementation style. Perhaps lets add to the repo and put in the .Rbuildignore to avoid having to add as a suggests?

As discussed I have gone ahead and merged this into develop as an experimental feature.