@fredshone you may be interested in this as well

thanks for sharing. I have feedback 😁

The mixture model is interesting. Assume this can be stacked on top of the LSTM units to do inference with more sequence "context". The primary critique is that you have to specify the number of components and/or this assumes normal distributions? In which case I can't recommend VAEs enough to introduce variance...

Silly comment but it's very traditional in ML to size layers using powers of 2. So rather than 50, use 64 (yes I know this sounds stupid but it will matter to certain audiences).

At the moment the models are not generative (not sure in the case of normal mixtures). So would expect to use a withheld test set for evaluation. Keras should also let you implement early stopping (ideally using a validation dataset), then you don't need to worry about specifying epochs.

It would be really interesting to see a roadmap of sorts for this work.

Engineering:

• In the past I have found seaborn to be a dependencies maintenance pain because it lagged behind on core project dependencies like plt and np - but maybe this is old news.
• adding keras and tensorflow as dependencies is very chonky and these models require a special skill set to maintain. I strongly suggest that they are developed in a separate project (pam-ml?) with pam as a dependency until there is some killer feature ready to add in later.

thanks @fredshone :)

The mixture model is interesting. Assume this can be stacked on top of the LSTM units to do inference with more sequence "context". The primary critique is that you have to specify the number of components and/or this assumes normal distributions? In which case I can't recommend VAEs enough to introduce variance...

The assumption of gaussian mixture distribution with pre-defined number of components was the intention here - it follows the review of observed distributions, and it feels like a good way to bound the solution space. However, I was also wondering - am I essentially moving towards a VAE? I guess the main difference is whether you choose to sample from a latent space or not. I feel the use cases are slightly different, but can't say what works better here. Maybe one of the next steps could be to solve the same problem (in notebook 2) with a VAE, and compare the results?

Silly comment but it's very traditional in ML to size layers using powers of 2. So rather than 50, use 64 (yes I know this sounds stupid but it will matter to certain audiences).

Good point - will update

At the moment the models are not generative (not sure in the case of normal mixtures). So would expect to use a withheld test set for evaluation. Keras should also let you implement early stopping (ideally using a validation dataset), then you don't need to worry about specifying epochs.

The aim is to combine the two models and end up with a generative model, that is recurrent, but is also able to sample results from a distribution.
Yes, I will introduce a test set, once we try this with more realistic data. For now, these are some toys for exploration and some basic checks - I am not formally reviewing performance yet.

Engineering:

• In the past I have found seaborn to be a dependencies maintenance pain because it lagged behind on core project dependencies like plt and np - but maybe this is old news.
• adding keras and tensorflow as dependencies is very chonky and these models require a special skill set to maintain. I strongly suggest that they are developed in a separate project (pam-ml?) with pam as a dependency until there is some killer feature ready to add in later.

Yeah, as you can see with the failing Windows tests, dependencies proved to be an issue here. I will see if there is a good way to resolve them with certain pinned versions. I don't have an answer yet; I am considering perhaps splitting to another repo, but we would still need to solve the problem if PAM is a dependency.

@fredshone I have added a very simplistic VAE example (03_vae_multimodal) that tries to solve the same problem as notebook 02_gaussian_mixture (ie predicting activity durations for an activity that follows a bi-modal distribution profile). In the spirit of this PR, I am trying to experiment with very minimalistic examples to build up our intuition.

After trying a few variations, my thoughts are:

• you can get a similar outcome with a VAE as with the MDN model.
• however, the VAE may require more training data and time
• this probably makes sense: the VAE is essentially trying to solve a more difficult problem (with more degrees of freedom)
• it may have more freedom to produce individual predictions that depart more from observed patterns
• as you say, the main benefit is the ability to fit any kind of distribution without prior assumptions about its type

The comparison probably comes down to someone's use case:

• what is the underlying temporal distributions of your data? Can they be approximated well with well-known distributions or do you want something more tightly-fit and agnostic?
• how much do you want to constrain the solution space of your model?
• how much do you care about interpretability?

Any thoughts appreciated @fredshone @panostsolerid @brynpickering !

thanks Theo, I think this is a nice conclusion.

Agree VAE has to do more work to reproduce a gaussian mixture, but as you say it can generalise more broadly. Related, my best models are big (0.5m params), which they use to reproduce the distributions in schedules. I am thinking how to combine the two approaches to make my models more parsimonious. For example if distributions can be incorporated in the vae decoder.

The comparison probably comes down to someone's use case:

what is the underlying temporal distributions of your data? Can they be approximated well with well-known distributions or do you want something more tightly-fit and agnostic?
how much do you want to constrain the solution space of your model?
how much do you care about interpret-ability?

My thinking is that once we get to sequence generation, the joint-distributions are gonna get pretty wild. We also need the models to incorporate less statistical rules, such as day duration, vehicle consistency and so on. Specifying and structuring all these will be tough. Expecting this to be interpret-able by looking at model params also not going to be a pleasant exercise...

If we want to chase structure then will tackle simpler things like total day duration. If we extend to household scheduling then interesting to think how these models could be structures to deal well with shared trips and shared activities.

I guess my inevitable answer is that I don't care about interpret-ability. Instead I want us to have a rigorous evaluation framework so we can just focus on results.

thanks @fredshone. I am taking these step-by-step :) . Some of the things you mention are what I am looking now as the next example (aiming for a summary of findings for Tuesday).

Joint distributions in plans: I am preparing an example now. It can indeed get out of hand, however, the large majority of plans tend to be quite simple with 1-2 non-home activities in the day. Trying to capture and interpret some of that interests me for scenario testing, and generally to understand what kind of signal the model is picking up.

Total day duration: I am also thinking about this. I consider a few approaches: a) use day duration as a metric: the results of a good model should predict durations that roughly up to 1. If that is true, simple adjust durations either proportionally, or according to some rule, b) The duration of the final activity is not predicted - it rather uses the remaining time budget in the day. c) The loss function penalises plans that don't add up well. Let me know if you have any other ideas.

Household structuring: I am probably far from that at the moment. I would expect a hybrid model that incorporates some sort of rule set or simulation to resolve the use of shared resources? That would be quite interesting.

total sequence duration seems tricky with RNNs - cannot deal with duration until first end of sequence token. I did try a combined loss function of activity type, duration and end time. The idea being that end time loss tries to keep total durations (so far) correct. But didn't see a useful result. Perhaps needs further work.

At the moment by models have average total duration error of only a few minutes in any case - so easy to fix after without too much stress.