Need to add a bit of code that picks an time that is not in HF already

That's just been added.

Thanks that was quick

Today, 46,000 / 95,000.
Seems to be about 30,000 a month, so will take another 6 weeks

@devsjc Is there an update on this? I'm looking to update the GCP disks for a new pvnet backtest and thought it could be good to update the 2018 as well: https://github.com/orgs/openclimatefix/projects/33/views/8?filterQuery=assignee%3Azakwatts&pane=issue&itemId=60827379

Not a backfill I was aware of, was probably being run via Jacobs login on a VM so not something I can easily track - going off of Peter's described cadence alone it sounds like it should have been finished a few months ago?

Thanks @devsjc anyways. @peterdudfield Do you know where this data is stored? I've checked the Google Storage and its not updated for 2018 yet (still NaNs). I can see that it gets uploaded to hugging face and is there: https://huggingface.co/datasets/openclimatefix/eumetsat-rss/tree/main/data/2018. I'm guess theres an upload script to get it onto the google storage that we might need to run?

@devsjc does the "gcp-sat-update" vm directly upload data to HF or the public Google storage?

Yeah, I was running it in the VM, I think it's in the attached disk there, I am not sure if it finished. The HF data might be good, but also might be NaNs so you'd need to check that.

Thanks @jacobbieker