Calling a second time IOUtils.copy(in, out) ?
Is that normal ?

Depending on you definition of "normal." 😸
I do not see the reason why it can not be called twice, and the idea is that bulk of the data will be copied by the first call without any blocking, but then second one would be executed while file store updates are blocked and any pipelined store operations are finished. It expected to be brief.

Unless I am mistaken, IOUtils.copy(in, out) is copying the full file in the (zip file entry) output stream. So it looks to me you will end up here with the full file copied twice (the two copies concatenated in the single zip file entry).
I guess this is not what is intended.

There is no position reset, so second copy operation should start where the previous one left off, and maybe pick up extra one or two chunks written by last store operation(s) caught in progress.

Please check IOUtils.copy(FileChannel in, OutputStream out) used here: the input is a FileChannel and is read from position 0.
I have quickly checked (using the TestBackup#testBackup test) and I can confirm the .mv.db file in the backup zip is twice as big as the original file.

@vreuland You right, my bad I missed the fact that absolutely positioned version of read() is used. Fixed it to start the second copy operation from where first one stopped.

Thank you @andreitokar 👍

I have two additional questions for you if I may 😃 :

1. Should we not also take a better care of the file header(s) (i.e. the two first blocks of the file) in this backup ? I mean it looks like this is the only part of the file that is potentially updated/rewritten once the reuseSpace flag is set to false. We could also copy those two blocks inside the section protected by the store lock and avoid any corruption of the header(s), no ?

2. Do we really need to copy the chunks written after the reuseSpace flag has been set to false? If we take a copy of the file as it was exactly when the reuseSpace flag was set to false, it looks to me we would get a consistent backup (at the start time of the backup) and we would also avoid taking a second time the lock.

I saw and know there is some extensive logic when opening the file store to deal with corrupted or obsolete file header(s) (ie. pointing to old block/chunk and not the last written one,...) but it seems to me we could actually end up with a perfectly consistent backup (at the start time of the backup) if we would do something like this (in pseudo-code):

underStoreLock() {
    flushStoreHeader();
    setReuseSpace(false);
    fileSizeAtBackupStart = getStoreFileSize();
    copy(in, out, 0, 2*BlockSize); // Copy only the headers (that should be very quick)
}

copy(in, out, 2*BlockSize, fileSizeAtBackupStart);

What do you think ? Do I miss something ?

@vreuland, I totally agree with you on the first point. In fact I had it implemented already, but had reservations. My concern was that this quick and dirty implementation writes "clean shutdown" mark into database first, then copies header into backup, and then erases flag from db file. So, what if db crashes before last step?
But now it occur to me that at this point db file would look exactly as backup, so should be fine.
On your second point:

If we take a copy of the file as it was exactly when the reuseSpace flag was set to false

You assume that chunk production is an atomic process and can be easy coordinated with setting that flag, but it is a pipeline where few chunks can be in-flight, and it is unclear which ones affected by flag change. It is possible that chunk will be written in the middle of the file even after reuseSpace was cleared.

Indeed, I was not aware of this non-atomic pipelined chunk writing.

It is possible that chunk will be written in the middle of the file even after reuseSpace was cleared.

I don't really see how we can guarantee that the backup will always be valid and consistent if that happens. There will always be a chance that what we are busy copying is being updated/rewritten at the same time...

Is there no way to avoid that ? No way for the backup thread to wait for the "inflight" pipeline to be completed ?

I don't think this is correct: The file header(s) will be copied at the tail of the backup file and not at the beginning.

@vreuland Yes, you are right again, thank you! I don't know what I've been smoking. 😸
Just don't have enough time to do it now.