Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework FileSystem::OpenFile call, and add FILE_FLAGS_NULL_IF_NOT_EXISTS #11297

Merged
merged 33 commits into from
Mar 22, 2024

Conversation

Mytherin
Copy link
Collaborator

Follow-up from #11259

This PR reworks the FileSystem::OpenFile call to take a single FileOpenFlags object, instead of the previous three parameters (uint8_t flags, FileLockType lock, FileCompressionType compression). The FileOpenFlags holds all this information and is more future-proof.

unique_ptr<FileHandle> OpenFile(const string &path, FileOpenFlags flags, optional_ptr<FileOpener> opener);

The FileOpenFlags can be bitwise or-d together similar to the previous flags, and it can also be bitwise or-ed with the FileLockType and FileCompressionType, i.e. this now works:

file = fs.OpenFile(file_name, FileFlags::FILE_FLAGS_READ | FileLockType::READ_LOCK | FileCompressionType::AUTO_DETECT);

Instead of doing bitwise operations, there are now methods that can be used to access various properties, e.g. flags.OpenForWriting() instead of flags & FileFlags::FILE_FLAGS_WRITE.

FILE_FLAGS_NULL_IF_NOT_EXISTS

This PR adds a new flag to OpenFile - FILE_FLAGS_NULL_IF_NOT_EXISTS. When this is passed into OpenFile, a nullptr is returned when the file cannot be opened because it does not exist, instead of throwing an exception. This allows for this pattern:

if (fs.FileExists(file_name)) {
	auto handle = fs.OpenFile(file_name, ...);
	...
}

To be replaced with this pattern:

auto handle = fs.OpenFile(file_name, FileFlags::FILE_FLAGS_NULL_IF_NOT_EXISTS | ...)
if (handle) {
	...
}

This has the following benefits:

  • It prevents an OS-level race-condition (FileExists -> OpenFile has an obvious race where the file can be removed in between calls)
  • It is more performant (we do a single OS call instead of two). This is not very relevant for local file systems, but is actually very relevant for remote file systems, as FileExists is very expensive for HTTPFS/S3FS/etc.

Using this new pattern for the write-ahead log fixes #9342, which was caused by precisely such a race (where FileExists would return true, followed by OpenFile failing because the file did not exist).

Mytherin added 30 commits March 20, 2024 21:36
…penFile with FILE_FLAGS_NULL_IF_NOT_EXISTS
@github-actions github-actions bot marked this pull request as draft March 21, 2024 20:26
@Mytherin Mytherin marked this pull request as ready for review March 21, 2024 20:28
@github-actions github-actions bot marked this pull request as draft March 22, 2024 11:56
@Mytherin Mytherin marked this pull request as ready for review March 22, 2024 11:57
@Mytherin Mytherin merged commit 9bc963f into duckdb:main Mar 22, 2024
46 checks passed
github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request Mar 22, 2024
Merge pull request duckdb/duckdb#11297 from Mytherin/tryopenfile2
krlmlr added a commit to duckdb/duckdb-r that referenced this pull request Mar 23, 2024
Merge pull request duckdb/duckdb#11297 from Mytherin/tryopenfile2
github-actions bot pushed a commit to duckdb/duckdb-r that referenced this pull request Mar 28, 2024
Merge pull request duckdb/duckdb#11297 from Mytherin/tryopenfile2
yajirobee added a commit to yajirobee/duckdb_msgpack_extension that referenced this pull request Apr 19, 2024
yajirobee added a commit to yajirobee/duckdb_msgpack_extension that referenced this pull request Apr 23, 2024
* bump duckdb version to 0.10.2

* follow duckdb/duckdb#11297
@Mytherin Mytherin deleted the tryopenfile2 branch June 7, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Attach fails with duckdb.IOException: IO Error: Cannot open file "foo.duckdb.wal": No such file or directory
1 participant