Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Add @__RELOCDIR__, a relocatable version of @__DIR__ #55146

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

fatteneder
Copy link
Member

This adds the @__RELOCDIR__ macro which shall not hinder relocation of packages, unlike @__DIR__.

It works by splitting @__DIR__ into depot and subpath at macro expansion time.
Resolution of @__RELOCDIR__ happens at runtime by trying to find a dir at depot/subpath.

If used in scripts or with julia -e <expr> the macro behaves like @__DIR__.
If resolution of @__RELOCDIR__ fails the value of @__DIR__ is returned.


Fixes #54430

@fatteneder fatteneder added the compiler:precompilation Precompilation of modules label Jul 16, 2024
@fatteneder
Copy link
Member Author

fatteneder commented Jul 16, 2024

Open questions:

  • Shall this just replace @__DIR__? Initially I thought no, because I am too paranoid that people use @__DIR__ to construct relative paths that refer outside a depot. On the other hand, introducing a new macro just misses out on easier migration in all the non-problematic cases.
  • If it would replace current @__DIR__, would we need a mechanism to opt out of relocation? Given that kwargs for macros aren't a thing, the only option that would come to my mind would be @__DIR__(false) which looks ugly. Edit: Could also use @__DIR__(:relocatable) or so.

@giordano
Copy link
Contributor

Initially I thought no, because I am too paranoid that people use @__DIR__ to construct relative paths that refer outside a depot.

I often use @__DIR__ in scripts, which have nothing to do with depots. And the script use case is also impossible to capture with PkgEval, so please absolutely don't do that.

@topolarity
Copy link
Member

I'm not a big fan of the fact that something as simple as:

const data = joinpath(@__RELOCDIR__, "myfile.data")

is still non-relocatable.

The alternative path in my mind would be to have @__RELOCDIR__ return a path with a "virtual depot root" (e.g. /@depot/path/to/file) and then the low-library File I/O's in Julia would resolve this virtual root to it's "real" location. That's a bit similar to how Rust transparently translates paths on Windows to their "verbatim" form for transparent long-path support.

That might be too extreme to be worth hooking into low-level libraries, but it's the only way I can imagine getting good relocatability behavior "transparently"

@vchuravy
Copy link
Member

If paths were more than just strings we could handle this elegantly. I can't find the issue for that just now.

We could perhaps handle the @depot handling during relocation. I.e. when we are writing an image we scan the ?beginning? of the string for @depot add it to an array of such strings and during deserialization we patch the result?

@topolarity
Copy link
Member

We could perhaps handle the @depot handling during relocation. I.e. when we are writing an image we scan the ?beginning? of the string for @depot add it to an array of such strings and during deserialization we patch the result?

I think that's too risky - Scanning all the String's in your application and modifying some of them seems like a great way to mess up programs. What happens if it was already used as a key in a Dict, etc.?

We wouldn't be able to mark String immutable any more either, since it may be modified by the serializer

@vchuravy
Copy link
Member

I don't disagree. But having the cost of prefix checking before operations in the file handling code seems suboptimal.

I kinda want it to return a DepotPath("..../...") object that we can handle correctly

@topolarity
Copy link
Member

I don't disagree. But having the cost of prefix checking before operations in the file handling code seems suboptimal.

The scan for a prefix is probably pretty cheap (compared to the syscall), but the fact that we have multiple depot paths is definitely a huge pain.. that would require additional syscalls and may not even be possible to soundly emulate

So yeah, maybe the DepotPath style solution is the only way forward

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:precompilation Precompilation of modules
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Some uses of @__DIR__ should be relocatable.
4 participants