This is awesome! ♥️
What's it's semantics though? How does it handle ambiguity? What is the result of parse(Union{Int32, Int64}, "15")? And given that unions are symmetrical, what if the two types in the union were switched?

@jakobnissen this is about parsing type objects themselves from strings. So in your example, it would be an error, since you've provided a number, not a type.

You could do parse(Type, "Union{Int, Int32}") and you'd get the union type as the result.

Does that make sense?

EDIT: Could I adjust anything in the PR description to make this clearer?

No this is clear enough, it was just me who didn't read it carefully enough :)

You might be interested in (for reference if nothing else) the approach I've taken in DataToolkitBase where I have a QualifiedType that represents a Type that may or may not be able to be realised at parse-time (e.g. relies on unloaded modules), that I may then construct later.

It is defined like so:

struct QualifiedType
    root::Symbol
    parents::Vector{Symbol}
    name::Symbol
    parameters::Tuple
end

And here is the parsing code (less capable than yours I suspect) https://github.com/tecosaur/DataToolkitBase.jl/blob/main/src/model/parser.jl#L5-L59, and the "make it an actual type" code
https://github.com/tecosaur/DataToolkitBase.jl/blob/main/src/model/qualifiedtype.jl#L33-L78.

This is how performance compares:

julia> @btime eval(Meta.parse("Main.Main.Base.Checked.Vector{Dict{Int, Base.Float64}}"))
  99.360 μs (176 allocations: 10.11 KiB)
Vector{Dict{Int64, Float64}} (alias for Array{Dict{Int64, Float64}, 1})

julia> @btime typeify(parse(QualifiedType, "Main.Main.Base.Checked.Vector{Dict{Int, Base.Float64}}"))
  11.224 μs (190 allocations: 10.59 KiB)
Vector{Dict{Int64, Float64}} (alias for Array{Dict{Int64, Float64}, 1})

Okay, I think this is ready for a real review! I would super appreciate another pass, @stevengj and @MasonProtter, and maybe a review from @topolarity, @c42f, or @LilithHafner?

The remaining open questions are I think:

• Is it okay that this is using Meta.parse under the hood, or should it have custom parsing code? I think I'd like to stick with Meta.parse, so that it's guaranteed to match what julia would parse as. So unless there is a very good reason to do otherwise, I'd like to stick with this.
• Should this automatically look into Base.loaded_modules to find top-level names? If you try to call parse(Type, "Random.UInt10Raw"), but Random is only an indirect dependency (e.g. you imported Test), should this parse?
  • I have currently decided no, but it's very easy to add this back in! I deleted the support for this in 883673d, so we could always add that back in if people disagree.
  • Instead, I currently require you to pass parse(Type, "Random.UInt10Raw"; module_context=Test), which will work. But you'd have to know a priori where to find it, or do something like import all the names from loaded_modules or something.
  • I'm mainly planning to use this PR in the PackageCompiler context (and a similar context for running precompile statements for PkgImages), where we will have already imported all names from loaded_modules, so we don't need this support for that. It's just about what others think is the friendliest API.
• Is the implementation for UnionAlls reasonable? Anything you'd change?

Thanks!!

CC: @kpamnany

tryparse throws sometimes

julia> Base.tryparse(Int, "5+4")

julia> Base.tryparse(Type, "5+4")
ERROR: MethodError: no method matching reinterpret(::typeof(+), ::Tuple{Int64, Int64})
The function `reinterpret` exists, but no method is defined for this combination of argument types.
...

The biggest thing that stands out to me is that this feels to me like a deserialization and not a parse.

In particular, it has to worry about a lot of classic "serialization" problems:

• Where are type names resolved?
• Is the de-serializer responsible for loading unavailable types?
• Is the serialization (i.e. string()) an invertible representation for all objects?

These questions make the meaning of a type string like this contextual, even for very "primitive" seeming type names:

module Test
   struct Union{A,B} end
   struct Float32 end
   struct Float64 end
   struct Foo
       x::Union{Float32,Float64}
   end
end

The Julia syntax here works double-duty as a serialization format (very similar to how raw Expr is stored and used in the deserializer), so that I think tying this to Base.parse(Type, ...) is not quite right.

@topolarity: I agree with you, but for better or worse, julia is already using string() and parse (via eval(Meta.parse())) as a serialization round-trip format for types, in the PackageCompiler precompilation process. That Meta.parse+eval is slow and unsafe, and so i'm trying to provide a safer faster route to support this.

The current process is that PackageCompiler runs your snoop workload with --trace-compile, and julia produces a bunch of precompile statements:

precompile(Tuple{typeof(Base.iterate), Base.Set{Char}})
precompile(Tuple{typeof(Base.copy), Array{ArgParse.ArgParseGroup, 1}})
precompile(Tuple{typeof(Base.pairs), NamedTuple{(:help, :arg_type), Tuple{String, DataType}}})
precompile(Tuple{typeof(Unrolled.type_length), RAICode.QueryEvaluator.var"#169#170"{String, ((), (1,))}})

then PackageCompiler parses and evals each of those types, and precompiles them, as in something like this:

ps = Meta.parse(statement)
@assert isexpr(ps, :call) && ps.args[1] == :precompile
ps = Core.eval(PrecompileStagingArea, ps)
precompile(ps...)

(We want to do this in our production servers, by having them run a set of precompile statements at startup, but I want to prevent the risk of someone having an arbitrary code-execution exploit by mutating the file that contains those statement - and also to make it faster.)

So I'm open to picking a different name for this mechanism, but i think the use of strings as a round-tripping format for types is already out there, and I'm mainly hoping to make that safer and faster. It does feel like parsing to me though: you're taking a string representation of a value, and constructing that julia value, according to julia's parse-tree.

Yeah, I agree with the motivation!

Better safety/performance for eval(Meta.parse(...))-style deserialization in PackageCompiler is a good thing.

My complaint was specifically about tying this to Base.parse. That API should return a valid interpretation of Julia code for all contexts where that code might appear - Even something innocent like evaluating what Float32 means breaks that rule.

On the other hand, doing eval(Meta.parse(...))-style deserialization isn't illegal on its own. Although it can be a bit fragile, I agree that ship is out of the gate. But the eval in eval(Meta.parse(...)) makes it very clear that its result depends on the Module it is evaluated in. For the round trip to work out, you have to make sure you think about that context and "match it up" between serialization/deserialization time.

I'd prefer this had a similar API that is either very clear about doing contextual evaluation (albeit much safer and more limited than eval()) or taking in the relevant context explicitly (similar to two-argument Core.eval)

Thanks cody. I'll have to think about what an alternative API should look like. --- Do you have any suggestions?

(FWIW, I did add a module_context=Main kwarg in my last push a few weeks ago. I agree that is a necessary part of something like this. 👍)

But yeah, i think your greater point stands that this probably isn't exactly "parsing." It's really close - it is at least parsing-adjacent. But indeed, maybe type-deserializing is a better term?

..... 😅 I keep coming back to this question in my head though: is deserializing a value from a string representation of julia source code anything other than parsing? 😅
Anyway, please help me pick a new name! 🙏 And let me know if you think this should live in a package instead.

I'm late to the party here (as usual sorry 😬)

But I totally agree with Cody. @topolarity I appreciate how well you explained why this "doesn't quite feel right as parse". I agree.

It seems that what you've implemented here is actually a restricted Julia interpreter:

• An interpreter where only some forms are allowed
• It operates on surface syntax, not lowered syntax
• Some extended cases are still allowed. For example bits types.

All this makes it surprisingly complex!

Amusingly we already have other interpreters a bit like this. For example, the runtime's jl_toplevel_eval_flex() evaluates some forms without lowering them (presumably for efficiency). But unlike the code here, it defers other forms to the full lowering code.

Should we really keep a zoo of special purpose interpreters? I don't know. If it matters performance-wise I guess we have to.

That's where the bulk of the remaining CPU time goes now.. Would it be preferential to write custom parsing code straight from the string to the Types?

This seems like a lot of work, I don't recommend it 😅
Does it help performance if you use JuliaSyntax.parsestmt() directly? That should be quite fast.