Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fields: expand lazy vals during fields, like modules #5294

Merged
merged 19 commits into from
Sep 1, 2016

Conversation

adriaanm
Copy link
Contributor

@adriaanm adriaanm commented Jul 20, 2016

(follow up for #5141 )

Fields phase fully expands lazy vals and modules

Mixins is relieved of its bitmap emission duties.

Important changes

  • the locking scope of a local lazy val is now tied to the lazy val, following Dotty's encoding
  • specialized lazy vals work because the full expansion happens before specialization

Internal details

Lazy vals remain ValDefs (with a method symbol for its accessor) until the fields phase, along with deferred vals and all trait vals. Code that previously only considered DefDefs when looking for accessors will need to widen this search to ValOrDefDef. Also, accessed will not lead to a field in these cases (ultimately, we should delay all field creation until fields.)

Lazy val accessors, fields and bitmaps are now mixed in during the fields phase, which means erasure and specialization get a better look at them. Should we also move mixins before erasure?

@lrytz did the bulk of the work in moving our encoding of local lazy vals to the scheme used by Dotty, based on a suggestion by @DarkDimius. Locking is now down on the holder of the lazy val's value / init bit, instead of on the enclosing class (which had widened from the closure class to the owner of the method, making this more prone to deadlocks).

All lazy vals and modules use double-checked locking, but modules no longer spin off their initialization to a compute method (it's short enough not to interfere with the inlining of the lazy accessor).

Notes from the implementation lab journal

  • remove lazy accessor logic
    now that we have a single ValDef for lazy vals,
    with the underlying machinery being hidden until the fields phase
    leave a @deprecated def lazyAccessor for scala-refactoring

  • don't skolemize in purely synthetic getters,
    but do skolemize in lazy accessor during typers

    Lazy accessors have arbitrary user code, so have to skolemize.
    We exempt the purely synthetic accessors (isSyntheticAccessor)
    for strict vals, and lazy accessors emitted by the fields phase
    to avoid spurious type mismatches due to issues with existentials
    (That bug is tracked as scala/scala-dev#165)

    When we're past typer, lazy accessors are synthetic,
    but before they are user-defined to make this hack less hacky,
    we could rework our flag usage to allow for
    requiring both the ACCESSOR and the SYNTHETIC bits
    to identify synthetic accessors and trigger the exemption.

    see also scala/scala-dev#165

    ok 7 - pos/existentials-harmful.scala
    ok 8 - pos/t2435.scala
    ok 9 - pos/existentials.scala

    previous attempt: skolemize type of val inside the private[this] val

    because its type is only observed from inside the
    accessor methods (inside the method scope its existentials are skolemized)

  • bean accessors have regular method types, not nullary method types

  • must re-infer type for param accessor
    some weirdness with scoping of param accessor vals and defs?

  • tailcalls detect lazy vals, which are defdefs after fields

  • can inline constant lazy val from trait

  • don't mix in fields etc for an overridden lazy val

  • need try-lift in lazy vals: the assign is not seen in uncurry
    because fields does the transform (see run/t2333.scala)

  • ensure field members end up final in bytecode

  • implicit class companion method: annot filter in completer

  • update check: previous error message was tangled up with unrelated
    field definitions (var s and val s_scope),
    now it behaves consistently whether those are val/vars or defs

  • analyzer plugin check update seems benign, but no way to know...

  • error message gen: there is no underlying symbol for a deferred var; look for missing getter/setter instead

  • avoid retypechecking valdefs while duplicating for specialize -- see pos/spec-private

  • Scaladoc uniformly looks to field/accessor symbol

  • test updates to innerClassAttribute by Lukas

TODO:

tests / review follow up

  • Introducing: the fields phase [ci: last-only] #5141 (comment)
  • scalacheck/quasiquotes
  • presentation/t5708
  • presentation/t8459
  • run/t7767.scala
  • scalacheck/HtmlFactoryTest.scala
  • SI-8873 @volatile var cannot be defined in case class with confusing error message
  • neg/t712.scala
  • run/existential-rangepos.scala
  • run/idempotency-case-classes.scala
  • run/reify_ann3.scala
  • run/reify_ann5.scala
  • run/repl-colon-type.scala
  • run/trait-fields-override-lazy.scala
  • run/analyzerPlugins.scala
  • neg/abstract-vars.scala update users of analyzer.underlyingSymbol to reflect abstract val/vars don't have a field
  • pos/spec-private.scala
  • run/t8549.scala (check bytecode of scala/collection/mutable/DoubleLinkedList.scala)
  • jvm/innerClassAttribute -- understand bytecode changes in jvm/innerClassAttribute/Classes_1.scala https://gist.github.com/adriaanm/833837b204141a130abaaa6a62ed8596 (thanks, Lukas!)
  • -Xcheckinit
  • look at bootstrapped bytecode diff

community build

  • bring back lazyAccessor for scala-refactoring
  • scalaz (missing sig for implicit val at position after use site)
  • scala-js ExportsTest fail (valdefs may also contribute exported methods: deferred val/trait val/lazy val)
  • akka (existential in lazy val accessor)

follow ups after RC1:

  • failing test in macro-paradise
  • rangepos

@adriaanm adriaanm added this to the 2.12.0-RC1 milestone Jul 20, 2016
@adriaanm adriaanm force-pushed the fields-lazies branch 2 times, most recently from 72eb820 to 822fd93 Compare July 21, 2016 23:51
@adriaanm adriaanm force-pushed the fields-lazies branch 2 times, most recently from be2c934 to 3d8a814 Compare August 12, 2016 00:00
@adriaanm adriaanm changed the title Fields: the lazies [ci: last-only] Fields: expand lazy vals during fields, like modules [ci: last-only] Aug 12, 2016
@lrytz
Copy link
Member

lrytz commented Aug 12, 2016

I checked the innerClassAttribute test, here's the fix: lrytz@e685a9a. note that the diff I see is smaller than the one you link to in your comment. The only differences are in numbering lifted definitions and anonymous classes.

@adriaanm
Copy link
Contributor Author

@retronym
Copy link
Member

I've taken a breadth-first pass over this and haven't found any issues so far. I'll dive in deeper tomorrow.

@lrytz
Copy link
Member

lrytz commented Aug 15, 2016

@adriaanm I started looking at implementing scala/scala-dev#133, but it's not clear to me what's the ultimate goal for the lazy val desugaring. Currently, the fields phase creates a variable and a getter for each lazy val. The lazyvals phase then splits up the getter into a getter and a lzy$compute method. Will this remain the same?

Also, there's still some duplication between lazyvals and mixin: both have a method mkLazyDef, the two implementations are basically the same. Will this also remain?

For scala/scala-dev#133, the idea is to expand a local lazy val v = init into

val v$lzy = new LazyRef()
def v = {
  if (!v$lzy.initialized) v$lzy.synchronized {
    if (!v$lzy.initialized) { v$lzy.initialized = true; v$lzy.value = init }
  }
  v$lzy.value
}

Should all of this be done in the fields phase? Or during lazyvals?

@adriaanm
Copy link
Contributor Author

Thanks! My goal is to move the remaining lazy desugaring from mixin to lazyvals, so that there's no duplication. I have started poking around to convince myself that Mixin's mkLazyDef isn't used anymore. I'm not sure I'll have time to move the logic that emits the init flags from Mixins to Lazyvals, but ultimately that would be the goal as well.

@adriaanm
Copy link
Contributor Author

adriaanm commented Aug 15, 2016

Now I've paged back in where I left this: mixins handles lazy vals in classes and uses the same init bit logic as checkinit, so moving that to lazyvals could be tricky to do within the time budget for RC1.

@lrytz
Copy link
Member

lrytz commented Aug 15, 2016

yeah, just found out the same :)

@lrytz
Copy link
Member

lrytz commented Aug 15, 2016

so lazyvals only handles local lazy vals? then i think we can get rid of the phase and move the translation for locals to the fields phase. i'm happy to take a look at this, doesn't look too hard (famous last words).

@adriaanm
Copy link
Contributor Author

Ah, that's an interesting approach, hadn't thought of that.

@adriaanm
Copy link
Contributor Author

adriaanm commented Aug 15, 2016

It looks like it's some strange mixture of the two (EDIT: local lazy vals and modules in classes --> both could be done during the fields phase) -- let's synch on slack how to divide the work here.

@adriaanm adriaanm force-pushed the fields-lazies branch 2 times, most recently from 5b5b55f to 34f967f Compare August 15, 2016 18:50
@lrytz
Copy link
Member

lrytz commented Aug 15, 2016

wip: lrytz@9f2f990

currently crashes in backend, will pick it up tomorrow.

warning: an unexpected type representation reached the compiler backend while compiling Test.scala: [T0]()T0. If possible, please file a bug on issues.scala-lang.org.
scala.MatchError: [T0]()T0 (of class scala.reflect.internal.Types$PolyType)
    at scala.tools.nsc.backend.jvm.BTypesFromSymbols.typeToBType(BTypesFromSymbols.scala:212)
    at scala.tools.nsc.backend.jvm.BCodeSkelBuilder$PlainSkelBuilder.tpeTK(BCodeSkelBuilder.scala:79)
    at scala.tools.nsc.backend.jvm.BCodeBodyBuilder$PlainBodyBuilder.genApply(BCodeBodyBuilder.scala:523)

@adriaanm adriaanm force-pushed the fields-lazies branch 3 times, most recently from cd61abf to 562b3ce Compare August 16, 2016 00:59
@lrytz
Copy link
Member

lrytz commented Aug 16, 2016

@adriaanm here's the desugaring for local vals: lrytz@26fbaf6. Note that this revision is not rebased on the current top of this PR.

I tried rebasing on top of this PR, the result is here: lrytz@3d56b1f. But this revision doesn't pass the test I wrote.

There are a few TODOs in the commit, please take a look. A few more open questions:

  • Can we remove the lazyvals phase? It seems it's still used for desugaring objects into a field, bitmap and lzycompute.
  • Should we write the LazyHolders in Java? It would save getter / setter accesses.

How should we proceed?

@adriaanm adriaanm force-pushed the fields-lazies branch 4 times, most recently from b086764 to 0581b6e Compare August 16, 2016 21:36
@retronym retronym mentioned this pull request Sep 28, 2016
2 tasks
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 28, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
@adriaanm adriaanm mentioned this pull request Sep 28, 2016
2 tasks
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 28, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 28, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
adriaanm added a commit to adriaanm/scala that referenced this pull request Sep 29, 2016
The motivation is to use the new fine-grained lock scoping that
local lazies have since scala#5294.

Fixes scala/scala-dev#235

Co-Authored-By: Jason Zaugg <jzaugg@gmail.com>
@adriaanm adriaanm modified the milestone: 2.12.0-RC1 Oct 29, 2016
@adriaanm adriaanm added the 2.12 label Oct 29, 2016
adriaanm added a commit to adriaanm/scala that referenced this pull request Nov 30, 2016
adriaanm added a commit to adriaanm/scala that referenced this pull request Dec 2, 2016
This likely regressed in scala#5294.

Review feedback from retronym:
 - Tie annotation triaging a bit closer together
 - Drop all annotations from static forwarders

durban kindly provided initial version of test/files/run/t10075.scala
And pointed out you must force `lazy val`, since `null`-valued field
is serializable regardless of its type.

Test test/files/run/t10075b courtesy of retronym
adriaanm added a commit to adriaanm/scala that referenced this pull request Dec 3, 2016
This likely regressed in scala#5294.

Review feedback from retronym:
 - Tie annotation triaging a bit closer together
 - Drop all annotations from static forwarders

durban kindly provided initial version of test/files/run/t10075.scala
And pointed out you must force `lazy val`, since `null`-valued field
is serializable regardless of its type.

Test test/files/run/t10075b courtesy of retronym
adriaanm added a commit to adriaanm/scala that referenced this pull request Dec 5, 2016
This likely regressed in scala#5294.

Review feedback from retronym:
 - Tie annotation triaging a bit closer together

durban kindly provided initial version of test/files/run/t10075.scala
And pointed out you must force `lazy val`, since `null`-valued field
is serializable regardless of its type.

Test test/files/run/t10075b courtesy of retronym
adriaanm added a commit to adriaanm/scala that referenced this pull request Apr 5, 2017
Fixes the problem reported with scala#5730 by xuwei-k in scala/scala-dev#352.

The problem was already present before the introduction of
`applyUnapplyMethodCompleter`, as 63f7b35 (in scala#5294) introduced
a similar bug where the `PolyTypeCompleter`'s `typeParams` override
was masked.
adriaanm added a commit to adriaanm/scala that referenced this pull request Apr 6, 2017
TODO: implement Lukas's observation and use CompleterWrapper for
other completers for polymorphic symbols

Fixes the problem reported with scala#5730 by xuwei-k in scala/scala-dev#352.

The problem was already present before the introduction of
`applyUnapplyMethodCompleter`, as 63f7b35 (in scala#5294) introduced
a similar bug where the `PolyTypeCompleter`'s `typeParams` override
was masked.
adriaanm added a commit to adriaanm/scala that referenced this pull request Apr 6, 2017
Fixes the problem reported with scala#5730 by xuwei-k in scala/scala-dev#352.

The problem was already present before the introduction of
`applyUnapplyMethodCompleter`, as 63f7b35 (in scala#5294) introduced
a similar bug where the `PolyTypeCompleter`'s `typeParams` override
was masked.
srowen referenced this pull request in apache/spark Mar 22, 2019
## What changes were proposed in this pull request?

- Support N-part identifier in SQL
- N-part identifier extractor in Analyzer

## How was this patch tested?

- A new unit test suite ResolveMultipartRelationSuite
- CatalogLoadingSuite

rblue cloud-fan mccheah

Closes #23848 from jzhuge/SPARK-26946.

Authored-by: John Zhuge <jzhuge@apache.org>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes worth highlighting in next release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants