Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental compilation recompiles all the files every time #923

Closed
piotrga opened this issue Oct 17, 2013 · 8 comments
Closed

Incremental compilation recompiles all the files every time #923

piotrga opened this issue Oct 17, 2013 · 8 comments
Assignees
@harrah
Milestone
0.13.1

Comments

@piotrga
Copy link

piotrga commented Oct 17, 2013

This is what I do:

  1. clean
  2. compile
  3. compile

No. 3 recompiles all the sources, even though I haven't touched the files.
I am on scala 2.10.2, sbt 0.13.0 (but it happens on the 0.12.4 and 0.13.Beta2 too)

See my build.sbt:

name := "model_fx"

version := "1.0"

scalaVersion := "2.10.2"

dependencyClasspath in Compile ++= EclipseClasspath(Seq("lib")).classpath

scalaSource in Compile := file( "main" )

autoCompilerPlugins := true

libraryDependencies ++= Seq(
  compilerPlugin("org.scala-lang.plugins" % "continuations" % "2.10.2")
)

logLevel := Level.Debug

scalacOptions += """-Xplugin:D:\x\y\our-own-compiler-plugin.jar"""

scalacOptions ++= "-P:entity:asyncOff:v1 -target:jvm-1.6 -language:_".split(" ").toSeq

And this is the sbt log when I do the step 3 (so the sources are fully compiled by this time):

>
> compile
[debug]
[debug] Initial source changes:
[debug]         removed:Set(D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\StringListExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\JumpVolMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\TradeWeightMap.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\TradeModelExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\PCMVMRMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\ModelPricingWeig
ht.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMarketModel.scala)
[debug]         added: Set(main\rice\distribution\fx\TradeWeightMap.scala, main\rice\cm\model\fx\PCMVMRMarketModel.scala, main\rice\cm\model\fx\SinglePairMarketModel.scala, main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, main\rice\cm\model\fx\JumpVolMarketModel.scala, main\rice\extensions\TradeModelExtensions.scala, main\rice\extensions\StringListExtensions.scala, main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, main\rice\distribution\fx\ModelPricingWeight.scala)
[debug]         modified: Set()
[debug] Removed products: Set()
[debug] Modified external sources: Set()
[debug] Modified binary dependencies: Set()
[debug] Initial directly invalidated sources: Set(D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\StringListExtensions.scala, main\rice\distribution\fx\TradeWeightMap.scala, main\rice\cm\model\fx\PCMVMRMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\JumpVolMarketModel.scala, main\rice\cm\model\fx\SinglePairMarketModel.scala, main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, main\rice\cm\model\fx\JumpVolMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\TradeWeightMap.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\Tra
deModelExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\PCMVMRMarketModel.scala, main\rice\extensions\TradeModelExtensions.scala, main\rice\extensions\StringListExtensions.scala, main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, main\rice\distribution\fx\ModelPricingWeight.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\ModelPricingWeight.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMarketModel.scala)
[debug]
[debug] Sources indirectly invalidated by:
[debug]         product: Set()
[debug]         binary dep: Set()
[debug]         external source: Set()
[debug] All initially invalidated sources: Set(D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\StringListExtensions.scala, main\rice\distribution\fx\TradeWeightMap.scala, main\rice\cm\model\fx\PCMVMRMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\JumpVolMarketModel.scala, main\rice\cm\model\fx\SinglePairMarketModel.scala, main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, main\rice\cm\model\fx\JumpVolMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\TradeWeightMap.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\TradeM
odelExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\PCMVMRMarketModel.scala, main\rice\extensions\TradeModelExtensions.scala, main\rice\extensions\StringListExtensions.scala, main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, main\rice\distribution\fx\ModelPricingWeight.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\ModelPricingWeight.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMarketModel.scala)
[debug] Recompiling all 9 sources: invalidated sources (18) exceeded 50.0% of all sources
[info] Compiling 9 Scala sources to D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\target\scala-2.10\classes...
[debug] Getting compiler-interface from component compiler for Scala 2.10.2
[debug] Getting compiler-interface from component compiler for Scala 2.10.2
[debug] Running cached compiler 9600450, interfacing (CompilerInterface) with Scala compiler version 2.10.2
[debug] Calling Scala compiler with arguments  (CompilerInterface):
[debug]         -Xplugin:D:\stratosphere\.stratosphere\dist\optimus\PROJ\prime\2013.10.11-4\common\lib\entityplugin\optimus.prime.entityplugin.jar
[debug]         -P:entity:asyncOff:v1
[debug]         -target:jvm-1.6
[debug]         -language:_
[debug]         -Xplugin:D:\piotrga\.ivy2\cache\org.scala-lang.plugins\continuations\jars\continuations-2.10.2.jar
[debug]         -bootclasspath
[debug]         C:\Progra~1\Java\jdk1.7.0_17\jre\lib\resources.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\rt.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\sunrsasign.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\jsse.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\jce.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\charsets.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\lib\jfr.jar;C:\Progra~1\Java\jdk1.7.0_17\jre\classes;D:\piotrga\.sbt\boot\scala-2.10.2\lib\scala-library.jar
[debug]         -classpath
[debug]         D:\stratosphere\gabryanc\gabryanc_ws\bundles\ (...removed to shorten the log...)
[warn] D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala:40: imported `EngineInputs' is permanently hidden by definition of object EngineInputs in package fx
[warn] import rice.cm.model.fx.EngineInputs
[warn]                         ^
[warn] there were 1 deprecation warning(s); re-run with -deprecation for details
[warn] two warnings found
[debug] Scala compilation took 5.706918459 s
[debug] Invalidated by transitive public inheritance: Set(main\rice\distribution\fx\TradeWeightMap.scala, main\rice\cm\model\fx\PCMVMRMarketModel.scala, main\rice\cm\model\fx\SinglePairMarketModel.scala, main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, main\rice\cm\model\fx\JumpVolMarketModel.scala, main\rice\extensions\TradeModelExtensions.scala, main\rice\extensions\StringListExtensions.scala, main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, main\rice\distribution\fx\ModelPricingWeight.scala)
[debug] Invalidated by direct dependency: Set()
[debug] New invalidations:
[debug]         Set()
[debug] Previously invalidated, but (transitively) depend on new invalidations:
[debug]         Set()
[success] Total time: 6 s, completed 17-Oct-2013 10:34:51
>
@gkossakowski
Copy link
Contributor

Hi,

I believe the culprit is first few lines line:

[debug] removed:Set(D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\StringListExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\JumpVolMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\TradeWeightMap.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\extensions\TradeModelExtensions.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\PCMVMRMarketModel.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\distribution\fx\ModelPricingWeig
ht.scala, D:\stratosphere\gabryanc\gabryanc_ws\bundles\risk_trunk_fx_dev\model_fx\main\rice\cm\model\fx\SinglePairMarketModel.scala)
[debug] added: Set(main\rice\distribution\fx\TradeWeightMap.scala, main\rice\cm\model\fx\PCMVMRMarketModel.scala, main\rice\cm\model\fx\SinglePairMarketModel.scala, main\rice\cm\pvrm\fx\PVRMRulesImpl.scala, main\rice\cm\model\fx\JumpVolMarketModel.scala, main\rice\extensions\TradeModelExtensions.scala, main\rice\extensions\StringListExtensions.scala, main\rice\cm\model\fx\SinglePairMonteCarloCalibratedMarketModel.scala, main\rice\distribution\fx\ModelPricingWeight.scala)

Not how first set of files uses fully qualified paths and the other doesn't. Are you running using some uncommon filesystem setup (e.g. are symlinks or network file system involved)?

@piotrga
Copy link
Author

piotrga commented Oct 17, 2013

It is on the freaking win7, so no symlinks I suppose...
I've overwritten the default source path though... maybe that's it... I'll give it a try!

@piotrga
Copy link
Author

piotrga commented Oct 17, 2013

that was it!:
scalaSource in Compile := file( "main" )

Now how do we go about it - I dont want everybody to hate me for changing the project directory structure. And this is clearly an oversight in the guts of sbt... :)

@gkossakowski
Copy link
Contributor

it's an oversight because we probably should always convert all paths to canonical from. @harrah, can you confirm if this is true?

However, you can easily work-around your problem by saying scalaSource in Compile := file(baseDirectory.value / "main" ) which will make scalaSource to use absolute path again.

@piotrga
Copy link
Author

piotrga commented Oct 17, 2013

Thank you Grzegorz - this worked!:)

You dont need the file bit:
scalaSource in Compile := baseDirectory.value / "main"

@harrah
Copy link
Member

harrah commented Oct 17, 2013

We do not convert relative paths because we don't have the information about the base directory. The current working directory is not the right one to use because that can change. At best, we can generate an error or provide some wrapper macro absFile(rel: String) = baseDirectory.value / rel (that only works in settings). See also https://groups.google.com/d/msg/simple-build-tool/w7hHi-eAn9c/ao4tl5gOaxEJ.

@gkossakowski
Copy link
Contributor

What about generating a runtime error whenever we see relative file like in the example above?

@harrah
Copy link
Member

harrah commented Oct 17, 2013

Yeah, that's what I meant by generating an error. The problem is you have to insert that check a bunch of places. Since this particular failure mode seems somewhat common, putting them in for sources might be a good start.

@harrah harrah closed this as completed in 2f683ef Nov 9, 2013
@ghost ghost assigned harrah Nov 9, 2013
harrah added a commit that referenced this issue Nov 11, 2013
@harrah
Fix source-dependencies/relative-source-error to use explicit reloads…
4f81512 
…. Ref #923.
harrah added a commit that referenced this issue Nov 19, 2013
@harrah
Generate an error when the incremental compiler is given relative sou…
de674a2 
…rce files. Fixes #923.

Review by @gkossakowski
harrah added a commit that referenced this issue Nov 19, 2013
@harrah
Fix source-dependencies/relative-source-error to use explicit reloads…
2fef20e 
…. Ref #923.
harrah added a commit that referenced this issue Mar 21, 2014
@harrah @eed3si9n
Generate an error when the incremental compiler is given relative sou…
08f4da7 
…rce files. Fixes #923.

Review by @gkossakowski

Conflicts:
	sbt/src/sbt-test/source-dependencies/relative-source-error/test
harrah added a commit that referenced this issue Mar 21, 2014
@harrah @eed3si9n
Fix source-dependencies/relative-source-error to use explicit reloads…
beddc73 
…. Ref #923.

Conflicts:
	sbt/src/sbt-test/source-dependencies/relative-source-error/test
@eed3si9n eed3si9n mentioned this issue Mar 22, 2019
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@harrah harrah

Labels
None yet
Projects
None yet
Milestone
0.13.1
Development

No branches or pull requests
3 participants
@gkossakowski @harrah @piotrga