Tracking Windows errors #6262
Comments
output was not createdError message:Affected commits:Attempted resolution:1 Clearing the cache through a separate PR NotesSeems to be reproducible and affects all builds. Did cleaning the cache fix the issue?1 seems to have worked for some PRs but not for all of them unclear why. |
undeclared inclusionsError message:Affected commits:b9fbba7 [build step] [test step] [exec log too large, 1.2GB] Note: b9fbba7 succeeded at the exact same build on the previous day: [build step], [test step], [Bazel execution log (zip)]. NotesWe’ve seen this issue numerous times before and believe it is due to working directories leaking into the cache. Did cleaning the cache fix the issue?Fix 2 from the previous issue also seems to workaround this one so far. |
undeclared inclusionsSame error as before but this is after we reset the cache. ResolutionSwitch to modifying the cache URL instead of relying on |
output was not createderror messageERROR: D:/a/2/s/compiler/scenario-service/protos/BUILD.bazel:67:1: output 'compiler/scenario-service/protos/_obj/scenario_service_haskell_proto/ScenarioService.o' was not created Affected commitshttps://dev.azure.com/digitalasset/daml/_build/results?buildId=45869&view=results ResolutionRerun succeded but on a different node. Builds on the same node failed. NotesCaches are separated by working directories. This only seems to have affected the node that ran the daily compat check which runs in a different working directory. |
output was not createderror messageAffected commitsf2cd820 build ResolutionBump Windows cache and |
Output was not createderror messageERROR: D:/a/1/s/compiler/damlc/daml-package-config/BUILD.bazel:9:1: output 'compiler/damlc/daml-package-config/_obj/daml-package-config/DA/Daml/Package/Config.o' was not created Affected commitsResolutionWe’ve tried selectively cleaning the cache via |
Output was not createderror messageAffected commitsResolutionSelectively cleaning the cache did not seem to help this time, so we instead bumped the cache counter (#6383), which worked. |
Output was not createderror messageAffected commitsResolution |
Output was not createdError messageERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Alpha.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Base.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Numeric.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Optics.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Pretty.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Recursive.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Subst.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: output 'compiler/daml-lf-ast/_obj/daml-lf-ast/DA/Daml/LF/Ast/Type.o' was not created ERROR: D:/a/1/s/compiler/daml-lf-ast/BUILD.bazel:6:1: not all outputs were created or valid ERROR: D:/a/1/s/compiler/damlc/BUILD.bazel:10:1 not all outputs were created or valid Affected commitsResolutionpending |
not all outputs were created or validHappened on the compat build for a 1.5-snapshot update PR. Machine was Also happened on #6884 on Also happened (twice) on ResolutionPending: trying to take out just that node for now, but may need to escalate to cache bumping. |
Output was not createdHappened on #7039 ResolutionTook out the affected node. |
Output was not createdHappened on rerun of #7007 run in ResolutionGoing to try resetting the cache, this is too annoying. |
Output was not createdResolutionDisabled machine, let’s see if it helps. |
Output was not createdWindows compat job. Killed the machine (PN4D). 🤞 |
Output not createdThis time in the Windows job. Killed the machine (Q1S0). |
Output not createdSame... Machine CW68 this time. Also killed. |
|
Note: it looks like rebasing made the above disappear. The PR on which those appeared (#7146) changes the stdlib so triggers targets we don't often rebuild. |
Output not created, masterOn commit c997f5d, node PN4D Node killed. |
Output not created, masterOn commit f54e193, node CW68. Node killed. |
|
Above kept happening so we've decided to bump the cache (#7212). |
Output not created, #7385Machine CW68: Disabled the machine for now. |
Output not created, masterThis is on |
Output not created, #7517On commit |
Output not created, #7542Interestingly this is the one that was modified in this PR. I’ve taken out the node for now, let’s see if that helps. Edit: It didn’t, took out a second node. |
Output not created, all over the placeLoads of this one today. We decided to go for a more targeted approach for the fix and are currently trying to only force a rebuild of the one file that fails: #7613. |
Output not created, #7767.On commit deb1c58. |
|
Closing since it seems to have done the job. We now have a workaround for the caching issue in place via #8668. Note that this does not stop the error from happening but it stops it from breaking the cache so we can simply take out the node and move on. We could automate that part as well but atm it seems to happen once a week or so at most so not that urgent of an issue. |
We have seen many errors on Windows recently. This issue is trying to keep track of them, so we can have a better view on what kinds of issues we are seeing and how frequent they are, in the hope that this will help us formulate concrete plans to solve them.
The text was updated successfully, but these errors were encountered: