Is this a flake? If so, could you please add "e2e flake: ..." in the title? That would help people find it.

Also, could you please paste the snippet of the log and the link to the failed tests? Thank you.

@pwittrock has anything changed with the PR node e2e job recently?

@kubernetes/goog-testing as fyi

I don't think it's a flake -- it seems to be happening in all PRs. Example: #23435

In a build yesterday, the first few lines in the build log showed

Cloning the remote Git repository
Cloning repository https://github.com/kubernetes/kubernetes
 > /usr/bin/git init /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes # timeout=10

Today, it's instead showing

Cloning repository https://github.com/kubernetes/kubernetes
 > /usr/bin/git init /var/lib/jenkins/workspace/node-pull-build-e2e-test # timeout=10
Fetching upstream changes from https://github.com/kubernetes/kubernetes

I'm not sure where the go/src/k8s.io/kubernetes went.

Looks like the jenkins config line to check out to a subdirectory was deleted. I added it back and things seem to be getting farther. Need to get the config into jjb so we have a reasonable way to track changes to the system.

+100 for JJB

I think I've traced the root cause. Spoiler: it's another bug in Jenkins (or one of its plugins)!

Basically, for this job, we've added the Check out to a sub-directory option to the Git SCM config, specifying go/src/k8s.io/kubernetes as the subdir. Saving the config seems to update the in-memory configuration for the job, but the config.xml file for the job on disk has no record of this setting.

Coincidentally, the PR Jenkins VM restarted for some reason around 7:30pm PDT yesterday. (No idea why, but it's sadly not uncommon.) When Jenkins came back, it lost the configuration parameter, since it wasn't stored in the disk-backed config.

We can expect this to break basically any time Jenkins restarts, now. Adding the config to JJB will help a bit (it should keep the in-memory version correct), though there may still be a short period where it'll fail between Jenkins starting and the config updater job running.

Aha - from https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin:

Version 2.4.4 (Mar 24, 2016)

• Fix git plugin 2.4.3 data loss when saving job definition (JENKINS-33695 and JENKINS-33564)
• Restore BuildData.equals lost in git plugin 2.4.2 revert mistake (JENKINS-29326)

We're currently running version 2.4.3 of the plugin, which explains the data loss. (Everything is terrible.)

/cc @kubernetes/sig-testing

Upgraded PR Jenkins to version 2.4.4 of the git plugin. The job's config.xml now contains the correct settings, too.

A bunch of the post-commit node e2e jobs have started failing because I restarted the Jenkins VM, and bad versions of these configs had been written with version 2.4.3 of the git plugin.

Unfortunately, the Jenkins job updater did not fix the problem. It seems that the XML it's producing looks like

  <scm class="hudson.plugins.git.GitSCM">
  ...
    <submoduleCfg class="list"/>
    <relativeTargetDir>go/src/k8s.io/kubernetes</relativeTargetDir>
    <reference/>

whereas the expected XML (looking at the Jenkins VM) looks like

  <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.4">
  ...
    <extensions>
      <hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
        <relativeTargetDir>go/src/k8s.io/kubernetes</relativeTargetDir>
      </hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
    </extensions>

We're using a pretty old version of the Jenkins job builder (1.3.1 - current is 1.4.1). However, it appears that the newest version still produces incorrect output, so we probably need to fix this.

@spxtr

I manually fixed kubelet-gce-e2e-ci in the Jenkins UI (hopefully it doesn't get reverted by the updater). I haven't touched the others yet.

Raw XML time :(

It this a reason why we're seeing 111s from gcloud calls all around the place?

I think the 111s are unrelated, due to the Jenkins VM being overloaded.