On Mon, Sep 15, 2014 at 02:43:48PM -0700, kiilerix wrote:

A fun self referencing problem:

File '/tmp/kallithea/kallithea/controllers/admin/settings.py', line 211 in settings_mapping
  install_git_hook=install_git_hooks)
File '/tmp/kallithea/kallithea/lib/utils.py', line 531 in repo2db_mapper
  git_repo._update_server_info()
File '/tmp/kallithea/kallithea/lib/vcs/backends/git/repository.py', line 697 in _update_server_info
  update_server_info(self._repo)
File '/tmp/kallithea/dist/v/lib/python2.7/site-packages/dulwich-0.9.7-py2.7-linux-x86_64.egg/dulwich/server.py', line 936 in update_server_info
  "".join(generate_info_refs(repo)))
File '/tmp/kallithea/dist/v/lib/python2.7/site-packages/dulwich-0.9.7-py2.7-linux-x86_64.egg/dulwich/repo.py', line 692 in _put_named_file
  f = GitFile(os.path.join(self.controldir(), path), 'wb')
File '/tmp/kallithea/dist/v/lib/python2.7/site-packages/dulwich-0.9.7-py2.7-linux-x86_64.egg/dulwich/file.py', line 84 in GitFile
  return _GitFile(filename, mode, bufsize)
File '/tmp/kallithea/dist/v/lib/python2.7/site-packages/dulwich-0.9.7-py2.7-linux-x86_64.egg/dulwich/file.py', line 109 in __init__
  os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, "O_BINARY", 0))
OSError: [Errno 2] No such file or directory: '/tmp/kallithea/dist/v/lib64/python2.7/site-packages/dulwich-0.9.7-py2.7-linux-x86_64.egg/dulwich/tests/data/repos/simple_merge.git/info/refs.lock'

Creating the info directory makes it work.

I guess it should create the dir if it is missing ... or skip locking.
Agreed. It should create the directory when it is updating the references file
and the directory doesn't exist.

• and arguably, it is confusing that dulwich itself contains (almost complete) raw git repos. I also wouldn't expect the pip package shouldn't contain these test data.
  We ship the raw git repos since we'd like to make sure dulwich works with
  the canonical git repo format, and not just with git-like repositories that
  were created by Dulwich itself.

Options I can think of: not ship test data and disable some tests in pip.
This seems like a hassle. The other option is to just not ship any of the tests
in pip.

Jelmer

While it is ugly to have binaries and zip/tar files in a repository, I would say that the git repo pretty much is a "binary". Storing it in a zip would make it more clear that it is something static that shouldn't be changed ... and prevent searches and refactorings from finding it.

On Wed, Sep 17, 2014 at 08:29:10AM -0700, kiilerix wrote:

While it is ugly to have binaries and zip/tar files in a repository, I would say that the git repo pretty much is a "binary". Storing it in a zip would make it more clear that it is something static that shouldn't be changed ... and prevent searches and refactorings from finding it.
Okay, we can certainly have a look at sticking these files in a tar file.

Obviously, that was just my thoughts and it is completely up to you how you do it.

Another thought, though: How come I saw this failure and it seems like nobody else had seen it before? I assume the problem also would show up when running the tests using this test data? So nobody ever ran the tests from pip? And probably also never from source? That might indicate that there is no reason to have the tests in pip ... or that having tests without having procedure for running them doesn't add much value.

Which brings me to: I see the value in having tests in the source repo and in running them when/before creating the release tar ball and uploading to pip. I am however not convinced I see any value in having them installed by pip. Why should anybody ever run the tests from pip? (I am however not that familiar with pip and don't know what people ususally do.)

On Wed, Sep 17, 2014 at 03:11:13PM -0700, kiilerix wrote:

Obviously, that was just my thoughts and it is completely up to you how you do it.

Another thought, though: How come I saw this failure and it seems like nobody else had seen it before? I assume the problem also would show up when running the tests using this test data? So nobody ever ran the tests from pip? And probably also never from source? That might indicate that there is no reason to have the tests in pip ... or that having tests without having procedure for running them doesn't add much value.

Which brings me to: I see the value in having tests in the source repo and in running them when/before creating the release tar ball and uploading to pip. I am however not convinced I see any value in having them installed by pip. Why should anybody ever run the tests from pip? (I am however not that familiar with pip and don't know what people ususally do.)

The reason these files are not tarred up at the moment is simply convenience. :-) Having the actual bare git repositories there makes it easier to introspect when debugging issues with tests, it reduces the complexity in the tests (since there is no need to unpack) and it makes it possible to easily manipulate them with git itself rather than having to unpack, manipulate and repack.

I'm also not a pip user, and I don't remember anybody mentioning that the tests fail. But, as you say, that is probably because nobody has run the tests after installing from pip.

Personally I like the idea of having the tests available of software I build and install, since that allows me to verify that it is properly installed and that it works on my platform. As a distro developer I don't tend to install from source often, though.

If the norm on pip is that tests are not installed, then I would be
happy to have dulwich conform to that for pip.

We'd need to make sure nobody actually relies on anything under
dulwich/tests before we stop installing it. E.g. bzr-git imports
from dulwich.tests.test_object_store.

Jelmer

Fixed the running of the testsuite from installed dulwich.