with locally-build builder image, make build-debs completes locally. CI translation-tests and staging-test-with-rebase checks are consistently failing however.

On further investigation, translation-tests was just flaky. staging-test-with-rebase is consistently timing out on the molecule verify phase, looks like the test_apache_logging_journalist_interface test is to blame. Am adding a skip temporarily to see if CI passes with that in place. Will poke at the test in the meantime to see if it can be gotten working locally.

Looks like a new file is added by venv: /opt/venvs/securedrop-app-code/pyvenv.cfg, which falls foul of the apache2 apparmor profile. It would be worth investigating if it's safe to include, and what else if anything has been added.

There are some significant differences between securedrop-app-code packages built by this branch and current develop:

• develop virtualenv includes a python3.5 binary under /opt/venvs/securedrop-app-code/bin and symlinks to it, while fix_debian_package symlinks to /usr/bin/python3.5
• develop includes the the wheel package of cryptography-2.6.1, whilefix_debian_package includes the egg package.
• develop includes pip-20.0.2 and setuptools 45.1, while fdp includes 8.1.1 and 20.7.0 respectively
• develop includes wheel 0.34.2, while this package is missing from fdp.

Probably worth discussing at least the first 2 of these, and pinning pip and setuptools.

The test_apache_journalist_interface_vhost test is stalling waiting for a curl as root to 127.0.0.1:8080 - same behaviour is observed logged in to app-staging.

I can explain a couple of those:

• develop virtualenv includes a python3.5 binary under /opt/venvs/securedrop-app-code/bin and symlinks to it, while fix_debian_package symlinks to /usr/bin/ptthon3.5

I believe this is expected with the venv module instead of virtualenv, and acceptable. The behavior could be changed to force symlinks by passing --symlinks to the venv module, except we apparently can't do that with dh-virtualenv 0.11.0.

• develop includes pip-20.0.2 and setuptools 45.1, while fdp includes 8.1.1 and 20.7.0 respectively

These are the versions in /usr/share/python-wheels, which the venv module is installing on the fdp branch.

The test_apache_journalist_interface_vhost test is stalling waiting for a curl as root to 127.0.0.1:8080 - same behaviour is observed logged in to app-staging.

Cannot reproduce. ☹️ Followed the test plan—with a single exception: used $(date +%Y_%m_%d) instead of 2020_01_29 in the custom build command—built packages, provisioned staging VMs. Confirmed that the following worked without issue:

• Load Journalist Interface in Tor Browser (after adding auth token to torrc)
• curl http://127.0.0.1:8080 as root on app-staging exits zero
• molecule verify -s libvirt-staging-xenial passes, after removing the pytest skip marker

Then I destroyed the VMs, rebooted host machine, and tried again. All the above checks passed without issue again. As a sanity check, made sure to confirm that e.g. /opt/venvs/securedrop-app-code/pyvenv.cfg was present, so I'm certainly using the new packaging logic on this branch.

Have not yet investigated the pip/pkg_resources/setuptools version variation.

Thanks for taking a look @conorsch - turns out it was the apparmor deny that was the problem. App was bombing out trying to load pyvenv.cfg. I've removed the skip.

During sprint planning, we identified one remaining task prior to merge: as @rmol flagged, we're downgrading both pip & setuptools in this branch. Would pinning the pip & setuptools dependencies to the versions (to 20.0.2 and 45.1, respectively) resolve? It may be that the packaging logic changes here force use of system deps for those packages—adding a pin and rebuilding to evaluate would shed light on how substantial the changes are.

@zenmonkeykstop will take a look at the pin strategy and report back to inform final review.

I may have missed this when we discussed it, but do we have any concern about the wheel vs. egg packaging changes Kev flagged here? Knowing little about both formats, it seems innocuous enough, but just wanted to highlight it in case there may be unintended side effects.

Picked apart the securedrop-app-code packages built on this branch and on develop, and verified all of @zenmonkeykstop's findings in #5110 (comment) . Have not yet pinned pip & setuptools, will try that next, then rebuild and compare the results of the third packages and see if they match the either of the other two. I suspect that if we ensure recent versions of both those packages, we'll see wheels instead of eggs again on e.g. cryptography & python_dateutil.

Spent a bit more time with the packaging logic today. @kushaldas what do you think of the following suggestion for the getting this PR in?

1. Pin setuptools & pip in the app-code requirements, as suggested by @zenmonkeykstop & @rmol.
2. Ensure that --allow-unsafe is passed to pip-compile for the app-code requirements, otherwise [pinned versions of setuptools & pip will be ignored.
3. Ensure that --no-deps is passed an extra pip arg

In other words, we'd be removing use of the --builtin-venv flag, and the corresponding python3-venv package import. Please let me know if you disagree, but I don't see any pressing reason to switch to --builtin-venv for the 1.2.1 point release.

Today I made a timeboxed attempt at the above, which you can find here: https://github.com/freedomofpress/securedrop/tree/fix_debian_packaging_again When I pick apart the resultant securedrop-app-code packages (one built from the develop branch, the other from the fix_debian-packaging_again branch), I see ostensibly similar packages:

==================
Report for build1
Description: Standard deb build from develop branch
==================
Number of .whl files: 18
Number of .egg dirs: 36
Versions:
    pip 20.0.2
    setuptools 45.2.0
    wheel 0.34.2

[..snip...]

==================
Report for build7
Description: pins pip, setuptools, and pip-tools in all reqs files
used pip-compile 4.4.0 LOCALLY to regen reqs files
==================
Number of .whl files: 18
Number of .egg dirs: 37
Versions:
    pip 20.0.2
    setuptools 45.2.0
    wheel 0.34.2

The off-by-one difference in eggs shipped is a setuptools egg. That's much closer to the original packaging logic, and so fairly predictable. I'm not appending those changes to your branch here, because I ended bumping pip-compile to 4.4.0 and running it locally (i.e. on my host) to get it working.

Traceback (most recent call last):
  File "/home/user/.virtualenvs/sd/bin/pip-compile", line 5, in <module>
    from piptools.scripts.compile import cli
  File "/home/user/.virtualenvs/sd/lib/python3.7/site-packages/piptools/scripts/compile.py", line 11, in <module>
    from .._compat import InstallCommand, install_req_from_line, parse_requirements
  File "/home/user/.virtualenvs/sd/lib/python3.7/site-packages/piptools/_compat/__init__.py", line 7, in <module>
    from .pip_compat import (
  File "/home/user/.virtualenvs/sd/lib/python3.7/site-packages/piptools/_compat/pip_compat.py", line 40, in <module>
    PackageFinder = do_import("index", "PackageFinder")
  File "/home/user/.virtualenvs/sd/lib/python3.7/site-packages/piptools/_compat/pip_compat.py", line 24, in do_import
    return getattr(imported, package)
AttributeError: module 'pip._internal.index' has no attribute 'PackageFinder'

In other words, we'd be removing use of the --builtin-venv flag, and the corresponding python3-venv package import. Please let me know if you disagree, but I don't see any pressing reason to switch to --builtin-venv for the 1.2.1 point release.

I am okay with this.

Hopefully, I think I found the solution, have to test first.

The error in your branch with the make update-python3-requirements is because of a API change inside of click. I found it as a chicken and egg problem, how to generate the dependency as to do so we need the new dependency inside of the docker image. I did the update manually via a click upgrade. Now testing the change, I will push to a different branch.

I will be adding click==7.0 into all of our requirements to solve the issue, one related issue with explanation.

@conorsch https://github.com/freedomofpress/securedrop/tree/fix_debian_packaging_again_and_again branch has the fix, if you are okay with it, we can use that branch, and repush into this branch to get things going.

After a lot of fighting over this, here are a few findings:

• virtualenv is always pulling the latest package set (for pip, setuptools, wheel) from pypi.
• The error which @rmol and @conorsch also saw, is due to the pip 20.0.2 version, I have asked the upstream to verify. https://pad.riseup.net/p/pip_20.0.2-keep has a better readable version of the error. I also filed pip is failing to verify the hashes of pip package itself. pypa/pip#7748

Now a few things we can do:

• Do not pin pip and setuptools and wheel
• Use the branch as it is (without --require-hashes)
• Go back to the original PR with venv module, and pass along the latest pip and setuptools and wheel via actual packages, we can ADD them into the build container in /usr/share/python-wheels/ directory. Because that way we can control what will there in that default virtual environment.

I like the 3rd option (but for 1.3.0). It introduces some duplication as it involves tracking dependencies both in requirements files and in the build Dockerfile, but the same would be true if a newer version of dh_virtualenv was available and it was being passed explicit pip etc versions.

@kushaldas has committed to coordinating further w/ upstream and recommending next steps for this change.

@kushaldas Is there any update on this yet?

@kushaldas Is there any update on this yet?

I have asked to retry the whole process again. I will do that sometimes this week and get back to the upstream and also here.

(We'll continue to monitor the situation, but we've agreed that we're not able to make a change here for 1.3.0.)

I have rebased and updated branch https://github.com/freedomofpress/securedrop/tree/fix_debian_packaging_again_and_again

This branch has some extra updates in the requirements files, we can fix those later in a different commit (after venv works). Those came in because I ran pip-compile command.

To use the venv inside of that, we still need to fix the molecule/builder-xenial/Dockerfile first.

Current issuse: ensurepip is not working inside of that container.

python3 -m venv /tmp/woo
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/tmp/woo/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

To work on this brach:

• mkdir wheels
• Save the following in in a download_wheels.txt file, so that we can download the wheels.

wheel==0.34.2 \
    --hash=sha256:8788e9155fe14f54164c1b9eb0a319d98ef02c160725587ad60f14ddc57b6f96 \
    --hash=sha256:df277cb51e61359aba502208d680f90c0493adec6f0e848af94948778aed386e

# The following packages are considered to be unsafe in a requirements file:
pip==20.0.2 \
    --hash=sha256:4ae14a42d8adba3205ebeb38aa68cfc0b6c346e1ae2e699a0b3bad4da19cef5c \
    --hash=sha256:7db0c8ea4c7ea51c8049640e8e6e7fde949de672bfa4949920675563a5a6967f
setuptools==46.1.3 \
    --hash=sha256:4fe404eec2738c20ab5841fa2d791902d2a645f32318a7850ef26f8d7215a8ee \
    --hash=sha256:795e0475ba6cd7fa082b1ee6e90d552209995627a2a227a47c6ea93282f4bfb1

• then download the wheels

pip download \
    --only-binary=:all: \
    --platform linux_x86_64 \
    --python-version 3 \
    --implementation cp \
    --abi cp35m -r download_wheels.txt 

• make build-container , this will build the container image with a tag containing the date. Next, inside of the docker container, try the following.

$ docker run -it --rm quay.io/freedomofpress/sd-docker-builder-xenial:2020_04_23
python3 -m venv /tmp/woo

^^ Python3 should be able to create a proper virtualenv. Instead I am getting the error pasted at the beginning of this comment.

Most probably we will have to install python3-distuils too.

This is now implemented in develop via #5484