Skip to content

Poetry cannot install dvc[ssh] package #4683

New issue
New issue
Closed
#5305
Closed
Poetry cannot install dvc[ssh] package#4683
#5305
Labels
kind/bugSomething isn't working as expected
@lambda-science

Description

@lambda-science
lambda-science
opened on Oct 28, 2021
  • I am on the latest Poetry version.
    I have searched the issues of this repo and believe that this is not a duplicate.
    If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
  • OS version and name: Windows 10 21H1 with WSL2 (Ubuntu 20.04)
  • Poetry version: 1.1.11
  • Link of a Gist with the contents of your pyproject.toml file: https://pastebin.com/avWt6UNk

Issue

This issue has been cross-posted in the dvc git repo. Here: iterative/dvc#6886
dvc[ssh] can't be added to a project using Poetry env manager.
When I create my package environnement using Poetry adding dvc with [ssh] extra create a weird unrelated error.
Here is my full error log from poetry https://pastebin.com/MBveHXbW

Reproductibility

Minimal failing example:

  1. poetry init
  2. poetry add dvc[ssh]
  3. poetry install

Full reproductibilty: get the pyproject.toml at the bottom, get an python installation with poetry installed and simply run poetry install with the file in the working directory.

I have the weird feeling that this issue is due to the fact that you need to first install dvc package and THEN dvc[ssh] (bcrypt looks missing) as described here: https://dvc.org/doc/install/linux#install-with-pip
But that's not possible with Poetry, I can't have two line called "dvc" (one vanilla, one with the extra) and I can't have a line starting with dvc[ssh]

KeyError

  Package('bcrypt', '3.2.0')

  at ~/miniconda3/lib/python3.9/site-packages/poetry/puzzle/solver.py:270 in _solve

By removing the dvc with extras line from .toml file and doing poetry add dvc it works, but I can't use ssh plugin. Doing poetry add dvc[ssh] fails with the erros in the previous pastebin log.

EDIT: By using a fresh .toml file from poetry with only dvc[ssh] added, the same error arise.

Activity

lambda-science
added
kind/bugSomething isn't working as expected
status/triageThis issue needs to be triaged
on Oct 28, 2021
lambda-science
mentioned this on Oct 28, 2021
sjawhar

sjawhar commented on Nov 6, 2021

@sjawhar
sjawhar
on Nov 6, 2021

To help in debugging this, the bcrypt package doesn't get visited in the DFS here:

poetry/poetry/puzzle/solver.py

Lines 206 to 213 in 1da3e72

def depth_first_search(
source: "PackageNode", aggregator: Callable
) -> List[Tuple[Package, int]]:
back_edges = defaultdict(list)
visited = {}
topo_sorted_nodes = []
dfs_visit(source, back_edges, visited, topo_sorted_nodes)

In this case, bcrypt is a dependency as an extra of an extra of a dependency: iterative/dvc#6886 (comment)

ssh extra depends on sshfs[bcrypt] which depends on asyncssh[bcrypt] which depends on bcrypt. So bcrypt is an indirect dependency of dvc[ssh].

austinbutler

austinbutler commented on Dec 7, 2021

@austinbutler
austinbutler
on Dec 7, 2021 · edited by austinbutler

To add some more data here, same issue with celery[sqs] since Celery 5.2.

Add the following to pyproject.toml:

celery = { extras = ["sqs"], version = "~5.2.0" }

Run poetry lock:

  KeyError

  Package('urllib3', '1.26.7')

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:270 in _solve
      266│ 
      267│                 continue
      268│ 
      269│             final_packages.append(package)
    → 270│             depths.append(results[package])
      271│ 
      272│         # Return the packages in their original order with associated depths
      273│         return final_packages, depths

This is working fine with Celery 5.1.x. So comparing the changes relating to SQS extras, my guess is that not explicitly defining dependencies for the extra causes this.

Seems pip-tools also has problems installing celery[sqs] as of 5.2: celery/celery#7070.

Explicitly adding the kombu[sqs] dependency to pyproject.toml works around this for me.

[tool.poetry.dependencies]
python = "^3.9"
celery = { extras = ["sqs"], version = "~5.2.0" }
kombu = {extras = ["sqs"], version = "^5.2.2"}
austinbutler
mentioned this on Dec 7, 2021
lucidyan

lucidyan commented on Dec 16, 2021

@lucidyan
lucidyan
on Dec 16, 2021

For those who wanted fast solution for adding dvc[ssh] with poetry:

poetry add bcrypt 'dvc[ssh]'

Clarification: I suppose the square brackets should be inside the quotes in the shell. Also, poetry has some problems with the bcrypt-dependency resolving as pointed above, so you must add it explicitly.

skshetry

skshetry commented on Dec 20, 2021

@skshetry
skshetry
on Dec 20, 2021 · edited by skshetry

We are also getting a similar issue with dvc[hdfs]. Resolving numpy versions is slow too.

  KeyError

  Package('pyarrow', '6.0.1')

  at .env/lib/python3.8/site-packages/poetry/puzzle/solver.py:270 in _solve
      266│ 
      267│                 continue
      268│ 
      269│             final_packages.append(package)
    → 270│             depths.append(results[package])
      271│ 
      272│         # Return the packages in their original order with associated depths
      273│         return final_packages, depths
      274│
Full log and stack trace 
PyPI: No release information found for pyarrow-0.1.0, skipping
PyPI: 9 packages found for pyarrow >=1
PyPI: Getting info for pyarrow (6.0.1) from PyPI
   1: fact: pyarrow (6.0.1) depends on numpy (>=1.16.6)
   1: selecting pyarrow (6.0.1)
   1: derived: numpy (>=1.16.6)
PyPI: No release information found for numpy-0.9.6, skipping
PyPI: No release information found for numpy-0.9.8, skipping
PyPI: No release information found for numpy-1.0.3, skipping
PyPI: No release information found for numpy-1.0.4, skipping
PyPI: No release information found for numpy-1.0b1, skipping
PyPI: No release information found for numpy-1.0b4, skipping
PyPI: No release information found for numpy-1.0b5, skipping
PyPI: No release information found for numpy-1.0rc1, skipping
PyPI: No release information found for numpy-1.0rc2, skipping
PyPI: No release information found for numpy-1.0rc3, skipping
PyPI: No release information found for numpy-1.1.1, skipping
PyPI: No release information found for numpy-1.2.0, skipping
PyPI: No release information found for numpy-1.2.1, skipping
PyPI: No release information found for numpy-1.4.0, skipping
PyPI: 29 packages found for numpy >=1.16.6
PyPI: Getting info for numpy (1.21.5) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.21.5.zip
   1: fact: numpy (1.21.5) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.5)
PyPI: Getting info for numpy (1.21.4) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.21.4.zip
   1: fact: numpy (1.21.4) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.4)
PyPI: Getting info for numpy (1.21.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.21.3.zip
   1: fact: numpy (1.21.3) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.3)
PyPI: Getting info for numpy (1.21.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.21.2.zip
   1: fact: numpy (1.21.2) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.2)
PyPI: Getting info for numpy (1.21.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: numpy-1.21.1.zip
   1: selecting numpy (1.21.1)
   1: Version solving took 92.668 seconds.
   1: Tried 1 solutions.

  Stack trace:

  8  .env/lib/python3.8/site-packages/clikit/console_application.py:131 in run
      129│             parsed_args = resolved_command.args
      130│ 
    → 131│             status_code = command.handle(parsed_args, io)
      132│         except KeyboardInterrupt:
      133│             status_code = 1

  7  .env/lib/python3.8/site-packages/clikit/api/command/command.py:120 in handle
      118│     def handle(self, args, io):  # type: (Args, IO) -> int
      119│         try:
    → 120│             status_code = self._do_handle(args, io)
      121│         except KeyboardInterrupt:
      122│             if io.is_debug():

  6  .env/lib/python3.8/site-packages/clikit/api/command/command.py:171 in _do_handle
      169│         handler_method = self._config.handler_method
      170│ 
    → 171│         return getattr(handler, handler_method)(args, io, self)
      172│ 
      173│     def __repr__(self):  # type: () -> str

  5  .env/lib/python3.8/site-packages/cleo/commands/command.py:92 in wrap_handle
       90│         self._command = command
       91│ 
    →  92│         return self.handle()
       93│ 
       94│     def handle(self):  # type: () -> Optional[int]

  4  .env/lib/python3.8/site-packages/poetry/console/commands/add.py:173 in handle
      171│             self._installer.whitelist([r["name"] for r in requirements])
      172│ 
    → 173│             status = self._installer.run()
      174│         except BaseException:
      175│             # Using BaseException here as some exceptions, eg: KeyboardInterrupt, do not inherit from Exception

  3  .env/lib/python3.8/site-packages/poetry/installation/installer.py:103 in run
      101│         local_repo = Repository()
      102│ 
    → 103│         return self._do_install(local_repo)
      104│ 
      105│     def dry_run(self, dry_run=True):  # type: (bool) -> Installer

  2  .env/lib/python3.8/site-packages/poetry/installation/installer.py:235 in _do_install
      233│             )
      234│ 
    → 235│             ops = solver.solve(use_latest=self._whitelist)
      236│         else:
      237│             self._io.write_line("Installing dependencies from lock file")

  1  .env/lib/python3.8/site-packages/poetry/puzzle/solver.py:65 in solve
       63│         with self._provider.progress():
       64│             start = time.time()
    →  65│             packages, depths = self._solve(use_latest=use_latest)
       66│             end = time.time()
       67│ 

  KeyError

  Package('pyarrow', '6.0.1')

  at .env/lib/python3.8/site-packages/poetry/puzzle/solver.py:270 in _solve
      266│ 
      267│                 continue
      268│ 
      269│             final_packages.append(package)
    → 270│             depths.append(results[package])
      271│ 
      272│         # Return the packages in their original order with associated depths
      273│         return final_packages, depths
      274│
skshetry
mentioned this on Dec 21, 2021
skshetry
mentioned this on Jan 12, 2022
dimbleby
mentioned this on Mar 13, 2022
pmrowla
mentioned this on Mar 17, 2022
radoering
closed this as completedin #5305on Mar 20, 2022

3 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      Poetry cannot install dvc[ssh] package · Issue #4683 · python-poetry/poetry