Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tensor not found in Tensorflow module #1820

Open
1 of 6 tasks
erlebach opened this issue Oct 8, 2023 · 4 comments
Open
1 of 6 tasks

Tensor not found in Tensorflow module #1820

erlebach opened this issue Oct 8, 2023 · 4 comments

Comments

@erlebach
Copy link

erlebach commented Oct 8, 2023 •
edited
Loading

Initial Checks

  • I have read and followed the docs and still think this is a bug

Description

I am using Poetry, and I added the docarray module, version 0.39 . Here is the pyproject.toml I am working with:

[tool.poetry]
name = "langchain-monorepo"
version = "0.0.1"
description = "LangChain mono-repo"
authors = []
license = "MIT"
readme = "README.md"
repository = "https://www.github.com/langchain-ai/langchain"

[tool.poetry.dependencies]
python = ">=3.9,<4.0"
python-dotenv = "^1.0.0"
openai = "^0.28.0"
chromadb = "^0.4.12"
tiktoken = "^0.5.1"
huggingface-hub = "^0.17.2"
langchain = "^0.0.300"
redlines = "^0.4.2"
pypdf = "^3.16.2"
yt-dlp = "^2023.9.24"
pydub = "^0.25.1"
lark = "^1.1.7"
scikit-learn = "^1.3.1"
instructorembedding = "^1.0.1"
torch = "^2.1.0"
sentence-transformers = "^2.2.2"
matplotlib = "^3.8.0"
panel = "^1.2.3"
docarray = "^0.39.0"

[tool.poetry.group.docs.dependencies]
autodoc_pydantic = "^1.8.0"
myst_parser = "^0.18.1"
nbsphinx = "^0.8.9"
sphinx = "^4.5.0"
sphinx-autobuild = "^2021.3.14"
sphinx_book_theme = "^0.3.3"
sphinx_rtd_theme = "^1.0.0"
sphinx-typlog-theme = "^0.8.0"
sphinx-panels = "^0.6.0"
toml = "^0.10.2"
myst-nb = "^0.17.1"
linkchecker = "^10.2.1"
sphinx-copybutton = "^0.5.1"
nbdoc = "^0.0.82"

[tool.poetry.group.codespell.dependencies]
codespell = "^2.2.0"

[tool.codespell]
skip = '.git,*.pdf,*.svg,*.pdf,*.yaml,*.ipynb,poetry.lock,*.min.js,*.css,package-lock.json,example_data,_dist,examples'
ignore-regex = '.*(Stati Uniti|Tense=Pres).*'
ignore-words-list = 'momento,collison,ned,foor,reworkd,parth,whats,aapply,mysogyny,unsecure,damon,crate'

Here is the error message:

AttributeError: module 'tensorflow' has no attribute 'Tensor'

(hard to believe this error could occur without being detected by the developers, since the tensorflow module is imported either directly or indirectly by the docarray module)
and the full stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[43], line 1
----> 1 cb = cbfs()
      3 file_input = pn.widgets.FileInput(accept='.pdf')
      4 button_load = pn.widgets.Button(name="Load DB", button_type='primary')

Cell In[42], line 14, in cbfs.__init__(self, **params)
     12 self.panels = []
     13 self.loaded_file = "docs/MachineLearning-Lecture01.pdf"
---> 14 self.qa = load_db(self.loaded_file,"stuff", 4)

Cell In[41], line 11, in load_db(file, chain_type, k)
      9 embeddings = OpenAIEmbeddings()
     10 # create vector database from data
---> 11 db = DocArrayInMemorySearch.from_documents(docs, embeddings)
     12 # define retriever
     13 retriever = db.as_retriever(search_type="similarity", search_kwargs={"k": k})

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/langchain/vectorstores/base.py:417, in VectorStore.from_documents(cls, documents, embedding, **kwargs)
    415 texts = [d.page_content for d in documents]
    416 metadatas = [d.metadata for d in documents]
--> 417 return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/langchain/vectorstores/docarray/in_memory.py:67, in DocArrayInMemorySearch.from_texts(cls, texts, embedding, metadatas, **kwargs)
     45 @classmethod
     46 def from_texts(
     47     cls,
   (...)
     51     **kwargs: Any,
     52 ) -> DocArrayInMemorySearch:
     53     """Create an DocArrayInMemorySearch store and insert data.
     54 
     55     Args:
   (...)
     65         DocArrayInMemorySearch Vector Store
     66     """
---> 67     store = cls.from_params(embedding, **kwargs)
     68     store.add_texts(texts=texts, metadatas=metadatas)
     69     return store

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/langchain/vectorstores/docarray/in_memory.py:38, in DocArrayInMemorySearch.from_params(cls, embedding, metric, **kwargs)
     20 @classmethod
     21 def from_params(
     22     cls,
   (...)
     27     **kwargs: Any,
     28 ) -> DocArrayInMemorySearch:
     29     """Initialize DocArrayInMemorySearch store.
     30 
     31     Args:
   (...)
     36         **kwargs: Other keyword arguments to be passed to the get_doc_cls method.
     37     """
---> 38     _check_docarray_import()
     39     from docarray.index import InMemoryExactNNIndex
     41     doc_cls = cls._get_doc_cls(space=metric, **kwargs)

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/langchain/vectorstores/docarray/base.py:19, in _check_docarray_import()
     17 def _check_docarray_import() -> None:
     18     try:
---> 19         import docarray
     21         da_version = docarray.__version__.split(".")
     22         if int(da_version[0]) == 0 and int(da_version[1]) <= 31:

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/__init__.py:5
      1 __version__ = '0.39.0'
      3 import logging
----> 5 from docarray.array import DocList, DocVec
      6 from docarray.base_doc.doc import BaseDoc
      7 from docarray.utils._internal.misc import _get_path_from_docarray_root_level

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/array/__init__.py:1
----> 1 from docarray.array.any_array import AnyDocArray
      2 from docarray.array.doc_list.doc_list import DocList
      3 from docarray.array.doc_vec.doc_vec import DocVec

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/array/any_array.py:22
      3 from typing import (
      4     TYPE_CHECKING,
      5     Any,
   (...)
     17     overload,
     18 )
     20 import numpy as np
---> 22 from docarray.base_doc.doc import BaseDocWithoutId
     23 from docarray.display.document_array_summary import DocArraySummary
     24 from docarray.exceptions.exceptions import UnusableObjectError

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/base_doc/__init__.py:1
----> 1 from docarray.base_doc.any_doc import AnyDoc
      2 from docarray.base_doc.base_node import BaseNode
      3 from docarray.base_doc.doc import BaseDoc

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/base_doc/any_doc.py:5
      1 from typing import Type
      3 from docarray.utils._internal.pydantic import is_pydantic_v2
----> 5 from .doc import BaseDoc
      8 class AnyDoc(BaseDoc):
      9     """
     10     AnyDoc is a Document that is not tied to any schema
     11     """

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/base_doc/doc.py:36
     34 from docarray.base_doc.base_node import BaseNode
     35 from docarray.base_doc.io.json import orjson_dumps_and_decode
---> 36 from docarray.base_doc.mixins import IOMixin, UpdateMixin
     37 from docarray.typing import ID
     38 from docarray.typing.tensor.abstract_tensor import AbstractTensor

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/base_doc/mixins/__init__.py:1
----> 1 from docarray.base_doc.mixins.io import IOMixin
      2 from docarray.base_doc.mixins.update import UpdateMixin
      4 __all__ = ['IOMixin', 'UpdateMixin']

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/base_doc/mixins/io.py:25
     22 from typing_inspect import get_args, is_union_type
     24 from docarray.base_doc.base_node import BaseNode
---> 25 from docarray.typing import NdArray
     26 from docarray.typing.proto_register import _PROTO_TYPE_NAME_TO_CLASS
     27 from docarray.utils._internal._typing import safe_issubclass

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/__init__.py:3
      1 from typing_extensions import TYPE_CHECKING
----> 3 from docarray.typing.bytes import AudioBytes, ImageBytes, VideoBytes
      4 from docarray.typing.id import ID
      5 from docarray.typing.tensor import ImageNdArray, ImageTensor

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/bytes/__init__.py:1
----> 1 from docarray.typing.bytes.audio_bytes import AudioBytes
      2 from docarray.typing.bytes.image_bytes import ImageBytes
      3 from docarray.typing.bytes.video_bytes import VideoBytes

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/bytes/audio_bytes.py:9
      7 from docarray.typing.bytes.base_bytes import BaseBytes
      8 from docarray.typing.proto_register import _register_proto
----> 9 from docarray.typing.tensor.audio import AudioNdArray
     10 from docarray.utils._internal.misc import import_library
     12 T = TypeVar('T', bound='AudioBytes')

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/__init__.py:5
      1 import types
      3 from typing_extensions import TYPE_CHECKING
----> 5 from docarray.typing.tensor.audio import AudioNdArray
      6 from docarray.typing.tensor.embedding import AnyEmbedding, NdArrayEmbedding
      7 from docarray.typing.tensor.image import ImageNdArray, ImageTensor

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/audio/__init__.py:4
      1 import types
      2 from typing import TYPE_CHECKING
----> 4 from docarray.typing.tensor.audio.audio_ndarray import AudioNdArray
      5 from docarray.typing.tensor.audio.audio_tensor import AudioTensor
      6 from docarray.utils._internal.misc import (
      7     _get_path_from_docarray_root_level,
      8     import_library,
      9 )

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/audio/audio_ndarray.py:3
      1 from docarray.typing.proto_register import _register_proto
      2 from docarray.typing.tensor.audio.abstract_audio_tensor import AbstractAudioTensor
----> 3 from docarray.typing.tensor.ndarray import NdArray
      6 @_register_proto(proto_type_name='audio_ndarray')
      7 class AudioNdArray(AbstractAudioTensor, NdArray):
      8     """
      9     Subclass of [`NdArray`][docarray.typing.NdArray], to represent an audio tensor.
     10     Adds audio-specific features to the tensor.
   (...)
     50     ---
     51     """

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/ndarray.py:31
     28 if tf_available:
     29     import tensorflow as tf  # type: ignore
---> 31     from docarray.typing.tensor.tensorflow_tensor import TensorFlowTensor  # noqa: F401
     33 if TYPE_CHECKING:
     35     from docarray.computation.numpy_backend import NumpyCompBackend

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/tensorflow_tensor.py:34
     31 T = TypeVar('T', bound='TensorFlowTensor')
     32 ShapeT = TypeVar('ShapeT')
---> 34 tf_base: type = type(tf.Tensor)
     35 node_base: type = type(BaseNode)
     38 # the mypy error suppression below should not be necessary anymore once the following
     39 # is released in mypy: https://github.com/python/mypy/pull/14135

AttributeError: module 'tensorflow' has no attribute 'Tensor'

Example Code

The error occurs in the constructor of class cbfs, more specifically, when calling DocArrayInMemorySearch.from_documents in method load_db(...):

import panel as pn
import param

def load_db(file, chain_type, k):
    # load documents
    loader = PyPDFLoader(file)
    documents = loader.load()
    # split documents
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
    docs = text_splitter.split_documents(documents)
    # define embedding
    embeddings = OpenAIEmbeddings()
    # create vector database from data
    print("hello 3")
    db = DocArrayInMemorySearch.from_documents(docs, embeddings)   # ERROR OCCURS <<<<<
    # define retriever
    print("hello 2")
    retriever = db.as_retriever(search_type="similarity", search_kwargs={"k": k})

class cbfs(param.Parameterized):
    chat_history = param.List([])
    answer = param.String("")
    db_query  = param.String("")
    db_response = param.List([])
    
    def __init__(self,  **params):
        super(cbfs, self).__init__( **params)
        self.panels = []
        self.loaded_file = "docs/MachineLearning-Lecture01.pdf"
        self.qa = load_db(self.loaded_file,"stuff", 4)


### Python, DocArray & OS Version

```Text
DocArray: 0.39.0
Python: 3.10.10

Affected Components
@erlebach
Copy link
Author

erlebach commented Oct 8, 2023

Here is some more information.

In tensorflow is imported, here is what help(tf) generates:

Help on package tensorflow:

NAME
    tensorflow

PACKAGE CONTENTS
    configure
    tensorflow (package)
    third_party (package)

FILE
    (built-in)

I have no idea what was actually imported via import tensorflow as tf, which occurs in method>

File ~/src/2023/deepleaarning.ai/.venv/lib/python3.10/site-packages/docarray/typing/tensor/ndarray.py:31
     28 if tf_available:
     29     import tensorflow as tf  # type: ignore
---> 31     from docarray.typing.tensor.tensorflow_tensor import TensorFlowTensor  # noqa: F401
     33 if TYPE_CHECKING:
     35     from docarray.computation.numpy_backend import NumpyCompBackend

The error is clearly in typing somewhere. Inside ndarray: :

tf_available = is_tf_available()
if tf_available:
    import tensorflow as tf  # type: ignore

    from docarray.typing.tensor.tensorflow_tensor import TensorFlowTensor  # noqa: F401

The command import ensorflow as tf is executed, but the full tensorflow is not imported. No idea why not.

It seems to me that some type-checking could be bypassed to avoid this issue. Thanks.

@JoanFM JoanFM moved this to Todo/Bug in DocArray backlog Oct 8, 2023
@JoanFM
Copy link
Member

JoanFM commented Oct 8, 2023

have u tried installing docarray[tensorflow]?

@erlebach
Copy link
Author

erlebach commented Oct 9, 2023

I did not. I just installed docarray into poetry. I'll try it. In the meantime, I switched to streamlit which has its own set of issues.

@JoanFM
Copy link
Member

JoanFM commented Oct 18, 2023

I think this must be related to something in your system.

huggingface/transformers#21135

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
DocArray backlog
Status: Todo/Bug
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erlebach @JoanFM