Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Infinite recursion for tesseract --list-langs with conda-forge binary #4230

Closed
jonashaag opened this issue Apr 30, 2024 · 16 comments
Closed

Infinite recursion for tesseract --list-langs with conda-forge binary #4230

jonashaag opened this issue Apr 30, 2024 · 16 comments

Comments

@jonashaag
Copy link

Current Behavior

tesseract --list-langs goes into infinite loop on macOS if TESSDATA_PREFIX is empty.

macOS Instruments shows infinite recursion in addAvailableLanguages, and a LOT of stat64 calls (multiple 10k per second).

Expected Behavior

Should not go into infinite recursion

Suggested Fix

No response

tesseract -v

tesseract 5.3.4
leptonica-1.83.1
libgif 5.2.1 : libjpeg 8d (libjpeg-turbo 3.0.0) : libpng 1.6.43 : libtiff 4.6.0 : zlib 1.2.13 : libwebp 1.3.2 : libopenjp2 2.5.2
Found NEON

Operating System

macOS 14 Sonoma

Other Operating System

No response

uname -a

Darwin ... 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:41 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T8103 arm64

Compiler

From conda-forge

CPU

No response

Virtualization / Containers

None

Other Information

No response
@zdenop
Copy link
Contributor

zdenop commented May 9, 2024

Can you please exactly describe how to reproduce the problem?
How did you install tesseract?

I do not have macOS, but on Windows, I got this result:

> set TESSDATA_PREFIX=""
> tesseract --list-langs
List of available languages in """/" (0):

>set TESSDATA_PREFIX=
>tesseract --list-langs
List of available languages in "./" (1):
Downloads/dotslayer

On the Linux I got this result:

$ export TESSDATA_PREFIX=""
$ tesseract --list-langs
List of available languages in "/usr/share/tesseract-ocr/5/tessdata/" (2):
eng
osd

@jonashaag
Copy link
Author

On macOS, your Linux commands reproduce the infinite loop

@zdenop
Copy link
Contributor

zdenop commented May 10, 2024

@jonashaag : how did you installed tesseract?

@stweil : Can you have a look at this?

@stweil
Copy link
Member

stweil commented May 10, 2024 •
edited
Loading

I don't get an infinite loop on macOS (installed with Homebrew). An empty TESSDATA_PREFIX is handled like an unset one:

% tesseract --list-langs|head -3
List of available languages in "/opt/homebrew/share/tessdata/" (682):
Fraktur
Greek
% TESSDATA_PREFIX= tesseract --list-langs|head -3
List of available languages in "/opt/homebrew/share/tessdata/" (682):
Fraktur
Greek

@stweil
Copy link
Member

stweil commented May 10, 2024 •
edited
Loading

@jonashaag, addAvailableLanguages starts searching for Tesseract model files at a well defined start location and is called recursively only if it finds a directory. Unless you have a cyclic directory structure, there won't be an infinite recursion. macOS and Linux use the same code.

Can you see the arguments of the stat64 calls? Where does the search start? Maybe there are simply a lot of files and directories below your start location. If there is really an endless recursion, it will have a first directory entry which will occur again and again.

@jonashaag
Copy link
Author

Aha, interesting, the Homebrew version works fine. The conda-forge version that I've been using doesn't.

Output from macOS's fs_usage:

11:50:07.100355  open              F=3        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000020   tesseract.1362835
11:50:07.100364  fstatfs64         F=3                                                                                                                                                                                        0.000002   tesseract.1362835
11:50:07.100439  getdirentries64   F=3    B=0x14d0                                                                                                                                                                            0.000074   tesseract.1362835
11:50:07.100443  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000003   tesseract.1362835
11:50:07.100450  open              F=4        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000007   tesseract.1362835
11:50:07.100451  fstatfs64         F=4                                                                                                                                                                                        0.000001   tesseract.1362835
11:50:07.100501  getdirentries64   F=4    B=0x14d0                                                                                                                                                                            0.000050   tesseract.1362835
11:50:07.100503  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000002   tesseract.1362835
11:50:07.100509  open              F=5        (R_____N____X)  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                            0.000006   tesseract.1362835
11:50:07.100510  fstatfs64         F=5                                                                                                                                                                                        0.000001   tesseract.1362835
11:50:07.100559  getdirentries64   F=5    B=0x14d0                                                                                                                                                                            0.000049   tesseract.1362835
11:50:07.100561  stat64                                 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/.pixi/envs/default/share/tessdata                                                                                                  0.000002   tesseract.1362835
[... repeated indefinitely ...]

@jonashaag
Copy link
Author

There are no symlinks in .pixi/envs/default/share/tessdata or any of its children.

$ file --no-dereference .pixi/envs/default/share/tessdata/** | grep sym
[empty]

@stweil stweil changed the title Infinite recursion in addAvailableLanguages Infinite recursion in addAvailableLanguages with conda-forge binary for macOS May 10, 2024
@stweil
Copy link
Member

stweil commented May 11, 2024

Conda links:

I am still searching for a description of the build process which was used for the Conda Tesseract package.

@jonashaag
Copy link
Author

It is here:

@stweil
Copy link
Member

stweil commented May 11, 2024

Thanks. I could build it locally, and the result works fine. It looks like the error is in libtesseract.5.dylib which is not surprising because that library contains the function addAvailableLanguages.

In my tests with the conda-forge binary I get some more strange results:

  • tesseract --list-langs works if the tessdata directory is empty.
  • It also terminates if there is a single entry (file or directy) below tessdata, but fails to report if that file is a model file.
  • It runs endless if there are two or more entries below tessdata.

I cannot explain that strange results with the Tesseract code, and debugging without debug symbols is rather time consuming. Therefore I suggest to report the issue to conda-forge.

@jonashaag
Copy link
Author

jonashaag commented May 11, 2024 •
edited
Loading

I think it is related to Conda's prefix replacement. The prefix ends up being empty or something like that. I will build a package with debug symbols/prints and check what's going on.

@stweil
Copy link
Member

stweil commented May 12, 2024 •
edited
Loading

That's a good hint which helped me find the root of the problem.

Patching TESSDATA_PREFIX in the compiled library does currently not work for Tesseract because that string is assigned to std::string datadir in CCUtil::main_setup. This datadir gets a different string value after the patch, but its length is not patched and still has the value of the unpatched string. Therefore datadir won't get the required terminating '/' character.

This is not restricted to installations on macOS. Installations with Conda on Linux will have the same problem.

I think there is a simple fix which I will try later.

stweil added a commit to stweil/tesseract that referenced this issue May 12, 2024
@stweil
Improve CCUtil::main_setup (fixes issue tesseract-ocr#4230)
a36598b 
Conda installations patch TESSDATA_PREFIX in the binary.
That does not work for std::string because the length
won't be patched, so use a normal C string which can be
patched.

Simplify also the code which checks the last character
of datadir.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
@stweil
Copy link
Member

stweil commented May 12, 2024

Pull request #4239 should fix this issue.

@stweil stweil changed the title Infinite recursion in addAvailableLanguages with conda-forge binary for macOS Infinite recursion for tesseract --list-langs with conda-forge binary for macOS May 12, 2024
@stweil stweil changed the title Infinite recursion for tesseract --list-langs with conda-forge binary for macOS Infinite recursion for tesseract --list-langs with conda-forge binary May 12, 2024
@jonashaag
Copy link
Author

Fantastic! Thanks a million!

stweil added a commit that referenced this issue May 12, 2024
@stweil
Improve CCUtil::main_setup (fixes issue #4230)
6a31e36 
Conda installations patch TESSDATA_PREFIX in the binary.
That does not work for std::string because the length
won't be patched, so use a normal C string which can be
patched.

Simplify also the code which checks the last character
of datadir.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
@zdenop zdenop closed this as completed May 12, 2024
@stweil stweil reopened this May 12, 2024
@stweil
Copy link
Member

stweil commented May 12, 2024

@jonashaag, I am sorry, but my patch does not work. I'll try a different fix tomorrow.

stweil added a commit to stweil/tesseract that referenced this issue May 13, 2024
@stweil
Fix setup of datadir on installations with Conda (issue tesseract-ocr…
54408da 
…#4230)

Signed-off-by: Stefan Weil <sw@weilnetz.de>
@stweil
Copy link
Member

stweil commented May 13, 2024

Pull request #4240 has a different fix which I now tested successfully on Linux and on macOS.

@stweil stweil mentioned this issue May 13, 2024
Open
1 task
stweil added a commit that referenced this issue May 15, 2024
@stweil
Fix setup of datadir on installations with Conda (issue #4230)
bcfdd5e 
Signed-off-by: Stefan Weil <sw@weilnetz.de>
@amitdo amitdo closed this as completed May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jonashaag @zdenop @stweil @amitdo