Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text extraction code for columns. #366

Merged
merged 54 commits into from
Jun 30, 2020
Merged
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
6fe0d20
Fixed filename:page in logging
peterwilliams97 May 19, 2020
22680be
Got CMap working for multi-rune entries
peterwilliams97 May 19, 2020
a9910e7
Treat CMap entries as strings instead of runes to handle multi-byte e…
peterwilliams97 May 20, 2020
0c54cec
Added a test for multibyte encoding.
peterwilliams97 May 20, 2020
6103fb8
Merge branch 'development' of https://github.com/unidoc/unipdf into cmap
peterwilliams97 May 20, 2020
e9c46fa
Merge branch 'cmap' into columns
peterwilliams97 May 24, 2020
6b13a99
First version of text extraction that recognizes columns
peterwilliams97 May 24, 2020
a5c538f
Added an expanation of the text columns code to README.md.
peterwilliams97 May 24, 2020
8303318
fixed typos
peterwilliams97 May 24, 2020
c515472
Abstracted textWord depth calculation. This required change textMark …
peterwilliams97 May 24, 2020
603b5ff
Added function comments.
peterwilliams97 May 25, 2020
fad1552
Fixed text state save/restore.
peterwilliams97 May 26, 2020
6b4314f
Adjusted inter-word search distance to make paragrah division work fo…
peterwilliams97 May 26, 2020
d21e2f8
Got text_test.go passing.
peterwilliams97 May 27, 2020
418f859
Reinstated hyphen suppression
peterwilliams97 May 27, 2020
2260e24
Handle more cases of fonts not being set in text extraction code.
peterwilliams97 May 28, 2020
a14d8e7
Fixed typo
peterwilliams97 May 28, 2020
49bbef0
More verbose logging
peterwilliams97 May 28, 2020
40806d7
Adding tables to text extractor.
peterwilliams97 Jun 1, 2020
29f2d9b
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 5, 2020
af9508c
Added tests for columns extraction.
peterwilliams97 Jun 5, 2020
16b3c1c
Removed commented code
peterwilliams97 Jun 5, 2020
30fc953
Check for textParas that are on the same line when writing out extrac…
peterwilliams97 Jun 5, 2020
b4d90b6
Absorb text to the left of paras into paras e.g. Footnote numbers
peterwilliams97 Jun 5, 2020
975e038
Removed funny character from text_test.go
peterwilliams97 Jun 15, 2020
e6be021
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 15, 2020
a7779a3
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 22, 2020
5d7e4aa
Commented out a creator_test.go test that was broken by my text extra…
peterwilliams97 Jun 22, 2020
acb5caa
Big changes to columns text extraction code for PR.
peterwilliams97 Jun 22, 2020
80b54ef
Updated extractor/README
peterwilliams97 Jun 22, 2020
91479a7
Cleaned up some comments and removed a panic
peterwilliams97 Jun 22, 2020
72155a0
Increased threshold for truncating extracted text when there is no li…
peterwilliams97 Jun 22, 2020
09ebbcf
Improved an error message.
peterwilliams97 Jun 22, 2020
1c54e01
Removed irrelevant spaces
peterwilliams97 Jun 22, 2020
17bee4d
Commented code and removed unused functions.
peterwilliams97 Jun 23, 2020
e65fb04
Reverted PdfRectangle changes
peterwilliams97 Jun 23, 2020
5933a3d
Added duplicate text detection.
peterwilliams97 Jun 23, 2020
933021c
Combine diacritic textMarks in text extraction
peterwilliams97 Jun 24, 2020
f3770ee
Reinstated a diacritic recombination test.
peterwilliams97 Jun 24, 2020
e8abebd
Small code reorganisation
peterwilliams97 Jun 24, 2020
3f1df97
Reinstated handling of rotated text
peterwilliams97 Jun 25, 2020
3cca581
Addressed issues in PR review
peterwilliams97 Jun 25, 2020
b39f205
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 25, 2020
d5c344d
Added color fields to TextMark
peterwilliams97 Jun 25, 2020
fe6afef
Updated README
peterwilliams97 Jun 25, 2020
8be2607
Reinstated the disabled tests I missed before.
peterwilliams97 Jun 25, 2020
a5e21a7
Tightened definition for tables to prevent detection of tables where …
peterwilliams97 Jun 25, 2020
8f64966
Compute line splitting search range based on fontsize of first word i…
peterwilliams97 Jun 26, 2020
25414d4
Use errors.Is(err, core.ErrNotSupported) to distinguish unsupported f…
peterwilliams97 Jun 27, 2020
cf91ad6
Fixed some naming and added some comments.
peterwilliams97 Jun 27, 2020
9caa40e
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 27, 2020
b7f91fd
errors.Is -> xerrors.Is and %w -> %v for go 1.12 compatibility
peterwilliams97 Jun 29, 2020
d3deac8
Removed code that doesn't ever get called.
peterwilliams97 Jun 29, 2020
fe35826
Removed unused test
peterwilliams97 Jun 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated extractor/README
  • Loading branch information
peterwilliams97 committed Jun 22, 2020
commit 80b54ef1de5586c8ee479782de2cf80b9c88f1b3
38 changes: 13 additions & 25 deletions extractor/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
TEXT EXTRACTION CODE
====================

BASIC IDEAS
-----------

There are two [directions](https://www.w3.org/International/questions/qa-scripts.en#directions)s\.

- *reading*
Expand All @@ -13,18 +10,6 @@ In English text,
- the *reading* direction is left to right, increasing X in the PDF coordinate system.
- the *depth* directon is top to bottom, decreasing Y in the PDF coordinate system.

*depth* is the distance from the bottom of a word's bounding box from the top of the page.
depth := pageSize.Ury - r.Lly

* Pages are divided into rectangular regions called `textPara`s.
* The `textPara`s in a page are sorted in reading order (the order they are read in, not the
*reading* direction above).
* Each `textPara` contains `textLine`s, lines with the `textPara`'s bounding box.
* Each `textLine` has extracted for the line in its `text()` function.
* Page text is extracted by iterating over `textPara`s and within each `textPara` iterating over its
`textLine`s.
* The textMarks corresponding to extracted text can be found.


HOW TEXT IS EXTRACTED
---------------------
Expand All @@ -36,13 +21,13 @@ HOW TEXT IS EXTRACTED
and spltting on space characters and the gaps between marks.
* The `textWords`s are grouped into `textParas`s based on their bounding boxes' proximities to other
textWords.
* The textWords in each textPara are arranged into textLines (textWords of similar depths).
* With each textLine, textWords are sorted in reading order each one that starts a whole word is marked.
See textLine.text()
* textPara.writeCellText() shows how to extract the paragraph text from this arrangment.
* The `textWord`s in each `textPara` are arranged into `textLine`s (`textWord`s of similar depth).
* Within each `textLine`, `textWord`s are sorted in reading order each one that starts a whole word is marked.
See `textLine.text()`.
* `textPara.writeCellText()` shows how to extract the paragraph text from this arrangment.
* All the `textPara`s on a page are checked to see if they are arranged as cells within a table and,
if they are, they are combined into `textTable`s and a textPara containing the textTable replaces the
the textParas containing the cells.
the `textPara`s containing the cells.
* The textParas, some of which may be tables, in sorted into reading order (the order in which they
are reading, not in the reading directions).

Expand All @@ -61,9 +46,12 @@ of about the same depth sorted left to right.
* textLine.markWordBoundaries() marks the textWords in each textLine that start whole words.

TODO
====
Remove serial code????
Reinstate rotated text handling.
Reinstate hyphen diacritic composition.
Reinstate duplicate text removal
-----

* Remove serial code????
* Remove verbose* logginng?
* Reinstate rotated text handling.
* Reinstate diacritic composition.
* Reinstate duplicate text removal.
* Reinstate creater_test.go extraction test.