Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text extraction code for columns. #366

Merged
merged 54 commits into from
Jun 30, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
6fe0d20
Fixed filename:page in logging
peterwilliams97 May 19, 2020
22680be
Got CMap working for multi-rune entries
peterwilliams97 May 19, 2020
a9910e7
Treat CMap entries as strings instead of runes to handle multi-byte e…
peterwilliams97 May 20, 2020
0c54cec
Added a test for multibyte encoding.
peterwilliams97 May 20, 2020
6103fb8
Merge branch 'development' of https://github.com/unidoc/unipdf into cmap
peterwilliams97 May 20, 2020
e9c46fa
Merge branch 'cmap' into columns
peterwilliams97 May 24, 2020
6b13a99
First version of text extraction that recognizes columns
peterwilliams97 May 24, 2020
a5c538f
Added an expanation of the text columns code to README.md.
peterwilliams97 May 24, 2020
8303318
fixed typos
peterwilliams97 May 24, 2020
c515472
Abstracted textWord depth calculation. This required change textMark …
peterwilliams97 May 24, 2020
603b5ff
Added function comments.
peterwilliams97 May 25, 2020
fad1552
Fixed text state save/restore.
peterwilliams97 May 26, 2020
6b4314f
Adjusted inter-word search distance to make paragrah division work fo…
peterwilliams97 May 26, 2020
d21e2f8
Got text_test.go passing.
peterwilliams97 May 27, 2020
418f859
Reinstated hyphen suppression
peterwilliams97 May 27, 2020
2260e24
Handle more cases of fonts not being set in text extraction code.
peterwilliams97 May 28, 2020
a14d8e7
Fixed typo
peterwilliams97 May 28, 2020
49bbef0
More verbose logging
peterwilliams97 May 28, 2020
40806d7
Adding tables to text extractor.
peterwilliams97 Jun 1, 2020
29f2d9b
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 5, 2020
af9508c
Added tests for columns extraction.
peterwilliams97 Jun 5, 2020
16b3c1c
Removed commented code
peterwilliams97 Jun 5, 2020
30fc953
Check for textParas that are on the same line when writing out extrac…
peterwilliams97 Jun 5, 2020
b4d90b6
Absorb text to the left of paras into paras e.g. Footnote numbers
peterwilliams97 Jun 5, 2020
975e038
Removed funny character from text_test.go
peterwilliams97 Jun 15, 2020
e6be021
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 15, 2020
a7779a3
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 22, 2020
5d7e4aa
Commented out a creator_test.go test that was broken by my text extra…
peterwilliams97 Jun 22, 2020
acb5caa
Big changes to columns text extraction code for PR.
peterwilliams97 Jun 22, 2020
80b54ef
Updated extractor/README
peterwilliams97 Jun 22, 2020
91479a7
Cleaned up some comments and removed a panic
peterwilliams97 Jun 22, 2020
72155a0
Increased threshold for truncating extracted text when there is no li…
peterwilliams97 Jun 22, 2020
09ebbcf
Improved an error message.
peterwilliams97 Jun 22, 2020
1c54e01
Removed irrelevant spaces
peterwilliams97 Jun 22, 2020
17bee4d
Commented code and removed unused functions.
peterwilliams97 Jun 23, 2020
e65fb04
Reverted PdfRectangle changes
peterwilliams97 Jun 23, 2020
5933a3d
Added duplicate text detection.
peterwilliams97 Jun 23, 2020
933021c
Combine diacritic textMarks in text extraction
peterwilliams97 Jun 24, 2020
f3770ee
Reinstated a diacritic recombination test.
peterwilliams97 Jun 24, 2020
e8abebd
Small code reorganisation
peterwilliams97 Jun 24, 2020
3f1df97
Reinstated handling of rotated text
peterwilliams97 Jun 25, 2020
3cca581
Addressed issues in PR review
peterwilliams97 Jun 25, 2020
b39f205
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 25, 2020
d5c344d
Added color fields to TextMark
peterwilliams97 Jun 25, 2020
fe6afef
Updated README
peterwilliams97 Jun 25, 2020
8be2607
Reinstated the disabled tests I missed before.
peterwilliams97 Jun 25, 2020
a5e21a7
Tightened definition for tables to prevent detection of tables where …
peterwilliams97 Jun 25, 2020
8f64966
Compute line splitting search range based on fontsize of first word i…
peterwilliams97 Jun 26, 2020
25414d4
Use errors.Is(err, core.ErrNotSupported) to distinguish unsupported f…
peterwilliams97 Jun 27, 2020
cf91ad6
Fixed some naming and added some comments.
peterwilliams97 Jun 27, 2020
9caa40e
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
peterwilliams97 Jun 27, 2020
b7f91fd
errors.Is -> xerrors.Is and %w -> %v for go 1.12 compatibility
peterwilliams97 Jun 29, 2020
d3deac8
Removed code that doesn't ever get called.
peterwilliams97 Jun 29, 2020
fe35826
Removed unused test
peterwilliams97 Jun 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'development' of https://github.com/unidoc/unipdf into c…
…olumns
  • Loading branch information
peterwilliams97 committed Jun 25, 2020
commit b39f205815050c4e162b9d7be1cbfa090de53e9a
15 changes: 13 additions & 2 deletions extractor/text.go
Original file line number Diff line number Diff line change
Expand Up @@ -847,12 +847,15 @@ func (to *textObject) renderText(data []byte) error {
translation(end),
math.Abs(spaceWidth*trm.ScalingFactorX()),
font,
to.state.tc)
to.state.tc,
fillColor,
strokeColor)

if !onPage {
common.Log.Debug("Text mark outside page. Skipping")
continue
}
if font == nil {
if font == nil {
common.Log.Debug("ERROR: No font.")
} else if font.Encoder() == nil {
common.Log.Debug("ERROR: No encoding. font=%s", font)
Expand Down Expand Up @@ -896,6 +899,14 @@ func (to *textObject) moveTo(tx, ty float64) {
to.tm = to.tlm
}









// PageText represents the layout of text on a device page.
type PageText struct {
marks []*textMark // Texts and their positions on a PDF page.
Expand Down
8 changes: 7 additions & 1 deletion extractor/text_mark.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ package extractor

import (
"fmt"
"image/color"
"math"

"github.com/unidoc/unipdf/v3/common"
Expand All @@ -27,13 +28,16 @@ type textMark struct {
trm transform.Matrix // The current text rendering matrix (TRM above).
end transform.Point // The end of character device coordinates.
originaBBox model.PdfRectangle // Bounding box without orientation correction.
fillColor color.Color // Text fill color.
strokeColor color.Color // Text stroke color.
}

// newTextMark returns a textMark for text `text` rendered with text rendering matrix (TRM) `trm`
// and end of character device coordinates `end`. `spaceWidth` is our best guess at the width of a
// space in the font the text is rendered in device coordinates.
func (to *textObject) newTextMark(text string, trm transform.Matrix, end transform.Point,
spaceWidth float64, font *model.PdfFont, charspacing float64) (textMark, bool) {
spaceWidth float64, font *model.PdfFont, charspacing float64,
fillColor, strokeColor color.Color) (textMark, bool) {
theta := trm.Angle()
orient := nearestMultiple(theta, orientationGranularity)
var height float64
Expand Down Expand Up @@ -117,6 +121,8 @@ func (to *textObject) newTextMark(text string, trm transform.Matrix, end transfo
trm: trm,
end: end,
orient: orient,
fillColor: fillColor,
strokeColor: strokeColor,
}
if verboseGeom {
common.Log.Info("newTextMark: start=%.2f end=%.2f %s", start, end, tm.String())
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.