Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify processing ASCII text files that have line ending character #295

Closed
3 tasks done
yruslan opened this issue May 25, 2020 · 0 comments · Fixed by #296
Closed
3 tasks done

Simplify processing ASCII text files that have line ending character #295

yruslan opened this issue May 25, 2020 · 0 comments · Fixed by #296
Assignees
@yruslan
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented May 25, 2020 •
edited
Loading

Background

Currently, all parameters must be specified in this code snippet:

    val parsedCopybook = CopybookParser.parseTree(ASCII(), copybook, dropGroupFillers = false, segmentRedefines = Seq(), stringTrimmingPolicy = StringTrimmingPolicy.TrimNone, ebcdicCodePage = CodePage.getCodePageByName("common"), nonTerminals = Seq())
    val cobolSchema = new CobolSchema(parsedCopybook, SchemaRetentionPolicy.CollapseRoot, false)
    val sparkSchema = cobolSchema.getSparkSchema

Feature

  • Simplify loading ASCII files by creating a method that has default options for all parameters.
  • Add direct support for such kind of files.
  • Update the documentation.
@yruslan yruslan added the enhancement New feature or request label May 25, 2020
@yruslan yruslan self-assigned this May 25, 2020
@yruslan yruslan changed the title Simplify copybook parsing for ASCII encoded data Simplify processing ASCII text files that have line ending character May 25, 2020
yruslan added a commit that referenced this issue May 27, 2020
@yruslan
#295 Add a parsing method that has default values for all fields.
cb3996a
yruslan added a commit that referenced this issue May 27, 2020
@yruslan
#295 Add 'is_text' parameter to read text files.
b3d2526 
Spark's support for text files is very limited for our use case since it does not support
encodings and custom line endings. We need to re-implement this feature using a
variable-length reader.
yruslan added a commit that referenced this issue May 27, 2020
@yruslan
#295 Add 'is_text' parameter to read text files.
db1ca6d 
Spark's support for text files is very limited for our use case since it does not support
encodings and custom line endings. We need to re-implement this feature using a
variable-length reader.
yruslan added a commit that referenced this issue May 27, 2020
@yruslan
#295 Add documentation for the new way of processing ASCII text files.
a586792
@yruslan yruslan mentioned this issue May 27, 2020
Merged
yruslan added a commit that referenced this issue May 28, 2020
@yruslan
#295 Add a parsing method that has default values for all fields.
0480152
yruslan added a commit that referenced this issue May 28, 2020
@yruslan
#295 Add 'is_text' parameter to read text files.
b47a92a 
Spark's support for text files is very limited for our use case since it does not support
encodings and custom line endings. We need to re-implement this feature using a
variable-length reader.
yruslan added a commit that referenced this issue May 28, 2020
@yruslan
#295 Add 'is_text' parameter to read text files.
51b8da3 
Spark's support for text files is very limited for our use case since it does not support
encodings and custom line endings. We need to re-implement this feature using a
variable-length reader.
yruslan added a commit that referenced this issue May 28, 2020
@yruslan
#295 Add documentation for the new way of processing ASCII text files.
0d9a4f8
@yruslan yruslan reopened this May 28, 2020
@yruslan yruslan closed this as completed Jun 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yruslan yruslan

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#295 Simplify loading ASCII text files AbsaOSS/cobrix
1 participant
@yruslan