Thanks for reporting the issue!

Looks interesting. Will take a look.

I can confirm the issue. Indeed, for variable-record-size files input_file_name() returns an empty string. That is due to the way we handle sparse indexes creation to parallelize the reading of such files.

It will take a while to fix this properly (probably need to create a custom RDD). But we can add a workaround to generate a column with the input file name for each record. That's what we are going to do first. It would look like this:

.option("with_input_file_name_col", "ISN_Source")

Just a double check. Which Spark version are you using?

We are planning to release Cobrix 2.0.0 first and all further changes will be made there. But it will support Spark 2.4 or above.

Great! Cobrix 2.0.0 is planned to be released this week. And the workaround for this issue can be expected sometime next week.

This should be fixed in the latest snapshot.
Please, try:

        <dependency>
            <groupId>za.co.absa.cobrix</groupId>
            <artifactId>spark-cobol_2.11</artifactId>
            <version>2.0.1-SNAPSHOT</version>
        </dependency>

and let me know if the issue is fixed.

Forgot to mention. In order to get input file names for each record of a variable record length file a workaround is used. In your case the option looks like this:

.option("with_input_file_name_col", "ISN_Source")

I'd also recommend using

.option("pedantic", "true")

So that unrecognized options cause errors.

Hi Kris,

Snapshot version linking requires additional configuration in .m2/settings.xml. It might be even harder for managed clusters.

Try setting the version to 2.0.1 which was released today.

And please let me know if it worked for you.

Thank you,
Ruslan

H2. environment: docker: jupyter/all-spark-notebook:latest + Apache Toree - Scala

H2. Issue

when using

.option("file_start_offset", "600")
.option("file_end_offset", "600")

input_file_name() no longer works

H3. Annonymized extract

%AddDeps za.co.absa.cobrix spark-cobol_2.11 2.0.3 --transitive

val sparkBuilder = SparkSession.builder().appName("Example")

val spark = sparkBuilder .getOrCreate()

`
import org.apache.spark.sql.functions._

import org.apache.spark.sql.SparkSession

spark.udf.register("get_file_name", (path: String) => path.split("/").last)
val cobolDataframe = spark
.read
.format("za.co.absa.cobrix.spark.cobol.source")
.option("pedantic", "true")
.option("copybook", "file:///home/jovyan/data/BRAND/COPYBOOK.txt")
.option("file_start_offset", "600")
.option("file_end_offset", "600")
.load("file:///home/jovyan/data/BRAND/initial_transformed/FILEPATTERN*")
.withColumn("DPSource", callUDF("get_file_name", input_file_name()))
`

cobolDataframe //.filter("RECORD.ID % 2 = 0") // filter the even values of the nested field 'RECORD_LENGTH' .take(20) .foreach(v => println(v))

Hi Kris,
When you use file offset a different reader is used. Use the workaround for this case instead of input_file_name():

.option("with_input_file_name_col", "DPSource")

Hi Ruslan,

"with_input_file_name_col" seems be intended for "is_record_sequence = true" only.

In this case I have a copy book (fixed lenth) where the copybook does not mention the Header and footer.

Possibly actions I should take are:

• get rid off the header and footer in a pre-prosessing (a less clean solution, to be avoided)
• try to rewrite the copybook to accomodate header and footer (ideal solution, maybe as it should) consisting of several record types. I will look into this next.

I value your opinion. Mainframe code can be messy. It is a trade off between handling source particuliarities out of the box and keeping the cobrix code maintainable.

Thanks in advance,

Regards, Bart,

a test of your suggestion:

`
import org.apache.spark.sql.functions._

import org.apache.spark.sql.SparkSession

spark.udf.register("get_file_name", (path: String) => path.split("/").last)

val cobolDataframe = spark
.read
.format("za.co.absa.cobrix.spark.cobol.source")
//.option("is_record_sequence", "true")
//.option("generate_record_id", "true") // for comparison with unconverted (windows) file only
.option("pedantic", "true")
//.option("with_input_file_name_col", "DPSourceTemp")
.option("copybook", "file:///home/jovyan/data/BRAND/COPYBOOK.txt")
.option("file_start_offset", "600")
.option("file_end_offset", "600")
.option("with_input_file_name_col", "DPSourceTemp")
.load("file:///home/jovyan/data/BRAND/initial_transformed")
`

the result:

Name: java.lang.IllegalArgumentException Message: Option 'with_input_file_name_col' is supported only when 'is_record_sequence' = true. StackTrace: at za.co.absa.cobrix.spark.cobol.source.parameters.CobolParametersParser$.validateSparkCobolOptions(CobolParametersParser.scala:467) at za.co.absa.cobrix.spark.cobol.source.parameters.CobolParametersParser$.parse(CobolParametersParser.scala:209) at za.co.absa.cobrix.spark.cobol.source.DefaultSource.createRelation(DefaultSource.scala:56) at za.co.absa.cobrix.spark.cobol.source.DefaultSource.createRelation(DefaultSource.scala:48) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)

Interesting. I will take a look. I think this can be easily fixed so that with_input_file_name_col would work in your case.

Opened #252 to continue the discussion there. Since the incompatibility between with_input_file_name_col and file_start_offset is a separate issue,