DataGen is a multi-process test data files generator.

This is handy when you want to generate large test data files (e.g. XMLs, JSONs, CSVs, etc), over multiple processes, utilising available CPU cores on your machine. It's also very easy to generate random numbers, dates, and strings as test data. You only need to create template files, no scripting involved.

npm install -g datagen 

Create example header, segment, and footer template files:

datagen init

Generate 8 data files containing 1 million segments in each file, written to data1 ... data8 output files, running over 8 processes, each process is used to generate 1 file:

datagen gen -s 1000000 -w 8 -o data

Limit the number of worker processes that can run concurrently to just 3:

datagen gen -s 1000000 -w 8 -m 3

DataGen uses three template files: header, segment, and footer. These templates are simple text files which are used to construct a data file in this format:

header
segment 1
segment 2
...
segment N (number of segments)
footer

Example header:

<?xml version="1.0" encoding="UTF-8"?>
<data>

Example segment:

<segment>
  <id>{gen_id}-{worker_id}-{segment_id}</id>
  <name>{first_name()} {last_name()}</name>
  <dob>{date('dd-mm-yyyy')}</dob>
</segment>

Example footer:

</data>

The above templates will generate an XML like this:

<?xml version="1.0" encoding="UTF-8"?>
<data>
<segment>
  <id>1-1-1</id>
  <name>Niels Bryant</name>
  <dob>12-08-1992</dob>
</segment>
<segment>
  <id>1-1-2</id>
  <name>John Bohr</name>
  <dob>01-11-1970</dob>
</segment>
...
</data>

Curly brackets in templates (e.g. JSON format) need to be escaped with double brackets:

{{ "id": "{gen_id}-{worker_id}-{segment_id}", "name": "{first_name()} {last_name()}", "dob": "{date('dd-mm-yyyy')}" }}

As for CSV, check out the example below.

Example header:

(Column1,Column2,Column3,Column4,Column5)
FirstName,LastName,Age,EmployeeNumber,DateJoint

Example segment:

john,Smith,{integer(18, 65)},{integer()},{date('yyyy-mm-dd', 2000, 2018)}

Footer is not required for CSV unless you ar using an ETL which my require a data record count.

Templates can contain the following parameters:

Developer's Guide

Build reports:

• Code complexity report
• Unit tests report
• Test coverage report
• Integration tests report
• API Documentation

Articles:

• DataGen Workers Optimisation
• DataGen: Generate Large Test Data Files – Like A Boss