Speed up JSONL to PostgreSQL import

Stare: Pending
Premiu: €120
Intrări primite: 5
Câștigător: richardm44

Winner

Sumarul concursului

The goal of this project is to significantly speed up an already existing data import script (see import_data.sh).

The steps needed to import the data consist of:
- Setup the necessary tables for the import to work (I already provide you with the postgres init script, see init.sql)
- Downloading/fetching the large ±24GB .JSONL data dump
- Importing and converting this single .JSONL-dump into two postgres tables running in a docker container

Requirements:
- The resulting tables should be exactly the same as when you run the original import script.
- The import time should be significantly reduced compared to the import time of the original import script using the same hardware requirements.
- It should be easy to run everything: clearly document how to run it

How you achieve this is completely up to you (Bash + SQL, pure SQL, Python, …).

Aptitudinile recomandate

Winner

Postează un concurs similar

Cele mai bune intrări pentru acest concurs

richardm44 India

0
Suptechy Pakistan

0
xslackx Brazil

0

Panoul public de clarificare

xslackx
- acum 3 zile
Hi the current file provider it's not working, i tried several times but the extract is getting an error, could you check again?
- acum 3 zile
richardm44
- acum 5 zile
Hi, Please check my entry #7 Thanks!!
- acum 5 zile
masond17
- acum 1 săptămână
Please rate entries and provide feedback so myself (and others) can get you the best solution possible
- acum 1 săptămână
blui88
- acum 1 săptămână
May I help you in this regard?

Let's chat to know more about the issue and about probable improvements in the Script.
Regards

Santosh
- acum 1 săptămână
kreativesystem91
- acum 1 săptămână
Hello. I can help you with your project contact me #1
- acum 1 săptămână