コンテストを投稿する

Speed up JSONL to PostgreSQL import

ステータス: Open
賞金: €120
受け取ったエントリー: 4

コンテスト概要

The goal of this project is to significantly speed up an already existing data import script (see import_data.sh).

The steps needed to import the data consist of:
- Setup the necessary tables for the import to work (I already provide you with the postgres init script, see init.sql)
- Downloading/fetching the large ±24GB .JSONL data dump
- Importing and converting this single .JSONL-dump into two postgres tables running in a docker container

Requirements:
- The resulting tables should be exactly the same as when you run the original import script.
- The import time should be significantly reduced compared to the import time of the original import script using the same hardware requirements.
- It should be easy to run everything: clearly document how to run it

How you achieve this is completely up to you (Bash + SQL, pure SQL, Python, …).

推奨スキル

このコンテストに参加する

このコンテストのトップエントリー

SakibKaiser Bangladesh

0

公開説明ボード

blui88
- 9時間前
May I help you in this regard?

Let's chat to know more about the issue and about probable improvements in the Script.
Regards

Santosh
- 9時間前
kreativesystem91
- 18時間前
Hello. I can help you with your project contact me #1
- 18時間前