Data Migration between Databases with Docker and Python

$10-11 USD

Fechado

Publicado

há aproximadamente 1 mês

$10-11 USD

Pago na entrega

Description: Perform data migration from a PostgreSQL database to MySQL using Docker Compose to configure containers. Develop ETL scripts in Python to manage the extraction, transformation, and loading (ETL) process, including data cleaning to treat missing, duplicate, and inconsistent values. Use a Kaggle database as a source, prioritizing large and complex datasets. Kaggle: [login to view URL] Tasks: Configuring Databases in Containers: Configure PostgreSQL and MySQL in different containers using Docker Compose. Create databases and tables in two databases, ensuring compatibility for migration. ETL Development: Extraction: Import data from the selected database in Kaggle to PostgreSQL. Transformation: Perform cleansing and standardization, addressing: Identifying and addressing missing values. Removing or adjusting duplicate records. Correcting data inconsistencies. Loading: Migrating the transformed data to MySQL. Data Validation: Perform robust validations, such as: Quantitative: Compare the number of records between databases to ensure consistency. Qualitative (optional): Check data samples to ensure the transformation was successful. Modeling and Architecture: Structure the project based on a Star Schema or Snowflake Schema diagram, as appropriate for the chosen data set. Document the overall architecture, including the relationship between tables and ETL processes. Deliverable: I present the Python scripts, the project diagram and a data validation report, and the scripts separated correctly to run on my machine along with an installation tutorial. -- I need a Star Schema model for data analysis. After that, I would like a complete tutorial on how to install and configure Docker Compose, MySQL, and PostgreSQL in VS Code. The modeling should be done using MySQL Workbench. -- I need this because later, I have to perform the entire installation process on my machine and see the project running.

ID do Projeto: 38827901

Sobre o projeto

11 propostas

Projeto remoto

Ativo há 1 mês

Quer ganhar algum dinheiro?

Endereço de e-mail

Benefícios de ofertar no Freelancer

Defina seu orçamento e seu prazo

Seja pago pelo seu trabalho

Descreva sua proposta

É grátis para se inscrever e fazer ofertas em trabalhos

11 freelancers estão ofertando em média $245 USD for esse trabalho

@raiquerlis

As an experienced full-stack developer with a solid grasp of Python and extensive knowledge of data migration and ETL processes, I feel I am the ideal candidate to tackle your data migration project. My familiarity with Docker Compose and setting up databases, coupled with my proficiency in Python scripting for ETL, aligns perfectly with your project requirements. During my 8-year career, I've handled diverse projects including complex E-commerce platforms using ReactJS, Vue.js, and Tailwind CSS. This experience has honed my ability to effectively extract, transform, and load the data while ensuring quality and consistency. Additionally, I have strong database management skills mainly in MySQL which will be invaluable in migrating your data. Through leveraging my up-to-date knowledge of data modeling techniques like Star Schema or Snowflake Schema, I can deliver an architecture that's scalable for your current needs, ensuring smooth data validation as a part of this process. You can look forward to a well-structured project documented comprehensively with the relationship between tables and ETL processes clearly outlined. So let's get started on achieving a seamless Kohana migration from a PostgreSQL database to MySQL for you using Docker Compose and Python!

$11 USD em 7 dias

5,0

(3 avaliações)

2,1

@colpbx

Hello, I am James from Match Data Pro. If you are open to other ideas, our service can import from your PG DB, do the ETL, maybe some deduplication and then export to your MySQL db as needed. Let me know if you would like to discuss.

$11 USD em 7 dias

0,0

(0 avaliações)

0,0

@yassinee17

Hi, I'm Yassine, a data engineer with a strong background in ETL processes, database migrations, and containerization. I’ve worked on projects involving complex data pipelines and containerized environments, which makes me confident in delivering exactly what you need. With my expertise in Python, PostgreSQL, MySQL, and Docker Compose, I can efficiently manage the entire migration process, including configuring containers, creating schemas, and handling ETL tasks. I’ve tackled large datasets like those on Kaggle before, ensuring data is cleaned, transformed, and validated for accuracy and consistency. I can also provide a clear Star Schema model using MySQL Workbench and create a detailed tutorial to help you set up Docker Compose, MySQL, and PostgreSQL in VS Code, ensuring you can run the project smoothly on your machine. Let’s bring this project to life!

$10 USD em 4 dias

0,0

(0 avaliações)

0,0

@mohamedk007

I am the ideal candidate for this project, having recently completed a similar task involving data migration and analysis. I used Docker Compose to configure PostgreSQL and MySQL containers, developed ETL scripts in Python to handle large Kaggle datasets, and performed data cleaning to address missing, duplicate, and inconsistent values. My experience includes creating a star schema for analysis and ensuring seamless setup with comprehensive documentation and tutorials, demonstrating proficiency in complex data workflows.

$11 USD em 4 dias

0,0

(0 avaliações)

0,0

@KaderAbdi

I will create a seamless data migration pipeline using Python and Docker Compose, ensuring data is cleaned, validated, and transformed. I will also design a Star Schema in MySQL Workbench and provide a clear tutorial for setup and execution in VS Code.

$11 USD em 7 dias

0,0

(0 avaliações)

0,0

@minaa65

A student at the College of Computers and Information, I have many skills in programming, accounting, information technology, and artificial intelligence

$11 USD em 7 dias