I am in need for a data engineer who can help me in building a pipeline for a required airflow job using pyspark.

₹600-1500 INR

Opravljeno

Objavljeno

pred 27 dnevi

₹600-1500 INR

Plačilo ob dostavi

To build a pipeline where we obtain data from a table in hadoop server and do some quality checks before updating that data to a postgress table. After that we need to filter that postgress data and do an upsert command and update that particular data into an s3 bucket. The data pipeline involves multiple data sources. The pipeline should include data validation to ensure accuracy and consistency. Ensure data quality checks such as checking the count of rows, comparing the data before and after, and collecting the new data inserted before updating the data to the Postgres table." Please use Apache Airflow for orchestration. The pipeline should run daily. Ensure the pipeline includes advanced data validation like integrity checks and statistical analysis.

ID projekta: 38860893

Več o projektu

3 ponudb

Projekt na daljavo

Aktivno pred 26 dnevi

Želite zaslužiti?

E-naslov

Prednosti oddajanja ponudb na Freelancerju

Nastavite svoj proračun in časovni okvir

Prejmite plačilo za svoje delo

Povzetek predloga

Registracija in oddajanje ponudb sta brezplačna

Dodeljeno:

@mohamedk007

With a strong background in AI, automation, and extensive experience in Python programming, I am well-equipped to handle your data engineering needs. I have developed robust projects utilizing advanced technologies and frameworks including the ones you mentioned like Apache Airflow and Pyspark. My expertise qualifies me not only to build a data pipeline between Hadoop and Postgres within your required daily timeframe but also to implement the necessary quality checks to ensure accuracy and consistency of your data. In addition, my proficiency in designing powerful software architectures that are scalable and maintainable will be a great asset in designing an efficient ETL process that incorporates advanced data validations such as statistical analysis and integrity checks. Furthermore, my familiarity with cloud platforms like AWS will be instrumental in storing your data securely and ensuring high-performance throughout the process. Above all, my dedication to continuous learning and keeping up-to-date with the latest industry trends means that you'll be working with someone who's not just qualified for the job but passionate about delivering innovative solutions. I look forward to discussing further how we can maximize the potential of your data pipeline project. Let's ensure your data operations are reliable, efficient, and future-proof!

₹3.500 INR v 2 dneh

0,0

(0 ocen)

0,0

3 freelancerjev je oddalo ponudbo s povprečno vrednostjo ₹2.517 INR za to delo

@ahmadkamaleddin

I develop web applications with a focus on ERP systems, using tools like Frappe platform and ERPNext. I specialize in data integration, ensuring smooth connectivity and automation across different services. Additionally, I build chatbots that enhance user interaction and support. My expertise also extends to setting up and customizing advanced chat systems for efficient communication.

₹1.050 INR v 7 dneh

5,0

(5 ocen)

3,9

@sarab01

Hello, I'm professional data engineer with strong experience in Pyspark. I've used this technology to process varous types of big data. I'm sure I can help you doing this task efficiently, and I guarantee a high quality jobs.

₹3.000 INR v 2 dneh