Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

ninaburg · 2024-11-01T13:34:56Z

Purpose

This pull request aims to achieve the following:

Containerize the Flexpart Application: The Flexpart application has been containerized to streamline deployment and execution. Jenkins is set up to build this container image automatically and push it to both NEXUS and AWS ECR repositories.
Add Job Automation Scripts:

Data Download Script: A script has been added to download the necessary input GRIB files from an S3 bucket. This input data is required to run Flexpart.
Flexpart Job Launch Script: A script to initiate the Flexpart job inside the container.
Data Upload Script: After processing, the output data will be uploaded to a specified S3 bucket.

Notes

Most of the code in this PR is adapted from the fdb branch in this repository, but the fdb option has been removed

victoria-cherkas · 2024-11-25T16:48:31Z

docker/Dockerfile

+ARG container_registry=dockerhub.apps.cp.meteoswiss.ch
+ARG base_tag=v0.2.1
+
+FROM ${container_registry}/flexpart-poc/spack-dependencies:${base_tag} as intermediate


If we depend on another image, we should also include in this repo the Dockerfile for that image, eg Dockerfile.base . Or we can simply merge the two into one Dockerfile but this will take a lot longer to build (the base only needs to be rebuild every now and then)

Also i would rename that base image, not to include the word 'poc'

victoria-cherkas · 2024-11-25T17:00:45Z

jenkins/test.sh

+wget -q https://nexus.meteoswiss.ch/nexus/repository/app-artifacts-mch/nwp-rzplus/flexpart-poc/flexpart/IGBP_int1.dat
+
+
+podman run --name flexpart-container-test-$BRANCH_NAME -v ${data_dir}:/data:ro $IMAGE_INTERN --fdb


does this --fdb flag do anything? Should be removed

victoria-cherkas · 2024-11-25T17:01:13Z

docker/genconf.yml

+IBTIME: 000006
+IEDATE: 20230918 
+IETIME: 000010
+FDBFLAG: 1


Should be removed

Suggested change

FDBFLAG: 1

victoria-cherkas · 2024-11-25T17:02:40Z

jenkins/Jenkinsfile

+                }
+            }
+        }
+        stage('Test container') {


Build and test should be done before Deploy stage . I would copy the layout here: https://github.com/C2SM-RCM/flexpart/blob/main/Jenkinsfile

victoria-cherkas · 2024-11-25T17:06:57Z

docker/Dockerfile

+    && apt-get -yqq install --no-install-recommends \
+    zlib1g && spack external find
+
+RUN spack env activate -p spack-env && spack concretize && spack install


This will always install the main branch of the image.. which may be what you want, but if we want to always build the current commit in Jenkins, I would do something like this: https://github.com/C2SM-RCM/flexpart/blob/main/Dockerfile#L19-L25

victoria-cherkas · 2024-11-25T17:22:11Z

docker/sandbox_generator.py

+    from yaml import Loader, Dumper
+
+
+def download_file(obj, file):


Would be neater if we had a new file called s3_utils.py with the contents of upload_s3.py and this function (and whatever is s3 related), removing then upload_s3.py

victoria-cherkas · 2024-11-25T17:23:56Z

docker/sandbox_generator.py

As mentioned above, I would restructure the python code into a library, and have separate functions rather than one script. Then we can also add pytest tests much easier.

victoria-cherkas · 2024-11-25T17:25:55Z

docker/spack.yaml

+spack:
+  # add package specs to the `specs` list
+  specs: 
+    - flexpart-ifs@main


Example of using spack develop for building from a local source (if we want to support building current commit in CICD).

spack: specs: - flexpart-cosmo@main config: install_tree: root: /home/vcherkas/dev/flexpart-cosmo-sandbox/spack-root develop: flexpart-cosmo: path: /home/vcherkas/dev/flexpart-cosmo-sandbox/flexpart spec: flexpart-cosmo@=main

also here in CICD: https://github.com/C2SM-RCM/flexpart/blob/main/Dockerfile.base#L91

victoria-cherkas · 2024-11-25T17:27:11Z

jenkins/Jenkinsfile

+        NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart-containerize"
+        IMAGE_INTERN = "docker-intern-nexus.meteoswiss.ch/$NAMESPACE:$TAG"
+        IMAGE_PUBLIC = "docker-public-nexus.meteoswiss.ch/$NAMESPACE:$TAG"
+        REPO_ECR = "493666016161.dkr.ecr.eu-central-2.amazonaws.com"


As discussed I would not expose AWS account ID. Would fetch this from Vault.

victoria-cherkas · 2024-11-25T17:28:23Z

jenkins/Jenkinsfile

+            script: 'if [ "$BRANCH_NAME" = "main" ]; then echo -n 1; elif [ -n "${TAG_NAME+1}" ]; then echo -n 1; else echo -n 0; fi'
+            )
+
+        NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart-containerize"


Suggested change

NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart-containerize"

NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart"

The name flexpart is already descriptive enough, or is there already an image for that?
OR could add ewc to the $TAG if not

No, there isn’t another flexpart-ifs/flexpart image, so I switched to that name to maintain consistency with the flexpart-cosmo naming convention.

victoria-cherkas and others added 3 commits September 25, 2024 16:36

add jenkins and docker directories

b7558a8

FDB: Use spack v0.20.1.3 base image dependencies (#7)

f06fc0c

Update Jenkinsfile

2df7b35

victoria-cherkas assigned ninaburg Nov 1, 2024

victoria-cherkas self-requested a review November 1, 2024 14:19

ninaburg force-pushed the containerize-dev branch from 1f6fca7 to b41e06a Compare November 5, 2024 22:35

Nina Burgdorfer and others added 4 commits November 22, 2024 10:50

remove fdb option

9ca6ffc

s3: adapt s3 settings for non-aws s3

17957c1

jenkins: push image to Elastic Container Registry

c5fafa5

s3: modify name of outputs directory

abb884c

ninaburg force-pushed the containerize-dev branch from 43441a1 to abb884c Compare November 22, 2024 10:00

victoria-cherkas reviewed Nov 25, 2024

View reviewed changes

ninaburg changed the base branch from containerize to main November 26, 2024 12:52

ninaburg added 12 commits November 26, 2024 14:59

add dependencies dockerfiles

436d8b7

rename base images

a0c1383

update jenkinsfile to rebuild base images

44bc28f

rm FDB flag

26ba322

move dockefiles in root

c662ffe

repo restructuration

2c08cef

update dockerfiles

ac21052

.py utils as package

cf03de0

update dockerfiles

e2bfca8

update entrpoint and jenkinsfile

4adbe16

update flexpart utils

0486a52

tests

50e0deb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

ninaburg commented Nov 1, 2024 •

edited

Loading

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

ninaburg Nov 26, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

victoria-cherkas Nov 25, 2024

ninaburg Nov 27, 2024 •

edited

Loading

		wget -q https://nexus.meteoswiss.ch/nexus/repository/app-artifacts-mch/nwp-rzplus/flexpart-poc/flexpart/IGBP_int1.dat


		podman run --name flexpart-container-test-$BRANCH_NAME -v ${data_dir}:/data:ro $IMAGE_INTERN --fdb

		from yaml import Loader, Dumper


		def download_file(obj, file):

	NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart-containerize"
	NAMESPACE = "numericalweatherpredictions/dispersionmodelling/flexpart-ifs/flexpart"

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

Are you sure you want to change the base?

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

Conversation

ninaburg commented Nov 1, 2024 • edited Loading

Purpose

Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ninaburg Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

ninaburg commented Nov 1, 2024 •

edited

Loading

ninaburg Nov 27, 2024 •

edited

Loading