Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
+5,998 −0
Open

Containerize Flexpart and Automate S3 Data Handling for GRIB Input #10

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b7558a8
add jenkins and docker directories
victoria-cherkas Nov 17, 2023
f06fc0c
FDB: Use spack v0.20.1.3 base image dependencies (#7)
victoria-cherkas Jan 4, 2024
2df7b35
Update Jenkinsfile
victoria-cherkas Jan 4, 2024
9ca6ffc
remove fdb option
Oct 1, 2024
17957c1
s3: adapt s3 settings for non-aws s3
Oct 1, 2024
c5fafa5
jenkins: push image to Elastic Container Registry
ninaburg Oct 25, 2024
abb884c
s3: modify name of outputs directory
ninaburg Nov 12, 2024
436d8b7
add dependencies dockerfiles
ninaburg Nov 26, 2024
a0c1383
rename base images
ninaburg Nov 26, 2024
44bc28f
update jenkinsfile to rebuild base images
ninaburg Nov 26, 2024
26ba322
rm FDB flag
ninaburg Nov 26, 2024
c662ffe
move dockefiles in root
ninaburg Nov 27, 2024
2c08cef
repo restructuration
ninaburg Nov 27, 2024
ac21052
update dockerfiles
ninaburg Nov 27, 2024
cf03de0
.py utils as package
ninaburg Nov 27, 2024
e2bfca8
update dockerfiles
ninaburg Dec 3, 2024
4adbe16
update entrpoint and jenkinsfile
ninaburg Dec 12, 2024
0486a52
update flexpart utils
ninaburg Dec 12, 2024
50e0deb
tests
ninaburg Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
.py utils as package
  • Loading branch information
@ninaburg
ninaburg committed Nov 27, 2024
commit cf03de01622fa196f7724ae7f1d5471c600f66dc
Empty file added utils/flexpart_ifs_utils/__init__.py
View file
Open in desktop
Empty file.
17 changes: 15 additions & 2 deletions upload_s3.py → utils/flexpart_ifs_utils/s3_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/python3
import boto3 # pip install boto3
import boto3
import os
import argparse
from datetime import datetime
Expand Down Expand Up @@ -54,3 +53,17 @@ def uploadDirectory(root_path, bucket_name, endpoint_url):
print("Skipping upload to S3 as S3_BUCKET_NAME environment variable is unset.")
else:
uploadDirectory(args.directory, args.bucket, args.endpoint_url)

def download_file(obj, file):
bucket = os.getenv('FLEXPART_S3_BUCKETS__INPUT__NAME')
endpoint_url = os.getenv('FLEXPART_S3_BUCKETS__INPUT__ENDPOINT_URL')

s3 = boto3.client('s3', endpoint_url=endpoint_url, aws_access_key_id=os.getenv('S3_ACCESS_KEY'),
aws_secret_access_key=os.getenv('S3_SECRET_KEY'),)

try:
with open(file, 'wb') as f:
print(f"Downloading {obj} to {f} from bucket: {bucket}")
s3.download_fileobj(bucket, obj, f)
except Exception as e:
raise(e)
19 changes: 2 additions & 17 deletions sandbox_generator.py → ...s/flexpart_ifs_utils/sandbox_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#!/usr/bin/python3
import os
import yaml
import glob
Expand All @@ -7,27 +6,13 @@
import argparse
import boto3

from flexpart_ifs_utils import s3_utils

try:
from yaml import CLoader as Loader, CDumper as Dumper
except ImportError:
from yaml import Loader, Dumper


def download_file(obj, file):
bucket = os.getenv('FLEXPART_S3_BUCKETS__INPUT__NAME')
endpoint_url = os.getenv('FLEXPART_S3_BUCKETS__INPUT__ENDPOINT_URL')

s3 = boto3.client('s3', endpoint_url=endpoint_url, aws_access_key_id=os.getenv('S3_ACCESS_KEY'),
aws_secret_access_key=os.getenv('S3_SECRET_KEY'),)

try:
with open(file, 'wb') as f:
print(f"Downloading {obj} to {f} from bucket: {bucket}")
s3.download_fileobj(bucket, obj, f)
except Exception as e:
raise(e)


with open('genconf.yml') as f:
conf: dict = yaml.load(f, Loader=Loader)

Expand Down
Loading