A utility python package that covers the common libraries we use.

Since this is hosted privately on git, you will need to be under VPN, then run

pip install git+ssh://git@github.com/hipagesgroup/data-ml-utils@v0.2.2

Almost one liner

import os
from data_ml_utils.pyathena_client.client import PyAthenaClient

os.environ["AWS_ACCESS_KEY_ID"] = "xxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxx"

pyathena_client = PyAthenaClient()

Almost one liner

query = """
    SELECT
        *
    FROM
        dev.test_tutorial_table
    LIMIT 10
"""

df_raw = pyathena_client.query_as_pandas(final_query=query)

Visit link

• You suggest, raise a feature request issue and we will review!

There is a jupyter notebook to show how to use the package utility package for pyathena: notebook

There is a jupyter notebook to show how to use the package utility package for EMR: notebook

There is a jupyter notebook to show how to use the package utility package for Sagemaker: notebook