FLOWViZ is a middleware for facilitating the integration of a state of the art data-centric workflow system, Apache Airflow, within web applications for phylogenetic analyses.

It is composed by two components: an HTTP Express server and a React client, both written in JavaScript.

With FLOWViZ, you can integrate your own phylogenetic tools, by filling a contract where you specify the tool's access, rules and guidelines.

• Docker
• Node.JS
• Tmux (optional)

Before going into the project's setup, follow this guide, in order to fulfill all setup's requirements.

If you use a Linux distribution that use systemd, you can easily execute the scripted setup by the following order (provided you have already installed the previously mentioned requirements):

1. Execute setupAirflow.sh. Make sure Airflow has all its services running, including the Web server (access it first, before going to step 2);

2. Execute setup.sh. This script will also copy the default .env files from defaults/. If you want to deploy the project or add your own values, you must change the copied .env files (in ./.env and ./client/.env), specially credentials and sensitive information;

3. (Optional, requires tmux) Execute start.sh to start both client and server in different tmux sessions.

Note: Execute setupAirflow.sh and setup.sh just once.

It is advised to follow the setup guide by the displayed order.

1. Download mongo image

docker pull mongo

2. Run and create mongo container, exposing port 27017

docker run --name mongodb -d -p 27017:27017 mongo

1. Configure the Airflow user (inside airflow/):

echo -e "AIRFLOW_UID=$(id -u)" > .env

2. Install and run the Airflow container, along with the required services by executing the following command inside airflow/:

Note: older versions require you to write docker-compose up instead docker compose up, this has now changed with the latest version. Depending on which package you use, both can be support or just one of them; you can also create an alias (bash tutorial, also compatible with zsh or fish) to better suit your needs. For more information check the head notes here and this article.

docker compose up

To check running containers:

docker ps

After all services are up and running, Airflow will expose a web client, which is accessible through the 8080 port (http://localhost:8080). The default credentials are username: airflow and password: airflow. After a successful login, you might see a dashboard containing a list of DAG examples.

3. Create a Docker network:

docker network create flowviz-docker-network

4. Add MongoDB container and all Airflow containers to the network (the containers must be running):

docker network connect flowviz-docker-network mongodb && \
for airflowContainer in $(docker ps --format {{.Names}} | grep "airflow-"); \
do \
    docker network connect flowviz-docker-network $airflowContainer; \
done

5. Inspect the assigned IP addresses inside the network (following command) and retrieve the MongoDB's container IP address:

docker network inspect flowviz-docker-network

Or, simply copy the result of this command (you may need to reset the container for this command to work):

docker inspect -f '{{with index .NetworkSettings.Networks "flowviz-docker-network"}}{{.IPAddress}}{{end}}' mongodb

6. Inside the Apache Airflow web client (http://localhost:8080), using the NavBar go to Admin and then Connections. Click add a new record (plus icon) and fulfill the displayed fields with the following information:

Connection Id: mongodb_flowviz
Connection Type: mongo
Host: [place the retrieved IP address from the MongoDB's container]

7. Copy the dag_generator.py script into the dags/ folder (must be in the same directory where the docker-compose.yaml is).

8. Also, copy the dag_template.py script into the include/ folder (in the same directory).

9. Inside the Airflow's dashboard and toggle on the dag_generator DAG (switch on the left of the DAG's name).

If everything went well, no errors should be displayed by the client (aka it must not appear that mongodb_flowviz connection, used by the dag_generator DAG, is not recognized).

1. Create a file called .env inside the main folder.

2. Fill it with the variables below (remove the curly brackets and change what is in between them):

PRODUCTION={true|false}
SERVER_NAME={server_name}
SERVER_PORT={server_port_number}
DATABASE_ADDRESS={database_address}
DATABASE_PORT={database_port}
AIRFLOW_ADDRESS={airflow_address}
AIRFLOW_PORT={airflow_port}
AIRFLOW_DAG_GENERATOR={airflow_dag_generator_name}
JWT_SECRET={jwt_secret}
AIRFLOW_USERNAME={airflow_username}
AIRFLOW_PASSWORD={airflow_password}

1. Create a file called .env inside the client/ folder.

2. Fill it with the variables below (remove the curly brackets and change what is in between them):

REACT_APP_SERVER_PROTOCOL={protocol}
REACT_APP_SERVER_ADDRESS={server_address}
REACT_APP_SERVER_PORT={server_port}

1. Install npm package dependencies (inside main folder and client/):

npm i

2. To run both client and server (concurrently dependency, run this command inside the main folder):

npm run dev

Allows you to isolate each log in a terminal individual session, providing better log visibility than the concurrently way.

1. Tmux setup (last item)

2. Execute the start.sh script

Note: if there are no execution permissions, execute:

chmod +x start.sh

Start:

./start.sh

To run unit tests (main or client/ folders):

npm test

Further documentation about the developed solution can be found in this repository wiki, namely the REST API's endpoints.

• Miguel Luís - A43504@alunos.isel.pt
• Cátia Vaz - cvaz@cc.isel.ipl.pt
• Alexandre Francisco - aplf@ist.utl.pt

This project was developed under the context of a Lisbon School of Engineering (ISEL) Master's degree final project, which was also funded by student grants, as follows:

• NGPHYLO PTDC/CCI-BIO/29676/2017, an INESC-ID project, funded by Science and Technology Foundation (FCT);
• IPL/ISEL/DIVA_ISEL, funded by Polytechnic Institute of Lisbon (IPL).

The following articles were also submitted under the context of this project:

• https://inforum.org.pt/sites/default/files/2022-09/Actas_INForum.pdf#page=224, single-column format, submitted and publicly presented at the INForum 2022 conference, which took place at the Polytechnic Institute of Guarda (IPG);