Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HiTL Dashboard - View and Update Model Checksum #1301

Open
wants to merge 10 commits into
base: hitl_dashboard_model_viz
Choose a base branch
from
Open

HiTL Dashboard - View and Update Model Checksum #1301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a0afc34
add frontend card
mialsy Jul 22, 2022
721b0cd
updated frontend
mialsy Jul 22, 2022
df6d6c2
add backend api for reteriving checksum from local path
mialsy Jul 22, 2022
fcefb35
impl backend api for computing checksum
mialsy Jul 22, 2022
9cb7cdc
Automatic style fix for droidlet
Jul 22, 2022
77d7a79
updated documentation
mialsy Jul 22, 2022
c3a9e07
Merge branch 'hitl_dashboard_model_checksum' of https://github.com/fa…
mialsy Jul 22, 2022
fd91113
upadte readme
mialsy Jul 22, 2022
54c3573
Update README.MD
mialsy Jul 22, 2022
4a7f707
fixed typo
mialsy Jul 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion droidlet/tools/artifact_scripts/tracked_checksums/locobot_perception.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0c6b0070ca1f30b1420c4db9259789c2d6442a3a
da39a3ee5e6b4b0d3255bfef95601890afd80709
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be checked in? We should only update the tracked checksums if there's an update to the associated artifacts.

2 changes: 1 addition & 1 deletion droidlet/tools/artifact_scripts/tracked_checksums/nlu.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
b15b887d3795140a5fab42a68d7369fcebf28e8c
da39a3ee5e6b4b0d3255bfef95601890afd80709
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above.

29 changes: 27 additions & 2 deletions droidlet/tools/hitl/dashboard_app/README.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
# Dashboard App for HITL
Updated July 13 by Chuxi.
Updated July 22 by Chuxi.

This is a Dashboard app prototype for the HITL system.

## Update Note
- July 22:
- Added backend api and frontend components for viewing current model checksum and updating checksum.
- Demo:
- View and update checksum:
- ![demo_view_n_update_model_checksum](https://user-images.githubusercontent.com/51009396/180574655-2ffac43a-7e50-473b-a4aa-5379753b04cd.gif)
- July 14:
- Added model visualization component, user can see text info about a model:
- Demo:
Expand Down Expand Up @@ -119,6 +124,26 @@ APIs are based on socket event, the following APIs are supported currently:
- the batch id of the run.
- the key for the model, could be any key from the model, or "COMPLETE", indicating getting the complete model dict
- output: the key and the value specific to the key for the model if the model exists and key is valid, otherwise error code

- get_model_checksum_by_name_n_agent
- get the checksum for a specific model and agent
- input:
- model name
- agent name
- the valid combinations for model name and agent are:
- nlu
- perception locobot
- perception craftassist
- output: the checksum if model and agent combination are valid, and if checksum has been computed; otherwise error code
- update_model_checksum_by_name_n_agent
- update the checksum for a specific model and agent
- input:
- model name
- agent name
- the valid combinations for model name and agent are:
- nlu
- perception locobot
- perception craftassist
- output: an success code if update success

## Demo
![backend_api_demo](https://user-images.githubusercontent.com/51009396/175696481-532cec55-5b2e-4bae-bceb-9e7d3f2aa7b7.gif)
29 changes: 29 additions & 0 deletions droidlet/tools/hitl/dashboard_app/backend/dashboard_model_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,12 @@
"""
import argparse
import collections
import os
import torch
import json

ROOTDIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../../../../")


def load_model(model_fpath: str):
"""
Expand Down Expand Up @@ -47,3 +50,29 @@ def get_complete_model(model):
for key in model.keys():
model_dict[key] = get_value_by_key(model, key)
return json.dumps(model_dict)


def get_model_checksum_by_name_n_agent(model_name, agent=None):
"""
helper method to get model checksum
"""
checksum_name = ""
if model_name == "nlu":
checksum_name = "nlu.txt"
elif model_name == "perception":
if agent == "locobot":
checksum_name = "locobot_perception.txt"
elif agent == "craftassist":
checksum_name = "craftassist_perception.txt"

checksum_path = os.path.join(
ROOTDIR, "droidlet/tools/artifact_scripts/tracked_checksums/" + checksum_name
)

if os.path.exists(checksum_path):
f = open(checksum_path)
checksum = f.read()
f.close()
return checksum, None
else:
return f"Cannot find checksum for model = {model_name}, agent = {agent}", 404
49 changes: 48 additions & 1 deletion droidlet/tools/hitl/dashboard_app/backend/dashboard_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
"""


import argparse
from enum import Enum
import json
from droidlet.tools.artifact_scripts.compute_checksum import compute_checksum_for_directory
from droidlet.tools.hitl.dashboard_app.backend.dashboard_aws_helper import (
get_dataset_by_name,
get_dataset_indices_by_id,
Expand All @@ -22,6 +22,7 @@
from droidlet.tools.hitl.dashboard_app.backend.dashboard_model_utils import (
get_complete_model,
get_keys,
get_model_checksum_by_name_n_agent,
get_value_by_key,
)
from flask import Flask, abort
Expand Down Expand Up @@ -49,6 +50,9 @@ class DASHBOARD_EVENT(Enum):
GET_MODEL_KEYS = "get_model_keys_by_id"
GET_MODEL_VALUE = "get_model_value_by_id_n_key"

GET_MODEL_CHECKSUM = "get_model_checksum_by_name_n_agent"
UPDATE_MODEL_CHECKSUM = "update_model_checksum_by_name_n_agent"


# constants for model related apis
KEY_COMPLETE = "complete_model"
Expand Down Expand Up @@ -225,5 +229,48 @@ def get_model_value(batch_id, key):
emit(DASHBOARD_EVENT.GET_MODEL_VALUE.value, [key, get_value_by_key(model, key)])


@socketio.on(DASHBOARD_EVENT.GET_MODEL_CHECKSUM.value)
def get_model_checksum(model_name, agent):
"""
get the checksum for a specific model and agent
- input:
- model name
- agent name
- the valid combinations for model name and agent are:
- nlu
- perception locobot
- perception craftassist
- output: the checksum if model and agent combination are valid, and if checksum has been computed; otherwise error code
"""
print(
f"Request received: {DASHBOARD_EVENT.GET_MODEL_CHECKSUM.value}, model = {model_name}, agent = {agent}"
)
checksum, error_code = get_model_checksum_by_name_n_agent(model_name, agent)
if error_code:
emit(DASHBOARD_EVENT.GET_MODEL_CHECKSUM.value, [model_name, agent, error_code])
else:
emit(DASHBOARD_EVENT.GET_MODEL_CHECKSUM.value, [model_name, agent, checksum])


@socketio.on(DASHBOARD_EVENT.UPDATE_MODEL_CHECKSUM.value)
def update_model_checksum(model_name, agent):
"""
update the checksum for a specific model and agent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to be really clear what this does, both in this comment as well as maybe some text description shown to the user. "Update checksum" implies to me that we're updating it to a value we specify. This computes the checksum based on the artifacts currently in place on your branch, and then pushes that checksum to the tracked file.

- input:
- model name
- agent name
- the valid combinations for model name and agent are:
- nlu
- perception locobot
- perception craftassist
- output: an success code if update success
"""
print(
f"Request received: {DASHBOARD_EVENT.UPDATE_MODEL_CHECKSUM.value}, model = {model_name}, agent = {agent}"
)
compute_checksum_for_directory(agent, "models", model_name)
emit(DASHBOARD_EVENT.UPDATE_MODEL_CHECKSUM.value, 200)


if __name__ == "__main__":
socketio.run(app)
4 changes: 2 additions & 2 deletions droidlet/tools/hitl/dashboard_app/dashboard_frontend/src/App.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import Main from './component/main';
import { Routes, Route, BrowserRouter } from "react-router-dom";
import NavBar from './component/navbar';
import { SUBPATHS } from './constants/subpaths';
import { BackTop, Layout, Typography } from 'antd';
import { BackTop, Layout } from 'antd';
import PipelinePanel from './component/pipeline/panel';
import DetailPage from './component/pipeline/detail/detailPage';
import JobInfoCard from './component/pipeline/detail/job/jobInfoCard';
Expand All @@ -36,7 +36,7 @@ function App() {
{/* Routes for different pipeline */}
<Layout>
<Routes>
<Route path={SUBPATHS.HOME.key} element={<div><Typography.Title>Welcome to Droidlet HiTL Dashboard</Typography.Title></div>} />
<Route path={SUBPATHS.HOME.key} element={<Main />} />
<Route path={SUBPATHS.NLU.key} element={<PipelinePanel pipelineType={SUBPATHS.NLU} />}>
<Route path=":batch_id" element={<DetailPage pipelineType={SUBPATHS.NLU.label} />}>
<Route path=":job" element={<JobInfoCard />} />
Expand Down
Loading