Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scale idled environments up if SSH connections are made #2583

Merged
merged 6 commits into from
Mar 29, 2021
64 changes: 64 additions & 0 deletions services/ssh/home/rsh.sh
Original file line number Diff line number Diff line change
@@ -16,6 +16,10 @@ USER_SSH_KEY=$2
REQUESTED_PROJECT=$3
shift 3

# get the value from an envvar override (can be added to the ssh deployment)
# default to false so we don't hold up the ssh for a long time
WAIT_TO_UNIDLE_SERVICES=${WAIT_TO_UNIDLE_SERVICES:-false}

# get the graphql endpoint, if set
eval "$(grep GRAPHQL_ENDPOINT /authorize.env)"

@@ -120,6 +124,36 @@ fi

# If there is a deployment for the given service searching for lagoon.sh labels
if [[ $($OC get deployment -l "lagoon.sh/service=${SERVICE}" 2> /dev/null) ]]; then
# get any other deployments that may have been idled by the idler and unidle them if required
# this only needs to be done for kubernetes
# we do this first to give the services a bit of time to unidle before starting the one that was requested
DEPLOYMENTS=$($OC get deployments -l "idling.amazee.io/watch=true" -o name)
if [ ! -z "${DEPLOYMENTS}" ]; then
# loop over the deployments and unidle them
for DEP in ${DEPLOYMENTS}
do
# if the deployment is idled, unidle it :)
DEP_JSON=$($OC get ${DEP} -o json)
if [ $(echo "$DEP_JSON" | jq -r '.status.replicas // 0') == "0" ]; then
REPLICAS=$(echo "$DEP_JSON" | jq -r '.metadata.annotations."idling.amazee.io/unidle-replicas" // 1')
if [ ! -z "$REPLICAS" ]; then
REPLICAS=1
fi
$OC scale --replicas=${REPLICAS} ${DEP} >/dev/null 2>&1
# for unidling an entire environment and waiting for the number of `readyReplicas`
# to be 1 for each deployment, could add considerable delays for the ssh connection to establish.
# WAIT_TO_UNIDLE_SERVICES will default to false so that it just scales the deployments
# and won't wait for them to be ready, but if set to true, it will wait for `readyReplicas` to be 1
if [[ "$WAIT_TO_UNIDLE_SERVICES" =~ [Tt][Rr][Uu][Ee] ]]; then
while [[ ! $($OC get ${DEP} -o go-template --template='{{.status.readyReplicas}}') == "1" ]]
do
sleep 1
done
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest two changes:

  1. can we split the actual unidling and the waiting into two for loops? In this case we first trigger scaling of all deployments and then maybe wait for them to be available, which would probably reduce the time that we need to wait.
  2. if a deployment had 2 replicas, the --template='{{.status.readyReplicas}}') == "1" will fail, I think we should just check that is has 1 or more replicas.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 1) waiting for all services in the environment to become available could add considerable delays to establishing the ssh connection though. Which is why I did it the way I did, where it unidles them but won't wait for them to become ready (unless the WAIT_TO_UNIDLE_SERVICES var is set to true, which defaults to false)

I'll fix the 1+ replica check now.

fi
done
fi
# then actually unidle the service that was requested
DEPLOYMENT=$($OC get deployment -l "lagoon.sh/service=${SERVICE}" -o name)
# If the deployment is scaled to 0, scale to 1
# .status.replicas doesn't exist on a scaled to 0 deployment in k8s so assume it is 0 if nothing is returned
@@ -138,6 +172,36 @@ fi
# If there is a deployment for the given service search for lagoon labels
# @DEPRECATED: Remove with Lagoon 2.0.0
if [[ $($OC get deployment -l lagoon/service=${SERVICE} 2> /dev/null) ]]; then
# get any other deployments that may have been idled by the idler and unidle them if required
# this only needs to be done for kubernetes
# we do this first to give the services a bit of time to unidle before starting the one that was requested
DEPLOYMENTS=$($OC get deployments -l "idling.amazee.io/watch=true" -o name)
if [ ! -z "${DEPLOYMENTS}" ]; then
# loop over the deployments and unidle them
for DEP in ${DEPLOYMENTS}
do
# if the deployment is idled, unidle it :)
DEP_JSON=$($OC get ${DEP} -o json)
if [ $(echo "$DEP_JSON" | jq -r '.status.replicas // 0') == "0" ]; then
REPLICAS=$(echo "$DEP_JSON" | jq -r '.metadata.annotations."idling.amazee.io/unidle-replicas" // 1')
if [ ! -z "$REPLICAS" ]; then
REPLICAS=1
fi
$OC scale --replicas=${REPLICAS} ${DEP} >/dev/null 2>&1
# for unidling an entire environment and waiting for the number of `readyReplicas`
# to be 1 for each deployment, could add considerable delays for the ssh connection to establish.
# WAIT_TO_UNIDLE_SERVICES will default to false so that it just scales the deployments
# and won't wait for them to be ready, but if set to true, it will wait for `readyReplicas` to be 1
if [[ "$WAIT_TO_UNIDLE_SERVICES" =~ [Tt][Rr][Uu][Ee] ]]; then
while [[ ! $($OC get ${DEP} -o go-template --template='{{.status.readyReplicas}}') == "1" ]]
do
sleep 1
done
fi
fi
done
fi
# then actually unidle the service that was requested
DEPLOYMENT=$($OC get deployment -l lagoon/service=${SERVICE} -o name)
# If the deployment is scaled to 0, scale to 1
# .status.replicas doesn't exist on a scaled to 0 deployment in k8s so assume it is 0 if nothing is returned