skypilot-org · Michaelvll · Mar 14, 2024 · Feb 28, 2024 · Feb 28, 2024 · Feb 28, 2024
diff --git a/docs/source/reference/cli.rst b/docs/source/reference/cli.rst
@@ -78,6 +78,10 @@ Sky Serve CLI
    :prog: sky serve logs
    :nested: full
 
+.. click:: sky.cli:serve_update
+   :prog: sky serve update
+   :nested: full
+
 Managed Spot Jobs CLI
 ---------------------------
 

diff --git a/docs/source/running-jobs/environment-variables.rst b/docs/source/running-jobs/environment-variables.rst
@@ -108,6 +108,9 @@ Environment variables for ``setup``
   * - ``SKYPILOT_SETUP_NODE_IPS``
     - A string of IP addresses of the nodes in the cluster with the same order as the node ranks, where each line contains one IP address.
     - 1.2.3.4
+  * - ``SKYPILOT_SERVE_REPLICA_ID``
+    - The ID of a replica within the service (starting from 1). Available only for a :ref:`service <sky-serve>`'s replica task.
+    - 1
 
 Since setup commands always run on all nodes of a cluster, SkyPilot ensures both of these environment variables (the ranks and the IP list) never change across multiple setups on the same cluster.
 
@@ -139,6 +142,9 @@ Environment variables for ``run``
        If a task is run as a :ref:`managed spot job <spot-jobs>`, then all
        recoveries of that job will have the same ID value. Read more :ref:`here <spot-jobs-end-to-end>`.
      - sky-2023-07-06-21-18-31-563597_myclus_id-1
+    * - ``SKYPILOT_SERVE_REPLICA_ID``
+      - The ID of a replica within the service (starting from 1). Available only for a :ref:`service <sky-serve>`'s replica task.
+      - 1
 
 The values of these variables are filled in by SkyPilot at task execution time.
 

diff --git a/docs/source/serving/update.rst b/docs/source/serving/update.rst
@@ -10,7 +10,12 @@ SkyServe supports *updating* a deployed service, which can be used to change:
 * Service spec in ``service`` (e.g., number of replicas or autoscaling spec)
 
 During an update, the service will remain accessible with no downtime and its
-endpoint will remain the same.
+endpoint will remain the same. By default, `rolling update <rolling-update>`__
+is applied, while you can also specify a `blue-green update <blue-green-update>`__.
+.. _rolling-update:
+
+Rolling Update
+---------------
 
 To update an existing service, use ``sky serve update``:
 
@@ -22,33 +27,17 @@ SkyServe will launch new replicas described by ``new_service.yaml`` with the fol
 
 * An update is initiated, and traffic will continue to be redirected to existing (old) replicas.
 * New replicas (with new settings) are brought up in the background.
-* Once ``min_replicas`` new replicas are ready, new traffic will start to be redirected to the new
-  replicas, while old replicas will stop receiving traffic and will be scaled down.
-
-
-For example, suppose we have a running service hosting an AI model with the following resource configuration:
-
-.. code-block:: yaml
-
-    resources:
-      memory: 32+
-      accelerators: T4
+* Whenever the total number of old and new replicas exceeds the expected number of replicas (based on autoscaler's decision), extra old replicas will be scaled down.
+* Traffic will be redirected to both old and new replicas until all new replicas are ready.
 
-It is possible to update it to use a new resource configuration for all replicas, such as:
+.. hint::
 
-.. code-block:: yaml
-
-    resources:
-      memory: 128+
-      accelerators: A100
-
-To support updates, a service and its replicas are versioned (starting from 1).
-During an update, traffic is entirely serviced by either old-versioned or
-new-versioned replicas.  :code:`sky serve status` shows the latest service
-version and each replica's version.
+  When only the ``service`` field is updated and no ``workdir`` or ``file_mounts`` is specified in the service task, SkyServe will reuse the old replicas
+  by applying the new service spec and bumping its version (See :code:`sky serve status` for the versions). This will significantly reduce the time to
+  update the service.
 
 Example
-~~~~~~~
+~~~~~~~~
 
 We first launch a `simple HTTP service <https://github.com/skypilot-org/skypilot/blob/master/examples/serve/http_server/task.yaml>`_:
 
@@ -73,8 +62,8 @@ We can use :code:`sky serve status http-server` to check the status of the servi
 
 Service ``http-server`` has an initial version of 1.
 
-Suppose we want to update the service to use 4 vCPUs instead of 2. We can update
-the task yaml ``examples/serve/http_server/task.yaml``, by changing the ``cpu``
+Suppose we want to update the service to have at least 3 replicas instead of 2. We can update
+the task yaml ``examples/serve/http_server/task.yaml``, by changing the ``replicas``
 field:
 
 .. code-block:: yaml
@@ -85,11 +74,11 @@ field:
       readiness_probe:
         path: /health
         initial_delay_seconds: 20
-      replicas: 2
+      replicas: 3
 
     resources:
       ports: 8081
-      cpus: 4+
+      cpus: 2+
 
     workdir: examples/serve/http_server
 
@@ -101,57 +90,185 @@ We can then use :code:`sky serve update` to update the service:
 
     $ sky serve update http-server examples/serve/http_server/task.yaml
 
-SkyServe will trigger launching two new replicas with 4 vCPUs. Before
-``min_replicas`` (set to ``service.replicas`` when unspecified; i.e., 2) new
-replicas are ready, SkyServe will only send traffic to the old replicas.  When
-the number of new replicas reaches ``min_replicas``, SkyServe will scale down
-old replicas to save cost. The service's version is updated from 1 to 2.
-Replicas 3 and 4 are the new replicas with 4 vCPUs.  Replicas 1 and 2 are the
-old replicas with 2 vCPUs.
+SkyServe will trigger launching three new replicas.
+
+.. code-block:: console
+
+    $ sky serve status http-server
+
+    Services
+    NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
+    http-server  2        6m 15s  READY   2/5       44.206.240.249:30002
+
+    Service Replicas
+    SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS        REGION
+    http-server   1   1        54.173.203.169  6 mins ago   1x AWS(vCPU=2)  READY         us-east-1
+    http-server   2   1        52.87.241.103   6 mins ago   1x AWS(vCPU=2)  READY         us-east-1
+    http-server   3   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
+    http-server   4   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
+    http-server   5   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
+
+
+Whenever a new replica is ready, the traffic will be redirected to both old and new replicas.
+
+.. code-block:: console
+
+    $ sky serve status http-server
+
+    Services
+    NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
+    http-server  1,2        10m 4s  READY   3/5       44.206.240.249:30002
+
+    Service Replicas
+    SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS         REGION
+    http-server   1   1        54.173.203.169  10 mins ago  1x AWS(vCPU=2)  READY          us-east-1
+    http-server   2   1        52.87.241.103   10 mins ago  1x AWS(vCPU=2)  READY          us-east-1
+    http-server   3   2        3.93.241.163    1 min ago    1x AWS(vCPU=2)  READY          us-east-1
+    http-server   4   2        -               1 min ago    1x AWS(vCPU=2)  PROVISIONING   us-east-1
+    http-server   5   2        -               1 min ago    1x AWS(vCPU=2)  PROVISIONING   us-east-1
+
+
+Once the total number of both old and new replicas exceeds the requested number, old replicas will be scaled down.
+
+.. code-block:: console
+
+    $ sky serve status http-server
+
+    Services
+    NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
+    http-server  1,2        10m 4s  READY   3/5       44.206.240.249:30002
+
+    Service Replicas
+    SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS         REGION
+    http-server   1   1        54.173.203.169  10 mins ago  1x AWS(vCPU=2)  SHUTTING_DOWN  us-east-1
+    http-server   2   1        52.87.241.103   10 mins ago  1x AWS(vCPU=2)  READY          us-east-1
+    http-server   3   2        3.93.241.163    1 min ago    1x AWS(vCPU=2)  READY          us-east-1
+    http-server   4   2        18.206.226.82   1 min ago    1x AWS(vCPU=2)  READY          us-east-1
+    http-server   5   2        -               1 min ago    1x AWS(vCPU=2)  PROVISIONING   us-east-1
+
+Eventually, we will only have new replicas ready to serve user requests.
+
+.. code-block:: console
+
+    $ sky serve status http-server
+
+    Services
+    NAME         VERSION  UPTIME   STATUS  REPLICAS  ENDPOINT
+    http-server  2        11m 42s  READY   3/3       44.206.240.249:30002
+
+    Service Replicas
+    SERVICE_NAME  ID  VERSION  IP             LAUNCHED    RESOURCES       STATUS  REGION
+    http-server   3   2        3.93.241.163   3 mins ago  1x AWS(vCPU=2)  READY   us-east-1
+    http-server   4   2        18.206.226.82  3 mins ago  1x AWS(vCPU=2)  READY   us-east-1
+    http-server   5   2        3.26.232.31    1 min ago   1x AWS(vCPU=2)  READY   us-east-1
+
+
+
+.. _blue-green-update:
+
+Blue-Green Update
+------------------
+
+SkyServe also supports blue-green updates, by the following command:
+
+.. code-block:: console
+
+    $ sky serve update --mode blue_green service-name new_service.yaml
+
+
+In this update mode, SkyServe will launch new replicas described by ``new_service.yaml`` with the following behavior:
+
+* An update is initiated, and traffic will continue to be redirected to existing (old) replicas.
+* New replicas (with new settings) are brought up in the background.
+* Traffic will be redirected to new replicas only when all new replicas are ready.
+* Old replicas are scaled down after all new replicas are ready.
+
+
+During an update, traffic is entirely serviced by either old-versioned or
+new-versioned replicas.  :code:`sky serve status` shows the latest service
+version and each replica's version.
+
+Example
+~~~~~~~
+
+We use the same service ``http-server`` as an example. We can then use :code:`sky serve update --mode blue_green` to update the service:
+
+.. code-block:: console
+
+    $ sky serve update http-server --mode blue_green examples/serve/http_server/task.yaml
+
+
+SkyServe will trigger launching three new replicas.
 
 .. code-block:: console
 
     $ sky serve status http-server
 
     Services
     NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
-    http-server  2        6m 15s  READY   2/4       44.206.240.249:30002
+    http-server  2        6m 15s  READY   2/5       44.206.240.249:30002
 
     Service Replicas
     SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS        REGION
     http-server   1   1        54.173.203.169  6 mins ago   1x AWS(vCPU=2)  READY         us-east-1
     http-server   2   1        52.87.241.103   6 mins ago   1x AWS(vCPU=2)  READY         us-east-1
-    http-server   3   2        -               21 secs ago  1x AWS(vCPU=4)  PROVISIONING  us-east-1
-    http-server   4   2        -               21 secs ago  1x AWS(vCPU=4)  PROVISIONING  us-east-1
+    http-server   3   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
+    http-server   4   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
+    http-server   5   2        -               21 secs ago  1x AWS(vCPU=2)  PROVISIONING  us-east-1
 
-The old replicas will be scaled down when the new replicas are ready. At this point, SkyServe will start sending traffic to the new replicas.
+
+When a new replica is ready, the traffic will still be redirected to old replicas.
 
 .. code-block:: console
 
     $ sky serve status http-server
 
     Services
     NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
-    http-server  2        10m 4s  READY   2/4       44.206.240.249:30002
+    http-server  1        10m 4s  READY   3/5       44.206.240.249:30002
+
+    Service Replicas
+    SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS         REGION
+    http-server   1   1        54.173.203.169  10 mins ago  1x AWS(vCPU=2)  READY          us-east-1
+    http-server   2   1        52.87.241.103   10 mins ago  1x AWS(vCPU=2)  READY          us-east-1
+    http-server   3   2        3.93.241.163    1 min ago    1x AWS(vCPU=4)  READY          us-east-1
+    http-server   4   2        -               1 min ago    1x AWS(vCPU=4)  PROVISIONING   us-east-1
+    http-server   5   2        -               1 min ago    1x AWS(vCPU=4)  PROVISIONING   us-east-1
+
+
+Once the total number of new replicas satisfies the requirements, traffics will be redirected to new replicas and old replicas will be scaled down.
+
+.. code-block:: console
+
+    $ sky serve status http-server
+
+    Services
+    NAME         VERSION  UPTIME  STATUS  REPLICAS  ENDPOINT
+    http-server  2        10m 4s  READY   3/5       44.206.240.249:30002
 
     Service Replicas
     SERVICE_NAME  ID  VERSION  IP              LAUNCHED     RESOURCES       STATUS         REGION
     http-server   1   1        54.173.203.169  10 mins ago  1x AWS(vCPU=2)  SHUTTING_DOWN  us-east-1
     http-server   2   1        52.87.241.103   10 mins ago  1x AWS(vCPU=2)  SHUTTING_DOWN  us-east-1
     http-server   3   2        3.93.241.163    1 min ago    1x AWS(vCPU=4)  READY          us-east-1
     http-server   4   2        18.206.226.82   1 min ago    1x AWS(vCPU=4)  READY          us-east-1
+    http-server   5   2        3.26.232.31     1 min ago    1x AWS(vCPU=4)  READY          us-east-1
 
-Eventually, we will only have new replicas ready to serve user requests.
+Eventually, same as rolling update, we will only have new replicas ready to serve user requests.
 
 .. code-block:: console
 
     $ sky serve status http-server
 
     Services
     NAME         VERSION  UPTIME   STATUS  REPLICAS  ENDPOINT
-    http-server  2        11m 42s  READY   2/2       44.206.240.249:30002
+    http-server  2        11m 42s  READY   3/3       44.206.240.249:30002
 
     Service Replicas
     SERVICE_NAME  ID  VERSION  IP             LAUNCHED    RESOURCES       STATUS  REGION
     http-server   3   2        3.93.241.163   3 mins ago  1x AWS(vCPU=4)  READY   us-east-1
     http-server   4   2        18.206.226.82  3 mins ago  1x AWS(vCPU=4)  READY   us-east-1
+    http-server   5   2        3.26.232.31    1 min ago   1x AWS(vCPU=4)  READY   us-east-1
+
+
+
diff --git a/sky/cli.py b/sky/cli.py
@@ -3756,6 +3756,14 @@ def serve_up(
                 nargs=-1,
                 **_get_shell_complete_args(_complete_file_name))
 @_add_click_options(_TASK_OPTIONS + _EXTRA_RESOURCES_OPTIONS)
+@click.option('--mode',
+              default=serve_lib.UpdateMode.ROLLING.value,
+              type=click.Choice([m.value for m in serve_lib.UpdateMode],
+                                case_sensitive=False),
+              required=False,
+              help=('Update mode. If "rolling", SkyServe will update the '
+                    'service with rolling update. If "blue_green", SkyServe '
+                    'will update the service with blue-green update. '))
 @click.option('--yes',
               '-y',
               is_flag=True,
@@ -3783,21 +3791,35 @@ def serve_update(
     memory: Optional[str],
     disk_size: Optional[int],
     disk_tier: Optional[str],
+    mode: str,
     yes: bool,
 ):
     """Update a SkyServe service.
 
+    service_yaml must point to a valid YAML file.
+
     SkyServe will reuse old replicas, if only the service section is changed
     and no file mounts are specified.
     Otherwise, SkyServe will terminate the old replicas and start new replicas.
 
-    service_yaml must point to a valid YAML file.
+    Two update modes are supported:
+    - "rolling": (default) SkyServe will update the service with rolling update,
+        i.e., it will terminate one old replica whenever one new replica is
+        ready. Traffic can be mixed on old and new replicas.
+    - "blue_green": SkyServe will update the service with blue-green update,
+        i.e., it will wait for new replicas to be ready and then terminate old
+        replicas. Traffic will only be switched from old to new replicas after
-        replicas. Traffic will only be switched from old to new replicas after
+        replicas. Traffic will only be served from old to new replicas after
-        replicas. Traffic will only be switched from old to new replicas after
+        replicas. Traffic will only be served from old to new replicas after
+        enough new replicas are ready.
 
     Example:
 
     .. code-block:: bash
 
+        # Update an existing service with rolling update
         sky serve update sky-service-16aa new_service.yaml
+        # Use blue-green update
+        sky serve update --mode blue_green sky-service-16aa new_service.yaml
+
     """
     task = _generate_task_with_service(
         service_name=service_name,
@@ -3835,7 +3857,7 @@ def serve_update(
                       abort=True,
                       show_default=True)
 
-    serve_lib.update(task, service_name)
+    serve_lib.update(task, service_name, mode=serve_lib.UpdateMode(mode))
 
 
 @serve.command('status', cls=_DocumentedCodeCommand)

diff --git a/sky/serve/__init__.py b/sky/serve/__init__.py
@@ -19,6 +19,7 @@
 from sky.serve.serve_utils import ServeCodeGen
 from sky.serve.serve_utils import ServiceComponent
 from sky.serve.serve_utils import SKY_SERVE_CONTROLLER_NAME
+from sky.serve.serve_utils import UpdateMode
 from sky.serve.service_spec import SkyServiceSpec
 
 os.makedirs(os.path.expanduser(SKYSERVE_METADATA_DIR), exist_ok=True)
@@ -43,4 +44,5 @@
     'tail_logs',
     'up',
     'update',
+    'UpdateMode',
 ]