Make PipeCD control plane support plugin-arched piped #5252

ffjlabo · 2024-10-04T04:04:33Z

What would you like to be added:

This issue tracks the progress for deprecating Platform Provider and Kind toward piped plugin architecture in pipedv1.

The purpose is to manage application kind and platform providers on the PipeCD plugin side in the future.

Considering the impact on users, pay attention to the following points:

In Pipedv1, platform providers, kind remain on the data and do not create logic with them. Use the value you define instead.
minify the migration for users.

current update process

Update data for the pipedv1

Add the label kind based on the current Application Kind to the Application.
Add deployTarget based on the current Application Platform Provider to the Application.

update to pipedv1 (users can use existing application at the time)

Details (under planning)

For Platform Provider

Instead of Platform Provider, we plan to introduce the config for the plugin and define deployTargets.

piped config

apiVersion: pipecd.dev/v1beta1
kind: Piped
spec:
...
  plugin:
    - name: k8s_plugin
      port: 8081
      deployTargets:
        - name: dev
          labels: 
            env: dev
          config: # depends on plugins
            masterURL: http://cluster-dev
            kubeConfigPath: ./kubeconfig-dev

type PipedDeployTarget struct {
	Name   string                     `json:"name"`
	Labels map[string]string          `json:"labels,omitempty"`
	Config json.RawMessage            `json:"config"`
}

We also plan to deploy the app to multiple targets at once in a multicluster feature for k8s.
So, we define DeployTargets as an array in Application and Deployment.

Application

message Application {
    reserved 3;
    ...
    // TODO: Add validation for this field.
    string platform_provider = 15;
    // 
    repeated string deploy_targets = 16;
    ...
}

Deployment

message Deployment {
    reserved 4;
    ...
    // The name of platform provider where to deploy this application.
    // This must be one of the provider names registered in the piped.
    string platform_provider = 11;
    
    repeated string deploy_targets = 12;
}

For the backward compatibility

Before updating the piped, users should migrate the current data on DB.

Former idea

During the migration, there might be both platform providers and deploy targets in the piped config.
So we need to convert the platform providers to deploy targets internally.

Refer the Platform Provider or Deploy Target

If the ApplicationKind is Application, just use DeployTarget
If the ApplicationKind is old one, convert PlatformProvider to DeployTarget

func (s *PipedSpec) FindDeployTarget(name string, t model.ApplicationKind) (*PipedDeployTarget, bool) {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If not, the deploy target is the platform provider.
	// For backward compatibility, the deploy target is the platform provider.
	if t != model.ApplicationKind_APPLICATION {
		p, found := s.FindPlatformProvider(name, t)
		if !found {
			return &PipedDeployTarget{}, false
		}
		return &PipedDeployTarget{
				Name:   p.Name,
				Labels: p.Labels,
				Config: p.Config,
			}, true
	}

	// If the application is supported by the plugin architecture, the deploy target is the deploy target.
	for _, dt := range s.DeployTargets {
		if dt.Name == name {
			return dt, true
		}
	}

	return &PipedDeployTarget{}, false
}

For Kind

Instead of Kind, we plan to introduce the label to represent the application kind.

apiVersion: pipecd.dev/v1beta1
kind: Application
metadata:
  labels:
    kind: KUBERNETES # <- like this
spec:
  name: myApp

For the builtin plugins, we define 5 labels as string.

KUBERNETES
ECS
LAMBDA
CLOUDRUN
TERRAFORM

For the backward compatibility

Before updating the piped, users should migrate the current data on DB.

Former idea

We need to support both before and after creating plugin architecture for now.
So, I propose the way to decide the application kind like this.

Define APPLICATION as ApplicationKind
Add new method to decide the actual kind for Application and Deployment.

enum ApplicationKind {
    KUBERNETES = 0;
    TERRAFORM = 1;
    LAMBDA = 3;
    CLOUDRUN = 4;
    ECS = 5;
    APPLICATION = 6; <- new! 
}

func (a * Application) GetKind() string {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If so, return the kind from the labels.
	if a.Kind == ApplicationKind_Application {
		return a.Labels["kind"]
	}

	// For backward compatibility, return the kind as string
	return a.Kind.String()
}

func (d *Deployment) GetKind() string {
	// First, check the application is supported by the plugin architecture. It means that the kind is set to "Application".
	// If so, return the kind from the labels.
	if d.Kind == ApplicationKind_Application {
		return d.Labels["kind"]
	}

	// For backward compatibility, return the kind as string
	return d.Kind.String()
}

TODO

Firstly, I listed the target features to require some modifications.
We need to further investigation to decide the way to fix.

investigation

implementation

Why is this needed:

The text was updated successfully, but these errors were encountered:

ffjlabo · 2024-10-22T06:58:37Z

Platform Provider; Livestatestore

The purpose of the component

Livestatestore stores the actual resource state for each app belonging to the platform.
Livestatestore is almost all for the platform-specific things such as detector, livestatereporter, and executor.

Modification idea

Put it on the plugin side
Store the app resource state per deployTarget
Introduce an aggregation layer to get multiple resource states by appID across the multiple deployTarget, In case of the multiple deployTarget per app,

ffjlabo · 2024-10-23T03:57:14Z

Platform Provider; Livestatereporter

The purpose of the component

Livestatereporter sends the application state to the Control Plane
Currently, There are livestatereporters for each platform provider

Modification idea

piped side
- Every minute, piped does below
  - List apps by plugin name
  - Get the app livestate from the plugin via gRPC call e.g. GetLivestate(app)
  - Send them to the Control Plane
plugin side
- Get the livestates grouped by deployTarget from livestatestore and return to the piped.
Control Plane side
- Receive and store the livestates

Concern

There is a different way to update livestate on Control Plane between k8s and other platforms.

other platform (ecs, cloud run, lambda) ref for cloud run:

pipecd/pkg/app/piped/livestatereporter/cloudrun/report.go

Lines 85 to 86 in 2f86f7d

case <-snapshotTicker.C:

r.flushSnapshots(ctx)
- Just flush livestates to the Control Plane every minute.

k8s:

pipecd/pkg/app/piped/livestatereporter/kubernetes/reporter.go

Lines 99 to 103 in 2f86f7d

    
           case <-snapshotTicker.C: 
        
           	r.flushSnapshots(ctx) 
        
           case <-ticker.C: 
        
           	r.flushEvents(ctx)

Flush livestates to the Control Plane every 10 minutes.
Send the new livestates to the Control Plane every 5 seconds during the 10 minutes

I think this is mainly for the speed to show it on the UI.
It would be nice to add an option like livestateFlushInterval to change them by users.

ffjlabo · 2024-10-23T04:11:07Z

Platform Provider; Detector

The purpose of the component

Detector check and send application sync state to the Control Plane
Currently, There are detectors for each platform provider

Modification idea

piped side
- Every minute, piped does below
  - List apps by plugin name
  - Get the application sync states from the plugin via gRPC call e.g. Detect(app)
  - Send them to the Control Plane
plugin side
- Get the liveManifests grouped by deployTarget from livestatestore
- Calculate the diff between live ones and git ones.
- Make the app sync state referencing the diff and send it to the Control Plane

Concern

There is a different interval to execute the detectors between terraform and other platforms.
- other platform: 1 minute
- terraform: 10 minutes
  - This might be to solve performance problem Update the interval for Terraform's drift detection #4181
It would be nice to add an option like detectInterval to change them by users.

ffjlabo · 2024-10-29T07:51:35Z

Platform Provider; Plan Preview

The purpose of the component

Make Plan Preview results ( the diffs, the plan result) for the apps which have some changes

Modification idea

piped side
- List commands for Plan Preview from the Control Plane by commandCheckInterval
- Every command, list apps managed by the piped using repoID related to the commands
- Every app, extract the apps that have some diffs
- Get the PlanPreviewResult from the plugin via gRPC call e.g. BuildPlanPreviewResult(app)
- Send all results of the apps to the ControlPlane with a command
plugin side
- Get the manifests on the latest commit
- Get the manifests on the commit used by the latest Deployment
- Make a PlanPreviewResult using the diffs between the above

Concern

It might be good to extract the apps that have some diffs on the plugin side

ffjlabo · 2024-10-31T07:48:23Z

Platform Provider; Metrics

The metrics deployment_status is affected because it has the label platform_provider.
https://github.com/pipe-cd/pipecd/blob/548a804d344d7c5faa70bcb6bd0889aa2a9269af/pkg/app/piped/controller/controllermetrics/metrics.go#L23C1-L40C2

deployment_status{application_id="xxxxxxxx",application_kind="KUBERNETES",application_name="simple",deployment="xxxxxxx",instance="127.0.0.1:9085",job="pipecd-ops",launcher_version="v0.49.2",pipecd_component="piped",piped="xxxxx",piped_version="v0.49.2",platform_provider="kubernetes",project="test",status="DEPLOYMENT_CANCELLED"}

Modification idea
There are two ideas.

Create new metrics like plugin_deployment_status
Add a new label deployTarget to the deployment_status

Concern

For idea 1.

pros
- Completely create it based on the new data model
cons
- We should observe both of the metrics (e.g. create two views for both on grafana)
- It is hard to check the deployment status from piped < v1 to pipedv1 because they are different metrics.

For idea 2.

pros
- We can continue to use it.
cons
- We have to modify the query to check both of them like deployment_status{platform_provider="kubernetes"} or deployment_status{deployTarget="kubernetes"}

ffjlabo · 2024-10-31T08:27:39Z

Kind; appconfigreporter

The purpose of the component

Detect and update the app config when it has some changes.
Send the changed app info to the Control Plane, then the Control Plane updates the app with it.

It has logic for checking the difference between the app info in the current app config and the already registered app.

pipecd/pkg/app/piped/appconfigreporter/appconfigreporter.go

Lines 279 to 285 in 4a2334a

    
           if appInfo.Kind != app.Kind { 
        
           	r.logger.Warn("kind in application config has been changed which isn't allowed", 
        
           		zap.String("app-id", app.Id), 
        
           		zap.String("repo-id", app.GitPath.Repo.Id), 
        
           		zap.String("config-file-path", app.GitPath.GetApplicationConfigFilePath()), 
        
           	) 
        
           }

Especially, compare values in model.ApplicationInfo and corresponding in the model.Application (e.g. Kind, Name...)

Modification idea

Compare model.Application.Labels["kind"] instead of model.Application.Kind

Concern

Nothing for now

ffjlabo · 2024-10-31T09:18:46Z

Kind; event watcher

The purpose of the component

Event Watcher handles events according to the rule set in the app config.
When it loads the app config, it calls config.LoadAplication(repoPath, configRelPath string, appKind model.ApplicationKind)

pipecd/pkg/app/piped/planpreview/builder.go

Line 307 in 4a2334a

appCfg, err := config.LoadApplication(repo.GetPath(), app.GitPath.GetApplicationConfigFilePath(), app.Kind)

Modification Idea

Modify config.LoadApplication(repoPath, configRelPath string, appKind model.ApplicationKind) to LoadApplication(repoPath, configRelPath string, appKind string)
Use model.Application.Labels["kind"] instead of model.Application.Kind

ffjlabo · 2024-11-05T08:26:29Z

Kind; livestatereporter

The purpose of the component

Livestatereporter sends the application state to the Control Plane
- pipecd/pkg/app/piped/livestatereporter/cloudrun/report.go
  
  Line 112 in 4a2334a
  
  Kind: app.Kind,
Currently, There are livestatereporters for each platform provider

Modification Idea

Add plugin_kind to the model.ApplicationLiveStateSnapshot
Use model.Application.Labels["kind"] instead of model.Application.Kind

message ApplicationLiveStateSnapshot {
    reserved 2;
    ...
    ApplicationKind kind = 5 [(validate.rules).enum.defined_only = true];
    ...
    string plugin_kind = 16;
}

ffjlabo · 2024-11-05T09:18:11Z

Kind; detector

The purpose of the component

Detector check and send application sync state to the Control Plane
Currently, There are detectors for each platform provider
It has logic for checking
- whether the kind in the config is supported
- whether the kind in the config and the one in the actual app are equal
- pipecd/pkg/app/piped/driftdetector/cloudrun/detector.go
  
  Line 300 in 44cfc93
  
  func (d *detector) loadApplicationConfiguration(repoPath string, app *model.Application) (*config.Config, error) {

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

ffjlabo · 2024-11-05T09:26:47Z

Kind; notifier

The purpose of the component

Notifier notifies the deployment progress to the slack
piped uses kind to show deployment info on the slack message

pipecd/pkg/app/piped/notifier/slack.go

Line 235 in 44cfc93

{"Kind", strings.ToLower(app.Kind.String()), true},

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

ffjlabo · 2024-11-05T12:39:31Z

Kind; Plan preview

The purpose of the component

Make Plan Preview results ( the diffs, the plan result) for the apps which have some changes
There are three purpose to use kind
1. Determine the logic to make diff result. This is a platform-specific thing.
  
  pipecd/pkg/app/piped/planpreview/builder.go
  
  Line 245 in 44cfc93
  
  switch app.Kind {
2. Determine the planner to make plan result. This is also a platform-specific thing.
  
  pipecd/pkg/app/piped/planpreview/builder.go
  
  Line 332 in 44cfc93
  
  p, ok := defaultPlannerRegistry.Planner(app.Kind)
3. Show it on the log.
  
  pipecd/pkg/app/piped/planpreview/builder.go
  
  Line 210 in 44cfc93
  
  zap.String("app-kind", app.Kind.String()),

Modification idea

For 1, the actual logic will be implemented in the plugin side.
For 2, we don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin.
For 3, just use labels["kind"]

ffjlabo · 2024-11-15T06:56:50Z

Kind; trigger

Trigger mainly creates a deployment based on commands.
There are two purposes for using kind
1. Add Kind to Deployment
  
  pipecd/pkg/app/piped/trigger/deployment.go
  
  Line 80 in 44cfc93
  
  Kind: app.Kind,
2. Load Application config with Kind
  
  pipecd/pkg/app/piped/trigger/trigger.go
  
  Line 225 in 44cfc93
  
  appCfg, err := config.LoadApplication(gitRepo.GetPath(), app.GitPath.GetApplicationConfigFilePath(), app.Kind)

Modification idea

For 1, No modification for now. But we need to use labels["kind"] in other logic using Deployment.Kind.
For 2, the same modification as event watcher. ref: Make PipeCD control plane support plugin-arched piped #5252 (comment)

ffjlabo · 2024-11-19T09:58:58Z

Kind; server

The purpose of the component

Server providers the gRPC API for web and other client (pipectl for now).
It uses Kind in the situations below.
- When registering the application
  
  pipecd/pkg/app/server/grpcapi/api.go
  
  Line 159 in 44cfc93
  
  Kind: req.Kind,
- When decoding application got from the datastore
  
  pipecd/pkg/datastore/applicationstore.go
  
  Line 98 in 44cfc93
  
  kind = app.Kind

Modification idea

just pass kind to the application same as current one. After pipedv1, we use Application as kind.

ffjlabo · 2024-11-20T02:23:12Z

Kind; Metrics

The purpose of the component
The metrics deployment_status is affected because it has the label application_kind.
https://github.com/pipe-cd/pipecd/blob/548a804d344d7c5faa70bcb6bd0889aa2a9269af/pkg/app/piped/controller/controllermetrics/metrics.go#L23C1-L40C2

deployment_status{application_id="xxxxxxxx",application_kind="KUBERNETES",application_name="simple",deployment="xxxxxxx",instance="127.0.0.1:9085",job="pipecd-ops",launcher_version="v0.49.2",pipecd_component="piped",piped="xxxxx",piped_version="v0.49.2",platform_provider="kubernetes",project="test",status="DEPLOYMENT_CANCELLED"}

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind

ffjlabo · 2024-11-20T02:56:10Z

Kind; planner

The purpose of the component

Planner builds the deployment pipeline based on the app config.
Currently piped use the kind for three purpose
1. Collect the telemetry data with open telemetry and its data has the kind as attribute.
  
  pipecd/pkg/app/piped/controller/planner.go
  
  Line 164 in 44cfc93
  
  attribute.String("kind", p.deployment.Kind.String()),
2. Decide which planner should be used using the kind
  
  pipecd/pkg/app/piped/controller/planner.go
  
  Line 211 in 44cfc93
  
  planner, ok := p.plannerRegistry.Planner(p.deployment.Kind)
3. Show log with the kind
  
  pipecd/pkg/app/piped/controller/planner.go
  
  Line 93 in 44cfc93
  
  zap.String("app-kind", d.Kind.String()),

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind
For 2, We don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin is used.
For 3, same way as for 1.

ffjlabo · 2024-11-20T03:13:29Z

Kind; scheduler

The purpose of the component

Scheduler controls the plan and deployment flow.
Currently piped use the kind in it for four purpose
1. Show log with it
  
  pipecd/pkg/app/piped/controller/scheduler.go
  
  Line 106 in 44cfc93
  
  zap.String("app-kind", d.Kind.String()),
2. Add it to the trace data
  
  pipecd/pkg/app/piped/controller/scheduler.go
  
  Line 278 in 44cfc93
  
  attribute.String("kind", s.deployment.Kind.String()),
3. Decide which rollback executer should be used
  
  pipecd/pkg/app/piped/controller/scheduler.go
  
  Line 439 in 44cfc93
  
  return s.executorRegistry.RollbackExecutor(s.deployment.Kind, in)
  
  Modification idea
4. Find platform provider using HasPlatformProvider
  
  pipecd/pkg/app/piped/controller/scheduler.go
  
  Line 524 in 44cfc93
  
  if !s.pipedConfig.HasPlatformProvider(s.deployment.PlatformProvider, s.deployment.Kind) {

Modification idea

Use model.Application.Labels["kind"] instead of model.Application.Kind
For 2. same as for 1
For 3. We don't need to determine by Kind in plugin architecture. Instead of this, we should determine which plugin is used.
For 4. create HasDeployTargets instead of the HasPlatformprvoders

khanhtc1202 · 2024-11-21T05:24:49Z

PlatformProvider; UI

Purpose

Mostly related to 2 model applications and deployment

Used in

API call AddApplication, UpdateApplication
For showing in Application, Deployment detail page

The source fetched to used in the drawer is come from the piped configuration

https://github.com/pipe-cd/pipecd/blob/master/web/src/components/application-form/index.tsx#L58-L80

Modification idea

Keep it as in the application and deployment model
Also, keep the configuration and parser for pipedv0 config the same as it is (no UI affected around that fetch)
Make a new Add Application form/suggestion form that doesn't show any platform provider-related information

ffjlabo added the epic label Oct 4, 2024

t-kikuc added this to v0.50.0 Oct 4, 2024

ffjlabo moved this to In Progress in v0.50.0 Oct 4, 2024

khanhtc1202 changed the title ~~Deprecate Platform Provider and Kind~~ Make PipeCD control plane support plugin-arched piped Oct 7, 2024

khanhtc1202 mentioned this issue Oct 7, 2024

PipeCD plugin architecture #5259

Open

ffjlabo mentioned this issue Dec 2, 2024

Add deploy target #5378

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make PipeCD control plane support plugin-arched piped #5252

Make PipeCD control plane support plugin-arched piped #5252

ffjlabo commented Oct 4, 2024 •

edited

Loading

ffjlabo commented Oct 22, 2024 •

edited

Loading

ffjlabo commented Oct 23, 2024

ffjlabo commented Oct 23, 2024

ffjlabo commented Oct 29, 2024

ffjlabo commented Oct 31, 2024

ffjlabo commented Oct 31, 2024 •

edited

Loading

ffjlabo commented Oct 31, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 15, 2024

ffjlabo commented Nov 19, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

khanhtc1202 commented Nov 21, 2024 •

edited

Loading

Make PipeCD control plane support plugin-arched piped #5252

Make PipeCD control plane support plugin-arched piped #5252

Comments

ffjlabo commented Oct 4, 2024 • edited Loading

Details (under planning)

For Platform Provider

For the backward compatibility

For Kind

For the backward compatibility

TODO

ffjlabo commented Oct 22, 2024 • edited Loading

Platform Provider; Livestatestore

ffjlabo commented Oct 23, 2024

Platform Provider; Livestatereporter

ffjlabo commented Oct 23, 2024

Platform Provider; Detector

ffjlabo commented Oct 29, 2024

Platform Provider; Plan Preview

ffjlabo commented Oct 31, 2024

Platform Provider; Metrics

ffjlabo commented Oct 31, 2024 • edited Loading

Kind; appconfigreporter

ffjlabo commented Oct 31, 2024 • edited Loading

Kind; event watcher

ffjlabo commented Nov 5, 2024 • edited Loading

Kind; livestatereporter

ffjlabo commented Nov 5, 2024 • edited Loading

Kind; detector

ffjlabo commented Nov 5, 2024 • edited Loading

Kind; notifier

ffjlabo commented Nov 5, 2024 • edited Loading

Kind; Plan preview

ffjlabo commented Nov 15, 2024

Kind; trigger

ffjlabo commented Nov 19, 2024 • edited Loading

Kind; server

ffjlabo commented Nov 20, 2024 • edited Loading

Kind; Metrics

ffjlabo commented Nov 20, 2024 • edited Loading

Kind; planner

ffjlabo commented Nov 20, 2024 • edited Loading

Kind; scheduler

khanhtc1202 commented Nov 21, 2024 • edited Loading

PlatformProvider; UI

Purpose

Modification idea

ffjlabo commented Oct 4, 2024 •

edited

Loading

ffjlabo commented Oct 22, 2024 •

edited

Loading

ffjlabo commented Oct 31, 2024 •

edited

Loading

ffjlabo commented Oct 31, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 5, 2024 •

edited

Loading

ffjlabo commented Nov 19, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

ffjlabo commented Nov 20, 2024 •

edited

Loading

khanhtc1202 commented Nov 21, 2024 •

edited

Loading