Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Add Deployment controller for managed connectors #1499

Closed
wants to merge 47 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a56aa5c
Initial stuff for managed connector specs
simlay Aug 24, 2021
05bc7ef
managed_connector spec
nacardin Aug 24, 2021
594d450
Fixed helm chart for managedconnectors
simlay Aug 25, 2021
75968a5
cargo fmt
simlay Aug 25, 2021
ad7d410
Added list and delete endpoints
simlay Aug 26, 2021
22e907f
Added ManagedConnectors to watch api
simlay Aug 26, 2021
f980f9f
more updates
simlay Aug 26, 2021
04f7748
Added deployment generation
simlay Aug 27, 2021
092e8c3
working deployment
nacardin Aug 27, 2021
225b76f
Added args to connector config
simlay Aug 30, 2021
691c901
Updated to use k8-api patch branch
simlay Aug 31, 2021
27a615f
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Aug 31, 2021
e4b4851
Image name is now correct
simlay Sep 2, 2021
a17edf5
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 2, 2021
06c8585
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 2, 2021
c80f1b1
fix warnings
simlay Sep 3, 2021
12b1799
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 3, 2021
ed364b3
updates to helm charts
simlay Sep 7, 2021
ad54a5e
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 7, 2021
a4fe234
Working again
simlay Sep 7, 2021
a100488
Added status to container
simlay Sep 8, 2021
5f4525c
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 10, 2021
4b9d84b
initial scaffolding to config map
simlay Sep 15, 2021
5cb88bf
Working for demo
simlay Sep 15, 2021
de2d6ba
Added fluvio-connector-helm chart
simlay Sep 16, 2021
d5d8ebc
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 22, 2021
6ee61de
deleted unused files
simlay Sep 22, 2021
cf1f326
updates
simlay Sep 22, 2021
f05f170
Updated helm chart
simlay Sep 22, 2021
dd4da1b
fix clippy
simlay Sep 22, 2021
ccc4e72
cargo fmt
simlay Sep 22, 2021
511a445
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 23, 2021
c57ac85
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 24, 2021
6596407
Added default for create_topic
simlay Sep 24, 2021
9bf6646
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 27, 2021
bdcb33c
Tested and working
simlay Sep 28, 2021
fd95342
cargo fmt
simlay Sep 28, 2021
8c9b190
Clean up unused comments and fmt
simlay Sep 28, 2021
20e903b
Updates from comments
simlay Sep 28, 2021
3399b25
added tls work around for configmap
simlay Sep 30, 2021
3a9bcf5
cargo fmt
simlay Sep 30, 2021
847557e
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 30, 2021
bacaa73
Added filter for managed connector
simlay Sep 30, 2021
cdf589d
Update from comments
simlay Sep 30, 2021
05dc77a
cargo fmt
simlay Sep 30, 2021
66b4971
Merge branch 'master' of github.com:infinyon/fluvio into add-spec-for…
simlay Sep 30, 2021
343e042
fix clippy
simlay Sep 30, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Initial stuff for managed connector specs
  • Loading branch information
simlay committed Aug 24, 2021
commit a56aa5c7e1b6bd2aa922e0595fba9e3f8ed7a751
1 change: 1 addition & 0 deletions src/controlplane-metadata/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ pub mod topic;
pub mod partition;
pub mod spg;
pub mod message;
pub mod managed_connector;

pub use fluvio_stream_model::core;

Expand Down
15 changes: 15 additions & 0 deletions src/controlplane-metadata/src/managed_connector/k8/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
//!
//! # Cluster
//!
//! Interface to the Cluster metadata in K8 key value store
//!
mod spec;

pub use self::spec::*;

use super::ManagedConnectorStatus;
use super::ManagedConnectorConfig;
use crate::k8_types::Status as K8Status;

/// implement k8 status for spu group status because they are same
impl K8Status for ManagedConnectorStatus {}
84 changes: 84 additions & 0 deletions src/controlplane-metadata/src/managed_connector/k8/spec.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
//!
//! # SPU Spec
//!
//! Interface to the SPU metadata spec in K8 key value store
//!
use serde::Deserialize;
use serde::Serialize;

use fluvio_types::defaults::{SPU_PUBLIC_PORT, SPU_LOG_BASE_DIR, SPU_LOG_SIZE};
use fluvio_types::defaults::SPU_PRIVATE_PORT;

use crate::spu::EncryptionEnum;
use super::super::ManagedConnectorStatus;
use crate::k8_types::{Spec, Crd, DefaultHeader, TemplateSpec, Env};

use crd::MANAGED_CONNECTOR_API;
mod crd {

use crate::k8_types::{Crd, CrdNames, GROUP, V1};

pub const MANAGED_CONNECTOR_API: Crd = Crd {
group: GROUP,
version: V1,
names: CrdNames {
kind: "ManagedConnector",
plural: "managedconnectors",
singular: "managedconnector",
},
};
}

impl Spec for K8ManagedConnectorSpec {
type Status = ManagedConnectorStatus;
type Header = DefaultHeader;
fn metadata() -> &'static Crd {
&MANAGED_CONNECTOR_API
}
}
use super::ManagedConnectorConfig;

#[derive(Deserialize, Serialize, Default, Debug, Clone)]
#[serde(rename_all = "camelCase", default)]
pub struct K8ManagedConnectorSpec {
pub name: String,
pub config: ManagedConnectorConfig,
}
mod convert {

use crate::managed_connector::*;

use super::*;

impl From<K8ManagedConnectorSpec> for ManagedConnectorSpec {
fn from(spec: K8ManagedConnectorSpec) -> Self {
Self {
name: spec.name,
config: spec.config,
}
}
}
/*
impl From<SpuTemplate> for SpuConfig {
fn from(template: SpuTemplate) -> Self {
Self {
rack: template.rack,
replication: template.replication.map(|r| r.into()),
storage: template.storage.map(|s| s.into()),
env: vec![], // doesn't really matter
}
}
}
*/

impl From<ManagedConnectorSpec> for K8ManagedConnectorSpec {
fn from(spec: ManagedConnectorSpec) -> Self {
Self {
name: spec.name,
config: spec.config,
}
}
}


}
65 changes: 65 additions & 0 deletions src/controlplane-metadata/src/managed_connector/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
mod spec;
mod status;
pub mod store;

pub use spec::*;
pub use status::*;

#[cfg(feature = "k8")]
mod k8;
#[cfg(feature = "k8")]
pub use k8::*;

mod convert {

use crate::core::{Spec, Status, Removable, Creatable};
use crate::extended::{ObjectType, SpecExt};
use super::*;

impl Spec for ManagedConnectorSpec {
const LABEL: &'static str = "SpuGroup";

type Status = ManagedConnectorStatus;

type Owner = Self;
type IndexKey = String;
}

impl SpecExt for ManagedConnectorSpec {
const OBJECT_TYPE: ObjectType = ObjectType::SpuGroup;
}

impl Removable for ManagedConnectorSpec {
type DeleteKey = String;
}

impl Creatable for ManagedConnectorSpec {}

impl Status for ManagedConnectorStatus {}

#[cfg(feature = "k8")]
mod extended {

use crate::store::k8::K8ExtendedSpec;
use crate::store::k8::K8ConvertError;
use crate::store::k8::K8MetaItem;
use crate::store::MetadataStoreObject;
use crate::k8_types::K8Obj;
use crate::store::k8::default_convert_from_k8;

use super::ManagedConnectorSpec;
use super::K8ManagedConnectorSpec;

impl K8ExtendedSpec for ManagedConnectorSpec {
type K8Spec = K8ManagedConnectorSpec;
type K8Status = Self::Status;

fn convert_from_k8(
k8_obj: K8Obj<Self::K8Spec>,
) -> Result<MetadataStoreObject<Self, K8MetaItem>, K8ConvertError<Self::K8Spec>>
{
default_convert_from_k8(k8_obj)
}
}
}
}
29 changes: 29 additions & 0 deletions src/controlplane-metadata/src/managed_connector/spec.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#![allow(clippy::assign_op_pattern)]

use dataplane::core::{Encoder, Decoder};
use fluvio_types::defaults::{SPU_LOG_BASE_DIR, SPU_LOG_SIZE};

#[derive(Encoder, Decoder, Default, Debug, PartialEq, Clone)]
#[cfg_attr(
feature = "use_serde",
derive(serde::Serialize, serde::Deserialize),
serde(rename_all = "camelCase")
)]
pub struct ManagedConnectorSpec {
pub name: String,
/// Configuration elements to be applied to each SPUs in the group
pub config: ManagedConnectorConfig,
}

#[derive(Encoder, Decoder, Default, Debug, Clone, PartialEq)]
#[cfg_attr(
feature = "use_serde",
derive(serde::Serialize, serde::Deserialize),
serde(rename_all = "camelCase")
)]
pub struct ManagedConnectorConfig {
pub r#type: String, // syslog, github star, slack
}

impl ManagedConnectorConfig {
}
72 changes: 72 additions & 0 deletions src/controlplane-metadata/src/managed_connector/status.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#![allow(clippy::assign_op_pattern)]

use std::fmt;

use dataplane::core::{Encoder, Decoder};

#[derive(Encoder, Decoder, Default, Debug, Clone, PartialEq)]
#[cfg_attr(
feature = "use_serde",
derive(serde::Serialize, serde::Deserialize),
serde(rename_all = "camelCase")
)]
pub struct ManagedConnectorStatus {
/// Status resolution
pub resolution: ManagedConnectorStatusResolution,

/// Reason for Status resolution (if applies)
pub reason: Option<String>,
}

impl fmt::Display for ManagedConnectorStatus {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "{:#?}", self.resolution)
}
}

impl ManagedConnectorStatus {
pub fn invalid(reason: String) -> Self {
Self {
resolution: ManagedConnectorStatusResolution::Invalid,
reason: Some(reason),
}
}

pub fn reserved() -> Self {
Self {
resolution: ManagedConnectorStatusResolution::Reserved,
..Default::default()
}
}

pub fn is_already_valid(&self) -> bool {
self.resolution == ManagedConnectorStatusResolution::Reserved
}
}

#[cfg_attr(feature = "use_serde", derive(serde::Serialize, serde::Deserialize))]
#[derive(Encoder, Decoder, Debug, Clone, PartialEq)]
pub enum ManagedConnectorStatusResolution {
Init,
Invalid,
Reserved,
}

// -----------------------------------
// Implementation - FlvSpuGroupResolution
// -----------------------------------
impl Default for ManagedConnectorStatusResolution {
fn default() -> Self {
Self::Init
}
}

impl fmt::Display for ManagedConnectorStatusResolution {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
match self {
Self::Init => write!(f, "Init"),
Self::Invalid => write!(f, "Invalid"),
Self::Reserved => write!(f, "Reserved"),
}
}
}
11 changes: 11 additions & 0 deletions src/controlplane-metadata/src/managed_connector/store.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
//!
//! Spu Group
//!

use crate::store::*;

use super::*;

pub type SpuGroupMetadata<C> = MetadataStoreObject<ManagedConnectorSpec, C>;

pub type SpuGroupLocalStore<C> = LocalStore<ManagedConnectorSpec, C>;