Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keeper backend doesn't work correctly with Custom Record Types #4063

Open
evilhamsterman opened this issue Oct 30, 2024 · 5 comments
Open

Keeper backend doesn't work correctly with Custom Record Types #4063

evilhamsterman opened this issue Oct 30, 2024 · 5 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@evilhamsterman
Copy link

evilhamsterman commented Oct 30, 2024

Describe the bug
When trying to access data from a custom record the fields are not processed correctly. The fields are identified by their type rather than their label. This leads to two problems

  1. If you have duplicate field types, like say multiple text fields you get the error Following keys are duplicated <type>
  2. If you don't have duplicates your fields are all called generic types like text or secret

To Reproduce
Steps to reproduce the behavior:

  1. Create a couple custom record types one with duplicate field types one without
  2. Try to access them with extract and no templating so you see all the keys in the resulting secret
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
 namespace: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: uid

The one with the duplicates will give the error above, if you don't have duplicate field types then you'll get a secret with field names like text or secret instead of useful identifiers.

Expected behavior
Custom records with duplicate field types don't cause a complete failure. Records regardless of the field types are accessible via the label, since labels are free form it will probably require some form of manipulation like replace spaces with _

@evilhamsterman evilhamsterman added the kind/bug Categorizes issue or PR as related to a bug. label Oct 30, 2024
@gusfcarvalho
Copy link
Member

Hi @evilhamsterman ! Keeper is a community maintained provider - and as such @ppodevlabs is your best shot at getting this implemented 😄

@ppodevlabs
Copy link
Contributor

Describe the bug When trying to access data from a custom record the fields are not processed correctly. The fields are identified by their type rather than their label. This leads to two problems

  1. If you have duplicate field types, like say multiple text fields you get the error Following keys are duplicated <type>
  2. If you don't have duplicates your fields are all called generic types like text or secret

To Reproduce Steps to reproduce the behavior:

  1. Create a couple custom record types one with duplicate field types one without
  2. Try to access them with extract and no templating so you see all the keys in the resulting secret
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
 namespace: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: uid

The one with the duplicates will give the error above, if you don't have duplicate field types then you'll get a secret with field names like text or secret instead of useful identifiers.

Expected behavior Custom records with duplicate field types don't cause a complete failure. Records regardless of the field types are accessible via the label, since labels are free form it will probably require some form of manipulation like replace spaces with _

Hi, duplicated types should be allowed, what can not be duplicated is the label on the field. Could you please confirm you do not have duplicated labels?? As those are used as keys as stated in

remoteRef.property is equated to one of the following options:
- Fields: [Record's field's Type](https://docs.keeper.io/secrets-manager/secrets-manager/about/field-record-types)
- CustomFields: Record's field's Label
- Files: Record's file's Name
If empty, defaults to the complete Record in JSON format

@evilhamsterman
Copy link
Author

evilhamsterman commented Nov 22, 2024

Yes I do not have duplicate labels

I created a Custom Record type called "Duplicate type exampled" and a record of that type. Here's the sanitized json of the record object as returned by the Keeper API

{
  "record_uid": "recordid",
  "title": "Test record",
  "type": "Duplicate type example",
  "fields": [
    {
      "type": "text",
      "label": "Field 1",
      "value": [
        "foo"
      ]
    },
    {
      "type": "text",
      "label": "Field 2",
      "value": [
        "bar"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T14:57:27",
  "user_permissions": [
    {
      "username": "myuser",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 12
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 12
}

When I create an ExternalSecret for that record like this

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: test-secret
spec:
 refreshInterval: 1m
 secretStoreRef:
   name: ksm-secretstore
   kind: ClusterSecretStore
 target:
   name: test-secret
 dataFrom:
   - extract:
       key: recordid

it fails to sync the secret with the event invalid Secret. Following keys are duplicated text When it says text it means the duplicate text type field, the labels are different but the type is the same. This is shown by if I create another custom type with the fields

"fields": [
    {
      "type": "date",
      "label": "Date Active",
      "value": [
        1732316915948
      ]
    },
    {
      "type": "date",
      "label": "Date of Birth",
      "value": [
        1731971318326
      ]
    }
  ]

I get a similar error invalid Secret. Following keys are duplicated date

You properly support Custom FIELDS in a record but this is regarding Custom RECORD Types. As far as I can tell none of the builtin in record types have fields with duplicate types, but with custom records it is entirely possible to have more than one of the same type of field with the labels differentiating them.

Even your own doc states this

remoteRef.property is equated to one of the following options:

  • Fields: [Record's field's Type]

@evilhamsterman
Copy link
Author

Further more if it doesn't have duplicate types the name of the fields is the field type which may not be very useful in a custom record.

If I create a custom record with a text field labelled login and a multiline field labelled password to get around the duplicate field issue. I recognize this is a bit of a contrived but it's for demonstration. So now I have a record json

{
  "record_uid": "recordid",
  "title": "Wrong names",
  "type": "Wrong field names",
  "fields": [
    {
      "type": "text",
      "label": "Login",
      "value": [
        "hunter2"
      ]
    },
    {
      "type": "multiline",
      "label": "Password",
      "value": [
        "thatsthepasswordonmyluggage"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T15:19:31",
  "user_permissions": [
    {
      "username": "user",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 13
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 13
}

It will successfully sync but the field names are not very useful they are just the names of the field types which my users may not be aware of because they don't have access to the record directly.

apiVersion: v1
data:
  multiline: dGhhdHN0aGVwYXNzd29yZG9ubXlsdWdnYWdl
  text: aHVudGVyMg==
immutable: false
kind: Secret
metadata:
  annotations: {}
  creationTimestamp: "2024-11-22T23:25:29Z"
  labels:
    reconcile.external-secrets.io/created-by: ee47526cfb7e5bedd021672be432e680
  name: test-secret
  namespace: it-github-runners
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: test-secret
    uid: 76fcec59-7d7b-4643-968d-83663135fdfd
  resourceVersion: "4"
  uid: eeafdad1-da15-4d75-b74a-8b657816c621
type: Opaque

It would be better if rather than the field types for Custom Records it uses a sanitized version of the field label. Say lowercased and replace spaces with _ or something

@ppodevlabs
Copy link
Contributor

Further more if it doesn't have duplicate types the name of the fields is the field type which may not be very useful in a custom record.

If I create a custom record with a text field labelled login and a multiline field labelled password to get around the duplicate field issue. I recognize this is a bit of a contrived but it's for demonstration. So now I have a record json

{
  "record_uid": "recordid",
  "title": "Wrong names",
  "type": "Wrong field names",
  "fields": [
    {
      "type": "text",
      "label": "Login",
      "value": [
        "hunter2"
      ]
    },
    {
      "type": "multiline",
      "label": "Password",
      "value": [
        "thatsthepasswordonmyluggage"
      ]
    }
  ],
  "custom": [],
  "version": 3,
  "shared": true,
  "client_modified_time": "2024-11-22T15:19:31",
  "user_permissions": [
    {
      "username": "user",
      "owner": true,
      "share_admin": true,
      "shareable": true,
      "editable": true
    }
  ],
  "shared_folder_permissions": [
    {
      "shared_folder_uid": "uuid",
      "reshareable": true,
      "editable": true,
      "revision": 13
    }
  ],
  "share_admins": [
    ""
  ],
  "revision": 13
}

It will successfully sync but the field names are not very useful they are just the names of the field types which my users may not be aware of because they don't have access to the record directly.

apiVersion: v1
data:
  multiline: dGhhdHN0aGVwYXNzd29yZG9ubXlsdWdnYWdl
  text: aHVudGVyMg==
immutable: false
kind: Secret
metadata:
  annotations: {}
  creationTimestamp: "2024-11-22T23:25:29Z"
  labels:
    reconcile.external-secrets.io/created-by: ee47526cfb7e5bedd021672be432e680
  name: test-secret
  namespace: it-github-runners
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: test-secret
    uid: 76fcec59-7d7b-4643-968d-83663135fdfd
  resourceVersion: "4"
  uid: eeafdad1-da15-4d75-b74a-8b657816c621
type: Opaque

It would be better if rather than the field types for Custom Records it uses a sanitized version of the field label. Say lowercased and replace spaces with _ or something

This was built to work with non custom records that is why fields used the types instead of labels. I need to do some testing in order to verify this wont break the existing implementation. I will try to work on this next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants