Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add auditor to coreServices #26372

Merged
merged 10 commits into from
Jan 21, 2025
Merged

Conversation

schultzp2020
Copy link
Contributor

Hey, I just made a Pull Request!

references: #23950

✔️ Checklist

  • A changeset describing the change and affected packages. (more info)
  • Added or updated documentation
  • Tests for new functionality and regression tests for bug fixes
  • Screenshots attached (for UI changes)
  • All your commits have a Signed-off-by line in the message. (more info)

@schultzp2020 schultzp2020 requested review from backstage-service and a team as code owners August 30, 2024 00:32
@schultzp2020 schultzp2020 marked this pull request as draft August 30, 2024 00:32
@backstage-goalie
Copy link
Contributor

backstage-goalie bot commented Aug 30, 2024

Important

This PR includes changes that affect public-facing API. Please ensure you are adding/updating documentation for new features or behavior.

Changed Packages

Package Name Package Path Changeset Bump Current Version
@backstage/backend-defaults packages/backend-defaults minor v0.7.0
@backstage/backend-plugin-api packages/backend-plugin-api minor v1.1.1
@backstage/backend-test-utils packages/backend-test-utils minor v1.2.1
@backstage/plugin-catalog-backend plugins/catalog-backend minor v1.30.0
@backstage/plugin-scaffolder-backend plugins/scaffolder-backend minor v1.29.0
@backstage/plugin-scaffolder-node plugins/scaffolder-node minor v0.6.3

@schultzp2020
Copy link
Contributor Author

@Rugvip what are your thought on this implementation of the event auditor?

@schultzp2020 schultzp2020 changed the title Event auditor feat: add eventAuditor and rootEventAuditor to coreServices Aug 30, 2024
@schultzp2020
Copy link
Contributor Author

schultzp2020 commented Sep 3, 2024

@Rugvip #26372 (comment)

Copy link
Member

@freben freben left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK some initial thoughts, we'll raise it with the team next week too

packages/backend-test-utils/api-report.md Outdated Show resolved Hide resolved
packages/backend-test-utils/api-report.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/api-report.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/api-report.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/api-report.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/api-report.md Outdated Show resolved Hide resolved
@schultzp2020 schultzp2020 changed the title feat: add eventAuditor and rootEventAuditor to coreServices feat: add auditor and rootAuditor to coreServices Sep 6, 2024
@schultzp2020
Copy link
Contributor Author

schultzp2020 commented Sep 9, 2024

@freben, do you have any further input on the unresolved comments? Thank you for reviewing!

@schultzp2020
Copy link
Contributor Author

@freben PTAL?

@schultzp2020 schultzp2020 changed the title feat: add auditor and rootAuditor to coreServices feat: add auditor to coreServices Sep 10, 2024
@schultzp2020 schultzp2020 marked this pull request as ready for review September 19, 2024 21:00
Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! 👍

Wanted to get some early feedback going to not have this sit completely silent, but haven't had a chance for a proper review yet. Just to set expectations I do think we have some work to do to iron out the API, it's still much too early to add a definition for this in the core APIs.

packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
@schultzp2020
Copy link
Contributor Author

@freben do you know why the PR is failing?

@freben
Copy link
Member

freben commented Sep 26, 2024

Ah yeah, sorry the nice message that explains what to do got lost, fix for that here. You were meant to see

*************************************************************************************
* You have uncommitted changes to the public API or reports of a package.           *
* To solve this, run `yarn build:api-reports` and commit all md file changes.       *
*************************************************************************************

Essentially you always need to run yarn build:api-reports when you change public contracts, and commit those generated changes along with the PR as well.

@schultzp2020
Copy link
Contributor Author

schultzp2020 commented Sep 26, 2024

@freben I've done that a few times and it still doesn't fix the issue. 😢 Let me try again.

packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
@schultzp2020
Copy link
Contributor Author

@Rugvip PTAL? In the meantime I will add the auditor to the scaffolder and catalog plugins.

@schultzp2020 schultzp2020 requested a review from a team as a code owner October 1, 2024 20:53
@github-actions github-actions bot added the area:catalog Related to the Catalog Project Area label Oct 1, 2024
@schultzp2020
Copy link
Contributor Author

@Rugvip How should I address the type error in the legacy backend package?

Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another round! 🍻

plugins/catalog-backend/report.api.md Outdated Show resolved Hide resolved
plugins/catalog-backend/src/service/createRouter.ts Outdated Show resolved Hide resolved
plugins/catalog-backend/README.md Show resolved Hide resolved
plugins/catalog-backend/README.md Outdated Show resolved Hide resolved
plugins/catalog-backend/README.md Outdated Show resolved Hide resolved
Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! continuing forward with the main bits. I like the new .createEvent API 👍

@schultzp2020 schultzp2020 requested a review from a team as a code owner October 8, 2024 20:49
@schultzp2020
Copy link
Contributor Author

@schultzp2020 can you rebase this please 🙏

@benjdlambert done!

Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright couple more bits to get this ready for merge after release on Tuesday. Main thing is still the default implementation and having it use the logger service by default.

.changeset/olive-boxes-hide.md Outdated Show resolved Hide resolved
packages/backend-plugin-api/report.api.md Outdated Show resolved Hide resolved
import type { Request } from 'express';
import type { mockServices } from './mockServices';

export class MockAuditorService implements AuditorService {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the default auditor service simply forwarding to the logger service I don't think we need a concrete mock implementation and can use the default implementation instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the auditor mock is still valuable to retain, as its implementation details don't exactly mirror those of the rootLogger. Let me know if I’m misunderstanding your point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks pretty much identical to me, except that it doesn't have support for the Winston transport ofc but it's not worth duplicating the code to remove that. I think it should be fine to leave this one without explicit mocks, similar to this and the ones below:

export namespace rootHealth {
export const factory = () => rootHealthServiceFactory;
export const mock = simpleMock(coreServices.rootHealth, () => ({
getLiveness: jest.fn(),
getReadiness: jest.fn(),
}));
}

packages/backend-test-utils/src/next/wiring/TestBackend.ts Outdated Show resolved Hide resolved
plugins/catalog-backend/src/service/CatalogBuilder.ts Outdated Show resolved Hide resolved
plugins/scaffolder-node/src/tasks/types.ts Show resolved Hide resolved
@schultzp2020 schultzp2020 force-pushed the event-auditor branch 4 times, most recently from 12b7f4e to 09f48ef Compare January 13, 2025 21:36
@schultzp2020
Copy link
Contributor Author

Alright couple more bits to get this ready for merge after release on Tuesday. Main thing is still the default implementation and having it use the logger service by default.

@Rugvip updated!

@schultzp2020 schultzp2020 force-pushed the event-auditor branch 2 times, most recently from 3453527 to e8a5702 Compare January 15, 2025 18:31
Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright just a couple of smaller nits but basically good to go. Skip the AuditorServiceCreateEventOptions comment, that one's intended for the future. If you do want to tackle it please do it in a separate PR.

.changeset/olive-boxes-hide-backend-defaults.md Outdated Show resolved Hide resolved
export type AuditorServiceCreateEventOptions = {
eventId: string;
severityLevel?: AuditorServiceEventSeverityLevel;
request?: Request_2<any, any, any, any, any>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize we likey want a way to supply credentials separately from a request, or essentially provide the equivalent fields of what we'd gather from the request manually. Thinking especially for situations when we use other protocols than plain HTTP. Happy to leave that as a followup though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When this PR was initially opened, the implementation included a way to provide credentials directly. However, that approach was updated to fallback onto service that triggered it. For reference, see. Although, let's discuss this more in a follow up.

Comment on lines +326 to +307
const impl = new DefaultRootAuditorService(
this.impl.child({}) as WinstonLogger,
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this indirection? Could just implement log internally in DefaultAuditorService instead right?

I'm thinking it's worth having DefaultAuditorService just take a log callback or something like that, receiving a message and meta, as in same signature as .info(...) but slimmed down to exactly what we need. That way it's actually kind of simple to reimplement this service with any kind of log function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're raising a valid point about simplifying the DefaultAuditorService by directly implementing the log method within it and accepting a callback. While seemingly straightforward, this approach would sacrifice crucial functionalities related to context and structured logging.

Currently, the DefaultRootAuditorService isn't just a wrapper around a Winston logger. It plays a vital role in managing the overall logging context, including consistent formatting, setting the service name, and propagating that context to individual plugin instances. This ensures uniformity across all audit logs, simplifying filtering and analysis.

By creating DefaultAuditorService instances via forPlugin, each plugin inherits the root context and can append its own specific details. This hierarchy makes it easy to track events originating from different plugins while maintaining a standardized format. A simple callback wouldn't allow for such structured context propagation. Consider a scenario where we want to filter logs by plugin or include common fields like service name in every log entry. The current design facilitates these use cases without requiring each plugin to re-implement the same logic.

Moreover, allowing plugins to completely customize logging via a callback could lead to inconsistencies and make analysis across plugins difficult. The present structure strikes a balance between flexibility for plugins (through metadata and events) and enforced consistency for the overall system. Therefore, while a callback approach appears simpler, it doesn't address the broader needs of a structured, consistent, and context-aware audit logging system.

Comment on lines +201 to +190
let credentials: BackstageCredentials =
await this.auth.getOwnServiceCredentials();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, doesn't feel quite right to default to service credentials here. I think it would be better to leave as undefined tbh. We kinda hide information by doing this, as in we don't know whether these credentials where explicitly passed or if we hit this fallback.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the concern, but tracking the actor—even if it’s the fallback—is important because auditors typically require an actor in audit logs. Having more detailed information allows us to trace where an event originated. For example, in the Scaffolder plugin, there are multiple scenarios where credentials (request objects) aren’t explicitly provided, and using the fallback helps us identify the service responsible. Leaving it undefined might result in gaps in the audit trail, which could create challenges when backtracking events.

fields = meta;
}

this.impl.info(eventId, fields);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking the default message should probably contain a lil bit more information than just the even ID right? Even if it's just something like Audit event ${eventId} ${status}

Copy link
Contributor Author

@schultzp2020 schultzp2020 Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d prefer to align with Google's format, where the event ID acts as a reference. Additionally, there’s an isAuditorEvent field to clearly distinguish between standard logs and auditor-specific logs.

Relevant implementation: https://github.com/schultzp2020/backstage/blob/event-auditor/packages/backend-defaults/src/entrypoints/auditor/Auditor.ts#L282-L284

import type { Request } from 'express';
import type { mockServices } from './mockServices';

export class MockAuditorService implements AuditorService {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks pretty much identical to me, except that it doesn't have support for the Winston transport ofc but it's not worth duplicating the code to remove that. I think it should be fine to leave this one without explicit mocks, similar to this and the ones below:

export namespace rootHealth {
export const factory = () => rootHealthServiceFactory;
export const mock = simpleMock(coreServices.rootHealth, () => ({
getLiveness: jest.fn(),
getReadiness: jest.fn(),
}));
}

schultzp2020 and others added 10 commits January 20, 2025 15:00
Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

create eventAuditor and rootEventAuditor mockServices

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rerun api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

apply requested changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

remove rootAuditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run build:api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

misc fixes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

misc fixes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add dynamic plugin auditor service factory

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rerun api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add comments to auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update api report

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

apply requested changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix tests

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor to the catalog plugin

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix tsc issues

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor service

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rename 'args' to 'options'

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor event shape

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add error option in auditor event shape

Signed-off-by: Paul Schultz <pschultz@pobox.com>

rename events

Signed-off-by: Paul Schultz <pschultz@pobox.com>

demo new api?

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update service impl

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add the auditor service to the scaffolder

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update tests

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

revert config secret enumerator changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>

clean up auditor api

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add documentation

Signed-off-by: Paul Schultz <pschultz@pobox.com>

update auditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add auditor to sign in/out

Signed-off-by: Paul Schultz <pschultz@pobox.com>

remove dynamic auditor

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add winston dep back

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run api-reports

Signed-off-by: Paul Schultz <pschultz@pobox.com>

run prettier

Signed-off-by: Paul Schultz <pschultz@pobox.com>

add the auditor service as a default service factory

Signed-off-by: Paul Schultz <pschultz@pobox.com>

fix eslint errror

Signed-off-by: Paul Schultz <pschultz@pobox.com>

revert auth changes

Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Co-authored-by: Kashish Mittal <113269381+04kash@users.noreply.github.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
Signed-off-by: Paul Schultz <pschultz@pobox.com>
@schultzp2020
Copy link
Contributor Author

@Rugvip done!

Copy link
Member

@Rugvip Rugvip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, thank you! 🎉 :shipit:

Gonna ship this for today's release and handle any smaller bits in followups.

Awesome work and thank you for enduring throughout the iteration 😅

@Rugvip Rugvip merged commit aba0aaf into backstage:master Jan 21, 2025
25 checks passed
Copy link
Contributor

Thank you for contributing to Backstage! The changes in this pull request will be part of the 1.36.0 release, scheduled for Tue, 18 Feb 2025.

@schultzp2020 schultzp2020 deleted the event-auditor branch January 21, 2025 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:catalog Related to the Catalog Project Area area:documentation Improvements or additions to documentation area:scaffolder Everything and all things related to the scaffolder project area
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants