Skip to content

Commit

Permalink
fix: cleaner completion traces (RUN-160) (#545)
Browse files Browse the repository at this point in the history
Runtime component: voiceflow/general-runtime#1053

Instead of having 3 different top level completion traces, we should be protective of the "trace.type" namespace - makes for easier documentation. The pattern we want to encourage for users is to build specialized handlers around each type of trace, and there will be a centralized switch/if else where this happens.

The less we have here, the better:
https://github.com/voiceflow/libs/blob/6cb7a7b7667cc18f7e49abc78afe9e34c187086d/packages/base-types/src/trace/index.ts#L111-L128

There is now a `trace.payload.status` property that denotes if it is `start` `continue` or `end`. In the future we can attach more specific metadata to the `start` state.

I believe users DO need an explicit start or stop message, because otherwise it would be on them to check the previous and next trace after the completion events to know to when to perform interface events. Imagine a webchat, there's likely an explicit function to create a chat bubble, then it gets streamed in text, and there is another function to end the bubble.
In the future for Audio, this is extremely obvious on web interfaces:
https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement

Giving the users tokens mid-stream is kind of useless and just redundant data. For example, OpenAI/Anthropic's API has token metrics only on the last chunk:
https://platform.openai.com/docs/api-reference/chat/object#chat/object-usage
https://docs.anthropic.com/en/api/messages-streaming

Finally, we will no longer be giving tokens are part of completion events. Even for regular text/speak steps we don't really give tokens are part of them. Instead we should have a unified type of trace for all token or billing usage in the future, including for TTS, AI Response, AI Set, etc. Today it's a debug trace, which you will still get with `completion_events=true`.

Example client code **before**:
```
        // completion events flag
        case "completion-start":
          process.stdout.write(trace.payload.completion ?? "");
          break;
        case "completion-continue":
          process.stdout.write(trace.payload.completion ?? "");
          break;
        case "completion-end":
          process.stdout.write(trace.payload.completion ?? "" + "\n");
          break;
```

example client code after:
```
        case "completion":
          process.stdout.write(trace.payload.content ?? "");

          if (trace.payload.state === "end") {
            process.stdout.write("\n");
          }
          break;
```

Co-authored-by: Tyler <tylerhan97@gmail.com>
  • Loading branch information
DecathectZero and DecathectZero committed Sep 30, 2024
1 parent 7b1e23b commit 98bdcd4
Show file tree
Hide file tree
Showing 7 changed files with 30 additions and 91 deletions.
4 changes: 1 addition & 3 deletions packages/base-types/src/node/utils/trace.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@ export enum TraceType {
ENTITY_FILLING = 'entity-filling',
CHANNEL_ACTION = 'channel-action',
KNOWLEDGE_BASE = 'knowledgeBase',
COMPLETION_START = 'completion-start',
COMPLETION_CONTINUE = 'completion-continue',
COMPLETION_END = 'completion-end',
COMPLETION = 'completion',
}

export interface BaseTraceFramePath<Event extends BaseEvent = BaseEvent> {
Expand Down
25 changes: 25 additions & 0 deletions packages/base-types/src/trace/completion.trace.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import type { BaseTraceFrame, TraceType } from '@base-types/node/utils';

export interface CompletionTrace
extends BaseTraceFrame<CompletionStartPayload | CompletionContentPayload | CompletionEndPayload> {
type: TraceType.COMPLETION;
}

export enum CompletionState {
START = 'start',
CONTENT = 'content',
END = 'end',
}

export interface CompletionStartPayload {
state: CompletionState.START;
}

export interface CompletionContentPayload {
state: CompletionState.CONTENT;
content: string;
}

export interface CompletionEndPayload {
state: CompletionState.END;
}

This file was deleted.

This file was deleted.

46 changes: 0 additions & 46 deletions packages/base-types/src/trace/completion/completion-start-trace.ts

This file was deleted.

3 changes: 0 additions & 3 deletions packages/base-types/src/trace/completion/index.ts

This file was deleted.

16 changes: 4 additions & 12 deletions packages/base-types/src/trace/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,9 @@ import type { IntentRequest } from '@base-types/request';
import type { Log as RuntimeLog } from '@base-types/runtimeLogs';
import type { AnyRecord } from '@voiceflow/common';

import type { CompletionContinueTrace, CompletionEndTrace, CompletionStartTrace } from './completion';

export {
CompletionContinueTrace,
CompletionEndTrace,
CompletionStartTrace,
CompletionStartTraceSpeak,
CompletionStartTraceText,
} from './completion';
import type { CompletionTrace } from './completion.trace';

export { CompletionTrace } from './completion.trace';
export { TraceFrame as CardV2 } from '@base-types/node/cardV2';
export { TraceFrame as Carousel } from '@base-types/node/carousel';
export { TraceFrame as End } from '@base-types/node/exit';
Expand Down Expand Up @@ -131,6 +125,4 @@ export type AnyTrace =
| CardV2Trace
| EntityFillingTrace
| ChannelActionTrace
| CompletionStartTrace
| CompletionContinueTrace
| CompletionEndTrace;
| CompletionTrace;

0 comments on commit 98bdcd4

Please sign in to comment.