Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: background tasks #3740

Merged
merged 69 commits into from
Jan 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4790100
wip(tasks): start with tasks
brunozoric Nov 29, 2023
526625c
wip(tasks): split plugin and definition
brunozoric Nov 30, 2023
c893c45
wip(tasks): task runs
brunozoric Dec 4, 2023
baa2829
feat(api-headless-cms): add json type field
brunozoric Dec 4, 2023
cdaec0a
feat(tasks): graphql and crud
brunozoric Dec 5, 2023
13f2297
refactor(tasks): database response manager
brunozoric Dec 5, 2023
b1a7083
chore: update ts configs
brunozoric Dec 5, 2023
f37579f
refactor(tasks): rename tests
brunozoric Dec 6, 2023
7c9c1ea
feat(tasks): add step fn token to input and output
brunozoric Dec 6, 2023
900cb92
refactor(tasks): add state machine id to event
brunozoric Dec 8, 2023
43bbadb
feat(tasks): task crud
brunozoric Dec 8, 2023
f1cbee2
refactor(pulumi-aws): background tasks
brunozoric Dec 8, 2023
0712207
refactor(pulumi-aws): background tasks logging policy
brunozoric Dec 11, 2023
4bbb104
refactor(tasks): simplify response types
brunozoric Dec 12, 2023
5cb1a27
refactor(tasks): move classes and abstractions
brunozoric Dec 12, 2023
97dba6c
refactor(pulumi-aws): update step function definition
brunozoric Dec 12, 2023
c7294a5
feat(pulumi-aws): add waiting possibility in background task
brunozoric Dec 13, 2023
1373616
test(tasks): add crud tests
brunozoric Dec 13, 2023
e7825d2
test(tasks): definitions crud
brunozoric Dec 13, 2023
8f8bdac
fix(tasks): expose graphql
brunozoric Dec 13, 2023
37b0232
test(tasks): task manager and event validation
brunozoric Dec 14, 2023
e4fed20
feat(pulumi-aws): deploy event bridge to step fn connection
brunozoric Dec 14, 2023
68fe0e7
feat(tasks): implement graphql
brunozoric Dec 14, 2023
3d38d48
feat(tasks): ability to stop a task
brunozoric Dec 15, 2023
1f35cd1
test(tasks): graphql to list tasks from database
brunozoric Dec 15, 2023
7a110d4
test(tasks): trigger crud
brunozoric Dec 15, 2023
1bf99e2
fix(tests): validate task definition id
brunozoric Dec 15, 2023
0be1a4e
test(tasks): modify tasks to camel cased id
brunozoric Dec 15, 2023
45e7a17
feat(tasks): add task values store method
brunozoric Dec 18, 2023
3cda7ca
feat(api-elasticsearch-tasks): indexing
brunozoric Dec 18, 2023
d3bb26b
feat(api-elasticsearch-task): code split task runner
brunozoric Dec 19, 2023
aa2520a
feat(api-background-tasks): wrapper for tasks and definitions
brunozoric Dec 19, 2023
034ad7f
fix(api-background-tasks): readme
brunozoric Dec 19, 2023
ee4b630
fix(tasks): store task management
brunozoric Dec 19, 2023
b061762
test(api-elasticsearch-tasks): wip reindexing
brunozoric Dec 19, 2023
6dbd877
feat(api-background-tasks-dynamodb): ddb tasks
brunozoric Dec 20, 2023
a13cc0d
feat(api-background-tasks-es): elasticsearch tasks
brunozoric Dec 20, 2023
2f3830f
feat(api-background-tasks-os): opensearch tasks
brunozoric Dec 20, 2023
c59aa4a
refactor: rename dynamodb tasks package
brunozoric Dec 20, 2023
112c55d
fix(cwp-template-aws): dynamodb setup
brunozoric Dec 20, 2023
0bf0c6a
fix(cwp-template-aws): os setup
brunozoric Dec 20, 2023
9a33a8d
fix(tasks): event input
brunozoric Dec 20, 2023
c920e8e
refactor(tasks): graphql type names
brunozoric Dec 21, 2023
15b129d
feat(pulumi-aws): step to transform event input into lambda input
brunozoric Dec 21, 2023
bd480da
feat: add count dynamodb items task to dev deployment
brunozoric Dec 21, 2023
3b7cec6
fix: deployment is missing some packages because they are private
brunozoric Dec 22, 2023
7af4aca
feat(tasks): enable users to wait certain amount of time to continue …
brunozoric Dec 22, 2023
a8abed7
refactor(pulumi-aws): wait task use new response var
brunozoric Dec 22, 2023
901ef88
refactor: continuing task
brunozoric Dec 22, 2023
720aceb
fix: task status after next merge
brunozoric Dec 26, 2023
21fbcd5
fix(project-utils): add sharp to webpack externals for all Lambda fun…
Pavel910 Dec 29, 2023
b478199
feat(pulumi-aws): add sharp layer to background task function
Pavel910 Dec 29, 2023
886701f
feat(aws-layers): add ts definition
brunozoric Dec 30, 2023
4084848
fix(pulumi-aws): remove unnecessary expect error
brunozoric Dec 30, 2023
7ad2edf
fix(cwp-template-aws): position of background tasks init
brunozoric Jan 2, 2024
1caf5ad
fix(tasks): updating task
brunozoric Jan 2, 2024
17b4f1d
fix(db-dynamodb): retry batch write on unprocessed items
brunozoric Jan 2, 2024
125ead4
fix(pulumi-aws): aborted status
brunozoric Jan 2, 2024
031632f
fix(api-elasticsearch-tasks): method to add logs is now user friendly
brunozoric Jan 2, 2024
b0fa4c1
refactor(tasks): export all types from root
brunozoric Jan 2, 2024
e80f538
feat(aws-sdk): add sfn package
brunozoric Jan 4, 2024
0f9c340
feat(pulumi-aws): pass execution name into lambda
brunozoric Jan 4, 2024
7148aab
refactor(tasks): add additional fields into the task model
brunozoric Jan 4, 2024
8250161
refactor(api-elasticsearch-tasks): implement changes from tasks package
brunozoric Jan 4, 2024
4b957fa
fix(tasks): convert Error into plain object for output
brunozoric Jan 4, 2024
26da803
refactor(tasks): rename task property values to input
brunozoric Jan 6, 2024
192ff82
refactor(tasks): move logs to own model
brunozoric Jan 9, 2024
134ae77
fix(api-elasticsearch-tasks): test failing after added the logs model
brunozoric Jan 9, 2024
e704c1a
fix(tasks): only pending and running tasks can be aborted
brunozoric Jan 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test(tasks): trigger crud
  • Loading branch information
brunozoric committed Dec 29, 2023
commit 7a110d490d8b2c5fed5b44442c027984b9a74a8e
3 changes: 2 additions & 1 deletion packages/aws-sdk/src/client-eventbridge/index.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
export {
EventBridgeClient,
PutEventsRequestEntry,
PutEventsCommand
PutEventsCommand,
PutEventsCommandInput
} from "@aws-sdk/client-eventbridge";
60 changes: 60 additions & 0 deletions packages/tasks/__tests__/crud/trigger.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
import { useHandler } from "~tests/helpers/useHandler";
import { createMockTaskDefinitions } from "~tests/mocks/definition";
import { createMockIdentity } from "~tests/mocks/identity";
import { TaskDataStatus } from "~/types";

jest.mock("@webiny/aws-sdk/client-eventbridge", () => {
return {
EventBridgeClient: class EventBridgeClient {
async send(cmd: any) {
return {
input: cmd.input
};
}
},
PutEventsCommand: class PutEventsCommand {
public readonly input: any;

constructor(input: any) {
this.input = input;
}
}
};
});

describe("trigger crud", () => {
const handler = useHandler({
plugins: [...createMockTaskDefinitions()]
});

it("should trigger a task", async () => {
const context = await handler.handle();

const result = await context.tasks.trigger({
definition: "myCustomTask-1",
name: "A test of triggering task",
values: {
myAnotherCustomValue: "myAnotherCustomValue",
myCustomValue: "myCustomValue"
}
});

expect(result).toEqual({
id: expect.any(String),
name: "A test of triggering task",
definitionId: "myCustomTask-1",
values: {
myAnotherCustomValue: "myAnotherCustomValue",
myCustomValue: "myCustomValue"
},
status: TaskDataStatus.PENDING,
createdBy: createMockIdentity(),
createdOn: expect.any(Date),
savedOn: expect.any(Date),
startedOn: undefined,
finishedOn: undefined,
log: [],
eventResponse: expect.any(Object)
});
});
});
9 changes: 6 additions & 3 deletions packages/tasks/__tests__/graphql/tasks.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,8 @@ describe("graphql - tasks", () => {
finishedOn: null,
createdBy: createMockIdentity(),
createdOn: expect.any(String),
savedOn: expect.any(String)
savedOn: expect.any(String),
eventResponse: null
},
{
name: "My Custom Task #2",
Expand All @@ -77,7 +78,8 @@ describe("graphql - tasks", () => {
finishedOn: null,
createdBy: createMockIdentity(),
createdOn: expect.any(String),
savedOn: expect.any(String)
savedOn: expect.any(String),
eventResponse: null
},
{
name: "My Custom Task #1",
Expand All @@ -92,7 +94,8 @@ describe("graphql - tasks", () => {
finishedOn: null,
createdBy: createMockIdentity(),
createdOn: expect.any(String),
savedOn: expect.any(String)
savedOn: expect.any(String),
eventResponse: null
}
],
meta: {
Expand Down
1 change: 1 addition & 0 deletions packages/tasks/__tests__/helpers/graphql/tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const createListTasksQuery = () => {
status
createdOn
savedOn
eventResponse
createdBy {
id
displayName
Expand Down
2 changes: 1 addition & 1 deletion packages/tasks/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"@webiny/pubsub": "0.0.0",
"@webiny/utils": "0.0.0",
"aws-lambda": "^1.0.7",
"zod": "^3.22.4"
"zod": "^3.21.4"
},
"devDependencies": {
"@babel/cli": "^7.22.6",
Expand Down
1 change: 1 addition & 0 deletions packages/tasks/src/crud/crud.tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ const convertToTask = (entry: CmsEntry<ITaskData>): ITaskData => {
definitionId: entry.values.definitionId,
values: entry.values.values,
status: entry.values.status,
eventResponse: entry.values.eventResponse,
startedOn: getDate(entry.values.startedOn),
finishedOn: getDate(entry.values.finishedOn),
log: entry.values.log
Expand Down
7 changes: 7 additions & 0 deletions packages/tasks/src/crud/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,13 @@ const modelPlugin = createCmsModel({
type: "json",
label: "Log",
multipleValues: true
},
{
id: "eventResponse",
fieldId: "eventResponse",
storageId: "object@eventResponse",
type: "json",
label: "Event Response"
}
],
layout: [],
Expand Down
19 changes: 16 additions & 3 deletions packages/tasks/src/crud/trigger.tasks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,19 @@ export const createTriggerTasksCrud = (

const task = await context.tasks.createTask<T>(input);

let event: Record<string, any> | null = null;
try {
const event = await createEventBridgeEvent(task);
console.log("EVENT: ", JSON.stringify(event));
event = await createEventBridgeEvent(task);

if (!event) {
throw new WebinyError(
`Could not create the Event Bridge Event!`,
"CREATE_EVENT_BRIDGE_EVENT_ERROR",
{
task
}
);
}
} catch (ex) {
/**
* In case of failure to create the Event Bridge Event, we need to delete the task that was meant to be created.
Expand All @@ -65,7 +75,10 @@ export const createTriggerTasksCrud = (
await context.tasks.deleteTask(task.id);
throw ex;
}
return task;

return await context.tasks.updateTask(task.id, {
eventResponse: event
});
},
stop: async (params: ITaskStopParams): Promise<ITaskData> => {
const task = await context.tasks.getTask(params.id);
Expand Down
2 changes: 2 additions & 0 deletions packages/tasks/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ export interface ITaskData<T = any> {
createdBy: ITaskIdentity;
startedOn?: Date;
finishedOn?: Date;
eventResponse: Record<string, any>;
log?: ITaskDataLog[];
}

Expand Down Expand Up @@ -76,6 +77,7 @@ export interface ITaskUpdateData<T = ITaskDataValues> {
log?: ITaskDataLog[];
startedOn?: string;
finishedOn?: string;
eventResponse?: Record<string, any>;
}

export interface OnTaskBeforeCreateTopicParams {
Expand Down
2 changes: 1 addition & 1 deletion yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -17948,7 +17948,7 @@ __metadata:
ttypescript: ^1.5.13
type-fest: ^4.8.3
typescript: 4.7.4
zod: ^3.22.4
zod: ^3.21.4
languageName: unknown
linkType: soft

Expand Down