Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: background tasks #3740

Merged
merged 69 commits into from
Jan 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4790100
wip(tasks): start with tasks
brunozoric Nov 29, 2023
526625c
wip(tasks): split plugin and definition
brunozoric Nov 30, 2023
c893c45
wip(tasks): task runs
brunozoric Dec 4, 2023
baa2829
feat(api-headless-cms): add json type field
brunozoric Dec 4, 2023
cdaec0a
feat(tasks): graphql and crud
brunozoric Dec 5, 2023
13f2297
refactor(tasks): database response manager
brunozoric Dec 5, 2023
b1a7083
chore: update ts configs
brunozoric Dec 5, 2023
f37579f
refactor(tasks): rename tests
brunozoric Dec 6, 2023
7c9c1ea
feat(tasks): add step fn token to input and output
brunozoric Dec 6, 2023
900cb92
refactor(tasks): add state machine id to event
brunozoric Dec 8, 2023
43bbadb
feat(tasks): task crud
brunozoric Dec 8, 2023
f1cbee2
refactor(pulumi-aws): background tasks
brunozoric Dec 8, 2023
0712207
refactor(pulumi-aws): background tasks logging policy
brunozoric Dec 11, 2023
4bbb104
refactor(tasks): simplify response types
brunozoric Dec 12, 2023
5cb1a27
refactor(tasks): move classes and abstractions
brunozoric Dec 12, 2023
97dba6c
refactor(pulumi-aws): update step function definition
brunozoric Dec 12, 2023
c7294a5
feat(pulumi-aws): add waiting possibility in background task
brunozoric Dec 13, 2023
1373616
test(tasks): add crud tests
brunozoric Dec 13, 2023
e7825d2
test(tasks): definitions crud
brunozoric Dec 13, 2023
8f8bdac
fix(tasks): expose graphql
brunozoric Dec 13, 2023
37b0232
test(tasks): task manager and event validation
brunozoric Dec 14, 2023
e4fed20
feat(pulumi-aws): deploy event bridge to step fn connection
brunozoric Dec 14, 2023
68fe0e7
feat(tasks): implement graphql
brunozoric Dec 14, 2023
3d38d48
feat(tasks): ability to stop a task
brunozoric Dec 15, 2023
1f35cd1
test(tasks): graphql to list tasks from database
brunozoric Dec 15, 2023
7a110d4
test(tasks): trigger crud
brunozoric Dec 15, 2023
1bf99e2
fix(tests): validate task definition id
brunozoric Dec 15, 2023
0be1a4e
test(tasks): modify tasks to camel cased id
brunozoric Dec 15, 2023
45e7a17
feat(tasks): add task values store method
brunozoric Dec 18, 2023
3cda7ca
feat(api-elasticsearch-tasks): indexing
brunozoric Dec 18, 2023
d3bb26b
feat(api-elasticsearch-task): code split task runner
brunozoric Dec 19, 2023
aa2520a
feat(api-background-tasks): wrapper for tasks and definitions
brunozoric Dec 19, 2023
034ad7f
fix(api-background-tasks): readme
brunozoric Dec 19, 2023
ee4b630
fix(tasks): store task management
brunozoric Dec 19, 2023
b061762
test(api-elasticsearch-tasks): wip reindexing
brunozoric Dec 19, 2023
6dbd877
feat(api-background-tasks-dynamodb): ddb tasks
brunozoric Dec 20, 2023
a13cc0d
feat(api-background-tasks-es): elasticsearch tasks
brunozoric Dec 20, 2023
2f3830f
feat(api-background-tasks-os): opensearch tasks
brunozoric Dec 20, 2023
c59aa4a
refactor: rename dynamodb tasks package
brunozoric Dec 20, 2023
112c55d
fix(cwp-template-aws): dynamodb setup
brunozoric Dec 20, 2023
0bf0c6a
fix(cwp-template-aws): os setup
brunozoric Dec 20, 2023
9a33a8d
fix(tasks): event input
brunozoric Dec 20, 2023
c920e8e
refactor(tasks): graphql type names
brunozoric Dec 21, 2023
15b129d
feat(pulumi-aws): step to transform event input into lambda input
brunozoric Dec 21, 2023
bd480da
feat: add count dynamodb items task to dev deployment
brunozoric Dec 21, 2023
3b7cec6
fix: deployment is missing some packages because they are private
brunozoric Dec 22, 2023
7af4aca
feat(tasks): enable users to wait certain amount of time to continue …
brunozoric Dec 22, 2023
a8abed7
refactor(pulumi-aws): wait task use new response var
brunozoric Dec 22, 2023
901ef88
refactor: continuing task
brunozoric Dec 22, 2023
720aceb
fix: task status after next merge
brunozoric Dec 26, 2023
21fbcd5
fix(project-utils): add sharp to webpack externals for all Lambda fun…
Pavel910 Dec 29, 2023
b478199
feat(pulumi-aws): add sharp layer to background task function
Pavel910 Dec 29, 2023
886701f
feat(aws-layers): add ts definition
brunozoric Dec 30, 2023
4084848
fix(pulumi-aws): remove unnecessary expect error
brunozoric Dec 30, 2023
7ad2edf
fix(cwp-template-aws): position of background tasks init
brunozoric Jan 2, 2024
1caf5ad
fix(tasks): updating task
brunozoric Jan 2, 2024
17b4f1d
fix(db-dynamodb): retry batch write on unprocessed items
brunozoric Jan 2, 2024
125ead4
fix(pulumi-aws): aborted status
brunozoric Jan 2, 2024
031632f
fix(api-elasticsearch-tasks): method to add logs is now user friendly
brunozoric Jan 2, 2024
b0fa4c1
refactor(tasks): export all types from root
brunozoric Jan 2, 2024
e80f538
feat(aws-sdk): add sfn package
brunozoric Jan 4, 2024
0f9c340
feat(pulumi-aws): pass execution name into lambda
brunozoric Jan 4, 2024
7148aab
refactor(tasks): add additional fields into the task model
brunozoric Jan 4, 2024
8250161
refactor(api-elasticsearch-tasks): implement changes from tasks package
brunozoric Jan 4, 2024
4b957fa
fix(tasks): convert Error into plain object for output
brunozoric Jan 4, 2024
26da803
refactor(tasks): rename task property values to input
brunozoric Jan 6, 2024
192ff82
refactor(tasks): move logs to own model
brunozoric Jan 9, 2024
134ae77
fix(api-elasticsearch-tasks): test failing after added the logs model
brunozoric Jan 9, 2024
e704c1a
fix(tasks): only pending and running tasks can be aborted
brunozoric Jan 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(tests): validate task definition id
  • Loading branch information
brunozoric committed Dec 29, 2023
commit 1bf99e20f1d6e9ffa7b4c1eecec207ba383eb4ff
6 changes: 5 additions & 1 deletion packages/tasks/__tests__/mocks/context.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { PluginsContainer } from "@webiny/plugins";
import { Context, ITaskUpdateData, IUpdateTaskResponse } from "~/types";
import { MOCK_TASK_DEFINITION_ID } from "~tests/mocks/definition";
import { PartialDeep } from "type-fest";
import { createMockIdentity } from "~tests/mocks/identity";

export const createMockContext = (params?: PartialDeep<Context>): Context => {
return {
Expand All @@ -15,9 +16,12 @@ export const createMockContext = (params?: PartialDeep<Context>): Context => {
return {
...data,
id,
startedOn: new Date(),
finishedOn: undefined,
createdOn: new Date(),
savedOn: new Date(),
definitionId: MOCK_TASK_DEFINITION_ID
definitionId: MOCK_TASK_DEFINITION_ID,
createdBy: createMockIdentity()
};
},
...params?.tasks
Expand Down
19 changes: 18 additions & 1 deletion packages/tasks/__tests__/task/plugin.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import WebinyError from "@webiny/error";
import { Context } from "../types";
import { ITaskDefinition, ITaskDefinitionField, ITaskRunParams } from "~/types";
import { createTaskDefinition, createTaskDefinitionField } from "~/task/plugin";
import { createMockTaskDefinition } from "~tests/mocks/definition";

const taskField: ITaskDefinitionField = {
fieldId: "url",
Expand Down Expand Up @@ -46,7 +48,7 @@ class MyTask implements ITaskDefinition<Context, MyInput> {
}
}

describe("task definition", () => {
describe("task plugin", () => {
it("should properly create a task - plain object", async () => {
const task: ITaskDefinition<Context, MyInput> = {
id: "myCustomTask",
Expand Down Expand Up @@ -138,4 +140,19 @@ describe("task definition", () => {
expect(task.onDone).toBeInstanceOf(Function);
expect(task.onError).toBeInstanceOf(Function);
});

it("should fail on invalid task id", async () => {
let error: WebinyError | undefined;
try {
createMockTaskDefinition({
id: "id-whichIsNotValid"
});
} catch (ex) {
error = ex;
}

expect(error?.message).toEqual(
`Task ID "id-whichIsNotValid" is invalid. It must be in camelCase format, for example: "myCustomTask".`
);
});
});
3 changes: 2 additions & 1 deletion packages/tasks/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
"@webiny/pubsub": "0.0.0",
"@webiny/utils": "0.0.0",
"aws-lambda": "^1.0.7",
"lodash": "^4.17.21",
"zod": "^3.21.4"
},
"devDependencies": {
Expand All @@ -39,7 +40,7 @@
"@webiny/project-utils": "0.0.0",
"rimraf": "^3.0.2",
"ttypescript": "^1.5.13",
"type-fest": "^4.8.3",
"type-fest": "^2.19.0",
"typescript": "4.7.4"
},
"publishConfig": {
Expand Down
13 changes: 13 additions & 0 deletions packages/tasks/src/task/plugin.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import camelCase from "lodash/camelCase";
import WebinyError from "@webiny/error";
import { Plugin } from "@webiny/plugins";
import { Context, ITaskDefinition, ITaskDefinitionField } from "~/types";

Expand Down Expand Up @@ -51,6 +53,7 @@ export class TaskDefinitionPlugin<C extends Context = Context, I = any>
if (typeof task.config === "function") {
task.config(this);
}
this.validate();
}

public getTask() {
Expand All @@ -65,6 +68,16 @@ export class TaskDefinitionPlugin<C extends Context = Context, I = any>
public addField(field: ITaskDefinitionField) {
this.task.fields = (this.task.fields || []).concat([field]);
}
/**
* TODO implement zod validation if validation becomes too complex
*/
private validate(): void {
if (camelCase(this.task.id) !== this.task.id) {
throw new WebinyError(
`Task ID "${this.task.id}" is invalid. It must be in camelCase format, for example: "myCustomTask".`
);
}
}
}

export const createTaskDefinition = <C extends Context = Context, I = any>(
Expand Down
1 change: 1 addition & 0 deletions packages/tasks/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ export interface ITaskBeforeTriggerParams<C extends Context = Context, I = ITask
export interface ITaskDefinition<C extends Context = Context, I = ITaskDataValues> {
/**
* ID of the task must be unique in the system.
* It should be in camelCase format, for example: "myCustomTask".
*/
id: string;
/**
Expand Down
17 changes: 9 additions & 8 deletions yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -17944,9 +17944,10 @@ __metadata:
"@webiny/pubsub": 0.0.0
"@webiny/utils": 0.0.0
aws-lambda: ^1.0.7
lodash: ^4.17.21
rimraf: ^3.0.2
ttypescript: ^1.5.13
type-fest: ^4.8.3
type-fest: ^2.19.0
typescript: 4.7.4
zod: ^3.21.4
languageName: unknown
Expand Down Expand Up @@ -44222,6 +44223,13 @@ __metadata:
languageName: node
linkType: hard

"type-fest@npm:^2.19.0":
version: 2.19.0
resolution: "type-fest@npm:2.19.0"
checksum: a4ef07ece297c9fba78fc1bd6d85dff4472fe043ede98bd4710d2615d15776902b595abf62bd78339ed6278f021235fb28a96361f8be86ed754f778973a0d278
languageName: node
linkType: hard

"type-fest@npm:^3.10.0":
version: 3.10.0
resolution: "type-fest@npm:3.10.0"
Expand All @@ -44231,13 +44239,6 @@ __metadata:
languageName: node
linkType: hard

"type-fest@npm:^4.8.3":
version: 4.8.3
resolution: "type-fest@npm:4.8.3"
checksum: cd8c9a20a32b9d813f5f6d84bd81e52e22fb779a7ef0ae53974bae5baac427974bc8243269e9832b22ce2b6277d857b44769b3664f97dcac344e600bdd95f186
languageName: node
linkType: hard

"type-is@npm:~1.6.17, type-is@npm:~1.6.18":
version: 1.6.18
resolution: "type-is@npm:1.6.18"
Expand Down