Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: background tasks #3740

Merged
merged 69 commits into from
Jan 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4790100
wip(tasks): start with tasks
brunozoric Nov 29, 2023
526625c
wip(tasks): split plugin and definition
brunozoric Nov 30, 2023
c893c45
wip(tasks): task runs
brunozoric Dec 4, 2023
baa2829
feat(api-headless-cms): add json type field
brunozoric Dec 4, 2023
cdaec0a
feat(tasks): graphql and crud
brunozoric Dec 5, 2023
13f2297
refactor(tasks): database response manager
brunozoric Dec 5, 2023
b1a7083
chore: update ts configs
brunozoric Dec 5, 2023
f37579f
refactor(tasks): rename tests
brunozoric Dec 6, 2023
7c9c1ea
feat(tasks): add step fn token to input and output
brunozoric Dec 6, 2023
900cb92
refactor(tasks): add state machine id to event
brunozoric Dec 8, 2023
43bbadb
feat(tasks): task crud
brunozoric Dec 8, 2023
f1cbee2
refactor(pulumi-aws): background tasks
brunozoric Dec 8, 2023
0712207
refactor(pulumi-aws): background tasks logging policy
brunozoric Dec 11, 2023
4bbb104
refactor(tasks): simplify response types
brunozoric Dec 12, 2023
5cb1a27
refactor(tasks): move classes and abstractions
brunozoric Dec 12, 2023
97dba6c
refactor(pulumi-aws): update step function definition
brunozoric Dec 12, 2023
c7294a5
feat(pulumi-aws): add waiting possibility in background task
brunozoric Dec 13, 2023
1373616
test(tasks): add crud tests
brunozoric Dec 13, 2023
e7825d2
test(tasks): definitions crud
brunozoric Dec 13, 2023
8f8bdac
fix(tasks): expose graphql
brunozoric Dec 13, 2023
37b0232
test(tasks): task manager and event validation
brunozoric Dec 14, 2023
e4fed20
feat(pulumi-aws): deploy event bridge to step fn connection
brunozoric Dec 14, 2023
68fe0e7
feat(tasks): implement graphql
brunozoric Dec 14, 2023
3d38d48
feat(tasks): ability to stop a task
brunozoric Dec 15, 2023
1f35cd1
test(tasks): graphql to list tasks from database
brunozoric Dec 15, 2023
7a110d4
test(tasks): trigger crud
brunozoric Dec 15, 2023
1bf99e2
fix(tests): validate task definition id
brunozoric Dec 15, 2023
0be1a4e
test(tasks): modify tasks to camel cased id
brunozoric Dec 15, 2023
45e7a17
feat(tasks): add task values store method
brunozoric Dec 18, 2023
3cda7ca
feat(api-elasticsearch-tasks): indexing
brunozoric Dec 18, 2023
d3bb26b
feat(api-elasticsearch-task): code split task runner
brunozoric Dec 19, 2023
aa2520a
feat(api-background-tasks): wrapper for tasks and definitions
brunozoric Dec 19, 2023
034ad7f
fix(api-background-tasks): readme
brunozoric Dec 19, 2023
ee4b630
fix(tasks): store task management
brunozoric Dec 19, 2023
b061762
test(api-elasticsearch-tasks): wip reindexing
brunozoric Dec 19, 2023
6dbd877
feat(api-background-tasks-dynamodb): ddb tasks
brunozoric Dec 20, 2023
a13cc0d
feat(api-background-tasks-es): elasticsearch tasks
brunozoric Dec 20, 2023
2f3830f
feat(api-background-tasks-os): opensearch tasks
brunozoric Dec 20, 2023
c59aa4a
refactor: rename dynamodb tasks package
brunozoric Dec 20, 2023
112c55d
fix(cwp-template-aws): dynamodb setup
brunozoric Dec 20, 2023
0bf0c6a
fix(cwp-template-aws): os setup
brunozoric Dec 20, 2023
9a33a8d
fix(tasks): event input
brunozoric Dec 20, 2023
c920e8e
refactor(tasks): graphql type names
brunozoric Dec 21, 2023
15b129d
feat(pulumi-aws): step to transform event input into lambda input
brunozoric Dec 21, 2023
bd480da
feat: add count dynamodb items task to dev deployment
brunozoric Dec 21, 2023
3b7cec6
fix: deployment is missing some packages because they are private
brunozoric Dec 22, 2023
7af4aca
feat(tasks): enable users to wait certain amount of time to continue …
brunozoric Dec 22, 2023
a8abed7
refactor(pulumi-aws): wait task use new response var
brunozoric Dec 22, 2023
901ef88
refactor: continuing task
brunozoric Dec 22, 2023
720aceb
fix: task status after next merge
brunozoric Dec 26, 2023
21fbcd5
fix(project-utils): add sharp to webpack externals for all Lambda fun…
Pavel910 Dec 29, 2023
b478199
feat(pulumi-aws): add sharp layer to background task function
Pavel910 Dec 29, 2023
886701f
feat(aws-layers): add ts definition
brunozoric Dec 30, 2023
4084848
fix(pulumi-aws): remove unnecessary expect error
brunozoric Dec 30, 2023
7ad2edf
fix(cwp-template-aws): position of background tasks init
brunozoric Jan 2, 2024
1caf5ad
fix(tasks): updating task
brunozoric Jan 2, 2024
17b4f1d
fix(db-dynamodb): retry batch write on unprocessed items
brunozoric Jan 2, 2024
125ead4
fix(pulumi-aws): aborted status
brunozoric Jan 2, 2024
031632f
fix(api-elasticsearch-tasks): method to add logs is now user friendly
brunozoric Jan 2, 2024
b0fa4c1
refactor(tasks): export all types from root
brunozoric Jan 2, 2024
e80f538
feat(aws-sdk): add sfn package
brunozoric Jan 4, 2024
0f9c340
feat(pulumi-aws): pass execution name into lambda
brunozoric Jan 4, 2024
7148aab
refactor(tasks): add additional fields into the task model
brunozoric Jan 4, 2024
8250161
refactor(api-elasticsearch-tasks): implement changes from tasks package
brunozoric Jan 4, 2024
4b957fa
fix(tasks): convert Error into plain object for output
brunozoric Jan 4, 2024
26da803
refactor(tasks): rename task property values to input
brunozoric Jan 6, 2024
192ff82
refactor(tasks): move logs to own model
brunozoric Jan 9, 2024
134ae77
fix(api-elasticsearch-tasks): test failing after added the logs model
brunozoric Jan 9, 2024
e704c1a
fix(tasks): only pending and running tasks can be aborted
brunozoric Jan 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(tasks): updating task
brunozoric committed Jan 2, 2024
commit 1caf5ad726277114ba4e9c65207054874473a818
18 changes: 12 additions & 6 deletions packages/tasks/src/runner/TaskManagerStore.ts
Original file line number Diff line number Diff line change
@@ -18,14 +18,14 @@ import deepEqual from "deep-equal";

const getValues = <T extends ITaskDataValues = ITaskDataValues>(
values: T,
param: ITaskManagerStoreUpdateTaskValuesParam<T>
input: ITaskManagerStoreUpdateTaskValuesParam<T>
) => {
if (typeof param === "function") {
return param(values);
if (typeof input === "function") {
return input(values);
}
return {
...values,
...param.values
...input
};
};

@@ -72,6 +72,7 @@ export class TaskManagerStore implements ITaskManagerStore {
param: ITaskManagerStoreUpdateTaskValuesParam<T>
): Promise<void> {
const values = getValues<T>(this.task.values, param);

/**
* No need to update if nothing changed.
*/
@@ -87,9 +88,14 @@ export class TaskManagerStore implements ITaskManagerStore {
return this.task.values as T;
}

public async addLog(log: ITaskDataLog): Promise<void> {
public async addLog(log: Omit<ITaskDataLog, "createdOn">): Promise<void> {
this.task = await this.context.tasks.updateTask(this.task.id, {
log: this.task.log.concat([log])
log: this.task.log.concat([
{
...log,
createdOn: new Date().toISOString()
}
])
});
}
}
2 changes: 1 addition & 1 deletion packages/tasks/src/runner/abstractions/TaskManagerStore.ts
Original file line number Diff line number Diff line change
@@ -37,5 +37,5 @@ export interface ITaskManagerStore<T extends ITaskDataValues = ITaskDataValues>
*/
updateValues: (param: ITaskManagerStoreUpdateTaskValuesParam<T>) => Promise<void>;
getValues: () => T;
addLog: (log: ITaskDataLog) => Promise<void>;
addLog: (log: Omit<ITaskDataLog, "createdOn">) => Promise<void>;
}