Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Operations queue #191

Merged
merged 12 commits into from
Jul 25, 2022
Prev Previous commit
added documentation for operations
  • Loading branch information
v1kko committed Jul 25, 2022
commit 6f07808a058c93ef21e0537367b17be38f6c4c7a
11 changes: 11 additions & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Operations

*Duqtools* uses delayed operations for filesystem-changing operations.
They are implemented mostly with decorators, but a function could be added directly
to the `op_queue`



::: duqtools.operations
options:
show_source: false
79 changes: 72 additions & 7 deletions duqtools/operations.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@


class Operation(BaseModel):
"""Operation, simple class which has a callable action."""
"""Operation, simple class which has a callable action.

Usually not called directly but used through Operations. has the
following members:
"""

description: str = Field(
description='description of the operation to be done')
Expand All @@ -28,20 +32,50 @@ class Operation(BaseModel):
description='keyword arguments that will be '
'passed to the action')

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def __init__(self, action, description, *args, **kwargs):
"""__init__.

Parameters
----------
description: str
description of the operation to be done
action: Callable
function to be eventually evaluated
*args: tuple
positional arguments to action
**kwargs: dict
keyword arguments to action
"""
super().__init__(action=action,
description=description,
*args,
**kwargs)
if not self.kwargs:
self.kwargs = {}

def __call__(self) -> Operation:
"""Execute the action with the args and kwargs.

Returns
-------
Operation
The operation that was executed
"""
logger.info(self.description)
self.action(*self.args, **self.kwargs)
return self


class Operations(deque):
"""Operations Queue which keeps track of all the operations that need to be
done."""
done.

It's basically dask_delayed, but custom made and a few drawbacks:
The return value from an action is eventually discarded,
communication between queue items is possible through references, or
global values, but not really recommended, and no guidance for this
is provided
"""

_instance = None
yes = False # Apply operations without prompt
Expand All @@ -54,7 +88,12 @@ def __new__(cls, *args, **kwargs):
return Operations._instance

def add(self, *args, **kwargs) -> None:
"""convenience Operation wrapper around put."""
"""convenience Operation wrapper around put. ```python from
duqtools.operations import add_to_op_queue.

op_queue.add(print, args=('Hello World,), description="Function
that prints hello world") ```
"""

self.append(Operation(*args, **kwargs))

Expand Down Expand Up @@ -111,7 +150,19 @@ def confirm_apply_all(self) -> bool:

def confirm_operations(func):
"""Decorator which confirms and applies queued operations after the
function."""
function.

```python
from duqtools.operations import confirm_operations, op_queue

@confirm_operations
def complicated_stuff()
op_queue.add(print, args=('Hello World,),
description="Function that prints hello world")
op_queue.add(print, args=('Hello World again,),
description="Function that prints hello world, again")
```
"""

def wrapper(*args, **kwargs):
ret = func(*args, **kwargs)
Expand All @@ -125,7 +176,21 @@ def wrapper(*args, **kwargs):
def add_to_op_queue(description: str):
"""Decorator which adds the function call to the op_queue, instead of
executing it directly, the string can be a format string and use the
function arguments."""
function arguments.

```python
from duqtools.operations import add_to_op_queue, op_queue

@add_to_op_queue("Printing hello world {name}")
def print_hello_world(name):
print(f"Hello World {name}")


print_hello_world("Snoozy")

op_queue.confirm_apply_all()
```
"""

def op_queue_real(func):

Expand Down
5 changes: 5 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ nav:
- config/status.md
- config/plot.md
- Dashboard: dash.md
- Operations: operations.md
- 🔗 Source code: https://github.com/CarbonCollective/fusion-dUQtools
- 🔗 Issues: https://github.com/CarbonCollective/fusion-dUQtools/issues

Expand Down Expand Up @@ -68,3 +69,7 @@ plugins:
- gen-files:
scripts:
- docs/gendocs.py

watch:
- duqtools/
- docs/