Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BashOperator: Execute templated bash script as file #42783

Merged
merged 5 commits into from
Oct 24, 2024
Merged

BashOperator: Execute templated bash script as file #42783

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3233f01
feat: execute templated shell script as file
joffreybienvenu-infrabel Oct 6, 2024
ae76bf8
feat: Add documentation
joffreybienvenu-infrabel Oct 6, 2024
4a1079a
fix: docstring typo
joffreybienvenu-infrabel Oct 6, 2024
8afa080
feat: Add newsfragment
joffreybienvenu-infrabel Oct 19, 2024
0dd1a4d
fix: mark tests using db with `@pytest.mark.db_test`
joffreybienvenu-infrabel Oct 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: Add documentation
  • Loading branch information
@joffreybienvenu-infrabel
joffreybienvenu-infrabel committed Oct 19, 2024
commit ae76bf80cb97555438d672658e89204455dcdcff
138 changes: 79 additions & 59 deletions docs/apache-airflow/howto/operator/bash.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,70 +231,21 @@ Executing commands from files
Both the ``BashOperator`` and ``@task.bash`` TaskFlow decorator enables you to execute Bash commands stored
in files. The files **must** have a ``.sh`` or ``.bash`` extension.

Note the space after the script name (more on this in the next section).

.. tab-set::

.. tab-item:: @task.bash
:sync: taskflow

.. code-block:: python
:emphasize-lines: 3

@task.bash
def run_command_from_script() -> str:
return "$AIRFLOW_HOME/scripts/example.sh "
With Jinja template
"""""""""""""""""""

You can execute bash script which contains Jinja templates. When you do so, Airflow
loads the content of your file, render the templates, and write the rendered script
into a temporary file. By default, the file is placed in a temporary directory
(under ``/tmp``). You can change this location with the ``cwd`` parameter.

run_script = run_command_from_script()

.. tab-item:: BashOperator
:sync: operator

.. code-block:: python
:emphasize-lines: 3

run_script = BashOperator(
task_id="run_command_from_script",
bash_command="$AIRFLOW_HOME/scripts/example.sh ",
)


Jinja template not found
""""""""""""""""""""""""

If you encounter a "Template not found" exception when trying to execute a Bash script, add a space after the
script name. This is because Airflow tries to apply a Jinja template to it, which will fail.

.. tab-set::

.. tab-item:: @task.bash
:sync: taskflow

.. code-block:: python

@task.bash
def bash_example():
# This fails with 'Jinja template not found' error
# return "/home/batcher/test.sh",
# This works (has a space after)
return "/home/batcher/test.sh "

.. tab-item:: BashOperator
:sync: operator
.. caution::

.. code-block:: python
Airflow must have write access to ``/tmp`` or the ``cwd`` directory, to be
able to write the temporary file to the disk.

BashOperator(
task_id="bash_example",
# This fails with 'Jinja template not found' error
# bash_command="/home/batcher/test.sh",
# This works (has a space after)
bash_command="/home/batcher/test.sh ",
)

However, if you want to use templating in your Bash script, do not add the space
and instead put your Bash script in a location relative to the directory containing
To execute a bash script, place it in a location relative to the directory containing
the DAG file. So if your DAG file is in ``/usr/local/airflow/dags/test_dag.py``, you can
move your ``test.sh`` file to any location under ``/usr/local/airflow/dags/`` (Example:
``/usr/local/airflow/dags/scripts/test.sh``) and pass the relative path to ``bash_command``
Expand Down Expand Up @@ -357,6 +308,75 @@ locations in the DAG constructor call.
bash_command="test.sh ",
)

Without Jinja template
""""""""""""""""""""""

If your script doesn't contains any Jinja template, disable Airflow's rendering by
adding a space after the script name.

.. tab-set::

.. tab-item:: @task.bash
:sync: taskflow

.. code-block:: python
:emphasize-lines: 3

@task.bash
def run_command_from_script() -> str:
return "$AIRFLOW_HOME/scripts/example.sh "


run_script = run_command_from_script()

.. tab-item:: BashOperator
:sync: operator

.. code-block:: python
:emphasize-lines: 3

run_script = BashOperator(
task_id="run_command_from_script",
bash_command="$AIRFLOW_HOME/scripts/example.sh ",
)


Jinja template not found
""""""""""""""""""""""""

If you encounter a "Template not found" exception when trying to execute a Bash script, add a space after the
script name. This is because Airflow tries to apply a Jinja template to it, which will fail.

.. tab-set::

.. tab-item:: @task.bash
:sync: taskflow

.. code-block:: python

@task.bash
def bash_example():
# This fails with 'Jinja template not found' error
# return "/home/batcher/test.sh",
# This works (has a space after)
return "/home/batcher/test.sh "

.. tab-item:: BashOperator
:sync: operator

.. code-block:: python

BashOperator(
task_id="bash_example",
# This fails with 'Jinja template not found' error
# bash_command="/home/batcher/test.sh",
# This works (has a space after)
bash_command="/home/batcher/test.sh ",
)

However, if you want to use templating in your Bash script, do not add the space
and instead check the `bash script with Jinja template <#with-jinja-template>`_ section.

Enriching Bash with Python
--------------------------

Expand Down