Skip to content

Commit

Permalink
benchmarks: cleanup document. (moveit#16)
Browse files Browse the repository at this point in the history 
Fix typos, formatting, layout and naming inconsistency.
  • Loading branch information
@gavanderhoorn @v4hn
gavanderhoorn authored and v4hn committed Sep 29, 2016
1 parent fec5893 commit d0e5f43
Showing 1 changed file with 40 additions and 35 deletions.
75 changes: 40 additions & 35 deletions doc/benchmarking_tutorial.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,57 +6,58 @@ Benchmarking Tutorial
.. note:: To use this benchmarking method, you will need to download and install the ROS Warehouse plugin. Currently this is not available from debians and requires a source install for at least some aspects. For source instructions, see `this page <http://moveit.ros.org/install/source/dependencies/>`_

The `benchmarking package <https://github.com/ros-planning/moveit/tree/kinetic-devel/moveit_ros/benchmarks>`_ provides methods to benchmark motion planning algorithms and aggregate/plot statistics using the OMPL Planner Arena.
The example below demonstrates how the benchamrking can be run for a Fanuc M10ia.
The example below demonstrates how the benchmarking can be run for a Fanuc M-10iA.

Example
-------

An example is provided in the examples folder. The launch file requires a moveit config package
for the Fanuc M10ia available from `here <https://github.com/ros-planning/moveit_resources>`_.
An example is provided in the ``examples`` folder. The launch file requires a MoveIt! configuration package
for the Fanuc M-10iA available from `here <https://github.com/ros-planning/moveit_resources>`_.

To run:

#. Launch the Fanuc M10ia demo.launch:
#. Launch the Fanuc M-10iA ``demo.launch``::

roslaunch moveit_resources demo.launch db:=true
roslaunch moveit_resources demo.launch db:=true

#. Within motion planning Rviz plugin, connect to the database by pressing the **Connect** button in the **Context** tab.
#. Save a scene on the **Stored Scenes** tab and name it "Kitchen1" by double clicking the scene in the list.
#. Move the start and goal states of the Fanuc M10ia using the interactive markers.
#. Save an associated query for the "Kitchen1" scene and name the query "Pick1".
#. Also save a start state for the robot on the **Stored States** tab and name it Start1.
#. The config file "moveit_ros/benchmarks/examples/demo1.yaml" refers to the scenes, queries and start states used for benchmarking. Modify them appropriately.
#. Bring down your previous launch file.
#. Change the location "output_directory" to export the benchmarked files:
#. Within the *Motion Planning* RViz plugin, connect to the database by pressing the *Connect* button in the *Context* tab.
#. Save a scene on the *Stored Scenes* tab and name it ``Kitchen1`` by double clicking the scene in the list.
#. Move the start and goal states of the Fanuc M-10iA by using the interactive markers.
#. Save an associated query for the ``Kitchen1`` scene and name the query ``Pick1``.
#. Also save a start state for the robot on the *Stored States* tab and name it ``Start1``.
#. The config file ``moveit_ros/benchmarks/examples/demo1.yaml`` refers to the scenes, queries and start states used for benchmarking. Modify them appropriately.
#. Bring down your previous launch file (``ctrl+c``).
#. Change the location ``output_directory`` to export the benchmarked files::

rosed moveit_ros_benchmarks demo_fanuc.launch
rosed moveit_ros_benchmarks demo_fanuc.launch

#. Run the benchmarks:
#. Run the benchmarks::

roslaunch moveit_ros_benchmarks demo_fanuc.launch
roslaunch moveit_ros_benchmarks demo_fanuc.launch


Viewing Results
---------------

The benchmarks are executed and many interesting parameters are aggregated and written to a logfile. A script (moveit_benchmark_statistics.py) is supplied to parse this data and plot the statistics.
Run:
The benchmarks are executed and many interesting parameters are aggregated and written to a logfile. A script (``moveit_benchmark_statistics.py``) is supplied to parse this data and plot the statistics.

rosrun moveit_ros_benchmarks moveit_benchmark_statistics.py <path_of_logfile>
Run::

To generate a PDF of plots:
rosrun moveit_ros_benchmarks moveit_benchmark_statistics.py <path_of_logfile>

python -p <plot_filename> moveit_benchmark_statistics.py <path_of_logfile>
To generate a PDF of plots::

Alternatively, upload the database file generated by moveit_benchmark_statistics.py to `plannerarena.org <http://plannerarena.org>`_ and interactively visualize the results.
python -p <plot_filename> moveit_benchmark_statistics.py <path_of_logfile>

Alternatively, upload the database file generated by ``moveit_benchmark_statistics.py`` to `plannerarena.org <http://plannerarena.org>`_ and interactively visualize the results.


Parameters of the BenchmarkOptions Class
----------------------------------------

This class reads in parameters and options for the benchmarks to run from the ROS parameter server. The format of the parameters is assumed to be in the following form:
This class reads in parameters and options for the benchmarks to run from the ROS parameter server. The format of the parameters is assumed to be in the following form::

benchmark_config:
benchmark_config:

warehouse:
host: [hostname/IP address of ROS Warehouse node] # Default localhost
Expand All @@ -72,9 +73,9 @@ benchmark_config:
start_states: [Regex for the stored start states in the warehouse to try] # Default ""
path_constraints: [Regex for the path constraints to benchmark] # Default ""

queries: [Regex for the motion plan queries in the warehouse to try] # Default .*
goal_constraints: [Regex for the goal constraints to benchmark] # Default ""
trajectory_constraints: [Regex for the trajectory constraints to benchmark] # Default ""
queries: [Regex for the motion plan queries in the warehouse to try] # Default .*
goal_constraints: [Regex for the goal constraints to benchmark] # Default ""
trajectory_constraints: [Regex for the trajectory constraints to benchmark] # Default ""

workspace: [Bounds of the workspace the robot plans in. This is an AABB] # Optional
frame_id: [The frame the workspace parameters are specified in]
Expand All @@ -86,6 +87,7 @@ benchmark_config:
x: [x-value]
y: [y-value]
z: [z-value]

planners:
- plugin: [The name of the planning plugin the planners are in] # REQUIRED
planners: # REQUIRED
Expand All @@ -96,16 +98,19 @@ benchmark_config:
- plugin: ...
- ...


Parameters of the BenchmarkExecutor Class
-----------------------------------------
This class creates a set of MotionPlanRequests that respect the parameters given in the supplied instance of BenchmarkOptions and then executes the requests on each of the planners specified. From the BenchmarkOptions, queries, goal_constraints, and trajectory_constraints are treated as separate queries. If a set of start_states is specified, each query, goal_constraints, and trajectory_constraints is attempted with each start state (existing start states from a query are ignored). Similarly, the (optional) set of path constraints is combined combinatorially with the start-query and start-goal_constraints pairs (existing path_constraints) from a query are ignored). The workspace, if specified, overrides any existing workspace parameters.

The benchmarking pipeline does not utilize MoveGroup, and PlanningRequestAdaptors are NOT invoked.
This class creates a set of ``MotionPlanRequests`` that respect the parameters given in the supplied instance of ``BenchmarkOptions`` and then executes the requests on each of the planners specified. From the ``BenchmarkOptions``, queries, ``goal_constraints``, and ``trajectory_constraints`` are treated as separate queries. If a set of ``start_states`` is specified, each query, ``goal_constraint``, and ``trajectory_constraint`` is attempted with each start state (existing start states from a query are ignored). Similarly, the (optional) set of path constraints is combined combinatorially with the start query and start ``goal_constraint`` pairs (existing ``path_constraint`` from a query are ignored). The workspace, if specified, overrides any existing workspace parameters.

The benchmarking pipeline does not utilize ``MoveGroup``, and ``PlanningRequestAdaptors`` are **not** invoked.

It is possible to customize a benchmark run by deriving a class from ``BenchmarkExecutor`` and overriding one or more of the virtual functions. Additionally, a set of functions exists for ease of customization in derived classes:

It is possible to customize a benchmark run by deriving a class from BenchmarkExecutor and overriding one or more of the virtual functions. Additionally, a set of functions exists for ease of customization in derived classes:
- preRunEvent (invoked immediately before each call to solve)
- postRunEvent (invoked immediately after each call to solve)
- plannerSwitchEvent (invoked when the planner changes during benchmarking)
- querySwitchEvent (invoked before a new benchmark problem begin execution)
- ``preRunEvent``: invoked immediately before each call to solve
- ``postRunEvent``: invoked immediately after each call to solve
- ``plannerSwitchEvent``: invoked when the planner changes during benchmarking
- ``querySwitchEvent``: invoked before a new benchmark problem begin execution

Note, in the above, a benchmark is a concrete instance of a PlanningScene, start state, goal constraints / trajectory_constraints, and (optionally) path_constraints. A run is one attempt by a specific planner to solve the benchmark.
Note, in the above, a benchmark is a concrete instance of a ``PlanningScene``, start state, goal constraints / ``trajectory_constraints``, and (optionally) ``path_constraints``. A run is one attempt by a specific planner to solve the benchmark.

0 comments on commit d0e5f43

Please sign in to comment.