ACM SIGGRAPH · Motion, Interaction and Games 2022

SCOOT Learning High-Risk
High-Precision
Motion Control

Learning to make precise decisions when a small error has irreversible consequences.

Aalto University, Finland

Read the paper Code Video BibTeX
Learned billiards control
SCOOT executes precise billiard shots from different initial ball positions
Different initial states.
Precise learned shots.
Target ball Cue ball

The research

Abstract

MIG 2022 paper

Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise actions may be corrected with later actions, thus allowing high returns with noisy actions. In contrast, we focus on an under-researched class of high-risk, high-precision motion control problems where actions carry irreversible outcomes, driving sharp peaks and ridges to plague the state-action reward landscape.

Using computational pool as a representative example of such problems, we propose and evaluate State-Conditioned Shooting (SCOOT), a novel DRL algorithm that builds on advantage-weighted regression (AWR) with three key modifications: 1) Performing policy optimization only using elite samples, allowing the policy to better latch on to the rare high-reward action samples; 2) Utilizing a mixture-of-experts (MoE) policy, to allow switching between reward landscape modes depending on the state; 3) Adding a distance regularization term and a learning curriculum to encourage exploring diverse strategies before adapting to the most advantageous samples.

We showcase our features' performance in learning physically-based billiard shots demonstrating high action precision and discovering multiple shot strategies for a given ball configuration.

01 / The challenge

A difficult reward landscape

In computational pool, a small change in cue-ball placement or shot angle can change the outcome. Successful shots lie on sparse, sharp, and multi-modal reward ridges.

Simulated billiards table and its state-action reward landscape with sparse reward ridges
Figure 1. Left: a simulated billiards table with one-dimensional state and action variables. Right: the corresponding reward landscape, where darker regions indicate higher reward.

02 / The approach

State-Conditioned Shooting

SCOOT builds on Advantage-Weighted Regression (AWR) with three modifications for high-precision motion control.

SCOOT policy optimization with a mixture-of-experts policy and elite samples from an off-policy replay buffer
Figure 2. Policy optimization using a mixture of experts and elite samples from the experience replay buffer.

  1. Learn from elite samples

    Optimize the policy using only elite samples, so rare, high-reward actions guide learning.

  2. Learn multiple strategies

    A mixture-of-experts policy switches between reward modes depending on the state.

  3. Explore before specializing

    Distance regularization and a learning curriculum encourage diverse strategies before adapting to the most advantageous samples.

03 / See it in action

Watch the research video

High-precision control, demonstrated through physically simulated billiard shots.

Watch on YouTube

Build on this work

Citation

If you use SCOOT in your research, please cite our paper.

@inproceedings{kim2022learning,
  title     = {Learning High-Risk High-Precision Motion Control},
  author    = {Kim, Nam Hee and Kirjonen, Markus and H{\"a}m{\"a}l{\"a}inen, Perttu},
  booktitle = {Proceedings of the 15th ACM SIGGRAPH Conference on Motion, Interaction and Games},
  pages     = {1--10},
  year      = {2022},
  doi       = {10.1145/3561975.3562943}
}