Monday, December 5, 2022
HomeArtificial IntelligenceNo TD Studying, Benefit Reweighting, or Transformers – The Berkeley Synthetic Intelligence...

No TD Studying, Benefit Reweighting, or Transformers – The Berkeley Synthetic Intelligence Analysis Weblog





An indication of the RvS coverage we be taught with simply supervised studying and a depth-two MLP. It makes use of no TD studying, benefit reweighting, or Transformers!

Offline reinforcement studying (RL) is conventionally approached utilizing value-based strategies based mostly on temporal distinction (TD) studying. Nonetheless, many latest algorithms reframe RL as a supervised studying drawback. These algorithms be taught conditional insurance policies by conditioning on purpose states (Lynch et al., 2019; Ghosh et al., 2021), reward-to-go (Kumar et al., 2019; Chen et al., 2021), or language descriptions of the duty (Lynch and Sermanet, 2021).

We discover the simplicity of those strategies fairly interesting. If supervised studying is sufficient to resolve RL issues, then offline RL might change into extensively accessible and (comparatively) simple to implement. Whereas TD studying should delicately steadiness an actor coverage with an ensemble of critics, these supervised studying strategies practice only one (conditional) coverage, and nothing else!

So, how can we use these strategies to successfully resolve offline RL issues? Prior work places ahead quite a few intelligent ideas and methods, however these methods are typically contradictory, making it difficult for practitioners to determine tips on how to efficiently apply these strategies. For instance, RCPs (Kumar et al., 2019) require rigorously reweighting the coaching information, GCSL (Ghosh et al., 2021) requires iterative, on-line information assortment, and Choice Transformer (Chen et al., 2021) makes use of a Transformer sequence mannequin because the coverage community.

Which, if any, of those hypotheses are right? Do we have to reweight our coaching information based mostly on estimated benefits? Are Transformers essential to get a high-performing coverage? Are there different crucial design choices which were disregarded of prior work?

Our work goals to reply these questions by making an attempt to determine the important components of offline RL through supervised studying. We run experiments throughout 4 suites, 26 environments, and eight algorithms. When the mud settles, we get aggressive efficiency in each surroundings suite we think about using remarkably easy components. The video above reveals the complicated conduct we be taught utilizing simply supervised studying with a depth-two MLP – no TD studying, information reweighting, or Transformers!

Let’s start with an summary of the algorithm we research. Whereas a lot of prior work (Kumar et al., 2019; Ghosh et al., 2021; and Chen et al., 2021) share the identical core algorithm, it lacks a typical identify. To fill this hole, we suggest the time period RL through Supervised Studying (RvS). We aren’t proposing any new algorithm however reasonably displaying how prior work could be considered from a unifying framework; see Determine 1.



Determine 1. (Left) A replay buffer of expertise (Proper) Hindsight relabelled coaching information

RL through Supervised Studying takes as enter a replay buffer of expertise together with states, actions, and outcomes. The outcomes could be an arbitrary operate of the trajectory, together with a purpose state, reward-to-go, or language description. Then, RvS performs hindsight relabeling to generate a dataset of state, motion, and consequence triplets. The instinct is that the actions which are noticed present supervision for the outcomes which are reached. With this coaching dataset, RvS performs supervised studying by maximizing the chance of the actions given the states and outcomes. This yields a conditional coverage that may situation on arbitrary outcomes at check time.

In our experiments, we give attention to the next three key questions.

  1. Which design choices are crucial for RL through supervised studying?
  2. How properly does RL through supervised studying truly work? We are able to do RL through supervised studying, however would utilizing a special offline RL algorithm carry out higher?
  3. What sort of consequence variable ought to we situation on? (And does it even matter?)



Determine 2. Our RvS structure. A depth-two MLP suffices in each surroundings suite we take into account.

We get good efficiency utilizing only a depth-two multi-layer perceptron. Actually, that is aggressive with all beforehand revealed architectures we’re conscious of, together with a Transformer sequence mannequin. We simply concatenate the state and consequence earlier than passing them by way of two fully-connected layers (see Determine 2). The keys that we determine are having a community with giant capability – we use width 1024 – in addition to dropout in some environments. We discover that this works properly with out reweighting the coaching information or performing any extra regularization.

After figuring out these key design choices, we research the general efficiency of RvS compared to earlier strategies. This weblog submit will overview outcomes from two of the suites we take into account within the paper.


The primary suite is D4RL Gymnasium, which comprises the usual MuJoCo halfcheetah, hopper, and walker robots. The problem in D4RL Gymnasium is to be taught locomotion insurance policies from offline datasets of various high quality. For instance, one offline dataset comprises rollouts from a completely random coverage. One other dataset comprises rollouts from a “medium” coverage skilled partway to convergence, whereas one other dataset is a mix of rollouts from medium and knowledgeable insurance policies.



Determine 3. General efficiency in D4RL Gymnasium.

Determine 3 reveals our ends in D4RL Gymnasium. RvS-R is our implementation of RvS conditioned on rewards (illustrated in Determine 2). On common throughout all 12 duties within the suite, we see that RvS-R, which makes use of only a depth-two MLP, is aggressive with Choice Transformer (DT; Chen et al., 2021). We additionally see that RvS-R is aggressive with the strategies that use temporal distinction (TD) studying, together with CQL-R (Kumar et al., 2020), TD3+BC (Fujimoto et al., 2021), and Onestep (Brandfonbrener et al., 2021). Nonetheless, the TD studying strategies have an edge as a result of they carry out particularly properly on the random datasets. This means that one would possibly choose TD studying over RvS when coping with low-quality information.


The second suite is D4RL AntMaze. This suite requires a quadruped to navigate to a goal location in mazes of various measurement. The problem of AntMaze is that many trajectories include solely items of the total path from the begin to the purpose location. Studying from these trajectories requires stitching collectively these items to get the total, profitable path.



Determine 4. General efficiency in D4RL AntMaze.

Our AntMaze ends in Determine 4 spotlight the significance of the conditioning variable. Whereas conditioning RvS on rewards (RvS-R) was the only option of the conditioning variable in D4RL Gymnasium, we discover that in D4RL AntMaze, it’s a lot better to situation RvS on $(x, y)$ purpose coordinates (RvS-G). Once we do that, we see that RvS-G compares favorably to TD studying! This was shocking to us as a result of TD studying explicitly performs dynamic programming utilizing the Bellman equation.

Why does goal-conditioning carry out higher than reward conditioning on this setting? Recall that AntMaze is designed so that straightforward imitation shouldn’t be sufficient: optimum strategies should sew collectively components of suboptimal trajectories to determine tips on how to attain the purpose. In precept, TD studying can resolve this with temporal compositionality. With the Bellman equation, TD studying can mix a path from A to B with a path from B to C, yielding a path from A to C. RvS-R, together with different conduct cloning strategies, doesn’t profit from this temporal compositionality. We hypothesize that RvS-G, however, advantages from spatial compositionality. It’s because, in AntMaze, the coverage wanted to succeed in one purpose is much like the coverage wanted to succeed in a close-by purpose. We see correspondingly that RvS-G beats RvS-R.

In fact, conditioning RvS-G on $(x, y)$ coordinates represents a type of prior information concerning the process. However this additionally highlights an essential consideration for RvS strategies: the selection of conditioning info is critically essential, and it might rely considerably on the duty.

General, we discover that in a various set of environments, RvS works properly while not having any fancy algorithmic methods (akin to information reweighting) or fancy architectures (akin to Transformers). Certainly, our easy RvS setup can match, and even outperform, strategies that make the most of (conservative) TD studying. The keys for RvS that we determine are mannequin capability, regularization, and the conditioning variable.

In our work, we handcraft the conditioning variable, akin to $(x, y)$ coordinates in AntMaze. Past the usual offline RL setup, this introduces an extra assumption, particularly, that now we have some prior details about the construction of the duty. We expect an thrilling route for future work can be to take away this assumption by automating the training of the purpose house.


We packaged our open-source code in order that it might probably robotically deal with all of the dependencies for you. After downloading the code, you may run these 5 instructions to breed our experiments:

docker construct -t rvs:newest .
docker run -it --rm -v $(pwd):/rvs rvs:newest bash
cd rvs
pip set up -e .
bash experiments/launch_gym_rvs_r.sh

This submit relies on the paper:

RvS: What’s Important for Offline RL through Supervised Studying?
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine
Worldwide Convention on Studying Representations (ICLR), 2022
[Paper] [Code]

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments