Dare "corpo" all'IA: il nostro futuro con i robot intelligenti
L'intersezione tra IA e robotica apre nuovi scenari applicativi, in cui l'intelligenza dei robot va oltre la semplice produzione di testi e ...
pubblicato
Un’altra tecnica è la sim-to-real reinforcement learning. Gli algoritmi del robot si esercitano, con tentativi ed errori, in una realtà simulata, miliardi di volte, prima di passare a quella fisica. Così funziona il robot umanoide Digit. Ne parla un paper <anchor>pubblicato</anchor> qualche giorno fa su Science Robotics e frutto di una collaborazione tra università dell’Oregon e della California, Berkeley.
Real-world humanoid locomotion with reinforcement learning | Science Robotics
Advertisement

Quick Search anywhere

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Searching:

Anywhere

###### Trending Terms:

- cancer
- climate
- [artificial intelligence](/action/doSearch?AllField=artificial intelligence)
- postdoc
- aging

Quick Search anywhere

Enter Search Term

AAAS Logo

No access

Research Article

ROBOT LOCOMOTION

Share on

# Real-world humanoid locomotion with reinforcement learning

Science Robotics

17 Apr 2024

Vol 9, Issue 89

## Editor’s summary

The ability of robots to navigate adaptively and robustly in varying terrain increases their chances of success when deployed in the real world. However, stable locomotion of full-size bipedal humanoid robots creates a challenge from a controls perspective. Radosavovic et al. developed a reinforcement learning approach for controlling locomotion of a humanoid robot, Digit. They trained their model in simulation and subsequently deployed it into the real-world zero-shot and showed the potential for robust locomotion on various indoor and outdoor environments. The robot could exhibit natural and adaptive walking behaviors, including an emergent arm-swing motion, and adapt to external perturbations. —Amos Matsiko

## Abstract

Humanoid robots that can autonomously operate in diverse environments have the potential to help address labor shortages in factories, assist elderly at home, and colonize new planets. Although classical controllers for humanoid robots have shown impressive results in a number of settings, they are challenging to generalize and adapt to new environments. Here, we present a fully learning-based approach for real-world humanoid locomotion. Our controller is a causal transformer that takes the history of proprioceptive observations and actions as input and predicts the next action. We hypothesized that the observation-action history contains useful information about the world that a powerful transformer model can use to adapt its behavior in context, without updating its weights. We trained our model with large-scale model-free reinforcement learning on an ensemble of randomized environments in simulation and deployed it to the real-world zero-shot. Our controller could walk over various outdoor terrains, was robust to external disturbances, and could adapt in context.

## Get full access to this article

View all available purchase options and get full access to this article.

ALREADY A SUBSCRIBER OR AAAS MEMBER? Sign in as an individual or via your institution

## Supplementary Materials

### The PDF file includes:

Text S1 and S2

Tables S1 to S4

### Other Supplementary Material for this manuscript includes the following:

Movies S1 to S6

## REFERENCES AND NOTES

1

I. Kato, Development of WABOT 1, in Biomechanism (University of Tokyo Press, 1973).

2

K. Hirai, M. Hirose, Y. Haikawa, T. Takenaka, The development of Honda humanoid robot, in IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 1998), vol. 2, pp. 1321–1326.

3

G. Nelson, A. Saunders, N. Neville, B. Swilling, J. Bondaryk, D. Billings, C. Lee, R. Playter, M. Raibert, Petman: A humanoid robot for testing chemical protective clothing. J. Robot. Soc. Jpn. 30, 372–377 (2012).

4

O. Stasse, T. Flayols, R. Budhiraja, K. Giraud-Esclasse, J. Carpentier, J. Mirabel, A. Del Prete, P. Souères, N. Mansard, F. Lamiraux, J. P. Laumond, L. Marchionni, H. Tome, F. Ferro, TALOS: A new humanoid research platform targeted for industrial applications, in IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids) (IEEE, 2017), pp. 689–695.

5

M. Chignoli, D. Kim, E. Stanger-Jones, S. Kim, The MIT humanoid robot: Design, motion planning, and control for acrobatic behaviors, in IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids) (IEEE, 2021), pp. 1–8.

6

M. H. Raibert, Legged Robots That Balance (MIT Press, 1986).

7

S. Kajita, F. Kanehiro, K. Kaneko, K. Yokoi, H. Hirukawa, The 3D linear inverted pendulum mode: A simple modeling for a biped walking pattern generation, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2001).

8

E. R. Westervelt, J. W. Grizzle, D. E. Koditschek, Hybrid zero dynamics of planar biped walkers. IEEE Trans Automat Contr 48, 42–56 (2003).

9

S. Collins, A. Ruina, R. Tedrake, M. Wisse, Efficient bipedal robots based on passive-dynamic walkers. Science 307, 1082–1085 (2005).

10

Y. Tassa, T. Erez, E. Todorov, Synthesis and stabilization of complex behaviors through online trajectory optimization, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2012), pp. 4906–4913.

11

S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, R. Tedrake, Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Auton Robots 40, 429–455 (2016).

12

J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, S. Kim, Dynamic locomotion in the MIT Cheetah 3 through convex model-predictive control, in IEEE/RSJ International Conference on Intelligent Robots And Systems (IROS) (IEEE, 2018), pp. 1–9.

13

M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, W. Zaremba, Learning dexterous in-hand manipulation. Int. J. Robot. Res. 39, 3–20 (2020).

14

OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, L. Zhang, Solving Rubik’s cube with a robot hand. arXiv: 1910.07113 (2019).

15

A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, DeXtreme: Transfer of agile in-hand manipulation from simulation to reality, in Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2023), pp. 5977–5984.

16

J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, M. Hutter, Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, eaau5872 (2019).

17

J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, M. Hutter, Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5, eabc5986 (2020).

18

A. Kumar, Z. Fu, D. Pathak, J. Malik, RMA: Rapid motor adaptation for legged robots, Proceedings of the Robotics: Science and Systems (RSS); Virtual Event, 12 to 16 July 2021.

19

H. Benbrahim, J. A. Franklin, Biped dynamic walking using reinforcement learning. Rob. Auton. Syst. 22, 283–302 (1997).

20

R. Tedrake, T. W. Zhang, H. S. Seung, Stochastic policy gradient reinforcement learning on a simple 3D biped, in 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2004), vol. 3, pp. 2849–2854.

21

Z. Xie, G. Berseth, P. Clary, J. Hurst, M. van de Panne, Feedback control for cassie with deep reinforcement learning, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2018), pp. 1241–1246.

22

J. Siekmann, Y. Godse, A. Fern, J. Hurst, Sim-to-real learning of all common bipedal gaits via periodic reward composition, in IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2021), pp. 7309–7315.

23

J. Siekmann, K. Green, J. Warila, A. Fern, J. Hurst, Blind bipedal stair traversal via sim-to-real reinforcement learning, in Proceedings of the Robotics: Science and Systems (RSS) (RSS, 2021).

24

S. Iida, S. Kato, K. Kuwayama, T. Kunitachi, M. Kanoh, H. Itoh, Humanoid robot control based on reinforcement learning, in Micro-Nanomechatronics and Human Science, 2004 and The Fourth Symposium Micro-Nanomechatronics for Information-Based Society, 2004 (IEEE, 2004), pp. 353–358.

25

D. Rodriguez, S. Behnke, Deepwalk: Omnidirectional bipedal gait by deep reinforcement learning, in 2021 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2021), pp. 3033–3039.

26

G. A. Castillo, B. Weng, W. Zhang, A. Hereid, Reinforcement learning-based cascade motion policy design for robust 3D bipedal locomotion. IEEE Access 10, 20135–20148 (2022).

27

L. Krishna, G. A. Castillo, U. A. Mishra, A. Hereid, S. Kolathaya, Linear policies are sufficient to realize robust bipedal walking on challenging terrains. IEEE Robot. Autom. Lett. 7, 2047–2054 (2022).

28

R. Antonova, S. Cruciani, C. Smith, D. Kragic, Reinforcement learning for pivoting task, arXiv: 1703.00472 (2017).

29

F. Sadeghi, S. Levine, Cad2rl: Real single-image flight without a single real image, in Proceedings of the Robotics: Science and Systems (RSS) (RSS, 2016).

30

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, P. Abbeel, Domain randomization for transferring deep neural networks from simulation to the real world, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2017), pp. 23–30.

31

X. B. Peng, M. Andrychowicz, W. Zaremba, P. Abbeel, Sim-to-real transfer of robotic control with dynamics randomization, in IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2018), pp. 3803–3810.

32

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. M. Candlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS, 2020), pp. 1877–1901.

33

A. K. Han, A. Hajj-Ahmad, M. R. Cutkosky, Bimanual handling of deformable objects with hybrid adhesion. IEEE Robot. Autom. Lett. 7, 5497–5503 (2022).

34

G. A. Castillo, B. Weng, W. Zhang, A. Hereid, Robust feedback motion policy design using reinforcement learning on a 3D digit bipedal robot, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2021), pp. 5136–5143.

35

Y. Gao, Y. Gong, V. Paredes, A. Hereid, Y. Gu, Time-varying alip model and robust foot-placement control for underactuated bipedal robotic walking on a swaying rigid surface, in 2023 American Control Conference (ACC) (IEEE, 2023), pp. 3282–3287.

36

Y. Gao, C. Yuan, Y. Gu, Invariant filtering for legged humanoid locomotion on a dynamic rigid surface. IEEE ASME Trans. Mechatron. 27, 1900–1909 (2022).

37

A. Adu-Bredu, N. Devraj, O. C. Jenkins, Optimal constrained task planning as mixed integer programming, in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2022), pp. 12029–12036.

38

K. S. Narkhede, A. M. Kulkarni, D. A. Thanki, I. Poulakakis, A sequential mpc approach to reactive planning for bipedal robots using safe corridors in highly cluttered environments. IEEE Robot. Autom. Lett. 7, 11831–11838 (2022).

39

A. Shamsah, Z. Gu, J. Warnke, S. Hutchinson, Y. Zhao, Integrated task and motion planning for safe legged navigation in partially observable environments. IEEE Trans. Robot. 39, 4913–4934 (2023).

40

D. J. Morton, D. D. Fuller, Human Locomotion and Body Form: A Study of Gravity and Man (Williams & Wilkins, 1952).

41

H. Herr, M. Popovic, Angular momentum in human walking. J. Exp. Biol. 211, 467–481 (2008).

42

S. H. Collins, P. G. Adamczyk, A. D. Kuo, Dynamic arm swinging in human walking. Proc. R. Soc. B Biol. Sci. 276, 3679–3688 (2009).

43

J. D. Ortega, L. A. Fehlman, C. T. Farley, Effects of aging and arm swing on the metabolic cost of stability in human walking. J. Biomech. 41, 3303–3308 (2008).

44

B. R. Umberger, Effects of suppressing arm swing on kinematics, kinetics, and energetics of human walking. J. Biomech. 41, 2575–2580 (2008).

45

M. Murray, S. Sepic, E. Barnard, Patterns of sagittal rotation of the upper limbs in walking. Phys. Ther. 47, 272–284 (1967).

46

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J.Wu, D. Amodei, Scaling laws for neural language models. arXiv: 2001.08361 (2020).

47

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Bińkowski, R. Barreira, O. Vinyals, A. Zisserman, K. Simonyan, Flamingo: A visual language model for few-shot learning. Adv. Neural Inf. Process. Syst. 35, 23716 (2022).

48

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in International Conference on Learning Representations (2021).

49

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding by generative pre-training (2018).

50

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS, 2017), pp. 6000–6010.

51

J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv: 1810.04805 (2018).

52

L. Dong, S. Xu, B. Xu, Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2018), pp. 5884–5888.

53

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in ECCV 2020: 16th European Conference, vol. 12346 of Lecture Notes in Computer Science (Springer, 2020), pp. 213–229.

54

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv: 1707.06347 (2017).

55

X. Da, O. Harib, R. Hartley, B. Griffin, J. W. Grizzle, From 2d design of underactuated bipedal gaits to 3d implementation: Walking with speed tracking. IEEE Access 4, 3469–3478 (2016).

56

Y. Gong, R. Hartley, X. Da, A. Hereid, O. Harib, J.-K. Huang, J. Grizzle, Feedback control of a cassie bipedal robot: Walking, standing, and riding a Segway, in 2019 American Control Conference (ACC) (IEEE, 2019), pp. 4559–4566.

57

V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, G. State, Isaac Gym: High performance GPU-based physics simulation for robot learning, arXiv: 2108.10470 (2021).

58

N. Rudin, D. Hoeller, P. Reist, M. Hutter, Learning to walk in minutes using massively parallel deep reinforcement learning, in Conference on Robot Learning (MLResearchPress, 2022).

59

S. Bai, J. Z. Kolter, V. Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv: 1803.01271 (2018).

60

S. Hochreiter, J. Schmidhuber, Long short-term memory. Neural Comput. 9, 1735–1780 (1997).

61

J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, Sim-to-real: Learning agile locomotion for quadruped robots, in Proceedings of the Robotics: Science and Systems (RSS) (RSS, 2018).

### Submit a Response to This Article

×

##### Compose eLetter

Title:

Title is required

Contents:

##### Contributors

remove contributor

First name:

Last name:

Email:

Role/occupation:

affiliation:

add another contributor

##### Statement of Competing Interests

Competing interests?

YES

NO

Please describe the competing interests

CANCELSUBMIT

## (0)eLetters

eLetters is a forum for ongoing peer review. eLetters are not edited, proofread, or indexed, but they are screened. eLetters should provide substantive and scholarly commentary on the article. Embedded figures cannot be submitted, and we discourage the use of figures within eLetters in general. If a figure is essential, please include a link to the figure within the text of the eLetter. Please read our Terms of Service before submitting an eLetter.

No eLetters have been published for this article yet.

## Information & Authors

### Information

#### Published In



Science Robotics

Volume 9 | Issue 89

April 2024

#### Copyright

Copyright © 2024 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works.

#### Article versions

#### Submission history

Received: 31 May 2023

Accepted: 26 March 2024

#### Permissions

Request permissions for this article.

#### Acknowledgments

We thank S. Kamat, B. Shi, and S. Cakir for help with experiments; Y. Liu for discussions of spring approximation in simulation; A. Srinivas, A. Gupta, A. Kumar, W. Peebles, T. Brooks, M. Tancik, S. Chen, Z. Li, B. McInroe, and R. Huang for helpful discussions; G. State, P. Reist, V. Makoviychuk, A. Handa, and the I. Gym team for simulation discussions; J. Thomas, J. Thompson, L. Allery, J. Hurst, and the Agility Robotics team for hardware discussions.

Funding: This work was supported in part by DARPA Machine Common Sense program, ONR MURI program (N00014-21-1-2801), NVIDIA, InnoHK of the Government of the Hong Kong Special Administrative Region via the Hong Kong Centre for Logistics Robotics, the AI Institute, and BAIR’s industrial alliance programs.

Author contributions: I.R., T.X., and B.Z. led the project. T.D., J.M., and K.S. advised the project.

Competing interests: J.M. is also (part-time) affiliated with Meta Platforms Inc., but this research was not sponsored or supported by Meta. The other authors declare that they have no competing interests.

Data and materials availability: Additional data and materials are present in the Supplementary Materials.

### Authors

#### Affiliations

University of California, Berkeley CA, USA.

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Supervision, Validation, Visualization, Writing - original draft, and Writing - review & editing.

University of California, Berkeley CA, USA.

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, and Writing - review & editing.

University of California, Berkeley CA, USA.

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, and Writing - review & editing.

TrevorDarrell‡

University of California, Berkeley CA, USA.

Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Visualization, and Writing - review & editing.

University of California, Berkeley CA, USA.

Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Validation, Writing - original draft, and Writing - review & editing.

University of California, Berkeley CA, USA.

Roles: Conceptualization, Funding acquisition, Investigation, Project administration, Resources, Supervision, Validation, Visualization, and Writing - review & editing.

#### Funding Information

#### Notes

*

†

These authors contributed equally to this work.

‡

These authors contributed equally to this work.

## Metrics & Citations

### Metrics

#### Article Usage

Note: The article usage is presented with a three- to four-day delay and will update daily once available. Due to this delay, usage data will not appear immediately following publication.

#### Altmetrics

### Citations

#### Cite as

- Ilija Radosavovic et al.

,

Real-world humanoid locomotion with reinforcement learning.Sci. Robot.9,eadi9579(2024).DOI: 10.1126/scirobotics.adi9579

#### Export citation

Select the format you want to export the citation of this publication.

Please select one from the listRIS (ProCite, Reference Manager)EndNoteBibTexMedlarsRefWorks

Direct import

## View Options

### Check Access

#### Log in to view the full text

Loading institution options

AAAS login provides access to Science for AAAS Members, and access to other journals in the Science family to users who have purchased individual subscriptions.

#### More options

Download and print this article within 24 hours for your personal scholarly, research, and educational
use.

### View options

#### PDF format

Download this article as a PDF file

#### Full Text

## Media

### Figures

### Multimedia

## Tables

## Share

### Share

#### Share article link

Copy Link

Copied!

Copying failed.

#### Share on social media

FacebookXLinkedinRedditWeChatWhatsAppemail

#### Current Issue

### Stretchable Arduinos embedded in soft robots

- By
  - Stephanie J. Woodman
  - Dylan S. Shah
  - et al.

### Restoration of grasping in an upper limb amputee using the myokinetic prosthesis with implanted magnets

- By
  - Marta Gherardini
  - Valerio Ianniciello
  - et al.

### Visual seafloor mapping with autonomous robots

- By
  - Amos Matsiko

#### Advertisement

### Sign up for ScienceAdviser

Subscribe to ScienceAdviser to get the latest news, commentary, and research, free to your inbox daily.

#### LATEST NEWS

#### Advertisement

#### Advertisement

Now Reading:

SHARE

Open in viewer

GO TO

GO TO

Show all references

Request permissionsExpand All

Collapse

Expand for more

SHOW ALL BOOKS

###### Follow Us

###### Follow Us

© 2024 American Association for the Advancement of Science. All rights reserved. AAAS is a partner of HINARI, AGORA, OARE, CHORUS, CLOCKSS, CrossRef and COUNTER. Science Robotics eISSN 2470-9476.

back to top

Get the latest news, commentary, and research, free to your inbox daily.

×

Back to article