Reinforcement Learning (RL) has boomed over the last decade after DeepMind’s breakthroughs with DQN and AlphaGo.

So what are the best books to learn Reinforcement Learning? It depends partly on what you’re looking for. There are so many options, it can be hard to differentiate between them.

Here’s a list of the pros and cons of 9 of the best options. Each review has:

1. How much theoretical background is provided
2. How much code it includes
3. How many practical applications are included
4. The level of accessibility
5. How complete the coverage of the subject is

We focuses on Reinforcement Learning as a subfield of machine learning (rather than animal behaviour).

1) Reinforcement Learning: An Introduction - Sutton and Barto (2018)

Printed version price: $65 | Free online: Yes - Source link | Amazon link

This is perhaps the classic Reinforcement Learning book. If you want an introduction to the theory, this is the place to go. It’s the most cited book in the field.

The book is written by iconic figures in the field Rich Sutton and Andrew Barto. The names Sutton and Barto are now almost synonymous with Reinforcement Learning.

It’s a wonderful reference text for Reinforcement Learning. The book's limitations are that it’s purely theoretical. It’s challenging to read through cover to cover and there is no code included. In the final section ‘Frontiers’, some applications are discussed, but only in a limited way. If you’re looking for a hands-on approach or applications, this isn’t for you.

Theory: 10/10 | Coding: 0/10 | Applications: 3/10 | Accessibility: 4/10 | Coverage: 9/10

2) Algorithms for Reinforcement Learning - Csaba Szepesvari (2010)

Printed version price: $34.99 | Free online: Yes - Source link | Amazon link Amazon link

This is a short and punchy book. It outlines the Reinforcement Learning problems and algorithms to solve them.

The author aimed for the book to be concise, and yet contain the major ideas underlying the state-of-the-art algorithms.

This is a purely theoretical book - there is no code, and it doesn’t try to say why Reinforcement Learning is useful. It’s like a concise version of Sutton and Barto’s classic work, focusing on the fundamentals.

Theory: 8/10 | Coding: 0/10 | Applications: 0/10 | Accessibility: 5/10 | Coverage: 5/10

3) Neuro-Dynamic Programming - Bertsekas and Tsitsiklis (1996)

Printed Version Price: $69.00 | Free Online: No | Amazon link

Neuro-Dynamic Programming is the oldest book in this list. If you’re looking for the original book on Reinforcement Learning, this is it. Originally published in 1996, it provides a window into the field before it exploded.

The authors - Dimitri Bertsekas and John Tsitsiklis - are both Professors of Computer Science at MIT.

The most recent version we could find was published in 1996, so the more recent algorithmic advances like DQN and AlphaGo aren’t included. However, it’s still a solid theoretical foundation for the fundamentals of Reinforcement Learning.

Theory: 8/10 | Coding: 0/10 | Applications: 0/10 | Accessibility: 5/10 | Coverage: 5/10

4) Reinforcement Learning: Industrial Applications of Intelligent Agents - Phil Winder (2020)

Printed version price: $33.60 | Free online: No | Amazon link

Phil Winder’s book falls somewhere between theory and application. The range of content covered is similar to Sutton and Barto, but with an applied focus rather than a theoretical focus. Also similarly to Sutton and Barto, there’s no code here. Algorithms are shown in pseudocode alongside a practical example.

Winder runs an AI consultancy, and you can tell that he’s applied these algorithms in business-relevant settings. It’s well illustrated, and the applications are clear.

This is most relevant for someone wanting to understand Reinforcement Learning applications and understand in depth how they might be applied. It’s less accessible, and not for those looking to implement algorithms directly.

Theory: 7/10 | Coding: 0/10 | Applications: 8/10 | Accessibility: 4/10 | Coverage: 8/10

5) Deep Reinforcement Learning Hands-On - Maxim Lapan (2020)

Printed Version Price: $54.99 | Free online: No | Amazon link

If you’re looking to get your hands dirty and build something in the real world, this is the book for you.

Maxim Lapan is a Reinforcement Learning practitioner, rather than an academic. And this practical approach comes through in the book in spades. The practical applications included in the book are Stock Trading, Chatbots, Web Navigation and Robotics.

The code is written in Python, and specifically PyTorch, with a range of other problem-specific libraries covered. While there aren’t exercises to complete, code is available in the book and online for all of the algorithms discussed.

Theory: 4/10 | Coding: 7/10 | Applications: 8/10 | Accessibility: 3/10 | Coverage: 5/10

6) Mastering Reinforcement Learning with Python - Enes Bilgin (2020)

Printed version price: $46.99 | Free online: No | Amazon link

This is another hands-on, coding and applications-focused Reinforcement Learning book. You’ll find the theoretical background you need interwoven with practical examples written in Python throughout.

Enes Bilgin’s role as an AI Engineer at Microsoft and Research Scientist at Amazon lends this book a very practical and applied lens. The theory is still here, but the focus isn’t on mathematics.

The book ends with a section on applications, with subjects covered including Supply Chain Management, Autonomous Systems, Marketing, Finance and Smart Cities. For each, code is given and walked through.

Overall it’s a highly practical way to go deep into Reinforcement Learning applications in Python.

Theory: 4/10 | Coding: 6/10 | Applications: 8/10 | Accessibility: 5/10 | Coverage: 7/10

7) Applied Reinforcement Learning with Python - Taweh Beysolow II (2019)

Printed Version Price: $34.99 | Free online: No | Amazon link

As the title indicates, this is a more applied book. It focuses on the Python packages Tensorflow and Keras, using OpenAI Gym. There is code embedded throughout and available online.

The book is focused on applications, though the applications are mostly focused on games and other components of the OpenAI Gym. You’ll find much less about real-world applications here.

The coverage of each algorithm is minimal and concise. The book covers the theoretical foundations without getting bogged down in them. It covers them to a degree needed to apply them, not to become a Reinforcement Learning researcher.

Theory: 4/10 | Coding: 8/10 | Applications: 5/10 | Accessibility: 7/10 | Coverage: 5/10

8) Applying Reinforcement Learning on Real-World Data with Practical Examples in Python - Philip Osborne, Kajal Singh, Matthew Taylor (2022)

Printed Version Price: $62.75 | Free Online: No | Amazon link

This is a concise and application focused book. It overviews Reinforcement Learning and Deep Learning, before diving into practical applications.

The authors are academics from the Universities of Manchester, Oxford and Alberta. However, the book reads with a much more practical focus than you might expect from academia. There are overviews of real applications by companies, ranging from chemical companies to text summarisation, interspersed with guides to running your first notebooks.

At around 100 pages, it’s short and sweet, serving as a neat introduction for those focused on practical Reinforcement Learning applications.

Theory: 4/10 | Coding: 4/10 | Applications: 8/10 | Accessibility: 8/10 | Coverage: 3/10

9) Keras Reinforcement Learning Projects - Giuseppe Ciaburro (2018)

Printed Version Price: $52.81 | Free online: No | Amazon link

This book is explicitly project-focused. After a brief introduction to Reinforcement Learning theory, the book goes through 9 practical applications of reinforcement learning and their implementation in Python package Keras. The projects range from mechanical modelling to stock market price forecasting and delivery vehicle routing.

The author, Giuseppe Ciaburro, has a background in machine learning engineering. The limitation of the book is that it’s purely focused on Keras, and doesn’t go into as much detail as other books. If this is the package you’d like to use, this might be a great place to start.

Theory: 3/10 | Coding: 6/10 | Applications: 5/10 | Accessibility: 5/10 | Coverage: 2/10

Have we missed a great resource here? If so, get in touch at matthew@joindeltaacademy.com.

If you found this useful, stay up to date with our latest blogs by subscribing to the mailing list. And if you need an extra motivational book to get going with RL that books can’t provide, check out our live courses to learn AI. They are designed from the ground up to be as motivating as possible.