How can agents learn from experience without an omniscient teacher explicitly telling them what to do? Reinforcement learning is the area within machine learning that investigates how an agent can learn an optimal behavior by correlating generic reward signals with its past actions. The discipline draws upon and connects key ideas from behavioral psychology, economics, control theory, operations research, and other disparate fields to model the learning process. In reinforcement learning, the environment is typically modeled as a does not have access to the transition structure of the environment and needs to learn how to choose appropriate actions to maximize its overall reward over time. This book by Prof. Masashi Sugiyama covers the range of reinforcement learning algorithms from a fresh, modern perspective. With a focus on the statistical properties of estimating parameters for reinforcement learning, the book relates a number of different approaches across the gamut of learning scenarios. The algorithms are divided into model-free approaches that do not explicitly model the dynamics of the environment, and model-based approaches that construct descriptive process models for the environment. Within each of these categories, there are policy iteration algorithms which estimate value functions, and policy search algorithms which directly manipulate policy parameters. optimization problems. A careful analysis is given for each of these cases, with an emphasis on understanding the statistical properties of the resulting estimators and learned parameters. Each chapter contains illustrative examples of applications of these algorithms, with quantitative comparisons between the different techniques. These examples are drawn from a variety of practical problems, including robot motion control and Asian brush painting.

In summary, the book provides a thought provoking statistical treatment of reinforcement learning algorithms, reflecting the author’s work and sustained research in this area. It is a contemporary and welcome addition to the rapidly growing machine learning literature. Both beginner students and experienced researchers will find it to be an important source for understanding the latest reinforcement learning techniques.