Reinforced Learning

Reinforcement learning is a goal-oriented algorithm. Let’s describe how it works briefly. The machine has an explicit goal, for example, a checkmate in chess. Also, we set the rules of the game (pieces’ movements, castling, and so on). Then, the machine has to choose actions to win the game. During learning, a computer explores an environment (in our case, a chessboard with pieces) and tries different actions to influence the environment (pieces’ movements). Each action of the machine produces a certain reward - for example, machines receive a high reward for capturing an opponent’s piece and a small reward for the regular piece’s movement. In this way, the machine learns the strategy that dictates the best movements for the current situation. So, the machine interacts with the environment and observes the result of its actions, learns from it and changes its behavior in response to the rewards received. We can say that a reinforced learning machine learns from its mistakes.

Reinforced learning has roots in behavioral psychology. This kind of learning is very close to the way in which humans and animals learn.

In comparison to other types of learning paradigms, reinforcement learning takes place in between supervised and unsupervised learning. In reinforced learning, the machine doesn’t know which actions are correct or incorrect (as is the case in supervised learning) but it “knows” whether it does a good job or not due to the resulting reward. Also, reinforced learning is different from unsupervised learning, since it is not trying to find hidden patterns, but rather trying to maximize a reward instead. So, it is a kind of semi-supervised method.

Let’s identify the main elements of a reinforced learning system.

The central elements consist of: an agent that takes actions and learns (e.g., robot, computer) and an environment (physical or virtual world). Sometimes, instead of an environment, there may be a model which mimics the environment's behavior.

The other elements of reinforcement learning include a policy, a reward, and a value function.

A policy is a strategy that an agent learns from the environment during interaction with it. The optimal policy provides the actions of an agent that promise the highest reward.

A reward is the feedback for an agent after each action. An agent tries to increase the reward at each step. There are also negative rewards (penalty) in some models when the received feedback aims to minimize unwanted actions.

A value function is an expected long-term return.

Reinforced learning is a very difficult task that requires the resolution of multiple problems:

1. The first problem is that the agent receives only the reward as a learning signal. The agent must find the best policy based on trial-and-error interactions with the environment that is based only on this feedback.
2. The second is the presence of strong temporal correlations, which appear when the agent bases the observations on its own actions.
3. Third, the agent needs many actions to realize whether its strategy was good or not. For example, in the case of indoor robotics navigation, a robot can overcome long distances to reach its goal and then reach a dead end.

Many algorithms were developed to solve these specific problems. All reinforced algorithms can be divided into two main types: approaches based on value function and those based on policy search. Among them are Monte Carlo methods, Q-learning, Deep reinforcement learning, and many other algorithms.

Software that uses reinforced learning is the next level in building autonomous systems. Some of the algorithms based on reinforced learning have been already applied in robotics, video games, and navigation. At the end of this article, we have given examples of applied reinforced learning.

Robotics. The idea of using reinforced learning in robotics is ambitious and complex. A robot reads raw video images from its camera. Then it uses a deep neural network for processing. The outputs in this case are the motor torques. Thus, the robot learns the policy from video images and maps these images to actions.

Games. Reinforced algorithms are used to solve games. The biggest success was achieved in Go. Algorithms AlphaGo and AlphaGo Zero, which are based on value network and Monte Carlo tree search, and have already achieved human performance.