Reinforcement Learning in AI: A Complete Guide to the Most Human Sort of Machine Learning

There are various combinations and cross-over approaches to these. In general, they can also be subdivided into approaches that are model-based reinforcement learning or model-free reinforcement learning… in other words, whether they will be given or learn a model of the system or are expected to behave according to pure trial and error experimentation.

In all cases, the process is iterative. RL algorithms experiment with different approaches as they go. Each has a balance between exploring new potential solutions that may be better than the current best reward, and optimizing solutions that have already scored high rewards. This is the explore vs exploit trade-off.

Essentially, it’s a kind of gamble: do you stand pat on a 16 in blackjack or hit on it? There are optimal statistical answers to such questions when it comes to RL. Finding those answers is what ML engineers do.

Multi Agent Reinforcement Learning Simulates AI Interaction

If it’s not complicated enough to build up a solid RL model for a single agent, imagine how you would reinforce learning for multiple agents at the same time. That’s multi-agent reinforcement learning, an investigation of how shared environments impact behavior and learning.

MARL doesn’t have a particular kind of utility in training, but offers invaluable insights into how AI will function in real-world applications. That’s because the advent of AI in almost every industry is going to lead to a large number of autonomous AI agents acting in various capacities. How will an AI learning to process invoices interact with another AI running accounts payable? What will both of them do when they tie into a reporting AI trying to develop projections for management purposes?

MARL is how AI researchers will learn to craft complex and complementary behavior through machine learning, rather than setting off a firestorm of conflicts and weirdness.

Inverse Reinforcement Learning Lets Algorithms Find Their Own Rewards

Taking RL a step further, researchers are exploring how to train algorithms without having to specify their reward function first. Instead, they ask the algorithm to make up its own reward.

This is accomplished by having the RL agent observe another agent, or a person, performing a task and by those observations attempting to infer what reward they are pursuing. By modeling the behavior that the example agent provides, the algorithm should be able to understand and identify the goals they are pursuing.

While IRL is still in early days, it offers a potentially powerful way for robust reward functions to be created… which, as you will see in the next section, is critical to the success of RL.

Reinforcement Learning From Human Feedback: Bringing the Personal to Artificial Intelligence

Although reinforcement learning is a kind of magic in that it allows machines to learn on their own, there are still many concepts that are pretty inefficient for AI to figure out on its own. In other cases, engineers may want to train AI on concepts that are still inherently human: humor, for example, is almost impossible to define in a reward function.

Reinforcement learning with human feedback is a way to use RL while incorporating direct human judgement into the reward function.

RLHF can be incorporated at various stages of training.

Pre-training may incorporate specifically created data to prime the process with optimal results.

Direct feedback happens when the model presents results directly to a human trainer, who then corrects or accepts the result as accurate.

Fine-tuning occurs after the bulk of model training has been completed. This is essentially a form of supervised training to reinforce learning specifically on finer points that additional RL might not easily achieve independently.

PPO Reinforcement Learning Helps Keep Algorithms on Track

Proximal Policy Reinforcement Learning puts some guardrails around the RL process to keep algorithms from going too far off the rails. Essentially, it helps offer human constraints that wouldn’t otherwise be present in the RL training process.

PPO is designed to help maximize the steps an RL algorithm can take to improve itself without overcorrecting.

As an RL algorithm experiments with the explore/exploit dichotomy, it has a choice with how dramatically it will alter its next move. Large steps risk overshooting optimal choices, taking longer or leading the algorithm astray. Small steps risk taking too long to find the right balance.

PPO imposes boundaries to keep the algorithm from following its natural inclination where that may result in guesses either too big or too small.

It’s proven so effective that it’s now the default RL algorithm used by OpenAI. Most organizations using RLHF now implement it with PPO as well.

Reinforcement Learning With Human Feedback Has Its Own Drawbacks

RLHF may seem like an ideal combination of training AI, but it does come with some drawbacks.

First, it’s expensive: humans are tremendously slow compared to machines. The time value of their contributions, especially in great quantity, can be quite costly.

Next, human judgement is inconsistent. For example, human evaluators training an AI model on humor will probably deliver a lot of contradictory information just based on different ideas of what’s funny.

Finally, humans bring human biases to the table when they offer feedback. In a sense this is benefit, since it is an accurate reflection of the world they represent and the expectations they set. But it’s also a drawback since some of the worst qualities of human judgement come from built-in biases.

When RLHF draws on human evaluators without sufficient diversity, it can lead to overfitting of the model to the problem. That loses the flexibility and power that AI should bring to the table.

Reinforcement Learning Comes With Trade-Offs and Limitations

Like any tool in the modern AI toolbox, there are trade-offs involved with the reinforcement learning technique.

Some machine learning problems don’t have consistent or clear objectives. It’s tough to define a reward function when you can’t tell the machine going in exactly what it should achieve.

In other cases, it’s absolutely necessary for algorithms to be explainable… they need to be clearly programed and trained from the ground up in ways that humans can understand, rather than training themselves by unknown means.

And sometimes the constellation of potential choices exceeds even the capacity of high-performance computer systems. An RL model may just not be able to explore the potential solution set in any reasonable timeframe.

Deep Reinforcement Learning Requires Careful Thought Around Rewards

Something that researchers found very early on with reinforcement learning is that reward functions have to be very carefully tuned to reflect the desired outcomes. And it can be challenging to do that when computers get creative.

In one famous instance, a researcher experimenting with RL in video games used a boat racing game to test an agent in developing pathfinding and competitive skills. But it turned out that the points model used by the game offered rewards for hitting target buoys along the route. And it also turned out that doing this could garner a higher score more quickly than actually completing the race.

The Paperclip Problem, where AI optimized to make paperclips runs amok turning every tool and source of metal into paperclip fodder, is an example of the result of a poorly designed reward function.

The RL agent simply found an area with repopulating buoys it could smack into repeatedly, occasionally catching fire, and not getting anywhere near the finish. It racked up a high score—but learned nothing the researchers had intended.

More complex learning environments offer even more pitfalls for reward tuning. In scenarios where multiple, even competing, goals exist, the ideal reward function may simply be unattainable.

Reinforcement Learning Can’t Cover Every Possible AI Use Case

Related to the requirement for large amounts of training data is the fact that RL can’t function very well with problems for which such data simply doesn’t exist. Any novel or low-frequency kind of task AI is expected to perform probably can’t be automated with RL techniques.

Reinforcement learning also falls down in situations where a particular action or attempt at solution may not be detectable until much later in the process. For example, modeling environmental issues that are highly dependent on small differences in initial conditions, but which do not play out for hundreds of years, doesn’t provide feedback directly tied to the actions in a useful timeframe.

In other cases, RL is inadvisable due to the stakes of the experiments. In the realm of robotics, for example, using reinforcement learning to teach a robot to pick up eggs will result in an awful lot of broken eggs before it lands on appropriate techniques.

Treating an autonomous vehicle like a Roomba when it comes to reinforcement learning would just create a lot of property damage.

For all that RL doesn’t represent a silver bullet to all machine learning challenges, however, it remains one of the most powerful techniques that AI engineers are using today.

Finding Fixes in Reinforcement Learning Limitations Is Where the Future of AI Is Heading

Yet the entire field of AI revolves around finding solutions to limitations that once seem insurmountable. Reinforcement learning itself was once thought to be a dead-end; only new processing power and the availability of large amounts of data rescued it from obscurity.

Similarly, AI researchers and scientists are finding ways around the limitations. For example, despite the apparent difficulty of using RL in autonomous vehicles, researchers are starting to train driving models in virtual environments where they can crash and burn as often as they like to figure out lane changes before turning them loose in real vehicles.

Other experimental approaches include: