There are two core components in any agent's architecture.

1. State Encoder: Encodes game information.
2. Action Scorer: Uses ended information to evaluate the actions taken in each state.

Recent architectural trends:
Papers in 2015 used LSTM, Bag of Words based approach for encoders and Deep Q-Network (DQN) for action selector. Recent approaches uses Gated Recurrent Unit’s for encoding and Actor-critic for action selectors. Some others works also used Knowledge Graphs and Pre-trained Transformers.

Below Table provides and some of the recent architectural trends.

Syracuse University

- The Possible Limitations in Text Game RL Problems
- Challenges and Proposed Solutions
- Benchmark Environments and Agents
- Agent Architectures
- Benchmark Results

Reinforcement Learning in Text Games

A Survey of Text Games for Reinforcement Learning informed by Natural Language

Environment Model: As in a text game, textual feedback is provided, the Decision Process for the Agent is defined as Discrete-Time Partially Observed Markov Decision Process.

Some possible limitations of generation tools to reduce the scope of challenges faced

Partial observability - Controlled by improving agents observations.

Complexity of the language - Can be reduced by providing in-game vocabulary to the agent and using symbolic tokens.

Rewards - Simplifies by rewarding agent based on state transitions and ground truth winning policy.

Action - Using word-level and syntax commands to reduce the complexity of actions to be taken

Possible Limitations in Text Game RL Problems

Partial Observability: When the agents observe the information which only represents a small part of the underlying truth. Requires extensive exploration and failure to understand the connection between the current observations and related actions. Additional challenge is when the agent changes to next state without completing pre-requisite of future states. 

Eg: Agent has use lantern to light its way but prior to this it needs to check if it has acquired the lantern in the previous state or not.

Possible Solution - Handcrafted Reward functions, using combination of a patience parameter and intrinsic motivation for new knowledge, dynamically learned knowledge graph

Large State Space: RL Problems requires exploration to find better solution however this can cause Natural Overfitting. For instance, setting reward signal for encountering new state removes the capability of learning new knowledge of the environment in favor of simply searching for unseen states. 

Eg: Treasure Hunter - used increase in complexity of the environment with obstacles to deal with this problem.

Normally, the solutions are related to the general problem of RL - Exploration vs Exploitation

Some additional problems include
Large, Combinatorial and Sparse Actions Spaces
Long-Term Credit Assignment
Understanding Parser Feedback & Language Acquisition
Commonsense Reasoning & Affordance Extraction
Knowledge Representation

The Challenges and Possible Solutions

Due to environments being independently generated by researchers, some uniformity in evaluation is needed for the agents in specified environment conditions.

Types of evaluation settings

Single Game: Same game with same conditions

Joint: Same set of games with some similar states

Zero-shot: Totally unseen in training

Below is the table of Environments with different properties published within past two years.

Benchmark Environments and Agents

Agent Architectures

Treasure Hunter: It is s classic game that requires to navigate a maze to find the specified object. 

In a zero-shot evaluation setting, 2 state-of-the-art agents BYU and Golovin are compared to choice-based random agent.

Learn Before

Related