2
\$\begingroup\$

I'm currently familiarizing myself with reinforcement learning (RL). For convenience, instead of manually entering coordinates in the terminal, I created a very simple UI for testing trained agents and play games against them. You can experiment and play around with it using different hyperparameters but as this is a very basic reinforcement learning algorithm I did not get good results for boards larger than the standard tic tac toe size (3 x 3).

For using it just run python3 ui.py the agent will be trained and then the UI will be loaded and you can play against the trained agent.

How to improve efficiency / memory requirements, they are negligible but an improvement does not hurt since memory scores are stored for updates. Can I to improve agent accuracy / ability to, win games / prevent human from winning specially on larger boards? And of course general feedback is appreciated. And as I'm currently learning RL concepts, I'm aware that there is a lot more to learn and sooner or later I might find answers to the RL technical questions I have however I'm always open for feedback.

Here's what the UI looks like:

enter image description here

helper.py

import numpy as np


class TicBase:
    """
    Base class with useful utils/helpers.
    """

    def __init__(self, board_size=3, empty_value=0, x_value=2, o_value=3):
        """
        Initialize common settings.
        Args:
            board_size: int, board cell x cell size.
            empty_value: int, representation of available cells.
            x_value: int, representation(internal not for display) of X on the board.
            o_value: int, representation(internal not for display) of O on the board.
        """
        self.board_size = board_size
        self.board = np.ones((board_size, board_size)) * empty_value
        self.empty_value = empty_value
        self.x_value = x_value
        self.o_value = o_value

    def get_empty_cells(self, board=None):
        """
        Get empty locations that are available.
        Args:
            board: numpy array of shape n, n where n: self.board_size.

        Returns:
            A list of (x, y) empty indices of the board.
        """
        board = board if board is not None else self.board
        empty_locations = np.where(board == self.empty_value)
        return list(zip(*empty_locations))

    def check_game_ended(self, board):
        """
        Check for empty cells.
        Args:
            board: numpy array of shape n, n where n: self.board_size.

        Returns:
            True if there are empty cells, False otherwise.
        """
        return not np.any(board == self.empty_value)

    def check_win(self, symbol, board=None):
        """
        Check if there is a winner.
        Args:
            symbol: int, representation of the player on the board.
            board: numpy array of shape n, n where n: self.board_size.

        Returns:
            symbol if symbol is winner else None.
        """
        board = board if board is not None else self.board
        result = (
            np.all(board == symbol, axis=0).any()
            or np.all(board == symbol, axis=1).any()
            or np.all(board.diagonal() == symbol)
            or np.all(board[::-1].diagonal() == symbol)
        )
        if result:
            return symbol

    def reset_game(self, symbol=None):
        """
        Reset board.
        Args:
            symbol: int, representation of the winning player on the board.

        Returns:
            None
        """
        self.board = (
            np.ones((self.board_size, self.board_size)) * self.empty_value
        )

agent.py

from helper import TicBase
import numpy as np


class TicAgent(TicBase):
    """
    Tic Tac Toe agent.
    """

    def __init__(
        self, agent_settings, board_size=3, empty_value=0, x_value=2, o_value=3
    ):
        """
        Initialize agent settings.
        Args:
            agent_settings: Dictionary that has the following keys:
                - 'epsilon': float, value from 0 to 1 representing degree
                    of random(exploratory) moves.
                - 'learning_rate': float, Learning rate used to update values.
                - 'symbol': int, representation of the agent on the board.
            board_size: int, board cell x cell size.
            empty_value: int, representation of available cells.
            x_value: int, representation(internal not for display) of X on the board.
            o_value: int, representation(internal not for display) of O on the board.
        """
        TicBase.__init__(self, board_size, empty_value, x_value, o_value)
        self.symbol = agent_settings['symbol']
        self.opponent_symbol = x_value if self.symbol == o_value else o_value
        self.learning_rate = agent_settings['learning_rate']
        self.epsilon = agent_settings['epsilon']
        self.game_scores = {}

    @staticmethod
    def get_game_id(board):
        """
        Convert game to string representation for storing game scores.
        Args:
            board: numpy array of shape n, n where n: self.board_size.

        Returns:
            string representation of the game.
        """
        return ''.join(board.reshape(-1).astype(int).astype(str))

    def calculate_score(self, board):
        """
        Calculate game score which depends on the winner/loser.
        Args:
            board: numpy array of shape n, n where n: self.board_size.

        Returns:
            score.
        """
        game_id = self.get_game_id(board)
        score = self.game_scores.get(game_id)
        if score is not None:
            return score
        score = 0
        winner = self.check_win(self.symbol, board) or (
            self.check_win(self.opponent_symbol, board)
        )
        if winner == self.symbol:
            score = 1
        if not winner and not self.check_game_ended(board):
            score = 0.5
        self.game_scores[game_id] = score
        return score

    def generate_move(self, board, empty_locations, learning=True):
        """
        Generate a random/best value move according to self.epsilon.
        Args:
            board: numpy array of shape n, n where n: self.board_size which
                represents the game most-recent state.
            empty_locations: A list of (x, y) empty indices of the board.
            learning: If True, exploration is allowed.

        Returns:
            Chosen empty_locations index.
        """
        probability = np.random.rand()
        if probability < self.epsilon and learning:
            return np.random.choice(len(empty_locations))
        possible_moves = [board.copy() for _ in empty_locations]
        for idx, empty_location in enumerate(empty_locations):
            r, c = empty_location
            possible_moves[idx][r, c] = self.symbol
        scores = [
            self.calculate_score(possible) for possible in possible_moves
        ]
        return np.argmax(scores)

    def step(self, board):
        """
        Take turn.
        Args:
            board: numpy array of shape n, n where n: self.board_size which
                represents the game most-recent state.

        Returns:
            None
        """
        empty_locations = self.get_empty_cells(board)
        move_idx = self.generate_move(board, empty_locations)
        r, c = empty_locations[move_idx]
        board[r, c] = self.symbol

    def update_game_scores(self, game_history, winner):
        """
        Update agent weights.
        Args:
            game_history: A list of numpy arrays(game boards) of 1 full game.
            winner: int representation of agent/opponent.

        Returns:
            None
        """
        target = 1 if winner == self.symbol else 0
        for game in reversed(game_history):
            game_id = self.get_game_id(game)
            old_score = self.calculate_score(game)
            updated_score = old_score + self.learning_rate * (
                target - old_score
            )
            self.game_scores[game_id] = updated_score
            target = updated_score

trainer.py

from helper import TicBase
from agent import TicAgent


class TicTrainer(TicBase):
    """
    Tool for agent training.
    """

    def __init__(
        self, agent_settings, board_size=3, empty_value=0, x_value=2, o_value=3
    ):
        """
        Initialize agents for training.
        Args:
            agent_settings: A list of 2 dictionaries for 2 agents.
            board_size: int, board cell x cell size.
            empty_value: int, representation of available cells.
            x_value: int, representation(internal not for display) of X on the board.
            o_value: int, representation(internal not for display) of O on the board.
        """
        TicBase.__init__(self, board_size, empty_value, x_value, o_value)
        self.agent_1 = TicAgent(
            agent_settings[0], board_size, empty_value, x_value, o_value
        )
        self.agent_2 = TicAgent(
            agent_settings[1], board_size, empty_value, x_value, o_value
        )

    def play_one(self, game_history):
        """
        Play one game.
        Args:
            game_history: A list of numpy arrays(game boards) of 1 full game.

        Returns:
            symbol or 'end' or None.
        """
        board = self.board
        for agent in [self.agent_1, self.agent_2]:
            agent.step(board)
            game_history.append(board.copy())
            if self.check_win(agent.symbol, board):
                return agent.symbol
            if self.check_game_ended(board):
                return 'end'

    def train_step(self):
        """
        1 training step/iteration and update.
        Returns:
            None
        """
        game_history = []
        while True:
            winner = self.play_one(game_history)
            if winner in ['end', self.x_value, self.o_value]:
                break
        self.agent_1.update_game_scores(game_history, winner)
        self.agent_2.update_game_scores(game_history, winner)
        self.reset_game(winner)

    def train_basic(self, iterations):
        """
        Train using the most basic value function.
        Args:
            iterations: Training loop size.

        Returns:
            self.agent_1, self.agent2
        """
        for i in range(iterations):
            percent = round((i + 1) / iterations * 100, 2)
            print(
                f'\r Training agent ... iteration: '
                f'{i}/{iterations} - {percent}% completed',
                end='',
            )
            self.train_step()
        return self.agent_1, self.agent_2

ui.py

from PyQt5.QtWidgets import (
    QMainWindow,
    QDesktopWidget,
    QApplication,
    QWidget,
    QHBoxLayout,
    QVBoxLayout,
    QPushButton,
    QLabel,
)
from PyQt5.QtCore import Qt
import numpy as np
import sys
from helper import TicBase
from trainer import TicTrainer


class TicCell(QPushButton):
    """
    Tic Tac Toe cell.
    """

    def __init__(self, location):
        """
        Initialize cell location.
        Args:
            location: Tuple, (row, col).
        """
        super(QPushButton, self).__init__()
        self.location = location


class TicUI(QMainWindow, TicBase):
    """
    Tic Tac Toe interface.
    """

    def __init__(
        self,
        window_title='Smart Tic Tac Toe',
        board_size=3,
        empty_value=0,
        x_value=2,
        o_value=3,
        agent=None,
    ):
        """
        Initialize game settings.
        Args:
            window_title: Display window name.
            board_size: int, the board will be of size(board_size, board_size).
            empty_value: int representation of an empty cell.
            x_value: int representation of a cell containing X.
            o_value: int representation of a cell containing O.
            agent: Trained TicAgent object.
        """
        super(QMainWindow, self).__init__()
        TicBase.__init__(
            self,
            board_size=board_size,
            empty_value=empty_value,
            x_value=x_value,
            o_value=o_value,
        )
        self.setWindowTitle(window_title)
        self.agent = agent
        if self.agent is not None:
            self.agent.board = self.board
        self.text_map = {
            x_value: 'X',
            o_value: 'O',
            empty_value: '',
            'X': x_value,
            'O': o_value,
            '': empty_value,
        }
        win_rectangle = self.frameGeometry()
        center_point = QDesktopWidget().availableGeometry().center()
        win_rectangle.moveCenter(center_point)
        self.central_widget = QWidget(self)
        self.main_layout = QVBoxLayout()
        self.score_layout = QHBoxLayout()
        self.human_score = 0
        self.agent_score = 0
        self.score_board = QLabel()
        self.update_score_board()
        self.setStyleSheet('QPushButton:!hover {color: yellow}')
        self.cells = [
            [TicCell((c, r)) for r in range(board_size)]
            for c in range(board_size)
        ]
        self.cell_layouts = [QHBoxLayout() for _ in self.cells]
        self.adjust_layouts()
        self.adjust_cells()
        self.update_cell_values()
        self.show()

    def adjust_layouts(self):
        """
        Adjust score board and cell layouts.

        Returns:
            None
        """
        self.main_layout.addLayout(self.score_layout)
        for cell_layout in self.cell_layouts:
            self.main_layout.addLayout(cell_layout)
        self.central_widget.setLayout(self.main_layout)
        self.setCentralWidget(self.central_widget)

    def adjust_cells(self):
        """
        Adjust display cells.

        Returns:
            None
        """
        self.score_layout.addWidget(self.score_board)
        self.score_board.setAlignment(Qt.AlignCenter)
        for row_index, row in enumerate(self.cells):
            for cell in row:
                cell.setFixedSize(50, 50)
                cell.clicked.connect(self.game_step)
                self.cell_layouts[row_index].addWidget(cell)

    def get_empty_cells(self, board=None):
        """
        Get empty cell locations.

        Returns:
            A list of indices that represent currently empty cells.
        """
        empty_locations = super().get_empty_cells()
        for empty_location in empty_locations:
            r, c = empty_location
            cell_text = self.cells[r][c].text()
            assert cell_text == self.text_map[self.empty_value], (
                f'location {empty_location} has a cell value of {cell_text}'
                f'and board value of {self.board[r][c]}'
            )
        return empty_locations

    def update_score_board(self):
        """
        Update the display scores.

        Returns:
            None
        """
        self.score_board.setText(
            f'Human {self.human_score} - ' f'{self.agent_score} Agent'
        )

    def reset_cell_colors(self):
        """
        Reset display cell text colors.

        Returns:
            None
        """
        for row_idx in range(self.board_size):
            for col_idx in range(self.board_size):
                self.cells[row_idx][col_idx].setStyleSheet('color: yellow')

    def reset_game(self, symbol=None):
        """
        Reset board and display cells and update display scores.
        Args:
            symbol: int, self.x_value or self.o_value.

        Returns:
            None
        """
        super().reset_game(symbol)
        self.update_cell_values()
        if symbol == self.x_value:
            self.human_score += 1
        if symbol == self.o_value:
            self.agent_score += 1
        self.update_score_board()
        self.reset_cell_colors()

    def modify_step(self, cell_location, value):
        """
        Modify board and display cells.
        Args:
            cell_location: tuple, representing indices(row, col).
            value: int, self.x_value or self.o_value.

        Returns:
            True if the clicked cell is not empty, None otherwise.
        """
        r, c = cell_location
        board_value = self.board[r, c]
        cell_value = self.cells[r][c].text()
        if not board_value == self.empty_value:
            return True
        assert cell_value == self.text_map[self.empty_value], (
            f'mismatch between board value({board_value}) '
            f'and cell value({cell_value}) for location {(r, c)}'
        )
        if value == self.x_value:
            self.cells[r][c].setStyleSheet('color: red')
        self.board[r, c] = value
        self.cells[r][c].setText(f'{self.text_map[value]}')

    def game_step(self):
        """
        Post cell-click step(human step and agent step)

        Returns:
            None
        """
        cell = self.sender()
        stop = self.modify_step(cell.location, self.x_value)
        if stop:
            return
        x_win = self.check_win(self.x_value)
        if x_win is not None:
            self.reset_game(self.x_value)
            return
        empty_locations = self.get_empty_cells()
        if not empty_locations:
            self.reset_game()
            return
        choice = np.random.choice(len(empty_locations))
        if self.agent:
            choice = self.agent.generate_move(
                self.board, empty_locations, False
            )
        self.modify_step(empty_locations[choice], self.o_value)
        o_win = self.check_win(self.o_value)
        if o_win is not None:
            self.reset_game(self.o_value)

    def update_cell_values(self):
        """
        Sync display cells with self.board

        Returns:
            None
        """
        for row_index, row in enumerate(self.board):
            for col_index, col in enumerate(row):
                update_value = self.text_map[self.board[row_index][col_index]]
                self.cells[row_index][col_index].setText(f'{update_value}')


if __name__ == '__main__':
    s = [
        {'symbol': 2, 'epsilon': 0.1, 'learning_rate': 0.5},
        {'symbol': 3, 'epsilon': 0.1, 'learning_rate': 0.5},
    ]
    train = TicTrainer(s)
    a1, a2 = train.train_basic(10000)
    test = QApplication(sys.argv)
    window = TicUI(agent=a2)
    sys.exit(test.exec_())
\$\endgroup\$
0
2
\$\begingroup\$

Cell value representation

It's good that you've distinguished this:

representation(internal not for display)

however, I'm not clear on the reason for making these integers configurable. You're probably better off making an Enum:

class TicValue(Enum):
    EMPTY = 0
    X = 2
    O = 3

This can still be compatible with Numpy, because .value will return those integers.

Truthy coalesce

board = board if board is not None else self.board

has shorthand:

board = board or self.board

as long as board doesn't have any funny business overriding __bool__ or __len__.

Boolean equivalent cast

target = 1 if winner == self.symbol else 0

can be

target = int(winner == self.symbol)

Unpacking

    self.agent_1 = TicAgent(
        agent_settings[0], board_size, empty_value, x_value, o_value
    )
    self.agent_2 = TicAgent(
        agent_settings[1], board_size, empty_value, x_value, o_value
    )

can be

self.agent_1, self.agent_2 = (
    TicAgent(settings, board_size, empty_value, x_value, o_value)
    for settings in agent_settings
)

In-band signalling

    Returns:
        symbol or 'end' or None.

is not very convenient for other functions to process. Consider splitting this apart into

def play_one(self, game_history: List[ndarray]) -> (
    bool,            # is end
    Optional[str],   # symbol or None
):
    # ...
    return False, agent.symbol
    # ...
    return True, None

Display map

    self.text_map = {
        x_value: 'X',
        o_value: 'O',
        empty_value: '',
        'X': x_value,
        'O': o_value,
        '': empty_value,
    }

is a little surprising. You're using this as a map in both directions. It would be safer, and a better guarantee of correctness, for this to be split into forward- and back-maps.

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Ah, so my warning about overriding __len__ turns out to be applicable. Adding type hints to your function signatures and member variables will make it clear which variables can receive this kind of treatment. \$\endgroup\$
    – Reinderien
    Jun 29 '20 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.