Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: lots of tweak to JOSS paper #241

Merged
merged 1 commit into from
Dec 14, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions paper/paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,24 @@ bibliography: paper.bib

# Summary

OpenTTD [@openttdteam2004openttd] is a business simulation game originally created for recreation, where one or more human players build companies through constructing and using a transportation network to transport passengers and goods. OpenTTD can be extended by allowing autonomous so-called AI players, and it is by leveraging this capability, allowing researchers to run experiments with their own AIs, that OpenTTDLab converts OpenTTD from a game into a system for researching algorithms and their effects on companies and supply chains, and helps to ensure the results of such research are reproducible.
OpenTTD [@openttdteam2004openttd] is a business simulation game created for recreational play, where one or more human players build companies by constructing a transportation network for use in moving passengers and goods. OpenTTD can be extended by allowing autonomous so-called AI players, and OpenTTDLab leverages this capability to allow researchers to run experiments with their own AIs. Doing so converts OpenTTD from a game into a system for researching algorithms and their effects on companies and supply chains, and helps to ensure the results of such research are reproducible.

![The OpenTTDLab logo. Adapted from the OpenTTD Logo, © OpenTTD Team, licenced under the GNU General Public License Version 2.](openttdlab-logo.pdf){height="100pt"}

# Statement of need

OpenTTD is remarkably flexible: by leveraging the OpenTTD AI system it has been successfully used as a tool to research algorithms including artificial intelligence (AI), machine learning (ML), and anomaly detection algorithms [@beuneker2019autonomous; @bijlsma2014evolving; @konijnendijk2015mcts; @lakomy2020railroad; @rios2009trains; @wisniewski2011artificial; @volna2017fuzzy]. However, OpenTTD does not include the capability to easily repeat experiments over ranges of configuration--unsurprising since it is designed to be a game--and this results in much of the existing research not being able to be reproduced [@charemza2024reusable, chap. 1]. OpenTTD addresses this by providing a framework through which experiments similar to those of the existing literature can be easily run by writing and running Python code, and encourages such code to be shared so future researchers can easily reproduce results.
OpenTTD is remarkably flexible: by leveraging the OpenTTD AI system it has been successfully used as a tool to research algorithms including artificial intelligence (AI), machine learning (ML), and anomaly detection algorithms [@rios2009trains; @wisniewski2011artificial; @bijlsma2014evolving; @konijnendijk2015mcts; @volna2017fuzzy; @beuneker2019autonomous; @lakomy2020railroad]. However, OpenTTD does not include the capability to easily repeat experiments over ranges of configuration--unsurprising given it was created for recreational play--and this results in much of the existing research not being reproducible [@charemza2024reusable, chap. 1]. OpenTTDLab addresses this by providing a framework through which experiments similar to those in the literature can be easily run via writing and running Python code, and encourages the code to be shared so future researchers can easily reproduce results.

There appears to be no open source framework that takes a game designed for recreation and converts it into a reusable simulator for reproducible research, and in this respect OpenTTDLab is novel. The closest equivalent found is Axelrod [@knight2016open], a Python framework that research into the Iterated Prisoner's dilema, allowing researchers to investigate existing and new strategies of playing the game. Similar to OpenTTDLab, Axelrod was created in response to existing research being found to not be reproducible.
There are no other similar frameworks that take a game created for recreational play and convert it into a reusable simulator for reproducible research, and in this respect OpenTTDLab is novel. The closest equivalent is Axelrod [@knight2016open], a Python framework that researches the Iterated Prisoner's Dilemma, allowing investigations of existing and new gameplay strategies. Similar to OpenTTDLab, Axelrod was created in response to difficulties in reproducing existing research.

# Usage and results

Through a single Python function call, `run_experiments`, OpenTTDLab runs OpenTTD over a range of configurations, most typically over a range of random seeds and range of parameters of an AI; it then returns data from every month of in-game time, which can be processed and visualised. \autoref{fig:example-results} shows example results: comparing how the distributions of money in the bank change over time for two configurations of a simple AI that builds a single bus route with a configurable number of buses.
Through a single Python function call, `run_experiments`, OpenTTDLab runs OpenTTD over a range of configurations, most often over a range of random seeds and a range of parameters of an AI. It then returns data from every month of in-game time, which can be processed and visualised. \autoref{fig:example-results} shows example results: comparing how the distributions of money in the bank change over time for two configurations of a simple AI that builds a single bus route with a configurable number of buses. The differences in the distributions show that OpenTTDLab can be used to investigate risk/benefit trades offs in supply chains.

![How the distribution of money in the bank changes over time for a simple AI [@charemza2024parameterised]. The results of 100 runs of OpenTTD are shown, 50 when the AI is configured to build 1 bus, and 50 for when it is configured to build 16 buses. The results are generated by a single call to OpenTTDLab's `run_experiments` function. Image adapted from [@charemza2024reusable, chap. 5].\label{fig:example-results}](example-results-charemza2004reproducible){height="150pt"}

# Acknowledgments

OpenTTDLab could not have been created without incredible work of all the [contributors to OpenTTD, its dependencies, and precursors](https://github.com/OpenTTD/OpenTTD/blob/master/CREDITS.md); thank you to Patric Stout (A.K.A. TrueBrain), who originally wrote the OpenTTD savegame parser that OpenTTDLab forked from; and finally, thank you to Michael Herrmann, whose advice from his supervision of the dissertation that resulted in the original versions of OpenTTDLab [@charemza2024reusable] was invaluable.
OpenTTDLab could not have been created without incredible work of all the [contributors to OpenTTD, its dependencies, and precursors](https://github.com/OpenTTD/OpenTTD/blob/master/CREDITS.md). A further thank you to Patric Stout (A.K.A. TrueBrain), who originally wrote the OpenTTD savegame parser that OpenTTDLab forked from. Finally, thank you to Michael Herrmann, whose invaluable supervision of the original dissertation resulted in the first versions of OpenTTDLab [@charemza2024reusable].

# References
Loading