Researchers at Gwangju Institute of Science and Technology are developing a smart observer for esports

It uses an object detection algorithm that learns human viewer data to find responsive viewports

GWANGJU, South Korea, November 25, 2022 /PRNewswire/ — Human match observers are an important part of the esports industry. They use extensive domain knowledge to decide what to show viewers. However, they can miss important events, necessitating the need for automatic observers. researchers out South Korea recently proposed a framework that uses an object detection method, Mask R-CNN, and human observation data to find the “Region of Common Interest” in StarCraft – a real-time strategy game.

Esports, already a billion-dollar industry, is growing, in part because of human game observers. You control the camera movement and show viewers the most captivating parts of the game screen. However, these viewers could miss important events happening simultaneously on multiple screens. They are also difficult to achieve in small tournaments. Consequently, the demand for automatic observers has grown. Artificial observation methods can be either rule-based or learning-based. Both predefine events and their importance, which requires extensive domain knowledge. In addition, they cannot capture undefined events or detect changes in the meaning of the events.

Recently, researchers have made South Koreaunder the direction of dr. Kyung Jong KimAssociate Professor at the Gwangju Institute of Science and Technology, have proposed an approach to overcome these problems. “We created an automatic observer using the Mask R-CNN object detection algorithm to learn human viewer data,” explains Dr. Kim. Their results were made available online at October 10, 2022 and published in Volume 213 Part B of Expert systems with applications Diary.

The novelty lies in defining the object as the two-dimensional space viewed by the viewer. In contrast, conventional object detection treats a single entity, such as a worker or a building, as the object. In this study, researchers first collected human observation data of in-game StarCraft from 25 participants. Next, the viewports—areas viewed by the viewer—were identified and labeled “one.” The rest of the screen was filled with “zeros”. While the game features are used as input data, the human observations formed the target information.

The researchers then fed the data into the convolution neural network (CNN), which learned the patterns of the viewports to find the region of common interest (ROCI) — the most engaging area for viewers. They then quantitatively and qualitatively compared the ROCI Mask R-CNN approach to other existing methods. The previous analysis showed that CNN’s predicted viewports were similar to the human observation data collected. In addition, the ROCI-based method outperformed others over the long run during generalization testing, which included different matchup races, starting locations, and playing maps. The proposed observer was able to capture the scenes of interest to humans. In contrast, this could not be achieved through behavior cloning – a technique of imitation learning.

dr Kim hints at the future applications of her work. “The framework can be applied to other games that represent part of the overall game state, not just StarCraft. As services like multi-screen broadcasting continue to grow in esports, the proposed automatic observer will play a part in these outcomes. It will also be actively used in other content to be developed in the future.”


Original Paper Title: Learning to Automatically Observe Matches for Esports Using Object Detection Mechanism

Journal: Expert systems with applications

*E-mail of the corresponding author: [email protected]

About Gwangju Institute of Science and Technology (GIST)

Chang Sung Kang
82 62 715 6253
[email protected]



View original content to download multimedia: -esports-301686149.html

SOURCE Gwangju Institute of Science and Technology


Leave a Reply

Your email address will not be published. Required fields are marked *