Medical Multimedia Information Systems
Congratulations to Natalia Sokolova, who got her journal paper on “Automatic detection of pupil reactions in cataract surgery videos” accepted in the PLOS ONE journal. This work has been (co-)authored by Natalia Sokolova, Klaus Schoeffmann, Mario Taschwer, Stephanie Sarny, Doris Putzgruber-Adamitsch, and Yosuf El-Shabrawi.
With his master thesis about “Animating Characters using Deep Learning based Pose Estimation”, Fabian Schober won the “Dynatrace Outstanding IT-Thesis Award” (DO*IT*TA). The award brings attention to extraordinary theses, motivates creativity, and provides insight into modern technologies.
In his thesis, Fabian Schober focuses on animating 2D (video game) characters using the PoseNet pose estimation model. He delivers a proof of concept on how new machine learning technologies can assist in video game development. Read more at the University press release (German only).
Our paper about LensID – a CNN-RNN-based framework for irregularity detection in cataract surgery videos has been accepted at the prestigious MICCAI 2021 conference (International Conference on Medical Image Computing & Computer Assisted Intervention). Negin Ghamsarian will present the details of her work on this topic in the end of September.
Authors: Negin Ghamsarian, Mario Taschwer, Doris Putzgruber-Adamitsch, Stephanie Sarny, and Klaus Schoeffmann
Link: https://www.miccai2021.org/en/
Authors: Nakisa Shams (ETS, Montreal, Canada), Hadi Amirpour (Alpen-Adria-Universität Klagenfurt), Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Bitmovin), and Mohammad Ghanbari (School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK)
Abstract: Cognitive radio networks by utilizing the spectrum holes in licensed frequency bands are able to efficiently manage the radio spectrum. A significant improvement in spectrum use can be achieved by giving secondary users access to these spectrum holes. Predicting spectrum holes can save significant energy that is consumed to detect spectrum holes. This is because the secondary users can only select the channels that are predicted to be idle channels. However, collisions can occur either between a primary user and secondary users or among the secondary users themselves. This paper introduces a centralized channel allocation algorithm in a scenario with multiple secondary users to control both primary and secondary collisions. The proposed allocation algorithm, which uses a channel status predictor, provides a good performance with fairness among the secondary users while they have the minimal interference with the primary user. The simulation results show that the probability of a wrong prediction of an idle channel state in a multi-channel system is less than 0.9%. In addition, the channel state prediction saves the sensing energy up to 73%, and the utilization of the spectrum can be improved more than 77%.
Keywords: Cognitive radio, Biological neural networks, Prediction, Idle channel.
Today, Klaus Schöffmann will present his keynote talk on “Deep Video Understanding and the User” at the ACM Multimedia 2020 Grand Challenge (GC) on “Deep Video Understanding”. The talk will highlight user aspects of automatic video content search, based on deep neural networks, and show several examples where users have serious issues in finding the correct content scene, when video search systems rely too much on the “automatic search” scenario and ignore the user behind. Registered users of ACMMM2020 can watch the talk online; the corresponding GC is scheduled for October 14 from 21:00-22:00 (15:00-16:00 NY Time).
Link: https://2020.acmmm.org/
Authors: Natalia Sokolova, Mario Taschwer, Stephanie Sarny, Doris Putzgruber-Adamitsch and Klaus Schoeffmann
Abstract: Automatically detecting clinically relevant events in surgery video recordings is becoming increasingly important for documentary, educational, and scientific purposes in the medical domain. From a medical image analysis perspective, such events need to be treated individually and associated with specific visible objects or regions. In the field of cataract surgery (lens replacement in the human eye), pupil reaction (dilation or restriction) during surgery may lead to complications and hence represents a clinically relevant event. Its detection requires automatic segmentation and measurement of pupil and iris in recorded video frames. In this work, we contribute to research on pupil and iris segmentation methods by (1) providing a dataset of 82 annotated images for training and evaluating suitable machine learning algorithms, and (2) applying the Mask R-CNN algorithm to this problem, which – in contrast to existing techniques for pupil segmentation – predicts free-form pixel-accurate segmentation masks for iris and pupil.
The proposed approach achieves consistent high segmentation accuracies on several metrics while delivering an acceptable prediction efficiency, establishing a promising basis for further segmentation and event detection approaches on eye surgery videos.
Authors: Philipp Moll, Veit Frick, Natascha Rauscher, Mathias Lux (Alpen-Adria-Universität Klagenfurt)
Abstract: The popularity of computer games is remarkably high and is still growing. Despite the popularity and economical impact of games, data-driven research in game design, or to be more precise, in-game mechanics – game elements and rules defining how a game works – is still scarce. As data on user interaction in games is hard to get by, we propose a way to analyze players’ movement and action based on video streams of games. Utilizing this data we formulate four hypotheses focusing on player experience, enjoyment, and interaction patterns, as well as the interrelation thereof. Based on a user study for the popular game Fortnite, we discuss the interrelation between game mechanics, enjoyment of players, and different player skill levels in the observed data.
Keywords: Online Games; Game Mechanics; Game Design; Video Analysis
Links: International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE)
Our Paper “Pixel-Based Tool Segmentation in Cataract Surgery Videos with Mask R-CNN” has been accepted for publication at IEEE 33rd International Symposium on Computer Based Medical Systems (CBMS – http://cbms2020.org).
Authors: Markus Fox, Klaus Schöffmann, Mario Taschwer
Abstract:
Automatically detecting surgical tools in recorded surgery videos is an important building block of further content-based video analysis. In ophthalmology, the results of such methods can support training and teaching of operation techniques and enable investigation of medical research questions on a dataset of recorded surgery videos. While previous methods used frame-based classification techniques to predict the presence of surgical tools — but did not localize them, we apply a recent deep-learning segmentation method (Mask R-CNN) to localize and segment surgical tools used in ophthalmic cataract surgery. We add ground-truth annotations for multi-class instance segmentation to two existing datasets of cataract surgery videos and make resulting datasets publicly available for research purposes. In the absence of comparable results from literature, we tune and evaluate the Mask R-CNN approach on these datasets for instrument segmentation/localization and achieve promising results (61\% mean average precision on 50\% intersection over union for instance segmentation, working even better for bounding box detection or binary segmentation), establishing a reasonable baseline for further research. Moreover, we experiment with common data augmentation techniques and analyze the achieved segmentation performance with respect to each class (instrument), providing evidence for future improvements of this approach.
Acknowledgments:
This work was funded by the FWF Austrian Science Fund under grant P 31486-N31.
The 1-page abstract “Pixel-Based Iris and Pupil Segmentation in Cataract Surgery Videos Using Mask R-CNN” was accepted at the workshop “Deep Learning for Biomedical Image Reconstruction” of the International Symposium on Biomedical Imaging that will take place in Iowa-City, Iowa, USA, 3-7 April.
Authors:
Natalia Sokolova, Mario Taschwer, Klaus Schoeffmann
Acknowledment:
This work was funded by the FWF Austrian Science Fund under grant P 31486-N31
At the 26th International Conference on MultiMedia Modeling (MMM 2020) in Daejeon, Korea, researchers from ITEC have successfully presented several scientific contributions to the multimedia community. First, Natalia Sokolova presented her first paper on “Evaluating the Generalization Performance of Instrument Classification in Cataract Surgery Videos”. Next, Sabrina Kletz presented her work on “Instrument Recognition in Laparoscopy for Technical Skill Assessment”. Finally, Andreas Leibetseder talked about “GLENDA: Gynecologic Laparoscopy Endometriosis Dataset”.
