Detecting and summarizing handwritten whiteboard content in lecture videos.
Lecture video is a useful record of a class, and it is hard to search. In a whiteboard lecture the useful signal is the handwriting, the math, and the sketches, and the lecturer keeps walking in front of it.
This line of work, with Bhargava Urala Kota, Kenny Davila, Srirangaraj Setlur, and Venu Govindaraju, detects that handwritten content and turns it into a compact summary. A text detector is adapted to handwriting, formulas, and drawings. Later methods represent the detected regions so that the same content can be tracked while it grows, disappears, or is occluded, and a spatio-temporal conflict criterion picks keyframes or key regions. The 2019 workshop paper on key handwritten content regions received a best paper award.
The experiments use the public AccessMath dataset. We also added timestamped bounding-box annotations for the handwritten content. The papers are listed below, with public PDFs.
Whiteboard lecture videos are summarized by detecting handwritten content regions and comparing their feature representations to find unique content. The method handles occluded, growing, and disappearing handwriting, and produces both key content and keyframes. Weighted spatio-temporal conflict minimization is used to segment the lecture.
@inproceedings{kota2021automated,title={Automated Whiteboard Lecture Video Summarization by Content Region Detection and Representation},author={Kota, Bhargava Urala and Stone, Alexander and Davila, Kenny and Setlur, Srirangaraj and Govindaraju, Venu},booktitle={2020 25th International Conference on Pattern Recognition (ICPR)},pages={10704--10711},year={2021},organization={IEEE},doi={10.1109/ICPR48806.2021.9412386},}
A framework for extracting and binarizing handwritten content in fixed-camera lecture videos, so the content can be used to index and navigate the video. A deep learning pipeline detects handwritten text, formulae, and sketches, and the spatio-temporal structure of those detections is used to segment the lecture. On the AccessMath dataset the framework reaches an F-measure of 94.32 percent for binary connected components.
@article{urala2019generalized,title={Generalized Framework for Summarization of Fixed-Camera Lecture Videos by Detecting and Binarizing Handwritten Content},author={Urala Kota, Bhargava and Davila, Kenny and Stone, Alexander and Setlur, Srirangaraj and Govindaraju, Venu},journal={International Journal on Document Analysis and Recognition (IJDAR)},volume={22},number={3},pages={221--233},year={2019},publisher={Springer},doi={10.1007/s10032-019-00327-y},}
Whiteboard lecture videos are summarized by key handwritten content regions. A detector finds semantically meaningful groups of handwriting, and a triplet-loss embedding distinguishes unique content across frames. A spatiotemporal index maps each summary region back to where it appears in the video. On AccessMath the method recalls 92.09 percent of unique ground-truth objects.
@inproceedings{kota2019summarizing,title={Summarizing Lecture Videos by Key Handwritten Content Regions},author={Kota, Bhargava Urala and Ahmed, Saleem and Stone, Alexander and Davila, Kenny and Setlur, Srirangaraj and Govindaraju, Venu},booktitle={2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)},volume={4},pages={13--18},year={2019},organization={IEEE},doi={10.1109/ICDARW.2019.30058},note={Best paper award}}
A scene-text detector is adapted to find handwritten text, math, and sketches on a whiteboard, including content occluded by the lecturer. The detections are used to summarize the lecture. The work also released timestamped bounding-box annotations for the AccessMath dataset.
@inproceedings{kota2018automated,title={Automated Detection of Handwritten Whiteboard Content in Lecture Videos for Summarization},author={Kota, Bhargava Urala and Davila, Kenny and Stone, Alexander and Setlur, Srirangaraj and Govindaraju, Venu},booktitle={2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR)},pages={19--24},year={2018},organization={IEEE},doi={10.1109/ICFHR-2018.2018.00013},}