Emerging online video-sharing websites such as YouTube allow users to access a huge number of videos and generate feedback via reviews and/or live comments (a.k.a. danmu), making it possible to summarize videos based on media and user responses collectively. A video summary produced by existing techniques may not fully capture an audience’s perception of and reaction to the source video, and thus may be less reflective. In this paper, we introduce VideoForest, a visualization system designed to convert an input video augmented with danmu commentary data into a treelike visual summary of content highlights under user supervision. The proposed visualization design employs a forest metaphor. The overall summary of different video sessions is illustrated as scene trees on top of the session timeline ground, with the roots depicting the corresponding danmu messages. VideoForest can also generate a detailed synopsis of user-selected video segment(s) as a compact storyline in the form of circle packing. We evaluate our system via case studies with real video data based on experts’ feedback. The results suggest the power and potential of the system.