Multimodal Summarization of Complex Sentences

This paper introduces ROCMMS, a system that automatically converts existing text to multimodal summaries (MMS) that capture the meaning of a complex sentence in a diagram containing pictures and simplified text related by structure extracted from the original sentence.

• Paper Link : https://www.cs.cmu.edu/~jbigham/pubs/pdfs/2011/multimodal_summarization.pdf
• Model : ROCMMS

Contributions

The main contributions are as follows:

• ROCMMS, a system that automatically converts existing text to multimodal summaries (MMS) that capture the meaning of a complex sentence in a diagram containing pictures and simplified text related by structure extracted from the original sentence.

Summary

Multimodal summarization (MMS) of complex sentences gives readers the main idea of the sentence using pictures and compressed text structured as simple sentence. Creating MMSs is challenging and involves many subtasks. The general steps in the MMS approach are the following:

• Identify both the main idea of the sentence and related entities and use them to create a compressed summary
• Extract pictures for the entities.
• Add structure to the pictures and text.

NOTE : This method is purely statistical and deterministic and does not use Deep Neural Networks.

The metrics used are :

• ROUGE-{1,2,L}
• FScore