Software tools for searching for similar fragments in a collection of educational documents
Abstract:
The article examines methods for automatic segmentation of electronic documents in docx and pdf formats into logical fragments (sections and chapters) with subsequent semantic analysis using the Sentence-BERT 2 model. The first method detects and analyzes changes in text fonts and styles, thereby identifying chapters or sections. The second method converts pdf documents to docx, locates chapters using structural markers such as 'Chapter,' 'Contents,' etc., and displays the identified chapters in a graphical interface for subsequent manual adjustment, which is particularly effective for files with nonstandard layouts. The third method utilizes built-in document bookmarks, ensuring maximum accuracy for documents with predefined structures. For semantic analysis of fragments, the Sentence-BERT model is employed to identify meaningful relationships between chapters and sections. Testing results on 74 documents are presented, demonstrating the advantages and limitations of each method. The developed tools automate the document processing workflow, enabling rapid segmentation of document collections into meaningful fragments for identifying similar content, as well as analysis of found content-related pairs through a graphical interface.
Keywords:
document processing, semantic analysis, text segmentation, PDF, Sentence-BERT