NAME "Text::Corpus::Summaries::Wikipedia" - Creates corpora for summarization testing. SYNOPSIS use Text::Corpus::Summaries::Wikipedia; use Data::Dump qw(dump); my $corpus = Text::Corpus::Summaries::Wikipedia->new; $corpus->create; dump $corpus->getListOfXmlFiles; DESCRIPTION "Text::Corpus::Summaries::Wikipedia" creates corpora for single document summarization testing using the featured articles of various Wikipedias. A criterion for an article in a Wikipedia to be *featured* is that it have a well written lead section, or introduction. So the featured articles of a Wikipedia can make an excellent corpus for testing single document summarization methods. This module creates a corpus from the featured articles of a Wikipedia by fetching and saving their content as HTML, text, and XML, with the appropriate sections labeled as either "summary" or "body". CONSTRUCTOR "new" The constructor "new" creates an instance of the "Text::Corpus::Summaries::Wikipedia" class with the following parameters: "languageCode" languageCode => 'en' "languageCode" is the language code of the Wikipedia from which the corpus of featured articles are to be created. The supported language codes are "af":Afrikaans, "ar":Arabic, "az":Azerbaijani, "bg":Bulgarian, "bs":Bosnian, "ca":Catalan, "cs":Czech, "de":German, "el":Greek, "en":English, "eo":Esperanto, "es":Spanish, "eu":Basque, "fa":Persian, "fi":Finnish, "fr":French, "he":Hebrew, "hr":Croatian, "hu":Hungarian, "id":Indonesian, "it":Italian, "ja":Japanese, "jv":Javanese, "ka":Georgian, "kk":Kazakh, "km":Khmer, "ko":Korean, "li":Limburgish, "lv":Latvian, "ml":Malayalam, "mr":Marathi, "ms":Malay, "mzn":Mazandarani, "nl":Dutch, "nn":Norwegian (Nynorsk), "no":Norwegian (Bokm?l), "pl":Polish, "pt":Portuguese, "ro":Romanian, "ru":Russian, "sh":Serbo-Croatian, "simple":Simple English, "sk":Slovak, "sl":Slovenian, "sr":Serbian, "sv":Swedish, "sw":Swahili, "ta":Tamil, "th":Thai, "tl":Tagalog, "tr":Turkish, "tt":Tatar, "uk":Ukrainian, "ur":Urdu, "vi":Vietnamese, "vo":Volap?k, and "zh":Chinese. If the language code is "all", then the corpus for each supported language is created (which takes a long time). The default is "en". "corpusDirectory" corpusDirectory => 'cwd' "corpusDirectory" is the corpus directory that the summaries and articles will be stored in; the directory is created if it does not exist. The default is the "cwd". A language subdirectory is created at "corpusDirectory/languageCode" that will contain the directories "log", "html", "unparsable", "text", and "xml". The directory "log" will contain the file "log.txt" that all errors, warnings, and informational messages are logged to using Log::Log4perl. The directory "html" will contain copies of the HTML versions of the featured article pages fetched using LWP. The directory "text" will contain two files for each article; one file will end with "_body.txt" and contain the body text of the article, the other will end with "_summary.txt" and will contain the summary. The directory "unparsable" will contain the HTML files that could not be parsed into *body* and *summary* sections. All files are UTF-8 encoded. METHODS "create" The method "create" fetches the featured articles and creates the text and XML versions of the files. use Text::Corpus::Summaries::Wikipedia; use Data::Dump qw(dump); my $corpus = Text::Corpus::Summaries::Wikipedia->new; $corpus->create; dump $corpus->getListOfXmlFiles; dump $corpus->getListOfTextFiles; "maxProcesses" maxProcesses => 1 "maxProcesses" is the maximum number of processes that can be running simultaneously to parse the files. Parsing the files for the summary and body sections may be computational intensive so the module Forks::Super is used for parallelization. The default is one. "test" test => 0 If "test" is a positive integer than it will be treated as the maximum number of pages that may be fetched and parsed. The default is zero, meaning all possible pages are fetched and parsed. "recreate" The method "recreate" recreates the text and XML versions of the files from the list of previously fetched HTML files in the "html" directory. use Text::Corpus::Summaries::Wikipedia; use Data::Dump qw(dump); my $corpus = Text::Corpus::Summaries::Wikipedia->new; $corpus->recreate; dump $corpus->getListOfXmlFiles; dump $corpus->getListOfTextFiles; "maxProcesses" maxProcesses => 1 "maxProcesses" is the maximum number of processes that can be running simultaneously to parse the files. Parsing the files for the summary and body sections may be computational intensive so the module Forks::Super is used for parallelization. The default is one. "test" test => 0 If "test" is a positive integer than it will be treated as the maximum number of pages that may be parsed. The default is zero, meaning all possible pages are parsed. "getListOfTextFiles" The method "getListOfTextFiles" returns an array reference with each item in the list having the form "{body => 'path_to_body_file', summary => 'path_to_summary_file'}". use Text::Corpus::Summaries::Wikipedia; use Data::Dump qw(dump); my $corpus = Text::Corpus::Summaries::Wikipedia->new; $corpus->create; dump $corpus->getListOfTextFiles; "getListOfXmlFiles" The method "getListOfXmlFiles" returns an array reference containing the path to each XML file. use Text::Corpus::Summaries::Wikipedia; use XML::Simple; use Data::Dump qw(dump); my $corpus = Text::Corpus::Summaries::Wikipedia->new; foreach my $xmlFile (@{$corpus->getListOfXmlFiles}) { my $article; eval { $article = XMLin ($xmlFile) }; if ($@) { dump \$@; } else { dump $article; } } XML FORMAT The XML files are created using XML::Code with the simple structure outlined below: The Article Title

This is the first paragraph of the summary.

This is the second paragraph of the summary.

Head of first section

First paragraph of this section.

Second paragraph of this section.

First paragraph of this section.

Second paragraph of this section.

Head of sub-section

First paragraph of this sub-section.

Second paragraph of this sub-section.

EXAMPLE The following example computes and prints the median, mean, and standard deviation of the fraction of words (ignoring repeats) in a summary that also occur in the body of the text for all the articles in the corpora. use Text::Corpus::Summaries::Wikipedia; use Statistics::Descriptive; use File::Slurp; use Encode; my $corpus = Text::Corpus::Summaries::Wikipedia->new; my $statistics = Statistics::Descriptive::Full->new; foreach my $textFilePair (@{$corpus->getListOfTextFiles}) { my $summary = lc decode ('utf8', read_file ($textFilePair->{summary}, binmode => ':raw')); my %summaryWords = map {($_, 1)} split (/\P{Letter}/, $summary); my $totalUniqueSummaryWords = keys %summaryWords; next unless $totalUniqueSummaryWords; my $body = lc decode ('utf8', read_file ($textFilePair->{body}, binmode => ':raw')); map {delete $summaryWords{$_}} split (/\P{Letter}/, $body); my $totalUniqueSummaryWordsNotInBody = keys %summaryWords; $statistics->add_data (1 - $totalUniqueSummaryWordsNotInBody / $totalUniqueSummaryWords); } print 'count: ', $statistics->count(), "\n"; print 'median: ', $statistics->median(), "\n"; print 'mean: ', $statistics->mean(), "\n"; print 'standard deviation: ', $statistics->standard_deviation(), "\n"; SCRIPTS The script create_summary_corpus.pl makes a corpus for summarization testing using this module. INSTALLATION Use CPAN to install the module and all its prerequisites: perl -MCPAN -e shell >install Text::Corpus::Summaries::Wikipedia BUGS This module creates corpora by parsing Wikipedia pages, the xpath expressions used to extract links and text will become invalid as the format of the various pages changes, causing some corpora not to be created. Please email bugs reports or feature requests to "bug-text-corpus-summaries-wikipedia@rt.cpan.org", or through the web interface at . The author will be notified and you can be automatically notified of progress on the bug fix or feature request. AUTHOR Jeff Kubina COPYRIGHT Copyright (c) 2010 Jeff Kubina. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. KEYWORDS corpus, information processing, summaries, summarization, wikipedia SEE ALSO create_summary_corpus.pl, Encode, Forks::Super, HTML::TreeBuilder::XPath, Log::Log4perl, LWP::UserAgent, XML::Code