Natural Language Processing with Python
Python 自然语言处理
2.8练习
1. ○ Create a variable phrase containing a list of words. Experiment with the operations described in this chapter, including addition, multiplication, indexing, slicing, and sorting.
2. ○ Use the corpus module to explore austen-persuasion.txt . How many word tokens does this book have? How many word types?
>>> import nltk
>>> from nltk.corpus import gutenberg
>>> gutenberg.fileids()
>>> len(gutenberg.words('austen-persuasion.txt'))
98171
>>> len(set(gutenberg.words('austen-persuasion.txt')))
6132
>>> len(set([w.lower() for w in gutenberg.words('austen-persuasion.txt')]))
5835
>>>
>>>
>>> persuasion = gutenberg.words('austen-persuasion.txt')
>>> len(persuasion) #tokens = len(persuasion)
>>> len(set(persuasion)) #types_1 = len(set(persuasion))
>>> len(set([w.lower for w in persuasion])) #types_2 = len(set([w.lower for w in persuasion]))
tokens:98171
types:6132
3. ○ Use the Brown Corpus reader nltk.corpus.brown.words() or the Web Text Corpus reader nltk.corpus.webtext.words() to access some sample text in two different genres.
4. ○ Read in the texts of the State of the Union addresses, using the state_union corpus reader. Count occurrences of men , women , and people in each document. What has happened to the usage of these words over time?
#
# NLP with Python
# ch2 2.8 EX2-8_q4
#
import nltk
from nltk.corpus import state_union
state_union.fileids()
cfd = nltk.ConditionalFreqDist((target,fileid[:4])
for fileid in state_union.fileids()
for w in state_union.words(fileid)
for target in ['women','man','people']
if w.lower().startswith(target))
cfd.plot()
5. ○ Investigate the holonym-meronym relations for some nouns. Remember that there are three kinds of holonym-meronym relation, so you need to use member_meronyms() ,part_meronyms() ,substance_meronyms() ,member_holonyms() ,part_holonyms() , and substance_holonyms() .
6. ○ In the discussion of comparative wordlists, we created an object called translate , which you could look up using words in both German and Italian in order to get corresponding words in English. What problem might arise with this approach? Can you suggest a way to avoid this problem?
7.○ According to Strunk and White’s Elements of Style, the word however, used at the start of a sentence, means “in whatever way” or “to whatever extent,” and not “nevertheless.” They give this example of correct usage: However you advise him,he will probably do as he thinks best. (http://www.bartleby.com/141/strunk3.html) Use the concordance tool to study actual usage of this word in the various texts wehave been considering. See also the LanguageLog posting “Fossilized prejudices about ‘however’” at http://itre.cis.upenn.edu/~myl/languagelog/archives/001913.html.
8.◑ Define a conditional frequency distribution over the Names Corpus that allowsyou to see which initial letters are more frequent for males versus females (see Figure 2-7).
#
# NLP with Python
# ch2 2.8 EX2-8_q8
#
import nltk
from nltk.corpus import names
name_types = names.fileids()
# ['male.txt','female.txt']
male_names = names.words('male.txt')
female_names = names.words('female.txt')
# [w for w in male_names if w in female_names]
cfd = nltk.ConditionalFreqDist(
(fileid, name[1])
for fileid in names.fileids()
for name in names.words(fileid)
)
cfd.plot()
9.◑ Pick a pair of texts and study the differences between them, in terms of vocabulary, vocabulary richness, genre, etc. Can you find pairs of words that have quite different meanings across the two texts, such as monstrous in Moby Dick and in Sense and Sensibility?
10.◑ Read the BBC News article: “UK’s Vicky Pollards ‘left behind’” at http://news.bbc.co.uk/1/hi/education/6173441.stm. The article gives the following statistic about teen language: “the top 20 words used, including yeah, no, but and like, account for around a third of all words.” How many word types account for a third of all word tokens, for a variety of text sources? What do you conclude about this statistic? Read more about this on LanguageLog, at http://itre.cis.upenn.edu/~myl/languagelog/archives/003993.html.
stopwords 语料库, 高频词汇
