在逻辑条件中包含更多停用词列表以过滤单词

Question

我需要在清理数据时添加更多条件，包括删除停用词、星期几和月份。对于星期几和月份，我创建了一个单独的列表（我不知道 python 中是否有一些已经内置的包来包含它们）。对于数字，我会考虑 isdigit。所以像这样：

days=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']

# need to put into lower case

months=['January','February','March', 'April','May','June','July','August','September','October','November','December']

# need to put into lower case

cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

我怎样才能包含在上面的代码中？我知道这是需要考虑额外的 if 语句，但我正在努力解决这个问题。

Answer 1

您可以将所有列表转换为集合，并将它们的并集作为最终集合。然后只需要检查你的单词在集合中的成员资格。像下面这样的东西会起作用：# existing codefrom nltk.corpus import stopwordsdays=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']# need to put into lower casemonths=['January','February','March', 'April','May','June','July','August','September','October','November','December']# need to put into lower case# add these linesstop_words = set(stopwords.words('english'))lowercase_days = {item.lower() for item in days}lowercase_months = {item.lower() for item in months}exclusion_set = lowercase_days.union(lowercase_months).union(stop_words)# now do the final checkcleaned = [w for w in remove_punc.split() if w.lower() not in exclusion_set and not w.isdigit()]