Iterate 2 objects with Nested Loop List Comprehension with Tokenisers
I am trying to grab a large sample of data from a corpus, and establishing what proportion of the tokens are stop-words.
from sussex_nltk.corpus_readers import MedlineCorpusReader
from nltk.corpus import stopwords
mcr = MedlineCorpusReader()
sample_size = 10000
stopwords = stopwords.words('english')
raw_sentences = mcr.sample_raw_sents(sample_size)
tokenised_sentences = [word_tokenize(sentence) for sentence in raw_sentences]
filter_tok=[[sentence.isalpha() for sentence in sentence and sentence not in stopwords] for sentence in tokenised_sentences]
raw_vocab_size = vocabulary_size(tokenised_sentences)
filter_vocab_size = vocabulary_size(filter_tok)
print("Stopwords produced a {0:.2f}% reduction in vocabulary size from {1} to {2}".format(
100*(raw_vocab_size - filter_vocab_size)/raw_vocab_size,raw_vocab_size,filter_vocab_size))
Though even after I tokenise my list, I still can't seem to traverse through it. Believe the problem is rooted on line 11, though I am unsure how to iterate with 2 different objects, both .isalpha() and stopwords.
python list loops token stop-words
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
add a comment |
I am trying to grab a large sample of data from a corpus, and establishing what proportion of the tokens are stop-words.
from sussex_nltk.corpus_readers import MedlineCorpusReader
from nltk.corpus import stopwords
mcr = MedlineCorpusReader()
sample_size = 10000
stopwords = stopwords.words('english')
raw_sentences = mcr.sample_raw_sents(sample_size)
tokenised_sentences = [word_tokenize(sentence) for sentence in raw_sentences]
filter_tok=[[sentence.isalpha() for sentence in sentence and sentence not in stopwords] for sentence in tokenised_sentences]
raw_vocab_size = vocabulary_size(tokenised_sentences)
filter_vocab_size = vocabulary_size(filter_tok)
print("Stopwords produced a {0:.2f}% reduction in vocabulary size from {1} to {2}".format(
100*(raw_vocab_size - filter_vocab_size)/raw_vocab_size,raw_vocab_size,filter_vocab_size))
Though even after I tokenise my list, I still can't seem to traverse through it. Believe the problem is rooted on line 11, though I am unsure how to iterate with 2 different objects, both .isalpha() and stopwords.
python list loops token stop-words
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
add a comment |
I am trying to grab a large sample of data from a corpus, and establishing what proportion of the tokens are stop-words.
from sussex_nltk.corpus_readers import MedlineCorpusReader
from nltk.corpus import stopwords
mcr = MedlineCorpusReader()
sample_size = 10000
stopwords = stopwords.words('english')
raw_sentences = mcr.sample_raw_sents(sample_size)
tokenised_sentences = [word_tokenize(sentence) for sentence in raw_sentences]
filter_tok=[[sentence.isalpha() for sentence in sentence and sentence not in stopwords] for sentence in tokenised_sentences]
raw_vocab_size = vocabulary_size(tokenised_sentences)
filter_vocab_size = vocabulary_size(filter_tok)
print("Stopwords produced a {0:.2f}% reduction in vocabulary size from {1} to {2}".format(
100*(raw_vocab_size - filter_vocab_size)/raw_vocab_size,raw_vocab_size,filter_vocab_size))
Though even after I tokenise my list, I still can't seem to traverse through it. Believe the problem is rooted on line 11, though I am unsure how to iterate with 2 different objects, both .isalpha() and stopwords.
python list loops token stop-words
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
I am trying to grab a large sample of data from a corpus, and establishing what proportion of the tokens are stop-words.
from sussex_nltk.corpus_readers import MedlineCorpusReader
from nltk.corpus import stopwords
mcr = MedlineCorpusReader()
sample_size = 10000
stopwords = stopwords.words('english')
raw_sentences = mcr.sample_raw_sents(sample_size)
tokenised_sentences = [word_tokenize(sentence) for sentence in raw_sentences]
filter_tok=[[sentence.isalpha() for sentence in sentence and sentence not in stopwords] for sentence in tokenised_sentences]
raw_vocab_size = vocabulary_size(tokenised_sentences)
filter_vocab_size = vocabulary_size(filter_tok)
print("Stopwords produced a {0:.2f}% reduction in vocabulary size from {1} to {2}".format(
100*(raw_vocab_size - filter_vocab_size)/raw_vocab_size,raw_vocab_size,filter_vocab_size))
Though even after I tokenise my list, I still can't seem to traverse through it. Believe the problem is rooted on line 11, though I am unsure how to iterate with 2 different objects, both .isalpha() and stopwords.
python list loops token stop-words
python list loops token stop-words
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
asked Nov 24 '18 at 20:17
bemzoobemzoo
7812
7812
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I know very little about the libraries you are using, but I know something about list comprehensions. The correct syntax is
[element for element in iterable if condition]
But you used
[element for element in iterable and condition]
So Python interpreted iterable and condition (or in your example sentence and sentence not in stopwords) as one expression. The result is a boolean and not iterable, so it throws a TypeError.
Just replace and with if and it will probably work. The nested list comprehensions are otherwise correct. I just wouldn't recommend having the same name for the element and the iterable (sentence), because that can lead to confusion.
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53461995%2fiterate-2-objects-with-nested-loop-list-comprehension-with-tokenisers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I know very little about the libraries you are using, but I know something about list comprehensions. The correct syntax is
[element for element in iterable if condition]
But you used
[element for element in iterable and condition]
So Python interpreted iterable and condition (or in your example sentence and sentence not in stopwords) as one expression. The result is a boolean and not iterable, so it throws a TypeError.
Just replace and with if and it will probably work. The nested list comprehensions are otherwise correct. I just wouldn't recommend having the same name for the element and the iterable (sentence), because that can lead to confusion.
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
add a comment |
I know very little about the libraries you are using, but I know something about list comprehensions. The correct syntax is
[element for element in iterable if condition]
But you used
[element for element in iterable and condition]
So Python interpreted iterable and condition (or in your example sentence and sentence not in stopwords) as one expression. The result is a boolean and not iterable, so it throws a TypeError.
Just replace and with if and it will probably work. The nested list comprehensions are otherwise correct. I just wouldn't recommend having the same name for the element and the iterable (sentence), because that can lead to confusion.
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
add a comment |
I know very little about the libraries you are using, but I know something about list comprehensions. The correct syntax is
[element for element in iterable if condition]
But you used
[element for element in iterable and condition]
So Python interpreted iterable and condition (or in your example sentence and sentence not in stopwords) as one expression. The result is a boolean and not iterable, so it throws a TypeError.
Just replace and with if and it will probably work. The nested list comprehensions are otherwise correct. I just wouldn't recommend having the same name for the element and the iterable (sentence), because that can lead to confusion.
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
I know very little about the libraries you are using, but I know something about list comprehensions. The correct syntax is
[element for element in iterable if condition]
But you used
[element for element in iterable and condition]
So Python interpreted iterable and condition (or in your example sentence and sentence not in stopwords) as one expression. The result is a boolean and not iterable, so it throws a TypeError.
Just replace and with if and it will probably work. The nested list comprehensions are otherwise correct. I just wouldn't recommend having the same name for the element and the iterable (sentence), because that can lead to confusion.
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
answered Nov 24 '18 at 20:34
BurningKarlBurningKarl
75948
75948
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53461995%2fiterate-2-objects-with-nested-loop-list-comprehension-with-tokenisers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
