python - How do I extract the id from an unsupervised text classification
So I have the following dataframe:
id text
342 text sample
341 another text sample
343 ...
And the following code:
X = tfidf_vectorizer.fit_transform(df['text']).todense()
pca = PCA(n_components=2)
data2D = pca.fit_transform(X)
clusterer = KMeans(n_clusters=n_clusters), random_state=10)
cluster_labels = clusterer.fit_predict(data2D)
silhouette_avg = silhouette_score(data2D, cluster_labels)
print(silhouette_avg)
y_lower = 10
for i in range(n_clusters):
# here I would like to get the id's of each item per cluster
# so that I know which list of id's falls into which cluster
Now, how can I see which id falls in which cluster, is this something that can be done? Also is my approach correct in order to "clusterize" these text documents?
Please not that I might have skipped some code in order to keep the question short
python-3.x k-means pca text-classification unsupervised-learning
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
add a comment |
So I have the following dataframe:
id text
342 text sample
341 another text sample
343 ...
And the following code:
X = tfidf_vectorizer.fit_transform(df['text']).todense()
pca = PCA(n_components=2)
data2D = pca.fit_transform(X)
clusterer = KMeans(n_clusters=n_clusters), random_state=10)
cluster_labels = clusterer.fit_predict(data2D)
silhouette_avg = silhouette_score(data2D, cluster_labels)
print(silhouette_avg)
y_lower = 10
for i in range(n_clusters):
# here I would like to get the id's of each item per cluster
# so that I know which list of id's falls into which cluster
Now, how can I see which id falls in which cluster, is this something that can be done? Also is my approach correct in order to "clusterize" these text documents?
Please not that I might have skipped some code in order to keep the question short
python-3.x k-means pca text-classification unsupervised-learning
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
add a comment |
So I have the following dataframe:
id text
342 text sample
341 another text sample
343 ...
And the following code:
X = tfidf_vectorizer.fit_transform(df['text']).todense()
pca = PCA(n_components=2)
data2D = pca.fit_transform(X)
clusterer = KMeans(n_clusters=n_clusters), random_state=10)
cluster_labels = clusterer.fit_predict(data2D)
silhouette_avg = silhouette_score(data2D, cluster_labels)
print(silhouette_avg)
y_lower = 10
for i in range(n_clusters):
# here I would like to get the id's of each item per cluster
# so that I know which list of id's falls into which cluster
Now, how can I see which id falls in which cluster, is this something that can be done? Also is my approach correct in order to "clusterize" these text documents?
Please not that I might have skipped some code in order to keep the question short
python-3.x k-means pca text-classification unsupervised-learning
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
So I have the following dataframe:
id text
342 text sample
341 another text sample
343 ...
And the following code:
X = tfidf_vectorizer.fit_transform(df['text']).todense()
pca = PCA(n_components=2)
data2D = pca.fit_transform(X)
clusterer = KMeans(n_clusters=n_clusters), random_state=10)
cluster_labels = clusterer.fit_predict(data2D)
silhouette_avg = silhouette_score(data2D, cluster_labels)
print(silhouette_avg)
y_lower = 10
for i in range(n_clusters):
# here I would like to get the id's of each item per cluster
# so that I know which list of id's falls into which cluster
Now, how can I see which id falls in which cluster, is this something that can be done? Also is my approach correct in order to "clusterize" these text documents?
Please not that I might have skipped some code in order to keep the question short
python-3.x k-means pca text-classification unsupervised-learning
python-3.x k-means pca text-classification unsupervised-learning
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
asked Nov 25 '18 at 22:07
Mihai VinagaMihai Vinaga
432419
432419
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
There are many ways to perform document classification. K-Means is one way. To say what you are doing is the best would be impossible with looking at the data and use case and exploring other methods.
If you'd like to stick with KMeans, I suggest you go read the documentation on the scikit-learn website one more time. You'll notice in the example how you can get the predicted class label for each point by calling the labels_ property on the fit classifier (note: not the result of fit_transform as you currently have).
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53472487%2fpython-how-do-i-extract-the-id-from-an-unsupervised-text-classification%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
There are many ways to perform document classification. K-Means is one way. To say what you are doing is the best would be impossible with looking at the data and use case and exploring other methods.
If you'd like to stick with KMeans, I suggest you go read the documentation on the scikit-learn website one more time. You'll notice in the example how you can get the predicted class label for each point by calling the labels_ property on the fit classifier (note: not the result of fit_transform as you currently have).
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
add a comment |
There are many ways to perform document classification. K-Means is one way. To say what you are doing is the best would be impossible with looking at the data and use case and exploring other methods.
If you'd like to stick with KMeans, I suggest you go read the documentation on the scikit-learn website one more time. You'll notice in the example how you can get the predicted class label for each point by calling the labels_ property on the fit classifier (note: not the result of fit_transform as you currently have).
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
add a comment |
There are many ways to perform document classification. K-Means is one way. To say what you are doing is the best would be impossible with looking at the data and use case and exploring other methods.
If you'd like to stick with KMeans, I suggest you go read the documentation on the scikit-learn website one more time. You'll notice in the example how you can get the predicted class label for each point by calling the labels_ property on the fit classifier (note: not the result of fit_transform as you currently have).
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
There are many ways to perform document classification. K-Means is one way. To say what you are doing is the best would be impossible with looking at the data and use case and exploring other methods.
If you'd like to stick with KMeans, I suggest you go read the documentation on the scikit-learn website one more time. You'll notice in the example how you can get the predicted class label for each point by calling the labels_ property on the fit classifier (note: not the result of fit_transform as you currently have).
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
answered Dec 24 '18 at 19:38
Alex LAlex L
307411
307411
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53472487%2fpython-how-do-i-extract-the-id-from-an-unsupervised-text-classification%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
