How can i clean my dataset from outliers as it includes numerical and categorical variables in Python?
I would like to clean my dataset from outliers but just in three specific columns, as the other 10 contain categorical variables. So how can I get my data cleaned by only referring to these specific columns?
I'd like to use iqr range method. That's the code i run so far:
import numpy as np
def outliers(x):
return np.abs(x- x.median()) > 1.5*(x.quantile(.75)-x.quantile(0.25))
ath2.Age[outliers(ath2.Age)]
ath2.Height[outliers(ath2.Height)]
ath2.Weight[outliers(ath2.Weight)]
After checking the number of outliers in the columns I'm interested in, I don't know how to proceed further.
python outliers
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
add a comment |
I would like to clean my dataset from outliers but just in three specific columns, as the other 10 contain categorical variables. So how can I get my data cleaned by only referring to these specific columns?
I'd like to use iqr range method. That's the code i run so far:
import numpy as np
def outliers(x):
return np.abs(x- x.median()) > 1.5*(x.quantile(.75)-x.quantile(0.25))
ath2.Age[outliers(ath2.Age)]
ath2.Height[outliers(ath2.Height)]
ath2.Weight[outliers(ath2.Weight)]
After checking the number of outliers in the columns I'm interested in, I don't know how to proceed further.
python outliers
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44
add a comment |
I would like to clean my dataset from outliers but just in three specific columns, as the other 10 contain categorical variables. So how can I get my data cleaned by only referring to these specific columns?
I'd like to use iqr range method. That's the code i run so far:
import numpy as np
def outliers(x):
return np.abs(x- x.median()) > 1.5*(x.quantile(.75)-x.quantile(0.25))
ath2.Age[outliers(ath2.Age)]
ath2.Height[outliers(ath2.Height)]
ath2.Weight[outliers(ath2.Weight)]
After checking the number of outliers in the columns I'm interested in, I don't know how to proceed further.
python outliers
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
I would like to clean my dataset from outliers but just in three specific columns, as the other 10 contain categorical variables. So how can I get my data cleaned by only referring to these specific columns?
I'd like to use iqr range method. That's the code i run so far:
import numpy as np
def outliers(x):
return np.abs(x- x.median()) > 1.5*(x.quantile(.75)-x.quantile(0.25))
ath2.Age[outliers(ath2.Age)]
ath2.Height[outliers(ath2.Height)]
ath2.Weight[outliers(ath2.Weight)]
After checking the number of outliers in the columns I'm interested in, I don't know how to proceed further.
python outliers
python outliers
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
asked Nov 25 '18 at 11:12
Emanuele ColaceEmanuele Colace
31
31
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44
add a comment |
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44
add a comment |
1 Answer
1
active
oldest
votes
If you want the code to be dynamic, you can 1st check the columns which are not categorical by below code:
cols = df.columns
num_cols = df._get_numeric_data().columns
##num_cols will contains list of column names which are numeric
## In your case, it should come Age,Height etc.
Alternatively, you can also use include or exclude parameters using df.select_dtypes according to your dataframe
After this run below code from columns from above:
df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
## Df is the dataframe and Data is the name of the column.
#In your case, it will be Age,Height etc.
OR
If you want to make a new df with only the numerical columns and find out the outliers in one shot, below is the code:
df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53466881%2fhow-can-i-clean-my-dataset-from-outliers-as-it-includes-numerical-and-categorica%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you want the code to be dynamic, you can 1st check the columns which are not categorical by below code:
cols = df.columns
num_cols = df._get_numeric_data().columns
##num_cols will contains list of column names which are numeric
## In your case, it should come Age,Height etc.
Alternatively, you can also use include or exclude parameters using df.select_dtypes according to your dataframe
After this run below code from columns from above:
df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
## Df is the dataframe and Data is the name of the column.
#In your case, it will be Age,Height etc.
OR
If you want to make a new df with only the numerical columns and find out the outliers in one shot, below is the code:
df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
add a comment |
If you want the code to be dynamic, you can 1st check the columns which are not categorical by below code:
cols = df.columns
num_cols = df._get_numeric_data().columns
##num_cols will contains list of column names which are numeric
## In your case, it should come Age,Height etc.
Alternatively, you can also use include or exclude parameters using df.select_dtypes according to your dataframe
After this run below code from columns from above:
df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
## Df is the dataframe and Data is the name of the column.
#In your case, it will be Age,Height etc.
OR
If you want to make a new df with only the numerical columns and find out the outliers in one shot, below is the code:
df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
add a comment |
If you want the code to be dynamic, you can 1st check the columns which are not categorical by below code:
cols = df.columns
num_cols = df._get_numeric_data().columns
##num_cols will contains list of column names which are numeric
## In your case, it should come Age,Height etc.
Alternatively, you can also use include or exclude parameters using df.select_dtypes according to your dataframe
After this run below code from columns from above:
df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
## Df is the dataframe and Data is the name of the column.
#In your case, it will be Age,Height etc.
OR
If you want to make a new df with only the numerical columns and find out the outliers in one shot, below is the code:
df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
If you want the code to be dynamic, you can 1st check the columns which are not categorical by below code:
cols = df.columns
num_cols = df._get_numeric_data().columns
##num_cols will contains list of column names which are numeric
## In your case, it should come Age,Height etc.
Alternatively, you can also use include or exclude parameters using df.select_dtypes according to your dataframe
After this run below code from columns from above:
df[np.abs(df.Data-df.Data.mean()) <= (3*df.Data.std())]
## Df is the dataframe and Data is the name of the column.
#In your case, it will be Age,Height etc.
OR
If you want to make a new df with only the numerical columns and find out the outliers in one shot, below is the code:
df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)]
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
answered Nov 25 '18 at 19:21
Rahul AgarwalRahul Agarwal
2,27151028
2,27151028
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53466881%2fhow-can-i-clean-my-dataset-from-outliers-as-it-includes-numerical-and-categorica%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
what is the error here??
– Rahul Agarwal
Nov 25 '18 at 12:57
@RahulAgarwal no errors up to this point. But then I need to remove outliers. I tried this code: ath2 = ath2[~ath2.apply(outliers).any(axis=1)]. But as there are also strings it cannot work on the entire dataset. How can i make it work on just these three columns? TypeError: (ValueError("could not convert string to float: 'Olesya Nikolayevna Zykina'",), 'occurred at index Name')
– Emanuele Colace
Nov 25 '18 at 14:00
What is the error u r getting when u r applying with just 3 columns as you have shown in your code. What is the problem with the above code u r trying?
– Rahul Agarwal
Nov 25 '18 at 14:32
AttributeError: 'float' object has no attribute 'median'
– Emanuele Colace
Nov 25 '18 at 16:15
Yes it worked, thanks again
– Emanuele Colace
Nov 29 '18 at 12:44