remove all lines in a file containing a string from another file
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
|
show 3 more comments
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08
|
show 3 more comments
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
I'd like to remove all the lines of a file based on matching a string from another file. This is what I have used but it only deletes some:
grep -vFf to_delete.csv inputfile.csv > output.csv
Here are sample lines from my input file (inputfile.csv):
Ata,Aqu,Ama3,Abe,0.053475,0.025,0.1,0.11275,0.1,0.15,0.83377
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Ata135,Atb,Aca,Am54,0.14695,0.1,0.2,0.05255,0.025,0.075,0.8005,
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
My file "to_delete.csv" looks like this for example:
Aqu
Aca
So any line with those strings should get deleted, in this case, lines 1 and 3 should get deleted. Sample desired output:
Ata135,Aru2,Aba301,A29,0.055525,0.025,0.1,0.082825,0.075,0.125
Adc,Aru7,Ama301,Agr84,0.002075,0,0.025,0.240075,0.2,0.
awk grep match carriage-return
awk grep match carriage-return
edited Nov 21 at 7:23
Inian
38.5k63669
38.5k63669
asked Nov 21 at 2:33
NKN
305
305
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08
|
show 3 more comments
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08
|
show 3 more comments
1 Answer
1
active
oldest
votes
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk
), append > temp_file && mv temp_file Input_file
in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
1
@NKN, also try to check by doingcat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?
– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404530%2fremove-all-lines-in-a-file-containing-a-string-from-another-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk
), append > temp_file && mv temp_file Input_file
in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
1
@NKN, also try to check by doingcat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?
– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
add a comment |
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk
), append > temp_file && mv temp_file Input_file
in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
1
@NKN, also try to check by doingcat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?
– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
add a comment |
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk
), append > temp_file && mv temp_file Input_file
in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
EDIT: Since OP had carriage characters in his files so adding solution for that too now.
cat -v Input_file ##To check if carriage returns are there or not.
tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
Since your samples of Input_file and expected output is not clear so couldn't fully test it, could you please try following.(if you are ok with awk
), append > temp_file && mv temp_file Input_file
in code to save output into Input_file itself.
awk -F, 'FNR==NR{a[$0];next} {for(i=1;i<=NF;i++){if($i in a){next}}} 1' to_delete.csv Input_file > temp_file && mv temp_file Input_file
Explanation: Adding explanation for above code too now.
awk -F, ' ##Setting field separator as comma here.
FNR==NR{ ##checking condition FNR==NR which will be TRUE when first Input_file is being read.
a[$0] ##Creating an array named a whose index is $0.
next ##next will skip all further statements from here.
}
{
for(i=1;i<=NF;i++){ ##Starting a for loop from value i=1 to till value of NF.
if($i in a){ ##checking if $i is present in array a if yes then go into this condition block.
next ##next will skip all further statements(since we DO NOt want to print any matching contents)
} ##Closing if block now.
} ##Closing for block here.
} ##Closing block which should be executed for 2nd Input_file here.
1 ##awk works on pattern and action method so making condition TRUE here and not mentioning any action so by default print of current line will happen.
' to_delete.csv Input_file ##Mentioning Input_file names here now.
edited Nov 21 at 3:43
answered Nov 21 at 2:51
RavinderSingh13
25.5k41438
25.5k41438
1
@NKN, also try to check by doingcat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?
– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
add a comment |
1
@NKN, also try to check by doingcat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doingtr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?
– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
1
1
@NKN, also try to check by doing
cat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doing tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?– RavinderSingh13
Nov 21 at 3:20
@NKN, also try to check by doing
cat -v Input_file
if you have carriage returns in your file? If yes then you could remove them by doing tr -d 'r' < Input_file > temp_file && mv temp_file Input_file
, let me know on same?– RavinderSingh13
Nov 21 at 3:20
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
AHHA! That was the problem. I thought checked for that initially but I guess they don't show up in some editors. Thank you!
– NKN
Nov 21 at 3:34
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404530%2fremove-all-lines-in-a-file-containing-a-string-from-another-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Not clear, please mention more clear examples in your post and let us know then?
– RavinderSingh13
Nov 21 at 2:35
I just updated the question with more exmaples and a sample desired output. I hope that is clear?
– NKN
Nov 21 at 3:03
Could you please try my code once and let me know?
– RavinderSingh13
Nov 21 at 3:05
When you tried your command, what was the actual output?
– Benjamin W.
Nov 21 at 3:07
@RavinderSingh13 - Based on wc -l it didnt delete any lines. @ Benjamin W - It deleted some lines but not all. I think it only deleted lines for which the string was first? But hard to say with thousands of lines.
– NKN
Nov 21 at 3:08