bash check for words in first file not contained in second file
I have a txt file containing multiple lines of text, for example:
This is a
file containing several
lines of text.
Now I have another file containing just words, like so:
this
contains
containing
text
Now I want to output the words which are in file 1, but not in file 2. I have tried the following:
cat file_1.txt | xargs -n1 | tr -d '[:punct:]' | sort | uniq | comm -i23 - file_2.txt
xargs -n1 to put each space separated substring on a newline.
tr -d '[:punct:] to remove punctuations
sort and uniq to make a sorted file to use with comm which is used with the -i flag to make it case insensitive.
But somehow this doesn't work. I've looked around online and found similar questions, however, I wasn't able to figure out what I was doing wrong. Most answers to those questions were working with 2 files which were already sorted, stripped of newlines, spaces, and punctuation while my file_1 may contain any of those at the start.
Desired output:
is
a
file
several
lines
of
bash
asked Nov 25 '18 at 11:11
LarsLars
8316
|
show 3 more comments
I have a txt file containing multiple lines of text, for example:
This is a
file containing several
lines of text.
Now I have another file containing just words, like so:
this
contains
containing
text
Now I want to output the words which are in file 1, but not in file 2. I have tried the following:
cat file_1.txt | xargs -n1 | tr -d '[:punct:]' | sort | uniq | comm -i23 - file_2.txt
xargs -n1 to put each space separated substring on a newline.
tr -d '[:punct:] to remove punctuations
sort and uniq to make a sorted file to use with comm which is used with the -i flag to make it case insensitive.
But somehow this doesn't work. I've looked around online and found similar questions, however, I wasn't able to figure out what I was doing wrong. Most answers to those questions were working with 2 files which were already sorted, stripped of newlines, spaces, and punctuation while my file_1 may contain any of those at the start.
Desired output:
is
a
file
several
lines
of
bash
asked Nov 25 '18 at 11:11
LarsLars
8316
What version ofcommare you using? Mine doesn't support-iand expects both input files to be sorted.
– choroba
Nov 25 '18 at 11:21
2
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@choroba I don't know how to find out the version. When I doman commit has the date January 26, 2005 at the bottom.
– Lars
Nov 25 '18 at 11:36
comm --versioncould tell you more.
– choroba
Nov 25 '18 at 11:37
|
show 3 more comments
I have a txt file containing multiple lines of text, for example:
This is a
file containing several
lines of text.
Now I have another file containing just words, like so:
this
contains
containing
text
Now I want to output the words which are in file 1, but not in file 2. I have tried the following:
cat file_1.txt | xargs -n1 | tr -d '[:punct:]' | sort | uniq | comm -i23 - file_2.txt
xargs -n1 to put each space separated substring on a newline.
tr -d '[:punct:] to remove punctuations
sort and uniq to make a sorted file to use with comm which is used with the -i flag to make it case insensitive.
But somehow this doesn't work. I've looked around online and found similar questions, however, I wasn't able to figure out what I was doing wrong. Most answers to those questions were working with 2 files which were already sorted, stripped of newlines, spaces, and punctuation while my file_1 may contain any of those at the start.
Desired output:
is
a
file
several
lines
of
bash
asked Nov 25 '18 at 11:11
LarsLars
8316
I have a txt file containing multiple lines of text, for example:
This is a
file containing several
lines of text.
Now I have another file containing just words, like so:
this
contains
containing
text
Now I want to output the words which are in file 1, but not in file 2. I have tried the following:
cat file_1.txt | xargs -n1 | tr -d '[:punct:]' | sort | uniq | comm -i23 - file_2.txt
xargs -n1 to put each space separated substring on a newline.
tr -d '[:punct:] to remove punctuations
sort and uniq to make a sorted file to use with comm which is used with the -i flag to make it case insensitive.
But somehow this doesn't work. I've looked around online and found similar questions, however, I wasn't able to figure out what I was doing wrong. Most answers to those questions were working with 2 files which were already sorted, stripped of newlines, spaces, and punctuation while my file_1 may contain any of those at the start.
Desired output:
is
a
file
several
lines
of
bash
bash
asked Nov 25 '18 at 11:11
LarsLars
8316
asked Nov 25 '18 at 11:11
LarsLars
8316
edited Nov 25 '18 at 11:34
Lars
asked Nov 25 '18 at 11:11
LarsLars
8316
asked Nov 25 '18 at 11:11
LarsLars
8316
asked Nov 25 '18 at 11:11
LarsLars
8316
8316
What version ofcommare you using? Mine doesn't support-iand expects both input files to be sorted.
– choroba
Nov 25 '18 at 11:21
2
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@choroba I don't know how to find out the version. When I doman commit has the date January 26, 2005 at the bottom.
– Lars
Nov 25 '18 at 11:36
comm --versioncould tell you more.
– choroba
Nov 25 '18 at 11:37
|
show 3 more comments
What version ofcommare you using? Mine doesn't support-iand expects both input files to be sorted.
– choroba
Nov 25 '18 at 11:21
2
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@choroba I don't know how to find out the version. When I doman commit has the date January 26, 2005 at the bottom.
– Lars
Nov 25 '18 at 11:36
comm --versioncould tell you more.
– choroba
Nov 25 '18 at 11:37
What version of
comm are you using? Mine doesn't support -i and expects both input files to be sorted.– choroba
Nov 25 '18 at 11:21
What version of
comm are you using? Mine doesn't support -i and expects both input files to be sorted.– choroba
Nov 25 '18 at 11:21
2
2
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@choroba I don't know how to find out the version. When I do
man comm it has the date January 26, 2005 at the bottom.– Lars
Nov 25 '18 at 11:36
@choroba I don't know how to find out the version. When I do
man comm it has the date January 26, 2005 at the bottom.– Lars
Nov 25 '18 at 11:36
comm --version could tell you more.– choroba
Nov 25 '18 at 11:37
comm --version could tell you more.– choroba
Nov 25 '18 at 11:37
|
show 3 more comments
4 Answers
4
active
oldest
votes
paste + grep approach:
grep -Eiv "($(paste -sd'|' <file2.txt))" <(grep -wo 'w*' file1.txt)
The output:
is
a
file
several
lines
of
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
add a comment |
I would try something more direct:
for A in `cat file1 | tr -d '[:punct:]'`; do grep -wq $A file2 || echo $A; done
flags used for grep: q for quiet (don't need output), w for word match
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
add a comment |
One in awk:
$ awk -F"[^A-Za-z]+" ' # anything but a letter is a field delimiter
NR==FNR { # process the word list
a[tolower($0)]
next
}
{
for(i=1;i<=NF;i++) # loop all fields
if(!(tolower($i) in a)) # if word was not in the word list
print $i # print it. duplicates are printed also.
}' another_file txt_file
Output:
is
a
file
several
lines
of
grep:
$ grep -vwi -f another_file <(cat txt_file | tr -s -c '[a-zA-Z]' 'n')
is
a
file
several
lines
of
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
add a comment |
This pipeline will take the original file, replace spaces with newlines, convert to lowercase, then use grep to filter (-v) full words (-w) case insensitive (-i) using the lines in the given file (-f file2):
cat file1 | tr ' ' 'n' | tr '[:upper:]' '[:lower:]' | grep -vwif file2
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53466876%2fbash-check-for-words-in-first-file-not-contained-in-second-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
4 Answers
4
active
oldest
votes
4 Answers
4
active
oldest
votes
active
oldest
votes
active
oldest
votes
paste + grep approach:
grep -Eiv "($(paste -sd'|' <file2.txt))" <(grep -wo 'w*' file1.txt)
The output:
is
a
file
several
lines
of
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
add a comment |
paste + grep approach:
grep -Eiv "($(paste -sd'|' <file2.txt))" <(grep -wo 'w*' file1.txt)
The output:
is
a
file
several
lines
of
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
add a comment |
paste + grep approach:
grep -Eiv "($(paste -sd'|' <file2.txt))" <(grep -wo 'w*' file1.txt)
The output:
is
a
file
several
lines
of
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
paste + grep approach:
grep -Eiv "($(paste -sd'|' <file2.txt))" <(grep -wo 'w*' file1.txt)
The output:
is
a
file
several
lines
of
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
answered Nov 25 '18 at 11:46
RomanPerekhrestRomanPerekhrest
55.8k32253
55.8k32253
add a comment |
add a comment |
I would try something more direct:
for A in `cat file1 | tr -d '[:punct:]'`; do grep -wq $A file2 || echo $A; done
flags used for grep: q for quiet (don't need output), w for word match
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
add a comment |
I would try something more direct:
for A in `cat file1 | tr -d '[:punct:]'`; do grep -wq $A file2 || echo $A; done
flags used for grep: q for quiet (don't need output), w for word match
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
add a comment |
I would try something more direct:
for A in `cat file1 | tr -d '[:punct:]'`; do grep -wq $A file2 || echo $A; done
flags used for grep: q for quiet (don't need output), w for word match
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
I would try something more direct:
for A in `cat file1 | tr -d '[:punct:]'`; do grep -wq $A file2 || echo $A; done
flags used for grep: q for quiet (don't need output), w for word match
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
edited Nov 25 '18 at 12:03
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
answered Nov 25 '18 at 11:27
OznOgOznOg
2,52411625
2,52411625
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
add a comment |
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
Wouldn't you still be troubled by punctuation?
– Lars
Nov 25 '18 at 11:38
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
you're right, punctuation removal is probably necessary as you mention it, I fix the proposition
– OznOg
Nov 25 '18 at 12:03
add a comment |
One in awk:
$ awk -F"[^A-Za-z]+" ' # anything but a letter is a field delimiter
NR==FNR { # process the word list
a[tolower($0)]
next
}
{
for(i=1;i<=NF;i++) # loop all fields
if(!(tolower($i) in a)) # if word was not in the word list
print $i # print it. duplicates are printed also.
}' another_file txt_file
Output:
is
a
file
several
lines
of
grep:
$ grep -vwi -f another_file <(cat txt_file | tr -s -c '[a-zA-Z]' 'n')
is
a
file
several
lines
of
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
add a comment |
One in awk:
$ awk -F"[^A-Za-z]+" ' # anything but a letter is a field delimiter
NR==FNR { # process the word list
a[tolower($0)]
next
}
{
for(i=1;i<=NF;i++) # loop all fields
if(!(tolower($i) in a)) # if word was not in the word list
print $i # print it. duplicates are printed also.
}' another_file txt_file
Output:
is
a
file
several
lines
of
grep:
$ grep -vwi -f another_file <(cat txt_file | tr -s -c '[a-zA-Z]' 'n')
is
a
file
several
lines
of
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
add a comment |
One in awk:
$ awk -F"[^A-Za-z]+" ' # anything but a letter is a field delimiter
NR==FNR { # process the word list
a[tolower($0)]
next
}
{
for(i=1;i<=NF;i++) # loop all fields
if(!(tolower($i) in a)) # if word was not in the word list
print $i # print it. duplicates are printed also.
}' another_file txt_file
Output:
is
a
file
several
lines
of
grep:
$ grep -vwi -f another_file <(cat txt_file | tr -s -c '[a-zA-Z]' 'n')
is
a
file
several
lines
of
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
One in awk:
$ awk -F"[^A-Za-z]+" ' # anything but a letter is a field delimiter
NR==FNR { # process the word list
a[tolower($0)]
next
}
{
for(i=1;i<=NF;i++) # loop all fields
if(!(tolower($i) in a)) # if word was not in the word list
print $i # print it. duplicates are printed also.
}' another_file txt_file
Output:
is
a
file
several
lines
of
grep:
$ grep -vwi -f another_file <(cat txt_file | tr -s -c '[a-zA-Z]' 'n')
is
a
file
several
lines
of
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
edited Nov 25 '18 at 12:13
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
answered Nov 25 '18 at 11:42
James BrownJames Brown
19.2k31735
19.2k31735
add a comment |
add a comment |
This pipeline will take the original file, replace spaces with newlines, convert to lowercase, then use grep to filter (-v) full words (-w) case insensitive (-i) using the lines in the given file (-f file2):
cat file1 | tr ' ' 'n' | tr '[:upper:]' '[:lower:]' | grep -vwif file2
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
add a comment |
This pipeline will take the original file, replace spaces with newlines, convert to lowercase, then use grep to filter (-v) full words (-w) case insensitive (-i) using the lines in the given file (-f file2):
cat file1 | tr ' ' 'n' | tr '[:upper:]' '[:lower:]' | grep -vwif file2
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
add a comment |
This pipeline will take the original file, replace spaces with newlines, convert to lowercase, then use grep to filter (-v) full words (-w) case insensitive (-i) using the lines in the given file (-f file2):
cat file1 | tr ' ' 'n' | tr '[:upper:]' '[:lower:]' | grep -vwif file2
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
This pipeline will take the original file, replace spaces with newlines, convert to lowercase, then use grep to filter (-v) full words (-w) case insensitive (-i) using the lines in the given file (-f file2):
cat file1 | tr ' ' 'n' | tr '[:upper:]' '[:lower:]' | grep -vwif file2
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
edited Nov 25 '18 at 16:53
Steven Kryskalla
10.7k12936
10.7k12936
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
answered Nov 25 '18 at 12:59
saurav omarsaurav omar
767
767
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
add a comment |
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
@Lars is it worked for you?
– saurav omar
Nov 25 '18 at 18:09
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53466876%2fbash-check-for-words-in-first-file-not-contained-in-second-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What version of
commare you using? Mine doesn't support-iand expects both input files to be sorted.– choroba
Nov 25 '18 at 11:21
2
Please add your desired output for that sample input to your question.
– Cyrus
Nov 25 '18 at 11:25
@Cyrus I added the desired output, hope that helps
– Lars
Nov 25 '18 at 11:35
@choroba I don't know how to find out the version. When I do
man commit has the date January 26, 2005 at the bottom.– Lars
Nov 25 '18 at 11:36
comm --versioncould tell you more.– choroba
Nov 25 '18 at 11:37