How to use Column.isin with list?
45
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
The code above throws the following exception.
Exception in thread "main" java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(a, b, c)
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.Column.isin(Column.scala:642)
Below is my attempt to fix it. It compiles and runs but doesn't return any match. Not sure why.
val items = List("a", "b", "c").mkString(""","","",""")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
scala apache-spark apache-spark-sql
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
add a comment |
45
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
The code above throws the following exception.
Exception in thread "main" java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(a, b, c)
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.Column.isin(Column.scala:642)
Below is my attempt to fix it. It compiles and runs but doesn't return any match. Not sure why.
val items = List("a", "b", "c").mkString(""","","",""")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
scala apache-spark apache-spark-sql
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
add a comment |
45
45
45
4
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
The code above throws the following exception.
Exception in thread "main" java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(a, b, c)
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.Column.isin(Column.scala:642)
Below is my attempt to fix it. It compiles and runs but doesn't return any match. Not sure why.
val items = List("a", "b", "c").mkString(""","","",""")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
scala apache-spark apache-spark-sql
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
The code above throws the following exception.
Exception in thread "main" java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(a, b, c)
at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:49)
at org.apache.spark.sql.functions$.lit(functions.scala:89)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:642)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.Column.isin(Column.scala:642)
Below is my attempt to fix it. It compiles and runs but doesn't return any match. Not sure why.
val items = List("a", "b", "c").mkString(""","","",""")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items))
.collect
.foreach(println)
scala apache-spark apache-spark-sql
scala apache-spark apache-spark-sql
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
edited May 18 '18 at 19:13
Jacek Laskowski
44.5k18132267
44.5k18132267
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
asked Sep 13 '15 at 16:32
NabeghNabegh
1,37931725
1,37931725
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
85
According to documentation, isin takes a vararg, not a list. List is actually a confusing name here. You can try converting your List to vararg like this:
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items:_*))
.collect
.foreach(println)
Your variant with mkString compiles, because one single String is also a vararg (with number of arguments equal to 1), but it is proably not what you want to achieve.
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
add a comment |
8
It worked like this in Java Api (Java 8)
.isin(sampleListName.stream().toArray(String::new))));
sampleListName is a List
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
add a comment |
-1
As Tomalak has mentioned it :
isin(java.lang.Object... list)
A boolean expression that is evaluated to true if the value
of this expression is contained by the evaluated values of the arguments.
Therefore, you just could fix this making the following change :
val items = List("a", "b", "c").map(c => s""""$c"""")
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
1
Why the map? My is that.filter($"c1".isin(List("a", "b", "c")))would work.
– Tomalak
Sep 13 '15 at 17:00
Considering your codeList("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.
– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f32551919%2fhow-to-use-column-isin-with-list%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
85
According to documentation, isin takes a vararg, not a list. List is actually a confusing name here. You can try converting your List to vararg like this:
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items:_*))
.collect
.foreach(println)
Your variant with mkString compiles, because one single String is also a vararg (with number of arguments equal to 1), but it is proably not what you want to achieve.
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
add a comment |
85
According to documentation, isin takes a vararg, not a list. List is actually a confusing name here. You can try converting your List to vararg like this:
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items:_*))
.collect
.foreach(println)
Your variant with mkString compiles, because one single String is also a vararg (with number of arguments equal to 1), but it is proably not what you want to achieve.
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
add a comment |
85
85
85
According to documentation, isin takes a vararg, not a list. List is actually a confusing name here. You can try converting your List to vararg like this:
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items:_*))
.collect
.foreach(println)
Your variant with mkString compiles, because one single String is also a vararg (with number of arguments equal to 1), but it is proably not what you want to achieve.
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
According to documentation, isin takes a vararg, not a list. List is actually a confusing name here. You can try converting your List to vararg like this:
val items = List("a", "b", "c")
sqlContext.sql("select c1 from table")
.filter($"c1".isin(items:_*))
.collect
.foreach(println)
Your variant with mkString compiles, because one single String is also a vararg (with number of arguments equal to 1), but it is proably not what you want to achieve.
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
edited Oct 8 '15 at 6:33
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
answered Sep 14 '15 at 8:19
NiemandNiemand
4,93142862
4,93142862
add a comment |
add a comment |
8
It worked like this in Java Api (Java 8)
.isin(sampleListName.stream().toArray(String::new))));
sampleListName is a List
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
add a comment |
8
It worked like this in Java Api (Java 8)
.isin(sampleListName.stream().toArray(String::new))));
sampleListName is a List
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
add a comment |
8
8
8
It worked like this in Java Api (Java 8)
.isin(sampleListName.stream().toArray(String::new))));
sampleListName is a List
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
It worked like this in Java Api (Java 8)
.isin(sampleListName.stream().toArray(String::new))));
sampleListName is a List
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
answered Sep 9 '16 at 17:55
AnandkumarAnandkumar
48946
48946
add a comment |
add a comment |
-1
As Tomalak has mentioned it :
isin(java.lang.Object... list)
A boolean expression that is evaluated to true if the value
of this expression is contained by the evaluated values of the arguments.
Therefore, you just could fix this making the following change :
val items = List("a", "b", "c").map(c => s""""$c"""")
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
1
Why the map? My is that.filter($"c1".isin(List("a", "b", "c")))would work.
– Tomalak
Sep 13 '15 at 17:00
Considering your codeList("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.
– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
add a comment |
-1
As Tomalak has mentioned it :
isin(java.lang.Object... list)
A boolean expression that is evaluated to true if the value
of this expression is contained by the evaluated values of the arguments.
Therefore, you just could fix this making the following change :
val items = List("a", "b", "c").map(c => s""""$c"""")
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
1
Why the map? My is that.filter($"c1".isin(List("a", "b", "c")))would work.
– Tomalak
Sep 13 '15 at 17:00
Considering your codeList("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.
– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
add a comment |
-1
-1
-1
As Tomalak has mentioned it :
isin(java.lang.Object... list)
A boolean expression that is evaluated to true if the value
of this expression is contained by the evaluated values of the arguments.
Therefore, you just could fix this making the following change :
val items = List("a", "b", "c").map(c => s""""$c"""")
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
As Tomalak has mentioned it :
isin(java.lang.Object... list)
A boolean expression that is evaluated to true if the value
of this expression is contained by the evaluated values of the arguments.
Therefore, you just could fix this making the following change :
val items = List("a", "b", "c").map(c => s""""$c"""")
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
edited Sep 13 '15 at 17:37
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
answered Sep 13 '15 at 16:45
Francis TothFrancis Toth
973718
973718
1
Why the map? My is that.filter($"c1".isin(List("a", "b", "c")))would work.
– Tomalak
Sep 13 '15 at 17:00
Considering your codeList("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.
– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
add a comment |
1
Why the map? My is that.filter($"c1".isin(List("a", "b", "c")))would work.
– Tomalak
Sep 13 '15 at 17:00
Considering your codeList("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.
– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
1
1
Why the map? My is that
.filter($"c1".isin(List("a", "b", "c"))) would work.– Tomalak
Sep 13 '15 at 17:00
Why the map? My is that
.filter($"c1".isin(List("a", "b", "c"))) would work.– Tomalak
Sep 13 '15 at 17:00
Considering your code
List("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.– Francis Toth
Sep 13 '15 at 17:10
Considering your code
List("a", "b", "c").mkString(""","","","""), I assumed that you wanted to surround each item by double quotes. The map does the same thing.– Francis Toth
Sep 13 '15 at 17:10
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
Not my code. ;)
– Tomalak
Sep 13 '15 at 17:14
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f32551919%2fhow-to-use-column-isin-with-list%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
