java.lang.Long and scala.Long
1
I don't know what happened in my code...
Logs are here.
[error] blahblahSampleApp.scala:22:53: overloaded method value reduce with alternatives:
[error] (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and>
[error] (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long
[error] cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long)
[error] val sumHundred = sparkSession.range(start, end).reduce(_ + _)
When I ran this code in scala 2.11.12, spark 2.3.2 it works without any ERROR.
And same code in scala 2.12.7, spark 2.4.0 it doesn't works - what?
Anybody knows about this?
private val (start, end) = (1, 101)
def main(args: Array[String]): Unit = {
val sumHundred = sparkSession.range(start, end).reduce(_ + _)
logger.debug(f"Sum 1 to 100 = $sumHundred")
close()
}
There's a parent trait that builds sparkSession etc.
What I've tried:
- Explicit declaration of type:
private val (start: Long, end: Long) = ...
- Similar things in
reducecode.
What I know:
Perfectly compatiable between scala.Long and java.lang.Long
java scala apache-spark
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
|
show 1 more comment
1
I don't know what happened in my code...
Logs are here.
[error] blahblahSampleApp.scala:22:53: overloaded method value reduce with alternatives:
[error] (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and>
[error] (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long
[error] cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long)
[error] val sumHundred = sparkSession.range(start, end).reduce(_ + _)
When I ran this code in scala 2.11.12, spark 2.3.2 it works without any ERROR.
And same code in scala 2.12.7, spark 2.4.0 it doesn't works - what?
Anybody knows about this?
private val (start, end) = (1, 101)
def main(args: Array[String]): Unit = {
val sumHundred = sparkSession.range(start, end).reduce(_ + _)
logger.debug(f"Sum 1 to 100 = $sumHundred")
close()
}
There's a parent trait that builds sparkSession etc.
What I've tried:
- Explicit declaration of type:
private val (start: Long, end: Long) = ...
- Similar things in
reducecode.
What I know:
Perfectly compatiable between scala.Long and java.lang.Long
java scala apache-spark
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
1
As a workaround, maybe you can try passing(a,b) => Long.box(a + b)to reduce instead of_ + _
– gilad hoch
Nov 22 '18 at 7:05
@giladhoch what the... Thank you SO much you saved my day. I don't even know thatLong.boxthing.
– nullmari
Nov 22 '18 at 7:07
@giladhoch Why this happens? What's the difference between_+_andbox?
– nullmari
Nov 22 '18 at 7:11
1
Long.boxwraps the primitiveLongin the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)
– gilad hoch
Nov 22 '18 at 15:21
1
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08
|
show 1 more comment
1
1
1
I don't know what happened in my code...
Logs are here.
[error] blahblahSampleApp.scala:22:53: overloaded method value reduce with alternatives:
[error] (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and>
[error] (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long
[error] cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long)
[error] val sumHundred = sparkSession.range(start, end).reduce(_ + _)
When I ran this code in scala 2.11.12, spark 2.3.2 it works without any ERROR.
And same code in scala 2.12.7, spark 2.4.0 it doesn't works - what?
Anybody knows about this?
private val (start, end) = (1, 101)
def main(args: Array[String]): Unit = {
val sumHundred = sparkSession.range(start, end).reduce(_ + _)
logger.debug(f"Sum 1 to 100 = $sumHundred")
close()
}
There's a parent trait that builds sparkSession etc.
What I've tried:
- Explicit declaration of type:
private val (start: Long, end: Long) = ...
- Similar things in
reducecode.
What I know:
Perfectly compatiable between scala.Long and java.lang.Long
java scala apache-spark
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
I don't know what happened in my code...
Logs are here.
[error] blahblahSampleApp.scala:22:53: overloaded method value reduce with alternatives:
[error] (func: org.apache.spark.api.java.function.ReduceFunction[java.lang.Long])java.lang.Long <and>
[error] (func: (java.lang.Long, java.lang.Long) => java.lang.Long)java.lang.Long
[error] cannot be applied to ((java.lang.Long, java.lang.Long) => scala.Long)
[error] val sumHundred = sparkSession.range(start, end).reduce(_ + _)
When I ran this code in scala 2.11.12, spark 2.3.2 it works without any ERROR.
And same code in scala 2.12.7, spark 2.4.0 it doesn't works - what?
Anybody knows about this?
private val (start, end) = (1, 101)
def main(args: Array[String]): Unit = {
val sumHundred = sparkSession.range(start, end).reduce(_ + _)
logger.debug(f"Sum 1 to 100 = $sumHundred")
close()
}
There's a parent trait that builds sparkSession etc.
What I've tried:
- Explicit declaration of type:
private val (start: Long, end: Long) = ...
- Similar things in
reducecode.
What I know:
Perfectly compatiable between scala.Long and java.lang.Long
java scala apache-spark
java scala apache-spark
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
asked Nov 22 '18 at 6:46
nullmarinullmari
6516
6516
1
As a workaround, maybe you can try passing(a,b) => Long.box(a + b)to reduce instead of_ + _
– gilad hoch
Nov 22 '18 at 7:05
@giladhoch what the... Thank you SO much you saved my day. I don't even know thatLong.boxthing.
– nullmari
Nov 22 '18 at 7:07
@giladhoch Why this happens? What's the difference between_+_andbox?
– nullmari
Nov 22 '18 at 7:11
1
Long.boxwraps the primitiveLongin the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)
– gilad hoch
Nov 22 '18 at 15:21
1
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08
|
show 1 more comment
1
As a workaround, maybe you can try passing(a,b) => Long.box(a + b)to reduce instead of_ + _
– gilad hoch
Nov 22 '18 at 7:05
@giladhoch what the... Thank you SO much you saved my day. I don't even know thatLong.boxthing.
– nullmari
Nov 22 '18 at 7:07
@giladhoch Why this happens? What's the difference between_+_andbox?
– nullmari
Nov 22 '18 at 7:11
1
Long.boxwraps the primitiveLongin the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)
– gilad hoch
Nov 22 '18 at 15:21
1
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08
1
1
As a workaround, maybe you can try passing
(a,b) => Long.box(a + b) to reduce instead of _ + _– gilad hoch
Nov 22 '18 at 7:05
As a workaround, maybe you can try passing
(a,b) => Long.box(a + b) to reduce instead of _ + _– gilad hoch
Nov 22 '18 at 7:05
@giladhoch what the... Thank you SO much you saved my day. I don't even know that
Long.box thing.– nullmari
Nov 22 '18 at 7:07
@giladhoch what the... Thank you SO much you saved my day. I don't even know that
Long.box thing.– nullmari
Nov 22 '18 at 7:07
@giladhoch Why this happens? What's the difference between
_+_ and box?– nullmari
Nov 22 '18 at 7:11
@giladhoch Why this happens? What's the difference between
_+_ and box?– nullmari
Nov 22 '18 at 7:11
1
1
Long.box wraps the primitive Long in the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)– gilad hoch
Nov 22 '18 at 15:21
Long.box wraps the primitive Long in the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)– gilad hoch
Nov 22 '18 at 15:21
1
1
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53425268%2fjava-lang-long-and-scala-long%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53425268%2fjava-lang-long-and-scala-long%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
As a workaround, maybe you can try passing
(a,b) => Long.box(a + b)to reduce instead of_ + _– gilad hoch
Nov 22 '18 at 7:05
@giladhoch what the... Thank you SO much you saved my day. I don't even know that
Long.boxthing.– nullmari
Nov 22 '18 at 7:07
@giladhoch Why this happens? What's the difference between
_+_andbox?– nullmari
Nov 22 '18 at 7:11
1
Long.boxwraps the primitiveLongin the corresponding java Object. Spark API should seamlessly do the conversion for you, but since the error message appears like they don't, it wouldn't hurt to do it yourself. I guess this counts as a bug which you can open an issue for on the spark repo (if doesn't already exist)– gilad hoch
Nov 22 '18 at 15:21
1
Sure. Glad I could help out :)
– gilad hoch
Nov 23 '18 at 8:08