Can't initialize graph on Datastax using Spark
up vote
0
down vote
favorite
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 at 16:30
Alex Ott
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
|
show 3 more comments
up vote
0
down vote
favorite
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 at 16:30
Alex Ott
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17
|
show 3 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 at 16:30
Alex Ott
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
I'm trying to initialize my Datastax graph using Spark as follow :
val graphBuilder = spark.dseGraph("GRAPH_NAME")
but I have the following exception :
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl
at com.datastax.bdp.graph.spark.graphframe.DseGraphFrameBuilder$.apply(DseGraphFrameBuilder.scala:257)
at com.datastax.bdp.graph.spark.graphframe.SparkSessionFunctions.dseGraph(SparkSessionFunctions.scala:20)
I search a dataxtax jar that containscom/datastax/bdp/graph/impl but I didn't find it.
Any help is really appreciated. Thanks in Advance!
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
apache-spark cassandra datastax-enterprise-graph dse-graph-frames
edited Nov 20 at 16:30
Alex Ott
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
edited Nov 20 at 16:30
Alex Ott
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
edited Nov 20 at 16:30
Alex Ott
26.8k35072
edited Nov 20 at 16:30
Alex Ott
26.8k35072
edited Nov 20 at 16:30
Alex Ott
26.8k35072
26.8k35072
asked Nov 20 at 15:52
BeginMoh
5019
asked Nov 20 at 15:52
BeginMoh
5019
asked Nov 20 at 15:52
BeginMoh
5019
5019
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17
|
show 3 more comments
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
how do you submit job? viadse spark-submitor you're using separate spark?
– Alex Ott
Nov 20 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
how do you submit job? via
dse spark-submit or you're using separate spark?– Alex Ott
Nov 20 at 16:49
how do you submit job? via
dse spark-submit or you're using separate spark?– Alex Ott
Nov 20 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17
|
show 3 more comments
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 at 9:25
Alex Ott
26.8k35072
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53396768%2fcant-initialize-graph-on-datastax-using-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 at 9:25
Alex Ott
26.8k35072
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
add a comment |
up vote
0
down vote
accepted
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 at 9:25
Alex Ott
26.8k35072
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 at 9:25
Alex Ott
26.8k35072
To use DseGraphFrames in your program running on external Spark, you need to link with so-called BYOS jar. This could be done as following (for Maven):
<dependency>
<groupId>com.datastax.dse</groupId>
<artifactId>dse-byos_2.11</artifactId>
<version>6.0.4</version>
</dependency>
And add DataStax repositories:
<repositories>
<repository>
<id>DataStax-Repo</id>
<url>https://repo.datastax.com/public-repos/</url>
</repository>
</repositories>
And if you check the jar fetched from DataStax repository, it contains necessary file:
unzip -l dse-byos_2.11-6.0.4.jar|grep AbstractVertexIdExternalImpl
2839 10-06-2018 15:22 com/datastax/bdp/graph/impl/element/vertex/id/AbstractVertexIdExternalImpl.class
answered Nov 21 at 9:25
Alex Ott
26.8k35072
answered Nov 21 at 9:25
Alex Ott
26.8k35072
answered Nov 21 at 9:25
Alex Ott
26.8k35072
answered Nov 21 at 9:25
Alex Ott
26.8k35072
26.8k35072
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
add a comment |
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
it works for me , i also added "dse-graph-frames" dependencies i used DseGraphFrameBuilder.dseGraph(cassandraConf.GRAPH_NAME, spark) to get DseGraphFrame . Thanks a lot
– BeginMoh
Nov 21 at 13:59
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
BYOS jar contains everything, including dse-graph-frame dependency...
– Alex Ott
Nov 21 at 14:18
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
Hi Alex , after adding byos dependencies i have a problem when i'm runing my spark jobs :18/12/07 14:12:00 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.commons.lang.SerializationException: java.lang.ClassNotFoundException: com.datastax.bdp.graph.spark.SerializableSchema at org.apache.commons.lang.SerializationUtils.deserialize(SerializationUtils.java:166) Do you have any idea about this error plz ?
– BeginMoh
Dec 7 at 13:39
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53396768%2fcant-initialize-graph-on-datastax-using-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
what version of DSE, how do you submit application?
– Alex Ott
Nov 20 at 16:29
version Dse : 6.0.4 , i tested with 5.0.4 also
– BeginMoh
Nov 20 at 16:47
how do you submit job? via
dse spark-submitor you're using separate spark?– Alex Ott
Nov 20 at 16:49
i'm using a separet spark
– BeginMoh
Nov 21 at 8:36
Please show what do you link to your application (Maven/SBT/...) do you use BYOS jar, or only DseGraphFrames?
– Alex Ott
Nov 21 at 9:17