apache spark mlib loading logistic regression model from http source
0
i want have a model on an extern http source, that I want to load in my spark streaming application for prediction of incoming data. Since the data is coming from different producers, I have to load individual models, depending on the data.
Spark runs on DCOS-mesos cluster and the model is a folder with data and metadata and parquet files.
Load directly from http is not possible, it needs "file:..."
I tried to download them into "./testmodel/" path, it lands also in the sandbox, but is not loading in the model because of the wrong path with following exception: Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/mnt/mesos/sandbox/testmodel/metadata
def fileDownloader(url: String, filename: String) = {
new URL(url) #> new File(filename) !!
}...
val modelFolder: File = new File("./testModel");
modelFolder.mkdir();
val modelDataFolder: File = new File("./testModel/data")
modelDataFolder.mkdir();
val modelMetaDataFolder: File = new File("./testModel/metadata");
modelMetaDataFolder.mkdir();
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/._SUCCESS.crc", "./testModel/data/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc", "./testModel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/_SUCCESS", "./testModel/data/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet", "./testModel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/._SUCCESS.crc", "./testModel/metadata/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/.part-00000.crc", "./testModel/metadata/.part-00000.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/_SUCCESS", "./testModel/metadata/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/part-00000", "./testModel/metadata/part-00000");
val model = LogisticRegressionModel.load(sc, "./testModel");
Thanks in advice
apache-spark model spark-streaming apache-spark-mllib
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
add a comment |
0
i want have a model on an extern http source, that I want to load in my spark streaming application for prediction of incoming data. Since the data is coming from different producers, I have to load individual models, depending on the data.
Spark runs on DCOS-mesos cluster and the model is a folder with data and metadata and parquet files.
Load directly from http is not possible, it needs "file:..."
I tried to download them into "./testmodel/" path, it lands also in the sandbox, but is not loading in the model because of the wrong path with following exception: Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/mnt/mesos/sandbox/testmodel/metadata
def fileDownloader(url: String, filename: String) = {
new URL(url) #> new File(filename) !!
}...
val modelFolder: File = new File("./testModel");
modelFolder.mkdir();
val modelDataFolder: File = new File("./testModel/data")
modelDataFolder.mkdir();
val modelMetaDataFolder: File = new File("./testModel/metadata");
modelMetaDataFolder.mkdir();
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/._SUCCESS.crc", "./testModel/data/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc", "./testModel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/_SUCCESS", "./testModel/data/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet", "./testModel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/._SUCCESS.crc", "./testModel/metadata/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/.part-00000.crc", "./testModel/metadata/.part-00000.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/_SUCCESS", "./testModel/metadata/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/part-00000", "./testModel/metadata/part-00000");
val model = LogisticRegressionModel.load(sc, "./testModel");
Thanks in advice
apache-spark model spark-streaming apache-spark-mllib
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
1
That happens because model should be placed on a distributed file system, as it is loaded usingDataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFilesor archives can do that).
– user10465355
Nov 24 '18 at 11:32
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09
add a comment |
0
0
0
i want have a model on an extern http source, that I want to load in my spark streaming application for prediction of incoming data. Since the data is coming from different producers, I have to load individual models, depending on the data.
Spark runs on DCOS-mesos cluster and the model is a folder with data and metadata and parquet files.
Load directly from http is not possible, it needs "file:..."
I tried to download them into "./testmodel/" path, it lands also in the sandbox, but is not loading in the model because of the wrong path with following exception: Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/mnt/mesos/sandbox/testmodel/metadata
def fileDownloader(url: String, filename: String) = {
new URL(url) #> new File(filename) !!
}...
val modelFolder: File = new File("./testModel");
modelFolder.mkdir();
val modelDataFolder: File = new File("./testModel/data")
modelDataFolder.mkdir();
val modelMetaDataFolder: File = new File("./testModel/metadata");
modelMetaDataFolder.mkdir();
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/._SUCCESS.crc", "./testModel/data/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc", "./testModel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/_SUCCESS", "./testModel/data/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet", "./testModel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/._SUCCESS.crc", "./testModel/metadata/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/.part-00000.crc", "./testModel/metadata/.part-00000.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/_SUCCESS", "./testModel/metadata/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/part-00000", "./testModel/metadata/part-00000");
val model = LogisticRegressionModel.load(sc, "./testModel");
Thanks in advice
apache-spark model spark-streaming apache-spark-mllib
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
i want have a model on an extern http source, that I want to load in my spark streaming application for prediction of incoming data. Since the data is coming from different producers, I have to load individual models, depending on the data.
Spark runs on DCOS-mesos cluster and the model is a folder with data and metadata and parquet files.
Load directly from http is not possible, it needs "file:..."
I tried to download them into "./testmodel/" path, it lands also in the sandbox, but is not loading in the model because of the wrong path with following exception: Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/mnt/mesos/sandbox/testmodel/metadata
def fileDownloader(url: String, filename: String) = {
new URL(url) #> new File(filename) !!
}...
val modelFolder: File = new File("./testModel");
modelFolder.mkdir();
val modelDataFolder: File = new File("./testModel/data")
modelDataFolder.mkdir();
val modelMetaDataFolder: File = new File("./testModel/metadata");
modelMetaDataFolder.mkdir();
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/._SUCCESS.crc", "./testModel/data/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc", "./testModel/data/.part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/_SUCCESS", "./testModel/data/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet", "./testModel/data/part-00000-beca47f5-4fa8-4af8-ba76-21be4e0c4763-c000.snappy.parquet");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/._SUCCESS.crc", "./testModel/metadata/._SUCCESS.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/.part-00000.crc", "./testModel/metadata/.part-00000.crc");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/_SUCCESS", "./testModel/metadata/_SUCCESS");
fileDownloader("http://extern-host.com/sparkmodel/testmodel/metadata/part-00000", "./testModel/metadata/part-00000");
val model = LogisticRegressionModel.load(sc, "./testModel");
Thanks in advice
apache-spark model spark-streaming apache-spark-mllib
apache-spark model spark-streaming apache-spark-mllib
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
asked Nov 23 '18 at 22:04
Roman ZounRoman Zoun
12
12
1
That happens because model should be placed on a distributed file system, as it is loaded usingDataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFilesor archives can do that).
– user10465355
Nov 24 '18 at 11:32
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09
add a comment |
1
That happens because model should be placed on a distributed file system, as it is loaded usingDataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFilesor archives can do that).
– user10465355
Nov 24 '18 at 11:32
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09
1
1
That happens because model should be placed on a distributed file system, as it is loaded using
DataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFiles or archives can do that).– user10465355
Nov 24 '18 at 11:32
That happens because model should be placed on a distributed file system, as it is loaded using
DataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFiles or archives can do that).– user10465355
Nov 24 '18 at 11:32
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453378%2fapache-spark-mlib-loading-logistic-regression-model-from-http-source%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453378%2fapache-spark-mlib-loading-logistic-regression-model-from-http-source%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
That happens because model should be placed on a distributed file system, as it is loaded using
DataFrameReader. So downloading is only the first step, the second one should be moving it to storage that can be used by each machine. You could also place a local copy on each node (SparkFilesor archives can do that).– user10465355
Nov 24 '18 at 11:32
Thank you, now I add the files and folders using sparkContext.add("http://...") as result I have the model on each executor...but somehow I got WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.0.20, executor 3): java.io.FileNotFoundException: File file:/tmp/spark-23f6d5c1-7c0d-48c0-b243-dbf5551b8228/userFiles-ca4025a5-9cae-4a4e-8518-c3d76cad06e8/testmodel/metadata/part-00000 does not exist but it should be there, I tested using the new File().exists function :(
– Roman Zoun
Nov 24 '18 at 19:49
Maybe its not on every executer? How can i check stuff like this?
– Roman Zoun
Nov 24 '18 at 20:09