Pyspark: How to convert a spark dataframe to json and save it as json file?
I am trying to convert my pyspark sql dataframe to json and then save as a file.
df_final = df_final.union(join_df)
df_final contains the value as such:
I tried something like this. But it created a invalid json.
df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)
{"Variable":"Col1","Min":"20","Max":"30"}
{"Variable":"Col2","Min":"25,"Max":"40"}
My expected file should have data as below:
[
{"Variable":"Col1",
"Min":"20",
"Max":"30"},
{"Variable":"Col2",
"Min":"25,
"Max":"40"}]
python-3.x pyspark apache-spark-sql pyspark-sql
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
add a comment |
I am trying to convert my pyspark sql dataframe to json and then save as a file.
df_final = df_final.union(join_df)
df_final contains the value as such:
I tried something like this. But it created a invalid json.
df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)
{"Variable":"Col1","Min":"20","Max":"30"}
{"Variable":"Col2","Min":"25,"Max":"40"}
My expected file should have data as below:
[
{"Variable":"Col1",
"Min":"20",
"Max":"30"},
{"Variable":"Col2",
"Min":"25,
"Max":"40"}]
python-3.x pyspark apache-spark-sql pyspark-sql
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
trydf.toJSON()
– Bala
Nov 22 '18 at 10:09
add a comment |
I am trying to convert my pyspark sql dataframe to json and then save as a file.
df_final = df_final.union(join_df)
df_final contains the value as such:
I tried something like this. But it created a invalid json.
df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)
{"Variable":"Col1","Min":"20","Max":"30"}
{"Variable":"Col2","Min":"25,"Max":"40"}
My expected file should have data as below:
[
{"Variable":"Col1",
"Min":"20",
"Max":"30"},
{"Variable":"Col2",
"Min":"25,
"Max":"40"}]
python-3.x pyspark apache-spark-sql pyspark-sql
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
I am trying to convert my pyspark sql dataframe to json and then save as a file.
df_final = df_final.union(join_df)
df_final contains the value as such:
I tried something like this. But it created a invalid json.
df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)
{"Variable":"Col1","Min":"20","Max":"30"}
{"Variable":"Col2","Min":"25,"Max":"40"}
My expected file should have data as below:
[
{"Variable":"Col1",
"Min":"20",
"Max":"30"},
{"Variable":"Col2",
"Min":"25,
"Max":"40"}]
python-3.x pyspark apache-spark-sql pyspark-sql
python-3.x pyspark apache-spark-sql pyspark-sql
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
asked Nov 22 '18 at 8:10
Shankar PandaShankar Panda
1741114
1741114
trydf.toJSON()
– Bala
Nov 22 '18 at 10:09
add a comment |
trydf.toJSON()
– Bala
Nov 22 '18 at 10:09
try
df.toJSON()– Bala
Nov 22 '18 at 10:09
try
df.toJSON()– Bala
Nov 22 '18 at 10:09
add a comment |
3 Answers
3
active
oldest
votes
A solution can be using collect and then using json.dump:
import json
collected_df = df_final.collect()
with open(data_output_file + 'createjson.json', 'w') as outfile:
json.dump(data, outfile)
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
add a comment |
If you want to use spark to process result as json files, I think that your output schema is right in hdfs.
And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :
with open('data.json') as f:
data = json.load(f)
You should try to read data line by line:
data =
with open("data.json",'r') as datafile:
for line in datafile:
data.append(json.loads(line))
and you can use pandas to create dataframe :
df = pd.DataFrame(data)
answered Nov 23 '18 at 2:49
chilunchilun
948
add a comment |
For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.
df_final.coalesce(1).write.format('json').save('/path/file_name.json')
and still you want to convert your datafram into json then you can used
df_final.toJSON().
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426420%2fpyspark-how-to-convert-a-spark-dataframe-to-json-and-save-it-as-json-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
A solution can be using collect and then using json.dump:
import json
collected_df = df_final.collect()
with open(data_output_file + 'createjson.json', 'w') as outfile:
json.dump(data, outfile)
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
add a comment |
A solution can be using collect and then using json.dump:
import json
collected_df = df_final.collect()
with open(data_output_file + 'createjson.json', 'w') as outfile:
json.dump(data, outfile)
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
add a comment |
A solution can be using collect and then using json.dump:
import json
collected_df = df_final.collect()
with open(data_output_file + 'createjson.json', 'w') as outfile:
json.dump(data, outfile)
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
A solution can be using collect and then using json.dump:
import json
collected_df = df_final.collect()
with open(data_output_file + 'createjson.json', 'w') as outfile:
json.dump(data, outfile)
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
answered Nov 22 '18 at 9:40
OmGOmG
7,97852743
7,97852743
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
add a comment |
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs
– Shankar Panda
Nov 22 '18 at 9:41
add a comment |
If you want to use spark to process result as json files, I think that your output schema is right in hdfs.
And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :
with open('data.json') as f:
data = json.load(f)
You should try to read data line by line:
data =
with open("data.json",'r') as datafile:
for line in datafile:
data.append(json.loads(line))
and you can use pandas to create dataframe :
df = pd.DataFrame(data)
answered Nov 23 '18 at 2:49
chilunchilun
948
add a comment |
If you want to use spark to process result as json files, I think that your output schema is right in hdfs.
And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :
with open('data.json') as f:
data = json.load(f)
You should try to read data line by line:
data =
with open("data.json",'r') as datafile:
for line in datafile:
data.append(json.loads(line))
and you can use pandas to create dataframe :
df = pd.DataFrame(data)
answered Nov 23 '18 at 2:49
chilunchilun
948
add a comment |
If you want to use spark to process result as json files, I think that your output schema is right in hdfs.
And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :
with open('data.json') as f:
data = json.load(f)
You should try to read data line by line:
data =
with open("data.json",'r') as datafile:
for line in datafile:
data.append(json.loads(line))
and you can use pandas to create dataframe :
df = pd.DataFrame(data)
answered Nov 23 '18 at 2:49
chilunchilun
948
If you want to use spark to process result as json files, I think that your output schema is right in hdfs.
And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :
with open('data.json') as f:
data = json.load(f)
You should try to read data line by line:
data =
with open("data.json",'r') as datafile:
for line in datafile:
data.append(json.loads(line))
and you can use pandas to create dataframe :
df = pd.DataFrame(data)
answered Nov 23 '18 at 2:49
chilunchilun
948
answered Nov 23 '18 at 2:49
chilunchilun
948
answered Nov 23 '18 at 2:49
chilunchilun
948
answered Nov 23 '18 at 2:49
chilunchilun
948
948
add a comment |
add a comment |
For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.
df_final.coalesce(1).write.format('json').save('/path/file_name.json')
and still you want to convert your datafram into json then you can used
df_final.toJSON().
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
add a comment |
For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.
df_final.coalesce(1).write.format('json').save('/path/file_name.json')
and still you want to convert your datafram into json then you can used
df_final.toJSON().
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
add a comment |
For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.
df_final.coalesce(1).write.format('json').save('/path/file_name.json')
and still you want to convert your datafram into json then you can used
df_final.toJSON().
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.
df_final.coalesce(1).write.format('json').save('/path/file_name.json')
and still you want to convert your datafram into json then you can used
df_final.toJSON().
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
answered Nov 23 '18 at 7:40
Sahil DesaiSahil Desai
1,3893822
1,3893822
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
add a comment |
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces
– Shankar Panda
Nov 23 '18 at 7:48
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426420%2fpyspark-how-to-convert-a-spark-dataframe-to-json-and-save-it-as-json-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
try
df.toJSON()– Bala
Nov 22 '18 at 10:09