Pyspark: How to convert a spark dataframe to json and save it as json file?

I am trying to convert my pyspark sql dataframe to json and then save as a file.

df_final = df_final.union(join_df)

df_final contains the value as such:

enter image description here

I tried something like this. But it created a invalid json.

df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)



{"Variable":"Col1","Min":"20","Max":"30"}

{"Variable":"Col2","Min":"25,"Max":"40"}

My expected file should have data as below:

[

{"Variable":"Col1",

"Min":"20",

"Max":"30"},

{"Variable":"Col2",

"Min":"25,

"Max":"40"}]

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

try df.toJSON()

– Bala
Nov 22 '18 at 10:09

add a comment |

I am trying to convert my pyspark sql dataframe to json and then save as a file.

df_final = df_final.union(join_df)

df_final contains the value as such:

enter image description here

I tried something like this. But it created a invalid json.

df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)



{"Variable":"Col1","Min":"20","Max":"30"}

{"Variable":"Col2","Min":"25,"Max":"40"}

My expected file should have data as below:

[

{"Variable":"Col1",

"Min":"20",

"Max":"30"},

{"Variable":"Col2",

"Min":"25,

"Max":"40"}]

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

try df.toJSON()

– Bala
Nov 22 '18 at 10:09

add a comment |

I am trying to convert my pyspark sql dataframe to json and then save as a file.

df_final = df_final.union(join_df)

df_final contains the value as such:

enter image description here

I tried something like this. But it created a invalid json.

df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)



{"Variable":"Col1","Min":"20","Max":"30"}

{"Variable":"Col2","Min":"25,"Max":"40"}

My expected file should have data as below:

[

{"Variable":"Col1",

"Min":"20",

"Max":"30"},

{"Variable":"Col2",

"Min":"25,

"Max":"40"}]

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

I am trying to convert my pyspark sql dataframe to json and then save as a file.

df_final = df_final.union(join_df)

df_final contains the value as such:

enter image description here

I tried something like this. But it created a invalid json.

df_final.coalesce(1).write.format('json').save(data_output_file+"createjson.json", overwrite=True)



{"Variable":"Col1","Min":"20","Max":"30"}

{"Variable":"Col2","Min":"25,"Max":"40"}

My expected file should have data as below:

[

{"Variable":"Col1",

"Min":"20",

"Max":"30"},

{"Variable":"Col2",

"Min":"25,

"Max":"40"}]

python-3.x pyspark apache-spark-sql pyspark-sql

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

asked Nov 22 '18 at 8:10

Shankar Panda

1741114

try df.toJSON()

– Bala
Nov 22 '18 at 10:09

add a comment |

try df.toJSON()

– Bala
Nov 22 '18 at 10:09

try df.toJSON()

– Bala
Nov 22 '18 at 10:09

add a comment |

3 Answers
3

active

oldest

votes

A solution can be using collect and then using json.dump:

import json

collected_df = df_final.collect()

with open(data_output_file + 'createjson.json', 'w') as outfile:

    json.dump(data, outfile)

answered Nov 22 '18 at 9:40

OmG

7,97852743

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

add a comment |

If you want to use spark to process result as json files, I think that your output schema is right in hdfs.

And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :

with open('data.json') as f:

  data = json.load(f)

You should try to read data line by line:

data = 

with open("data.json",'r') as datafile:

  for line in datafile:

    data.append(json.loads(line))

and you can use pandas to create dataframe :

df = pd.DataFrame(data)

answered Nov 23 '18 at 2:49

chilun

948

add a comment |

For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.

df_final.coalesce(1).write.format('json').save('/path/file_name.json')

and still you want to convert your datafram into json then you can used
df_final.toJSON().

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53426420%2fpyspark-how-to-convert-a-spark-dataframe-to-json-and-save-it-as-json-file%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

3 Answers
3

active

oldest

votes

3 Answers
3

active

oldest

votes

A solution can be using collect and then using json.dump:

import json

collected_df = df_final.collect()

with open(data_output_file + 'createjson.json', 'w') as outfile:

    json.dump(data, outfile)

answered Nov 22 '18 at 9:40

OmG

7,97852743

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

add a comment |

A solution can be using collect and then using json.dump:

import json

collected_df = df_final.collect()

with open(data_output_file + 'createjson.json', 'w') as outfile:

    json.dump(data, outfile)

answered Nov 22 '18 at 9:40

OmG

7,97852743

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

add a comment |

A solution can be using collect and then using json.dump:

import json

collected_df = df_final.collect()

with open(data_output_file + 'createjson.json', 'w') as outfile:

    json.dump(data, outfile)

answered Nov 22 '18 at 9:40

OmG

7,97852743

A solution can be using collect and then using json.dump:

import json

collected_df = df_final.collect()

with open(data_output_file + 'createjson.json', 'w') as outfile:

    json.dump(data, outfile)

answered Nov 22 '18 at 9:40

OmG

7,97852743

answered Nov 22 '18 at 9:40

OmG

7,97852743

answered Nov 22 '18 at 9:40

OmG

7,97852743

answered Nov 22 '18 at 9:40

OmG

7,97852743

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

add a comment |

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

Actually this correct but it is not creating the file directly in hdfs. It creates on the container where the script runs

– Shankar Panda
Nov 22 '18 at 9:41

add a comment |

If you want to use spark to process result as json files, I think that your output schema is right in hdfs.

And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :

with open('data.json') as f:

  data = json.load(f)

You should try to read data line by line:

data = 

with open("data.json",'r') as datafile:

  for line in datafile:

    data.append(json.loads(line))

and you can use pandas to create dataframe :

df = pd.DataFrame(data)

answered Nov 23 '18 at 2:49

chilun

948

add a comment |

If you want to use spark to process result as json files, I think that your output schema is right in hdfs.

And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :

with open('data.json') as f:

  data = json.load(f)

You should try to read data line by line:

data = 

with open("data.json",'r') as datafile:

  for line in datafile:

    data.append(json.loads(line))

and you can use pandas to create dataframe :

df = pd.DataFrame(data)

answered Nov 23 '18 at 2:49

chilun

948

add a comment |

If you want to use spark to process result as json files, I think that your output schema is right in hdfs.

And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :

with open('data.json') as f:

  data = json.load(f)

You should try to read data line by line:

data = 

with open("data.json",'r') as datafile:

  for line in datafile:

    data.append(json.loads(line))

and you can use pandas to create dataframe :

df = pd.DataFrame(data)

answered Nov 23 '18 at 2:49

chilun

948

If you want to use spark to process result as json files, I think that your output schema is right in hdfs.

And I assumed you encountered the issue that you can not smoothly read data from normal python script by using :

with open('data.json') as f:

  data = json.load(f)

You should try to read data line by line:

data = 

with open("data.json",'r') as datafile:

  for line in datafile:

    data.append(json.loads(line))

and you can use pandas to create dataframe :

df = pd.DataFrame(data)

answered Nov 23 '18 at 2:49

chilun

948

answered Nov 23 '18 at 2:49

chilun

948

answered Nov 23 '18 at 2:49

chilun

948

answered Nov 23 '18 at 2:49

chilun

948

add a comment |

For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.

df_final.coalesce(1).write.format('json').save('/path/file_name.json')

and still you want to convert your datafram into json then you can used
df_final.toJSON().

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

add a comment |

For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.

df_final.coalesce(1).write.format('json').save('/path/file_name.json')

and still you want to convert your datafram into json then you can used
df_final.toJSON().

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

add a comment |

For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.

df_final.coalesce(1).write.format('json').save('/path/file_name.json')

and still you want to convert your datafram into json then you can used
df_final.toJSON().

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

For pyspark you can directly store your dataframe into json file, there is no need to convert the datafram into json.

df_final.coalesce(1).write.format('json').save('/path/file_name.json')

and still you want to convert your datafram into json then you can used
df_final.toJSON().

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

answered Nov 23 '18 at 7:40

Sahil Desai

1,3893822

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

add a comment |

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

Yeah, but it stores data line by line {"Variable":"Col1","Min":"20","Max":"30"} {"Variable":"Col2","Min":"25,"Max":"40"} instead it should be separated by , and enclosed with square braces

– Shankar Panda
Nov 23 '18 at 7:48

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Tukukkk