Create single row dataframe from list of list pyspark












3















I have a data like this data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]] I want to create a pyspark dataframe



I already use dataframe = SQLContext.createDataFrame(data, ['features']), but I always get 



+--------+---+

|features| _2|

+--------+---+

|     1.1|1.2|

|     1.3|1.4|

|     1.5|1.6|

+--------+---+


how can I get result like below?



+----------+

|features  |

+----------+

|[1.1, 1.2]|

|[1.3, 1.4]|

|[1.5, 1.6]|

+----------+





python apache-spark pyspark spark-dataframe







share|improve this question























  • You can create a schema and provide while creating a dataframe

    – Shankar Koirala
    Feb 12 '18 at 11:11
















3















I have a data like this data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]] I want to create a pyspark dataframe



I already use dataframe = SQLContext.createDataFrame(data, ['features']), but I always get 



+--------+---+

|features| _2|

+--------+---+

|     1.1|1.2|

|     1.3|1.4|

|     1.5|1.6|

+--------+---+


how can I get result like below?



+----------+

|features  |

+----------+

|[1.1, 1.2]|

|[1.3, 1.4]|

|[1.5, 1.6]|

+----------+





python apache-spark pyspark spark-dataframe







share|improve this question























  • You can create a schema and provide while creating a dataframe

    – Shankar Koirala
    Feb 12 '18 at 11:11














3












3








3


2






I have a data like this data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]] I want to create a pyspark dataframe



I already use dataframe = SQLContext.createDataFrame(data, ['features']), but I always get 



+--------+---+

|features| _2|

+--------+---+

|     1.1|1.2|

|     1.3|1.4|

|     1.5|1.6|

+--------+---+


how can I get result like below?



+----------+

|features  |

+----------+

|[1.1, 1.2]|

|[1.3, 1.4]|

|[1.5, 1.6]|

+----------+





python apache-spark pyspark spark-dataframe







share|improve this question














I have a data like this data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]] I want to create a pyspark dataframe



I already use dataframe = SQLContext.createDataFrame(data, ['features']), but I always get 



+--------+---+

|features| _2|

+--------+---+

|     1.1|1.2|

|     1.3|1.4|

|     1.5|1.6|

+--------+---+


how can I get result like below?



+----------+

|features  |

+----------+

|[1.1, 1.2]|

|[1.3, 1.4]|

|[1.5, 1.6]|

+----------+





python apache-spark pyspark spark-dataframe




python apache-spark pyspark spark-dataframe






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Feb 12 '18 at 11:08









Yanfa Adi PutraYanfa Adi Putra

629




629













  • You can create a schema and provide while creating a dataframe

    – Shankar Koirala
    Feb 12 '18 at 11:11



















  • You can create a schema and provide while creating a dataframe

    – Shankar Koirala
    Feb 12 '18 at 11:11

















You can create a schema and provide while creating a dataframe

– Shankar Koirala
Feb 12 '18 at 11:11





You can create a schema and provide while creating a dataframe

– Shankar Koirala
Feb 12 '18 at 11:11












3 Answers
3






active

oldest

votes


















1














I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.



You can get your desired output by making each element in the list a tuple:



data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]

dataframe = sqlCtx.createDataFrame(data, ['features'])

dataframe.show()

#+----------+

#|  features|

#+----------+

#|[1.1, 1.2]|

#|[1.3, 1.4]|

#|[1.5, 1.6]|

#+----------+


Or if changing the source is cumbersome, you can equivalently do:



data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]

dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])

dataframe.show()

#+----------+

#|  features|

#+----------+

#|[1.1, 1.2]|

#|[1.3, 1.4]|

#|[1.5, 1.6]|

#+----------+





share|improve this answer

































    0














    You need a map function to convert the tuples to array and use it in createDataFrame 



    dataframe = sqlContext.createDataFrame(sc.parallelize(data).map(lambda x: [x]), ['features'])


    You should get as you desire 



    +----------+
    
|  features|
    
+----------+
    
|[1.1, 1.2]|
    
|[1.3, 1.4]|
    
|[1.5, 1.6]|
    
+----------+





    share|improve this answer































      0














      You should use the Vector Assembler function, from your code I guess you are doing this to train a machine learning model, and vector assembler works the best for that case. You can also add the assembler in the pipeline.



      assemble_feature=VectorAssembler(inputCol=data.columns,outputCol='features')
      
pipeline=Pipeline(stages=[assemble_feature])
      
pipeline.fit(data).transform(data)





      share|improve this answer























        Your Answer






        StackExchange.ifUsing("editor", function () {
        StackExchange.using("externalEditor", function () {
        StackExchange.using("snippets", function () {
        StackExchange.snippets.init();
        });
        });
        }, "code-snippets");

        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "1"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        autoActivateHeartbeat: false,
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














        draft saved

        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48745029%2fcreate-single-row-dataframe-from-list-of-list-pyspark%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        3 Answers
        3






        active

        oldest

        votes








        3 Answers
        3






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes









        1














        I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.



        You can get your desired output by making each element in the list a tuple:



        data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]
        
dataframe = sqlCtx.createDataFrame(data, ['features'])
        
dataframe.show()
        
#+----------+
        
#|  features|
        
#+----------+
        
#|[1.1, 1.2]|
        
#|[1.3, 1.4]|
        
#|[1.5, 1.6]|
        
#+----------+


        Or if changing the source is cumbersome, you can equivalently do:



        data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]
        
dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])
        
dataframe.show()
        
#+----------+
        
#|  features|
        
#+----------+
        
#|[1.1, 1.2]|
        
#|[1.3, 1.4]|
        
#|[1.5, 1.6]|
        
#+----------+





        share|improve this answer






























          1














          I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.



          You can get your desired output by making each element in the list a tuple:



          data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]
          
dataframe = sqlCtx.createDataFrame(data, ['features'])
          
dataframe.show()
          
#+----------+
          
#|  features|
          
#+----------+
          
#|[1.1, 1.2]|
          
#|[1.3, 1.4]|
          
#|[1.5, 1.6]|
          
#+----------+


          Or if changing the source is cumbersome, you can equivalently do:



          data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]
          
dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])
          
dataframe.show()
          
#+----------+
          
#|  features|
          
#+----------+
          
#|[1.1, 1.2]|
          
#|[1.3, 1.4]|
          
#|[1.5, 1.6]|
          
#+----------+





          share|improve this answer




























            1












            1








            1







            I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.



            You can get your desired output by making each element in the list a tuple:



            data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]
            
dataframe = sqlCtx.createDataFrame(data, ['features'])
            
dataframe.show()
            
#+----------+
            
#|  features|
            
#+----------+
            
#|[1.1, 1.2]|
            
#|[1.3, 1.4]|
            
#|[1.5, 1.6]|
            
#+----------+


            Or if changing the source is cumbersome, you can equivalently do:



            data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]
            
dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])
            
dataframe.show()
            
#+----------+
            
#|  features|
            
#+----------+
            
#|[1.1, 1.2]|
            
#|[1.3, 1.4]|
            
#|[1.5, 1.6]|
            
#+----------+





            share|improve this answer















            I find it's useful to think of the argument to createDataFrame() as a list of tuples where each entry in the list corresponds to a row in the DataFrame and each element of the tuple corresponds to a column.



            You can get your desired output by making each element in the list a tuple:



            data = [([1.1, 1.2],), ([1.3, 1.4],), ([1.5, 1.6],)]
            
dataframe = sqlCtx.createDataFrame(data, ['features'])
            
dataframe.show()
            
#+----------+
            
#|  features|
            
#+----------+
            
#|[1.1, 1.2]|
            
#|[1.3, 1.4]|
            
#|[1.5, 1.6]|
            
#+----------+


            Or if changing the source is cumbersome, you can equivalently do:



            data = [[1.1, 1.2], [1.3, 1.4], [1.5, 1.6]]
            
dataframe = sqlCtx.createDataFrame(map(lambda x: (x, ), data), ['features'])
            
dataframe.show()
            
#+----------+
            
#|  features|
            
#+----------+
            
#|[1.1, 1.2]|
            
#|[1.3, 1.4]|
            
#|[1.5, 1.6]|
            
#+----------+






            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Feb 12 '18 at 17:04

























            answered Feb 12 '18 at 16:19









            paultpault

            14.4k31947




            14.4k31947

























                0














                You need a map function to convert the tuples to array and use it in createDataFrame 



                dataframe = sqlContext.createDataFrame(sc.parallelize(data).map(lambda x: [x]), ['features'])


                You should get as you desire 



                +----------+
                
|  features|
                
+----------+
                
|[1.1, 1.2]|
                
|[1.3, 1.4]|
                
|[1.5, 1.6]|
                
+----------+





                share|improve this answer




























                  0














                  You need a map function to convert the tuples to array and use it in createDataFrame 



                  dataframe = sqlContext.createDataFrame(sc.parallelize(data).map(lambda x: [x]), ['features'])


                  You should get as you desire 



                  +----------+
                  
|  features|
                  
+----------+
                  
|[1.1, 1.2]|
                  
|[1.3, 1.4]|
                  
|[1.5, 1.6]|
                  
+----------+





                  share|improve this answer


























                    0












                    0








                    0







                    You need a map function to convert the tuples to array and use it in createDataFrame 



                    dataframe = sqlContext.createDataFrame(sc.parallelize(data).map(lambda x: [x]), ['features'])


                    You should get as you desire 



                    +----------+
                    
|  features|
                    
+----------+
                    
|[1.1, 1.2]|
                    
|[1.3, 1.4]|
                    
|[1.5, 1.6]|
                    
+----------+





                    share|improve this answer













                    You need a map function to convert the tuples to array and use it in createDataFrame 



                    dataframe = sqlContext.createDataFrame(sc.parallelize(data).map(lambda x: [x]), ['features'])


                    You should get as you desire 



                    +----------+
                    
|  features|
                    
+----------+
                    
|[1.1, 1.2]|
                    
|[1.3, 1.4]|
                    
|[1.5, 1.6]|
                    
+----------+






                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Feb 12 '18 at 11:23









                    Ramesh MaharjanRamesh Maharjan

                    27k52047




                    27k52047























                        0














                        You should use the Vector Assembler function, from your code I guess you are doing this to train a machine learning model, and vector assembler works the best for that case. You can also add the assembler in the pipeline.



                        assemble_feature=VectorAssembler(inputCol=data.columns,outputCol='features')
                        
pipeline=Pipeline(stages=[assemble_feature])
                        
pipeline.fit(data).transform(data)





                        share|improve this answer




























                          0














                          You should use the Vector Assembler function, from your code I guess you are doing this to train a machine learning model, and vector assembler works the best for that case. You can also add the assembler in the pipeline.



                          assemble_feature=VectorAssembler(inputCol=data.columns,outputCol='features')
                          
pipeline=Pipeline(stages=[assemble_feature])
                          
pipeline.fit(data).transform(data)





                          share|improve this answer


























                            0












                            0








                            0







                            You should use the Vector Assembler function, from your code I guess you are doing this to train a machine learning model, and vector assembler works the best for that case. You can also add the assembler in the pipeline.



                            assemble_feature=VectorAssembler(inputCol=data.columns,outputCol='features')
                            
pipeline=Pipeline(stages=[assemble_feature])
                            
pipeline.fit(data).transform(data)





                            share|improve this answer













                            You should use the Vector Assembler function, from your code I guess you are doing this to train a machine learning model, and vector assembler works the best for that case. You can also add the assembler in the pipeline.



                            assemble_feature=VectorAssembler(inputCol=data.columns,outputCol='features')
                            
pipeline=Pipeline(stages=[assemble_feature])
                            
pipeline.fit(data).transform(data)






                            share|improve this answer












                            share|improve this answer



                            share|improve this answer










                            answered Feb 12 '18 at 12:04









                            pratiklodhapratiklodha

                            686718




                            686718






























                                draft saved

                                draft discarded




















































                                Thanks for contributing an answer to Stack Overflow!


                                • Please be sure to answer the question. Provide details and share your research!

                                But avoid …



                                • Asking for help, clarification, or responding to other answers.

                                • Making statements based on opinion; back them up with references or personal experience.


                                To learn more, see our tips on writing great answers.




                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f48745029%2fcreate-single-row-dataframe-from-list-of-list-pyspark%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                -

                                Popular posts from this blog

                                404 Error Contact Form 7 ajax form submitting

                                Image
                                0 The contact form does not work, and affects loading after one try. The chrome console shows the error message: POST http://example.com/wp-json/contact-form-7/v1/contact-forms/50/feedback 404 (Not Found) send @ jquery.js?ver=1.12.4:4 ajax @ jquery.js?ver=1.12.4:4 wpcf7.submit @ scripts.js?ver=5.0.3:346 (anonymous) @ scripts.js?ver=5.0.3:53 dispatch @ jquery.js?ver=1.12.4:3 r.handle @ jquery.js?ver=1.12.4:3 How to solve this issue? thanks in advance javascript wordpress contact-form-7 share | improve this question edited Jul 17 '18 at 5:39 Kiran Shahi 4,745 3 16 43 ...
                                Read more

                                How to know if a Active Directory user can login interactively

                                Image
                                1 I would like to know if and how is it possible to know if a AD user can login interactively (on a server) in a domain. I need to know if I can find it out using a LDAP search. Thanks. windows active-directory login user-permissions user-profile share | improve this question edited 1 hour ago yagmoth555 ♦ 11.6k 3 17 42 asked 1 hour ago Luigi Luigi 6 1 New contributor ...
                                Read more

                                Refactoring coordinates for Minecraft Pi buildings written in Python

                                Image
                                0 $begingroup$ I'm fairly new to coding and I wrote this script in Python for a house with a roof in Minecraft Pi. I've found different ways to write the coordinates to make the code base shorter, but I'm wondering if it is possible for me to refactor it any further so I don't have to list each coordinate line by line. import mcpi.minecraft as minecraft import mcpi.block as block mc = minecraft.Minecraft.create() p = mc.player.getTilePos() mc.setBlocks(p.x+1, p.y, p.z+1, #Main structure starting coordinates (DO NOT ALTER) p.x+19, p.y+4, p.z+6, #Structure size/coordinates (Alters cube size) block.STONE_BRICK) mc.setBlocks(p.x+2, p.y+1, p.z+2, #Hollowing out main structure interior p.x+18, p.y+3, p.z+5, block.AIR) mc.setBlocks(p.x, p.y...
                                Read more