Thread waiting for sun.misc.Unsafe.park in spark createDataFrame











up vote
0
down vote

favorite












I am using spark to filter out data from a JavaRDD dataframe.
It has around 2900000 record of which I need to filter 29000.
There is no memory issue as I am using a super server with enought memory.



Below is the code



    JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};

    Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());


It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.



I checked similar answers none helped.



There is no CPU utilization as well. And there can't be any memory issue as I am using super server.






apache-spark threadpool deadlock







share|improve this question


























    up vote
    0
    down vote

    favorite












    I am using spark to filter out data from a JavaRDD dataframe.
    It has around 2900000 record of which I need to filter 29000.
    There is no memory issue as I am using a super server with enought memory.



    Below is the code



        JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
    
    Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());


    It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.



    I checked similar answers none helped.



    There is no CPU utilization as well. And there can't be any memory issue as I am using super server.






    apache-spark threadpool deadlock







    share|improve this question
























      up vote
      0
      down vote

      favorite









      up vote
      0
      down vote

      favorite











      I am using spark to filter out data from a JavaRDD dataframe.
      It has around 2900000 record of which I need to filter 29000.
      There is no memory issue as I am using a super server with enought memory.



      Below is the code



          JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
      
    Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());


      It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.



      I checked similar answers none helped.



      There is no CPU utilization as well. And there can't be any memory issue as I am using super server.






      apache-spark threadpool deadlock







      share|improve this question













      I am using spark to filter out data from a JavaRDD dataframe.
      It has around 2900000 record of which I need to filter 29000.
      There is no memory issue as I am using a super server with enought memory.



      Below is the code



          JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
      
    Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());


      It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.



      I checked similar answers none helped.



      There is no CPU utilization as well. And there can't be any memory issue as I am using super server.






      apache-spark threadpool deadlock




      apache-spark threadpool deadlock






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 19 at 14:57









      Mehaboob Khan

      102113




      102113





























          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377284%2fthread-waiting-for-sun-misc-unsafe-park-in-spark-createdataframe%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown






























          active

          oldest

          votes













          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377284%2fthread-waiting-for-sun-misc-unsafe-park-in-spark-createdataframe%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          404 Error Contact Form 7 ajax form submitting

          Image
          0 The contact form does not work, and affects loading after one try. The chrome console shows the error message: POST http://example.com/wp-json/contact-form-7/v1/contact-forms/50/feedback 404 (Not Found) send @ jquery.js?ver=1.12.4:4 ajax @ jquery.js?ver=1.12.4:4 wpcf7.submit @ scripts.js?ver=5.0.3:346 (anonymous) @ scripts.js?ver=5.0.3:53 dispatch @ jquery.js?ver=1.12.4:3 r.handle @ jquery.js?ver=1.12.4:3 How to solve this issue? thanks in advance javascript wordpress contact-form-7 share | improve this question edited Jul 17 '18 at 5:39 Kiran Shahi 4,745 3 16 43
          Read more

          How to know if a Active Directory user can login interactively

          Image
          1 I would like to know if and how is it possible to know if a AD user can login interactively (on a server) in a domain. I need to know if I can find it out using a LDAP search. Thanks. windows active-directory login user-permissions user-profile share | improve this question edited 1 hour ago yagmoth555 ♦ 11.6k 3 17 42 asked 1 hour ago Luigi Luigi 6 1 New contributor
          Read more

          TypeError: fit_transform() missing 1 required positional argument: 'X'

          Image
          0 I am trying to do Feature Scaling in a dataset, but I get an error and have no idea how to proceed: > Traceback (most recent call last): > > File "<ipython-input-10-71bea414b4d0>", line 22, in <module> > x_train = sc_X.fit_transform(x_train) > > TypeError: fit_transform() missing 1 required positional argument: 'X' and here is my code: import pandas as pd # Importing the dataset dataset = pd.read_csv('Data.csv') X = dataset.iloc[:, :-1].values y = dataset.iloc[:, 3].values # Taking care of missing data from sklearn.preprocessing import Imputer imputer = Imputer(missing_values="NaN", strategy="mean", axis=0) imputer = Imputer.fit(imputer,X[:,1:3]) X[:, 1:3] = Imputer.transform(imputer,X[
          Read more