How can I use a variable from web scraping?












0














I scrape a jobs portal using this code:



const puppeteer = require('puppeteer')

export default function scrape() {
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://www.example.de/jobs/javascript')

const position = await page.evaluate(() =>
Array.from(document.querySelectorAll('h2')).map(
position => position.innerText
)
)

// const logo = await page.evaluate(() =>
// Array.from(document.querySelectorAll('div.job-element__logo img')).map(
// logo => logo.src
// )
// )

console.log(JSON.stringify(position))

await browser.close()
})()
}


I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.



My question:



I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...



I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:



Can't resolve './scraper' in .../components'.



Here is a screenshot of my data tree: My data structure



If I put the scraper.js into components it says:



./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'



I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?










share|improve this question


















  • 1




    puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
    – Chris G
    Nov 21 '18 at 18:45










  • We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
    – pguardiario
    Nov 22 '18 at 0:09










  • Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
    – user9845001
    Nov 22 '18 at 9:02
















0














I scrape a jobs portal using this code:



const puppeteer = require('puppeteer')

export default function scrape() {
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://www.example.de/jobs/javascript')

const position = await page.evaluate(() =>
Array.from(document.querySelectorAll('h2')).map(
position => position.innerText
)
)

// const logo = await page.evaluate(() =>
// Array.from(document.querySelectorAll('div.job-element__logo img')).map(
// logo => logo.src
// )
// )

console.log(JSON.stringify(position))

await browser.close()
})()
}


I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.



My question:



I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...



I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:



Can't resolve './scraper' in .../components'.



Here is a screenshot of my data tree: My data structure



If I put the scraper.js into components it says:



./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'



I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?










share|improve this question


















  • 1




    puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
    – Chris G
    Nov 21 '18 at 18:45










  • We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
    – pguardiario
    Nov 22 '18 at 0:09










  • Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
    – user9845001
    Nov 22 '18 at 9:02














0












0








0







I scrape a jobs portal using this code:



const puppeteer = require('puppeteer')

export default function scrape() {
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://www.example.de/jobs/javascript')

const position = await page.evaluate(() =>
Array.from(document.querySelectorAll('h2')).map(
position => position.innerText
)
)

// const logo = await page.evaluate(() =>
// Array.from(document.querySelectorAll('div.job-element__logo img')).map(
// logo => logo.src
// )
// )

console.log(JSON.stringify(position))

await browser.close()
})()
}


I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.



My question:



I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...



I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:



Can't resolve './scraper' in .../components'.



Here is a screenshot of my data tree: My data structure



If I put the scraper.js into components it says:



./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'



I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?










share|improve this question













I scrape a jobs portal using this code:



const puppeteer = require('puppeteer')

export default function scrape() {
;(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto('https://www.example.de/jobs/javascript')

const position = await page.evaluate(() =>
Array.from(document.querySelectorAll('h2')).map(
position => position.innerText
)
)

// const logo = await page.evaluate(() =>
// Array.from(document.querySelectorAll('div.job-element__logo img')).map(
// logo => logo.src
// )
// )

console.log(JSON.stringify(position))

await browser.close()
})()
}


I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.



My question:



I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...



I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:



Can't resolve './scraper' in .../components'.



Here is a screenshot of my data tree: My data structure



If I put the scraper.js into components it says:



./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'



I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?







javascript node.js reactjs web-scraping






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 21 '18 at 18:33







user9845001















  • 1




    puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
    – Chris G
    Nov 21 '18 at 18:45










  • We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
    – pguardiario
    Nov 22 '18 at 0:09










  • Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
    – user9845001
    Nov 22 '18 at 9:02














  • 1




    puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
    – Chris G
    Nov 21 '18 at 18:45










  • We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
    – pguardiario
    Nov 22 '18 at 0:09










  • Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
    – user9845001
    Nov 22 '18 at 9:02








1




1




puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45




puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45












We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09




We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09












Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02




Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02












1 Answer
1






active

oldest

votes


















0














you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .



When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data



If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that






share|improve this answer





















    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418490%2fhow-can-i-use-a-variable-from-web-scraping%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown
























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    you can't import your scrapper component in react.js component
    when you scrapping you are running node.js. that is not reactjs
    it's a server side thing .



    When you running your react.js it's a browser based thing.
    i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
    then used that data



    If you want dynamic result, you need a server side code. So overall design is like
    you scrape the data, save it in database. make a api that and use react to show that






    share|improve this answer


























      0














      you can't import your scrapper component in react.js component
      when you scrapping you are running node.js. that is not reactjs
      it's a server side thing .



      When you running your react.js it's a browser based thing.
      i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
      then used that data



      If you want dynamic result, you need a server side code. So overall design is like
      you scrape the data, save it in database. make a api that and use react to show that






      share|improve this answer
























        0












        0








        0






        you can't import your scrapper component in react.js component
        when you scrapping you are running node.js. that is not reactjs
        it's a server side thing .



        When you running your react.js it's a browser based thing.
        i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
        then used that data



        If you want dynamic result, you need a server side code. So overall design is like
        you scrape the data, save it in database. make a api that and use react to show that






        share|improve this answer












        you can't import your scrapper component in react.js component
        when you scrapping you are running node.js. that is not reactjs
        it's a server side thing .



        When you running your react.js it's a browser based thing.
        i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
        then used that data



        If you want dynamic result, you need a server side code. So overall design is like
        you scrape the data, save it in database. make a api that and use react to show that







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Nov 21 '18 at 18:38









        Tanvir RajTanvir Raj

        3113




        3113






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418490%2fhow-can-i-use-a-variable-from-web-scraping%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            404 Error Contact Form 7 ajax form submitting

            How to know if a Active Directory user can login interactively

            TypeError: fit_transform() missing 1 required positional argument: 'X'