How can I use a variable from web scraping?

I scrape a jobs portal using this code:

const puppeteer = require('puppeteer')



export default function scrape() {

  ;(async () => {

    const browser = await puppeteer.launch()

    const page = await browser.newPage()

    await page.goto('https://www.example.de/jobs/javascript')



    const position = await page.evaluate(() =>

      Array.from(document.querySelectorAll('h2')).map(

        position => position.innerText

      )

    )



    // const logo = await page.evaluate(() =>

    //   Array.from(document.querySelectorAll('div.job-element__logo img')).map(

    //     logo => logo.src

    //   )

    // )



    console.log(JSON.stringify(position))



    await browser.close()

  })()

}

I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.

My question:

I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...

I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:

Can't resolve './scraper' in .../components'.

Here is a screenshot of my data tree: My data structure

If I put the scraper.js into components it says:

./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'

I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?

asked Nov 21 '18 at 18:33

user9845001

1

puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45

We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09

Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02

add a comment |

I scrape a jobs portal using this code:

const puppeteer = require('puppeteer')



export default function scrape() {

  ;(async () => {

    const browser = await puppeteer.launch()

    const page = await browser.newPage()

    await page.goto('https://www.example.de/jobs/javascript')



    const position = await page.evaluate(() =>

      Array.from(document.querySelectorAll('h2')).map(

        position => position.innerText

      )

    )



    // const logo = await page.evaluate(() =>

    //   Array.from(document.querySelectorAll('div.job-element__logo img')).map(

    //     logo => logo.src

    //   )

    // )



    console.log(JSON.stringify(position))



    await browser.close()

  })()

}

I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.

My question:

I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...

I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:

Can't resolve './scraper' in .../components'.

Here is a screenshot of my data tree: My data structure

If I put the scraper.js into components it says:

./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'

I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?

asked Nov 21 '18 at 18:33

user9845001

1

puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45

We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09

Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02

add a comment |

I scrape a jobs portal using this code:

const puppeteer = require('puppeteer')



export default function scrape() {

  ;(async () => {

    const browser = await puppeteer.launch()

    const page = await browser.newPage()

    await page.goto('https://www.example.de/jobs/javascript')



    const position = await page.evaluate(() =>

      Array.from(document.querySelectorAll('h2')).map(

        position => position.innerText

      )

    )



    // const logo = await page.evaluate(() =>

    //   Array.from(document.querySelectorAll('div.job-element__logo img')).map(

    //     logo => logo.src

    //   )

    // )



    console.log(JSON.stringify(position))



    await browser.close()

  })()

}

I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.

My question:

I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...

I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:

Can't resolve './scraper' in .../components'.

Here is a screenshot of my data tree: My data structure

If I put the scraper.js into components it says:

./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'

I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?

asked Nov 21 '18 at 18:33

user9845001

I scrape a jobs portal using this code:

const puppeteer = require('puppeteer')



export default function scrape() {

  ;(async () => {

    const browser = await puppeteer.launch()

    const page = await browser.newPage()

    await page.goto('https://www.example.de/jobs/javascript')



    const position = await page.evaluate(() =>

      Array.from(document.querySelectorAll('h2')).map(

        position => position.innerText

      )

    )



    // const logo = await page.evaluate(() =>

    //   Array.from(document.querySelectorAll('div.job-element__logo img')).map(

    //     logo => logo.src

    //   )

    // )



    console.log(JSON.stringify(position))



    await browser.close()

  })()

}

I pasted a example URL here, of course I use a real one in my example. Otherwise the code should work I thought.

My question:

I can console log my extracted data under the const position. That works! But I would like to reuse it now in another react component. Which does not work...

I tried to export it as a function and import it in my component called JobCard.js. But it prints a warning saying:

Can't resolve './scraper' in .../components'.

Here is a screenshot of my data tree: My data structure

If I put the scraper.js into components it says:

./node_modules/puppeteer/lib/WebSocketTransport.js
Module not found: Can't resolve 'ws' in '/remote-jobs-app/node_modules/puppeteer/lib'

I just want to reuse the const position in another component. But I cannot get it done. Any ideas maybe?

javascript node.js reactjs web-scraping

asked Nov 21 '18 at 18:33

user9845001

asked Nov 21 '18 at 18:33

user9845001

asked Nov 21 '18 at 18:33

user9845001

asked Nov 21 '18 at 18:33

user9845001

asked Nov 21 '18 at 18:33

user9845001

1

puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45

We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09

Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02

add a comment |

1

puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45

We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09

Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02

puppeteer is a server-side module, but a React app runs on the client (browser). Two very different things. You need to set up a separate node server, then use fetch() on the client to request data from it. You can use the http module to set up a minimal server.
– Chris G
Nov 21 '18 at 18:45

We're not talking about an electron app by any chance? That could work but @Chris is right, not on a web page. Not even a chrome extension.
– pguardiario
Nov 22 '18 at 0:09

Thanks for your replies. No its not an electron app but a web app I am building with React.js. Ok I see, I will then setup a server and do it like you guys suggested. Thx for your help.
– user9845001
Nov 22 '18 at 9:02

add a comment |

1 Answer
1

active

oldest

votes

you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .

When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data

If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53418490%2fhow-can-i-use-a-variable-from-web-scraping%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .

When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data

If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

add a comment |

you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .

When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data

If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

add a comment |

you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .

When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data

If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

you can't import your scrapper component in react.js component
when you scrapping you are running node.js. that is not reactjs
it's a server side thing .

When you running your react.js it's a browser based thing.
i suggest you that, you scrape data and save it a file like jobdata.json and then import this in your component
then used that data

If you want dynamic result, you need a server side code. So overall design is like
you scrape the data, save it in database. make a api that and use react to show that

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

answered Nov 21 '18 at 18:38

Tanvir Raj

3113

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Tukukkk