How to allow a puppeteer bot to log in through google to an online account? Preferably without scraping the...
So I have an account through which I sign in with google. This is a third party account that uses Google's OAuth services. I wanted to create a bot using Puppeteer that would login through google to that site every ten minutes or so. Then it would scrape my pages for data that I would use for automation.
However, I realized that Google probably doesn't like bots accessing their login pages.
In fact, that specific page is disallowed in their robots.txt file. Is their a way that I can give my bot credentials to login to my account that Google would approve of? Maybe an OAuth token of some sort?
If that's not possible what are the risks of me continuing to directly log in through their portal with my bot? Could Google ban my IP address? Would it even be able to detect that I'm using a bot? Is their any shady behavior that I should avoid so as not to alert Google?
Edit: Actually after looking closer I realized that https://www.google.com/accounts/signin is not disallowed (it doesn't specifically say that it is allowed though). However, https://www.google.com/accounts/ClientLogin and https://www.google.com/accounts/ClientAuth are disallowed. If possible could someone also explain what these last two links are?
node.js bots puppeteer
asked Nov 20 at 21:00
Joshua Avery
1069
add a comment |
So I have an account through which I sign in with google. This is a third party account that uses Google's OAuth services. I wanted to create a bot using Puppeteer that would login through google to that site every ten minutes or so. Then it would scrape my pages for data that I would use for automation.
However, I realized that Google probably doesn't like bots accessing their login pages.
In fact, that specific page is disallowed in their robots.txt file. Is their a way that I can give my bot credentials to login to my account that Google would approve of? Maybe an OAuth token of some sort?
If that's not possible what are the risks of me continuing to directly log in through their portal with my bot? Could Google ban my IP address? Would it even be able to detect that I'm using a bot? Is their any shady behavior that I should avoid so as not to alert Google?
Edit: Actually after looking closer I realized that https://www.google.com/accounts/signin is not disallowed (it doesn't specifically say that it is allowed though). However, https://www.google.com/accounts/ClientLogin and https://www.google.com/accounts/ClientAuth are disallowed. If possible could someone also explain what these last two links are?
node.js bots puppeteer
asked Nov 20 at 21:00
Joshua Avery
1069
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05
add a comment |
So I have an account through which I sign in with google. This is a third party account that uses Google's OAuth services. I wanted to create a bot using Puppeteer that would login through google to that site every ten minutes or so. Then it would scrape my pages for data that I would use for automation.
However, I realized that Google probably doesn't like bots accessing their login pages.
In fact, that specific page is disallowed in their robots.txt file. Is their a way that I can give my bot credentials to login to my account that Google would approve of? Maybe an OAuth token of some sort?
If that's not possible what are the risks of me continuing to directly log in through their portal with my bot? Could Google ban my IP address? Would it even be able to detect that I'm using a bot? Is their any shady behavior that I should avoid so as not to alert Google?
Edit: Actually after looking closer I realized that https://www.google.com/accounts/signin is not disallowed (it doesn't specifically say that it is allowed though). However, https://www.google.com/accounts/ClientLogin and https://www.google.com/accounts/ClientAuth are disallowed. If possible could someone also explain what these last two links are?
node.js bots puppeteer
asked Nov 20 at 21:00
Joshua Avery
1069
So I have an account through which I sign in with google. This is a third party account that uses Google's OAuth services. I wanted to create a bot using Puppeteer that would login through google to that site every ten minutes or so. Then it would scrape my pages for data that I would use for automation.
However, I realized that Google probably doesn't like bots accessing their login pages.
In fact, that specific page is disallowed in their robots.txt file. Is their a way that I can give my bot credentials to login to my account that Google would approve of? Maybe an OAuth token of some sort?
If that's not possible what are the risks of me continuing to directly log in through their portal with my bot? Could Google ban my IP address? Would it even be able to detect that I'm using a bot? Is their any shady behavior that I should avoid so as not to alert Google?
Edit: Actually after looking closer I realized that https://www.google.com/accounts/signin is not disallowed (it doesn't specifically say that it is allowed though). However, https://www.google.com/accounts/ClientLogin and https://www.google.com/accounts/ClientAuth are disallowed. If possible could someone also explain what these last two links are?
node.js bots puppeteer
node.js bots puppeteer
asked Nov 20 at 21:00
Joshua Avery
1069
asked Nov 20 at 21:00
Joshua Avery
1069
edited Nov 21 at 5:30
asked Nov 20 at 21:00
Joshua Avery
1069
asked Nov 20 at 21:00
Joshua Avery
1069
asked Nov 20 at 21:00
Joshua Avery
1069
1069
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05
add a comment |
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401443%2fhow-to-allow-a-puppeteer-bot-to-log-in-through-google-to-an-online-account-pref%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401443%2fhow-to-allow-a-puppeteer-bot-to-log-in-through-google-to-an-online-account-pref%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Google would definitely shut this down (and force recaptcha or some other measure to reduce automated usage). They do however have a lot of API's available; perhaps your use-case falls under one of these? developers.google.com/apis-explorer/#p
– Art
Nov 21 at 3:05