Getting error trying to scrape data from FiveThirtyEight
1
I am trying to scrape data from FiveThirtyEight's presidential approval rating to get the dates, pollsters, sample sizes, and percentages into a data frame in R. My first attempt was an approach using html_nodes:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")
pres_approval <- pres_approval %>%
html_nodes(css = "table") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)
Which returned
Error in nodes_duplicated(nodes) : Expecting an external pointer: [type=NULL].`
And then again using the Selector Gadget:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")`
pres_approval <- pres_approval %>%
html_nodes(css = "td , .heat-map , .pollster a") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)`
Which returned
Error in html_table.xml_node(., header = TRUE, fill = TRUE) : html_name(x) == "table" is not TRUE`
What can I do from here?
r web-scraping rvest
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
add a comment |
1
I am trying to scrape data from FiveThirtyEight's presidential approval rating to get the dates, pollsters, sample sizes, and percentages into a data frame in R. My first attempt was an approach using html_nodes:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")
pres_approval <- pres_approval %>%
html_nodes(css = "table") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)
Which returned
Error in nodes_duplicated(nodes) : Expecting an external pointer: [type=NULL].`
And then again using the Selector Gadget:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")`
pres_approval <- pres_approval %>%
html_nodes(css = "td , .heat-map , .pollster a") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)`
Which returned
Error in html_table.xml_node(., header = TRUE, fill = TRUE) : html_name(x) == "table" is not TRUE`
What can I do from here?
r web-scraping rvest
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
I only see one table on that page, so I thinknth(2)should benth(1).
– Marius
Nov 23 '18 at 2:40
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58
add a comment |
1
1
1
1
I am trying to scrape data from FiveThirtyEight's presidential approval rating to get the dates, pollsters, sample sizes, and percentages into a data frame in R. My first attempt was an approach using html_nodes:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")
pres_approval <- pres_approval %>%
html_nodes(css = "table") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)
Which returned
Error in nodes_duplicated(nodes) : Expecting an external pointer: [type=NULL].`
And then again using the Selector Gadget:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")`
pres_approval <- pres_approval %>%
html_nodes(css = "td , .heat-map , .pollster a") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)`
Which returned
Error in html_table.xml_node(., header = TRUE, fill = TRUE) : html_name(x) == "table" is not TRUE`
What can I do from here?
r web-scraping rvest
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
I am trying to scrape data from FiveThirtyEight's presidential approval rating to get the dates, pollsters, sample sizes, and percentages into a data frame in R. My first attempt was an approach using html_nodes:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")
pres_approval <- pres_approval %>%
html_nodes(css = "table") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)
Which returned
Error in nodes_duplicated(nodes) : Expecting an external pointer: [type=NULL].`
And then again using the Selector Gadget:
pres_approval <- read_html("https://projects.fivethirtyeight.com/trump-approval-ratings/")`
pres_approval <- pres_approval %>%
html_nodes(css = "td , .heat-map , .pollster a") %>%
nth(2) %>%
html_table(header = TRUE, fill = TRUE)`
Which returned
Error in html_table.xml_node(., header = TRUE, fill = TRUE) : html_name(x) == "table" is not TRUE`
What can I do from here?
r web-scraping rvest
r web-scraping rvest
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
edited Nov 23 '18 at 2:39
Ronak Shah
36.1k103959
36.1k103959
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
asked Nov 23 '18 at 2:31
LaurenGLaurenG
111
111
I only see one table on that page, so I thinknth(2)should benth(1).
– Marius
Nov 23 '18 at 2:40
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58
add a comment |
I only see one table on that page, so I thinknth(2)should benth(1).
– Marius
Nov 23 '18 at 2:40
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58
I only see one table on that page, so I think
nth(2) should be nth(1).– Marius
Nov 23 '18 at 2:40
I only see one table on that page, so I think
nth(2) should be nth(1).– Marius
Nov 23 '18 at 2:40
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58
add a comment |
1 Answer
1
active
oldest
votes
1
They generally load data asynchronously via XHR requests which you can see if you open Developer Tools in your browser and re-load the page. In Network -> XHR you'll see lots of lovely JSON:
I don't know which one you want (I skimmed the Q) but you can get all the main JSON files easily:
polls <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/polls.json")
str(polls, 1)
## 'data.frame': 3401 obs. of 14 variables:
## $ id : int 77261 77265 77272 77249 77257 77266 77596 77246 77263 77253 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ sampleSize : int 1992 1500 1190 1043 1500 2692 1712 1500 1500 1991 ...
## $ population : chr "rv" "a" "rv" "rv" ...
## $ weight : num 0.946 0.245 1.645 1.166 0.639 ...
## $ grade : chr "B-" "B" "A-" "B" ...
## $ multiversions: logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ url : chr "http://www.politico.com/story/2017/01/poll-voters-liked-trumps-inaugural-address-234148" "http://www.gallup.com/poll/201617/gallup-daily-trump-job-approval.aspx" "https://poll.qu.edu/national/release-detail?ReleaseID=2415" "http://www.publicpolicypolling.com/pdf/2015/PPP_Release_National_12617.pdf" ...
## $ created_at : chr "2017-01-23" "2017-01-23" "2017-01-26" "2017-01-25" ...
## $ startDate : chr "2017-01-20" "2017-01-20" "2017-01-20" "2017-01-23" ...
## $ endDate : chr "2017-01-22" "2017-01-22" "2017-01-25" "2017-01-24" ...
## $ pollster : chr "Morning Consult" "Gallup" "Quinnipiac University" "Public Policy Polling" ...
## $ tracking : chr "" "T" "" "" ...
## $ answers :List of 3401
approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/approval.json")
str(approval, 1)
## 'data.frame': 2751 obs. of 9 variables:
## $ date : chr "2017-01-23" "2017-01-23" "2017-01-23" "2017-01-24" ...
## $ future : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ subgroup : chr "Adults" "All polls" "Voters" "Adults" ...
## $ approve_estimate : chr "45" "45.46693" "46" "45" ...
## $ approve_hi : chr "51.1347" "50.88971" "52.29238" "50.98562" ...
## $ approve_lo : chr "38.8653" "40.04416" "39.70762" "39.01438" ...
## $ disapprove_estimate: chr "45" "41.26452" "37" "45.74659" ...
## $ disapprove_hi : chr "51.1347" "46.68729" "43.29238" "51.73221" ...
## $ disapprove_lo : chr "38.8653" "35.84175" "30.70762" "39.76097" ...
historic_approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/historical-approval.json")
str(historic_approval, 1)
## 'data.frame': 26001 obs. of 6 variables:
## $ president : chr "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" ...
## $ date : chr "1945-06-06" "1945-06-07" "1945-06-08" "1945-06-09" ...
## $ days : int 55 56 57 58 59 60 61 62 63 64 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ approve_estimate : chr "87" "87" "87" "87" ...
## $ disapprove_estimate: chr "3" "3" "3" "3" ...
I'd run the resultant data frame(s) through readr::type_convert() to get better types.
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439989%2fgetting-error-trying-to-scrape-data-from-fivethirtyeight%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
They generally load data asynchronously via XHR requests which you can see if you open Developer Tools in your browser and re-load the page. In Network -> XHR you'll see lots of lovely JSON:
I don't know which one you want (I skimmed the Q) but you can get all the main JSON files easily:
polls <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/polls.json")
str(polls, 1)
## 'data.frame': 3401 obs. of 14 variables:
## $ id : int 77261 77265 77272 77249 77257 77266 77596 77246 77263 77253 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ sampleSize : int 1992 1500 1190 1043 1500 2692 1712 1500 1500 1991 ...
## $ population : chr "rv" "a" "rv" "rv" ...
## $ weight : num 0.946 0.245 1.645 1.166 0.639 ...
## $ grade : chr "B-" "B" "A-" "B" ...
## $ multiversions: logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ url : chr "http://www.politico.com/story/2017/01/poll-voters-liked-trumps-inaugural-address-234148" "http://www.gallup.com/poll/201617/gallup-daily-trump-job-approval.aspx" "https://poll.qu.edu/national/release-detail?ReleaseID=2415" "http://www.publicpolicypolling.com/pdf/2015/PPP_Release_National_12617.pdf" ...
## $ created_at : chr "2017-01-23" "2017-01-23" "2017-01-26" "2017-01-25" ...
## $ startDate : chr "2017-01-20" "2017-01-20" "2017-01-20" "2017-01-23" ...
## $ endDate : chr "2017-01-22" "2017-01-22" "2017-01-25" "2017-01-24" ...
## $ pollster : chr "Morning Consult" "Gallup" "Quinnipiac University" "Public Policy Polling" ...
## $ tracking : chr "" "T" "" "" ...
## $ answers :List of 3401
approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/approval.json")
str(approval, 1)
## 'data.frame': 2751 obs. of 9 variables:
## $ date : chr "2017-01-23" "2017-01-23" "2017-01-23" "2017-01-24" ...
## $ future : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ subgroup : chr "Adults" "All polls" "Voters" "Adults" ...
## $ approve_estimate : chr "45" "45.46693" "46" "45" ...
## $ approve_hi : chr "51.1347" "50.88971" "52.29238" "50.98562" ...
## $ approve_lo : chr "38.8653" "40.04416" "39.70762" "39.01438" ...
## $ disapprove_estimate: chr "45" "41.26452" "37" "45.74659" ...
## $ disapprove_hi : chr "51.1347" "46.68729" "43.29238" "51.73221" ...
## $ disapprove_lo : chr "38.8653" "35.84175" "30.70762" "39.76097" ...
historic_approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/historical-approval.json")
str(historic_approval, 1)
## 'data.frame': 26001 obs. of 6 variables:
## $ president : chr "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" ...
## $ date : chr "1945-06-06" "1945-06-07" "1945-06-08" "1945-06-09" ...
## $ days : int 55 56 57 58 59 60 61 62 63 64 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ approve_estimate : chr "87" "87" "87" "87" ...
## $ disapprove_estimate: chr "3" "3" "3" "3" ...
I'd run the resultant data frame(s) through readr::type_convert() to get better types.
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
add a comment |
1
They generally load data asynchronously via XHR requests which you can see if you open Developer Tools in your browser and re-load the page. In Network -> XHR you'll see lots of lovely JSON:
I don't know which one you want (I skimmed the Q) but you can get all the main JSON files easily:
polls <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/polls.json")
str(polls, 1)
## 'data.frame': 3401 obs. of 14 variables:
## $ id : int 77261 77265 77272 77249 77257 77266 77596 77246 77263 77253 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ sampleSize : int 1992 1500 1190 1043 1500 2692 1712 1500 1500 1991 ...
## $ population : chr "rv" "a" "rv" "rv" ...
## $ weight : num 0.946 0.245 1.645 1.166 0.639 ...
## $ grade : chr "B-" "B" "A-" "B" ...
## $ multiversions: logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ url : chr "http://www.politico.com/story/2017/01/poll-voters-liked-trumps-inaugural-address-234148" "http://www.gallup.com/poll/201617/gallup-daily-trump-job-approval.aspx" "https://poll.qu.edu/national/release-detail?ReleaseID=2415" "http://www.publicpolicypolling.com/pdf/2015/PPP_Release_National_12617.pdf" ...
## $ created_at : chr "2017-01-23" "2017-01-23" "2017-01-26" "2017-01-25" ...
## $ startDate : chr "2017-01-20" "2017-01-20" "2017-01-20" "2017-01-23" ...
## $ endDate : chr "2017-01-22" "2017-01-22" "2017-01-25" "2017-01-24" ...
## $ pollster : chr "Morning Consult" "Gallup" "Quinnipiac University" "Public Policy Polling" ...
## $ tracking : chr "" "T" "" "" ...
## $ answers :List of 3401
approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/approval.json")
str(approval, 1)
## 'data.frame': 2751 obs. of 9 variables:
## $ date : chr "2017-01-23" "2017-01-23" "2017-01-23" "2017-01-24" ...
## $ future : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ subgroup : chr "Adults" "All polls" "Voters" "Adults" ...
## $ approve_estimate : chr "45" "45.46693" "46" "45" ...
## $ approve_hi : chr "51.1347" "50.88971" "52.29238" "50.98562" ...
## $ approve_lo : chr "38.8653" "40.04416" "39.70762" "39.01438" ...
## $ disapprove_estimate: chr "45" "41.26452" "37" "45.74659" ...
## $ disapprove_hi : chr "51.1347" "46.68729" "43.29238" "51.73221" ...
## $ disapprove_lo : chr "38.8653" "35.84175" "30.70762" "39.76097" ...
historic_approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/historical-approval.json")
str(historic_approval, 1)
## 'data.frame': 26001 obs. of 6 variables:
## $ president : chr "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" ...
## $ date : chr "1945-06-06" "1945-06-07" "1945-06-08" "1945-06-09" ...
## $ days : int 55 56 57 58 59 60 61 62 63 64 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ approve_estimate : chr "87" "87" "87" "87" ...
## $ disapprove_estimate: chr "3" "3" "3" "3" ...
I'd run the resultant data frame(s) through readr::type_convert() to get better types.
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
add a comment |
1
1
1
They generally load data asynchronously via XHR requests which you can see if you open Developer Tools in your browser and re-load the page. In Network -> XHR you'll see lots of lovely JSON:
I don't know which one you want (I skimmed the Q) but you can get all the main JSON files easily:
polls <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/polls.json")
str(polls, 1)
## 'data.frame': 3401 obs. of 14 variables:
## $ id : int 77261 77265 77272 77249 77257 77266 77596 77246 77263 77253 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ sampleSize : int 1992 1500 1190 1043 1500 2692 1712 1500 1500 1991 ...
## $ population : chr "rv" "a" "rv" "rv" ...
## $ weight : num 0.946 0.245 1.645 1.166 0.639 ...
## $ grade : chr "B-" "B" "A-" "B" ...
## $ multiversions: logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ url : chr "http://www.politico.com/story/2017/01/poll-voters-liked-trumps-inaugural-address-234148" "http://www.gallup.com/poll/201617/gallup-daily-trump-job-approval.aspx" "https://poll.qu.edu/national/release-detail?ReleaseID=2415" "http://www.publicpolicypolling.com/pdf/2015/PPP_Release_National_12617.pdf" ...
## $ created_at : chr "2017-01-23" "2017-01-23" "2017-01-26" "2017-01-25" ...
## $ startDate : chr "2017-01-20" "2017-01-20" "2017-01-20" "2017-01-23" ...
## $ endDate : chr "2017-01-22" "2017-01-22" "2017-01-25" "2017-01-24" ...
## $ pollster : chr "Morning Consult" "Gallup" "Quinnipiac University" "Public Policy Polling" ...
## $ tracking : chr "" "T" "" "" ...
## $ answers :List of 3401
approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/approval.json")
str(approval, 1)
## 'data.frame': 2751 obs. of 9 variables:
## $ date : chr "2017-01-23" "2017-01-23" "2017-01-23" "2017-01-24" ...
## $ future : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ subgroup : chr "Adults" "All polls" "Voters" "Adults" ...
## $ approve_estimate : chr "45" "45.46693" "46" "45" ...
## $ approve_hi : chr "51.1347" "50.88971" "52.29238" "50.98562" ...
## $ approve_lo : chr "38.8653" "40.04416" "39.70762" "39.01438" ...
## $ disapprove_estimate: chr "45" "41.26452" "37" "45.74659" ...
## $ disapprove_hi : chr "51.1347" "46.68729" "43.29238" "51.73221" ...
## $ disapprove_lo : chr "38.8653" "35.84175" "30.70762" "39.76097" ...
historic_approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/historical-approval.json")
str(historic_approval, 1)
## 'data.frame': 26001 obs. of 6 variables:
## $ president : chr "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" ...
## $ date : chr "1945-06-06" "1945-06-07" "1945-06-08" "1945-06-09" ...
## $ days : int 55 56 57 58 59 60 61 62 63 64 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ approve_estimate : chr "87" "87" "87" "87" ...
## $ disapprove_estimate: chr "3" "3" "3" "3" ...
I'd run the resultant data frame(s) through readr::type_convert() to get better types.
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
They generally load data asynchronously via XHR requests which you can see if you open Developer Tools in your browser and re-load the page. In Network -> XHR you'll see lots of lovely JSON:
I don't know which one you want (I skimmed the Q) but you can get all the main JSON files easily:
polls <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/polls.json")
str(polls, 1)
## 'data.frame': 3401 obs. of 14 variables:
## $ id : int 77261 77265 77272 77249 77257 77266 77596 77246 77263 77253 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ sampleSize : int 1992 1500 1190 1043 1500 2692 1712 1500 1500 1991 ...
## $ population : chr "rv" "a" "rv" "rv" ...
## $ weight : num 0.946 0.245 1.645 1.166 0.639 ...
## $ grade : chr "B-" "B" "A-" "B" ...
## $ multiversions: logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ url : chr "http://www.politico.com/story/2017/01/poll-voters-liked-trumps-inaugural-address-234148" "http://www.gallup.com/poll/201617/gallup-daily-trump-job-approval.aspx" "https://poll.qu.edu/national/release-detail?ReleaseID=2415" "http://www.publicpolicypolling.com/pdf/2015/PPP_Release_National_12617.pdf" ...
## $ created_at : chr "2017-01-23" "2017-01-23" "2017-01-26" "2017-01-25" ...
## $ startDate : chr "2017-01-20" "2017-01-20" "2017-01-20" "2017-01-23" ...
## $ endDate : chr "2017-01-22" "2017-01-22" "2017-01-25" "2017-01-24" ...
## $ pollster : chr "Morning Consult" "Gallup" "Quinnipiac University" "Public Policy Polling" ...
## $ tracking : chr "" "T" "" "" ...
## $ answers :List of 3401
approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/approval.json")
str(approval, 1)
## 'data.frame': 2751 obs. of 9 variables:
## $ date : chr "2017-01-23" "2017-01-23" "2017-01-23" "2017-01-24" ...
## $ future : logi FALSE FALSE FALSE FALSE FALSE FALSE ...
## $ subgroup : chr "Adults" "All polls" "Voters" "Adults" ...
## $ approve_estimate : chr "45" "45.46693" "46" "45" ...
## $ approve_hi : chr "51.1347" "50.88971" "52.29238" "50.98562" ...
## $ approve_lo : chr "38.8653" "40.04416" "39.70762" "39.01438" ...
## $ disapprove_estimate: chr "45" "41.26452" "37" "45.74659" ...
## $ disapprove_hi : chr "51.1347" "46.68729" "43.29238" "51.73221" ...
## $ disapprove_lo : chr "38.8653" "35.84175" "30.70762" "39.76097" ...
historic_approval <- jsonlite::fromJSON("https://projects.fivethirtyeight.com/trump-approval-ratings/historical-approval.json")
str(historic_approval, 1)
## 'data.frame': 26001 obs. of 6 variables:
## $ president : chr "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" "Harry S. Truman" ...
## $ date : chr "1945-06-06" "1945-06-07" "1945-06-08" "1945-06-09" ...
## $ days : int 55 56 57 58 59 60 61 62 63 64 ...
## $ subgroup : chr "All polls" "All polls" "All polls" "All polls" ...
## $ approve_estimate : chr "87" "87" "87" "87" ...
## $ disapprove_estimate: chr "3" "3" "3" "3" ...
I'd run the resultant data frame(s) through readr::type_convert() to get better types.
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
answered Nov 23 '18 at 3:06
hrbrmstrhrbrmstr
60.6k687148
60.6k687148
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439989%2fgetting-error-trying-to-scrape-data-from-fivethirtyeight%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I only see one table on that page, so I think
nth(2)should benth(1).– Marius
Nov 23 '18 at 2:40
They have alot of data here: github.com/fivethirtyeight/data (not sure if this is there, tho)
– hrbrmstr
Nov 23 '18 at 2:58