Tukukkk

I'm currently working with data from an online database. I access the data via API, which works when I retrieve all data at once. But this makes my system slow, so I want to make a request only for filtered data (which I did not make until now). This is the way to get the whole dataset:

#-------------------------------#

#          packages             #

#-------------------------------#

library(httr)

library(jsonlite)



#-------------------------------#

#         API requests          #

#-------------------------------#



##  get all data at once  ##

url <- "https://www.eter-project.com/api/3.0/HEIs/full"

raw_result <- GET(url)



#-------------------------------#

#       data processing         #

#-------------------------------#



# 'status_code' (if request worked) and 'content' (APIs answer) important

names(raw_result) 



# '200' tells us that server received request

raw_result$status_code 



# translate Unicode into text

this.raw.content <- rawToChar(raw_result$content) 



# transform json into workable format for R

mydata <- fromJSON(this.raw.content, flatten = TRUE) 



class(mydata)

dim(mydata)

According to the documentation (https://www.eter-project.com/api/doc/#/) I need a POST request using url https://www.eter-project.com/api/3.0/HEIs/queryand a filter embedded in the following structure:

{

 "filter": {},

  "fieldIds": {}

}

I want to filter for years and countries in order to only get the data I currently want to work with. The structure for the filter would be
{ "BAS.REFYEAR.v": 2011, "BAS.COUNTRY.v": "AT"}.

Has anyone an idea, how I could implement this into a POST request?

Until now, I made some desperate attempts to include the filter into the POST requests (e.g. raw_result <- POST(url, body = list({
"filter": {"BAS.REFYEAR.v" = 2011}}), encode = "json")
and played around with the mongolitepackage, which was not even close.

UPDATE: the filtering problem has been solved. I used the following solution:

myquery <- '{

  "filter": {"BAS.REFYEAR.v": 2015, "BAS.COUNTRY.v": "LV"},

  "fieldIds": {},

  "searchTerms": 

  }'



url <- "https://www.eter-project.com/api/3.0/HEIs/query"



raw_result <- POST(url, body = myquery, content_type_json())

Now, I face another problem: the data include many special characters, which are not displayed properly in R (e.g. Alberta koledža in the dataset is displayed as Alberta koledžain R). Is there a way to solve this, for example by using UTF-8 in the request call?

You can try to build the required JSON as list of lists. However, I find it easier to supply the JSON explicitly and add the content type manually:

query <- '{

  "filter": { "BAS.REFYEAR.v": 2011,  "BAS.COUNTRY.v": "AT"},

  "fieldIds": {},

  "searchTerms": 

}'

url <- "https://www.eter-project.com/api/3.0/HEIs/query"

raw_result <- httr::POST(url = url, body = query, content_type_json())

After this you can apply your processing as before.