Tukukkk

I am trying to parse the articles from this site: https://derstandard.at/r2000026008978/Wirtschaftspolitik?_chron=t

When you visit the website for the first time, you are prompted to accept cookies. It seems that they store the consent in DSGVO_ZUSAGE_V1:true because when I scrape like this it works:

def start_requests(self):

    urls = [

        'https://derstandard.at/r2000026008978/Wirtschaftspolitik?_chron=t'

    ]

    for url in urls:

        yield scrapy.Request(url=url, callback=self.parse,

                              cookies={'DSGVO_ZUSAGE_V1':'true'})



def parse(self, response):

    base_query = '//div[contains(@class,"contentLeft")]/ul[contains(@class, "stories")]/li'

    articles = response.xpath(base_query)

    for index, value in enumerate(articles):

        article_url = validateResponse(response,

                                       base_query + '/div[contains(@class,"text")]/h3/a/@href',

                                       index)

        request = scrapy.Request(response.urljoin(article_url), callback=self.parseSingleArticle, cookies={'DSGVO_ZUSAGE_V1':'true'})

        yield request



def parseSingleArticle(self, response):

    article_content = ''

    article_date = validateResponse(response, "//h6[contains(@class,'info')]/span[contains(@class,'date')]/text()")

    article_title = validateResponse(response, "//h1[contains(@itemprop,'headline')]/text()")

    query = "//div[contains(@class,'copytext')]//child::text()"

    article_content_response = response.xpath(query)

    for index, value in enumerate(article_content_response):

        article_content += " " + validateResponse(response, query, index)

    yield self.article_to_pipeline(article_content, response.url, article_date, article_title)



def article_to_pipeline(self, article_content, url, article_date, article_title):

    article_item = ArticleItem()

    # some other stuff

    return article_item

And for some articles this works perfect. From the debug:

2018-11-24 10:01:58 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://derstandard.at/2000091420120/60-Stunden-pro-Woche-Fuer-Unternehmer-ist-das-die-Realitaet>

Cookie: DSGVO_ZUSAGE_V1=true; MGUID=GUID=e2b4bb79-069f-4778-8062-f34d7b3d2b9d&Timestamp=2018-11-24T09:01:52&DetectedVersion=Web&Version=&Hash=4F3A498256857F21390A3DD636588B38

And I get everything I wanted for this article.

But then there are some articles where it doesn't work. For example this doesn't return anything:

2018-11-24 10:02:07 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://derstandard.at/2000090136491/Lufthunderter-fuer-E-Autos-faellt>

Cookie: DSGVO_ZUSAGE_V1=true; MGUID=GUID=e2b4bb79-069f-4778-8062-f34d7b3d2b9d&Timestamp=2018-11-24T09:01:52&DetectedVersion=Web&Version=&Hash=4F3A498256857F21390A3DD636588B38

Also when I access this website through Chrome I get the "accept cookie prompt". Despite having accepted this cookie before for some other page on their website. When I do that they again save the confirmation under DSGVO_ZUSAGE_V1:true and MGUID (Deleting them from chrome puts the "accept cookie prompt" back).

Anybody any ideas? I have tried other cookies but MGUID and DSGVO_ZUSAGE_V1 are the only ones that make a difference.