Building a webscraper site
![building a webscraper site building a webscraper site](https://webscraper.io/documentation/images/open-web-scraper/open-web-scraper-chrome.png)
set( 'SESSION', 'Y2IxOWE0YWQtNmM0Ni00ZWYzLTkzYmItOGI4YWQ0MDI1MTg4') # copy & paste the SESSION cookie value obtained from request headers This is how I tried to get the same response from the server programmatically: main.py 1 The requests package is more than capable of setting cookies and making simple HTTP requests. The SESSION cookie looks promising, doesn’t it? I did what anybody does when they need to quickly put together a script to get something done I created a main.py file. I looked at the cookies in the Request Cookies section: Firsty, I needed to find out what additional information I had to send with my request to make the server treat me as a logged-in user. The first step of my plan was to write a script that makes a GET request to that endpoint and checks if delivery is available for my neighborhood within the next 4 days. I found out that there was an endpoint for checking live delivery availability, and it was getting hit automatically when a logged-in user visited the website.
![building a webscraper site building a webscraper site](https://i.ytimg.com/vi/yMFt8byX0es/maxresdefault.jpg)
I looked at the Network tab to see which HTTP request was returning the above response. I didn’t want to resort to violence but after yet another failed attempt yesterday, I got really frustrated and finally unsheathed Chrome developer tools. Probably as soon as new slots are opened, they get occupied in minutes by a few dozen lucky people who happen to be online at that time. “We do not have delivery to the neighborhood you have chosen for the next 4 days.” I’ve frequently visited the online store for the past two weeks and attempted to place an order, but I always got this message (translated from Turkish): Even though their delivery system works from 8:30 AM to 10:00 PM every day, it’s virtually impossible to find an empty slot, that is if you play nicely. When everybody decided to switch to online shopping all of a sudden, they couldn’t handle that demand spike. But they don’t have unlimited resources, obviously. Migros Sanalmarket is one of them and it’s arguably the best one. Here’s the deal very few supermarket chains in Turkey have online stores. It’s good to know that people are acting responsibly, but I never thought this could mean that I won’t be able to do online grocery shopping anymore! Now it’s not a matter of personal preference anymore everybody stays at home and especially avoids crowded indoor places like supermarkets. Unlike many people I know, I always preferred online shopping over going to the supermarket. Grocery shopping has been one of my least favorite chores, even before the pandemic.