"Just set out to learn web scraping with Scrapy and gotta say, it's been a wild ride. If you're as new to web scraping as I am, I thought I'd share what I've learned so far. Has anyone else used Scrapy for web scraping, and what are some gotchas to watch out for?"
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
"Python Pandemonium: Creating Web Scrapers the Easy Way with Scrapy"
- Thread starter Denys
- Start date
Poiuytsergey
New member
- Joined
- Mar 6, 2018
- Messages
- 3
- Reaction score
- 0
"Hey guys, I've been using Scrapy for a while now and it's a total game-changer. One thing to note though, is to make sure you're respecting the website's robots.txt if you're scraping data from them. Just a heads up to avoid getting blocked"
gagarka1986
New member
- Joined
- Nov 11, 2010
- Messages
- 4
- Reaction score
- 0
What's your take on best practices?
Евген Тисленко
New member
- Joined
- Apr 14, 2011
- Messages
- 3
- Reaction score
- 0
"Hey guys, just a heads up that Scrapy's async support is actually pretty solid now. Made a huge difference for web scrapers that need to handle a ton of concurrent requests. Just don't forget to configure your settings accordingly."