reinhard.fun Blog
Go Back
Python / Scrapy
2025-11-22
Tool
The tool being used is
Scrapy
Hints
settings.py
#Obey robots.txt rules
ROBOTSTXT_OBEY
=
False
