Photo credit to Clément Hélardot — that’s JavaScript in the photo, not Python, but I liked the way this looked.

Python Tutorial: Let’s build a web crawler for sitemaps

A quick, niche tutorial for crawling internal webpages on a website.

Ben Scheer
3 min readOct 12, 2021

--

Have you ever wanted to crawl a website, but specifically one that only visits each page of that website, and ignores other stuff like external links? Okay, no worries if not, but if so, this article could be helpful.

--

--