This term refers to the spider or crawler that goes to a website to examine the content. The spider or crawler is a search engine robot that has the job of categorizing Internet sites. As the robot works to categorize a site, the site is “being spidered” at that time.