A CDE Definition
A text file placed in the root directory of a website that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.
(1) Also known as a "crawler," "robot" (bot) and "intelligent agent," a spider is a program that searches for information on the Web. Spiders are widely used by Web search engines to index all the pages on a site by following the links from page to page. The search engine summarizes the content and adds the links to their indexes. Spiders are also used to locate Web pages that sell a particular product or to find blogs that have opinions about a product. See surface Web and bot.
(2) Software that indexes a single website for browsing offline. See offline browser.
(3) (Spider) A gaming platform from AMD. See Phenom.
Before/After Your Search Term
|robot||Robson flash memory|
|robotic assistant||Rock Ridge file system|
|robotic process automation||Rocket eBook|
Terms By Topic
Click any of the following categories for a list of fundamental terms.