Alan Freedman -- The Computer Language Company - Computer Desktop Encyclopedia
Computer Desktop Encyclopedia
Longest-Running Tech Reference on the Planet

A CDE Definition

You'll love The Computer Desktop Encyclopedia (CDE) for Tech Term of the Day (TTOD)

LOOK UP ANOTHER TERM


robots.txt

A text file placed in the root directory of a website that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.



spider

(1) Also known as a "crawler," "robot" (bot) and "intelligent agent," a spider is a program that searches for information on the Web. Spiders are widely used by Web search engines to index all the pages on a site by following the links from page to page. The search engine summarizes the content and adds the links to their indexes. Spiders are also used to locate Web pages that sell a particular product or to find blogs that have opinions about a product. See surface Web and bot.

(2) Software that indexes a single website for browsing offline. See offline browser.

(3) (Spider) A gaming platform from AMD. See Phenom.



Personal Use Only

Before/After Your Search Term
BeforeAfter
RoboCADRobson
robocallRobson cache
robotRobson flash memory
robot carrobust
robot taxiROC
robotic assistantRock Ridge file system
robotic process automationRocket eBook
robotic surgeryRocketBook
roboticsROCOF
robots exclusionROFL

Terms By Topic
Click any of the following categories for a list of fundamental terms.
Computer Words You Gotta KnowSystem design
Job categoriesUnix/Linux
Interesting stuffPersonal computers
InternetIndustrial Automation/Process Control
Communications & networkingAssociations/Standards organizations
HistoryDesktop publishing
Audio/VideoGraphics
MainframesSecurity
ProgrammingHealthcare IT
System design