Class BRobotsTxt 

java.lang.Object
   |
   +----java.util.Vector
           |
           +----BRobotsTxt
public class BRobotsTxt
extends Vector
This class uses a weak implementation of the Robot Exclusion Standard as it is used commonly throughout the web. There is an Internet-Draft for a more rigid version. Since this is yet not a standard and the author believes that the existing convention offers good oppotunity to protect websites from being indexed unwillingly, the following implementation fits the requirements ;-) For more information on this topic vistit http://info.webcrawler.com/mak/projects/robots/.

Version:
0.1
Author:
Simon Berg

Constructor Index

o BRobotsTxt(URL, String)
An object of this class represents the robot-protection information of one internet site.

Method Index

o isAllowed(URL)
Returns true if robot-access of url is allowed, false otherwise

Constructors

o BRobotsTxt 
 public BRobotsTxt(URL url,
                   String robotName) throws NoRobotInformationException
An object of this class represents the robot-protection information of one internet site.

Parameters:
url - can be a site name with directory name and file name.
robotName - will be processed case-insensitive
Throws: NoRobotInformationException
if no information is found.

Methods

o isAllowed 
 public boolean isAllowed(URL url)
Returns true if robot-access of url is allowed, false otherwise