Are you playing with the possibilities of Java? This article explores in detail how to use Java's Web Crawler class and methods. It is excerpted from chapter six of The Art of Java, written by Herbert Schildt and James Holmes (McGraw-Hill, 2004; ISBN: 0072229713).
Crawling the Web with Java - Compiling and Running the Search Web Crawler (Page 15 of 15 )
As mentioned earlier, SearchCrawler takes advantage of Java’s new regular expression package: java.util.regex. The regular expression package was introduced in JDK 1.4; thus you will need to use JDK 1.4 or later to compile and run SearchCrawler.
Compile SearchCrawler like this:
Run SearchCrawler like this:
Search Crawler has a simple, yet feature-rich, interface that’s easy to use. First, in the Start URL field, enter the URL at which you want your search to begin. Next, choose the maximum number of URLs you want to crawl and whether or not you want to limit crawling to only the Web site specified in the Start URL field. If you want the crawler to continue crawling until it has exhausted all links it finds, you can leave the Max URLs to Crawl field blank. Be forewarned, however, that choosing not to set a maximum value will likely result in a search that will run for a very long time.
Next you’ll notice a Matches Log File specified for you. This text field is prepopulated, specifying that the log file be written to a file called crawler.log in the directory that you are running the Search Crawler from. If you’d like to have the log file written to a different file, simply enter the new filename. Next, enter the string you want to search for and then select whether or not you want the search to be case sensitive. Note that entering a search string containing multiple words requires matching pages to include all the words specified.
Once you have entered all your search criteria and configured the search constraints, click the Search button. You’ll notice that the search controls become disabled and the Search button changes to a Stop button. After searching has completed, the search controls will be reenabled, and the Stop button will revert back to being the Search button. Clicking the Stop button will cause the crawler to stop crawling after it has finished crawling the URL it is currently crawling. Figure 6-2 shows the Search Crawler in action.
A few key points about Search Crawler’s functionality:
Only HTTP links are supported, not HTTPS or FTP.
URLs that redirect to another URL are not supported.
Similar links such as “http://osborne.com” and “http://osborne.com/” (notice the trailing slash) are treated as separate unique links. This is because the Search Crawler cannot categorically know that both are the same in all instances.
Figure 6-2.The Search Crawler in action
Web Crawler Ideas
Search Crawler is an excellent illustration of Java’s networking capabilities. It is also an application that demonstrates the core technology associated with Web crawling. As mentioned at the start of this chapter, although Search Crawler is useful as is, its greatest benefit is as a starting point for your own crawler-based projects.
To begin, you might try enhancing Search Crawler. Try changing the way it follows links, perhaps to use depth-first crawling rather than breadth-first. Also try adding support for URLs that redirect to other URLs. Experiment with optimizing the search, perhaps by using additional threads to download multiple pages at the same time.
Next, you will want to try creating your own crawler-based projects. Here are some ideas:
Broken Link Crawler A broken link crawler could be used to crawl a Web site and find any links that are broken. Each broken link would be recorded. At the end of crawling, a report would be generated listing each page that had broken links along with a breakdown of each broken link on that page. This application would be especially useful for large Web sites where there are hundreds if not thousands of pages to check for broken links.
Comparison Crawler A comparison crawler could be used to crawl several Web sites in an effort to find the lowest price for a list of products. For example, a comparison crawler might visit Amazon.com, Barnes& Noble.com, and a few others to find the lowest prices for books. This technique is often called “screen scraping” and is used to compare the price of many different types of goods on the Internet.
Archiver Crawler An archiver crawler could be used to crawl a site and save or “archive” all of its pages. There are many reasons for archiving a site, including having the ability to view the site offline, creating a backup, or obtaining a snapshot in time of the Web site. In fact, a search engine’s use of crawler technology is actually in the capacity of an archiver crawler. Search engines crawl the Internet and save all the pages along the way. Afterward they go back and sift through the data and index it so it can be searched rapidly.
As the preceding ideas show, crawler-based technology is useful in a wide variety of Web-based applications.
DISCLAIMER: The content provided in this article is not warranted or guaranteed by Developer Shed, Inc. The content provided is intended for entertainment and/or educational purposes in order to introduce to the reader key ideas, concepts, and/or product reviews. As such it is incumbent upon the reader to employ real-world tactics for security and implementation of best practices. We are not liable for any negative consequences that may result from implementing any information covered in our articles or tutorials. If this is a hardware review, it is not recommended to open and/or modify your hardware.