[ Previous | Home | Index | Next ]Part 9: Understand Your Engines Effective searching requires understanding how best to utilize the features of your search services. But, Internet searching is a highly-competitive, dynamic area. New search engines are cropping up continually, others are folding or being acquired, and feature sets change almost daily in order to keep pace. This part is a comprehensive overview of the state of search services on the Internet as of Spring 1999, updated from the first version a year previous. When published, it was already possibly dated. The authors therefore take no responsibility for the accuracy or completeness of the information herein. Hey, we're just doing the best job we can. But we do make mistakes .... Topic 34: Some Caveats: The Dynamic Search Business Topic 34: Some Caveats: The Dynamic Search Business Searching on the Internet extends from the quick question, for which a lot of information is known to exist, to serious and purposeful research on esoteric topics. Casual users simply surfing or posing the quick question likely do not need an understanding of query syntax and construction nor search engine features and operation. This tutorial is definitely geared to those who want to spend the time to get more enjoyment and results from serious searches. As of early 1997, some 600 search services were known to exist on the Internet. Recent citations have noted as many as 1,800 and one Web site, www.beaucoup.com, includes references to more than 1,400 [15] our own estimate is 11,000 on the low side and likely many times more that amount [16, 38]. Recent major engines including Galaxy, Magellan and WebCrawler, have gone out of business or been acquired by competitors. Major partnerships have been formed and some apparently separate engines, such as AOL NetFind, are branded implementations of other services (in this case, Excite). Entirely new services, such as Direct Hit, have also begun in the past year and achieved early prominence. The industry is clearly in flux. This dynamism makes it impossible to keep absolutely current on the state of Internet search services. The information presented herein is a best-faith effort to provide an accurate snapshot of its state as of Spring 1999. The authors or BrightPlanet.com LLC make no representations as to the accuracy or completeness of the information presented. The authors do not intend endorsement by virtue of whether a search service is listed herein. The decision as to which engines to include as major ones comes from one of the more authoritative Web sites on search engines, www.searchenginewatch.com [39]. The engines included in that service were used to define which search services in this tutorial were classified as "major." Additional updates of this tutorial are likely. The authors welcome identification of errors or provision of additional, useful information. These updates and corrections will be reflected in future versions. |
| Topic 35: Duplication, Coverage and Responsiveness
The best estimates of the number of publicly-available documents on the Internet hover around 800 million [11]. The fact that the numbers available are simply estimates and differ greatly is an indication of how little is truly known about the size of the Internet and the completeness with which search services cover it. The same Science article by Steve Lawrence and Lee Giles of the NEC Research Institute from which the larger estimate was drawn also is the reference for much of the information on search engine duplication and coverage. Lawrence and Giles (L&G)analyzed coverage of 575 mostly scientific or technical queries posed by researchers at their institute in December 1997. Krishna Bharat and Andrei Broder (B&B) of the Digital Systems Research Center recently conducted a similar study with nearly equivalent methodology [21]. Here are their findings for coverage of the Internet by six of the major services, all of which do full-text indexing (in other words, a directory service like Yahoo was not included in their analysis): |
|
||||||||||||||||||||||||||||||||||||||||
| The combined coverage figure refers to what percentage of searches were successfully returned by that engine. Because none of the engines comprehensively covered the Internet, the percent coverage of the total Web represents the authors' estimate of gaps and overlap.
Three additional conclusions from these L&G study deserve mention. First, submitting queries to multiple search engines greatly increases the amount of results obtainable. They estimated that combining queries to the six engines studied increased the likelihood of finding results by a factor of 3.5. Second, they found surprisingly little duplication between the engines. With the largest two engines, HotBot and AltaVista, the number of duplicates was only 18% [40]. And, third, they found that "dead links," that is pages listed on the search engines but no longer in existence, ranged from 1.6% to 5.3%. Though not universally true, there tended to be a correlation of engines that indexed more documents, such as HotBot, with a higher incidence of dead links. This result should not be surprising, in that significant effort must be expended to maintain a larger database, and the room for error and untimeliness is higher. Of course, size is not all that matters on the Internet. Many search engines justifiably make the argument that better and more accurate beats bigger. As a searcher, your interests should be on the quality of results. What perhaps is most disturbing, then, is that many quality results may not be indexed by the major engines in use. This possible lack of coverage is likely not a concern if the search topic is one of a broad, widespread nature. But, if looking for technical information or that which is inherently not part of the mainstream, these results are not comforting. There is perhaps a serious methodological flaw at the heart of the Science article analysis. Recall two things: First, the subject of the analysis was technical queries; and, second, the nature of how items get listed initially by search engines. Full-text search engines get their listings in one of two ways. Either a site developer submits one or more Web addresses asking the engine to index it (in which case it is then scheduled for a later full-site indexing). Or, the 'spiders' used by the engines to find new content on the Web encounter the site and then include it. Spiders depend on linkages from prior sites to identify new ones. Information tucked away in the nooks and crannies of the Internet — in other words, some of the most specific information you may be trying to obtain — may have few if any links to them. Without links, or without prior notification by the developers, spiders will only chance upon new sites. Because businesses tend to actively seek listings on search engines, it is not at all clear that the lack of coverage implied by the Science article would apply to this sector. By focusing on technical searches, the authors could therefore have significantly overestimated the lack of coverage on the Internet. Whether coverage is better or worse for different subject areas or for different focuses on the Web is unknown at this time. As professional information searchers have come to well understand, individual search engines can return outstanding results that are found on no other engines [41]. For this reason, and the reason of inadequate coverage by those engines, you should always submit your important queries to multiple search engines. |
![[You should always use multiple search services for your important queries.]](images/srchtip16.gif)
One of the main conclusions of both studies is that no search engine indexes more than about one-third to one-half of the publicly-available documents on the Internet. By applying these figures to the known documents these services have indexed as of late 1997, the authors were able to come up with their estimates of 200 million to 320 million total documents on the Web. Even still, the authors believed their size estimate to be a lower bound, expecting the "true size of the Web to be much larger" than their methodology suggests. | Topic 36: Boolean or Not?
For serious searching, perhaps the most important first choice facing you is choice of search engines. Which search engines better cover the topics you are interested in? Which support the search features that will enable you to find what you want? Not all searches are created equal. The increasing ability of some search engines to take your requests in context, and then enable you to narrow results based on your first attempt, is a promising development. Certainly being able to type in a few words and then begin receiving documents of value bodes well for common-topic searches. We ourselves use this approach when quick searches are needed. We doubt, however, the ability of search engines in the near term to improve on this process for complicated searches or for hard-to-find information. Not only is coverage of such topics weak for a given engine, but the ability to anticipate refinements is weakened by the need to categorize information into levels insufficiently specific to the difficult query.
This recommendation, however, exacerbates the lack of coverage of any given search engine. By definition, hard-to-find information is not well-indexed, meaning you will likely need to use more than one search engine to get the robust results you desire. Topic 37: A Comparison of 100 Search Services A listing comparing major features of 100 of the largest search services on the Web is shown below. For a larger listing of about 2,500 to 3,000 search services, see [16]. |
![[Use search engines with full-text indexing and Boolean support for your most demanding queries.]](images/srchtip17.gif)
Thus, for difficult search topics, we still must recommend the use of search engines with full Boolean support. Only you know what information you are seeking (even though it may be ill-defined or abstract). With full Boolean searching, you have complete control to find what you seek.
|
| Topic 38: Features of the Top 10 Search Services
Based on March 1999 rankings from Media Metrix [42], and including service characterizations from Search Engine Watch [39] and Search Engine Showdown [43], the chart below compares features for the major search services on the Web. Included in this listing are search engines (SE), and directories (D). For a further description of search service types, see Topic 2; for a description of the features listed, see Topic 33. Specific notes on some of the services are appended at the end of the table. Some of the listed features are coded. These codes represent the authors judgment as to the completeness of a feature compared to other services in the listing:
or does not provide full functionality A blank means that service does not offer the feature shown. As before, we do not imply endorsement nor claim complete accuracy for the features presented. You are always advised to consult the online help topics for any given services. Features and sometimes syntax change on a periodic basis. |
![[ y ]](images/srchyoption.gif){kind=small}
means the feature is deemed to be as complete as others![[ y/n ]](images/srchynoption.gif){kind=small}
means the feature is not as complete as others offered\
|
| The key for how the services determine relevance is: 1 -- 3/4 star review; 2 -- metatag keywords; 3 -- title keywords, popularity; 4 -- none; 5 -- in title, higher in category tree.
All of the directory services link to a standard search engine if their own listings do not satisfy the query. The directories and their associated engines are: Yahoo! - Inktomi; About.com - AltaVista; and LookSmart - AltaVista. There are also differences between the services that license the Inktomi engine: Snap, HotBot and GoTo. While all of these index and score pages in a similar manner, the options presented to the user can differ quite substantially, with HotBot providing the most power, GoTo the simplest interface. Specific service notes are: YA — people searching uses the Four11 specialty engine The options shown in the table are often noted by different terms by the services that support them, and usually involve special syntax rules. Sometimes, too, the descriptions of how these features operate is difficult to find from the main pages of the services. Directly consult each service's home page; and, then, try consulting advanced or power searching, the help sections or the frequently asked questions (FAQ) areas to read about the special operators and their rules. Another useful resource, though based on relatively small sample sizes, is Greg Notess' Search Engine Showdown [43]. This site reports dead link percentages, unique hits, overlap and some other different statistics. Specialty engines have the advantage of cataloging information particular to a narrow topic area, thus potentially increasing coverage versus the general search services. This advantage, however, often comes at the cost of not providing you with the search options and flexibility that the general services provide. One of the most complete catalogs of Internet search engines is found at www.beaucoup.com, listed below for English-oriented services, by its breakdown of more than 1,400 search engines and major topic area [44]: |
|
| Be aware some of these services catalog information that is not normally spidered or indexed by the general search services.
Not shown on the table above are searches in languages other than English. For example, major search alternatives are provided in the languages of Dutch, Spanish, German, Japanese, French, and specialty search services are offered in perhaps another 30 languages or so. For regional alternatives, Yahoo alone provides 12 different country-based search services and another 12 focused specifically on individual U.S. metropolitan areas. Similar diversification is occurring with other major search services. These specializations are natural and reflect the huge size of the Internet (plus, obviously, the fact that English is not the only language used on the Web!). This specialization trend is likely to continue. Other search engine directories that are comprehensive listings of other specialty engines on the Web are: http://www.searchpower.com The latter in fact seems to have the largest listing; you will need to poke around some to get them. To page individually through about 26 pages (about 2,500 engines), try this URL, and then continue paging using the 'Next Page' option at the bottom of each page: http://www.finderseeker.com/cgi-bin/search.cgi?disp=99&sp=1&cat=&key=&country= Depending on the topics of your searches, you are encouraged to test out and try these listing services. There are a number of other search services that bear watching, either because of new and unique search technology, or because of partnering or plans that may cause them to become big players:
Topic 41: Some Perplexing Behaviors Search engines may not always perform as indicated on their help pages. These differences are due to constant changes in how they handle their service, strange quirks relating to their scoring and indexing methodologies, errors made by the developers of Web pages, and decisions the service may make to speed performance at high-traffic volume periods. We can illustrate some of these quirks using our standard AltaVista search source [1]. Let's take the most simple example we've used throughout this tutorial: posing the query query. Depending on the time of day, during a 24 hour period AltaVista returned results ranging from 671,424 counts to 712,851 counts. This indicates that at times of high traffic limits are placed internally on the search. In the worst cases, AltaVista or other engines may even provide a message that the server is busy, and prompt you to return at a later time to obtain results. Most often when this error occurs, a quick re-issue of the query will obtain results. Here's another example of quirky behavior when comparing results counts from two different formats (text vs. phrase) using our basic query, and using the simple and advanced search forms on AltaVista:
The advanced form returned only half of the results of the simple (this query was posed at a low-demand time). Sometimes, too, the unquoted version on the advanced search turns up no results matching the query. Clearly, both time of day and search form can affect results. Order on a seemingly equivalent pair of queries can also produce slightly different results. Let's compare these two queries: "bird*" AND falcon The first query produces a results count of 9,581; the second a count of 9,228. Of course, counts are not what you the searcher wants when you search. The actual results pages for these examples were quite similar. But it's useful to realize that how an engine operates exactly may not be clear or consistent. As Topic 21 and Topic 26 note, you must also exercise care in the use of capitalization, special characters and special terms. What you are thinking you are posing as a query may not be evaluated as such by specific engines. This is not meant to be a criticism of the search engines, or of AltaVista in particular. Many anomalies occur because of improperly formatted Web pages. And, after all, engines are indexing millions of pages in very short periods of time and need to provide snappy response in all instances. The fact they do accurately index very high percentages is remarkable. But, you, as a searcher, should be aware results are not foolproof. |
|||||||||
|
|
Copyright © 2001. BrightPlanet.com LLC. All rights reserved.
Privacy and site use policies.
Problems or suggestions? Report quality suggestions or site problems here.