articles

The Scoop on Search Engines

Reprinted with permission from the STC Intercom magazine - January 2001 Volume 48, Issue 1.

Whether you are building a personal Web site to showcase the oh-so-cute photos of your newborn child or building a corporate Web site for your company, once the site is complete and launched, you probably breathe a sigh of relief, thinking your work is done. On the contrary--that’s when most of your work begins! You need to make certain that people can find your site. A couple of years ago, I redesigned a site for a Native American newspaper, the Oklahoma Indian Times (OKIT), whose old site was getting 6,000 hits a month. I registered the new site with five search engines, but there was still a lot I could do to help users find it.

Get Searched

Think of search engines as big phone books: People enter keywords to be searched for or scroll through directories to find what they’re looking for. If your site is not registered with a particular search engine, its users may not find your site. To complicate matters, there are a lot of search engines out there with varying search capabilities. It’s up to you to help them locate your site.

A directory groups registered sites into a searchable categorized listing: A familiar example is Yahoo, whose homepage is divided into subject topics. Other big ones include LookSmart, About.com, and NBCi. A database holds information that you can access through a search query. Both directories and databases can use robots or spiders to gather information from the Internet.

You can target specific areas within the directories. For example, when I registered the OKIT site with the Yahoo directory, I told Yahoo how to categorize my site. So the Yahoo directory includes the OKIT site under “News and Media/Newspapers/By Region/U.S. States/Oklahoma,” as well as under “Society and Culture/Cultures and Groups/Cultures/American/Native American/News and Media/Newspapers.” Whenever I failed to find a Native American category under the Society and Culture section of a search engine or directory, I sent an email politely asking if that category could be added. So if you don’t find the category you are looking for, ask the directory to add it. (Your site will be the only one listed in that category, at least for a while.)

If you don’t believe that registering with search engines will increase your site’s hits, I am here to tell you it works! Once I registered the new OKIT site with Yahoo, the hits increased to 20,000 in just two weeks.

Get Crawled

Some directories and databases allow users to perform sophisticated searches using deep or general search engines. These engines use a robot (also called a spider) program to "crawl" the Internet searching for new or updated pages. These pages are added to a large database maintained by the search engine. When a user submits a search query, the database returns the links to the relevant Web pages. The quality of the database (and thus the search results) depends on the sophistication of the robot.

A robot finds pages in two ways: by visiting pages of sites that are registered with its search engine and by following links on these pages. The robot visits any subpages connected to the page it's reading, then visits all the external links on the pages, and so on. (Note: The robot can follow links on a page providing that the page is not in frames.) So if your page isn't linked from anywhere else and you haven't registered it, there's no way it will ever be found--it will be invisible. You'll want to register with the major general search engines: AltaVista, Excite, Hotbot, Infoseek, Lycos, MSN search, and Webcrawler. Other search engines to target include Anzwers, Go2Net (you will pay for listings on this one), and Northern Light.

I redesigned the OKIT site a couple of times. In the second redesign, I had around 500 pages on the site. So I placed both sets of site navigation (side and bottom) into separate HTML files and wrote a script that called these pages into the main page. This was a great idea for maintenance--I had to update only one file, instead of 500-plus pages, when I added a new section to the Web site. But the separate navigation pages cut down on the chances of the search engine spider crawling the new pages. To increase the chances of the site getting crawled, I added a "crawler page."

An article I found on the Webmonkey site showed me how to create a crawler page--an HTML page listing every URL I wanted indexed by search engines. I linked the crawler page from the top of my main page so the spider would find it immediately. The more URLs you list, the better the chances that users will find your site.

But there are still more ways to get crawled. Spiders gather information about a Web site through its TITLE tags, META tags, and keywords. Confused? Relax. There are a lot of Web sites that teach you how to deal with all those evil META tags and keywords. It's easy once you get the hang of it. You might visit the Art of Business Web Site Promotion Web site. This site has a tutorial on how search engines work and how to optimize Web pages for the best results. It teaches you how to use META tags correctly and how to register your site with search engines. Another excellent site, Search Engine Watch (formerly called A Webmaster's Guide To Search Engines), explains how search engines find and rank Web pages, with an emphasis on how Webmasters can improve the way search engines list their Web sites.

A tip that I learned at the Web Design 99 Conference in Atlanta is that search engines look at TITLE tags first. So put as many of your keywords as possible in the TITLE tag; of course, the title will appear on your Web site, so be sure to make it legible and appealing.

Subject-Specific Search Engines

There are subject-specific search engines, but you have to find them. Subject-specific search engines are large, highly developed databases or Internet directories that concentrate in particular subject areas. Since OKIT features Native American news, issues, and events, I registered the site with several Native American-specific search engines, such as www.nativesearch.com.

Some directories and databases catalog selected Web sites and search engines by subject. The Directory Guide lists about 400 directories and search engines in various categories. You can also try I-Sleuth and Beaucoup.

There are services that offer to submit your site to 400 search engines for a price, but they do not consider the subject matter of your site. And most of the search engines to which they submit sites are not often used anyway. I prefer to submit my sites myself--it is time-consuming but well worth the effort.

Figuring out search engines is not so difficult, and it’s well worth the effort. All it really takes is doing some homework. By spending a little time and effort, I increased the hits on the OKIT site from 6,000 to over 170,000 a month.

References

Boutin, Paul. "Sending Search-Engine Traffic to Your Site." Webmonkey. July 27, 1999.

Return to top.