Major Search Engine Robot User-Agents

Here are a list of the top search engines and the names of their spiders.

  • AltaVista - Scooter
  • AllTheWeb/Fast - Fast
  • Google - Googlebot
  • Inktomi - Slurp
  • Lycos - Lycos

Why Hackers Love The Robots.txt File

Did you know that unless you cloak (hide) your robots.txt file, anyone can access the file by simply entering the robots.txt file address into a web browser. I've read stories about unscrupulous people searching the robots.txt file to uncover hidden private files. So be careful what you include in the robots.txt file.

2 Free Robots.txt File Management Tools

Here are 2 free tools to help you manage your robots.txt files:

How To Use Meta Refresh Tag & JavaScript Redirect

If you want to automatically redirect or forward a visitor to another webpage or site, use a meta refresh tag or JavaScript redirect.

How To Use A Meta Refresh Tag

The meta refresh tag will instruct the web browser to refresh (reload) the page, using the same or different URL, after a designated time has elapsed. To add a meta refresh tag to a page, place the tag in between the HEAD section of the page.

Here is a sample refresh tag that reloads the page with "new-page.html" after 3 seconds:

<META HTTP-EQUIV="refresh" CONTENT="3; url=new-page.html">

Here's a sample refresh tag to reload the page a website URL after 5 seconds:

<META HTTP-EQUIV="refresh" CONTENT="5; url=">

If you want to refresh the same page, simply leave out the second part of the CONTENT attribute.

For example:

<META HTTP-EQUIV="refresh" CONTENT="30'>

If you want to refresh your page after a certain period of time, I suggest that you set a time of 30 seconds or more, as most search engines consider anything less than 30 seconds as spam.

How To Use A JavaScript Redirect

If you really must forward or redirect your visitors to a new page or site within 30 seconds, I recommend using a JavaScript redirect instead of the Meta refresh tag to keep the search engines happy.

Search engines are less likely to detect the use of JavaScript redirects to redirect users, because JavaScript has so many uses. The problem with the Meta refresh tag is that it only has one use.

To add a JavaScript redirect to a page, place the tag in between the HEAD section of the page.

Sample JavaScript redirect to replace the current page with a new page:

<SCRIPT LANGUAGE="javascript">

You should also include a Meta refresh tag, just in case the user has switched off JavaScript, or is using an older browser that does not support JavaScript. Approximately 10 percent of internet users use web browsers that either have JavaScript manually switched off, or don't support JavaScript.

The Meta refresh tag should be placed in a "noscript" element, so that a browser does not execute both the JavaScript redirect and Meta refresh.

Here is a sample Meta refresh tag within a "NOSCRIPT" element:

<META HTTP-EQUIV="refresh" CONTENT="30; url=">

How To Change Header Font Sizes With CSS

One common use for CSS is to change the properties of header tags and links. This is useful when you want to highlight an important keyword phrase in a page, but don't want to have a huge heading on the page.

For example, this reduces the size of the heading text to medium.

<H1 STYLE="font-size: medium">Search Engine Optimization</H1>