Major Search Engine Robot User-Agents
Here are a list of the top search engines and the names of their spiders.
- AltaVista - Scooter
- AllTheWeb/Fast - Fast
- Google - Googlebot
- Inktomi - Slurp
- Lycos - Lycos
Why Hackers Love The Robots.txt File
Did you know that unless you cloak (hide) your robots.txt file, anyone can access the file by simply entering the robots.txt file address into a web browser. I've read stories about unscrupulous people searching the robots.txt file to uncover hidden private files. So be careful what you include in the robots.txt file.
2 Free Robots.txt File Management Tools
Here are 2 free tools to help you manage your robots.txt files:
- Search Engine Promotion Help - Analyses your robots.txt file and reports any syntax errors and warnings on the use of new features that are not yet widely deployed.
- he Web Robots Database - A database of currently known robots with descriptions and contact details.
How To Use A Meta Refresh Tag
The meta refresh tag will instruct the web browser to refresh (reload) the page, using the same or different URL, after a designated time has elapsed. To add a meta refresh tag to a page, place the tag in between the HEAD section of the page.
Here is a sample refresh tag that reloads the page with "new-page.html" after 3 seconds:
<META HTTP-EQUIV="refresh" CONTENT="3; url=new-page.html">
Here's a sample refresh tag to reload the page a website URL after 5 seconds:
<META HTTP-EQUIV="refresh" CONTENT="5; url=http://www.google.com">
If you want to refresh the same page, simply leave out the second part of the CONTENT attribute.
<META HTTP-EQUIV="refresh" CONTENT="30'>
If you want to refresh your page after a certain period of time, I suggest that you set a time of 30 seconds or more, as most search engines consider anything less than 30 seconds as spam.
Here is a sample Meta refresh tag within a "NOSCRIPT" element:
<META HTTP-EQUIV="refresh" CONTENT="30; url=http://www.google.com">
How To Change Header Font Sizes With CSS
One common use for CSS is to change the properties of header tags and links. This is useful when you want to highlight an important keyword phrase in a page, but don't want to have a huge heading on the page.
For example, this reduces the size of the heading text to medium.
<H1 STYLE="font-size: medium">Search Engine Optimization</H1>