Разбираем статистику сайта. Поисковые системы.

При просмотре логов сайта натыкаешься на названия поисковых ботов, их огромное количество, хотя мы привыкли пользоватся лишь несколькими основными. Как можно использовать эту информацию? Во-первых, правильно прописать действия роботов при обращении к файлу robot.txt. Во-вторых, зная название поисковика можно заставить его быстрее индексировать информацию на сайте, например с помощью пингования. У многих стоит на сайте собственная поисковая система. Так вот палю тему тем, кто еще не в курсе. Результаты запросов локального поисковика тоже можно скармливать для индексации поисковым системы и сервисам социальных закладок, т.к. поисковик будет видеть такую ссылку как отдельную страницу на сайте. Названия поисковых ботов нужны и при написании собственных систем статистики.

Основные поисковые системы в русском интернете

yandex.ru, rambler.ru, google.ru, mail.ru, aport.ru, nigma.ru, metabot.ru

Глобальные поисковики

google.com, yahoo.com, search.msn.com, aol.com, myway.com, ask.com, search.netscape.com

Названия поисковых ботов (USER_AGENT)

abcdatos=ABCdatos BotLink

acme.spider=Acme.Spider

ahoythehomepagefinder=Ahoy! The Homepage Finder

alkaline=Alkaline

anthill=Anthill

antibot=Antibot

aport=Aport

aport=Aport Bot

appie=Walhello appie

arachnophilia=Arachnophilia

arale=Arale

araneo=Araneo

architext=ArchitextSpider

archive_org=Archive.org Bot

aretha=Aretha

ariadne=ARIADNE

arks=arks

Ask Jeeves=Ask Jeeves Bot

ask jeeves=Ask Jeeves Bot

aspider=ASpider (Associative Spider)

atn.txt=ATN Worldwide

atomz=Atomz.com Search Robot

auresys=AURESYS

awbot=AWBot

backrub=BackRub

baiduspider=BaiDuSpider

bbot=BBot

BecomeBot=Become Bot

bigbrother=Big Brother

bjaaland=Bjaaland

blackwidow=BlackWidow

blindekuh=Die Blinde Kuh

bloodhound=Bloodhound

bobby=Bobby

booch=Booch Bot

borg-bot=Borg-Bot

boris=Boris

brightnet=bright.net caching robot

bspider=BSpider

bumblebee=Bumblebee (relevare.com)

cactvschemistryspider=CACTVS Chemistry Spider

calif=Calif

cassandra=Cassandra

cgireader=Digimarc Marcspider/CGI

checkbot=Checkbot

christcrawler=ChristCrawler.com

churl=churl

cienciaficcion=cIeNcIaFiCcIoN.nEt

collective=Collective

combine=Combine System

conceptbot=Conceptbot

coolbot=CoolBot

core=Web Core / Roots

cosmos=XYLEME Robot

cruiser=Internet Cruiser Robot

cscrawler=CsCrawler

cusco=Cusco

cyberspyder=CyberSpyder Link Test

daviesbot=DaviesBot

desertrealm=Desert Realm Spider

deweb=DeWeb© Katalog/Index

dienstspider=DienstSpider

digger=Digger

digout4u=Digout4u

diibot=Digital Integrity Robot

direct_hit=Direct Hit Grabber

dnabot=DNAbot

download_express=DownLoad Express

dragonbot=DragonBot

dwcp=DWCP (Dridus' Web Cataloging Project)

ebiness=EbiNess

echo=EchO!

e-collector=e-collector

elfinbot=ELFINBOT

emacs=Emacs-w3 Search Engine

emcspider=ananzi

esther=Esther

evliyacelebi=Evliya Celebi

exactseek=ExactSeek Crawler

ezresult=sEzresult

fastcrawler=FastCrawler

fast-webcrawler=Fast-Webcrawler

fdse=Fluid Dynamics Search Engine robot

felix=Felix IDE

ferret=Wild Ferret Web Hopper #1, #2, #3

fetchrover=FetchRover

fido=fido

finnish=Hдmдhдkki

fireball=KIT-Fireball

fish=Fish search

fouineur=Fouineur

francoroute=Robot Francoroute

freecrawl=Freecrawl

funnelweb=FunnelWeb

gama=gammaSpider, FocusedCrawler

gazz=gazz

gcreep=GCreep

getbot=GetBot

GetSmart=GetSmart Bot

geturl=GetURL

GigaBlast=Gigablast Bot

gigabot=GigaBot

gnodspider=sGNOD Spider

golem=Golem

googlebot=Google Bot

googlebot=Google Bot

googlebot=Googlebot

grapnel=Grapnel/0.01 Experiment

griffon=Griffon

gromit=Gromit

grub=Grub.org

grub-client=GrubClient Bot

gulliver=Northern Light Gulliver

gulperbot=Gulper Bot

hambot=HamBot

harvest=Harvest

havindex=havIndex

henrythemiragorobot=Mirago

holmes=Holmes

hometown=Hometown Spider Pro

htdig=ht://Dig

htmlgobble=HTMLgobble

hyperdecontextualizer=Hyper-Decontextualizer

ia_archiver=Alexa (IA Archiver)

ia_archiver=Archive.org Bot

ia_archiver=IA.Archiver Bot

iajabot=iajaBot

iconoclast=Popular Iconoclast

ilse=Ingrid

imagelock=Imagelock

incywincy=IncyWincy

informant=Informant

infoseek=InfoSeek Robot 1.0

infoseeksidewinder=Infoseek Sidewinder

infospider=InfoSpiders

inspectorwww=Inspector Web

intelliagent=IntelliAgent

internetseer=InternetSeer

irobot=I, Robot

iron33=Iron33

israelisearch=Israeli-search

javabee=JavaBee

jbot=JBot Java Web Robot

jcrawler=JCrawler

jeeves=AskJeeves

jennybot=JennyBot

Jetbot=Jet Bot

jobo=JoBo Java Web Robot

jobot=Jobot

joebot=JoeBot

jubii=The Jubii Indexing Robot

jumpstation=JumpStation

justview=JustView

kapsi=image.kapsi.net

katipo=Katipo

kilroy=Kilroy

ko_yappo_robot=KO_Yappo_Robot

labelgrabber.txt=LabelGrabber

larbin=larbin

legs=legs

libwww=Punto Bot

linkbot=LinkBot

linkchecker=LinkChecker

linkidator=Link Validator

linkscan=LinkScan

linkwalker=LinkWalker

lockon=Lockon

logo_gif=logo.gif Crawler

lycos_=Lycos

Lycos=Lycos Bot

lycos=Lycos.com Bot

macworm=Mac WWWWorm

magpie=Magpie

marvin=marvin/infoseek

mattie=Mattie

mediafox=MediaFox

mediapartners-google=Google AdSense

mercator=Mercator

merzscope=MerzScope

meshexplorer=NEC-MeshExplorer

metager-linkchecker=MetaGer LinkChecker

microsoft_url_control=Microsoft URL Control

mindcrawler=MindCrawler

MnoGoSearch=mnoGoSearch Bot

mnogosearch=mnoGoSearch search engine software

moget=moget

momspider=MOMspider

monster=Monster

motor=Motor

msiecrawler=MSIECrawler

msnbot=MSN Bot

msnbot=MSN Bot

msnbot=MSNBot

muncher=Muncher

muscatferret=Muscat Ferret

mwdsearch=Mwd.Search

myweb=Internet Shinchakubin

nagios=Nagios

NaverBot=Naver Bot

ndspider=NDSpider

nederland.zoek=Nederland.zoek

netcarta=NetCarta WebMap Engine

netcraft=Netcraft

netmechanic=NetMechanic

netscoop=NetScoop

newscan-online=newscan-online

nhse=NHSE Web Forager

nomad=Nomad

northstar=The NorthStar Robot

nzexplorer=nzexplorer

objectssearch=ObjectsSearch

oBot=oBot

occam=Occam

octopus=HKU WWW Octopus

Openbot=Openfind Bot

openfind=Openfind data gatherer

orb_search=Orb Search

packrat=Pack Rat

pageboy=PageBoy

parasite=ParaSite

patric=Patric

pegasus=pegasus

perignator=The Peregrinator

perlcrawler=PerlCrawler 1.0

perman=Perman surfer

petersnews=Petersnews

phantom=Phantom

phpdig=PhpDig

piltdownman=PiltdownMan

pimptrain=Pimptrain.com's robot

pioneer=Pioneer

pitkow=html_analyzer

pjspider=Portal Juice Spider

plumtreewebaccessor=PlumtreeWebAccessor

pompos=Pompos

poppi=Poppi

portalb=PortalB Spider

psbot=psbot

python=The Python Robot

rambler=Rambler Bot

Rambler=Rambler Bot

rambler=StackRambler

raven=Raven Search

rbse=RBSE Spider

redalert=Red Alert

resumerobot=Resume Robot

rhcs=RoadHouse Crawling System

road_runner=Road Runner: The ImageScape Robot

robbie=Robbie the Robot

robi=ComputingSite Robi/1.0

robocrawl=RoboCrawl Spider

robofox=RoboFox

robozilla=Robozilla

roverbot=Roverbot

rules=RuLeS

safetynetrobot=SafetyNet Robot

scooter=Altavista Bot

scooter=Scooter

search_au=Search.Aus-AU.COM

search-info=Sleek

searchprocess=SearchProcess

senrigan=Senrigan

sgscout=SG-Scout

shaggy=ShagSeeker

shaihulud=Shai'Hulud

shoutcast=Shoutcast Directory Service

sift=Sift

simbot=Simmany Robot Ver1.0

sitetech=SiteTech-Rover

site-valet=Site Valet

skymob=Skymob.com

slcrawler=SLCrawler

slurp@inktomi=Hot Bot

slurp@inktomi=Hot Bot

slurp=Inktomi Slurp

slysearch=SlySearch

smartspider=Smart Spider

snooper=Snooper

solbot=Solbot

speedy=Speedy Spider

spider_monkey=spider_monkey

spiderbot=SpiderBot

spiderline=Spiderline Crawler

spiderman=SpiderMan

spiderview=SpiderView™

spry=Spry Wizard Robot

ssearcher=Site Searcher

StackRambler=Rambler Bot

StackRambler=StackRambler Bot

suke=Suke

suntek=suntek search engine

surveybot=SurveyBot

sven=Sven

tach_bw=TACH Black Widow

tarantula=Tarantula

tarspider=tarspider

techbot=TechBOT

templeton=Templeton

titan=TITAN

titin=TitIn

tkwww=The TkWWW Robot

tlspider=TLSpider

turnitinbot=Turn It In

TurnitinBot=TurnitinBot

turtle=Turtle

turtlescanner=Turtle

TurtleScanner=TurtleScanner Bot

ucsd=UCSD Crawl

udmsearch=UdmSearch

ultraseek=Ultraseek

unlost_web_crawler=Unlost Web Crawler

urlck=URL Check

valkyrie=Valkyrie

Vampire=Net_Vampire Bot

verticrawl=Verticrawl

victoria=Victoria

visionsearch=vision-search

voidbot=void-bot

voila=Voila

voyager=Voyager

vwbot=VWbot

w3index=The NWI Robot

w3m2=W3M2

wallpaper=WallPaper (alias crawlpaper)

wanderer=the World Wide Web Wanderer

wapspider=w@pSpider by wap4.com

webbandit=WebBandit Web Spider

webbase=WebBase

webcatcher=WebCatcher

webclipping.com=WebClipping.com

webcompass=webcompass

webcopy=WebCopy

WebCrawler=Fast Bot

webfetcher=webfetcher

webfoot=The Webfoot Robot

webinator=Webinator

weblayers=weblayers

weblinker=WebLinker

webmirror=WebMirror

webmoose=The Web Moose

webquest=WebQuest

webreader=Digimarc MarcSpider

webreaper=WebReaper

websnarf=Websnarf

webspider=WebSpider

webvac=WebVac

webwalk=webwalk

webwalker=WebWalker

webwatch=WebWatch

WebZIP=WebZIP Bot

WhatUSeek=What You Seek Bot

whatuseek=What You Seek Bot

whatuseek=whatUseek Winona

whowhere=WhoWhere Robot

wired-digital=Wired Digital

wisenutbot=WISENutbot

wmir=w3mir

wolp=WebStolperer

wombat=The Web Wombat

wonderer=spider: Web Wombat Redback Spider

worm=The World Wide Web Worm

wwwc=WWWC Ver 0.2.5

wz101=WebZinger

xget=XGET

Yahoo!=Yahoo Bot

Yahoo-MMCrawler=Yahoo-MMCrawler Bot

yahoo-verticalcrawler=Yahoo Vertical Crawler

Yandex=Yandex Bot

zealbot=ZealBot

ZipppBot=ZipppBot

zyborg=Zyborg