Visiting and Crawling a Website

The Search Engine process starts by a crawler visiting a website and downloading the content.  A crawler is a program written with the task of downloading content from website pages on a schedule basis.  The list of websites and pages for a crawler to visit come from many sources, such as:

• Submitting a website or page to a Search Engine via a form at their website
• The discovery of new pages or sites by the crawler when downloading a page
• Submission or download of a sitemap

When a crawler visits a website, there are many ways to control its behavior so that content you do or don’t want to show in the search results has the desired effect.  The crawler behavior can be modified using the following techniques.

• robots.txt – The robots.txt file goes in the website root and contains rules for a crawler to follow when visiting your website.  You can allow or deny a crawler access to specific directories or files based on the user agent.  Wikipedia has an excellent article on the details of the robots.txt file and links to additional resources.
• Meta Tags – Meta Tags can be used to control the behavior of a Search Engine crawler on a page by page basis.  In most cases, using the robots.txt file is all that is needed to control what the crawler sees on your website.
• Nofollow Attribute – This attribute allows the webmaster to control which links will pass credit on to the pages they link to and which ones won’t.  This is useful for links added in forums or blog comments.

The Search Engine Index

Each Search Engine has their own algorithm used to sort and rank the crawled web pages.  Most Search Engines have many similarities in the way they sort the pages.  The built index determines what pages will be shown to the user for a particular search query.

The algorithms used to build an index look for things such as how many times a word or phrase shows up on a page, the font size of a word or phrase, the amount of content on the page, formatting, user experience, meta tags, and the number of links to a page.  There are many tutorials and guides available on the Internet on best practices when developing web sites and pages to maximize the user experience and the ability of a Search Engine to crawl the website pages.

Searching and the Display of Search Results

A search is performed using a keyword or group of words to form a phrase.  Most Search Engines include more advanced search functions, such as using Boolean operators and keywords meant to perform a specific function.

Once the search query is entered and submitted, the Search Engine consults its index to return a number of pages that best fix the query according to their algorithm.  Search results can vary from search to search and even more so in different geographic regions.  The search results page is constantly changing and it is rare for a website or page to remain at the same ranking for a long period of time.

Some Important SEO Factors

Some important factors for SEO are HTML syntax, page title, page description, keyword research, links, and the amount of content on a page. Well-formed HTML syntax makes the web page much easier to read, not only by other people, but by search engines as well. The goal is to make your pages as easy for the search engines to crawl as possible.

The page title and description are very important for search ranking, as well as the search results page. The title is placed in between the “title” tags in the header section. The description is placed in a “meta” tag and is also located in the header section. When looking at a search results page, the title will most often show as the title of the search result, and in many cases, the description with be used for the search entry description.

Keyword research is important to determine what terms to use in your page. There are a number of keyword tools available to aid in the search. The most important factor to consider in keyword research is to try and figure out what keywords a visitor would enter into a search engine to find your page. Working these keywords into the page content can help increase the visibility of your page for that particular search.

Links are important to SEO for many reasons. You can look at a link as a vote for your web page. Some websites will have more influence than others when linking to your page. For example, an established and popular website will have more influence than a new site.

The amount of content on a page is important because the search engines need to have enough content to figure out what your page is about. Having a good amount of content is important to the user experience as well. The search engines are trying to provide their searchers with the best experience possible when returning a list of results. By keeping the user experience in mind when developing and writing content for your website, you should achieve good results with the search engines.