- A sitemap will boost my SERP Rank.
- A sitemap is needed for good SEO
- If it is in my sitemap, then it will be crawled and included in the Search Engine Index.
- If it is not in my sitemap, the Search Engine will just ignore it.
- With a sitemap my pages will be listed in the Search Engine Immediately!
- What I put on my sitemap is more important and has a higher priority.
- Sitemaps are a requirement on the Internet.
Have you ever heard, or thought, any of the above? If yes, you are normal and have studied the issue of sitemaps on the Internet. However, you may be surprised to find out that All of the above are myths! Nothing could be farther from the truth!
Mysteries of the Internet!
With DNS being the number one mystery on the Internet; .htaccess being some kind of voodoo magic; and robots.txt (not) blocking anything; sitemaps rate high on the list of misunderstood parts of the Internet! So this post is going to set the record straight and give you a clue as to what sitemaps actually are used for.
I Can’t Find You!
Believe it or not, a search engine like “Ask Bing Google” (No, Bing Google is not a person!), cannot find anything on its own! NOTHING! So a search engine must be told where to look. It starts with a list of domain names supplied by the domain registry. alantait.net is on that list. And that is all a search engine needs to get started!
Through the magic of DNS, if you type in alantait.net it will bring you to a server. That server was programed that if you do not include a file name, it will serve you a default file. That default file has (almost always) links to other pages on the Internet. Links are the basic search engine food!
The First Site Map
If you had a link to every page on your site, all nicely listed on the default page, Then people could get to every page. That is exactly how website used to be at one very early point, and basically that was the very first Site Map!
Groups and Categories
Later websites started grouping their pages, like pages for boys and pages for girls. Pages for fruits in one section and pages for Vegetables in another. Everything was ok as long as the links to get from here to there and all the pages in between were intake. But someone deleted one of those pages and created a “broken link!” Broken links are still a problem for search engines today.
Site Map Version 2.0
Instead of going back to putting a link to every page of our site on the landing page (default page, home page, front page – it is all the same thing!), a link to a “Site Map”, “Site-Map” or Sitemap page was made and the sitemap was actually a list of links to all the other pages in the site.
Beauty and Function
It took only a day or two and sitemaps started looking more beautiful. In fact, some website owners added some helpful information, like the date the site was created and, more importantly, the last date and time a webpage was modified! Some website owners went as far as to suggest the importance of a particular page expressed as a percentage. 100% would be very important indeed!
Suggestions and Requests
Nonetheless, these pages were still just list of links to internal pages and at best just suggestions or requests of what may be of some importance. Other pages (not on the sitemap) may be of more importance or value to the reader (or search engine). The request was to the search engine to please include these pages, but it was still the search engine who decided what to crawl and what to index.
THIS HAS NOT CHANGED!
Sitemaps Are Inclusive, Not Exclusive!
The sitemaps list of URLs will be included into the list of URLs the search engine found in other ways. Like Links! So let’s say you have a well designed website that is divided into three categories, ‘services’, ‘new products’ and ‘how-to’! Some pages are on more than one list or category. Now let’s say you create a sitemap list of links, and you decide to list every page that is NOT on the ‘how-to’ list. That will not help you, because the ‘how-to’ pages will most likely be linked to, by links on the ‘services’ or ‘new products’ pages!
The sitemap URLs will be included into the URLs the search engine found via the links. Nothing is exclusive in sitemaps. So you went to a lot of work for nothing! Your unlisted pages will still be discovered, and listed in the search engine in just exactly the same way the pages on the sitemap are.
Sitemap in Different Terms…
“Sir, in case you have not noticed these URL’s, here is a list I think are important.”
Now consider yourself, giving that list, on a real piece of paper, to the president of the USA! What do you think he will do with it?
Ok, it is a little better than that.
It’s All About URLs
A search engine Mainly finds URLs by links to them from other places. A sitemap is simply an “other place” that you tell the search engine about via one of 3 or 4 ways.
- A Link On A Website (xml or html)
- Ping A Sitemap Address
- Submit To Webmaster Tools
- A robots.txt File
#4 Frankly I really do not like the last #4, the robots.txt file. The reason is because robots.txt is a prime target for evil spiders and bad bots to attack your site. Why help them?
#3 Is a better option, but it does mean Manually submitting your sitemap – for each site! If you are an admin with a few thousand or even a few hundred sites, that is really the pits and very time consuming. If you only have one site, go for it! I hardly every use this.
#2 Is the one I am in love with! I set it up once, set it and forget it. My sitemaps are built automatically, on the fly, and renewed each time I publish a new post or page or edit one, or when one automatically goes active at a future date! The problem is most XML sitemap plugins do not do this well so I have to write my own handler, but it works great! I am hoping a new sitemap plugin will have something like my code for everyone!
#1 You may laugh, but this is a fall back to the old days and every search engine will follow a normal link. Look at the bottom of my page and you will see, right after the copyright, is a link to the Sitemap.
This page has users from about 100 countries, but it has no sitemap listed on the robots.txt file. In fact you will notice that I have almost nothing listed in robots.txt – what ever you put there is just food for the evil people! I get all this coverage and attention from Ask Bing Google despite the fact I have never submitted a sitemap to Webmaster Tools (either Google or Bing!).
I depending solely upon methods #2 and #1 to provide Search Engines with my sitemap!
It works great!
How a Search Engine Works
When a search engine finds a link it puts it on a list and gets around to crawling it.
A sitemap is just a list of links.
Before it crawls your site, it decides if that link has any value. For example, if you are coming from a well known evil site, they likely have you blacklisted already! If they do not like you (your domain name, network, or IP address), they will just ignore you!
After they crawl your site, they will evaluate it to see if they want to add you to their index. It is completely up to them and totally optional.
In the early days, a sitemap was just a page on your website that had a link to every other page, so a crawler (spider, bot) could find all your pages. Today, it is basically the same thing, with a little added useful information like when was the page last modified!
But it is still just a link, like any other link.
A search engine includes this link. Nothing is excluded.
A Sitemap is Only a Suggestion!
The Sitemap Myths
Now let’s take a look at each of the Myths About Sitemaps, and see if we can determine how they error and how they got started.
A sitemap will boost my SERP Rank.
The Search Engine Results Page Rank is based on a lot of different things, but you will find good original content and quality incoming links related to your topic matter the largest factors. A sitemap is not a factor in SERP Rank. However, a sitemap may be helpful to a search engine to be able to find your webpage so it can crawl and rank it. In other words, you need some pages linked to your page.
A sitemap is needed for good SEO
Search Engine Optimization is not necessarily related to a sitemap. Organization, content, people with genuine interest and links to your site are much more important to SEO. On the other hand, a sitemap may help you to organize your thoughts in regard to these, and will insure that a search engine can find your pages to crawl and index them.
If it is in my sitemap, then it will be crawled and included in the Search Engine Index.
Most likely true in the case of anyone reading this, however, all major search engines blacklist domain names, networks and IP address. Only if you are a safe friendly publisher will Search Engines be interested in crawling your site. Even if your site is crawled, it does not mean it will be Indexed! Especially duplicate content is rejected.
If it is not in my sitemap, the Search Engine will just ignore it.
Wrong! If it is not linked to the search engine will ignore it – but almost everything is linked to at someplace.
With a sitemap my pages will be listed in the Search Engine Immediately!
All search engines need time to crawl pages, and then they decide IF they choose to index them!
What I put on my sitemap is more important and has a higher priority.
Except for no index directives, everything else should be considered a request or a suggestion, nothing more!
Sitemaps are a requirement on the Internet.
No sitemap has ever been required. Remember a sitemap is just included into the list of URLs that has already been discovered for your domain. Although not required, a good XML sitemap is a very good idea for suggesting pages to search engines.
