Through the normal course of events, search engines will eventually crawl through the links of your site and index your files. A proper structure would include a general sitemap that contains every link you want indexed (with a link from your homepage running directly to it for easy access by robots) and every page linked in some manner to other pages so as there are clear flowing file paths for robots to follow. Any files that are not linked to other pages or a sitemap risk not being indexed, as they are isolated & cut off from the normal file flow pattern of pathways.

Once you are confident every file is linked in some way so that it is visible to robots, you will want to create a sitemap.xml file. The structure looks like this:

-<urlset xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
<url><loc>http://rjwattenhofer.com/</loc><lastmod>2013-05-21T13:55:02+00:00</lastmod>
     <changefreq>daily</changefreq><priority>1.00</priority></url>
-<url><loc>http://rjwattenhofer.com/technical/ping/</loc><lastmod>2013-05-21T13:55:02+00:00</lastmod>
     <changefreq>daily</changefreq><priority>0.85</priority></url>
-<url><loc>http://rjwattenhofer.com/contact-wattenhofer/</loc><lastmod>2013-05-21T13:55:02+00:00</lastmod>
     <changefreq>daily</changefreq><priority>0.85</priority></url>

I used three pages as examples of the structure. You will want to include every page of your website on this .xml sitemap. Fortunately, you do not have to do this by hand. There are free sitemap generators online that you can use which will quickly index your entire site and produce a downloadable .xml file.

Upload this file to your root directory. Pull up a browser window and locate the direct path to your new file: It will be 'http://yourdomain.com/sitemap.xml'. Open up your index.html (or whatever your homepage is named: index.php, etc) and place the following code in the head section:

<link rel="sitemap" type="application/xml" title="Sitemap" href="http://rjwattenhofer.com/sitemap.xml"/>

As mentioned, it is best to place the .xml file in the root directory because this is where robots begin their crawl and they will easily locate it in this spot.

Create Google & Bing accounts and access their Webmaster Tools section. They will have a handy area which prompts you to submit your sitemap. Copy & paste the full sitemap path into this submit form and send it on its way. Now these search engines know your sitemap exists and will index your pages on a more frequent basis.

Sometimes you will create fresh content on your site that you would like indexed right away. You can ping these search engines, alerting them to the fact you have new content (I will ping them as soon as I finish this article). Here are the Ping pathways:

http://www.google.com/webmasters/sitemaps/ping?sitemap=http://yourdomain.com/sitemap.xml

http://www.bing.com/webmaster/ping.aspx?siteMap=http://yourdomain.com/sitemap.xml

Place your domain in the proper spot in the pathways, paste the pathway into your address bar and hit enter. You will immediately receive ping success notification.

~ RJ Wattenhofer, May 21, 2013