Google Robots.txt Generator – New Feature Introduced

Google announced a new tool in Google Web Master Tool for robots.txt generation. This Robots.txt generator tool gives and easy and interactive way to create a robots.txt file. To create a Robotos.txt file using Google webmaster tool simple go to Google Webmaster Tool, after login to webmaster tool go to Tools section and then click on Generate robots.txt. This tool is easy to use, enter the url or directories that you don’t want to crawled by search engines.

Isn’t it simple?

Once you are finished with creating robotos.txt don’t forget to analysis with robots.txt analyzer, this tool is also available in Google Web Master Tool under Tools section.

Regards,
Bhavesh Goswami.

PR Update - 29th Feb Sat 2008

Hi All,

At saturday morning when i came to my office i noticed some changes in PR of my websites. If anyone noticed PR update then let me know and also let me know the web url of your website so i can check it.

Regards,:-)
Bhavesh Goswami.

MSN Announced: New Improved Live Search Engine Crawler

On 12th Feb MSN officially announced a new improved MSN crawler that is more powerful in caching and indexing more pages of website. Many webmaster complaints about MSN crawler crawling less pages but with this new improved crawler you will see more pages indexed by MSN. May be this new improved feature will help webmaster to make website more visible to MSN Search Engine.

HTTP Compression: This feature compressed the file size of webpage and reduces time of downloading data from server to their crawler.

Condition Get: This feature will ask the server if the page has been changed before last time crawl the page. MSN crawler will include “If-Modified-Since” header and get the time of last download using GET request. If the page has not been changed after the last download the server will respond 304 HTTP Response.

To reflect these changes they upgraded their user agent to msnbot/1.1.

PR Updation has taken place

Hi All,

Today in the morning when i visited my website i found all my website get Google page rank.

Also Google now showed up more link back to all my websites.

This is great and very quick.

Regards,
Bhavesh Goswami.

How to remove URL from Google and other search engines?

There are many ways to remove pages from Google cache

.htaccess
Do not link to the page
Robots.txt
Nofollow Meta tag
Noindex
url removal tool

.htaccess:- Using htaccess file you can redirect one url to another url. Also using htaccess you can make sub directory password protected so Google and other search engines can’t guess the password. You can find ready tools that generate .htaccess password protected file. This is very handy tool to remove your web urls from Google and other search engine.

Do not link to the page: - This is not a good way to remove pages from search engine. Do not link to the page that you don’t want to search engine to find. This is not recommended because if someone link to the page that you didn’t link then also search engine find such page from other website that linked to that particular page.

Robots.txt: - Here you can mention pages or files or sub directories that you don’t want search engine to crawl. Google provide robotos.txt feature in webmaster tool to check weather Google allowed to get to the url or not.

Nofollow attribute: - Nofollow link is another week approach to avoid search engine crawling. Suppose if you have some pages that you don’t want them to crawl by search engine like your sing up page or contact us page then simple put nofollow link to that page. But if someone link to those pages of your website then search engine find that link and crawl the page and showed up in their index. So, this also not a secure way to remove pages from search engine.

NoIndex: - Noindex Meta tag is simple and easy to remove pages from search engine. Pt a simple noindex Meta tag in the page that you don’t want search engine to find. Even other websites linked to that particular page you don’t want search engine to crawl and if you have noindex Meta tag on that page then search engine will not know about that page and don’t include in cache.

Google URL Removal Tool: - Google introduced this tool before 5 years but before it was in Google service but now you can find this handy tool in Google Webmaster Tool. Now you can remove the page that was cached by Google but now it is useless or unnecessary page for your website. So make that page 404 and then submit it Google URL removal toll.

Regards,
Bhavesh Goswami.