On 12th Feb MSN officially announced a new improved MSN crawler that is more powerful in caching and indexing more pages of website. Many webmaster complaints about MSN crawler crawling less pages but with this new improved crawler you will see more pages indexed by MSN. May be this new improved feature will help webmaster to make website more visible to MSN Search Engine.
HTTP Compression: This feature compressed the file size of webpage and reduces time of downloading data from server to their crawler.
Condition Get: This feature will ask the server if the page has been changed before last time crawl the page. MSN crawler will include “If-Modified-Since” header and get the time of last download using GET request. If the page has not been changed after the last download the server will respond 304 HTTP Response.
To reflect these changes they upgraded their user agent to msnbot/1.1.
