The web crawleris a program that translates web pages and any existing links relevant to the page. The web crawler begins by looking through the web addresses that are available in its database or index. Any other page on the internet is added to the database should the web crawler consider it relevant to its existing index. Thus, the database continually grows and the web crawler also goes back to the index to check for updates and again search for new available links.So it is important to let the crawleris cralw your site,then how? Any tips?