“`
Product Overview of GSiteCrawler
GSiteCrawler is a powerful, free, and highly customizable Google Sitemap generator designed for website owners and webmasters. Here’s a detailed look at what the product does and its key features.
Primary Function
GSiteCrawler automates the creation of Google Sitemap files, helping webmasters to ensure their websites are indexed optimally by search engines like Google, Yahoo!, and MSN/Live.com. The tool crawls a website to identify all pages, links, and relevant metadata, and then generates a sitemap file in various formats.
Key Features
Crawling Capabilities
- GSiteCrawler can capture URLs through a normal website crawl, emulating a Googlebot to find all links and pages within the website.
- It also supports importing URLs from existing Google Sitemap files, server log files, or any text file containing URLs.
- The crawler respects `robots.txt` files and robots meta tags for index and follow instructions.
Customization and Control
- Users can set various options such as the maximum depth of the crawl, exclude specific pages, and define the frequency of updates.
- The tool allows for filtering pages based on custom criteria, manual editing, adding, and deleting pages, and adjusting settings globally.
Performance and Efficiency
- GSiteCrawler can run up to 15 crawlers in parallel and can be throttled with a user-defined wait time between URLs to avoid overwhelming the server.
- It checks the date, size, title, description, and keyword tags of each page and tracks the time required to download and crawl the page.
Export Options
- The tool generates sitemaps in multiple formats, including XML, HTML, CSV, and RSS feeds. It also supports exporting to Excel, Google Base Bulk-Import files, ROR (Resources of Resources) XML files, and static HTML sitemap files.
Automation and Integration
- GSiteCrawler can automatically upload the generated sitemap file via FTP, and it supports both XML and compressed XML.GZ files. This feature is particularly useful for larger sites with more than 40,000 URLs.
- The tool can be run in a network environment, splitting crawlers over multiple computers and sharing the same database (supporting MS-Access, SQL-Server, or MSDE databases).
Additional Functionality
- GSiteCrawler provides a general site overview, including the number of URLs, oldest URLs, and other statistics.
- It identifies broken URLs, slow pages, and duplicate content, with the option to automatically disable duplicate content pages from the sitemap file.
- The tool also tests for and recognizes non-standard file-not-found pages without HTTP result code 404.
User Experience
- GSiteCrawler features an integrated wizard that guides users through the process of adding a new project, adjusting settings, crawling the website, generating the sitemap file, and uploading it to the server.
- The tool is compatible with various Windows versions, from Windows 95b to the latest server versions, making it versatile for different user environments.
In summary, GSiteCrawler is a robust and flexible tool that not only generates Google Sitemap files but also offers extensive customization, automation, and integration features, making it an invaluable asset for webmasters aiming to optimize their website’s indexing and performance.
“`