Should You Build Your Website Offline?
One way to ensure your website isn’t indexed before you’re ready is to build it offline using a local virtual server and upload it once it’s complete. This is not necessary if you have proper web hosting because most web hosts allow you to access and edit your robots.txt file to disallow search engines from indexing your content. That being said, there are a few good reasons to make a website on your local machine before uploading it to the web.
First, all major search engines respect the robots.txt web standard but unscrupulous engines and bots don’t. Your site could potentially be scrapped before you allow it in your robots.txt file though it’s probably unlikely. A scenario where this would matter is if you’re using a paywall to protect premium content. If you start uploading your premium newsletters or other member-online content before installing your paywall it could potentially be indexed and end up ‘free’ in a search engine or crawler’s cache for months or years to come.
Another reason to build your site locally using a virtual server is to test it for design and function without having to risk anyone seeing it in an ‘unprepared’ state. This is especially worth consideration if you’ve already released your domain name through social media or if your domain is likely to be stumbled upon during the construction of your site. For most website owners, building your website online and simply using robots.txt to prevent premature indexing is the best solution.
Using Robots.txt to Control Indexing
The simplest way to make sure your content doesn’t get crawled by search engines before it’s ready for the limelight of the web is to edit your robots.txt file to disallow search engine crawling during the construction phase of your website.
If you’re not running WordPress or simply want to edit the file manually you can locate it using your web server’s file manager or FTP portal. Look in your Public_HTML folder or other root folder for an existing robots.txt file. If you can’t locate the file on your server you may need to manually create it.
Download the file and open it with Notepad or any other text editor. Creating a new robots.txt is just as easy, simply open your favorite text editor and create a file named “robots.txt” Use the following syntax to disallow or allow certain portions of your website to be crawled and indexed by Google and other search engines:
- User-agent: [Name of crawler or bot to disallow]
- Disallow: [Folder path or URL string for which to prevent indexing]
Some of the common user-agents for search engines include Bingbot and Googlebot but you can see a complete list of popular bots on the web or you can block specific bots or IP Addresses.
How to Check Over Your SEO
One of the most important tasks you’ll perform when you decide to make a website is Search Engine Optimization. In order for search engines to find your website and decide to rank it above other competing sites you’ll need to show them your site has what they are looking for. This includes high quality content, fast loading media, proper spelling, and optimized images.
You should start by finding a host that will help your website load quickly when visitors and bots browse your content. From there, add HTML tags and captions to describe your images. HTML tags help bots and individuals better understand the meaning and placement of your images and graphics. Don’t forget to make sure your HTML meta tags are accurate and concise.
There are plugins like Yoast SEO to help with this if you’re new to SEO or prefer using a graphical menu to manually editing HTML files.
Adding a Sitemap to your Website
The final thing you should do to prepare your website for indexing is add a sitemap. A sitemap is basically an XML file that helps search engines and other bots understand the hierarchical structure of your website. If you’re using WordPress or another Content Management System then a sitemap has likely already been automatically generated for you but you can improve its accuracy by installing a sitemap handler like the Google XML Sitemapsplugin.
Check over your website before editing your robots.txt file to allow for search engines to crawl your content or before uploading it to the web if you’re building it offline. Look for SEO factors, optimized media, a proper sitemap, and good load times to make sure your website ranks well in the search engine results pages and you’ll start seeing organic search traffic in no time.