Sitebulb Server - start here

If you've found your way here, this means you are probably interested in the Sitebulb Server beta.

The purpose of this post is to introduce you to Sitebulb Server and clarify what it is and what it is not. Hopefully by the end of it you will have a clear idea if you think it is for you or not (to be clear: it is not for everybody).

What is Sitebulb Server?

First things first, Sitebulb Server is not a cloud crawler.

In fact, it is a solution which means that cloud crawling is not necessary.

Crawling websites uses a lot of computer resources, and requires a lot of storage, so this is mostly what you are paying for when you use a cloud crawler.

Cloud crawlers have stringent limits on the number of projects/domains you can crawl and the number of URLs you can crawl - this is because of the elastic way in which cloud hosting prices grow. If there were no limits, the prices would explode out of control.

Sitebulb Server is different to this - it is designed to be run on a dedicated server where you are working with fixed (or capped) costs, that means your usage does not need to be limited by arbitrary thresholds.

It is a much cheaper way to achieve the same thing.

And it does this by ticking off all the things you want from a cloud crawler in the first place:

  • One central place that your whole team can access audit data
  • Crawl multiple sites concurrently
  • Access your crawl data from anywhere
  • Crawl websites with millions of URLs

Who is Sitebulb Server for?

Sitebulb Server version is designed with certain types of users in mind:

  • Agencies that want shared access to audit data between team members
  • Brands/in-house teams that need to crawl gigantic websites
  • Consultants that want access to their audit data from anywhere in the world
  • Publishers that need to stay abreast of lots of sites all the time

Of course there will be some amount of crossover here, but in general most users will fall into one of these categories, and there will be some features of Sitebulb Server that are more important to you than others.

And if you don't need any of the things above, you probably don't really need Sitebulb Server.

To use Sitebulb Server you also need Sitebulb Pro

Sitebulb Server is designed so that after the initial install, you don't need to keep logging back onto the server again via Remote Desktop Connection.

Instead, you can connect to the server on your desktop or laptop machine running Sitebulb Pro. This means it is perfect if you are already using Sitebulb Pro - but if you have not already got a Sitebulb Pro subscription, you will need to get one if you want to try Sitebulb Server.

You will need a Pro license for every user that wants/needs to connect to the server.

Getting started with Sitebulb Server

To get set up, you need to follow these three steps, in order:

  • Step 1: Access a server/machine that you can use
  • Step 2: Configure the server and install Sitebulb on there
  • Step 3: Connect your local desktop/laptop machines to the server version

What sort of machine do you need?

As with everything in SEO, 'it depends.' The options range from 'essentially free' to hundreds of dollars a month in terms of cost. Which option is suitable for you will depend on what resources you currently have available, your budget for technical SEO, and how 'mission critical' website crawl data is to your business.

These are the options you can go for:

Use a spare desktop machine

If you happen to have an spare Windows desktop computer lying around in the office, this could be a good starting point. This machine can be set up to act as the server, and as long as it stays on (and connected to the internet), you'll be able to connect to it from your other desktop/laptop machines.

This spare computer would need an absolute minimum spec of 8 cores with 16GB RAM and SSD hard drives - if you happen to have a more powerful machine available, bigger is definitely better. More cores will allow you to crawl faster or support concurrent crawling, and more RAM will allow you to process bigger websites.

Please check out our server recommendations guide for more details about suitable machine specs.

Fire up a cloud instance

AWS/Google Compute is a really quick option that will allow you to test Sitebulb Server without a monthly commitment to a rented server.

They literally take 10 minutes to set up (see our AWS EC2 or Google Cloud Compute video guides) so they are a great short-term or test solution.

However if you want to use Sitebulb Server on an ongoing basis, a cloud solution can cost as much as 5X more.

This is why our top recommendation is to:

Rent a dedicated server

If you don't have a spare machine, or want a more dedicated resource to handle your crawling, renting a dedicated server is the best bet here. These days it is pretty straightforward to get a decent spec 'out the box' server set up in a matter of hours, for a few hundred dollars/pounds a month.

The benefits of renting a server like this is that you can avoid an initial outlay on buying your own hardware and the ongoing management/maintenance. It also means you will end up with a machine connected directly to a fast internet connection, without sucking up all your office bandwidth.

Also, a server can be rented on a monthly contract, so you can switch it off if you no longer need it (e.g. large client cancels) or you can easily upgrade to a bigger server as you scale.

In order to move forward with this solution, we suggest you read our server recommendations guide.

Buy your own hardware

This option won't be for everyone, but if you have invested in your in-house hardware stack, it is probably the most effective long-term solution.

This is suitable for publishers or larger in-house brands where you can define exactly the right server you need for your specific crawling needs, with more clarity about the long-term requirement (versus an agency situation where the requirement is dependent on changeable client needs).

Please contact us on [email protected] if you are planning to purchase your own servers, so we can help make sure you get the right spec.

Let Sitebulb run a managed server

Sitebulb will put together a customised server build specifically for your needs, that is fully managed and taken care of for you.

This is the most expensive option, as it does not require you to set up or run servers at all, and instead let us take care of it.

This is suitable for large enterprise brands or agencies who need to regularly crawl large websites, but don't want to pay the eye-watering fees that DeepCrawl, OnCrawl or Botify charge them.

If this sounds like you, please get it touch with us via [email protected] so we can discuss your requirements. We'll have a meeting to properly understand your needs, then put together a bespoke quotation for you to consider.

Next steps

Having read all of the above, you now need to make a decision about the right option for your business.

To help give you further guidance in this, please read our server recommendations guide.