TroyGrosfield.com TroyGrosfield.com

Headline

Django Production Environment Setup with MongoDB

Author
by Troy Grosfield
Date
November 22nd, 2011
Category
Developer
Story

Many developers are often curious on what a possible web environment setup looks like in production. That answer can vary greatly depending on what exactly you want your application to do. A few question to ask yourself:

  • Are you expecting heavy traffic on the site?
  • How will you be notified if something in your system fails?
  • Do you have tasks that could potentially take a long time to run?
  • How are you backing up your data?
  • If a piece of hardware fails, will your system handle the failure gracefully?
  • If a piece of software fails, will your system handle the failure gracefully?

These are just some of the questions to address when thinking about your web environment setup.

Are you Expecting Heavy Traffic?

There are many ways to handle heavy loads of requests to your site. You can always throw bigger hardware up which will likely handle increased traffic. However, what if that hardware fails?

Web Environment Setup

Web Environment Setup

One way to handle this would be to setup multiple web servers and put a load balancer in front of them. Say we have 3 web servers that can handle our requests to the site. A load balancer will direct traffic to the healthy servers. Say one of the web servers fails. The load balancer will see one of the web servers is down and will only direct traffic to the two healthy web server instances.

So, the load balancer not only divvy’s up the requests to each of our web servers, but also saved us against a web server hardware failure.

How will you be notified if something in your system fails?

Ok, the load balancer handled the failed web server gracefully, great! But, now you only have two healthy web servers up and running.

  • How will you be notified that a server went down?

There are 3rd party sites that will ping your servers to make sure they are up and running smoothly. One of those companies is pingdom. They will send pings to your servers at intervals you set that will verify a the server is up and running. If they get unsuccessful responses from the server pings they will shoot off emails to notify system administrators.

Ok, pingdom just emailed me. Now what do I do? Go fix/replace the unhealthy server!

Logging

Logging is extremely helpful for troubleshooting problems.  Setting up logging in a Django project is relatively easy.  Just follow Django logging documentation.

Database Setup

Now that you have your web servers up and running, what about your database servers?

  • Are you running a single server on a single piece of hardware?
  • What happens if the hard drive get corrupted somehow?
  • What happens if the database server goes down?
  • Do you have a backup server in place?
  • Do you have your data backed up?

One way to handle this is to setup multiple database servers and make sure your data is replicated.  Since we’re running MongoDB as our database, let’s setup 3 hardware servers and create replica sets each of the boxes. We will have one primary master MongoDB server instance that will do reads and writes and 2 other database servers that will replicate the data.  Now, if one of the database servers goes down, one of the secondary MongoDB replicas will automatically step up to be the primary master MongoDB instance.  And since we already setup pingdom to monitor the health of our servers we now should be expecting an email about the database server going down.

Note: MongoDB replica sets are great ways to replicate your data, however, this should not replace your backup methods.  Do mongodumps to backup your data!

Now let’s say the server wasn’t the issue, but instead we had a hard drive failure?  This is a tricky one which can cause downtime.  The MongoDB replicas didn’t step because they still thing the primary master MongoDB instance is still running.  And we didn’t get an email from pingdom because the server is still up.  How do we handle this hard drive failure?  RAID 10!

RAID 10, Redundant Array of Independent Disks, mirrors the data and if one part of the drive goes bad everything will still run smoothly since the data is mirrored.  If one part of the array goes bad, the bad portion of the RAID will drop out of the array and we will still be up and running.  Then we setup scripts to monitor the RAID so if we do have any issues we are notified immediately.

The best thing about all the issues we came across? We didn’t have a single second of downtime and we were notified immediately as soon as we had a failure in our system.

Do you have tasks that could potentially take a long time to run?

Now that we have a system setup to handle unforeseen failures that may arise, lets talk about the web application itself. There are times when you will want to perform tasks that would take too long for your user to wait for or will timeout before the task has a chance to complete.

Once solution to this problem is to setup a queue of tasks to run in the background. Celery handles this case for us. It will kickoff background process that run behind the scenes so your users can continue browsing without having to wait long periods of time to perform certain actions, like sending mass emails.

Along with Celery, you can also setup Celerybeat which will perform periodic, cron like, tasks that will run in the background.

Whew, that’s it!  You now have a very responsive, proactive system setup ready to handle all the web can throw at it…for now.

Tags
Comments
1 Comment »

One Comment

Leave a reply

 
  1. Author
    Your Questions About Free Advice On Starting Your Own Business - Just another WordPress site - Mikey Likes It
    Date
    February 25th, 2012 at 6:37 pm
    Comment

    […] […]