DSCng installation

Prerequisites - what you need before you start


DSCng in its current form is just a new front-end to data gathered by older versions of DSC. Therefore you need to have your DSC set up in order to use DSCng.


DSCng uses the following software components:

  • PostgreSQL (>=8.4)
  • Apache
  • Python (>=2.6)
  • Django (>=1.5)
  • Python GeoIP bindings
  • memcached (optional)

Any computer running this software is (at least in theory) able to run DSCng. The system is developed and tested on Linux, namely on Ubuntu 12.04 LTS and Ubuntu 12.10, and if you have the choice, we would recommend to use it as well, to minimize the chance of problem. (Of course as the development moves forward, we will certainly test on more platforms, and we certainly welcome any reports of both successful and unsuccessful installations.)

Software installation

In Ubuntu or other Debian based Linux systems, you will probably find the necessary software under the following names:

  • postgresql
  • python-psycopg2
  • libapache2-mod-wsgi
  • python-geoip
  • memcached
  • python-memcached

You just need to run:

sudo apt-get install postgresql libapache2-mod-wsgi python-psycopg2 python-geoip memcached python-memcache

All other necessary packages should be installed as part of installation of these. NOTE: Since latest version of DSCng requires Django 1.5 and it's not probable this version is in the package repository, it's necessary to install it manually. Easiest method is using pip:

sudo apt-get install python-pip
sudo pip install Django==1.5.1

On other system, you have to find packages equal to these, which should not pose a big problem.

NOTE: It's recommended to use separate Python virtual environment for DSCng, instructions how to set one up can be found in the Setting up Python virtual environment part of this readme.

Software configuration

In order to start using DSCng, you need to setup Apache and PostgreSQL.


It is really not possible to describe all the possibilities of setting up Apache here. DSC does not require any special generic setup beyond what is standard. Any configuration specific to DSC will be described in 'Installing DSCng' below. For now it is enough that your Apache installation is capable of displaying the default 'It works' page.


Make sure your PostgreSQL database server is up and running. It does not necessarily need to run on the same machine. If you set up PostgreSQL server so that is listens on a non-default port, you will have to specify so later on, when setting up DSCng.


Memcached server is optional but recommended. Memcached is used to reduce DB load by caching some of the frequent queries and it'd be possibly used more in the future. Currently DSCng assumes the server is running on localhost on default port 11211, otherwise the caching would not be used, slightly reducing the overall performance. For configuration of the memcached server see the official documentation, yet default installation from the Ubuntu repository is sufficient.

Installing DSCng

Creating and setting up the database

To create a new databases and roles in PostgreSQL, you usually use the postgres user who by default has the privileges to do so. Choose a name for your database and for the user that will own it. By default, both the database and the user are named "dsc", but you can set it up any way you want - just do not forget to change the config file of DSCng later on. Here is a set of commands that would set up the default database and user on an Ubuntu system:

sudo su postgres
createuser -P dsc
createdb -O dsc dsc

The createuser command will ask for a password for the user. Choose something safe and do not forget to add it into your config file later. You can safely answer no to all questions PostgreSQL will ask you about the new users privileges.

Unpacking and configuring DSCng

Obtaining DSCng

Because you are reading this, it is likely that you already have unpacked the sources or obtained them from GIT. If not, you can unpack the distribution tarball by running:

tar xvzf /path/to/dscng-x.y.z.tar.gz

in the directory where you want it unpacked. It would typically be /var/www/, but it could be anything else as well. It is up to you.

If you want to use the git sources, just do:

git clone git://git.nic.cz/dscng

It would create a directory named dscng in the directory where you currently are.


The configuration of DSCng itself is split into two parts. One is global and you should probably not touch it. The other is local and it is not part of either the GIT repository or the distribution packages. Once you create this file from an example, this configuration will be preserved by future updates, so you do not need to reconfigure DSCng again.

In the DSCng directory, you will find a directory named dsc. In this directory resides a file settings_local.example.py. If you are setting DSCng up for the first time, you have to rename this file to settings_local.py. Otherwise just copy over your old settings_local.py.

The {SCRIPT_NAME} value is a value which should start with a slash (/). Under this prefix, all of DSCng will be visible on your server, so you can either put it directly to the top of the structure (by using just /) or bury it deeper into the structure of your web, by using something like "/dns/monitoring/tools/dsc/ng/" or whatever suits your setup. Please remember this value as you will need it later when configuring Apache.

Modify the settings_local.py file to reflect your database setup by changing the entries beginning with DATABASE. Also set up a secret key further in the file. It has to be a long random value. You can create one for example by running the following command on the command line:

head -c 36 /dev/urandom | base64

You'll probably have to change the ALLOWED_HOST too, this variable specifies under which host names is your server available. Usually this means just adding another entry to the list of the names, for more information see the explanation in the settings file. By default DSCng is available only via localhost hostname and IP.

Feel free to modify any other parts of the file as you need.

Database initialization

To fill the database you created with the right tables, you need to run create_tables.py in the root of the DSCng directory tree structure:

python create_tables.py

NOTE: If you have much data to import, it is also possible to create the database structure without many database constraints, which makes the import of data significantly faster. To do so, run:

python create_tables.py --no-constraints

instead of the version without '--no-constraints' and after the import is finished, add the missing constraints by running:

python create_tables.py --constraints-only

Frontend intialization

To make the frontend part (UI) of the DSCng work, you need to collect all the static files (i.e. all the files that don't change, like JS, CSS or images) and initialize the Django database used for saving the persistent data of the application. This step is simple and you just have to run the following two commands in the dsc directory of your DSCng installation:

./manage.py syncdb
./manage.py collectstatic

The first command asks you if you want to create a superuser. This user has access to the administration site (available at URL http://yourdsc/admin/) and if you have USE_AUTHENTICATION set to True, it's the only user who has access to DSCng after the installation.

For more information about authentication and administration site, see Password protection and the admin site.


At this point, you can test if your setup works. You can do so by starting a development version of the Django server (warning - it is not intended for running a production system and you have to set the DEBUG option in your settings_local.py file to True, the debug server doesn't serve the static files in production mode).

Go to the dsc directory and run the following:

python manage.py runserver

If everything works as expected, you should see something like:

Validating models...

0 errors found
April 23, 2013 - 21:42:42
Django version 1.5.1, using settings 'dsc.settings'
Development server is running at
Quit the server with CONTROL-C.

If you are installing on a remote server, you can change the command line to:

python manage.py runserver

to make the test server listen on all interfaces and not only the local address. You should now be able to go to the address http://your_server:8000/ You should be presented with an interface without any servers or data types available and on the server's command line should appear several log messages. If this is so, you have done everything right so far. Otherwise you need to retrace your steps to find where it went wrong and/or write us for help.

Configuring Apache

The last configuration step is to serve the content via the Apache webserver, rather than by using the insecure and slow development server. However, because the import of old DSC data can be a very lengthy process, you can now safely jump to the section Initial data import from old DSC and return back once the import script is merrily munching on your data. At least you will have something to do while you wait for it to finish :)

To set up Apache to work properly with DSC, you need to adjust the configuration of the virtual server that will be hosting the system. The following should appear somewhere in the configuration, typically in one of the config files in /etc/apache2/sites-available on Debian systems:

# DSCng configuration
<VirtualHost *:80>
    # ServerName www.example.com
    # ServerAlias example.com
    ServerAdmin webmaster@localhost

    Alias {SCRIPT_NAME}/robots.txt {PATH_TO_DSC_NG}/dsc/static/robots.txt
    Alias {SCRIPT_NAME}/favicon.ico {PATH_TO_DSC_NG}/dsc/static/favicon.ico

    Alias {SCRIPT_NAME}/media/ {PATH_TO_DSC_NG}/dsc/media/
    Alias {SCRIPT_NAME}/static/ {PATH_TO_DSC_NG}/dsc/static/

    WSGIDaemonProcess dsc processes=2 threads=15 display-name=%{GROUP} python-path={PATH_TO_DSC_NG}/dsc/
    WSGIProcessGroup dsc

    WSGIScriptAlias {SCRIPT_NAME} {PATH_TO_DSC_NG}/dsc/dsc/wsgi.py

    <Directory {PATH_TO_DSC_NG}/dsc/static>
        Order deny,allow
        Allow from all

    <Directory {PATH_TO_DSC_NG}/dsc/media>
        Order deny,allow
        Allow from all

    <Directory {PATH_TO_DSC_NG}/dsc/dsc>
        <Files wsgi.py>
            Order allow,deny
            Allow from all

{SCRIPT_NAME} is the root URL for DSCng you configured above. Do not append a trailing slash to WSGIScriptAlias, if you're not running it in the root of the server (i.e. using only / as {SCRIPT_NAME}), otherwise it won't work correctly. {PATH_TO_DSC_NG} is where you installed DSCng. Below is a sample configuration for a typical installation to make it clear:

# DSCng configuration
<VirtualHost *:80>
    # ServerName www.example.com
    # ServerAlias example.com
    ServerAdmin webmaster@localhost

    Alias /dscng/robots.txt /home/user/dscng/dsc/static/robots.txt
    Alias /dscng/favicon.ico /home/user/dscng/dsc/static/favicon.ico

    Alias /dscng/media/ /home/user/dscng/dsc/media/
    Alias /dscng/static/ /home/user/dscng/dsc/static/

    WSGIDaemonProcess dsc processes=2 threads=15 display-name=%{GROUP} python-path=/home/user/dscng/dsc/
    WSGIProcessGroup dsc

    WSGIScriptAlias /dscng /home/user/dscng/dsc/dsc/wsgi.py

    <Directory /home/user/dscng/dsc/static>
        Order deny,allow
        Allow from all

    <Directory /home/user/dscng/dsc/media>
        Order deny,allow
        Allow from all

    <Directory /home/user/dscng/dsc/dsc>
        <Files wsgi.py>
            Order allow,deny
            Allow from all

By default, sites-available contains files default and default-ssl, that's used to configure localhost server on port 80 for HTTP and 443 for HTTPS respectively. It's fine to put the configuration in default, however in production you should make a separate and properly configured vhost. For more information about Apache configuration files, see official Apache documentation for Virtual Hosts and documentation for mod_wsgi configuration directives

If you put the configuration in a new file, you should enable the site using command (site_name stands for the name of the file you have created):

sudo a2ensite site_name

Once you have created and enabled the above configuration, you can reload or restart Apache (in Ubuntu: sudo service apache2 reload) and test if http://dscng.server.addr/{SCRIPT_NAME}/, where {SCRIPT_NAME} is the root URL you set up above, returns the same as the test server running on port 8000. If you already imported your data or the import script is running, you should already see some plot on the rendered page as well.

Setting up Python virtual environment

It's strongly recommended to run DSCng in a separate Python virtual environment which allows running multiple WSGI applications on a same host even with conflicting requirements. It takes just a few minutes to set up an environment that doesn't depend on the global Python installation.

First you need to download few packages with development headers and virtualenv package itself:

sudo apt-get install -y libgeoip-dev python-dev postgresql-server-dev-9.1 python-virtualenv

Then create a directory for the environments (if you already don't have some) and cd into it:

sudo mkdir /usr/local/pythonenv
cd /usr/local/pythonenv

Once you are in the directory, create a baseline virtual environment (an environment that'll be used for all the WSGI apps):

sudo virtualenv --no-site-packages BASELINE

Then you have to apply this baseline environment in Apache configuration:

sudo sh -c "echo 'WSGIPythonHome /usr/local/pythonenv/BASELINE' > /etc/apache2/conf.d/python-virtualenv-baseline"

Finally you need to create a separate DSCng environment and install required Python packages:

sudo virtualenv --no-site-packages DSCNG
sudo /usr/local/pythonenv/DSCNG/bin/pip install python-memcached Django==1.5.1 psycopg2==2.4.5
sudo /usr/local/pythonenv/DSCNG/bin/easy_install http://geolite.maxmind.com/download/geoip/api/python/GeoIP-Python-1.2.8.tar.gz

Last thing you need to do is to modify the WSGIDaemonProcess directive in site's configuration file (discussed earlier), that means add the path to the DSCNG virtual environment's site-packages. To do so, just append :/usr/local/pythonenv/DSCNG/lib/python2.7/site-packages to the content of python-path attribute. In the example above the resulting line would read:

WSGIDaemonProcess dsc processes=2 threads=15 display-name=%{GROUP} python-path=/home/user/dscng/dsc/:/usr/local/pythonenv/DSCNG/lib/python2.7/site-packages

And don't forget to reload the Apache configuration using:

sudo service apache2 reload

Now if you want to run scripts in the context of this virtual environment, run . /usr/local/pythonenv/DSCNG/bin/activate in your shell to enter the DSCNG environment. To deactivate it, simply execute deactivate.

Initial data import from old DSC

Import of data

To import your data from the original DSC (lets call it DSC 1), you need to locate the data. While DSCng can import old XML transport files that DSC 1 uses, it is much faster to import the already parsed text files that DSC 1 uses for displaying the actual content. This data typically resides in /usr/local/dsc/data/ and is made of a directory structure of server names and dates. Each directory contains data for one day in form of many files ending with .dat. Once you find where this directory structure resides on your system, you can run the import_dsc_dat_files.py script in root of DSCng installation like this:

python import_dsc_dat_files.py /path/to/dsc1/dat/dir/

The path can be path to one day worth of data, data for one whole server or the top dir containing all the servers. The import script will search the directory structure and start importing data into the database. It will tell you how much data there is and periodically inform you about the progress and how much time probably remains to the end.

The script dumps some logging information into a log file. The name of the file is part of the output of the script. You can cause more output to be generated by adding the -v argument to the import script.

WARNING: Once the script is running, do not modify the database content by hand in any way and do not try to run a second instance of the import script in parallel. The script uses some optimizations that are not compatible with parallel changes to the database.

On the other hand, you can safely use the web interface during the import because it uses only a read only access to the database. However, the imported data are not guaranteed to show immediately in the web interface, so do not be alarmed if you see no data while the import script is running.

The import script is capable of detecting double import of data, so it is safe to kill the import (Ctrl-C is the preferred method) if needed and start it again later - already imported data will be skipped.

How long does it take?

The import of data into DSCng is a pretty lengthy process. Even though we did much to speed it up, it is likely to take a few hours. On our modest dual core test server, it takes about 30 minutes to process 1 GB of original .dat files, which in our case corresponds to about 4.5 server-years processed per hour. This means that if you have 10 servers and 3 years worth of data, it would take about 7-8 hours to finish. Of course it depends on the speed of your system and amount of accumulated data. The import script does relatively good job in keeping you informed about the progress of the import and it is also usually pretty good in estimating the duration of the process.

NOTE: If you use a database running on the same server as the import script, it is really beneficial if the machine that you run the import script on has at least two processors/cores. Also a speedy harddrive (or an SSD) helps much, as the import is pretty IO intensive on the database part.

NOTE: If you have much data to import, make sure to see the Database initialization section for information about speeding the database for initial import.

How big would it be?

In our testing, the database occupied about 2.5x the space of the original text files from DSC 1.

Password protection and the admin site

It's possible to protect the access to the DSCng pages using the authentication system. This option could be toggled in the settings_local.py file by changing the USE_AUTHENTICATION directive. During the initialization of the database, you are asked to create a superuser, this is the only user who's able to access the DSCng after installation if the authentication is enabled. If you didn't create one, you can create the superuser later (or simply create another one) by running ./manage.py createsuperuser.

Superuser has access to the admin site, available at http://yourdsc/admin/ Admin site is currently used for two operations - the first one is adding the users who have access to the DSCng - to add or edit users, use the link Users (or the links Add and Change). Help for the available options is described in the edit form. Basically, if you want to give the user rights only to access the DSCng graphs, use the defaults. It is planned to add a more complex authentication/authorization system with custom permissions in the future. Another feature of the admin site is the possibility of tweaking the daily data graph parameters. To do this, use the Daily data display configs link. Edit these only if you understand what are you doing, otherwise it might break the display and render nonsense graphs.

Final touches

Once the import script is done, or even during its work (but not before it started), there is one more script to run. It imports text values for some numeric constants, such as the value 15 representing an MX record type. To do so, run the script fill_default_value_remaps.py from the same directory as the import script. It takes one argument and that should be a directory containing the remap definitions:

python fill_default_value_remaps.py data/

This script will print some information about what it imported and finish in a second or two.

What next?

At this point, you should have DSCng set up and filled with all your data from the original DSC. If you do not only want to test the new interface, but also use it for everyday purposes, you need to set up updates from the DSC .xml files which are arriving into your original DSC. This topic is discussed in the updates documentation.