Michael's Blog

Home | Gallery | Stats | Downloads | Links | Scripts | Fuel Stats | Wiki | RSS
There's no crying in webhosting.
-- JoeDoss     Add quote.

How to fix graphite user creation in FreeBSD 10

2016-03-09 14:22:00 by Michael 0 Comments
Tags: freebsd graphite sysadmin sql

If you see an error like below when you try to create a graphite user you will need to update the graphite database in order for user create to work.

django.db.utils.IntegrityError: NOT NULL constraint failed: auth_user.last_login

Unfortunately sqlite doesn't support the drop constraint syntax so you will need to create a new temporary table, copy over the existing table, and then rename.

CREATE TABLE "auth_user2" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "password" varchar(128) NOT NULL, "last_login" datetime NULL, "is_superuser" bool NOT NULL, "username" varchar(30) NOT NULL UNIQUE, "first_name" varchar(30) NOT NULL, "last_name" varchar(30) NOT NULL, "email" varchar(75) NOT NULL, "is_staff" bool NOT NULL, "is_active" bool NOT NULL, "date_joined" datetime NOT NULL);
insert into auth_user2 select * from auth_user ;
drop table auth_user ;
alter table auth_user2 rename to auth_user ;

Now user creation should work.

root@graphite:/usr/local/lib/python2.7/site-packages/graphite # python manage.py createsuperuser
Username (leave blank to use 'root'): wattersm
Email address: wattersm@watters.ws
Password (again): 
Superuser created successfully.

Prevent iptables from spamming your console

2015-08-14 13:41:00 by Michael 2 Comments
Tags: linux kernel iptables sysadmin netfilter

How to disable firewall "spam" on your console.

I worked on a ticket recently for a customer concerned about firewall messages being sent to every user's console by the kernel. After doing a bit of research I discovered that the nf_ct_ftp module logs messages to syslog as *emergency* level by default which results in every console being spammed by firewall messages. To prevent this you can make a few simple changes as follows.

First, set up a custom rsyslog conf file to send iptables messages to a different file.

cat << EOF > /etc/rsyslog.d/iptables.conf 
:msg, contains, "nf_ct_ftp:" -/var/log/iptables.log
& ~

The first line means send all messages that contain the “nf_ct_ftp:” string to /var/log/iptables.log. The second line causes rsyslog to discard messages that were matched on the previous line. Adjust this rule according to your needs.

Second, update sysctl.conf with the following lines and then run "sysctl -p".

kernel.printk = 4 4 1 7

sysctl -p

See https://www.kernel.org/doc/Documentation/sysctl/kernel.txt for a description of these values.

Now restart rsyslog and test your changes using the "logger" command.

service rsyslog restart
logger -p kern.emerg -t kernel "nf_ct_ftp: dropping packet test"

You should not see anything on the console. cat /var/log/iptables.log to confirm that the entry was logged properly. After you have confirmed that the messages are being logged properly you can set up logrotate to manage the logs. Create a config file to do this similar to below.

cat << EOF > /etc/logrotate.d/iptables 
	rotate 7
		invoke-rc.d rsyslog rotate > /dev/null

There is nothing else to do at this point.

How to Manually Change Domain in Magento

2014-10-10 10:20:00 by Michael 0 Comments
Tags: magento linux mysql sysadmin

Changing the domain name on a Magento install requires a few steps to update the site URL in mysql. The procedure should be similar to below.

Update your core_config_data table to edit the two records for web/unsecure/base_url and web/secure/base_url


mysql> update core_config_data set value = 'http://dev.example.com/' where path = 'web/unsecure/base_url';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
mysql> update core_config_data set value = 'http://dev.example.com/' where path = 'web/secure/base_url';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

After this is done delete the contents of WEBROOT/var/cache. The location of the WEBROOT varies depending on how your server is set up.

cd /home/username/public_html/var/
rm -rf ./cache/*

Update any .htaccess redirects you may have added.

That's it, you're done! Open the site in a new browser tab to make sure that everything loads properly.

Storm VPS Lustre Benchmarks

2013-04-11 13:56:00 by Michael 0 Comments
Tags: linux sysadmin lustre storage

After reading about various cluster file systems I decided to set up a small cluster running Lustre using Storm VPS instances. All nodes have the same hardware configuration and use a 50 GB SAN volume connected through iSCSI as the lustre block device. Specs are as follows.

Node configuration:

OS: CentOS 6.3 x86_64
Kernel: 2.6.32-279.19.1.el6_lustre.x86_64
RAM: 3556 MB (Storm 4 GB)
Primary Disk: 300 GB virtual disk
Secondary Disk (iscsi): 50 GB SAN volume
CPU: Two Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz cores

Lustre configuration: 1 management server, 1 metadata server, 1 object storage server.  LNET was configured to use a private network interface.

Disk performance was tested with the sgpdd_survey script from the Lustre IOkit. Write speed appears to average around 35-40 MB/s.

Wed Apr 10 10:29:39 EDT 2013 sgpdd-survey on /dev/sda from oss1.watters.ws
total_size  8388608K rsz 1024 crg     1 thr     1 write   49.32 MB/s     1 x  49.32 =   49.32 MB/s read   68.15 MB/s     1 x  68.15 =   68.15 MB/s
total_size  8388608K rsz 1024 crg     1 thr     2 write   77.15 MB/s     1 x  77.15 =   77.15 MB/s read   92.85 MB/s     1 x  92.85 =   92.85 MB/s
total_size  8388608K rsz 1024 crg     1 thr     8 write   36.15 MB/s     1 x  36.14 =   36.14 MB/s read   94.08 MB/s     1 x  94.09 =   94.09 MB/s
total_size  8388608K rsz 1024 crg     1 thr    16 write   35.84 MB/s     1 x  35.85 =   35.85 MB/s read  101.59 MB/s     1 x 101.59 =  101.59 MB/s
total_size  8388608K rsz 1024 crg     2 thr     2 write   35.34 MB/s     2 x  17.67 =   35.34 MB/s read   67.38 MB/s     2 x  33.69 =   67.39 MB/s
total_size  8388608K rsz 1024 crg     2 thr     4 write   39.09 MB/s     2 x  19.55 =   39.10 MB/s read   79.20 MB/s     2 x  39.60 =   79.19 MB/s
total_size  8388608K rsz 1024 crg     2 thr     8 write   40.40 MB/s     2 x  20.20 =   40.40 MB/s read   98.16 MB/s     2 x  49.09 =   98.17 MB/s
total_size  8388608K rsz 1024 crg     2 thr    16 write   37.73 MB/s     2 x  18.86 =   37.73 MB/s read   99.31 MB/s     2 x  49.66 =   99.32 MB/s
total_size  8388608K rsz 1024 crg     2 thr    32 write   38.08 MB/s     2 x  19.04 =   38.07 MB/s read   97.30 MB/s     2 x  48.66 =   97.31 MB/s
total_size  8388608K rsz 1024 crg     4 thr     4 write   38.38 MB/s     4 x   9.59 =   38.38 MB/s read   98.17 MB/s     4 x  24.55 =   98.19 MB/s
total_size  8388608K rsz 1024 crg     4 thr     8 write   38.25 MB/s     4 x   9.57 =   38.26 MB/s read  100.06 MB/s     4 x  25.01 =  100.06 MB/s
total_size  8388608K rsz 1024 crg     4 thr    16 write   39.42 MB/s     4 x   9.85 =   39.41 MB/s read   99.96 MB/s     4 x  25.00 =   99.98 MB/s
total_size  8388608K rsz 1024 crg     4 thr    32 write   39.43 MB/s     4 x   9.86 =   39.44 MB/s read   99.93 MB/s     4 x  24.99 =   99.95 MB/s
total_size  8388608K rsz 1024 crg     4 thr    64 write   38.22 MB/s     4 x   9.56 =   38.22 MB/s read   97.80 MB/s     4 x  24.45 =   97.81 MB/s
total_size  8388608K rsz 1024 crg     8 thr     8 write   38.73 MB/s     8 x   4.84 =   38.76 MB/s read   87.71 MB/s     8 x  10.97 =   87.74 MB/s
total_size  8388608K rsz 1024 crg     8 thr    16 write   39.70 MB/s     8 x   4.96 =   39.67 MB/s read   81.09 MB/s     8 x  10.14 =   81.10 MB/s
total_size  8388608K rsz 1024 crg     8 thr    32 write   43.40 MB/s     8 x   5.43 =   43.41 MB/s read   81.21 MB/s     8 x  10.16 =   81.25 MB/s
total_size  8388608K rsz 1024 crg     8 thr    64 write   38.88 MB/s     8 x   4.86 =   38.91 MB/s read   67.10 MB/s     8 x   8.39 =   67.14 MB/s
total_size  8388608K rsz 1024 crg     8 thr   128 write   42.19 MB/s     8 x   5.27 =   42.19 MB/s read   65.92 MB/s     8 x   8.24 =   65.92 MB/s

IOPS performance was tested using iozone, here are the results.

	OPS Mode. Output is in operations per second.
	Include fsync in write timing
	No retest option selected
        Record Size 4 KB
        File size set to 4194304 KB
        Command line used: iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Min process = 32
        Max process = 32
        Throughput test with 32 processes
        Each process writes a 4194304 Kbyte file in 4 Kbyte records

        Children see throughput for 32 initial writers  =   27764.87 ops/sec
        Parent sees throughput for 32 initial writers   =   26692.16 ops/sec
        Min throughput per process                      =     840.07 ops/sec
        Max throughput per process                      =     903.35 ops/sec
        Avg throughput per process                      =     867.65 ops/sec
        Min xfer                                        =  975918.00 ops

        Children see throughput for 32 readers          =   26758.37 ops/sec
        Parent sees throughput for 32 readers           =   26755.12 ops/sec
        Min throughput per process                      =     448.79 ops/sec
        Max throughput per process                      =    1372.74 ops/sec
        Avg throughput per process                      =     836.20 ops/sec
        Min xfer                                        =  342845.00 ops

As you can see lustre is a relatively high performance file system and is easily scalable to store petabytes of data. Adding more space is as simple as building a new object server and running mkfs.lustre.

create an rpm mirror using wget

2013-04-10 13:35:00 by Michael 0 Comments
Tags: wget linux sysadmin scripting

If you want to set up a yum repo you can easily mirror an existing site using wget. To do this you will need to run this command.

wget --mirror -np --no-host-directories -A rpm,srpm http://downloads.whamcloud.com/public/lustre/latest-maintenance-release/

In this case we are mirroring the lustre rpm repo.

After the files are downloaded you can run the createrepo command to create yum metadata.

quickly remove old ssh keys with sed

2012-04-24 10:13:00 by Michael 0 Comments
Tags: sed linux ssh sysadmin

If you work on a lot of servers and do a lot of reinstalls you will see the following error often.

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /var/root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:9948
RSA host key for host.example.com has changed and you have requested strict checking.
Host key verification failed.

To save some time you can quickly remove the old host key with a single sed command:

sed -i '9948d' .ssh/known_hosts

Xen with File Server Replication

2008-11-19 14:33:36.756208 by Michael 0 Comments
Tags: xen solaris unix sysadmin zfs iscsi AVS

I've been working on a project at work that has kept me pretty busy this week, it involves shared storage and computing clusters which has me pretty geeked out. I must say that I've learned A LOT about Solaris clustering, iSCSI, and disk replication, throw ZFS with Xen on top of that and things get pretty complicated.

Here's a diagram of the current system I have built.

With this setup the file server has ZFS pools that replicate each disk over to the secondary, the concept is the same as a local disk mirror. I've tested out a few different fail over situations which have worked so far, the one wrench in the works is that Linux doesn't like having iSCSI targets moved around while the device is open. This means that the xen server must shut down all running domains, take the volume offline, and then restart everything. Naturally this is not desirable in production, I will be testing out a Solaris server running xVM to see how that handles moving iSCSI targets later this week.

Open Solaris Desktop

2008-10-05 19:04:18.964429 by Michael 0 Comments
Tags: solaris unix sysadmin

I've recently switched to OpenSolaris on my desktop at work and I just wanted to write a bit about my experiences.


Installing the OS is about the same as any other unix system. Boot the CD, enter a host name, root password and select the drive you want to install to. One nice thing is that you can set up a ZFS mirror out of the box, if not you can easily mirror your pool later without having to mess around too much, one command takes care of it.

Hardware support:

All of the hardware on my computer was detected and loaded the proper drivers without me having to intervene. As long as your hardware is listed on the compatibility list you'll be fine. Setting up X with multiple monitor support is also very easy, just run the Nvidia settings app and configure your screens.


One issue I did have is that mp3 support isn't included as part of the default install, you have to download the codec package from Fluendo if you want mp3 support in totem or anything else that uses the gstreamer backend. Flash also requires a manual install, the plugin is pretty easy to set up however.


After using the system for a few weeks I'd have to say I'm impressed. If you have any experience at all with running a Linux desktop it shouldn't take long to adjust and you'll have access to zfs and dtrace which simply don't have equivalents in Linux. In short, give it a try, you might like it.