![]() |
Michael's Blog |
Home | Gallery | Stats | Downloads | Links | Scripts | Fuel Stats | Wiki | RSS |
I’ve always been a big fan of beauty. Sure, you can’t judge a book by its cover but who wants to have sex with a book?-- Stephen Colbert Add quote.
After reading about various cluster file systems I decided to set up a small cluster running Lustre using Storm VPS instances. All nodes have the same hardware configuration and use a 50 GB SAN volume connected through iSCSI as the lustre block device. Specs are as follows.
Node configuration: OS: CentOS 6.3 x86_64 Kernel: 2.6.32-279.19.1.el6_lustre.x86_64 RAM: 3556 MB (Storm 4 GB) Primary Disk: 300 GB virtual disk Secondary Disk (iscsi): 50 GB SAN volume CPU: Two Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz cores Lustre configuration: 1 management server, 1 metadata server, 1 object storage server. LNET was configured to use a private network interface.
Disk performance was tested with the sgpdd_survey script from the Lustre IOkit. Write speed appears to average around 35-40 MB/s.
Wed Apr 10 10:29:39 EDT 2013 sgpdd-survey on /dev/sda from oss1.watters.ws total_size 8388608K rsz 1024 crg 1 thr 1 write 49.32 MB/s 1 x 49.32 = 49.32 MB/s read 68.15 MB/s 1 x 68.15 = 68.15 MB/s total_size 8388608K rsz 1024 crg 1 thr 2 write 77.15 MB/s 1 x 77.15 = 77.15 MB/s read 92.85 MB/s 1 x 92.85 = 92.85 MB/s total_size 8388608K rsz 1024 crg 1 thr 8 write 36.15 MB/s 1 x 36.14 = 36.14 MB/s read 94.08 MB/s 1 x 94.09 = 94.09 MB/s total_size 8388608K rsz 1024 crg 1 thr 16 write 35.84 MB/s 1 x 35.85 = 35.85 MB/s read 101.59 MB/s 1 x 101.59 = 101.59 MB/s total_size 8388608K rsz 1024 crg 2 thr 2 write 35.34 MB/s 2 x 17.67 = 35.34 MB/s read 67.38 MB/s 2 x 33.69 = 67.39 MB/s total_size 8388608K rsz 1024 crg 2 thr 4 write 39.09 MB/s 2 x 19.55 = 39.10 MB/s read 79.20 MB/s 2 x 39.60 = 79.19 MB/s total_size 8388608K rsz 1024 crg 2 thr 8 write 40.40 MB/s 2 x 20.20 = 40.40 MB/s read 98.16 MB/s 2 x 49.09 = 98.17 MB/s total_size 8388608K rsz 1024 crg 2 thr 16 write 37.73 MB/s 2 x 18.86 = 37.73 MB/s read 99.31 MB/s 2 x 49.66 = 99.32 MB/s total_size 8388608K rsz 1024 crg 2 thr 32 write 38.08 MB/s 2 x 19.04 = 38.07 MB/s read 97.30 MB/s 2 x 48.66 = 97.31 MB/s total_size 8388608K rsz 1024 crg 4 thr 4 write 38.38 MB/s 4 x 9.59 = 38.38 MB/s read 98.17 MB/s 4 x 24.55 = 98.19 MB/s total_size 8388608K rsz 1024 crg 4 thr 8 write 38.25 MB/s 4 x 9.57 = 38.26 MB/s read 100.06 MB/s 4 x 25.01 = 100.06 MB/s total_size 8388608K rsz 1024 crg 4 thr 16 write 39.42 MB/s 4 x 9.85 = 39.41 MB/s read 99.96 MB/s 4 x 25.00 = 99.98 MB/s total_size 8388608K rsz 1024 crg 4 thr 32 write 39.43 MB/s 4 x 9.86 = 39.44 MB/s read 99.93 MB/s 4 x 24.99 = 99.95 MB/s total_size 8388608K rsz 1024 crg 4 thr 64 write 38.22 MB/s 4 x 9.56 = 38.22 MB/s read 97.80 MB/s 4 x 24.45 = 97.81 MB/s total_size 8388608K rsz 1024 crg 8 thr 8 write 38.73 MB/s 8 x 4.84 = 38.76 MB/s read 87.71 MB/s 8 x 10.97 = 87.74 MB/s total_size 8388608K rsz 1024 crg 8 thr 16 write 39.70 MB/s 8 x 4.96 = 39.67 MB/s read 81.09 MB/s 8 x 10.14 = 81.10 MB/s total_size 8388608K rsz 1024 crg 8 thr 32 write 43.40 MB/s 8 x 5.43 = 43.41 MB/s read 81.21 MB/s 8 x 10.16 = 81.25 MB/s total_size 8388608K rsz 1024 crg 8 thr 64 write 38.88 MB/s 8 x 4.86 = 38.91 MB/s read 67.10 MB/s 8 x 8.39 = 67.14 MB/s total_size 8388608K rsz 1024 crg 8 thr 128 write 42.19 MB/s 8 x 5.27 = 42.19 MB/s read 65.92 MB/s 8 x 8.24 = 65.92 MB/s
IOPS performance was tested using iozone, here are the results.
OPS Mode. Output is in operations per second. Include fsync in write timing No retest option selected Record Size 4 KB File size set to 4194304 KB Command line used: iozone -l 32 -O -i 0 -i 1 -i 2 -e -+n -r 4K -s 4G Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Min process = 32 Max process = 32 Throughput test with 32 processes Each process writes a 4194304 Kbyte file in 4 Kbyte records Children see throughput for 32 initial writers = 27764.87 ops/sec Parent sees throughput for 32 initial writers = 26692.16 ops/sec Min throughput per process = 840.07 ops/sec Max throughput per process = 903.35 ops/sec Avg throughput per process = 867.65 ops/sec Min xfer = 975918.00 ops Children see throughput for 32 readers = 26758.37 ops/sec Parent sees throughput for 32 readers = 26755.12 ops/sec Min throughput per process = 448.79 ops/sec Max throughput per process = 1372.74 ops/sec Avg throughput per process = 836.20 ops/sec Min xfer = 342845.00 ops
As you can see lustre is a relatively high performance file system and is easily scalable to store petabytes of data. Adding more space is as simple as building a new object server and running mkfs.lustre.