Hybrid RAID1 performance tests

* Home :: Get a random password :: Get a multi-word password :: What is my IP? (v4 | v6)
* My projects

Hybrid RAID1 performance tests

Tags: howto,benchmark

Date: 20250204

So I ended up with this HP MicroServer Gen8 that I'd like to turn into a NAS or NextCloud box.

The hw specs are:
Intel Xeon E3-1220L V2, 16 GB ECC RAM and the following drives - 2TB each:
2x SAMSUNG HD204UI (SATA II, 3Gbps)
2x Samsung SSD 870 QVO (SATA 3, 6Gbps)

OS: Debian 12 x64

My aim is to combine each SSD with each SATA (rotational) in a RAID1 and put LVM on top of everything for maximum flexibility and space usage.

Now the problem is that by default such setup will have bad performance, because the lowest common denominator here is the old and slow 5400 RPM SAMSUNG HD204UI, so reads and writes will be brought down to that speed..

BTW, for all the tests I am using the "fio" utility with the following parameters:

fio --loops=5 --size=1000m --filename=fiotest.tmp --stonewall --ioengine=libaio --direct=1 \
--name=Seqread --bs=1m --rw=read --name=Seqwrite --bs=1m --rw=write \
--name=512Kread --bs=512k --rw=randread  --name=512Kwrite --bs=512k --rw=randwrite \
--name=4kQD32read --bs=4k --iodepth=32 --rw=randread \
--name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite

Luckily this is Linux and I learned that the Linux RAID kernel modules can mitigate this to a degree, by means of using the option "--write-mostly" when adding a rotational drive to a RAID1 array where there is also an SSD. For clarification: Linux will not do this for you automatically, this all must be done by yourself.

root@microserver8:~# cat /sys/block/md2/md/dev-sdc2/state 
in_sync,write_mostly

By using this method we have now limited the slowness just to the write operations, as the SSD will be prefered for the reads.

This is pretty brilliant already, but I am not very happy about the slow writes, so what else can we do to speed things up? Well, several things it turns out:
1 - Just as we enable "write-mostly" for drives, we can also enable "write-behind", this will buffer a number of writes and it only applies to "write-mostly" setups, btw
2 - The SATA rotational drive has a cache of 32MB! Let's try to use it
3 - There's something called dm-writecache where you can use a fast LVM PV to cache writes for a generally slow Logical Volume - this was measured as well with an 8GB cache LV!

In all 3 cases I used the same "fio" command and here is what happened...

Results:

	Seqread IOPS	Seqread BW (MiB/s)	Seqwrite IOPS	Seqwrite BW (MiB/s)	512Kread IOPS	512Kread BW (MiB/s)	512Kwrite IOPS	512Kwrite BW (MiB/s)	4kQD32read (k) IOPS	4kQD32read BW (MiB/s)	4kQD32write IOPS	4kQD32write BW (MiB/s)
SATA only	125	126	51	52	76	38	67	33	0.2	0.9	184	0.7
SSD only	418	419	178	178	649	325	292	146	88	345	9680	38
RAID+write-mostly	482	482	50	50	766	383	65	33	85	166	0.6	0.7
RAID+write-mostly+write-behind=512	461	462	50	50	754	377	62	31	86	337	165	0.7
RAID+write-mostly+write-behind=2048	456	457	51	51	737	369	65	32	98	382	171	0.7
RAID+write-mostly+write-behind=2048+drivecache	475	476	124	125	725	363	90	45	91	358	249	1
RAID+write-mostly+drivecache	464	464	123	123	736	368	115	58	99	386	255	1
RAID+write-mostly+lvm-writecache	394	395	195	196	584	292	333	167	86	335	13000	50

Conclusion:

1 - --write-mostly made a very significant difference to reads, really pleased with it
2 - --write-behind appears pretty useless, so I am not going to use it
3 - the SATA drive's hardware cache of 32MB has improved things significantly and is a tempting solution, especially given how simple it is to enable with just a "hdparm -W 1" command and it is reasonaly safe data-wise
4 - the LVM/dm-writecache solution seems to have affected the read performance somewhat but generally improved results dramatically, so a very worthy solution, however this is also more complex to set up and comes with more data-loss risks

So which solution do I like best for my current use-case? I think I'll go for the RAID+write-mostly+drivecache setup. Hits a good balance of simplicity, data-safety and performance.

Hope this helps!

(c)The Nux(tm) ;-)

No CSS, no Javascript, no some HTML.
No ads, no tracking, no cookies.

Optimised for Dillo browser, great in Lynx, too :)