Performance Issues

kspare · Tue Oct 01, 2013 6:06 pm

Anatoly you had asked some questions in another post about my config, I thought rather than hijack someone elses post i'll start my own.

Our server is as follows:
Super Micro 6047R-E1R24L Storage Server
Dual 6 core cpu's
64gb of ram
LSI 2108 Controller
15 2tb WD Black Drive
Intel X520-DA 10GB Nics

We are configured with a raid 5 64KB Stripe across 14 of the drives with one as a hot spare.
NTFS volume formated quick with 64k
Firmware on the controller is up to date.
I'm not interested in HA, I simply want the fastest storage I can get with my drives.
I've done some tests moving to and from my starwind server.
The server I move from is a HP DL380 G6 with 8 300GB 15K dual port drives.
The servers are directly connected with identical 10gb cards using copper running jumbo frames.

I've only been able to get the server to write at about 4.5gbps and read between 6 and 7gbps

Once we were satisfied with the speed we attempted to put our test box into production and see how it performed. The results were not good. everything slowed the crawl and we had to pull our vms off.

This was running server 2012. I've read that some other had issues with 2012 and 2008r2 would be better. So i'm planning to re-insall 2008r2 and try that instead.

I've tried a virtual disk and a redirected device. Both performed about the same.

I've read about others having similar performace with freenas so I gave it a shot, out of the box I was able to write at 6gb/s and read at the same. Perfect! Except that i'm not running windows or starwind.

So I know the hardware is capable, but I don't want to run freenas. The managment of the lsi controller and iscsi is much better under windows.

I'm not sure what 'm doing wrong, but I just can't get the performance of freenas.

For anyone wondering why I don't just stick with freenas, there are a couple reasons.
1. No UPS support.
2. Limit array controller support. we had a drive fail and it just rebooted rather than tell us a drive failed.

Those are pretty big reasons for us.

Tue Oct 01, 2013 8:21 pm

First of all you're not going to have a good performance with a parity RAID. LSFS is doing a good job for wide stripe writes but LSFS is not for everyone.
Second you did not say anything about your software config. How much WB cache did you use? Also 2012 R2 is a preferred OS for now. Publish numbers for raw disk
(ATTO is enough to start) and TCP (IPperf and NTtcp). Writes and reads should not have gaps with a smaller packets and TCP should do wire speed. Then configure RAM
disk and make sure it does write speed with iSCSI. Then we'll see what's up and running with image file layered on top of your config...

kspare · Wed Oct 02, 2013 4:26 am

Raid 5 is irrelevant. I'm hitting my numbers with freenas. I've tested the controller with raid 6 and still hit my numbers, so lets move past the controller. I also tested with a raid 0 stripe and a raid 10.. All the same numbers it doesn't make a difference.

When I configured the WB cache I checked off max cache, which gave it about 48gb of cache.

I'll upgrade to 2012 R2, I actually forgot it was out! my bad.

I'll work on the tests for you tomorrow and get back to you.

Wed Oct 02, 2013 8:59 am

Thanks! We`ll look forward to hear back from you.

kspare · Wed Oct 02, 2013 4:50 pm

I did some more testing today and I found that I'm getting the same speeds whether I have write cache on or not. The only time we speed a speed increase or decrease is if I turn off the writeback cache on the array controller, and things come to a complete crawl. This is running the latest 6.0 sofware and server 2012.

I do hit about 3.5gb/s but more average of about 2.0. It's a far cry from the 6.0gb/s I hit with freenas but the zfs arc caching was working, so it looks like that is my bottle neck?

Wed Oct 02, 2013 11:17 pm

1) You cannot use parity RAIDs for VM storage.

2) What test do you use to test performance?

3) You need to remove / disable all caches to get raw performance numbers or it would be measuring the weather on the Mars

4) How did you manage to make 48 GB cache non-visible for writes (back to 2)? It can adsorb damn a lot...

kspare wrote:I did some more testing today and I found that I'm getting the same speeds whether I have write cache on or not. The only time we speed a speed increase or decrease is if I turn off the writeback cache on the array controller, and things come to a complete crawl. This is running the latest 6.0 sofware and server 2012.

I do hit about 3.5gb/s but more average of about 2.0. It's a far cry from the 6.0gb/s I hit with freenas but the zfs arc caching was working, so it looks like that is my bottle neck?

kspare · Wed Oct 02, 2013 11:26 pm

for my tests i'm simply moving a eager zeroed vm from local storage to iscsi storage on a vmware server. (esxi 5.1)

I actually think i've determined it to be a problem with the network, again it all works perfectly fine under freenas.

I created a 40gb ram drive and I still got the same transfer numbers. I was able to move data at 5gb/s to two nics at the same time, however even with a ram disk I couldn't get over the 5gb/s

I'm not sure where else to go, i've tried all the latest drivers for the intel x520da nic on server 2012. i'm working on rebuilding the box with server 2008r2 and we'll see where that goes. I know it's not the esxi box as nothing changed, so it's something in windows.....

Wed Oct 02, 2013 11:32 pm

Check TCP optimization settings with our forum. Also try to run TCP benchmarks (I;m not aware of the ones for ESXi <-> Windows so could try a VM but that's not the best bet...)

kspare · Sat Oct 05, 2013 6:32 am

I ran the tcp optimizations in 2008r2, it didn't help. I have to try it again in 2012

We're back testing freenas, and we found once we turned off sync writes in zfs we got amazing performance, to the tune of 8gb/s sustained to 3 servers. pretty amazing for 16 sata drives.

I took your advice and i'm running them in a raid 10 array.

The problem seems to be within windows, even with a 40gb ram drive I can't hit over 3gb/s, but I would see spikes all the way up to 8gb but my sustained network speed is terrible.

On a server with 3 gig links running mpio using starwind I get about 30% of my expected speed per nic, but on freenas I can saturate all the nics.

I know it's not the starwind software, the ram drive kind of proves that I think, and I know it's not hardware, freenas proves that.

So i'm kind of stuck in a corner here. I've tried all kinds of combinations of the intel x520 driver and nothing seems to work. Something limits the connections to 30% and thats it.

Any ideas?

Sat Oct 05, 2013 11:22 am

If RAM disk works fine with Windows I mean you can saturate your network (please confirm!) then something is with disk subsystem rather then with drivers for NIC and so on.
Can you confirm 1) RAM disk is fine and 2) disk can give the same numbers in a loopback under FreeBSD and Windows.

?

kspare · Sun Oct 06, 2013 6:17 am

I can't saturate the network with a ram disk. I get the exact same speed as I do with my raid 10 lun.

Sun Oct 06, 2013 3:16 pm

Indeed that sounds like a network issue

Do you have a spare Windows box to re-mount the NICs from ESXi to check TCP performance in Windows-to-Windows config? Also Google a bit (I'm not comfortable to publish out-of-site links here) for ESXi-to-Windows TCP performance tests. If your 10 GbE gear cannot do wire speed with TCP - no iSCSI @ wire speed obviously...

kspare · Mon Oct 07, 2013 5:04 am

I guess i'm hoping someone else can chime in. We're using a good super micro server, and people rave about these intel x520-da nics, so how can I not get the same performance in windows as I do in freebsd? I'm confused by this.

I can't be the only person with this problem. Or is everyone satisfied with 33% saturation of a 10gb nic?

Mon Oct 07, 2013 2:45 pm

Hi guys!

I jjust want to add one important thing that Anton mentined above, but it just drowned in conversation for some reason: you need to use the special tools to measure the performance (IOmeter is preffered for disk, the network can be benchmarked with iperf or ntttcp), otherwise

it would be measuring the weather on the Mars

Maybe it wil be a good idea if you could read through our Benchmarking Guide.
The bottom line is that every chain in the system need to be properly benchmarked, because if not we can only guess about what is the bottleneck.

kspare · Sat Oct 19, 2013 5:20 am

We found that the array controller had bad memory on it, it just never showed up anywhere until it finally started to fail. We're now hitting 5gb/s write speed to a 16 disk raid 5 set on 2tb sata drive, and reading at between 6-7gb/s.

We're also using lsi's cache cache on a 9266 vs the 9260 that came with our server and it works great!

We still have to tinker with trying raid 10 vs raid 5 and virtual disk vs physical disk redirection.

I'm not really sold on the whole virtual disk as we use one large iscsi volume to present to esxi. We also don't run HA. so i'm not sure if there is an advantage to the virtual disk at all?