Write Cache. Bad Performance. [RUS]

Wed Nov 07, 2012 9:20 am

It's interesting. You are using software RAID on StarWind server that stores image files for virtual devices, is it right?
Could you please write what software have you used and what settings affect the performance on software RAID?

flysats · Wed Nov 07, 2012 10:33 am

Yes that's right.
I certainly would like to try raw, but StarWind with this long-standing problem ...

I used cache software
When I turned off the lazy write(Defer-Write in terms of program) - is exactly the same drop in performance as in starwind
Сaching on the disk level significantly improve performance. StarWind working at the time with no cache.

Wed Nov 07, 2012 10:42 am

Thank you for the information!
We are researching the issues now.

flysats · Wed Nov 07, 2012 11:45 am

I'm really interested in using StarWind. And I've spent an insane amount of time testing. It would be nice to be rewarded for their work, even if it is only working as I need the product.

Compare...[iometer 64k 100%write 10%random] (StarWind cache is worse than without it )

Wed Nov 07, 2012 12:26 pm

Can you give some more hints (maybe in private) what you run to test us against with so we could run the cross-tests here to check everything.
For now what you say sounds pretty much unrealistic. With dirty cache and sustained write there's no way cache will do anything except overhead so performance increase sounds very suspicious. So I want to see it myself.

flysats wrote:I conducted additional tests using caching software from another manufacturer. has doubled and in some cases triple performance on write scenario.
My preliminary conclusion: the type and caching algorithms in Starwind, conflict when interacting with software RAID in WINDOWS Server. I found the following settings of the additional caching software, at which a similar drop in performance.

You need to give more options to configure caching.

p.s. in paragraph 2 of the previous question was referring to Write back cache. Additional software that I use now for caching can do is what I wrote.

Wed Nov 07, 2012 12:28 pm

I think we can arrange this. Please check your e-mail.

flysats wrote:I'm really interested in using StarWind. And I've spent an insane amount of time testing. It would be nice to be rewarded for their work, even if it is only working as I need the product.

[ ... ]

Wed Nov 07, 2012 12:31 pm

That's as it should be. If you have sustained writes your cache get filled very fast and after that the only thing it does - increases latency and I/O path.
Cache assumes you have production-type rather then sequential type load. So the only sceanrio you're covering with your test now is - linear video capturing. So
for this scenario indeed you need to turn cache OFF.

Can you post "other software" benchmark results here?

flysats wrote:
...

Compare...[iometer 64k 100%write 10%random] (StarWind cache is worse than without it )

flysats · Wed Nov 07, 2012 12:43 pm

it's much easier ...
I can give you access to a test machine

The last screenshot is not visible overflow cache - because it is not crowded! just a test host is under production load and not corny enough performance

Wed Nov 07, 2012 12:56 pm

Weird... OK, will love to have access to.

flysats wrote:it's much easier ...
I can give you access to a test machine

The last screenshot is not visible overflow cache - because it is not crowded! just a test host is under production load and not corny enough performance

flysats · Wed Nov 07, 2012 1:13 pm

Access Granted. See PM

Wed Nov 07, 2012 1:39 pm

Got it. Please give us some time for investigation. Thanks!

flysats wrote:Access Granted. See PM

maximp · Sat Nov 10, 2012 1:26 am

Hello, I ran into similar issues (poor write performance with write-back cache enabled). My setup is somewhat similar I guess. I use HP Gen8 385 and hardware raid controller with 10 SATA disks in a RAID5 configuration. Server has 16GB of RAM and 10GB was assigned to cache. I use free version of Starwind target and "image disk" as a target. I did use CrystalMark (i will re-test with IOMeter) for tests. All tests I ran in a VM against data stores from different storage arrays, including EMC VMAx and HP EVA. I was indeed surprised by somewhat poor write performance see attached.

Thanks,
Max

Sat Nov 10, 2012 8:56 am

In the original poster config the whole thing we've been missing - he was comparing FancyCache (pinpointed physical memory kernel running cache software) Vs. StarWind running
inside VIRTUAL MACHINE. These are not comparable configurations as VM gives overhead and memory access speeds are very different. We'll look further into this to see what could be done to improve performance (as running inside ESXi VM should be a supported configuration with the next release) but we're not expected to compare with non-virtualized solutions in any case. For your scenario it's not clear as well - where do you have StarWind running? Are you comparing StarWind running on a stand-alone hardware Vs. EMC VNX boxes or do you have StarWind running inside guest VM? Could you please draw an interconnection diagram and tell a little bit more about your configuration? Thanks!

maximp · Sun Nov 11, 2012 6:57 am

Thanks Anton for prompt response. I am running StarWind on a physical server (target) but all of the tests I did were done on a VM (initiator). Let me run same tests on a physical initiator using IOMeter. There is a good chance that my tests using CrystalMark and VM might be indeed skewed.

My end goal is to come up with a cheaper SAN alternative for some of the apps that don't require high end storage.

Max

Mon Nov 12, 2012 1:41 pm

OK, so we`ll wait some news from you.

BTW - you are definitely able to create cheap high-performable storage, but I`d recomend you to consider increasing of its availability: if you`ll use stand-alon SAN mode you will have downtime during every planned server reboot. I hope you get my point.