ESXi iSCSI initiator WRITE speed

Software-based VM-centric and flash-friendly VM storage + free version

Moderators: anton (staff), art (staff), Max (staff), Anatoly (staff)

User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Fri Apr 01, 2011 9:47 am

1) Oh, good! So it's the same Windows VM inside ESX hypervisor and it's only ATTO setting making HUGE difference in your tests. Right? So I'll be checking what's really different...

2) ESX initiator is branched out old Cisco one (also Linux-based) so I expect there's some mis-config between Windows and Linux initiator settings. Again, let us compare Vs. OpenFiler.

3) So no heavy swap. This is good. Can cross this out :)

4) I understand. So doing Write-Back Cache helps but not completely. Let's run cross-tests here and complete cache line size changes.

Thank you for your feedback. We really appreciate it!
CyberNBD wrote:1) It's an atto setting. You can check the "direct IO" box or not within atto before testing.

2) The other target software I tested was OpenFiler. Works very well and performs well out of the box but it but it doesn't support Windows 2008 R2 Clustering (something to do with persistent reservations etc) so that made me look for other solutions. I have no specific preference for Linux or Windows OS SAN software BUT is has to perform well for both Win and ESX(i) initiators. Some windows boxes are going to connect directly through MS initiator because of the Clustering. All other hosts (mixed Win and Linux) will use ESXi disks and thus ESXi iSCSI initiator.

Allright. Most of the time I have a taskmanager window open during testing but i haven't seen anything abnormal on the Starwind box. CPU usage around 0-3%, sometimes peaks to 10%. No swapping memory neither. Fills up nicely when testing using cache mode. Also took a look at the resource monitor (disk queue length etc) but also nothing abnormal there.

I'm testing using the 2048MB WB cache right now. Some improvement using atto. Clean W2k8 install is going faster compared to no cache but it still took about 40 minutes to complete where I'm expecting 15 to max 20 mins based on previous tests.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
michal
Posts: 7
Joined: Wed Feb 23, 2011 4:04 am

Sun Apr 03, 2011 8:03 pm

Hi All,

I'm a Mohawk College student trying to use star-wind in a similar lab environment(WS 2008 R2 clustering AND VMWare ESXi). I have ALSO tried openfiler, as well as MS iSCSI Target 3.3(from my technet account), Open-E, and NexentaStor. I've also looked at DataCore but haven't as of yet tried it.

FYI, this lab is not to do with school directly, its out of personal interest and learning, to get me ahead and give me an edge in the workforce. Plus, while some might find this strange, its a hell of a lot of fun for me, I enjoy playing with enterprise level systems and hardware, as well as tinkering and tweaking the hell out of consumer level stuff(I have an overclocked Dual Intel Xeon 5650(6core, 12 threads)on a EVGA SR2 motherboard overclocked to 4.2GHz, thats 12cores, 24threads in total :lol: ).

In short, this stuff is fun to me, and I greatly enjoy it. Anyways, enough Off Topic babbling;

The reason I have been largely coming back to starwind is because it provides the easy, simple, and well performing HA, AND in my Windows tests seems to perform very well. Thats the other thing, I originally did much of my performance tests using Windows iSCSI initiator, and it was one of the better performing solutions... That is, until I tried some testing inside a VM on one of my ESXi 4.1 Update 1 Hosts.

I get the SAME issues as he is getting. I will post a few screenshots of it and the various tests I've done as soon as I can.


P.S. Is there any way of getting a longer than 60 day licence, or do you guys have some sort of free version? It's being used in a lab only environment.
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Mon Apr 04, 2011 9:57 am

In a nutshell: for some unknown yet reason under some circumstances StarWind's non-cached I/O shines at the time cached StarWind's I/O just suck. I mean *locally* cached. And with ESX iSCSI initiator mostly (only). At this moment we gave this issue a highest priority and trying to figure out what we do to make Windows Cache Manager very unhappy. If you'd find some time to help us with testing on your "problematic" hardware we'd very much appreciate this :)

We have a free version but it has no HA...

P.S. I've dropped you an e-mail so please check your Inbox.
michal wrote:Hi All,

I'm a Mohawk College student trying to use star-wind in a similar lab environment(WS 2008 R2 clustering AND VMWare ESXi). I have ALSO tried openfiler, as well as MS iSCSI Target 3.3(from my technet account), Open-E, and NexentaStor. I've also looked at DataCore but haven't as of yet tried it.

FYI, this lab is not to do with school directly, its out of personal interest and learning, to get me ahead and give me an edge in the workforce. Plus, while some might find this strange, its a hell of a lot of fun for me, I enjoy playing with enterprise level systems and hardware, as well as tinkering and tweaking the hell out of consumer level stuff(I have an overclocked Dual Intel Xeon 5650(6core, 12 threads)on a EVGA SR2 motherboard overclocked to 4.2GHz, thats 12cores, 24threads in total :lol: ).

In short, this stuff is fun to me, and I greatly enjoy it. Anyways, enough Off Topic babbling;

The reason I have been largely coming back to starwind is because it provides the easy, simple, and well performing HA, AND in my Windows tests seems to perform very well. Thats the other thing, I originally did much of my performance tests using Windows iSCSI initiator, and it was one of the better performing solutions... That is, until I tried some testing inside a VM on one of my ESXi 4.1 Update 1 Hosts.

I get the SAME issues as he is getting. I will post a few screenshots of it and the various tests I've done as soon as I can.


P.S. Is there any way of getting a longer than 60 day licence, or do you guys have some sort of free version? It's being used in a lab only environment.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
User avatar
Bohdan (staff)
Staff
Posts: 435
Joined: Wed May 23, 2007 12:58 pm

Mon Apr 04, 2011 12:15 pm

Hi,

May I ask you for some additional information? How did you provide the iscsi target to the virtual machine (I mean ESX initiator case)? Is it raw device mapping or creating of new vmdk over the iSCSI datastore?

We performed similar tests in our testlab and here are the results:

MS iSCSI Initiator, no Direct io


MS iSCSI Initiator, Direct io


ESX iSCSI Initiator RDM, no Direct io


ESX iSCSI Initiator RDM, Direct io


ESX iSCSI Initiator Datastore, no Direct io


ESX iSCSI Initiator DataStore, Direct io


Also could you please show us the screenshots of task manager captured during the ATTO benchmark testing of the target connected via ESX iSCSI initiator?
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Mon Apr 04, 2011 12:25 pm

Tiny remark: Windows Task Manager from inside the virtual machine running ATTO Disk Benchmark.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
CyberNBD
Posts: 25
Joined: Fri Mar 25, 2011 10:56 pm

Mon Apr 04, 2011 11:42 pm

I have tested using VMDK Datastore.

80 Gig iSCSI Target formatted as one big VMDK Datastore:
- 50 gig allocated to the Windows VM as second disk for testing with Atto. Used local datastore for OS Disk.
- 18 gig allocated as OS disk to another VM for the non-scientific "installation timing" test.

I will see if i can make the taskmgr screenshots tomorrow.
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Tue Apr 05, 2011 5:56 am

Good. Please keep us updated! Thank you very much for cooperation!
CyberNBD wrote:I have tested using VMDK Datastore.

80 Gig iSCSI Target formatted as one big VMDK Datastore:
- 50 gig allocated to the Windows VM as second disk for testing with Atto. Used local datastore for OS Disk.
- 18 gig allocated as OS disk to another VM for the non-scientific "installation timing" test.

I will see if i can make the taskmgr screenshots tomorrow.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
michal
Posts: 7
Joined: Wed Feb 23, 2011 4:04 am

Thu Apr 07, 2011 4:53 am

Exams are coming up for this semester, and thus I haven't had much time to test/play with my lab. I will have a couple weeks between semesters(I'm doing a summer semester as well, so no summer break).

Anyways, I did play around, after Anton's comment about how the local cache(inside the vm) was a party to the issue somehow. When I turn off write caching(inside the vm), vm performance generally goes up. It's mostly subjective as the atto bench doesn't change, but when I tested the write performance(ESXi initiator) by coping a file from my file server, I went from 10 megabytes/s transfer speed, to 40megabytes per second.
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Thu Apr 07, 2011 6:21 am

This still sucks/// Good news (not for you and others unfortunately) is we've managed to reproduce the same issue with both MS and other big iSCSI SAN vendor software. So we're not alone and it's EMC (VMware) to blame in "whooping" performance :) But it does not bring anybody any joy so we continue our investigation.
michal wrote:Exams are coming up for this semester, and thus I haven't had much time to test/play with my lab. I will have a couple weeks between semesters(I'm doing a summer semester as well, so no summer break).

Anyways, I did play around, after Anton's comment about how the local cache(inside the vm) was a party to the issue somehow. When I turn off write caching(inside the vm), vm performance generally goes up. It's mostly subjective as the atto bench doesn't change, but when I tested the write performance(ESXi initiator) by coping a file from my file server, I went from 10 megabytes/s transfer speed, to 40megabytes per second.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
CyberNBD
Posts: 25
Joined: Fri Mar 25, 2011 10:56 pm

Thu Apr 07, 2011 7:36 pm

Below you can find taskmanager screenshot of the full test.

Since I have tested some other iSCSI targets, this one is taken after a clean installation of the StarWind server. I also switched from the demo to a licensed (thanks Anton) version. Only modification made on starwind side so far is the MTU change on the iSCSI interfaces to 9014. VMWare /virtual machines didn't change.

Atto performance using non-direct IO has improved somehow. Direct IO stays the same. Also performed my "non scientific" W2k8 clean install time test again which also stays as slow as before.


Bohdan (staff) wrote:Hi,
Also could you please show us the screenshots of task manager captured during the ATTO benchmark testing of the target connected via ESX iSCSI initiator?
CyberNBD
Posts: 25
Joined: Fri Mar 25, 2011 10:56 pm

Thu Apr 07, 2011 7:43 pm

Same thing I discovered during my tests with other software. All windows based targets seem to have this ESXi performance issue. Linux performs great and last week I did some tests using ZFS (OpenIndiana) which also doesn't have the issue.
anton (staff) wrote:This still sucks/// Good news (not for you and others unfortunately) is we've managed to reproduce the same issue with both MS and other big iSCSI SAN vendor software. So we're not alone and it's EMC (VMware) to blame in "whooping" performance :) But it does not bring anybody any joy so we continue our investigation.
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Fri Apr 08, 2011 8:59 am

So you've basically had confirmed our results. Good :)

P.S. Everything is not that fine with Linux-based targets, I've forwarded you some test results. Please keep them private as I'm not sure our competitors would be happy to see them published w/o their permission.
CyberNBD wrote:Same thing I discovered during my tests with other software. All windows based targets seem to have this ESXi performance issue. Linux performs great and last week I did some tests using ZFS (OpenIndiana) which also doesn't have the issue.
anton (staff) wrote:This still sucks/// Good news (not for you and others unfortunately) is we've managed to reproduce the same issue with both MS and other big iSCSI SAN vendor software. So we're not alone and it's EMC (VMware) to blame in "whooping" performance :) But it does not bring anybody any joy so we continue our investigation.
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
CyberNBD
Posts: 25
Joined: Fri Mar 25, 2011 10:56 pm

Sat Apr 09, 2011 8:05 pm

Well, seems unchecking a little box solved everything :?

I just found out ESXi also has a delayed ack setting, which is on by default. After disabling and rebooting the ESXi box speed went up, clean 2k8 install on iSCSI datastore completes within 14 minutes, like it should :)


vSphere Client, Configuration, Storage Adapters, iSCSI adapter Properties ==> Advanced, last option.
User avatar
anton (staff)
Site Admin
Posts: 4021
Joined: Fri Jun 18, 2004 12:03 am
Location: British Virgin Islands
Contact:

Sat Apr 09, 2011 10:44 pm

Interesting... Please let us check everything on our side. Thank you!
Regards,
Anton Kolomyeytsev

Chief Technology Officer & Chief Architect, StarWind Software

Image
michal
Posts: 7
Joined: Wed Feb 23, 2011 4:04 am

Sun Apr 10, 2011 8:21 am

I just want to note, at least in my limited google-fu research, Delayed Ack feature in ESXi is not the same thing as the naggle algorithm

http://virtualgeek.typepad.com/virtual_ ... phere.html

I'm going to try this out myself(disabling delayed ack), will pipe back in with the results :D
Post Reply