Update.
I worked with a VMware technician, and he showed me this article - so that explains the differences between my testing Windows 2003 vs. Windows 2008 at least at an operating system level.
http://kb.vmware.com/selfservice/micros ... Id=2036863
So, if Windows 2008 is trying to send larger I/O block sizes then what could be preventing it?
VMware's guess at the moment is that it has to do with my MTU size of my VMkernel. I have my VMkernel's set to a block size of 8972 - which was the recommended setting provided by Avaya for my Avaya Baystack 5510's. I remember we tried using 9000 but it was dropping the packets every so often when doing ping tests. Another reason why they believe it's my MTU size, we used I/O Meter to see that our I/O response time was on average 164MS during the testing which is pretty terrible. They are looking over some performance logs now to see if that poor behavior only occurred during the testing which I imagine to be the case.
So, what I'm going to try next is to set my VMkernels to 4000, set my SAN NIC's to 4088(Intel Quad ports aren't manual, you can select 4088 or 9014) and try the tests again. I have an HP ProCurve that is capable of 9K Jumbo Frames so I may also just try and increase the 8972 to 9000 on my VMkernels and see. I plan on doing this with my test subnet/NIC's/SAN target I created in my previous post.
The Latest Gartner® Magic Quadrant™Hyperconverged Infrastructure Software