Poor read performance, good write performance

[anton (staff)]

Yes. The answer is "it depends". So if OP did try only one combination I've suggested to try all.

Again there's no "one and only" answer. Somebody should experiment with cache settings and his app of choice as a benchmark tool to get proper numbers.

Yes. Both initiator and target sides please.

i have the same problem with v5.7 Why do u say to put 2 instead of 1 ? 1 disables it right ?

During some tests in the past I found that disabling Delayed Ack only on the initiator side (using MS initiator) gave a slightly better performance than disabling it on both starwind and initiator machines. Did you compare?

The system is using a PERC 5/E adapter, which has 256MB DDR2 memory for its cache.

It is enabled, but I don't know how much this 256MB will help.

And yes, it's battery-backed and the write policy is write-back.

How did you set disk cache policy? Using a similar MD1000 + Perc-5/E Setup I disabled it on the fast 15K SAS Spindles and enabled it on the slower 7.2K SATA spindles. This means the SAS array doesn't use the physical disk cache which gives better performance. For slower disks like the SATA's it's better to enable disk cache.

[sjors]

The system is using a PERC 5/E adapter, which has 256MB DDR2 memory for its cache.

It is enabled, but I don't know how much this 256MB will help.

And yes, it's battery-backed and the write policy is write-back.

How did you set disk cache policy? Using a similar MD1000 + Perc-5/E Setup I disabled it on the fast 15K SAS Spindles and enabled it on the slower 7.2K SATA spindles. This means the SAS array doesn't use the physical disk cache which gives better performance. For slower disks like the SATA's it's better to enable disk cache.

Are you referring to the Read, Write or general cache policy on the PERC?

[CyberNBD]

The system is using a PERC 5/E adapter, which has 256MB DDR2 memory for its cache.

It is enabled, but I don't know how much this 256MB will help.

And yes, it's battery-backed and the write policy is write-back.

How did you set disk cache policy? Using a similar MD1000 + Perc-5/E Setup I disabled it on the fast 15K SAS Spindles and enabled it on the slower 7.2K SATA spindles. This means the SAS array doesn't use the physical disk cache which gives better performance. For slower disks like the SATA's it's better to enable disk cache.

Are you referring to the Read, Write or general cache policy on the PERC?

None of them There is another setting called disk cache policy. It enables or disables the physical disk cache (has nothing to do with the perc cache unlike the other settings).

[anton (staff)]

You mean hard disk on-board cache memory?

[CyberNBD]

You mean hard disk on-board cache memory?

Yes

[sjors]

I just upgraded to v5.7 and redid all testing.
Node 1 exhibits good performance throughout now, so I'm happy.

Now it's node 2 that worries me.
Performance testing (ATTO) gives the same weird performance dip around 1K and 2K.
I mean REAL dips.

Hyper-V Node 1

Node1.png (24.12 KiB) Viewed 20233 times

Hyper-V Node 2

Node2.png (24.15 KiB) Viewed 20233 times

Is there something I may have overlooked in the configs of the two servers?

Also, is there a way to add caching to existing IMG volumes?

[starneo]

As I posted in another forum thread, I do have the same issues.
We use MPIO (round robin) over 4 1xGibit connections (iSCSI).
I have 4targets each with 4GB Cache and 5000ms expery period.

Some Benchmarks:

[Anatoly (staff)]

Not good
So just to clarify:
What's the cache size?
Have you dissabled delayed ACK for all adapters taking part in iSCSI connections (sync channel too)?

[sjors]

@starneo,

Did you follow the ACK recommendations along with the global chimney ones?

If not, perform the following commands:
netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled

Verify the settings with this command: netsh int tcp show

Also, be sure to have disabled the RSS settings in the NIC properties.
(Only do this or the iSCSI adapters.)

That's all I can say for now.

[anton (staff)]

Use "Direct I/O" and "Overlapped I/O" option as with options you've selected we do all discuss bogus values. Thanks!

As I posted in another forum thread, I do have the same issues.
We use MPIO (round robin) over 4 1xGibit connections (iSCSI).
I have 4targets each with 4GB Cache and 5000ms expery period.

Some Benchmarks:

[starneo]

@starneo,

Did you follow the ACK recommendations along with the global chimney ones?

If not, perform the following commands:
netsh int tcp set global chimney=disabled
netsh int tcp set global rss=disabled

Verify the settings with this command: netsh int tcp show

Also, be sure to have disabled the RSS settings in the NIC properties.
(Only do this or the iSCSI adapters.)

That's all I can say for now.

Thanks for the input! Currenty both options are enabled (I thought this was written in a white paper).
I have to change the TcpAckFrequency in the registry, too. That's quite difficult, because this is a productive system.
Will the change of the RSS & Chimney setting interrupt the iSCSI Sesseions?

[Anatoly (staff)]

Actually to changes applied on the system after modifying registry setting you need to reboot anyway.

[sjors]

Thanks for the input! Currenty both options are enabled (I thought this was written in a white paper).
I have to change the TcpAckFrequency in the registry, too. That's quite difficult, because this is a productive system.
Will the change of the RSS & Chimney setting interrupt the iSCSI Sesseions?

Changing settings on the NIC will reset the connection, so yes, you will have to schedule downtime.
(It's almost weekend anyway, so it'll probably be less difficult. )

I'd do the changes, reboot and perform the tests like Anton suggested.

[Anatoly (staff)]

Keep us updated please

[starneo]

At Saturday I set the following parameters to all my iSCSI NICs on Storage and Hyper-V side (except Syncchannel):

=> HKLM->System->CurrentControlSet->Services->Tcpip->Parameters->Interfaces->“<Interface GUI>“
DWORD32-> TcpAckFrequency = 1

for INTEL NICs on Server-Core:
=> HKLM->System->CurrentControlSet->Control->Class->{4D36E972-E325-11CE-BFC1-08002BE10318}->“<Adpater>“:
 RSS = 0 -> off
 TCPChecksumOffload = 0 -> off

for Windows2k8R2 SP1 OS:
=> Netsh int tcp set global chimney=disbaled
=> Netsh int tcp set global rss=disbaled

Now everything is running really smooth and quite fast. I have done a ATTO diskbenchmark on all 4 CSV HA targets simultaniously (queue depth=10, 2GB size) and received 26000 I/Os at the Starwind Performance Monitor (overall). I know physically this is not passible (for our Storages), but with over all 30GB Cache (18GB for Starwind and 12GB on each Raidcontroller), it can be true

So I am happy for now, hopefully this will be the last time with performance issues