One large iSCSI target or multiple small ones?

[adns]

I'd like to resurrect this thread as I think some of the answers given over-simplified the question. I've done some reading on this and I'm not sure what my stance is on this as I've read arguments going both ways. My take on this question is that one big target makes it easy to manage, but management isn't my only concern. I’m also concerned about backups, overall performance and disaster recovery. My fear (like an earlier post suggested) is that if one IMG file goes corrupt then all VMs are down until a backup can be restored.

In the VMware SAN Configuration Guide page 37 it brings up some pros and cons that should be considered. For convenience I'll paste it below. Considering the VMware specific points, what does Starwind consider as a best practice?

Choosing Larger or Smaller LUNs
You need to plan how to set up storage for your ESX Server systems before you perform installation. You can choose one of these approaches:

• Many LUNs with one VMFS volume on each LUN
• Many LUNs with a single VMFS volume spanning all LUNs

You can have at most one VMFS volume per LUN. You could, however, decide to use one large LUN or multiple small LUNs.

You might want fewer, larger LUNs for the following reasons:

• More flexibility to create virtual machines without going back to the SAN administrator for more space.
• More flexibility for resizing virtual disks, doing snapshots, and so on.
• Fewer LUNs to identify and manage.

You might want more, smaller LUNs for the following reasons:

• Less contention on each VMFS due to locking and SCSI reservation issues.
• Different applications might need different RAID characteristics.
• More flexibility (the multipathing policy and disk shares are set per LUN).
• Use of Microsoft Cluster Service, which requires that each cluster disk resource is in its own LUN.

NOTE: You can divide your datacenter into servers that are best configured with fewer, larger LUNs and other servers that use more, smaller LUNs

Making LUN Decisions
When the storage characterization for a virtual machine is not available, there is often no simple answer when you need to decide on the LUN size and number of LUNs to use. You can use one of the following approaches when making the decision:

Predictive Scheme: In the predictive scheme, you:

• Create several LUNs with different storage characteristics.
• Build a VMFS volume in each LUN (label each volume according to its characteristics).
• Locate each application in the appropriate RAID for its requirements.
• Use disk shares to distinguish high‐priority from low‐priority virtual machines. Note that disk shares are relevant only within a given ESX Server host. The shares assigned to virtual machines on one ESX Server host have no effect on virtual machines on other ESX Server hosts.

Adaptive Scheme: In the adaptive scheme, you:

• Create a large LUN (RAID 1+0 or RAID 5), with write caching enabled.
• Build a VMFS in that LUN.
• Place four or five virtual disks on the VMFS.
• Run the applications and see whether disk performance is acceptable.
• If performance is acceptable, you can place additional virtual disks on the VMFS. If it is not, you create a new, larger LUN, possibly with a different RAID level, and repeat the process. You can use cold migration so you don’t lose virtual machines when recreating the LUN.

Tips
When making your LUN decision, keep in mind the following:

• Each LUN should have the right RAID level and storage characteristic for applications in virtual machines that use it.
• One LUN must contain only one single VMFS volume.
• If multiple virtual machines access the same LUN, use disk shares to prioritize virtual machines. administrator for more space.

[TomHelp4IT]

I get asked this question quite often too, and as you have realised there are a lot of factors at play. For a start how many hard drives is your storage system made up of? Say you have 8 as that's a fairly standard number of bays for a decent server, you could make them all one big RAID5 array to maximise capacity whilst keeping some redundancy. That's ok but then you will be taking a performance hit, especially on the write speed, and with 10+ VMs running on it you could potentially find your disk IO is a big bottleneck. RAID1+0 will halve your capacity but give you much better performance whilst still keeping the redundancy. If you have some database type servers (i.e. Exchange and SQL) then the disk IO is a major consideration since they typically do a lot of small random read/write operations which can quickly slow your storage system right down, the more users the worse it get usually.

Its very difficult to find any definitive guides on this subject, MS have published a good article on Exchange IO requirements, but for other server types the typical advice is to run some performance testing on your existing systems. Its why storage/SAN consultants claim they are worth the money they charge, its a black art and lot of it is down to experience and allowing plenty of over-specification. You can improve things by creating separate LUNs for specific uses, e.g. with 8 disks you could set 4 as RAID1+0 for your Exchange/SQL data, and then 4 as a RAID5 for your servers, that way at least they wont slow each other down. For bigger deployments the recommendation is to create separate array for transaction logs as it consists of mostly sequential writes, which a couple of disks can handle well enough if they aren't being interrupted by random read requests all the time.

Having said that, in general performance terms though there is no substitute for having as many hard drive spindles as possible, so a 1TB array made up of 10x 100GB disks will be 5x faster than one made of 2x 500GB disks. Also your RAID controller will be much better at improving performance than you are, so make sure you have a good one with as much battery backed cache as possible.

You will probably find that once you have configured your arrays it will probably dictate to a certain extent what your iSCSI target sizes will be, and while larger ones are certainly more efficient the DR aspect always worries me. I would be very reluctant to have all my VMs on one large LUN as if it was lost for some reason I would have no VMs until the whole thing was restored. At least with 2 or 3 LUNs you will be able to get the most critical VMs back online faster. Like I said before, over-specify your storage and then you can have the luxury of not having to worry too much about efficiency.

[adns]

Thank you for your input on this. Let's narrow down the scope of this question since at the time of this writing StarWind Enterprise has a fairly defined target customer base. Their solution is great for SMBs in regards to features and pricing, and in my opinion they are a perfect backend for VMware Essentials and Essentials Plus. In our typical SMBs with VMware Essentials we'll have 3 hosts, usually 2 SAN servers (with 12 SATA 1TB drives in RAID 10 totaling to 6TB) and maybe 10-20 virtual servers (including Exchange, SQL Servers, Remote Desktop servers, file servers, etc.) I consider this to be a pretty common environment in the SMB space, so using this as a baseline what are people's thoughts on how to carve out targets for your VMs?

Undoubtedly it's a piece of cake to manage 1 big target. You need to touch the SAN once and then all future storage work is done in the virtual hosts. But the VMware document mentions "contention on each VMFS due to locking and SCSI reservation issues." Does anyone know what this translates into performance wise? And is there a real risk when you put all VMs in one IMG file?

If an iSCSI target is created for each VM, then not only is there a lot of management overhead, but it's also an inefficient use of disk space as other VMs can't tap into the unused storage sitting out there. But it *feels* safer when one starts considering the disaster recovery plan.

Or is a hybrid approach a better way to look at it as has been suggested by some. Maybe three 2TB targets should be created on that 6TB SAN so that if there is a problem at least it's not all the VMs that are immediately affected.

[Aitor_Ibarra]

This is what I do, although I use Hyper-V so my experiences may not be directly comparable with VMware...

Original Hyper-V (before CSV)

• For failover reasons, you have to have one windows volume for each virtual machine, or one windows volume for each VHD if you want them to be on different iSCSI targets
• Therefore in Starwind you have to create at least one iSCSI target per virtual machine
• I use img files exclusively
• I create the img files as close as possible to the size required, but extend them if necessarry (e.g. if a VM needs a bigger boot disk than necessarry
• I put multiple imgs on each windows volume on the starwind box, but not more than 3 or 4 per volume to minimise disk contention
• With limited numbers of disks and relatively small VMs, I tend to use RAID-1. I would only use RAID-10 if I needed an img bigger than what could fit into a RAID-1 (i.e. bigger than a single disk). Multiple RAID-1s give more random IOPs than one big RAID-10, which is important when there are multiple VMs sharing the same space. This is less of an issue if you are using SSDs due to their faster random i/o (no heads to move).
• I never use parity RAID (5 or 6) for VMs even though the write penalty is lessened due to my fast RAID cards. The main reason is again that random i/o is going to be worse than multiple RAID-1s, and for me this out weighs the capacity gain. The other reason is that a RAID rebuild is going to affect far more VMs.

Hyper-V R2 (with CSV) - prob more applicable to vmware

• Now you can have multiple VMs on one iSCSI target
• I sometimes use diskbridge as well as img (although as Stawind 5 HA only works with img at the moment, I will be switching back to img exclusively)
• I create one big img that fills the windows volume.
• again I try not to have more than 3 or 4 VMs jostling for access to a single windows volume, although it depends on the i/o needs of each vm
• again I rarely use RAID 10 and never use RAID 5 or 6.

The basic rule I follow is - you want those drive heads to move as little as possible. Therefore you don't want lots of VMs fighting for the same drive, unless they have relatively low i/o requirements. This drives you to use multiple RAID-1s or RAID-10s with the ideal being one RAID volume per VM or one VM's virtual disk.

Backup is a seperate issue! I've tried DPM to backup VMs at the host level. This is with a DPM agent installed on each hyper-v host. I've not had much success with DPM 2007 and it doesn't work with CSV (although the latest DPM 2010 beta does). Instead I've moved to giving each VM its own iSCSI target specifically for backup and using backup internal to the VM. The major drawback of this approach is that you lose centralised reporting. The plus is that the owner of each VM has full control over backup (I rent virtual machines).

Hope this makes sense!

cheers,

Aitor

[Aitor_Ibarra]

Just like to add - I've never experienced a corrupt img file, although this would be a similar scenario to a corrupt windows volume - you'd have to restore from backup. I have experienced disk failures and that's why I'd always use RAID. Having smaller RAID volumes, with fewer VMs sharing them, obviously makes for faster recoveries and lesser impact of failure, but obviously you pay in terms of capacity.

If you can afford it, it's better to use smaller, faster drives for your VMs and leave the 500GB+ drives for backup, or at least data that tends to be read sequentially. I mostly use 2.5" 300GB or 320GB drives at 7200rpm, with my favourites being 300GB 10krpm SAS drives. A RAID-1 of these can rebuild in less than 2 hours and they are very good (by HD standards) at random i/o, so they can handle multiple VMs quite well. However I don't think 15K drives are worth the premium - better to get SSDs. So far I've had good results from the Intel X25-M in RAID-1 and it has much better random i/o than a 15K SAS drive.

[Constantin (staff)]

Do you have any additional questions?