[Solved] Hyper-V Server Setup - problem formatting HA target

[jwillis84]

Hello,

I've a problem trying to format the iSCSI HA device created as a target during Hyper-V Server 2012 installation.

Specifics:

HP BL470c G7 blade server
NC553i Emulex iSCSI Select
isolated iSCSI switch
HP DL380p G8 server
StarWind Version 6.0.5569
Multiple HA device targets have been created for multiple HA devices

BL490cG7 server (iscsi initatior, microsoft hyper-v) <--> GbE2/3 Layer switch <--> DL380pG8 server (iscsi target, starwind)

BL490cG7 server media
-- ilo DVD virtual media (hyper-v server 2012 iso)
-- ilo Floppy virtual media(iscsi driver)
-- NC553i Emulex ethernet dual port adapter

DL380pG8 server volume
-- RAID5 direct attached SAS
-- 331FLR Four port adapter - 1 port dedicated to iscsi service, 2 port dedicated sync & heartbeat, 1 port dedicated mgmt

-- 172.16.110.x - sync & heartbeat
-- 172.16.111.x - sync & heartbeat
-- 172.16.114.x - iscsi service
-- 172.25.5.x - mgmt

Symptoms:

Hyper-V Server 2012 boots and begins install
I provide the Emulex device driver to support iSCSI boot volumes
Drive list refreshes and displays Drive 0
I select the Drive 0 for install and it proceeds
The drive is successfully partitioned
Then it attempts to format the drive and fails

Final Error Message:

Microsoft Hyper-V Server Setup

Windows could not format a partition on disk 0. The error occurred while
preparing the partition selected for installation. Error code 0x80070057

If I boot into repair mode and attempt the same process with 'diskpart'

clean - succeeds
partition - succeeds
format fs=ntfs quick - fails

diskpart > 'list disk' indicates there are no disks available

NOTE: I have read the Microsoft Initiator may not work with LUNs that present as 4096 blocks and only work with LUNs that present as 512 blocks. Is there any way to determine the block size Starwind is presenting?

NOTE: I found the diskpart command 'Filesystems' which did list the LUN block size as 512

NOTE: I tried creating just a virtual image iSCSI target instead of an HA device iSCSI target, it behaved in the same manner

NOTE: I tried Upgraded Firmware on the Embedded NC553i Emulex iSCSI Controller from v3.102.517.7 to v4.2.401.2215, it behaved in the same manner

NOTE: I tried Upgraded Firmware and Drivers on the 331FLR Controller, it behaved in the same manner

DL380p G8
Upgrade Firmware from
Boot Code 1.29 to 1.33
PXE 15.2.51 to 15.4.2
NCSI 1.0.88 to 1.1.15

DL380p G8
HP Broadcom 1Gb Driver for Windows Server 2008 x64
From 15.4.0.17
To 15.4.0.19

Attached: I noticed in other similar cases people have been asked to increase the logging verbosity level and upload a copy of the starwind log. I increased it from level 1 to level 2 and have attached it after trying to format the partition again.

[Max (staff)]

Hi Jwillis84,
I've searched the web for cases similar to yours, and so far the results seem to be quite contradictory.
In most cases it looks like a driver issue (which shouldn't be your case since you've already updated them).

Is there any chance you could try to use a single image file target for the installation.
If the installation succeeds - we're looking at an HA compatibility issue, if not - then the issue lies on the client server side.
Look forward to hearing from you.

[jwillis84]

Max,

Thank you.

Yes I tried that.

I also deleted all targets and devices.

Then created a new target with a single virtual image file, same behavior it still failed to format with the same error number.

[jwillis84]

I had to stop work on this to continue work on a different project.

However I did get further information before I had to stop.

1st - I restored a converted VHD to IMG file of a Hyper-V Server 2012 install that was achieved by using the Microsoft iSCSI Target Service.

That is the VHD was created by making an iSCSI target using Microsoft iSCSI Target Service, mounting that with the server and installing Hyper-V Server 2012. Then using the Starwind Converter I converted the VHD to an IMG file and copied that over to the server running Starwind iSCSI SAN software and created an iSCSI Target to share that IMG file.

2nd - That IMG file booted and ran the Hyper-V Server 2012. I then tried to perform a new install over the same iSCSI target being served by the Starwind iSCSI SAN software. The Hyper-V Server 2012 did not have to reformat the NTFS file system and moved the original install to windows.old and began installing a new installation. It failed however at the the very end while trying to install the boot loader components and could not boot from the IMG image after the attempt.

I am not sure what to think.

Perhaps the Installer is confused about the Volume enumeration and is trying to write the Boot loader to the wrong device, or perhaps it thinks the device is write protected. In any event it seems to be able to write files during install to an existing partition and file system, it just can't manipulate the boot partitions properly.

As I said before, dropping to a CMD prompt and running 'diskpart' I can successfully list the disk, select the disk and clean the disk. I can create a primary partition. However whenever I attempt to format that partition the disk seems to "disappear".

I have not tried to read the more verbose Starwind logs to see if perhaps the service is reacting to attempts to partition or format and disconnecting or resetting the iSCSI login.

I have not setup CHAP or Access permissions, perhaps it expects some exclusive access?

I can't help guessing that Starwind is reacting to the MBR access is a bad way and logging the Installers connection out.. I would assume that would be revealed in the Starwind logs if that is happening.

I plan to pursue this problem as time presents itself and would welcome any suggestions for collecting more information to work the problem.

[jwillis84]

A thought occurred to me.

The VHD format can handle "multiple" partitions in a single volume.

Hyper-V Server 2012 like many Windows installs "partitions" a recovery partition of 350 MB at the top of the Volume and installs to the second partition in the lower half of the volume.

The IMG format may not handle "multiple" partitions in a single volume.

I know the Starwind VHD to IMG converter must perform some type of format conversion. Perhaps it is selecting the larger of the partitions and adapting the MBR to point at the newly created "single" partition?

That could explain the incompatibility.

Starwind iSCSI SAN Volumes seem to be intended to host "Data" volumes and perhaps are not suppose to handle "Multi-partition" Boot volumes.

This is pure speculation on my part.

I never used 'diskpart' to see what would happen if I tried to create more than one Primary Partition on a IMG backed iSCSI target.

[Max (staff)]

That's a good thought.
However StarWind's img file doesn't care about the contents of the file, it is just a file which allocates some space on the SAN's NTFS volume.
Honestly,I fall to the impression that it is a problem on the client HBA side. StarWind logs don't show any write errors.

[jwillis84]

I worked on the problem some more after work last night.

I setup an entirely different test bed at home with different hardware

Using Windows 2008R2 as the host for the Starwind iSCSI SAN software and an Intel iSCSI Remote boot card for the client

I was able to create the Target and install Hyper-V Server 2012 to the iSCSI Volume without issue. The Volume was partitioned into two as it normally would be expected.

So this narrows the problem to either the Host Operating System, or the Hardware at my place of work.

I further researched the iSCSI Client it is an HP second sourced Emulex Converged NIC. The Emulex website had more recommendations for supporting iSCSI and specific things to do to improve performance. The HP website strangely did not include any of the information for the same hardware. It was more superficial focusing on the use of the card as a Network NIC.

More concerning the Server NIC had a recent Driver update that indicated "Many" of the Windows Server 2012 issues were just being addressed as of the latest Feb 2013 driver release.. several mentioned "Sudden Connection drops".

I obviously have the ability to work the problem by process of elimination now: 1. Host OS, 2. Client hardware , 3. Host hardware

I'll be very interested to see which turns out to be the culprit.

[jeddyatcc]

Actually, I'm really confused by the whole process here. Why are you booting a Hyper-V server to an iSCSI volume? Even if you get it running, I'm almost certain that it isn't supported. I understand that you may be testing boot to iSCSI or something like that, but the title of the post is misleading. It should be something like Problem formatting when booting from iSCSI.

I have had issues with many drivers in server 2012 with boot to iSCSI. The only real success has been the devices that provide a bios and present the iSCSI drives as local volumes. My only experience is with DELL Broadcom NIC's, and the TOE offload ones seem to work well, but the ones that have required drivers have been very difficult to make work. I only did this to test and demo iSCSI boot, as I really see no need for this where I work at this point. Maybe someday we will be running thin clients this way, but it is way too difficult at this time to worry about it.

[jwillis84]

Actually, I'm really confused by the whole process here. Why are you booting a Hyper-V server to an iSCSI volume? Even if you get it running, I'm almost certain that it isn't supported. I understand that you may be testing boot to iSCSI or something like that, but the title of the post is misleading. It should be something like Problem formatting when booting from iSCSI.

I have had issues with many drivers in server 2012 with boot to iSCSI. The only real success has been the devices that provide a bios and present the iSCSI drives as local volumes. My only experience is with DELL Broadcom NIC's, and the TOE offload ones seem to work well, but the ones that have required drivers have been very difficult to make work. I only did this to test and demo iSCSI boot, as I really see no need for this where I work at this point. Maybe someday we will be running thin clients this way, but it is way too difficult at this time to worry about it.

Diskless boot has been a default for Hypervisors for some time.

The reason is to get HA diskless boot.

The standards have been around for a good while now, HBAs have been doing it for years. The new thing is using a Converged NIC for diskless boot as opposed to SD or USB.

I do agree based on recent experience that Windows 2012 doesn't seem "old enough" to have received proper hardware driver treatment yet, it seems they didn't introduce the hardware vendors to all of the new features early enough and turned them all on.. and its kind of breaking things. Hardware vendors like Emulex seem to be universally recommending "turn it all off" and things sort of work.

The ramped up release schedule with WS2012 R2 already on the doorstep also seems to be contributing to a "reliability" log jam with Windows Server at the moment. So I'm seriously considering going back to Windows Server 2008 R2 until things shake out at Microsoft.

"I'm sorry if the title for the Posting offended you.. I will change it as you suggest"

[jwillis84]

I got time to run another test this evening.

I took the Windows Server 2008 R2 that was running Starwind iSCSI SAN and placed it on the network the Blade server client was booting from.

Same behavior, so that indicated to me (a) its probably not specific to WS2012 or WS2008R2 (b) its probably not specific to the original server hardware

Unfortunately it indicates to me the problem is somewhere on the Client Blade Server.

Since a different hardware client works, that means its not the Hyper-V Installer, its either the 1. Client hardware/firmware or 2. The driver supplied as part of the installation to use the hardware, there are a few parameters to the .inf file included with the driver, or there is a possibility of using an off-brand driver for the hardware -- it leaves me wondering however how it could succeed with the Microsoft SCSI Target

[jeddyatcc]

I was not offended by the title lol, just confused.

I don't really understand wanting to do diskless boot for a Hyper-V host though? What is gained aside from problems? I have been getting some serious performance improvements out of 2012, so I can't wait to move my production hosts to 2012 instead of 2008 R2.

Thanks for taking the time to explain.

[jwillis84]

I found I could download the Hyper-V Server 2012 setup logs from X:\Windows\panther

Nothing more useful than what I already knew:

It says "all is in order" drivers supported, IBS in place, even partitions correctly, but then fails to format.

I plan to take in the other hardware client to confirm this is an Emulex firmware/driver issue and not some random Switch issue.

I would guess the next level would be to get debug information from the firmware or driver of the converged NIC.

2012-07-25 18:36:51, Info IBS DetermineDeviceSupport:Disk 0 has the necessary driver support
2012-07-25 18:36:51, Info PublishDiskInfoOnBlackboard: Successfully serialized disk info.
2012-07-25 18:36:51, Error [0x060442] IBS PerformInstallDiskPreparationOps:Failed to format install drive; hr = 0x80070057
2012-07-25 18:36:51, Error [0x060350] IBS CallBack_PrepareInstallDrive:An error occurred while preparing install drive hr = 0x80070057
2012-07-25 18:36:51, Info IBSLIB PublishCriticalError: Publishing critical error message [Windows could not format a partition on disk 0. The error occurred while preparing the partition selected for installation. Error code: 0x80070057]

[jwillis84]

I setup [both] Microsoft iSCSI Software Target [and] Starwind iSCSI SAN on the same server, in the same environment.

I created a "logical" Interface for the Microsoft iSCSI Software Target on the same server so that I could continue to run on port 3260

I also used the same iSCSI Target IQN the only difference was IP address.

Hyper-V Server 2012 will install to Microsoft Target

Hyper-V Server 2012 will not install to Starwind Target (unable to format)

[jwillis84]

Had an awesome meeting with Mac Craft this morning.

He identified an issue with the HP NC553i 10 Gb Dual Port Lan on Motherboard Adapter in the BL490c Gen 7 blade server we are using.

It has Emulex iSCSISelect Firmware version v4.2.401.2215 currently installed.

Max was more precise, but I believe the issue was with the "FirstBurstLength" parameter of the iSCSI Initiator configuration.

The parameter is set per Target, but in the Emulex iSCSISelect Firmware it is was fixed and not modifiable at 4 Kilobytes.

The Starwind log reported the "FirstBurstLength" used by the Adapter appeared to be less than that negotiated and dropped the traffic.

After Max modified the "FirstBurstLength" in the Starwind Server -> Configuration (tab) -> Server Settings -> Advanced Settings -> Modify (hyperlink) -> PopUp -> SCSI (nav) -> "First Burst Length" (textbox) -> from 65536 to 4096

And then restarted the Starwind service (in order to force the Adapter to relogin and renegotiate the connection) the Hyper-V Server 2012 then could partition and a complete restart of the Installation succeeded normally.

Max indicated this might be such a low number it could effect performance and that I might want to take it up with the Firmware provider to see if there is a way of configuring the parameter on the Adapter to a higher value.

- John

[Max (staff)]

Mac Craft

I already feel more Scottish
John, thank you for the informative update!