Friday, April 4, 2014

Create Windows stack using Heat Orchestration Template in Openstack

The blog explains the process of creating a basic Windows stack in Openstack using Heat Orchestration Template. The hypervisor being used is VMware ESXi5.5.

Image preparation to upload in glance

  • Lets start with creating a Windows 2012 R2 VM in ESXi. In this server, download and install the cloudbase-init package for windows .The beta version is available at this link:

Follow the steps in this link for installation :

  • Once installation is completed, edit the 'setup.exe" registry key at HKLocal machine/SYSTEM/SETUP/STATUS/ChildCompletion and change the value from 1 to 3. This is to avoid a system restart exception when the image boots up for the first time in openstack
  • If you want to do any custom configurations in the windows machine, like open a specific firewall port, enable ping ,rdp can do it at this point
  •  Run Syprep and shutdown the VM
 C:\Windows\System32\sysprep\sysprep.exe /generalize /oobe /shutdown

  • Use a VMware standalone convertor ,select the prepared VM as source and convert it to a VM suitable for Vmware workstation 10.0.x. When the conversion process is completed, you will get a vmdk and .vmx file at the destination
  • Using winscp or any other similar tools, copy the converted vmdk to your openstack glance  server. 
  • Create an image from this vmdk using the following command

glance image-create --name <image name>--disk-format=vmdk --container-format=bare --is-public=true --property vmware_disktype="sparse" --property vmware_adaptertype="ide" --property vmware_ostype="windows8Server64Guest" < Openstack_win2012.vmdk

Sample yaml template

The template given below uses heat orchestrator to spin up an instance with the image that we created and install IIS in it

heat_template_version: 2013-05-23

description: >
  Basic windows+IIS installation


    type: string
    description : Name of a KeyPair to enable access to the instance
    type: string
    description: Instance type for Windows server
    default: m1.small
      - allowed_values: [m1.small, m1.medium, m1.large]
        description: instance_type must be one of m1.small, m1.medium or m1.large
    type: string
    description: ID of the image to use for the Windows erver
   type: string
   description: windows feature to be installed

    type: OS::Nova::Server
      image: { get_param: image_id }
      flavor: { get_param: instance_type }
      key_name: { get_param: key_name }
          template: |

            Install-WindowsFeature -Name feature -IncludeManagementTools
            feature: { get_param: windows_feature }


 Get password of the instance

Once the instance is up and running, you can use the keypair used while spinning up the stack to retrieve the password. In your openstance machine, run the following command

nova get-password <instance-id> <private key>

instance - id: This is the Id of instance created by the stack , can be obtained from horizon dashboard
privaye key : This is the pem file you downloaded while creating the keypair. Please note that you should copy it over to openstack machine using winscp before running the get-password command


The VMware specific configuration options for creating glance image
The Vmware_ostype derived from the enumerator VirtualMachineGuestOsIdentifier in the API reference:

Tuesday, February 11, 2014

How to create new Local SR in XenServer

In this blog, I will explain the process of creating an additional local SR in XenServer. In this scenario, there is an additional hard disk present in the server which I plan to add as a local SR in addition to the existing Local storage

First you need to identify which disk is being currently used for your existing local SR, so that it is not accidentally deleted

1)SSH to your server using putty

type the command xsconsole

In the console that is displayed go to Disks & Storage repositories -> Current Storage Repositories

You need to locate your current local repository and view the properties on the right pane

Note down the scsi ID

2)Type the following command

#ll /dev/disk/by-id

Note down which device the ID maps to. In this case it is sda3.So that is your present local SR

Also you need to note the SCSI id of the disk that you are planning to use for the new SR. In my case it is sdb

3)Next step is to create the new SR. You need run the following command and note down the "uuid"

#xe host-list

4) Create new SR using the following command

# xe sr-create content-type=user device-config:device=/dev/disk/by-id/<scsi-xxxxxxxxxxxxxxxxxxxxxxxxx> host-uuid=<host-uuid> name-label=”Local Storage 2” shared=false type=lvm

you need to replace the scsi id and host uuid

Now you should be able to view the new Local SR in the XenCenter!!

Friday, February 7, 2014

Install nested KVM in VMware ESXi 5.1

In this blog, I will explain the steps required to run a nested KVM hypervisor on  Vmware ESXi. The installation of KVM is done on Ubuntu 13.10(64 bit).

Note: It is assumed that you have already installed your Ubuntu 13.10 VM in ESXi, and hence we will not look into the Ubuntu installation part.

1) Upgrade VM Hardware version to 9.

In my ESXi server, the default VM hardware version was 8. So I had to shutdown my VM and upgrade the Hardware version to 9 to get the KVM hypervisor working. You can right click the VM and select the Upgrade hardware option to do this.

2)In the ESXi host In /etc/vmware edit the 'config' file and add the following setting

vhv.enable = "TRUE"

3)Edit the VM settings and go to VM settings > Options > CPU/MMU
Virtualization .
Select the Intel EPT option

4) Go to Options->CPUID mask> Advanced-> Level 1, add the following CPU mask level

ECX  ---- ---- ---- ---- ---- ---- --H- ----

5) Open the vmx file of the Ubuntu VM and add the following setting

monitor.virtual_exec = “hardware”
hypervisor.cpuid.v0 = “FALSE”

KVM installation steps

Once the above configurations are done in ESXi and the VM, hardware virtualization would be supported in the VM. Inorder to confirm that, run the following command in the Ubuntu VM.
egrep -c '(vmx|svm)' /proc/cpuinfo

You should get output as 1, which means hardware  virtualization is supported. Also you should see hvm flags in the output when you run the following command

cat /sys/hypervisor/properties/capabilities 
Or else you could also run the following command
The output should be as follows:
INFO: /dev/kvm exists

KVM acceleration can be used
 If the above command succeeds, proceed with the remaining steps below

1)Install KVM

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils

2)Add user to libvirtd group

$ sudo adduser <username> libvirtd
username should be the same as the admin user that you are logged in with(the one created during Ubuntu installation)

3)Set permissions

The sock file permission should be as follows
$ sudo ls -la /var/run/libvirt/libvirt-sock

srwxrwx--- 1 root libvirtd 0 2010-08-24 14:54 /var/run/libvirt/libvirt-sock

The /dev/kvm permission should be as follows

 $ ls -l /dev/kvm

crw-rw----+ 1 root libvirtd 10, 232 Jul  8 22:04 /dev/kvm

sometimes, the device group would be root, in that case change it to libvirtd

sudo chown root:libvirtd /dev/kvm

Log off and log in for the changes to take effect

4)Install GUI

In the default Ubuntu installation GUI is not installed. You could choose to install it followed by
if you plan to use the graphical user interface virt-manager to manage the VMs in KVM

First install the Ubuntu GUI

sudo apt-get install --no-install-recommends ubuntu-desktop

This will get a bare minimum installation of the GUI, that would serve our purpose of using Virt-manager

Reboot the machine after installation, and you can login to the GUI from the VM console

Now, Install the virt-manager using the following command

sudo apt-get install virt-manager

5)Access Virt-manager to create/manage VMs in KVM

Once you login to GUI, the virt manager can  be found by browing /Usr/share/applications  and select Virt-manager

If all turns out well, you should be able to see the Virt-manager connected to localhost(QEMU). Now you can start creating your VMs!!!

Monday, January 27, 2014

Timeout error during OpenStack installation

While trying to install OpenStack using RDO, following the instructions in the link here , you might face the following timeout error

Applying <IP address>_prescript.pp   [ ERROR ]

ERROR : Error appeared during Puppet run: <IP address>_prescript.pp
Error: Command exceeded timeout

The reason for this error, in all probabilities would be the delay in your network, which could be causing the netns component packages to be timed out You can increase the timeout by editing the following file


Add a timeout line in the file

    if $::netns_support != "true" {
        exec { "netns_dependecy_install":
            path => "/usr/bin/",
            command => "yum update -y kernel iputils iproute",
            timeout => 1900,

Now you can retry the installation. It is advisable to use the answer file that is created during the initial installation, otherwise the reinstall would fail

Sample command for doing the reinstall using answer file is

packstack --answer-file=/root/packstack-answers-20140125-195230.txt

replace the text file name with the name of your answer file that was created during the initial installation

Thursday, January 9, 2014

XenCenter troubleshooting

Issue 1:

While trying to add a new host to a Pool in XenCenter, I was getting error "internal connection failed no route to host "


It was quite simple actually, but since I couldnt find it mentioned in any knowledge bases or articles, I am noting it down here. The machine where XenCenter was installed couldnt resolve the Xen host name. The machines were not added to domains and hence no DNS entries were there. So I had to add a host entry to the XenCenter machine to help with the name resolution

Issue 2:

Enabling AD authentication via XenCenter fails with error "clock skew detected with active directory server"


This happens because of the time difference between XenServer and AD server


Update NTP server list of the XenServer and point it to the AD server of the domain. Steps below

From the XenServer console, go to 'Network and management interface"->Network time(NTP)

Select option "Remove all NTP servers" to remove the default NTP server configuration

Then select option "Add an NTP server"

Enter the name or IP address of the NTP server

This will update the NTP configuration. On the same menu, click on 'Enable NTP time synchronization" to complete the configuration


Wednesday, January 8, 2014

Xen learnings

This week, I was trying to get my head around a new Hypervisor(new for me, obiviously), ie XenServer. Though it is pretty much same as ESXi and is free as well, there are some striking differences as well.The observations are based on the free version of Citrix XenServer version 6.2.0

- While ESXi needs a paid vCenter to manage multiple hosts, you can use the free XenCenter software to manage multiple Xen hosts

- Latest version of Xen server doesnt have the equalent of DRS or DPM. There was a feature named workload balancing, which was strangely discontinued in version 6.2.0 stating reason that there are no takers for it.

- It does offer a High Availability of VMs using pool based clustering of hosts

-XenMotion is the equalent of VM live migration, but it is restricted to one VM at a time

-XenCenter doesnt have a web client like vCenter

-There was a tool named XenConvert used for physical to virtual conversion, but it is retired as well.

-There is an option named Dynamic Memory Control(DMC), which can be used for dynamic allocation of memory for VMs. We can set a maximum and minimum memory options for VMs which will be used by XenServer to manage memory crunch situations

-Thin provisioning is supported  for local storages only

-Distributed vSwitch controller appliance  is available for centralized management of networks in XenCenter. However, this too is being depreciated in v6.2.0

PS: One interesting point to note is that the configuration limits document of xenServer is very small when compared to VMware and it doesnt have much details mentioned. For example, VMware specified the maximum number of vCPUs that we can create per physical processor core. For v5.1 its 25 and for v5.5 its 32. However Xen doesnt give you a hardcoded value for that. When we contacted Xen support regarding the same, they mentioned that there is no limit!!! Obiviously, that means you have to keep performance in mind while deciding on the number of vCPUs

Here is a good comparison matrix between various hypervisors available in market

Tuesday, December 10, 2013

Windows Azure: Powershell script to update instance type in .csdef file

Here is a simple PowerShell script to change your azure instance type in .csdef file. You need to run this from your code repository and enter the instance type ie "small", ExtraSmall","Medium" etc when prompted

$allCsDefFiles = Get-ChildItem -Recurse -filter *.csdef | ForEach-Object -Process {$_.FullName}
$newvmsize = Read-Host 'Enter the instance type'

foreach ($thisCsDefFile in $allCsDefFiles)
    [xml]$thisCsDefXml = Get-Content $thisCsDefFile
$root = $thisCsDefXml.get_DocumentElement();

If (!$root.WebRole.vmsize)
{ Write-Host "No webrole found in $($ "
    $root.WebRole.vmsize = $newvmsize
Write-host "Webrole size of $($ changed to $($root.WebRole.vmsize)"
If (!$root.Workerrole.vmsize)
{ Write-Host "No Workerrole found in $($ "
$root.Workerrole.vmsize = $newvmsize
Write-host "Workerrole size of $($ changed to $($root.Workerrole.vmsize)"



Monday, December 2, 2013

Virtual fibre channel in Hyper V

Virtual fibre channel option in Hyper V allows the connection to pass through from physical  fibre channel HBA to virtual fibre channel HBA, and still have the flexibilities like live migration.


  • VM should be running Windows Server 2008, 2008 R2 or Windows Server 2012
  • Supported physical HBA with N_Port Virtualization(NPIV) enabled in the HBA. This can be enabled using any management utility provided by the SAN manufacturer.
  • If you need to enable live migration, each host should be having two physical HBAs and each HBA should have two World Wide Names(WWN). WWN is used to established connectivity to FC storage.When you perform migration, the second node can use the second WWN to connect to the storage and then the first node can release its connection. Thereby the storage connectivity is maintained during live migration
Configuring virtual fibre channel is a two step process

Step 1: Create a Virtual SAN in the Hyper-V host

First you need to click on Virtual SAN manager available on Hyper-V manager,

Select option to create a new Virtual fibre channel SAN , give a name and  select your available physical SAN HBAs .

 Thus the physical SANs are made available to the Virtual machines, but you will still need to add those SANs to your VMs when required . Single host can be connected to multiple SAN Volumes

Step 2: Add the Virtual fibre channel adapter to VM and connect to your SAN

Upto four virtual fibre channel adapters are possible on a VM, however you cannot add virtual fibre channel adapter when VM is switched on.

Right click on your VM-> Settings->New hardware and select Fibre channel adapter and click ok

Select the Virtual SAN that we created in Step 1 and click OK

Now start the VM and use the virtual HBA to connect to the physical SAN storage

What happens during Live migration?

Each Virtual HBA will have two sets of addresses to facilitate live migration.

Lets find out what exactly happens during a live migration.

Initially the SAN is connected to the VM, it will be using one of the World wide address A to connect to the SAN.

When we initiate the Live migration, it will start using the second set ie world wide set B.

Thus the  FC connectivity is maintained and once migration is completed connection is flicked over to the second set

This will ensure availability during Live migration

Image courtesy / Ref: &

Tuesday, November 26, 2013

Hyper V Server 2012 remote management from Windows 8

Now that we have had  a look at how to do the Installation and initial configuration of Hyper V Server 2012 in my previous blog post, lets start on the management part.

In this blog, I will explain how to manage your Hyper V installation from a Windows 8 machine

Remote management:

Since Hyper V Server 2012 server core machine, you may want to manage it remotely using the familiar GUIs and MMC consoles. You can do so remotely, but before that you need to set the firewall rules to allow that.In the command prompt windows of the server, get a poweshell prompt by typing in "powershell". Now you can execute the following powershell command

Enable-NetFirewallRule -DisplayGroup *

Note: I used this command since it is my test network, you may want to lockdown the firewall rules a bit if in case of production network

Inorder to connect to the Hyper V server using MMC from my PC, I had to run the following command in the PC command prompt

cmdkey /add:<ServerName> /user:<UserName> /pass:<password>

Servername - Used IP of the Hyper-v server
Username,Password -> Provided credentials of the Hyper-V


In this scenario, Hyper V was not member of Domain , by my PC was..hence by default when we try connecting to Hyper V server through MMC, it will try to connect using your domain credentials and you will get an error.

Managing from Windows 8 PC:

1) Hyper V management tools is available as a feature in Windows 8. You can install the same from "Turn Windows features on or off" windows

PS: You can also manage Hyper V from your windows 7 PC by installing the Remote administration tools pack . Somehow the installation was taking ages in my Windows 7 machine and hence I opted for Windows 8

2) Now you need to set the windows firewall rules in Windows 8 to allow access to the Hyper V server. In an elevated powershell window, run the following command
Enable-NetFirewallRule -DisplayGroup *

3) In my test scenario, both Hyper V and Windows 8 PC(hereby referred to as client machine) were not members of the domain. So if you want to manage the hypervisor from the client machine, you need to create a local admin account in Hyper V that matches your client admin credentials. You can do so by using option 3 in sconfig.cmd window

4) Now if you try conecting to the Hyper V Server from the Hyper V manager, you might get the
“Access denied. Unable to establish communication between Client and Server”. You will have to tweak the COM security permissions on your client to sort this out. This can be done from the DCOMCNFG MMC.

Open the console, go to Component Services > Computers > My Computer
Right click, select properties of "My Computer" -> COM Security Tab
Select "Edit Limits" on the Access permissions area

Scroll down to find the "Anonymous Login" group and ensure that "Remote access" is allowed

 5) You can set the Server name in the Hyper V server 2012 using the 2nd option in the sconfig.cmd window and use this server name in Hyper V manager to connect to the Hypervisor

Note: If you client machine is not in domain, you will need to add a host entry in the client host file to ensure that the name resolution happens

After this, you should be able to connect to the Hyper V Server from the  management console and create VMs !!!.. 

Hyper V Server 2012 installation on VMware Workstation 8

Having heard a lot about the latest free virtualization from Microsoft, Hyper-V server 2012 , I coudnt resist giving it a whirl.. After all, it is not daily that Microsoft comes out with "free" offerings ;)

Let us admit it..Ever since the advent of virtualization, we have few physical servers lying around.All of them have joined the virtualization bandwagon. My case was no different, so I decided to try out  Hyper-V server 2012 as a virtual machine in VMware Workstation 8 installed in my PC.

Installation preparation:

Few things to be taken care before you start the actual installation

1)Download the Hyper V Server 2012 ISO from Microsoft site:

2)VMware workstation 8 does not have Server 2012 in the Windows OS list.Hence you need to select the option "Windows Server 2008 R2 x64" when you create the Virtual machine

3) There is a small tweak to the Processor settings that should be done before starting the installation. Edit the Virtual machine settings->Processors and select the option "Virtualize Inter VT-x/EPT or AMD-V/RVI"

4) The last step is to tweak the vmx file of the VM and add the following setting

PS: The vmx file can be found in the installation directory of the VM,Go to VM settings-> Options->Genaral and refer to the Working directory setting on the right pane

All done now!! You can connect the downloaded ISO and start the installation..

Installation procedure:

It is pretty straight forward, screenshots below

1)Select the language,time & keyboard format

2)Accept the License agreement

 3)Now the installation will start

4)Once completed, you will get a prompt saying that the administrator password needs to be changed.

5)Set the administrator password and login!!

Now that you have logged in , you will be welcomed by two windows, One command prompt in a normal shade of black and another command prompt in a pretty shade of blue , called sconfig.cmd

As you guessed correctly, this is a stripped down server core edition of Windows Server 2012 with Hyper-V and hence there will not be any GUI. You need to do the initial configurations from the sconfig.cmd prompt

Initial Configuration:

First things first, lets get the network configured

1)Select option 8. It will show the current network connection settings.


2)If you already have a DHCP server in your network, you will automatically get the IP from it. However, it is always good to set a static IP from a management perspective. Inorder to set a static IP, select the Index number of the adapter. You will  get options to set the IP address, DNS server as well as to clear the DNS settings

3) While all was setup and done, I realized that I was unable to ping to the hyper-V server from any of the other machines in the network. However, the server was able to ping to other machines. Turned out that ping is not enabled by default, we need to enable it through the Renote management option in sconfig. Select option 4 to do this

4)You need to select the option number 3 in above menu  ie "Configure Server response to Ping" to enable ping to the machine

In my next blog post, I will explain how to manage your Hyper V server remotely..

Reference for Installation prep: This nice blog from Veeam