Tuesday 22 November 2016

Convert Digital Ocean Droplet to VMware VM

This guide explains a method for converting a Digital Ocean Droplet to a VMDK which can be used under VMware ESXi Hypervisor or other virtualization software. This process is one way. It is currently impossible to convert a VMDK to a Digital Ocean Droplet.


Requirements:
1.Root access to the Digital Ocean Droplet.
2.Password for Root on the Digital Ocean Droplet.
3.Destination Storage Location with SSH access enabled.
4.Destination Storage Location with 'qemu-utils' installed.
5.Favorite Live Linux Distribution (Procedure uses Ubuntu Desktop) using either an Existing Linux VM or a Live Boot Capable ISO


Procedure:
1.Log into the Digital Ocean Droplet.
2.Prepare the Droplet for Backup.
   a.Reset root's password if you do not already know it.
   b.Change to runlevel 1 if possible and enter root's password to enter maintenance mode. Otherwise skip to step 3.
       i.Stop the rsyslog, udev, and dbus daemons.
initctl --system stop rsyslog
initctl --system stop udev
initctl --system stop dbus

*It is normal to receive an error from initctl when stopping dbus
**The above examples assume the Droplet is using upstart. Debian snapshots may not be using upstart.
3.Use DD to zero out any deleted data on the partition, so that compression size of the backup is smaller
dd if=/dev/zero of=0bits bs=20M; rm 0bits
4.Use DD to byte copy the Digital Ocean partition, feeding it into gzip, and then transfer it over SSH to the Storage Location.
dd if=/dev/vda | gzip -1 - | ssh @ dd of=/storage/location/snapshot.image.gz
*While it is possible to create the backup image from a runlevel other than 1, it increases the risk of the backup image containing corrupt files and dirty data.
5.Extract the gzipped image. 
gunzip /storage/location/snapshot.image.gz  
*At this point, you could manually mount the dd image file with 'losetup /dev/loop[0-7] /storage/location/snapshot.image' and then mount /dev/loop[0-7] to the fs. This is useful for then extracting files from the images or modifying it in any way before converting it to a VMDK.
6.Convert the DD image to a Virtual Machine Disk (VMDK) with the 'qemu-img' utility.
qemu-img convert -O vmdk /storage/location/snapshot.image /storage/location/snapshot.vmdk 
*The purpose of converting this to a VMDK is to make it easily accessible from a VM. You can skip converting this to a VMDK and mount the DD image directly inside a VM (from the host's filesystem) by mapping it to a loopback device. 
7.Create a new Virtual Machine (VirtualBox, VMware, etc.) with a new virtual disk that is at least 2GB larger than the DD image size. 
*On the ESXi Shell, a new VMDK can be created with the 'vmkfstools' utility to avoid creating a new Virtual Machine. Assign the newly created VMDK to an existing Virtual Machine instead.
8.With both the Converted DD Image VMDK and the Freshly Created VMDK attached to a VM, boot to a Linux environment (verified with Ubuntu 13.10 Desktop Live Environment) to transfer the partition. 
For the purposes of this procedure, we will assume the following: 
a. /dev/sda Is the Converted DD Image VMDK 
b. /dev/sdb Is the Freshly Created VMDK which is at least 2GB bigger than /dev/sda 
c. We are using Ubuntu for the partition transfer.
9.Partition /dev/sdb 
a. /dev/sdb1 Maximum Partition Size - ext4
10.Using DD, transfer byte-for-byte /dev/sda to /dev/sdb1.
dd if=/dev/sda of=/dev/sdb1 conv=notrunc,noerror
11.Disconnect the Converted DD Image VMDK from the VM (/dev/sda)
12.Using FSCK, check the integrity of the newly copied partition.
fsck -t ext4 /dev/sdb1
13.Mount /dev/sdb1 to delete any remnant GRUB files from Digital Ocean. Unmount when finished.
mkdir /droplet 
mount /dev/sdb1 /droplet 
rm -rf /droplet/boot/grub 
umount /droplet
14.Install GRUB to /dev/sdb to make the new VMDK bootable.
*For Ubuntu, Boot-Repair (https://help.ubuntu.com/community/Boot-Repair) is a quick and easy way to install and repair a GRUB installation.
15.Reboot the Virtual Machine to the new VMDK, at this point, the Droplet should now boot!
a.For additional cleanup to make booting a little more sane perform the following:
echo -e "# Fixes boot message during VMware Startup \n # since VMware does not offer the same CPU access as DO \n blacklist i2c_piix4" >> /etc/modproble.d/blacklist.conf