Storage on the bin

Linux: Replace Disk in MD

Mon, 28 Dec 2015 00:00:00 +0000

On my lab machines I use a pair of physical disks, configured in a RAID 1 with MD, with LVM on top of that. This gives me a lot of flexibility with full redundancy. However recently I had a failure of a drive.

Identify the Failure

Here is what MD was telling me after I identified and removed the drive from the chassis to send it off for an RMA, notice how each device only has one member (md0 has sda1, md1 has sda2), this does not make a good RAID 1. But this is easily resolved.

ZFS: Send and Receive

Tue, 24 Sep 2013 00:00:00 +0000

ZFS gives us the ability to move data with ZFS send and receive. This can be used to combine snapshots, create full or incremental backups, or replicate data between servers on your LAN or even over the WAN.

Local Backup with Send

This is very simple, simply perform a send of a snapshot then output it to the file, I have complicated it a bit to include a folder which is named off of the current date (as would be helpful with backups).

Linux-LVM: Resize Partition to Grow LVM Volume Group

Mon, 20 Aug 2012 00:00:00 +0000

Logical Volume Manager makes the dynamic expansion of file systems dead stupid simple. However there is a weakness, if you are using a partitioned file system as your Physical Volume (PV) then you will end up needing to expand the file system if you ever need to grow the actual storage. This can be avoided by using the actual physical device /dev/sda as the PV, however if this is the same volume where your /boot partition lives then you will have no choice but to partition it out since /boot cannot reside on a Logical Volume (LV). The other way you can avoid this is by simply creating a second partition on the same disk and adding it as a new PV to the existing Volume Group (VG). This is preferable as there is far less room for error than in this procedure, but I personally find that method a little more confusing to inherit then a straight forward single partition made an LV.

LVM: Using 2TB Volumes for Volume Groups

Mon, 28 May 2012 00:00:00 +0000

Occasionally you will have the need to use physical volumes which exceed the limits of the standard DOS partitioning scheme. When you run into this a quick fdisk -l will reveal that you will need a new way to partition your disks.

Fdisk will produce output similar to this…

 text
 
 
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

WARNING: The size of this disk is 2.3 TB (2290828181504 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).

Use Parted to Create a Partition Using the Whole Disk

Parted is able to handle large volumes which are not able to be modified with fdisk. Here we are going to launch parted, and then create the partition with a size of 1 -1 which means that it will use it all until it gets to the end. You can additionally specify the number in megabytes, so 1 1024 would start at 1 and go until 1024 giving you a 1GB partition.

Bash: Duplicate Logical Volume Configuration for a Volume Group

Mon, 30 Apr 2012 00:00:00 +0000

I recently ran into a situation where I needed to create a large number of Logical Volumes for a server replacement situation. I could have done some manual data input, but why not take the opportunity to quickly write a script to do the hard work for me?

For my situation this was a server that pulls data from other servers in order to optimize getting that data to tape. So we had 2 VGs, one for the system and one for the data we wanted to move. We planned on simply performing a rsync to move the data itself, but the volumes had to be pre-staged.

Adventures in ZFS: Configuring iSCSI Targets

Mon, 27 Feb 2012 00:00:00 +0000

The most critical thing about any storage implementation is flexibility. This means that you need to be able to adapt your solution to changing environmental considerations. The nice thing about a ZFS-based solution is that all of the building blocks are available. If you want to use it as a NAS device, you can use CIFS, NFS. If you want to provision Fibre Channel that is available too. Even some of the more sought after SAN features are included, replication, deduplication, thin-provisioning, and many more. Now in todays article we are going to discuss how you can go about setting up iSCSI targets. I personally am not a fan of iSCSI (I have been bitten by the hard knee too many times), however you really can’t argue with the flexibility that is provided by being able to quickly provision storage on already existing or readily available hardware. The best thing about this process is that it is really similar to provisioning fibre channel targets, so if you are familiar with this process (available here) then you will be able to quickly adapt to the iSCSI variation.

Linux-KVM: Converting Raw Disk Images to LVM Logical Volumes Redux

Mon, 07 Nov 2011 00:00:00 +0000

This really is an update to my previous article “Linux-KVM: Converting Raw Disk Images to LVM Logical Volumes“, but the process is significantly different and thus requires more explaining than a simple inline update would do it justice. Now the big flaw with my first article was that by using dd on the hypervisor level, we are opening ourselves up to unintentional data loss due to incompetence or even a very competent slip of the finger.

Adventures in ZFS: Configuring Fibre Channel Targets

Wed, 24 Aug 2011 00:00:00 +0000

In our environment ZFS has become a critical component of our Storage Infrastructure. We have been able to provision Fibre Channel storage from our ZFS file systems which give us the benefit of data integrity, deduplication, performance through use of the ARC, L2ARC, and ZIL as needed. Additionally the real benefit to ZFS is our storage can run on commodity hardware which reduces the TCO significantly over the major Storage vendors. Now administering ZFS is really quite simple, however getting Fibre Channel can be a bit more work. However after reading this you ought to be able to handle this with your eyes closed, well maybe not with your eyes closed. But you will be much better prepared.

Linux-KVM Management: Live Migration Without Shared Storage

Tue, 16 Aug 2011 00:00:00 +0000

Everyday I learn something new about Linux-KVM, this is why I love this platform. Today we are going to talk about Live Migration. The kicker? Shared storage not required. Now please keep in mind that just because shared storage is not required, doesn’t mean there is not a benefit to it. Shared storage will accelerate this process, since the data will not have to move. However virtualization is all about flexibility and so is Linux for that matter, so why shouldn’t we be able to make the choice? That said this process is not documented by the libvirt team, as such it sounds like an “undocumented” feature. Please use this at your own risk, and for heaven’s sake please test and understand thoroughly before using this in production.

Adventures in ZFS: Faulted Cache Devices

Mon, 15 Aug 2011 00:00:00 +0000

We recently had an issue with Crucial M4 Solid State Disks when using them with ZFS on Solaris 11 Express (snv_151a). Basically the disks were showing a whole bunch of write errors and had been “FAULTED” by ZFS. Now to make this problem even worse when we tried to zfs clear them it locked up my SSH session, as well as subsequent sessions, it would allow me to initiate subsequent sessions (and authenticate), however would not deliver me to the prompt. Additionally when I went onto the console if I ran a “zfs list, zpool list, or zpool status” it would lock up the shell every time. When I say lock up, I mean lock up, no CTRL + C or Z or anything. So since these devices were cache devices only this wasn’t the end of the world for us.

Bash: Using pv to Display Progress of dd

Tue, 09 Aug 2011 00:00:00 +0000

One of the biggest weaknesses of dd is that it has no way to display progress on its actions. You can send a signal to the process which will pause, display statistics, and resume the process however this takes up alot of your terminal screen if you are doing any sort of long running copy. Enter pv. Pv allows us to monitor the progress of data through a pipe.

Create A 1GB Test File

This is the file that we will use to peform the following tests.

So You Want to Learn ZFS Part One

Mon, 04 Apr 2011 00:00:00 +0000

Enterprise Storage is currently broken. IT departments big and small spend way too much money on completely ineffective storage solutions which only partially fulfil the companies storage need. All the while having to justify massive expenditures to the business which do not meet the need. Essentially IT departments everywhere have become storage sales reps, sure we work with vendors who provide us with quotes, but when the rubber meets the road it is the IT department which is selling the storage to the business. If any of you have ever been in this position as I have and literally felt sick to your stomach because you knew it was all wrong, then this article is for you. This article does not provide an Enterprise Storage cure-all. What it does provide is an interesting look at a solution which really revolutionises the entire Enterprise Storage model. More importantly it allows you the flexibility, features, and densities needed to make it far superior to any modern day Enterprise Storage solution. Also it is pretty darn cheap.

Linux-KVM: Migrating Hyper-V VHD Images to KVM

Wed, 23 Mar 2011 00:00:00 +0000

I have always been a big fan of Hyper-V, and was actually a Virtualization MVP 2 years in a row. However recently I haven’t been able to get past the really poor support for Linux on the Hyper-V platform. We run a large number of hypervisors in our international sales offices (15+) and have a need to run both Linux and Windows on them. This has necessitated our switch to Linux-KVM as their Windows support is far better than Hyper-V’s Linux support. As such with an upcoming trip planned to our European offices I have been planning exactly how our migration will look, of course the first iteration of this plan consisted of me deploying replacement VMs on the KVM stack then migrating to the VMs individually. Though this plan was lengthy and cumbersome it would work.

Linux-KVM: Managing Disk Images

Wed, 16 Mar 2011 00:00:00 +0000

Disk images have long been the traditional form of storage for virtualized environments, they are essentially containers in the form of a file on the host’s file system. These files can be either fully allocated or sparsely allocated at time of disk creation, a fully allocated 20GB disk image will take up 20GB of storage on the host file system, a sparsely allocated 20GB disk image will only take up as much storage on the host file system as has actually been written inside of the disk, so if you have a newly created sparse file the file will be just a few KB in size.

Linux-KVM: Converting Raw Disk Images to LVM Logical Volumes

Wed, 09 Mar 2011 00:00:00 +0000

Linux Logical Volume Manager provides a better method for managing your disks in the home and even more so in the datacenter. I briefly outlined some of the benefits afforded us by LVM2 in a previous article Linux LVM2: Flexible Local Storage Management. In this article I will provide you with some very simple steps and scenarios to migrate your existing KVM VMs to LVM2 Logical Volumes.

In this article I am going to make an assumption that your disk images are raw disk images and that you are not using qcow2, vhd, vdi, vmdk or any other disk image format. Additionally this process should work for a physical or RAID device as well, though I have not verified that.

Linux LVM2: Flexible Local Storage Management

Tue, 01 Mar 2011 00:00:00 +0000

Linux LVM provides a different way of looking at storage. On the server side this provides much more flexibility than standard disks with partitions. It allows us to migrate from old to new storage as well as expand the available storage to a particular file system easily (assuming that there is free space available).

Basically LVM abstracts the physical storage by creating Physical Volumes (PV), and each PV is divided into Physical Extents (PE) which are a consistent size. Physical Volumes are then added to a Volume Group (VG) which essentially pools all of the PE on all member PV. Once you have a VG you can then create a Logical Volume (LV) which can be formatted and used as if it were a physical disk, the difference is that the LV can be expanded as many times as needed up until all PE have been allocated within the VG. In this event you could then simply add another PV to the VG, which would make available additional PE to be added to a LV.