...making Linux just a little more fun!
Amit Kumar Saha [amitsaha.in at gmail.com]
Hello!
Does any one here have experience using a RAID Simulator or know of any? Basically, what I am looking for is 'emulating' RAID on a single computer, single hard disk.
Looking forward to some insights
Regards, Amit
-- Amit Kumar Saha Writer, Programmer, Researcher https://amitsaha.in.googlepages.com https://amitksaha.blogspot.com
Jim Jackson [jj at franjam.org.uk]
On Tue, 1 Jan 2008, Amit Kumar Saha wrote:
> Hello! > > Does any one here have experience using a RAID Simulator or know of > any? Basically, what I am looking for is 'emulating' RAID on a single > computer, single hard disk.
check out lvm - logical volume management. It doesn't do all RAID modes but does mirroring. There's an IBM developerworks article at
https://www.ibm.com/developerworks/linux/library/l-lvm/
which might be a bit long in the tooth nowadays, but is ok as an intro.
> > Looking forward to some insights > > Regards, > Amit > > -- > Amit Kumar Saha > Writer, Programmer, Researcher > https://amitsaha.in.googlepages.com > https://amitksaha.blogspot.com > > +-+--------------------------------------------------------------------+-+ > You've asked a question of The Answer Gang, so you've been sent the reply > directly as a courtesy. The TAG list has also been copied. Please send > all replies to tag@lists.linuxgazette.net, so that we can help our other > readers by publishing the exchange in our monthly Web magazine: > Linux Gazette (https://linuxgazette.net/) > +-+--------------------------------------------------------------------+-+ > _____________________________________________ > TAG mailing list > TAG@lists.linuxgazette.net > https://lists.linuxgazette.net/mailman/listinfo/tag >
Amit Kumar Saha [amitsaha.in at gmail.com]
On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote:
> > > > > On Tue, 1 Jan 2008, Amit Kumar Saha wrote: > > > Hello! > > > > Does any one here have experience using a RAID Simulator or know of > > any? Basically, what I am looking for is 'emulating' RAID on a single > > computer, single hard disk. > > check out lvm - logical volume management. > It doesn't do all RAID modes but does mirroring. > There's an IBM developerworks article at > > https://www.ibm.com/developerworks/linux/library/l-lvm/ > > which might be a bit long in the tooth nowadays, but is ok as an intro.
Its a good intro on LVM, but actually what I am looking at is simulating a RAID environment on a single Linux box.
Any suggestions?
Thanks, Amit
-- Amit Kumar Saha Writer, Programmer, Researcher https://amitsaha.in.googlepages.com https://amitksaha.blogspot.com
Jimmy ORegan [joregan at gmail.com]
On 01/01/2008, Amit Kumar Saha <amitsaha.in@gmail.com> wrote:
> Its a good intro on LVM, but actually what I am looking at is > simulating a RAID environment on a single Linux box. > > Any suggestions? >
Have you tried qemu?
Francis Daly [francis at daoine.org]
On Wed, Jan 02, 2008 at 12:11:21AM +0530, Amit Kumar Saha wrote:
> On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote:
Hi there,
> > check out lvm - logical volume management. > > It doesn't do all RAID modes but does mirroring. > > There's an IBM developerworks article at > > > > https://www.ibm.com/developerworks/linux/library/l-lvm/ > > > > which might be a bit long in the tooth nowadays, but is ok as an intro. > > Its a good intro on LVM, but actually what I am looking at is > simulating a RAID environment on a single Linux box.
I thought it was a useful answer, particularly if you follow the links to the (similarly old) articles on Linux software RAID.
The precise details may not be relevant to your specific distribution and kernel, but the fundamentals of "what is RAID, and why do I want it?" remain.
> Any suggestions?
What do you think you mean by "simulating a RAID environment"?
RAID == Redundant Array of Inexpensive Disks (or something similar).
From a kernel point of view, that means "multiple block devices", which can mean "different disks" or "different partitions on a disk" (or "different partitions on different disks").
So: decide what physical block devices you have available; decide which of the many different types of RAID you want to play with today; and find your distribution's utilities that probably begin "raid" or "md" (multiple disk, or maybe multiple device) to start playing.
Once you recognise that you can partition your /dev/sdb into /dev/sdb1 and /dev/sdb2, then what the RAID part does is to join /dev/sdb1 and /dev/sdb2 into a new logical entity /dev/md0. Outside the RAID config, you ignore /dev/sdb[12], and only use /dev/md0 as a single block device onto which you (typically) create a filesystem, trusting the lower-level code to mirror/stripe/whatever the bits the way you asked it to.
(Granted, using two partitions on a single disk may not be the cleverest for redundancy, resilience, or speed; but once you understand the concept, you are better placed to choose what partitions to use for what part of your simulation.)
LVM can be used to join multiple physical disks or partitions into a single "disk" (volume group); and can then split that again into "partitions" (logical volumes).
And then, if you really want to, you can use the md-utils on those logical volumes as if they were /dev/sdb1 and /dev/sdb2 in the earlier suggestion.
The disadvantage of doing all of that is that you'd have to learn both lvm and md. The advantages are that lvm is more convenient than fdisk in resizing your "partitions" (so you can start over more easily), and that lvm can possibly do what you were wanting to do with RAID without directly involving md at all.
And depending on your interpretation, it is possible that none of this is "simulating RAID", but is "real, live RAID", especially if you are using inexpensive disks in a redundant configuration.
Good luck,
f
-- Francis Daly francis@daoine.org
Jim Jackson [jj at franjam.org.uk]
On Wed, 2 Jan 2008, Amit Kumar Saha wrote:
> On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote: >> >> On Tue, 1 Jan 2008, Amit Kumar Saha wrote: >> >>> Hello! >>> >>> Does any one here have experience using a RAID Simulator or know of >>> any? Basically, what I am looking for is 'emulating' RAID on a single >>> computer, single hard disk. >> >> check out lvm - logical volume management. >> It doesn't do all RAID modes but does mirroring. >> There's an IBM developerworks article at >> >> https://www.ibm.com/developerworks/linux/library/l-lvm/ >> >> which might be a bit long in the tooth nowadays, but is ok as an intro. > > Its a good intro on LVM, but actually what I am looking at is > simulating a RAID environment on a single Linux box. > > Any suggestions?
yes. LVM. that's why I answered.
LVM doesn't care where the partitions are is managers. So on the one disk you can mirror (one of the raid modes) 2 partitions quite happily. You can stripe etc etc. Of course there will be a performance hit 'cso of the one disk.
Maybe I misunderstood what you meant by "simulating". If so, you'll have to be a lot more specific about what you want to do.
Peter Knaggs [peter.knaggs at gmail.com]
On Jan 1, 2008 7:55 AM, Amit Kumar Saha <amitsaha.in@gmail.com> wrote:
> Hello! > > Does any one here have experience using a RAID Simulator or know of > any? Basically, what I am looking for is 'emulating' RAID on a single > computer, single hard disk.
Hi Amit,
Well, let's say you want to use a single file from your filesystem, and make it available as multiple linux block devices?
From there, you could use whatever RAID software you like, feeding it these tasty new block devices. Sound tempting?
OK, so let's start with creating the single file in your filesystem. Let's make it blank, of size 1GB:
dd if=/dev/zero of=./somefile bs=1024k count=1024 losetup /dev/loop1 ./somefile fdisk /dev/loop1Here, let's create a few 100MB partitions, e.g.:
Disk /dev/loop1: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x68ed631d
Device Boot Start End Blocks Id System
/dev/loop1p1 1 10 80293+ 83 Linux
/dev/loop1p2 11 20 80325 83 Linux
/dev/loop1p3 21 30 80325 83 Linux
Now you can use the handy script by Christophe Saout
https://lkml.org/lkml/2003/7/18/141
(also attached to this email, I needed to change /bin/sh to /bin/bash
for it to work
on Ubuntu, apparently because /bin/sh is a symlink to a speedier stripped-down
shell used for booting called "dash"), to make each of these
"partitions" of your
file into a linux block device, using device-mapper:
dmsetup-partitions.sh /dev/loop1The output would be e.g.
loop1p1 63 160587 loop1p2 160650 160650 loop1p3 321300 160650and now you have these new block devices to play with:
/dev/mapper/loop1p1 /dev/mapper/loop1p2 /dev/mapper/loop1p3This method can also be handy for "looking inside" raw backups of flash memory sticks or other kinds of image files which have been hastily dumped using "dd", and have a partition table at the beginning.
For course, this answer might be completely unrelated to what you're looking for with your original question.
Cheers, Peter.
Amit Kumar Saha [amitsaha.in at gmail.com]
On 1/2/08, Jimmy O'Regan <joregan@gmail.com> wrote:
> On 01/01/2008, Amit Kumar Saha <amitsaha.in@gmail.com> wrote: > > Its a good intro on LVM, but actually what I am looking at is > > simulating a RAID environment on a single Linux box. > > > > Any suggestions? > > > > Have you tried qemu?
Are you telling that I simulate multiple computers using 'qemu' and then combine all their 'virtual HDDs' to form a RAID array.
Correct me, if I am wrong.
Thanks, Amit
-- Amit Kumar Saha Writer, Programmer, Researcher https://amitsaha.in.googlepages.com https://amitksaha.blogspot.com
Amit Kumar Saha [amitsaha.in at gmail.com]
On 1/2/08, Francis Daly <francis@daoine.org> wrote:
> And depending on your interpretation, it is possible that none of this is > "simulating RAID", but is "real, live RAID", especially if you are using > inexpensive disks in a redundant configuration.
"Real, live RAID"- it is.
Just to put things into better perspective and context, I am working on a project which requires the design of a tool for "Trend analysis of RAID Services.." which will enable better "observability" of systems in a large scale computing infrastructure which will have "live" RAID implemented to ensure more reliable data storage.
Now, since such an infrastructure is absent at my college, my project mentor suggested me that there are "RAID simulators" available which help emulate RAID devices. The simulated RAID environment will then be uses to emulate disk failures, and other dummy RAID operations to enable an "analysis" and hence the design of the application .
I think, i have put my query into better perspective now.
All comments, suggestions are welcome!
Thanks, Amit
-- Amit Kumar Saha Writer, Programmer, Researcher https://amitsaha.in.googlepages.com https://amitksaha.blogspot.com
René Pfeiffer [lynx at luchs.at]
On Jan 03, 2008 at 0937 +0530, Amit Kumar Saha appeared and said:
> On 1/2/08, Francis Daly <francis@daoine.org> wrote: > > And depending on your interpretation, it is possible that none of this is > > "simulating RAID", but is "real, live RAID", especially if you are using > > inexpensive disks in a redundant configuration. > > "Real, live RAID"- it is.
Real and live RAID will have different implementations, especially for RAID levels such as 4, 5 or 6. You have to be careful picking test systems. A colleague of mine is specialised in data recovery and told me that RAID5 controllers from different vendors have their own way of distributing the data among the disks. This might be an issue when it comes to comparing real and live RAID systems.
> [...] > Now, since such an infrastructure is absent at my college, my project > mentor suggested me that there are "RAID simulators" available which > help emulate RAID devices. The simulated RAID environment will then be > uses to emulate disk failures, and other dummy RAID operations to > enable an "analysis" and hence the design of the application.
The easiest way to do this is to use virtualised GNU/Linux systems with multiple virtual disk partition combined by the Linux software RAID. mdadm can simulate drive failures (https://tldp.org/HOWTO/Software-RAID-HOWTO-6.html). This setup misses the fun caused by stubborn hardware, but it's a start. I don't know if this is sufficient for your testing environment.
Best wishes, René.
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Thu, 03 Jan 2008, Amit Kumar Saha wrote:
> Just to put things into better perspective and context, I am working > on a project which requires the design of a tool for "Trend analysis > of RAID Services.." which will enable better "observability" of > systems in a large scale computing infrastructure which will have > "live" RAID implemented to ensure more reliable data storage. > > Now, since such an infrastructure is absent at my college, my project > mentor suggested me that there are "RAID simulators" available which > help emulate RAID devices. The simulated RAID environment will then be > uses to emulate disk failures, and other dummy RAID operations to > enable an "analysis" and hence the design of the application .
As suggested use loopback devices or "real" partitions to create pseudo-disks. You can emulate the failure of one or more disks in a number of different ways:
a. write directly to some blocks on the block device associated with one of the pseudo-disks. b. use blockdev to "suddenly" mark the blockdevice "read-only". c. use an external USB disk as one of the elements of the array and un-plug it during operation. d. use LVM to interleave a "zero/error" LVM device with actual read/write disk space. Use that as one of your pseudo-disk devices. e. suspend your array and replace one of the pseudo-devices with another such device whose data is almost but not entirely identical with the one before. Then resume the array.and other such innovative techniques!
Have fun,
Kapil. --
Amit Kumar Saha [amitsaha.in at gmail.com]
On 1/3/08, Ren? Pfeiffer <lynx@luchs.at> wrote:
> On Jan 03, 2008 at 0937 +0530, Amit Kumar Saha appeared and said: > > On 1/2/08, Francis Daly <francis@daoine.org> wrote: > > > And depending on your interpretation, it is possible that none of this is > > > "simulating RAID", but is "real, live RAID", especially if you are using > > > inexpensive disks in a redundant configuration. > > > > "Real, live RAID"- it is. > > Real and live RAID will have different implementations, especially for > RAID levels such as 4, 5 or 6. You have to be careful picking test > systems. A colleague of mine is specialised in data recovery and told me > that RAID5 controllers from different vendors have their own way of > distributing the data among the disks. This might be an issue when it > comes to comparing real and live RAID systems. > > > [...] > > Now, since such an infrastructure is absent at my college, my project > > mentor suggested me that there are "RAID simulators" available which > > help emulate RAID devices. The simulated RAID environment will then be > > uses to emulate disk failures, and other dummy RAID operations to > > enable an "analysis" and hence the design of the application. > > The easiest way to do this is to use virtualised GNU/Linux systems with > multiple virtual disk partition combined by the Linux software RAID. > mdadm can simulate drive failures > (https://tldp.org/HOWTO/Software-RAID-HOWTO-6.html). This setup misses > the fun caused by stubborn hardware, but it's a start. I don't know if > this is sufficient for your testing environment.
Thanks, Rene. For the moment, Virtualized GNU/Linux systems is possibly the way to start off my experiments. The Software RAID How-To will be a good starting point, and I plan to use 'qemu' for my works.
Regards, Amit
-- Amit Kumar Saha *NetBeans Community Docs Coordinator* Writer, Programmer, Researcher https://amitsaha.in.googlepages.com https://amitksaha.blogspot.com