Active TopicsActive Topics  Display List of Forum MembersMemberlist  Search The ForumSearch  HelpHelp
  RegisterRegister  LoginLogin
Projects
 Harmony : Projects
Subject Topic: PC Upgrade - Virtualisation Post ReplyPost New Topic
Author
Message << Prev Topic | Next Topic >>
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 09 December 2010 at 9:37pm | IP Logged Quote SteveM

Ok, in case anyone is in the slightest bit interested in the following, here is the issue I have.

I used to have a fully functional 1TB NAS drive from Buffalo (never again!!!! ), hence been very quiet on this forum recently (head, wall … bang!!! ). It decided to go south taking most of my data with it, only to find the back routine supplied was as much use as wet tissue paper.
After recovering the data almost literally bit by bit from the raid using some very nice software (UFS Explorer), I have all my data back, so I need a new NAS drive. Looking at options available most of the main providers either have poor functions or poor support and in one instance even pre-sales was terrible!!!

I have also a stand alone PC running the home automation (something called Harmony, don’t know what version though ), see thread on new name for Harmony).

I have come to the conclusion that the best NAS to run is going to be a product called FreeNAS, but it needs a PC to run it, so I will end up with two PC’s constantly on.

Idea !!! Why not virtualise them, so here is the game plan;

1)     Acquire VMware ESXi 4.1 (free …. like this idea already, so does wife)
2)     Build a decent size PC, triple core AMD 3.1Ghz processor on an ASUS motherboard with a 6 SATA onboard RAID controller (have managed to convince the wife this is cost effective, 450 ukp later)
3)     4 * 500gb SATA drives, with external eSATA disk drive docker (Icy Box, like it no screws to hold the drives they just stand vertical) for backup device running up to 3gbps, will use rsync on FreeNAS to backup.

Using Vmware you can load the host controller on the PC as it has its own operating system and manages via GUI based control panel, from this it will then manage all of the other operating systems you require and only limited by the processing power and memory. You can then segment the hardware per O/S or allow all O/Ss access to all hardware.

This will allow me to then only have 1 PC up and running and run multiple O/Ss underneath it ……………… good eh?

Well that is the theory, hardware now arrived so guess what I will be doing over the weekend …………….. swearing at hardware and software !!!!!

If anyone is interested I will report progress, otherwise wish me good luck.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 

Guest
Guest


Joined: 01 October 2003
Online Status: Online
Posts: 25
Posted: 09 December 2010 at 9:42pm | IP Logged Quote 

Is it finished yet
Back to Top View 's Profile Search for other posts by 
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 09 December 2010 at 11:26pm | IP Logged Quote Rob Iles

SteveM wrote:
If anyone is interested I will report progress, otherwise wish me good luck.


Can I go for both please?

Good Luck! And *please* report on progress!!!   

What happened to the Buffalo? (I ask, nervously eyeing the 2 terastations holding most of my worldly (digital) assets)?? I've 4 * 2TB in Raid5, giving 6TB usable, which is mirrored onto a duplicate setup - - so 8 * 2TB drives providing 6TB usable. Yes, it's quite an overhead, but so far, it's proven to be very reliable. I **DO** have all source code backed up off site (monthly trip to bank vault, weekly USB Drive to offsite location, Daily full backup to hosted server, and hourly differentials of subversion repository) - - belt and braces and string and another belt etc... Would still be "put out" if Buffalo's packed up :s


Also, VMware ESXi 4.1 - - from what you've written / how I've read it - - is that the "bare metal virtualisation" version? If so - wow - will definitely have to revisit that - - last time I ventured into that arena, it cost silly ££££s!!

So, Good luck, Keep us posted, and please tell me more about VMware ESXi!

R.
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 10 December 2010 at 9:36am | IP Logged Quote SteveM

Homer, if only you lived close to me ........ I could throw something heavy at you

Rob, not a problem was just not sure if people would be interested.

Buffalo - running RAID 5 across 4 disks, worked fine for 3 years (obviously expect disks to start failing in the period 3/5 years just due to wear and tear). When one disk did start to have sector issues the following happened (still not convinced that there were issues across two disks). The disks are partitioned into at least 4, one of those hold the kernel. Not sure exactly how it “spreads” this across the disks and if it actually does, can’t get this inform from Buffalo. However the kernel failed after I had shut the system down due to power work. (Interestingly 3 weeks previously I had upgraded the firmware, did a power cycle and all was ok), bottom line was that trying various methods I was unable to recover the kernel or reload the firmware. Buffalo’s final answer was “you will have to get a data recovery company to recover your disk”, not at 15000 ukp!!!

Bottom line was the technical support I got form them was extremely poor, as usual you end up speaking to a trained semi technical person who just flounders when the answer is not in his scripted page.

Sounds from what you have in place you should be ok, I could not afford to have another 1Tb NAS drive mirroring the live box, I also relied upon Buffalo’s backup software to copy correctly. I was extremely lucky that the main errors were on the kernel partition, I lost part of an outlook file but that was historical emails that I was able to recover form another quarterly “drag and drop” backup I do. The UFS software was great, I was able to rebuild the RAID array and recover 99.9% of the data.

VMware has moved on we use this software to test reverse engineer malware & viruses in a controlled environment. The following link is to the version of software that I intend to use and it’s free (well at least at present) VMware EXSI 4.1

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 10 December 2010 at 10:53pm | IP Logged Quote Rob Iles

Thanks for info.

Ouch - very disappointed with Buffalo "support". What's the UFS Software you refer to? (Sorry, I've been of sys-admin mode for far too long - - PC's / servers/storage are now (almost) commodities - black boxes - I've *really* lost touch with the tools of the trade (though I'm slowly getting back up to speed). I've been a code-monkey for sooo long now ;-)

VMware - great - I can see me "repurposing" a machine for that, and consolidating a couple of others onto it - - Cheers!

Rob
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 11 December 2010 at 10:39am | IP Logged Quote SteveM

Rob,

UFS Explorer is a piece of software (around 60 ukp) that allows you to rebuild RAID arrays and then browse the directories and files once it has rebuilt it. You can the restore all or some elements back to another disk. You obviously require additional hardware to mount the disks you had in the RAID, as outlined i bought these ICY Box docking stations and 2 two port eSATA controllers to mount 4 disk used in my array.

As I say it worked a treat cost be about 160 ukp in total and was able to recover 99.9% of the data, cheaper than the quotes from a data recovery company. The joy is i can re-use the ICY Box/eSATA controllers (at 3gbps'ish) device now as backup devices and use RSYNC on FreeNAS to backup.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 12 December 2010 at 7:53pm | IP Logged Quote Rob Iles

Looks very handy - definitely one to buy, but hope to never have to use in anger!

R.
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 05 January 2011 at 6:10pm | IP Logged Quote SteveM

Ok update for those that are interested…….

WORD OF WARNING … before you start this, especially buying hardware, check compatibility list …… guess who did not

I bought an ASUS motherboard with onboard Realtek network adapter, to cut a long story short ESXI does not like the onboard NIC, fortunately I was able to borrow an Intel PCIe NIC and now it is running. Further piece of info, my initial idea was to run a hardware RAID with 4 of the supplied SATA interfaces but it would appear that ESXI does not like loading itself onto a RAID setup, so I have had to install a 5th standalone SATA drive to run the ESXI hosting software.

Now this is functioning need to now build the separate operating systems in each of their own virtual environments 1) Windows XP for Harmony 2) Linux for FreeNAS, will keep you post on events.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 05 January 2011 at 9:42pm | IP Logged Quote Rob Iles

Ouch,

I'll be sure to double check the list first! By sheer co-incidence, I was planning on using a "spare" PC/Server to build an ESXI host over the weekend - - - But it, too, has an Asus mobo with Realtek Nic!!

Sorry to hear of your troubles, if it's any consolation, your experience and post have been of great help to me!

I'll keep y'all posted on how I get on (assuming I do actually get time to build it - - these plans have a habit of being replaced by other tasks).

Rob
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 05 January 2011 at 9:53pm | IP Logged Quote SteveM

Rob,

Suggest you have a couple of spare NICs handy, I was able to install the Intel NIC in one of the expansion slots. From what I read becuase it loads the host O/S across the Internet, the NIC is important, not sure if all Intel NICs are supported but just try, it does nto take long to build, the time consuming bit for me was creating the initial boot disk.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 05 January 2011 at 10:01pm | IP Logged Quote Rob Iles

Thanks for that Steve - all good stuff to know via someone else's experience! I didn't even know the host O/S was loaded from the cloud.......clearly I need to do a lot more research before tackling this one!

Rob
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 07 January 2011 at 5:40pm | IP Logged Quote SteveM

Ok part 2 and the learning experience goes on

Managed to get a virtual machine (VM) installed, select Windows XP 32 bit and allowed the system to select default options (big mistake!!!, huge)
Loaded O/S and the plethora of patches and service packs, day later ready to install Harmony

As per instructions in this forum deregistered old version from old PC, re-registered on new VM, it comes up and works

Made backup of old Harmony config, worked a treat

Restored backup, although I received an error message saying backup did not work everything appeared to be functioning, this is looking good ……


THEN


XP reports disk space low (default assigns 8gb), me thinks not a problem I setup the virtual disk (VD) to expand as required using a thin install as per the defaults ……

Do you think the disk space would expand, over my dead body!!!! After reading various web based comments on how to expand VD and then of course the Windows partition, took the view that the time spent trying to get this functioning better spent starting from scratch.

So, instead of this time accepting defaults to VM build for XP, I would be clever ……

It appears that if you define the VD as being an IDE, disk expansion is difficult and convoluted, so I will use SCSI instead (makes no real difference except for having the right drivers).   

Anyone trying to load XP and pressing F6 at install to load drivers (let alone find them in the first place) knows it’s not easy.

However, playing with the VM hardware settings, stumbled across the ability to connect a virtual floppy disk. If you then select this from an existing datstore, surprise surpise you get a folder with a number of pre-provided SCSI drivers.

So at present I am in the process of yet again starting the OS build from scratch, will keep you posted on events.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 08 January 2011 at 1:20am | IP Logged Quote Rob Iles

Thanks Steve,

It sounds a little like ESXi for free being "too good to be true" is
manifesting itself

The VD issue is frustrating, though maybe partially fault of
Microsoft....I'm not bashing MS - though I have had problems
resizing the OS partition/disk in other circumstances. In their
defence, it's not a routine activity for XP users.....

Sorry to be a pain - can I ask for more info on "datastore" - is that an
ESXi concept, similar to nas/san, or something else?

Having re-read this thread - I'm slightly concerned about ESXi
residing on a single disk.......if I'm not mistaken, you could end up in
the same situation as when the buffalo kernel disk went
bad....hopefully I'm missing something here!!

BTW, did you successfully uninstall Harmony before restarting the
rebuild? If not, you're going to run into Harmony licensing issues
soon, and I'd hate to be the cause of more grief!! Email me your
Harmony serial number, and I'll check all is well on the activation /
licensing front!

Thanks for the updated info; I've decided to put off my ESXi build
until you get yours working!! Not being lazy, just 'prioritizing'

Rob

P.s. "F6 for drivers", takes me back many many years, installing NT
Server on a Mylex DAC960 RAID card, with *12* **huge, 1 Gigabyte
Seagate Barracuda disks ** !! Several thousand pounds worth of
storage subsystem for 4GB usable (but for the time, immensely
quick) Raid 50 storage, with 'hot spares'! I'm typing this on a phone
with >16GB storage!!! Now I feel old! By Moore's law, shouldn't I be
*much* faster by now? ;-)


Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 08 January 2011 at 1:25pm | IP Logged Quote SteveM

Rob,

Let me firstly put this into context, at heart I am a technician, show me a techie who reads a manual that’s what the term RTFM was created for . Personally I prefer to learn from my own experiments/mistakes and reading others guidance, very last resort (after hours of trying) is RTFM.

From what I have seen and used of ESXi I like it, think the sysadmin needs to do some work.

I have now completed the Harmony virtual machine (VM), running Harmony on a 30gb VD (lesson learnt) and it all appears to be functioning well. I still got the same error when restoring the Harmony backup, not sure if you want to follow up on that?
I have managed to get the VM configured to a SCSI interface and it looks like I do now have the option on the VM hardware settings to expand the VD, but Windows partition maybe another thing. I believe however that recent versions of Windows do have this capability written in, but not XP.

Yes thx for the reminder I did deregister Harmony before the 2nd swap (not really, running multiple instances of Harmony managed to hack it, but I wont tell if you don’t ).

Datastores, this appears to be a label the vendor uses to identify an area for storing data i.e. I have 5 hard disks and they have been labelled datastore1 thru 5.

With respect to single disk, yes ideally I would have liked to run the host and guest O/Ss on a RAID array, but could not get the ASUS motherboard RAID recognised by ESXi (even though at BIOS level I had configured it, probably driver required), please refer to first paragraph, this may actually be possible.

However my main concern for RAID is part two of this project, use FreeNAS on a 2nd VM to store my 0.75Tb of data. As you outlined, I will in the short term have to find a way to backup the 5th disk which houses the host and guest O/Ss.

One slight issue I did have and could not easily find the solution, the BBSB controller. I could not remember how to get it to talk to a new IP address for the PC running the instance of Harmony, it kept polling the old address and I could not remember how to change this, so I cheated and configured the VM with the original IP address.

So at the end of the day I now appear to have a fully functional Harmony system running on a virtual device, NAS next. I may also in the future use ESXi to run a VM for CCTV.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 08 January 2011 at 10:54pm | IP Logged Quote Rob Iles

Hi,
Please forgive a short reply.

Re single ESXi disk - maybe pull it, and take a ghost image for easy
restoration (assuming content remains pretty static).

Re multiple Harmony installs - I'll not tell, but if the boss reads this,
expect an invoice.

BBS Controller - press the little button on the back of the unit to put
it into programming mode, then press "Controller : Harmony" in the
BBS Harmony component on the Harmony VM within 10 seconds.

I'm surprised at datastore 1-5 being visible to guest OS (unless I've
misunderstood). Does this mean ESXi doesn't provide any raid
functionality natively? I'd hoped that the guest OS's would each see a
single partition, constructed from the 5 disks....... having to run
FreeNas to use the disks "safely" sounds like an additional point of
failure.

I agree about hands on over RTFM! This thread has answered a lot of
questions, and opened up a lot more! Going to have to take the
plunge.

One more thought / suggestion re XP & hotfixes. Look into
"slipstreaming" - you end up with an ISO that contains all SP's &
hotfixes (up to the date at which you create it). Can even "pre-enter"
your license key so that you don't have to hunt for it. IIRC, "nlite" is
the software for the job.

Re cctv - does ESXi "share" hardware such as PCI capture cards?....or
more importantly, can a guest OS access such resources directly?

Cheers for an interesting side-thread. I'll be sure to post when I get
going on my build!

Rob
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 
SteveM
Senior Member
Senior Member
Avatar

Joined: 02 August 2010
Location: United Kingdom
Online Status: Offline
Posts: 184
Posted: 09 January 2011 at 9:59am | IP Logged Quote SteveM

Rob,

Let me just answer a couple of queries, not sure everyone else maybe interested in the continuation of this thread as its not directly related to Harmony.

Ghost image, yes that’s possible, also as the guest O/S is in affect a virtual disk/file(s) on the host disk, then just copying the file(s) would also provide a backup. Just on that point and your reference to slipstreaming, build a virgin XP build and patch, then copy the virtual disk/file(s) and you have a ready to go O/S which can be easily loaded just by creating a new guest and point it to the copied files, I may well explore that route down the line.

BBSB Controller – thx, remember that now, need to put that nugget of information somewhere, I may add it to the existing thread on the BBSB topic.

Datastores – by adding “hardware” by means of additional hard disks to the guest O/S you can share the physical disks to any guest O/S, you can also share virtual disks residing on those physical disks in the same manner, makes it very flexible for sharing data. With respect to CCTV, yes again as far as I can tell you can share physical hardware across multiple guest O/Ss i.e. I have two USB ports, these are shared, same as CD ROM/DVD. I am sure there will be some limitations somewhere i.e. can you have a CCTV card “sending” data to multiple O/Ss, in theory I believe you can.

With respect to the FreeNAS, I do not have a NAS system at present so using the FreeNAS software I will in effect be running another guest O/S which is in effect a NAS drive. This can then be shared by all members of my family as if it was a stand alone NAS, just using the same physical box saving money at the same time by consolidating multiple pieces of hardware/systems in to one physical box. As outlined previously the alternative would have been a stand alone NAS to share our data, which reminds me I must get the wife to delete some of the files, sure she does not need all that lot.

Eventually you could take this virtualisation to a situation where I merge all my existing PC’s (7 around the house) into on device and use smaller physical and footprint PC’s just to run the GUI to interface to the main system.

Steve
Back to Top View SteveM's Profile Search for other posts by SteveM
 
Rob Iles
Admin Group
Admin Group
Avatar

Joined: 10 August 2005
Location: United Kingdom
Online Status: Offline
Posts: 5377
Posted: 09 January 2011 at 11:51pm | IP Logged Quote Rob Iles

Steve,

No problem with Harmony Relevance - this is in the Projects section - so perfectly fine. Nobody is forced to read it, and I'm sure it will come in useful to other people - heck - it's worth it just for my info

I've decided to order hardware to give ESXi a go. Will post info as and when.

Please, do keep posting anything that may be of interest.

Rob
Back to Top View Rob Iles's Profile Search for other posts by Rob Iles Visit Rob Iles's Homepage
 

If you wish to post a reply to this topic you must first login
If you are not already registered you must first register

  Post ReplyPost New Topic
Printable version Printable version

Forum Jump
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot delete your posts in this forum
You cannot edit your posts in this forum
You cannot create polls in this forum
You cannot vote in polls in this forum