Where to start with Linux RAID NAS for media storage/fault-tolerant backup?

[SpacemanSpiff2000]

I usually think these through before posting, so, sorry …

In this forum, I just spec’d an HTPC. Now I need a NAS for 2 main purposes:

P1) contain my key files: Quicken/Office/scanned file cabinet backup, irreplaceable pictures/camcorder videos

P2) serve out all my entertainment content to my HTPC (well, really to the TV), no extenders or 2nd TVs for a while

P2.5) POSSIBLY perform HTPC-sourced transcoding, since the HTPC only has an Intel i3-2100

P1 files to not need to be fast but must be redundant/fault tolerant, so there’s a VERY small chance to lose everything (masters are on the PCs, except I’m running out of room on the master for HD home videos).  Ability to pop out/replace a drive or 2 and store elsewhere for disaster recovery is also desired.

P2 has 2 sides to it:

a) recorded content: should play up to speed but also don’t want to lose them (but life will go on if I do)

b) ripped content: should play up to speed but can re-rip if I lose them

Presently, I only have a WD double WorldBook white box to serve P1 files (sans scanned file cabinet), that I PAINstakingly upgraded over the holidays from 2x1TB to 2x2TB, mirrored for fault tolerance. It’s getting low too fast (due to new cameras and HD movies of the 18mo old). The rest is still only an idea.

I’m not fond of paying Microsoft, am recently $100 lighter because of them, so I’ve been thinking this could be the one dedicated-Linux/FREE OS, implementation in my house, esp. after my ugly experience hacking Linux on the WorldBook. The limitations of this experience also led me to want to roll my own box for this.

So I’ve got some vague ideas forming, like: cores & disks being dedicated to separate arrays to serve these separate purposes, RAID levels, hot swapping … all in one box.

 

With that background, the Q is: Where do I start?  Where should I begin my search for knowledge, hardware, software, architecture, etc. to pull this off?

Thanks for any starting pointers. – Spiff

[santanan]

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

[mikinho]

A couple things:

1- What type of video content do you primary haveplan to have?

I’m asking the Core i3-2100 will do a better job of transcoding content assuming QuickSync supports it.

In general, you don’t want o build a NAS for power.  An ideal NAS does three things very well in my opinion.  It is 1) Stable 2) Efficient 3) Flexible

2- Are you tied down toe Linux or would you be open to FreeBSD solutions as well?

3- Did you want suggestions for off-the-shelf products like the Synology, Netgair and Drobo?

4- If you do go DIY, do you have any existing hardware you would be re-using?

If I was doing a DIY NAS I would go w/ an Intel Atom D525 board that had dual Intel NICs for teaming.  To be specific the Supermicro X7SPA-HF-D525.  Low poer, 4GB RAM, good SATA controller, 6x SATA ports, IPMI (huge IMHO) and extremely important to me dual Intel NICs.  I use this board for my routers and a backup NAS.

[swoon]

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you’re looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

[SpacemanSpiff2000]

[quote=santanan]

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

[/quote]

[quote=santanan]

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

[/quote]Sorry … other issues. So much data. I’ll take these 1 at a time.

Budget: I expected that question. Answer: Whatever it takes without overkill … just basic fiscal responsibility. I know that’s hard for some people to wrap their head around … until you look at it from a 100% noob POV who has nothing to base a gut-feel upon. I need a basic ballpark.

e.g., I never knew that my budget for an HTPC was ~$700 until I analyzed a lot of build specs, found/read a good guide or 2, read a lot about each part, and basically took weeks to spec a build that should do what I wanted without overkill for the best prices I could find … then have people make recommendations from that, I made changes, repriced … and after all that, the prices came to the high $600s. So that was my budget. That final product (which must work with the NAS) is here:  http://www.missingremote.com/forums/quick-anything-wrong-my-1st-htpc-build#comment-36694

I’m in a similar position again. I’ve done some initial reading and need more of an experienced opinion and concept direction to start out. Like, I don’t know if I need hardware RAID, firmware RAID, software RAID, etc.  Basically, I’d like to save the weeks of self-torture this time and see if someone can help me narrow the 1000s of directions and millions of pages/posts that I don’t have time to read into some common, tried/true, current concepts and selections. Then I can research those things.

And “roll my own” above refers to my wanting to build it … but that’s just so I don’t risk running into limitations like the ones I had simply upgrading a WD WorldBookII with non-WD drives. But if someone has experience with an “almost” system, and can adjust it without many limitations, and can articular how that’s better, I’m certainly listening.

[SpacemanSpiff2000]

[quote=mikinho]

1- What type of video content do you primary haveplan to have?

I’m asking the Core i3-2100 will do a better job of transcoding content assuming QuickSync supports it.

In general, you don’t want o build a NAS for power.  An ideal NAS does three things very well in my opinion.  It is 1) Stable 2) Efficient 3) Flexible

[/quote]

1) Good points. Will have to lookup QuickSync.

I envision the video content being my ripped DVDs (and Blurays, as I accumulate them), direct DVR’d recordings from TV (think Ceton, later), PVR recordings from my present DVR and camcorders … can’t think of more (or what’s possible). FYI, I’ll probably want it to serve music, too … when I get an AVR and speakers.

I guess I should clarify “transcoding” … does that mean just making files smaller and/or cutting commercials? I’m not sure what else is possible in this realm, so I’m not really sure how to use this word yet.

And I’m surprised the hear the 2100 does so well … but is that still true while it’s recording other stuff? If I’m going to go for any automated transcoding, I just figured it should be on another box, and this NAS should be the only other box in the house that no one is trying to sit in front of and actually use … or I’ll have to build ANOTHER box (this $1K HDTV is getting expensive).

[quote=mikinho]

2- Are you tied down toe Linux or would you be open to FreeBSD solutions as well?

[/quote]

I am open to anything right now. I just suggested Linux b/c I figured that’s the common, user supported, home-use Unix now. It just has to serve the HTPC with WMC somehow. (I figure anything will take a backup from my other XP & Win7 PCs.)

[quote=mikinho]

3- Did you want suggestions for off-the-shelf products like the Synology, Netgair and Drobo?

[/quote]

Have not researched these. Please refer to my recent post. If someone thinks that’s better and can fulfill all the needs in the OP. (e.g., office/home backup, serve the HTPC, transcode?, easy disk pulls for offsite storage/disaster recovery/hot-swapping?)

[quote=mikinho]

4- If you do go DIY, do you have any existing hardware you would be re-using?

[/quote]

Just a bunch of old PATA disks, so probably not (unless I can reuse them vs. chucking them). I’ve got those 2x 2TBs in my current WD NAS, but they will be needed to initially load this new NAS. Would love to use some old cases, too, vs. chucking.  I used to be all about rebuilding with old parts until the tech starting leaping too fast.

I suppose I could also retire my wife’s Pentium E2160 PC (and spec her a new build), if that would work. She would like that. :^{D

[quote=mikinho]

If I was doing a DIY NAS I would go w/ an Intel Atom D525 board that had dual Intel NICs for teaming.  To be specific the Supermicro X7SPA-HF-D525.  Low poer, 4GB RAM, good SATA controller, 6x SATA ports, IPMI (huge IMHO) and extremely important to me dual Intel NICs.  I use this board for my routers and a backup NAS.

[/quote]

Thank you so much for the specific recommendation. I can look into those things and perhaps other similars (never heard of IPMI … will look).

[SpacemanSpiff2000]

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you’re looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

I’m still considering, yes. Does that make things like the above (that I want) so much easier than a free option?

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

Though if it does what my HTPC now does, that makes me feel like it negates the $700 I just spent on the HTPC, reducing it to an extender and disc player, and I should have just bought a standalone Bluray player. 🙁

[jrandeck]

[quote=SpacemanSpiff2000]

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

[/quote]

 

If you just have one HTPC I wouldn’t bother with trying to put  the InfiniTV in some other box.  The only reasons to do that (IMO) would be if it doesn’t fit in the HTPC or if for some reason you don’t have a cable outlet near your HTPC.   Even if the InfiniTV physically resides in another computer, such as a WHS, all the recording still happens on the HTPC, but now the data from the card needs to come over your network so there can be issues that you won’t have with a local tuner.

[swoon]

[quote=SpacemanSpiff2000]

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you’re looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

I’m still considering, yes. Does that make things like the above (that I want) so much easier than a free option?

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

Though if it does what my HTPC now does, that makes me feel like it negates the $700 I just spent on the HTPC, reducing it to an extender and disc player, and I should have just bought a standalone Bluray player. 🙁

[/quote]

WHS2011 can act as a file server + more. If you want a RAID under WHS, you can do that. It also gives you benefits as I mentioned before. It should be fast considering it is essentially a special version of Server 2008 R2. If you are more comfortable configuring and working with Windows, then it probably could be considered easier to use.

The primary reason you might want to house the InfiniTV in a WHS box other than your HTPC being SFF (which I know your chassis is not) is because it will likely be always on and if you want to have multiple HTPCs accessing the tuners, you can configure that and let your HTPCs sleep without adding any additional delays to the startup process. It’s certainly not a requirement to network the InfiniTV. It also doesn’t make your HTPC an extender. Content is still recorded to the HTPC( s ).

[swoon]

 

Regarding your question regarding QuickSync and transcoding:

QuickSync is dedicated logic in the Core i-series Sandy Bridge processors. It is specifically targeted towards transcoding content (decoding (which was already present in Clarkdale) + encoding). Basically, applications that target the QuickSync feature such as CyberLink Media Espresso and ArcSoft MediaConverter will be able to transcode video at a much faster rate than is possible with CPU encoding (or even other vendors’ GPU encoding). In doing so, the system will also consume much less power performing the operation. Sounds great, right?

Well, the cost is that the transcoding quality will not be at the same level as what you might expect from something done with software + CPU. Personally, the only use I have for transcoding is if I am going to place the content on a small screen portable device so I think the tradeoffs of using QuickSync for that application are worthwhile.

[santanan]

My personal suggestions is to get yourself a ReadyNAS.  It should meet most, if not all, of ur current needs.  I started with that and now i’m building an HTPC/NAS server.  Put that device on your network first and with time you will learn more of what you want.  And you can always sell it, which is what i’m doing, in order to grow.   We have to crawl before we walk.    

[SpacemanSpiff2000]

You all have been very helpful so far and I appreciate it. You’ve offered some pre-made solutions, some OS and hardware suggestions, and explained some concepts I may use, etc.  I will consider it all.

But I have a gap at this point that I want to fill by returning to the main priorities:

[quote=SpacemanSpiff2000]

P1) backup key files: Quicken/Office docs, scanned file cabinet backup, irreplaceable pictures/camcorder videos

musts: redundant/fault tolerant, easy disaster recov; speed is secondary

P2) serve entertainment to my HTPC

P2-a) recorded content (recording off the DVR or direct from TV, so prob 1080i max)

musts: plays well; fault tolerance is a nice to have (some are hard to re-obtain)

P2-b) ripped DVDs and BluRays

musts: plays well; fault tolerance is unnecessary

[/quote]

The area of help I’m looking for in this follow-up are things like:

Would you setup different RAIDs for the above 3 functions? I’ll make this up: like a RAID5 for P1, a separate RAID3 for P2a, and a RAID0 for P2b? Because P1 simply cannot be risked, P2a needs to be faster but shouldn’t be lost, and P2b just needs to be fast? And I still mean all in the same box here.

Would anyone actually do that or is that just crazy talk? If crazy=yes, then what? Would keeping you critical backups on the same fault-tolerant RAID as your networked BluRay content impede either function in some way? Or does it really not matter at all and they can be all in one?

Also, I’ve read about some ripping methods but most are for movie-only. I’m looking for whole disc (so I don’t have to go find the discs to see extras I want). How might that affect my specs here?

I’m trying to visualize an appropriate architecture at this stage, so I know what I’m shopping for.

[santanan]

RAID config options:

Please keep in mind that RAID options are more of a cost and preference issue.  The options to create a RAID config are motherboard based RAID, software based RAID, and PCIe controller based RAID.  PCIe being the most expensive route is the best choice, and my personal preference.

The RAID functionality, that you’re looking for can be done within the same box.  The only limitation you would have is the number of HDD you can install for the array.  This limitation is set by the motherboard and or the RAID controller(i prefer the LSI MegaRAID line).

As for the fault tolerance concern, you have the following options:

If bandwidth is of more concern, I strongly recommend creating a RAID 10 VD.  The sacrifice with the RAID 10 configuration is that you loose the additional HDD space you can have by creating a RAID 5 or 6.  You get the most bandwidth with a RAID 10 config.

If space is more important, I would recommed RAID 5(can handle the failure, and rebuilding, of 1 HDD) or RAID 6(can handle the failure, and rebuilding, of 2 HDD). These config will allow you to gain more space for the VD and only use one (or two, depending on RAID config) hard drive at the point of failure to rebuild. But you sacrifice speed.

Either configuration can be expensive, but RAID 10 is more expensive due to the need of the 1-for-1 HDD fault tolerance config.

Although I’ve been in the IT tech field for 20 yrs, I’m new to the Media Enthusiast arena, but in my adventure I have found there are many apps available for ripping cd/dvd media.  It really is just a matter of preference, I happend to like “Magic ISO” to create ISO images of the media.  And for creating MKV files i use MakeMKV, HandBrake and AnyDVD HD depending on my need.

As for the OS to make this happen, make sure you pick something you feel comfortable with and meets ur file sharing/streaming expectations.

Hope this helps thru ur quest.

[mikinho]

While I do agree with the use of hardware RAID, most home users do not need RAID 1+0 (aka RAID 10 but less accurate).  The times where RAID 1+0 shine is if you need a high performance writes.  For a business, even small business, RAID 1+0 is a given for servers.  For home HTPC usage and media storage that typically is not a concern and if it (write performance) is then I would suggest a different setup entirely then.

Note: While RAID 1+0 is typically the fastest overall RAID level (RAID 0 is faster but not redundant so it is arguably not a RAID level and wasn’t one of the original RAID levels) with 4 disks RAID 5 arrays can perform better than RAID 1 + 0 so if you are only looking at a 4 disk array the performance benefits can be moot.

My SAN is setup for RAID 1+0 and my media storage is RAID 6.  For NAS media storage I recommend RAID 6 over RAID 5.  In addition to being able to lose an additional disk, RAID 6 has a higher level of protection against block failures and controller errors by the extra parity over RAID 5.  RAID 6 has a lower impactful on disks when rebuilding. 

If you do build a DYI NAS I strongly recommend a discrete RAID hardware controller with a Battery Backup Unit (BBUs) in addition to a UPS.  I tend to go with 3wareLSI.  Not having a BBU will kill performance and put you at a higher risk of data loss.

BTW, In terms of Magic ISO please see http://aluigi.altervista.org/misc/magiciso_gpl_violation.txt.  I highly recommend using ImgBurn + Virtual CloneDrive instead.

[mikinho]

[quote=SpacemanSpiff2000]Would you setup different RAIDs for the above 3 functions? I’ll make this up: like a RAID5 for P1, a separate RAID3 for P2a, and a RAID0 for P2b? Because P1 simply cannot be risked, P2a needs to be faster but shouldn’t be lost, and P2b just needs to be fast? And I still mean all in the same box here.[/quote]

That all depends on how much you are looking to spend, how much data you have and how many disks you havewant.  You can easily spend several thousand if you look to implement multiple RAID configurations.

[quote=SpacemanSpiff2000]Would anyone actually do that or is that just crazy talk? If crazy=yes, then what? Would keeping you critical backups on the same fault-tolerant RAID as your networked BluRay content impede either function in some way? Or does it really not matter at all and they can be all in one?[/quote]

I have a 16-port RAID 6 for my media storage and 8-port RAID 1+0 for my SAN. And a RAID 1 backup for source code and documents.

[quote=SpacemanSpiff2000]Also, I’ve read about some ripping methods but most are for movie-only. I’m looking for whole disc (so I don’t have to go find the discs to see extras I want). How might that affect my specs here?[/quote]

The easiest method is AnyDVD HD.  You are looking on average ~40 GB per Blu-ray and 8 GB per DVD.

[Author]

Posts