Where to start with Linux RAID NAS for media storage/fault-tolerant backup?

20 posts / 0 new
Last post
Offline
Joined: 22 Jul 2011
Where to start with Linux RAID NAS for media storage/fault-tolerant backup?

I usually think these through before posting, so, sorry ...

In this forum, I just spec'd an HTPC. Now I need a NAS for 2 main purposes:

P1) contain my key files: Quicken/Office/scanned file cabinet backup, irreplaceable pictures/camcorder videos

P2) serve out all my entertainment content to my HTPC (well, really to the TV), no extenders or 2nd TVs for a while

P2.5) POSSIBLY perform HTPC-sourced transcoding, since the HTPC only has an Intel i3-2100

P1 files to not need to be fast but must be redundant/fault tolerant, so there's a VERY small chance to lose everything (masters are on the PCs, except I'm running out of room on the master for HD home videos).  Ability to pop out/replace a drive or 2 and store elsewhere for disaster recovery is also desired.

P2 has 2 sides to it:

a) recorded content: should play up to speed but also don't want to lose them (but life will go on if I do)

b) ripped content: should play up to speed but can re-rip if I lose them

Presently, I only have a WD double WorldBook white box to serve P1 files (sans scanned file cabinet), that I PAINstakingly upgraded over the holidays from 2x1TB to 2x2TB, mirrored for fault tolerance. It's getting low too fast (due to new cameras and HD movies of the 18mo old). The rest is still only an idea.

I'm not fond of paying Microsoft, am recently $100 lighter because of them, so I've been thinking this could be the one dedicated-Linux/FREE OS, implementation in my house, esp. after my ugly experience hacking Linux on the WorldBook. The limitations of this experience also led me to want to roll my own box for this.

So I've got some vague ideas forming, like: cores & disks being dedicated to separate arrays to serve these separate purposes, RAID levels, hot swapping ... all in one box.

 

With that background, the Q is: Where do I start?  Where should I begin my search for knowledge, hardware, software, architecture, etc. to pull this off?

Thanks for any starting pointers. - Spiff

Offline
Joined: 6 Jul 2011

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

mikinho's picture
Offline
Joined: 19 Mar 2007

A couple things:

1- What type of video content do you primary have\plan to have?

I'm asking the Core i3-2100 will do a better job of transcoding content assuming QuickSync supports it.

In general, you don't want o build a NAS for power.  An ideal NAS does three things very well in my opinion.  It is 1) Stable 2) Efficient 3) Flexible

2- Are you tied down toe Linux or would you be open to FreeBSD solutions as well?

3- Did you want suggestions for off-the-shelf products like the Synology, Netgair and Drobo?

4- If you do go DIY, do you have any existing hardware you would be re-using?

If I was doing a DIY NAS I would go w/ an Intel Atom D525 board that had dual Intel NICs for teaming.  To be specific the Supermicro X7SPA-HF-D525.  Low poer, 4GB RAM, good SATA controller, 6x SATA ports, IPMI (huge IMHO) and extremely important to me dual Intel NICs.  I use this board for my routers and a backup NAS.

Aaron Ledger's picture
Offline
Joined: 6 Aug 2010

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you're looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

Senior Editor | @swoon_

Offline
Joined: 22 Jul 2011

santanan wrote:

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

santanan wrote:

How much is your budget?  Are you looking to purchase a NAS that already has most, if not all, of the functions you want? Or, are you looking to build a NAS?

Sorry ... other issues. So much data. I'll take these 1 at a time.

Budget: I expected that question. Answer: Whatever it takes without overkill ... just basic fiscal responsibility. I know that's hard for some people to wrap their head around ... until you look at it from a 100% noob POV who has nothing to base a gut-feel upon. I need a basic ballpark.

e.g., I never knew that my budget for an HTPC was ~$700 until I analyzed a lot of build specs, found/read a good guide or 2, read a lot about each part, and basically took weeks to spec a build that should do what I wanted without overkill for the best prices I could find ... then have people make recommendations from that, I made changes, repriced ... and after all that, the prices came to the high $600s. So that was my budget. That final product (which must work with the NAS) is here:  http://www.missingremote.com/forums/quick-anything-wrong-my-1st-htpc-build#comment-36694

I'm in a similar position again. I've done some initial reading and need more of an experienced opinion and concept direction to start out. Like, I don't know if I need hardware RAID, firmware RAID, software RAID, etc.  Basically, I'd like to save the weeks of self-torture this time and see if someone can help me narrow the 1000s of directions and millions of pages/posts that I don't have time to read into some common, tried/true, current concepts and selections. Then I can research those things.

And "roll my own" above refers to my wanting to build it ... but that's just so I don't risk running into limitations like the ones I had simply upgrading a WD WorldBookII with non-WD drives. But if someone has experience with an "almost" system, and can adjust it without many limitations, and can articular how that's better, I'm certainly listening.

Offline
Joined: 22 Jul 2011

mikinho wrote:

1- What type of video content do you primary have\plan to have?

I'm asking the Core i3-2100 will do a better job of transcoding content assuming QuickSync supports it.

In general, you don't want o build a NAS for power.  An ideal NAS does three things very well in my opinion.  It is 1) Stable 2) Efficient 3) Flexible

1) Good points. Will have to lookup QuickSync.

I envision the video content being my ripped DVDs (and Blurays, as I accumulate them), direct DVR'd recordings from TV (think Ceton, later), PVR recordings from my present DVR and camcorders ... can't think of more (or what's possible). FYI, I'll probably want it to serve music, too ... when I get an AVR and speakers.

I guess I should clarify "transcoding" ... does that mean just making files smaller and/or cutting commercials? I'm not sure what else is possible in this realm, so I'm not really sure how to use this word yet.

And I'm surprised the hear the 2100 does so well ... but is that still true while it's recording other stuff? If I'm going to go for any automated transcoding, I just figured it should be on another box, and this NAS should be the only other box in the house that no one is trying to sit in front of and actually use ... or I'll have to build ANOTHER box (this $1K HDTV is getting expensive).

mikinho wrote:

2- Are you tied down toe Linux or would you be open to FreeBSD solutions as well?

I am open to anything right now. I just suggested Linux b/c I figured that's the common, user supported, home-use Unix now. It just has to serve the HTPC with WMC somehow. (I figure anything will take a backup from my other XP & Win7 PCs.)

mikinho wrote:

3- Did you want suggestions for off-the-shelf products like the Synology, Netgair and Drobo?

Have not researched these. Please refer to my recent post. If someone thinks that's better and can fulfill all the needs in the OP. (e.g., office/home backup, serve the HTPC, transcode?, easy disk pulls for offsite storage/disaster recovery/hot-swapping?)

mikinho wrote:

4- If you do go DIY, do you have any existing hardware you would be re-using?

Just a bunch of old PATA disks, so probably not (unless I can reuse them vs. chucking them). I've got those 2x 2TBs in my current WD NAS, but they will be needed to initially load this new NAS. Would love to use some old cases, too, vs. chucking.  I used to be all about rebuilding with old parts until the tech starting leaping too fast.

I suppose I could also retire my wife's Pentium E2160 PC (and spec her a new build), if that would work. She would like that. :^{D

mikinho wrote:

If I was doing a DIY NAS I would go w/ an Intel Atom D525 board that had dual Intel NICs for teaming.  To be specific the Supermicro X7SPA-HF-D525.  Low poer, 4GB RAM, good SATA controller, 6x SATA ports, IPMI (huge IMHO) and extremely important to me dual Intel NICs.  I use this board for my routers and a backup NAS.

Thank you so much for the specific recommendation. I can look into those things and perhaps other similars (never heard of IPMI ... will look).

Offline
Joined: 22 Jul 2011

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you're looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

I'm still considering, yes. Does that make things like the above (that I want) so much easier than a free option?

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

Though if it does what my HTPC now does, that makes me feel like it negates the $700 I just spent on the HTPC, reducing it to an extender and disc player, and I should have just bought a standalone Bluray player. Sad

Offline
Joined: 12 Jan 2010

SpacemanSpiff2000 wrote:

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

 

If you just have one HTPC I wouldn't bother with trying to put  the InfiniTV in some other box.  The only reasons to do that (IMO) would be if it doesn't fit in the HTPC or if for some reason you don't have a cable outlet near your HTPC.   Even if the InfiniTV physically resides in another computer, such as a WHS, all the recording still happens on the HTPC, but now the data from the card needs to come over your network so there can be issues that you won't have with a local tuner.

Aaron Ledger's picture
Offline
Joined: 6 Aug 2010

SpacemanSpiff2000 wrote:

And just to throw another option out there for you, have you considered Windows Home Server 2011? Yes, it would mean paying MS, but the price is now only $50-$60 and it can offer a lot more functionality such as taking care of your backup strategy, housing an InfiniTV (which I know you're looking to get) to share amongst multiple HTPCs, file server duties, remote access to all your PCs, audio and video serving and transcoding on the fly when you are away from home, etc.

I'm still considering, yes. Does that make things like the above (that I want) so much easier than a free option?

Also, I never considered putting the Ceton in the NAS, so far away from the TV, but (as it happens) a lot closer to the router. I just assumed it had to be in my HTPC right by the TV. Any other considerations to account for there?

Though if it does what my HTPC now does, that makes me feel like it negates the $700 I just spent on the HTPC, reducing it to an extender and disc player, and I should have just bought a standalone Bluray player. Sad

WHS2011 can act as a file server + more. If you want a RAID under WHS, you can do that. It also gives you benefits as I mentioned before. It should be fast considering it is essentially a special version of Server 2008 R2. If you are more comfortable configuring and working with Windows, then it probably could be considered easier to use.

The primary reason you might want to house the InfiniTV in a WHS box other than your HTPC being SFF (which I know your chassis is not) is because it will likely be always on and if you want to have multiple HTPCs accessing the tuners, you can configure that and let your HTPCs sleep without adding any additional delays to the startup process. It's certainly not a requirement to network the InfiniTV. It also doesn't make your HTPC an extender. Content is still recorded to the HTPC( s ).

Senior Editor | @swoon_

Aaron Ledger's picture
Offline
Joined: 6 Aug 2010

 

Regarding your question regarding QuickSync and transcoding:

QuickSync is dedicated logic in the Core i-series Sandy Bridge processors. It is specifically targeted towards transcoding content (decoding (which was already present in Clarkdale) + encoding). Basically, applications that target the QuickSync feature such as CyberLink Media Espresso and ArcSoft MediaConverter will be able to transcode video at a much faster rate than is possible with CPU encoding (or even other vendors' GPU encoding). In doing so, the system will also consume much less power performing the operation. Sounds great, right?

Well, the cost is that the transcoding quality will not be at the same level as what you might expect from something done with software + CPU. Personally, the only use I have for transcoding is if I am going to place the content on a small screen portable device so I think the tradeoffs of using QuickSync for that application are worthwhile.

Senior Editor | @swoon_

Offline
Joined: 6 Jul 2011

My personal suggestions is to get yourself a ReadyNAS.  It should meet most, if not all, of ur current needs.  I started with that and now i'm building an HTPC/NAS server.  Put that device on your network first and with time you will learn more of what you want.  And you can always sell it, which is what i'm doing, in order to grow.   We have to crawl before we walk.    

Offline
Joined: 22 Jul 2011

You all have been very helpful so far and I appreciate it. You've offered some pre-made solutions, some OS and hardware suggestions, and explained some concepts I may use, etc.  I will consider it all.

But I have a gap at this point that I want to fill by returning to the main priorities:

SpacemanSpiff2000 wrote:

P1) backup key files: Quicken/Office docs, scanned file cabinet backup, irreplaceable pictures/camcorder videos

musts: redundant/fault tolerant, easy disaster recov; speed is secondary

P2) serve entertainment to my HTPC

P2-a) recorded content (recording off the DVR or direct from TV, so prob 1080i max)

musts: plays well; fault tolerance is a nice to have (some are hard to re-obtain)

P2-b) ripped DVDs and BluRays

musts: plays well; fault tolerance is unnecessary

The area of help I'm looking for in this follow-up are things like:

Would you setup different RAIDs for the above 3 functions? I'll make this up: like a RAID5 for P1, a separate RAID3 for P2a, and a RAID0 for P2b? Because P1 simply cannot be risked, P2a needs to be faster but shouldn't be lost, and P2b just needs to be fast? And I still mean all in the same box here.

Would anyone actually do that or is that just crazy talk? If crazy=yes, then what? Would keeping you critical backups on the same fault-tolerant RAID as your networked BluRay content impede either function in some way? Or does it really not matter at all and they can be all in one?

Also, I've read about some ripping methods but most are for movie-only. I'm looking for whole disc (so I don't have to go find the discs to see extras I want). How might that affect my specs here?

I'm trying to visualize an appropriate architecture at this stage, so I know what I'm shopping for.

Offline
Joined: 6 Jul 2011

RAID config options:

Please keep in mind that RAID options are more of a cost and preference issue.  The options to create a RAID config are motherboard based RAID, software based RAID, and PCIe controller based RAID.  PCIe being the most expensive route is the best choice, and my personal preference.

The RAID functionality, that you're looking for can be done within the same box.  The only limitation you would have is the number of HDD you can install for the array.  This limitation is set by the motherboard and or the RAID controller(i prefer the LSI MegaRAID line).

As for the fault tolerance concern, you have the following options:

If bandwidth is of more concern, I strongly recommend creating a RAID 10 VD.  The sacrifice with the RAID 10 configuration is that you loose the additional HDD space you can have by creating a RAID 5 or 6.  You get the most bandwidth with a RAID 10 config.

If space is more important, I would recommed RAID 5(can handle the failure, and rebuilding, of 1 HDD) or RAID 6(can handle the failure, and rebuilding, of 2 HDD). These config will allow you to gain more space for the VD and only use one (or two, depending on RAID config) hard drive at the point of failure to rebuild. But you sacrifice speed.

Either configuration can be expensive, but RAID 10 is more expensive due to the need of the 1-for-1 HDD fault tolerance config.

Although I've been in the IT tech field for 20 yrs, I'm new to the Media Enthusiast arena, but in my adventure I have found there are many apps available for ripping cd/dvd media.  It really is just a matter of preference, I happend to like "Magic ISO" to create ISO images of the media.  And for creating MKV files i use MakeMKV, HandBrake and AnyDVD HD depending on my need.

As for the OS to make this happen, make sure you pick something you feel comfortable with and meets ur file sharing/streaming expectations.

Hope this helps thru ur quest.

mikinho's picture
Offline
Joined: 19 Mar 2007

While I do agree with the use of hardware RAID, most home users do not need RAID 1+0 (aka RAID 10 but less accurate).  The times where RAID 1+0 shine is if you need a high performance writes.  For a business, even small business, RAID 1+0 is a given for servers.  For home HTPC usage and media storage that typically is not a concern and if it (write performance) is then I would suggest a different setup entirely then.

Note: While RAID 1+0 is typically the fastest overall RAID level (RAID 0 is faster but not redundant so it is arguably not a RAID level and wasn't one of the original RAID levels) with 4 disks RAID 5 arrays can perform better than RAID 1 + 0 so if you are only looking at a 4 disk array the performance benefits can be moot.

My SAN is setup for RAID 1+0 and my media storage is RAID 6.  For NAS media storage I recommend RAID 6 over RAID 5.  In addition to being able to lose an additional disk, RAID 6 has a higher level of protection against block failures and controller errors by the extra parity over RAID 5.  RAID 6 has a lower impactful on disks when rebuilding. 

If you do build a DYI NAS I strongly recommend a discrete RAID hardware controller with a Battery Backup Unit (BBUs) in addition to a UPS.  I tend to go with 3ware\LSI.  Not having a BBU will kill performance and put you at a higher risk of data loss.

BTW, In terms of Magic ISO please see http://aluigi.altervista.org/misc/magiciso_gpl_violation.txt.  I highly recommend using ImgBurn + Virtual CloneDrive instead.

mikinho's picture
Offline
Joined: 19 Mar 2007
SpacemanSpiff2000 wrote:
Would you setup different RAIDs for the above 3 functions? I'll make this up: like a RAID5 for P1, a separate RAID3 for P2a, and a RAID0 for P2b? Because P1 simply cannot be risked, P2a needs to be faster but shouldn't be lost, and P2b just needs to be fast? And I still mean all in the same box here.
That all depends on how much you are looking to spend, how much data you have and how many disks you have\want.  You can easily spend several thousand if you look to implement multiple RAID configurations.
SpacemanSpiff2000 wrote:
Would anyone actually do that or is that just crazy talk? If crazy=yes, then what? Would keeping you critical backups on the same fault-tolerant RAID as your networked BluRay content impede either function in some way? Or does it really not matter at all and they can be all in one?
I have a 16-port RAID 6 for my media storage and 8-port RAID 1+0 for my SAN. And a RAID 1 backup for source code and documents.
SpacemanSpiff2000 wrote:
Also, I've read about some ripping methods but most are for movie-only. I'm looking for whole disc (so I don't have to go find the discs to see extras I want). How might that affect my specs here?
The easiest method is AnyDVD HD.  You are looking on average ~40 GB per Blu-ray and 8 GB per DVD.
Offline
Joined: 6 Jul 2011

Dude, at the end of the day, since you aren't sure what you need, just get an off-the-shelf product. At least until you understand the benefits of building your own. Remember you can always sell it.

What it really comes down to is Storage Capacit vs. Performance. There are many factors that come into play when deciding which route to take.  And remember the cost factor also. 

I choose best performance over anything else. Bandwidth is important to me since networks tend to grow and the performance is crucial. With RAID 10 I have the comfort of having performance and fault tolerence needs met, at the expense of storage capacity. But this is the most costly route to take. Besides, you can have multiple RAID configs.

I suggest you decide the storage capacity you need first and then go from there. Building your own HTPC/NAS can become rather expensive. Remember, at the end of the day, it's a computer and you can always change configs and/or upgrade if you need to. But as long as you have the foundation, the posibilities are endless. Start with something simple and modify as your needs expand/grow.

 

To simplify RAID 5/6 vs. RAID 10:

RAID 0 or RAID 1 are not options for your wants/needs/requirements, that is the reason i didn't bother mentioning them.

Raid 5(RAID 6 is similar but IS NOT always offered as an option with controllers):

RAID 5 is the preference for HOME use since you get more storage capacity and you get the fault tolerence needed.  This RAID option requires extral processing power to write parity information, which benchmarks show a performance decrease on writes.

The storage capacity is equal to the total of all drives minus 1.

RAID 1+0 (currently better known as RAID 10):

A Raid 10 VD performance is similar to its equivalent Raid 0 VD.  A minimum of 4 HDD is required.

Example: "2" HDD Raid 0 VD = "4" HDD Raid 10 VD

This RAID config will ALWAYS have the best read/write performance hands down.  Any benchmark will prove that. 

 

Offline
Joined: 22 Jul 2011

Thanks to everyone again. This is interesting to watch and learn.

Further clarifications:

VD = Virtual Disk? No one else is talking "VD" here (or elsewhere in my reading), so I'm curious how this comes into play.

Wants: I do know what I want ... in the end (see OP). Just trying to figure out how best to get there. The best advice so far here may be to start small but expandable and build from there. This is not my 1st build (it's my 3rd, but I also have a 20 yr history in IT and I am new to the entertainment media party).

Off-the-shelf: I am a bit resistant b/c I've kinda "been there done that" with a WD WorldBook w/ 2x1TB disks (RAID1) that I filled. Wanted to upgrade to 2x2TB Samsung disks in-hand. WD only allowed it to recognize WD HDDs ... until I hacked the Linux down to reading the code myself and found/commented out the 1 line where it detected the WD drives (and told it all HDDs were WD), AND figured out how to keep the mirrored drive from overwriting the hack. That (+ the 1TB copying) only took 3 weeks of my life around Christmas of doing little else. Never again. So, I'm a bit wary, but still will look @ ReadyNAS.

Performance: I'm not looking for "best" performance but "good enough" performance. P1 files (fault tolerant) are written overnight (don't care about speed) and are only read if files are lost or for local sharing (don't care again). P2 is all about serving movies, so I need the performance to serve a ripped Bluray w/ 5.1 sound at full speed with 0 lag and 0 dropped frames ... a little more is a buffer ... any more than that is overkill.

Performance Q: So what performance is good enough to serve BRs over the network? Or what will give me that performance?

If it matters, the network will be at least wireless-N (when I upgrade the router), up to gigabit later (when I figure out how to run cable from the office in my basement, horizontally thru a half-bath and along an external wall to the family room, which is also in my fully-finished basement ... worst case scenario there). Thanks once again!

Offline
Joined: 6 Jul 2011

Dude, you claim to have experience in IT and built several computers, but yet you know nothing.  I see you have posted several threads on building your first HTPC.  Before you keep wasting everyone's time, read the comments and suggestions everyone has been kind enough to post.  ALL the info you need is within them, repeated over and over.  GOOD LUCK! seems you're gonna need it.

Offline
Joined: 22 Jul 2011

santanan wrote:

Although I've been in the IT tech field for 20 yrs, I'm new to the Media Enthusiast arena, but in my adventure ...

Hmm. That describes me, too. Thought we had more in common. As I said, the best overall approach idea so far came from you (if not the technicals for "best" perf required). Oh, well.

My apologies for not being as far along or on the same adventure as you, or not coming from similar areas of IT, or that my questions do not conform to your way of thinking (or whatever has offended your sensibilities). This is a discussion, which has been interesting and informative until now. What is now clear is that you are not patient enough to assist a fellow Media Enthusiast noob. Someone who is would have simply linked or pointed out where a given question is already answered. If we are of similar age, I would have expected a similar level of maturity.

Aaron Ledger's picture
Offline
Joined: 6 Aug 2010

Folks, let's stay on topic here. There are no stupid questions and no apologies must be made. The goal of these forums is to share and exchange ideas. We all have different experiences and we all can stand to learn a thing or two no matter who we are. Sometimes we may not see things obvious to some which is understandable given the complex nature of technology.

Public Service Announcement over, carry on.

Senior Editor | @swoon_

Login or register to post comments
Website design by Yammm Software
Powered by Drupal