Data Server Upgrade

[torrentking]

Hey guys and girls

This is kinda related to the server segments on the show but this is more for home use so odviously I don't have heeps of money to spent.

My current setup is P4 3.0Ghz, 1GB ram, 1TB( 2x500GB) in RAID 0 (software raid) running under Ubuntu. It stores pretty much everything I have and its full :( It is setup with uTorrent with wine (windows emulation) that has time scheuleing for downloads and looks at rss feeds for my latest TV shows and downloads it for me. I am sharing all this content over my gigabit network (majority of computers are macs).

I want to upgrade to something more powerfull and much more space. Call me crazy but I am thinking of getting the these stuff

Let me know what you think for an OS and any hardware changes that you would make. cheers.

Note: All prices in AUD.

[Sparda]

I don't think you need any thing more powerful, more efficient would be nice from the less electricity side. P4 at 3GHz is crap for a desktop, for a computer that doesn't do much that's fine. You probably don't need 1GB RAM either, 512MB would do.

RAID0? Unless you don't care about the data on there that's not particularly good. You should consider getting either two 2TB drives and putting them in RAID1 or 3 1TB drives and putting them in RAID5. Each will result with 2TB of relatively secure storage, the two 2TB drives would be more efficient in terms of power usage, but the three 1TB are probably cheaper initially.

[torrentking]

I don't think you need any thing more powerful, more efficient would be nice from the less electricity side. P4 at 3GHz is crap for a desktop, for a computer that doesn't do much that's fine. You probably don't need 1GB RAM either, 512MB would do.

RAID0? Unless you don't care about the data on there that's not particularly good. You should consider getting either two 2TB drives and putting them in RAID1 or 3 1TB drives and putting them in RAID5. Each will result with 2TB of relatively secure storage, the two 2TB drives would be more efficient in terms of power usage, but the three 1TB are probably cheaper initially.

Oh k cool. I suppose keeping my current setup would be cheaper.

Yeah I should have mentioned that I don't want RAID 0 because there is no data redundancy. My brother did that. I am thinking of a hardware RAID 5 with 3 or more 1TB drives. If I decide to buy more 1TB drives can i add them to the raid or to I need to rebuild the raid again and loose the data?

[Ryan]

I don't think you need any thing more powerful, more efficient would be nice from the less electricity side. P4 at 3GHz is crap for a desktop, for a computer that doesn't do much that's fine. You probably don't need 1GB RAM either, 512MB would do.

RAID0? Unless you don't care about the data on there that's not particularly good. You should consider getting either two 2TB drives and putting them in RAID1 or 3 1TB drives and putting them in RAID5. Each will result with 2TB of relatively secure storage, the two 2TB drives would be more efficient in terms of power usage, but the three 1TB are probably cheaper initially.

i agree that stuff is over kill for you purposes you would be wasting you money IMHO i think you should consider extending you storage for sure maybe get a couple of 1TB drives and use them externally as additional storage maybe move the old data on the externals for an archive and use the internals for everyday use. that's essentially what i do with my whs system. i have about 1TB of internal storage that i use to run my nightly diffs and to stream content to my 360 and other computers and i have a RAID array running on usb and i use sync toy every couple off weeks to move my video and stuff like that over to the RAID to keep the internal storage free. a bit inefficient i know but i haven't had the time to implement an automatic solution. btw that server is an old p D 820 running 2.8 and 1.5 gigs of RAM. even with every computer streaming at the same time it never breaks a sweat i use maybe 50% cpu at the max

[Sparda]

If I decide to buy more 1TB drives can i add them to the raid or to I need to rebuild the raid again and loose the data?

In theory yes, but you'll need to resize the partition on the drive to make use of the space (arguably a high risk operation). It maybe the case that if you opt for a hardware RAID that when adding to the array you have no choice but to brake the raid then create a new one with the new drive added, which may result in the drives been reinitialised thus loosing all the data on them. If a controller is able to add drives to an array without loosing data it will likely be listed as one of it's features. Should be able to add disks to the raid with software raid.

The disadvantage with software raid been that ideally you have an additional drive (or SSD) to run the OS off.

[torrentking]

i agree that stuff is over kill for you purposes you would be wasting you money IMHO i think you should consider extending you storage for sure maybe get a couple of 1TB drives and use them externally as additional storage maybe move the old data on the externals for an archive and use the internals for everyday use. that's essentially what i do with my whs system. i have about 1TB of internal storage that i use to run my nightly diffs and to stream content to my 360 and other computers and i have a RAID array running on usb and i use sync toy every couple off weeks to move my video and stuff like that over to the RAID to keep the internal storage free. a bit inefficient i know but i haven't had the time to implement an automatic solution. btw that server is an old p D 820 running 2.8 and 1.5 gigs of RAM. even with every computer streaming at the same time it never breaks a sweat i use maybe 50% cpu at the max

Okay I guess I will keep my current CPU, RAM and motherboard. I will just look into getting a RAID 5 card and Drives... I was thinking of just getting the 8 drives so i won't need to buy more later and therefore have to rebuild the raid... Alternatively I could get a 4 port RAID card and buy another one later when i need it. I like the USB backup storage Idea. I might do that also when i get everything cleaned up

[h3%5kr3w]

BTW, just to add, you are looking at the WD Caviar drives? You might want to look either at the WD 1T Green Drives *better efficiency* or the WD 1T Black Drives *better performance*.

And I also agree that your going a lil' overkill with that. Why do you want so much space btw? You should invest in some hot-swap bays as well, and I would go with a large case w/ @ least 5 large front bays to accomodate the hot-swap bays.

[torrentking]

BTW, just to add, you are looking at the WD Caviar drives? You might want to look either at the WD 1T Green Drives *better efficiency* or the WD 1T Black Drives *better performance*.

And I also agree that your going a lil' overkill with that. Why do you want so much space btw? You should invest in some hot-swap bays as well, and I would go with a large case w/ @ least 5 large front bays to accomodate the hot-swap bays.

Oh woops, yeah the black drives are the better ones. I was reading that... I don't know why i chose that Caviar ones.

lol yeah I might just look for a 4 port raid card. and get another raid card when I run out of space... I am the 'torrent king' after all ahha i need my space. I am downloading about 200GB/month and I thought I might as well just get enough space to last a while. But now that I come to think of it I might just get a 4 port card because most of the drives would be on and not needed. dw I have a case.... It is a rack mountable server case.... it has 6 large front bays... I was thinking of looking into getting hot swap bays for them, that would be kinda cool.

[Swathe]

My mate has a coolermaster stackewr with 50 HDD and 22.5 TB of storage space.

I can link to the build log if there is any interest ( or non-beleivers lol)

[torrentking]

My mate has a coolermaster stackewr with 50 HDD and 22.5 TB of storage space.

I can link to the build log if there is any interest ( or non-beleivers lol)

shit thats sweet... yes do link. I am very interested in seeing this setup... The more storage space the better i always say

[BieRHeDD]

I want to upgrade to something more powerfull and much more space. Call me crazy but I am thinking of getting the these stuff

Let me know what you think for an OS and any hardware changes that you would make. cheers.

Note: All prices in AUD.

Hi, I did a similar upgrade last week. I use the server to also run a few virtual machines (vmware server 2) as well as SMB and NFS. Let me start by saying that if you use NFS for your fileshares that the GA-EX58-UD5 uses two Realtek 8111D chips for its onboard dual LAN. And the linux kernel doesn't play well with this chip currently (apparently a fix is in store for kernel 2.6.30 but that is cutting edge and even when i compiled it myself (rc8) it still didn't quite work.

The problem was the motherboard basically does a hard reset randomly during a heavy load NFS<-->RAID operation. Sometimes I could get 10gig+ transferred before it would reboot but other times much smaller.This is obviously not ideal when you're talking multi gigabyte raid volumes that need to resync / fsck after each random reboot. This problem had me trying different distros (with different kernels) for a full week but each one ended up the same.

In the end i relegated the i7 to be my desktop machine, and now iuse my E8500/8gig/ASUS-P5E-VM-HDMI box as the server which has worked flawlessly using ubuntu 8.04.2 server (no gui).

My advice to you would be to use your current machine if you only want it for filesharing. If you have a small case, buy a big one. Check out Lian-Li PC-A77 or PC-343. (I got the A77) then as you have on your list a 4 or 8 port hardware RAID card that is capable of SAS and using expanders. I was looking at an Adaptec ASA1405 (up to 128 drives with expanders). Of course check your current motherboard has a PCIe 4x or above slot free to house the card.

Bottom line is unless you want to use Windows then the currently available 1366 motherboards all seem to have realtek 8111 LANs. (well the good ones) So you may run into the same problems as I had.

Good luck with it.

Cheers,

BieRHeDD

[torrentking]

Hi, I did a similar upgrade last week. I use the server to also run a few virtual machines (vmware server 2) as well as SMB and NFS. Let me start by saying that if you use NFS for your fileshares that the GA-EX58-UD5 uses two Realtek 8111D chips for its onboard dual LAN. And the linux kernel doesn't play well with this chip currently (apparently a fix is in store for kernel 2.6.30 but that is cutting edge and even when i compiled it myself (rc8) it still didn't quite work.

The problem was the motherboard basically does a hard reset randomly during a heavy load NFS<-->RAID operation. Sometimes I could get 10gig+ transferred before it would reboot but other times much smaller.This is obviously not ideal when you're talking multi gigabyte raid volumes that need to resync / fsck after each random reboot. This problem had me trying different distros (with different kernels) for a full week but each one ended up the same.

In the end i relegated the i7 to be my desktop machine, and now iuse my E8500/8gig/ASUS-P5E-VM-HDMI box as the server which has worked flawlessly using ubuntu 8.04.2 server (no gui).

My advice to you would be to use your current machine if you only want it for filesharing. If you have a small case, buy a big one. Check out Lian-Li PC-A77 or PC-343. (I got the A77) then as you have on your list a 4 or 8 port hardware RAID card that is capable of SAS and using expanders. I was looking at an Adaptec ASA1405 (up to 128 drives with expanders). Of course check your current motherboard has a PCIe 4x or above slot free to house the card.

Bottom line is unless you want to use Windows then the currently available 1366 motherboards all seem to have realtek 8111 LANs. (well the good ones) So you may run into the same problems as I had.

Good luck with it.

Cheers,

BieRHeDD

Thanks for the heads up BieRHeDD...thats anoying aye.... I have decided to keep my hardware and just get the drives, a new PSU and a RAID card. I will most likely load up Ubuntu server 8.04 server and install gnome coz i feal that i will have more controll with that over no gui. Nice case btw... i already have a Antec 4U22ATX Case which has enough room for me... i am going to get a 5 bay hot swap for it... idn what i will do when i need more drives, i may be able to take out the fan on the other side of the case and get another 5 slots out of it so i can have 8 HDDS in there in a RAID 5 config. i am not sure if i want to put them inside the case coz its a bitch to get out and open coz it weight >22KG :P I don't think i will ever go over 8 HDDs... its only for home use and I am 19... I don't know what i will do in the future so I am thinking that 8 Ports might be good enough... what do u think?

[BieRHeDD]

Thanks for the heads up BieRHeDD...thats anoying aye.... I have decided to keep my hardware and just get the drives, a new PSU and a RAID card. I will most likely load up Ubuntu server 8.04 server and install gnome...

etc

Looks like a good setup. I use the WD 1TB blacks in my setup and they are awesome (6x drives, 4TB raid5 with one hot spare) I'll probably grow it to 5TB and lose the hot spare when i need the space.

I know you're getting the RAID card so this may not be relevant but just in case you're interested i've only ever used linux software raid and usually setup four partitions on each drive:

100MB /boot (/boot2, /boot3 etc for remaining drives each one set as bootable and grubbed after install)

2GB swap per drive

20GB linux autodetect software raid

remaining (ie 900+GB) linux autodetect software raid

all the 20GB partitions get put into a raid1 or raid5 for the OS (ie /)

all the 900+GB partitions get put into a raid 5 or raid 6 volume for my storage (ie. /mnt/storage)

this way i don't need a separate drive for the OS, and if a drive falls off the array it's easy enough to boot up off one of the "backup" /boot partitions to get mdadm into action to remove/add a freshly partitioned drive. Add to this fast swap space across multiple drives if your RAM fills up. It's worked very well for me and I move files at about 80MB/s between machines on the network using NFS.

Of course the hardware raid card should be awesome and allow hotswap for failed drives but check if it has linux management software to make the most of it's features, a lot of them i've only seen windows software(drivers?) mentioned in their blurb but like i said i haven't used one so i could be completely in the dark about this.

If you want to use GNOME just download the *alternate* version of the ubuntu install .iso. (alternate lets you play with software raid during the install process). If you're going to use gnome i can't see any advantage to downloading the *server* .iso as its designed for no gui.

Anyway it sounds like you've got it under control. I don't want this to sound like a lecture, i'm just annoyed after a week of trying to get that GA-EX58-UD5 to work how i want and failing. May as well give you forewarning since you mentioned this exact board :D. Until the kernel catches up with these 8111's i'm using this massive case and an i7 with 12gig of ram for my desktop.. pretty stupid hey.

Cheers,

BieRHeDD

[torrentking]

Looks like a good setup. I use the WD 1TB blacks in my setup and they are awesome (6x drives, 4TB raid5 with one hot spare) I'll probably grow it to 5TB and lose the hot spare when i need the space.

I know you're getting the RAID card so this may not be relevant but just in case you're interested i've only ever used ...

Yeah It should be good. oh right yeah thats a good idea... have a spare 1TB in case one dies and you need to get the raid up quick.

ohh right the software RAID, cool... I think i will do a hardware RAID... the hightpoint cards now have Ubuntu support with their software so i should be good with that...i hope :P

If i have issues i will ask you how u did the software raid. Nice speed man... 80MB/s... is hardware RAID faster?

So yeah there is software and driver support for linux and drivers for Ubuntu.

Yeah I might not use Ubuntu server if i am gonna use GNOME.

Something just clicked man... I have a P4 and my mother board doesn't have PCI Express support... what should I do? they don't have PCI express motherboards for P4 do they? what do you recommend i get?

[miT]

It is setup with uTorrent with wine (windows emulation) that has time scheuleing for downloads and looks at rss feeds for my latest TV shows and downloads it for me.

One word: Torrentflux

I can't believe you're running an ubuntu torrent server with uTorrent + Wine. I cant even imagine whats that like, lol

[torrentking]

One word: Torrentflux

I can't believe you're running an ubuntu torrent server with uTorrent + Wine. I cant even imagine whats that like, lol

lol yeah i suppose i didn't know that linux had a torrent client like torrent flux and i certainly didn't know they had azurues on linux... lol idn how i went past that one :P yeah ubuntu with uTorrent & Wine is alright.... the odd lock up iof the OS every month or so but other than that i really didn't see an issue. Now that i have time and money... i want to fix that mistake :)

[miT]

lol yeah i suppose i didn't know that linux had a torrent client like torrent flux and i certainly didn't know they had azurues on linux... lol idn how i went past that one :P yeah ubuntu with uTorrent & Wine is alright.... the odd lock up iof the OS every month or so but other than that i really didn't see an issue. Now that i have time and money... i want to fix that mistake :)

I couldn't recommend Torrentflux any more than i already have.

I don't know if you use newsgroups at all, but if you do, check out SABnzbd. It's the same concept of a web interface with everything working in the background (like torrentflux) and it handles everything for you (par repair, unrar and even clean up of files that are no longer needed).

On top of those, i run museeqd to download music from the soulseek servers (p2p). You use a client app (museek+) to connect to your daemon and queue up all your downloads. Works wonderfully.

This trifecta on my linux server makes me very VERY happy.

[torrentking]

I couldn't recommend Torrentflux any more than i already have.

I don't know if you use newsgroups at all, but if you do, check out SABnzbd. It's the same concept of a web interface with everything working in the background (like torrentflux) and it handles everything for you (par repair, unrar and even clean up of files that are no longer needed).

On top of those, i run museeqd to download music from the soulseek servers (p2p). You use a client app (museek+) to connect to your daemon and queue up all your downloads. Works wonderfully.

This trifecta on my linux server makes me very VERY happy.

K i just installed torrent flux on my VM image of Ubuntu 8.04 (testing image)

i have a number of questions about getting it to work propperly but I should be studying for exams so i will ask them later... One thing.. whats the point of torrentflux? it needs a torrent client as well right? and can it scheduale downloads say like at midnight till 9am then stop downloading?

nah haven't used newsgroups... ah right sweet... sounds good... looks loke a good holiday project for me all this stuff :)

[BieRHeDD]

I couldn't recommend Torrentflux any more than i already have.

I don't know if you use newsgroups at all, but if you do, check out SABnzbd. It's the same concept of a web interface with everything working in the background (like torrentflux) and it handles everything for you (par repair, unrar and even clean up of files that are no longer needed).

etc

I was thinking wine+utorrent sounded pretty horrible but didnt want to start a client war :P. I've used azureus (Vuze?) up until now. It runs well on ubuntu and can be installed as easy as #apt-get install vuze. I'll try out torrentflux though, especially if it runs headless with an html interface :)

SABnzbd also sounds good. I don't have a big usenet quota so haven't bothered doing much research into the best client. I'll check it out too.

Now gotta go watch s5e17!

[miT]

K i just installed torrent flux on my VM image of Ubuntu 8.04 (testing image)

i have a number of questions about getting it to work propperly but I should be studying for exams so i will ask them later... One thing.. whats the point of torrentflux? it needs a torrent client as well right? and can it scheduale downloads say like at midnight till 9am then stop downloading?

The point of torrentflux is to have a torrent interface that you can access from anywhere. From your LAN, from your WAN and even from your iPhone (they have a kickass iphone theme for iphone support).

You can schedule it when you want it to start/stop or when you want it to start throttling the DL/UL speeds. You can also setup multiple user accounts so you can share your fibre connection with some buddies :) After the torrent is done, they can download the finished torrent directly from the torrent flux interface, its pretty slick :f34r:

It does not need a client to connect, it's all managed via the web front end interface (see quoted reply below)

Did you confuse torrentflux with museeqd that does require a client to connect to?

If you have any questions or need help with anything, feel free to drop by our IRC chat @ http://www.omgirc.com/chat.php

I was thinking wine+utorrent sounded pretty horrible but didnt want to start a client war :P. I've used azureus (Vuze?) up until now. It runs well on ubuntu and can be installed as easy as #apt-get install vuze. I'll try out torrentflux though, especially if it runs headless with an html interface :)

Torreflux is the same! sudo apt-get install torrentflux

Then browse to http://host/torrentflux

Totally runs headless (thats why its great for servers)

;)

SABnzbd also sounds good. I don't have a big usenet quota so haven't bothered doing much research into the best client. I'll check it out too.

I bought the boys over at OMGIRC a platinum giganews account with SSL support, so its safe to say that our newsgroups usage has gone up

[torrentking]

Fair enough I understand it now... sounds pretty cool... although I am able to RDP to my server and download torrents this will be better :)

haha fibre connection :P I wish i had a fibre connection... bloody Australia internet is hopeless at the moment.

When i go to the administration settings it has an option about default bitTorrent client... whats that all about. What do u have it set to miT?

sure I will most likely be on the IRC after the exams.

So is Newsgroup realy good? Does it have everything there and fast speeds?

[BieRHeDD]

So is Newsgroup realy good? Does it have everything there and fast speeds?

Usenet doesn't get much attention in Oz as far as the media goes. It's p2p that has the attention of a certain Federal Minister. It would be good to keep it that way so he doesn't get some half assed idea about filtering it.

As far as the user experience goes torrents and usenet are very similar (with a good client app). Usenet uses .nzb files instead of .torrent files. And usenet "binaries" are stored on centralised servers instead of the whole p2p web type structure where leechers also act as seeders. As far as using usenet, you basically pay in $$ instead of seeding in MB. Pay $$ to the server providers (eg giganews) for different levels of access.

A few ISPs here do provide usenet access standard with their DSL plans. Internode and Netspace that i know of, probably more. Of course you can just use any ISP and buy your own giganews access. A high quota tpg plan and a big giganews account would be hard (maybe impossible) to beat here as far as bang for buck goes. So bottom line is your paying for access, can download generally at full speed (depending on your ISP and the whole state of their network, tpg run "at their networks limit" let's say), and there is no upstream(seeding) as its not p2p.

Any more information than that and this post could be taken by some people as promoting copyright infringement. Which it isn't. So let google do the rest for you. Whatever you may want to download. Linux .ISOs i assume ;)

[torrentking]

hey your from Australia as well BieRHeDD :)

Yeah true they don't aye. haha yeah bloody filtering idea is crap.

oh right wow i learned something about usenet and .nzb files

Yeah I have TPG 200GB plan, its sick... thats why i need the new HDD's... run out of space too quickly :P esp when my ADSL 2 lets me download at 2MB/s max :) constant 2MB/s for everything would be sweet.

Have you got a giganews account or something similar to that?

[BieRHeDD]

Have you got a giganews account or something similar to that?

One of my connections is with internode so i've had a bit of usenet experience through that. It's only a low quota plan though because it's not primarily for downloading stuff. They basically provide i dunno what you'd call it.. maybe "proxy/peering" access to giganews servers in the US.

With TPG you'd need to buy your own account with giganews/astraweb/rapidshare/useneXt .. etc etc etc. There're probably a million of them to choose from.

[torrentking]

One of my connections is with internode so i've had a bit of usenet experience through that. It's only a low quota plan though because it's not primarily for downloading stuff. They basically provide i dunno what you'd call it.. maybe "proxy/peering" access to giganews servers in the US.

With TPG you'd need to buy your own account with giganews/astraweb/rapidshare/useneXt .. etc etc etc. There're probably a million of them to choose from.

sweet... yeah I will have to think about which account to get.