Cheap, low power Linux fileserver

[barry99705]

They also make liquid electrical tape. Stuff is awesome. Not very undoable, but for this it would be better than tape. http://www.amazon.com/Permatex-85120-Liquid-Electrical-Tape/dp/B003ERU04W/ref=pd_sim_sg_4?ie=UTF8&refRID=0QVGSDNP4BS8E0A7X1SE I've used the crap out of this stuff to seal phone crimps outside. They say the crimps are water tight, but I've had them corrode due to weather.

[cooper]

Cool stuff. I've never seen anything like it. First similar thing that pops to mind would be to simply incase it with hot glue (which I also don't have).

Back to the SATA cable, I tried prying the original cable from the clamps on the connector but quickly stopped when the inside metal strip came out along with the wire so I guess this means snapping off the current thing and soldering on the new one... Yay...

So next stop will be to rough up the wires using sanding paper, tinning it and then, using a folded chunk of aluminium (yes America, that's what it's called) have something to keep the solder from flowing too much and possibly touching the other wires. I hope that sounded more confident than I actually am about this sort of thing.

Edited February 11, 2015 by Cooper

[cooper]

Darren's HakAcrossEurope road trip took a pit stop in Amsterdam 2 weeks ago, to top up its supplies of fermented malts, hops and other goodness that city has on offer, and, like quite a few others, I decided to drive up to join in the good times.
Met Darren & Seb, and I'd love to mention the names of the other people there, but as it turned out, very few of the people present actually even visited the forums, let alone have an account here. I showed this little project to some of the folks there and mentioned at some point that I was going to mod that SATA cable since it needed to be so much shorter than normal. It was then that new forum member Killergeek mentioned he had a stack of short SATA cables from those Intel NUC barebones machines which might be useful. We exchanged details and not long thereafter I got a package in the mail with 2 14cm long SATA cables.



Nice!

This saturday I finished up the electrics (there's one finicky job I hope to not have to repeat anytime soon) and today I put together the keystone jacks for the network. Here's the finished product:


Full size

You'll notice the little gray/brown line going across that blue SATA cable. Since it needed to make that 90 degree bend I used a highly sophisticated industrial device known to the more technical people amongst us as "a piece of string".
The other thing you might notice is that between the harddisks there are a number of USB plugs. The reason for that is that, in spite of my 1337 electrical taping skillz, I failed to account for the sheer size of those plugs. They're so long that, even when inserted into the nano with the SATA cable, it touches against the harddisk. Rather than trying to force things, I decided to just not attach the cables. The other nano does have one of them attached, the other being needed for the extra network connection.

Since I decided against making another 19" rack, I removed the handles from the side of this chassis since making this picture, put the lid on and... Well, I would've liked to say that it's contently purring in the corner of the closet, but it's not even doing that. It's dead-quiet, even when accessing files on the harddisks.

I still need to construct the metal plate into which the keystone jacks would snap neatly as currently the rear of this chassis is still wide open and a quick tug on a network cable to simply pull all the innards right out, but that'll be something for another day. For now, I'm calling this project a success. I'll be messing about with the software side of things some more since the issues there haven't been resolved yet, but suspect a kernel upgrade, moving on to mainline, will end up fixing those issues.

Thanks to everybody for the views, the helpful comments and suggestions, and Killergeek for the contribution of the SATA cable!
Next up... clusters!

Edited March 1, 2015 by Cooper

[killergeek]

Nice build :) love the bleu sata cables :P
for the usb there are hubs with small connectors like this:

[cooper]

I'll see if I can find some cables/connectors with that plug someplace cheap *cough*china*cough* and some of that heatshrink tube stuff so I can do things properly.

Somethiing like this perhaps, with the added bonus of giving me 2 male connectors rather than just 1, but the plug itself still looks large-ish.

For now though, I'm battling the software side once more, trying to get mostly-mainline linux on there (genuine mainline doesn't support the gigabit adapter it seems).

Edited March 2, 2015 by Cooper

[cooper]

Another update... Guess I just can't help myself.

I initially reported problems with the network connection to PcDuino3 Nano, whereby it every so often would just drop the connection, forcing me to use a tool like screen to run stuff from so I can reconnect to it from another session if/when the current one locked up. It turns out that's a known problem for which subsequent fixes have been provided. To quote the relevant part:

set the GMAC_TX_DELAY parameter to 3. This adjusts the relative timing of the clock and data signals to the PHY in order to compensate for differing trace lengths on the PCB

The result of not having this parameter set is that you get *LOTS* of dropped packets when the interface negotiated at GBit speed. Once that's compensated for, things are rock-solid.

On with my problems trying to rebuild packages. Every so often the build would either break or stall which was annoying. Turns out, I only have myself to blame for this. When I rebuilt my kernel, I deactivated THUMB support. An ARM chip has 2 operating modes - arm and thumb. When it's in thumb mode, it's using a reduced instruction set allowing it to express operations with fewer bytes. Generally speaking, when you compile your program or library to a thumb binary, it's anywhere between 15 and 25% smaller than when compiled as regular arm binary. It's also, again generally speaking, about 5% slower. Note that ARM chips can switch between these modes very quickly and easily.

Since I knew for a fact I wouldn't be building thumb binaries on this system (I've got all the diskspace I could want and then some, so I'd much rather have that 5% of extra performance) I also disabled support for it in the kernel. However it turns out certain programs provide hand-optimised ASM implementations of routines that, as it happens, run considerably faster in thumb mode, mainly due to the fact that the smaller code allows you to fit more of it into the processor cache. Since I disabled support for this at the kernel level, whenever a context switch occurs, which is frequent when you run 3 concurrent build jobs on a 2-core board, and the executing code placed the processor in thumb mode, things go very, very wrong.

It took a lot of trial and error, compiling kernels to try and compiling glibc as proof that it did or didn't work. But luckily the culprit was found (me), the setting returned to its former glory with in the kernel config and this, too, has now been running perfectly.

One thing to note is that in my quest for solving the memory issue I've moved from the heavily patched 3.4 kernel on these boards to 4.0-rc1. Since I did this work on the Gateway board I don't know how the port multiplication stuff works just yet. At least one factor here is going to be that the file to modify doesn't exist anymore in this kernel, but I've got 5 instances for the flag to remove so it shouldn't be a lengthy process to figure out which, if any, I should flip for this. Hell, it might just work out of the box. Stranger things have happened....

Remaining things to work on here:

1. SATA PMP on the fileserver board.

2. Network performance tuning - I got about 12 MB/s reading data from an SD card that's capable of 16 MB/s. When I moved the file to a ramdisk which itself is capable of 130 MB/s I got 46 MB/s throughput. That, while decent, still leaves quite a bit of room for improvement.

3. Storage performance tuning - I need to install Samba, get a bit of a feel for what I can expect and then see if there's room for improvement here aswell. I have a few settings I know already I'm going to want to play with. The relevance here is that I want the gateway board to be a torrent client that downloads directly onto a mounted samba share on the fileserver board. To do that with any semblance of performance, I need that throughput both at the network and the storage level.

4. Setup the network such that internet goes through the gateway box. Currently everything, including my router, is connected to my switch. I want to change the router to bridge mode, have it connect to the 100 mbit USB thing on my gateway board, have it decide what to route and what not to route, and have the network access the internet through it, aided in part by a simple squid proxy that would cache a fair chunk of the traffic, again improving the overall performance of the rest of the network.

5. Some guy's been working on figuring out how to improve the memory performance of these nano boards. He's managed to get things quite stable at 648 MHz with 1.325v where the original is 408 MHz with 1.250v. I'm quite interested to see if that'll work for me, and to what extent this would improve performance. I would imagine that when you dd a file from a ramdisk to /dev/null at 130 MB/s originally, this change could increase that to 200 MB/s which would be quite a feat.

[cooper]

1. SATA PMP on the fileserver board - done
As it turned out, Linux Mainline simply provides a module parameter that allows you to run the SATA controller in PMP mode - it can't detect which it should be on its own, but if you provide "enable_pmp=1" to the module or the kernel, it just works. Much, much cleaner way to achieve this. Go Linux devs!

2+3. Performance tuning of network and storage - done

I started out at 32 MB/s for cache-cold items off an already spinning disk capable of delivering 60 MB/s. After tweaking the network settings, the harddisk readahead and the samba configuration I'm getting 65+ MB/s off of a harddisk that now does 90 MB/s. You could argue that this still leaves considerable room for improvement, but since at this stage the samba process serving me is claiming 70% of a processor core I think that I can only fill the line fully when I have multiple cconcurrent clients.

What I configured:

Network settings:

sysctl net.core.rmem_max=2801664
sysctl net.core.wmem_max=2097152
sysctl net.ipv4.tcp_rmem="4096        87380   2801664"
sysctl net.ipv4.tcp_wmem="4096        16384   2097152"
sysctl net.core.optmem_max=65535
sysctl net.core.netdev_max_backlog=5000

Harddisk readahead:

for disk in /dev/sd?
do hdparm -a 1024 $disk
done

Samba (smb.conf):

[global]
...
# Because I'm serving symlinks that travel off to files outside of the exported directory.
wide links = yes
# To serve clients a file rather than a symlink which would be interpreted as a link to a client-local file
unix extensions  = no
# This one alone took samba from a CPU-bound process delivering 50 MB/s to an IO-bound process delivering 65+ MB/s
use sendfile = true
# Far more efficient write processing, resulting in considerable performance increase.
strict allocate = yes

I'm still investigating if jumbo frames are supported. I read somewhere that a similar A20 board doesn't support it - all I can say right now that I don't seem to manage to enable it just yet.

With this now sorted all I've got remaining are numbers 4 and 5, which are more long-term items. Happy days!

Edited March 11, 2015 by Cooper

[cooper]

Time for an update I'm afraid. Because yes, even at home, the only constant is change.

I originally made the decision to use one of my PcDuino3 Nano's as a gateway box. It would have its gigabit ethernet port directed at my lan and a USB 100MBit ethernet adapter directed at my internet router which would be fine since I'm only getting 20 MBit down and 2 MBit up. At the time it was a reasonable decision and so far it's been working swimmingly.

However recently my local municipality decided to provide all homes with a free fiber connection and, of course, the choice is yours to just leave things as they are or to acquire various networking services (tv, internet, telephone) using this cable. So I'm leafing through the documentation thinking "meh, whatever. I'm happy with what I've got for the money I'm paying" when I notice that for about 10 euro a month over my current monthly fee I can upgrade to a 500 MBit up, 500 MBit down internet service.

Blinked a few times. Reread the whole thing quite intently and yes, it's true. A tenner more and I get 25 times the down speed and 250 times the up speed.



While they're working out how to get the cable into my home, I took another long look at my network and, clearly, that PcDuino3 Nano can't dream of coping with that. So sadly, It's gonna have to go.
Its replacement is going to have to be a 5V-powered board that either natively has, or through USB3 can be expanded to dual gigabit ethernet and thankfully I've already found such a thing in the form of the MinnowBoard Max. It's an Atom, but when it comes to just pushing data around those things still kick ass. The biggest problem though is that this board is quite a bit larger than the Nano. Not Mini-ITX large, but about 1 inch wider and half an inch longer. This means I might have to redo the plastic baseplate everything's attached to. And yes, that kinda, sorta, really sucks. But in the end, it'll be soooooo worth it. I'll probably get 2 so I'll have the main node for my cluster project aswell.

Edited June 11, 2015 by Cooper

[barry99705]

Dude, the internet in the US sucks. The fastest I can get right now is 50mbps down and 5mbps up. Technically I'm in the 100mbps down tier, but they don't have that in my area yet, so when it does come, I will automatically jump to that. I have seen 60mbps a few times though.

[cooper]

Before fiber the best you could get (and pay through the nose for) here was 120 down and probably 12 up via the TV cable network.

The TV cable providers here, in general, *SUCK*. You're not their customer, you're their bitch.

But once it finally works it does keep working so at least there's that, and price/performance, once working, has always been better than the competition, which would've been ADSL.

It's funny that they used to run ads like this one - the voice over says "Sucks huh? To not have any choice? That won't happen with cable. Tons of channels straight into your living room".

There was another one where, in a very sad, emotional scene, a guy was saying goodbye to a cute girl at a trainstation. As the train rolls away she waves out of the train window with her hankerchief then accidentally let it go and he quickly grabbed it and held it tightly, smelling her smell on the piece of cloth. Moments later he's taking a dump at the station's public toilet and when he's done he, to his shock, discoveres there's no paper on the roll. All he has is the hankerchief his girlie dropped. "Sucks huh? To not have any choice?"

It's funny because your TV cable provider is a monopolist. There's only 1 in your region and they thoroughly abuse this position. To make matters worse, Ziggo and UPC are the two main cable providers here in .nl who together cover 90% of the country (and UPC is about as loved as Comcast). A few months ago the owner of UPC, LibertyGlobal, bought Ziggo and unified the companies so now there's just Ziggo controlling 90% of the cable market. Sucks, huh? To not have any choice? Indeed...

Even if these fiber providers only deliver half the performance it would totally own.

[sud0nick]

Dude, the internet in the US sucks. The fastest I can get right now is 50mbps down and 5mbps up. Technically I'm in the 100mbps down tier, but they don't have that in my area yet, so when it does come, I will automatically jump to that. I have seen 60mbps a few times though.

I concur. The most I'm willing to pay for is 15Mbps down and 1Mbps up. Anything above that from TWC is way too expensive. I even think what I'm paying for now is too expensive. I called TWC last week and told them to cancel the few television channels I have because I never watch them but they insisted I keep them. They said I would be paying the same amount to keep my current internet speed without television which I argued for about 30 minutes because the concept is absurd. Finally, the guy offered to lower my monthly bill from the current rate and boost my internet speed up to 20Mbps down and 2Mbps up for the next year while keeping the TV channels. I still think the prices are ridiculous but they are at least a little better than before.

I think I need to move to a city where Google Fiber is being implemented.

[cooper]

Just spent a touch over 300 euros for 2 MinnowBoard Max's to get shipped here.

Hopefully customs isn't going to screw me over on this, but I'm expecting an additional 70 euro charge.

[jason_guy_yeah]

The most we get here is 100Mbs down and 5Mbs up for cable $30/month recently, we only have 2 providers, Oceanic and Hawaiiantel. It's either cable (shared pipe) or DSL, or if you have cash to burn, you can do satellite internet. At work we have 550mbs but, we're paying out of the f*&%ing ass for it but it is fiber coming in :-) . I'm curious about the setup for the MinnowBoard Max's.

[cooper]

Well, the device is intended as a drop-in replacement for the gateway pcduino (the one nearest to the edge of the case, with only a USB network adapter and an SD card plugged into it). Putting a MinnowBoard in there should for the most part just be a drop-in replacement. This will be the USB network adapter it's going to use, resulting in 2 GbE network adapters on the board allowing it to fulfill its destiny.

[cooper]

Having been shafted by customs and DHL earlier this week to the tune of 85 euro I've today received notice that my MinnowBoard Max boards are awaiting pickup.

[Sinn3rman]

I wish I never saw that keyboard, now it wants one

[cooper]

... keyboard?

[Sinn3rman]

In one of the pics "das keyboard" is it?

[cooper]

Hehehe. Yup. You can't buy this specific model anymore.

The current model that comes closest is the Model S Ultimate (their Ultimate versions are the ones with the all-blank buttons, which is what I use) but I personally don't like this new design with the extension above the numeric pad. Mine has the blue mx mechanical buttons so when I'm typing, I make a lot of noise.

But to give you some idea, I bought 3 (work, home, spare) of these years ago (I think it's rapidly approaching 10 years now) and they're all still going strong. In fact, some have the rough plastic around the buttons worn smooth where for instance I typically rest my thumb or around the arrow keys where I'm often just tapping to the beat of a song I'm listening to. One I've taken apart and run through the dishwasher to clean it a bit. I've smashed one repeatedly viciously when the machine was slow to respond... They truly are fantastic pieces of equipment. Costs a pretty penny, but you really do get a keyboard for life (and before that my keyboard would, on average, die in about a year).

[cooper]

Got my prezzies today!

Placed it on a CD for size reference. They're smaller than I expected but still larger than the PcDuino3 Nano.

[Anathae]

That port multiplier board... any chance you have measurements for it? thinking about ordering one to use with the cubieboard 2 (arm sbc similar to the pc-duino) but would like measurements to see what kind of space I'd be dealing with.

[cooper]

I'll measure it up for you when I get home, about 6 hours from now.

[cooper]

In centimetres, it's 10x5.5. The 4 holes for the screws are your standard M5's (meaning the hole has a diameter of 0.3 cm) which you get with the board, along with brass the stand-offs (basically a 0.7 cm nut - it's open on both ends). Where the holes are near the corner the center of the hole is 0.4 x 0.4 centimetre from the corner. On the other end it's 1.6 x 0.4 cm. All exterior connectors are flush with the edge of the board, but you're going to insert some connectors which would mean you need to accomodate some extra space on that 5.5 cm axis. The barrel plug that I chose to use for power extends a full 3cm from the edge of the board. If you use the molex connector the white block extends 0.7 cm from the edge of the board, and assuming a fairly tight 90 degree bend you should get away with 0.9 to 1 cm of clearance. Similarly, if you attach a sata cable, even a 90 degree one, only 0.4 cm of the connector slides over the connector on the board. The rest is overhang which on the cables I have is anywhere from 1.0 to 1.1 cm. Note that for ALL SATA connectors the little notch on the L-tab is facing DOWN. For most cables I own with a 90 degree bend, that means the cable ends up going down too so be aware of this.

I can't tell you what the dimensions of the barrel plug is without disconnecting it, and if at all possible I'd like to prevent doing so. The outside dimension of the barrel is 0.4 cm (diameter) and I've got a whole stack of USB to barrel plugs with my Orange Pi's that have that very same external dimension for their barrel plug. Those have an internal dimension of 0.1 cm, but I'd have to unplug the board to be sure.

Hope that helps.

Edited September 9, 2015 by cooper

[Anathae]

Perfect. Sounds like it's about the length of a 2.5 inch hard drive, but not quite as wide, so it might be possible to mount one on a carrier that could be mounted like a 2.5 inch drive... I have plans.

[cooper]

According to Wikipedia a 2.5" HDD's dimensions in cm is 6.985 × 0.95 × 10.0 but this board with connectors attached to it will be at best 7.5 x 0.95 x 10 meaning that it'll be wider unless you find a way to make your 90 degree SATA plugs considerably smaller (i.e. from the edge of the connector to the part of the bend can't be more than 1.25 cm (just under half an inch) and the smallest I have are 1.5 cm so you're looking at a mostly custom part, some modding or accommodating a slightly wider shape to be mounted within your carrier. Careful!