Cheap, low power Linux fileserver

[cooper]

Woot! Got things going. The stalling of the ssh session is now gone, the time after reboot was nicely corrected by ntp and everything appears to be running smoothly. I'll create an image of this SD card and use it to create the SD for the other board aswell. Once that's done I'll get to work installing samba on the one, rtorrent and its ilk on the other and start work on getting that 1-to-5 SATA thing going since at the moment the board is complaining about the SATA device being slow to respond, which may be caused by me damaging the cable, or just as easily from me simply not having attached any harddisks to the thing just yet.

One thing I did notice was that the default config I used to compile my kernel actually disables sata port multiplication so I'm going to have to turn that on at least and see if that makes things better. Following that I'm probably going to have to change what was mentioned in that forum post in my very first post. I've located the appropriate line in the appropriate file (for my 3.4.104 kernel it's drivers/ata/sw_ahci_platform.c line 252) and the change itself is entirely trivial.

Good times!

Edited January 12, 2015 by Cooper

[cooper]

Did a big software update. Tried to get Gentoo to recompile close to everything and while I was at it I recompiled the kernel for the whole gmac thing being a module. The board didn't really approve and I got what looked like a full-on PANIC once glibc finished installing. That's either because, well, it is glibc that I was replacing at the time things died or because I replaced the kernel file on the boot partition (it may have been memory mapped in some way) or because I flipped the bootable bit on the boot partition with everything mounted which didn't sit well with the kernel either.

Regardless, the thing locked solid and I had to reboot. Figured I'd strike while the iron was hot and attached 3 harddisks to the 1-to-5 adapter card. Sure enough, booting went quite a bit faster and I ended up with a /dev/sda and associated partition devices for the harddisk on SATA1. The harddisks on SATA3 and SATA5 were nowhere to be seen. Due to the way the device works, this is to be expected - without PMP it can only expose a single harddisk to the OS and since the driver in the OS doesn't currently support PMP (not compiled in, aside from the patch that I also didn't apply yet) this is about as good as it can get.

The upside of this is that I did NOT butcher the SATA cable when trying to make the sharp bends to get it to go under the board.

It's late and I have to get up early again tomorrow so I'm just going to let it continue recompiling everything (it's on package 2 of 136...) and get back to this tomorrow.

One thing I did notice is that it doesn't appear to have a fixed mac address (which is typical - my Odroid has the same problem but it has a config file to deal with that) causing my router to constantly assign a different IP to it. That's something I'm going to have to remedy aswell.

[cooper]

Since I'll probably be making the final push on this project any day now I offlined the machines this setup is going to replace and made a few snaps of them.

Here's my fileserver:


The Mobo is an ASRock E350M1 and I cut away the chassis' guide rails between the 5.25" slots so that I could insert the two IcyDock drive cages. It looks quite spiffy, but the only time I needed to take out a harddisk was when I had the thing completely full and the PSU was unable to handle the load of all of them spinning up at once. If I took one out it would start and before the OS starts loading I simply re-insert the removed one and hit reset. It did have activity leds at the front which looked really impressive when you make the box do IO.

I'll probably use this chassis to replace the one I'm currently using for my main machine due to the relative abundance and more sensible placement of the USB ports. I'll probably also start using that Seasonic PSU as I'm confident the PSU in my main machine is overkill for the hardware and at best bronze-rated whereas this one is gold.

And this was my gateway machine:

VIA Eden board with a "flash harddisk" which was like a pre-SSD - CompactFlash with an IDE interface. It was dreadfully slow and its total capacity was a whopping 4GB! To be honest, I wasn't even aware that thing was in there anymore. For sure, it didn't get mounted on boot or anything. Main storage was that 250GB Seagate drive. Fat power brick to feed the Pico-PSU that powered the thing. That small board at the front near the center is another CompactFlash to IDE adapter. Came with the chassis. There's a second one under the harddisk.
The flat cables are for the USB connector on the front panel.

I'll be reusing this chassis for this little project. The 3 harddisks will be spaced evenly across the front, the board will be mounted left from where the current board is (the USP of that chassis was that it could house 2 ITX boards side-by-side) and the space to the right will house the power brick. Contemplated trying to run cables from the USB connectors on the front to the actual boards, but with the excellent network performance of these puppies it doesn't really make a lot of sense to do so. I might even yank those connectors out but then I'd have these two big holes in the front... We'll see what it becomes but whatever it will be, it really will be glorious.

Edited January 13, 2015 by Cooper

[cooper]

If you look at the back of that chassis that I'll be using to house this little project you'll notice that next to each of the openings for the back plates for the motherboards there's a square opening. This was originally used to house the connector for the power plug. Since the chassis was intended for 2 independent ITX boards it came with a (complete rubbish) PSU for each.

Anyways, that's all gone now as you can see. My concern right now is that once I've put all the new stuff in, cables will connect from the outside directly to the boards and the power brick and such. If anything were to yank that, the boards are almost guaranteed to just snap off so I need something in the way of strain relief. I would also prefer the chassis to be closed at the back so that if I'm trying to work out what goes where in an underlit closet I don't have to worry about putting my fingers in though there and causing damage that way.

I've ordered 4 of these keystone jacks (one extra in case I mess things up) which I intend to mount on a sheet of aluminium which itself will get mounted in the square opening in the middle. It'll be 1 connector in each of the corners for 3 of them which will leave 1 corner where I intend to have an RP-SMA connector come out of the box, allowing me to hook up an antenna to the gateway board.

I'll probably place a metal cover over the bigger openings and see if I can find a suitable surface-mountable plug that I can use to attach the power cable to which will end up feeding the power brick in the chassis. That way the chassis will be neatly closed up, exposing only those ports that need to be and in a way that if something yanks at it the damage should be minimal and mostly to the connector itself. What you haven't seen yet is that the metal cover for the chassis has ventilation slits across the top over where the ITX is so heat should be able to escape. I'm also rather less worried these days about heat since even during a full glibc recompile the heatsink never goes over 40 degrees celcius. If anything heats up, it'll be the harddisks I reckon.

Edited January 14, 2015 by Cooper

[cooper]

I was thinking (I know, I know), having an RP-SMA connector on the back of your server is fun and all, but is that really a good location for an antenna? Probably not so much. Rather than have an RP-SMA connector there and use a long(ish) coax cable to get the antenna somewhere where I'd like it to be I'm probably much better off with a nice USB port that I can attach a long extension cord to, and have a wifi adapter with a kick-ass antenna directly attached to it on the other end, right where I want it to sit.

To achieve that, I was thinking of ordering this and have a USB port on the back. But then I took a closer look at those 2 USB ports I already have on the front of the chassis. They go on to a 10-pin block you're supposed to connect to the mobo. It's all fairly obvious: 4 wires - red, black, green, white. Standard for USB. I could attach 2 USB connectors to each of those cables, connect that to the appropriate ports on the Nanos and leave it at that. Obviously one port can't be used since it'll provide the extra ethernet adapter, but other than that I'm really liking this idea. Sure, the wire is going to end up with a massive chunk of tape attached to it (I know my skill level) but this WILL work, do what it's supposed to do, reuse stuff that's already there (I've cut off the part that I want from the USB cables used to power the Nanos, so that's 2 connectors in the bag already) and from the outside look pretty damned pro.

In other news, my kernel is quite consistently OOPSing on me when I put it under some sustained load. At first I didn't trust the filesystem code, but when I check the filesystem itself using my main box and a USB micro SD card reader everything's golden. That kernel is configured incorrectly, or the memory settings are off - there was some writing on the subject over at the Linux-sunxi website mostly related to u-boot but I'll have to check. It spits out a ton of voltage info into the system log so that should at least get me going here.

And finally I've got family coming over this weekend so I guess the real move for the hardware to the new chassis will have to wait another week, which at least gives me some time to figure out what's up with the kernel.

At least I managed to set the mac address so the IP on the board stays constant across reboots.

[cooper]

Managed to create a working kernel after doing some culling in 'make menuconfig'. Was a wee bit hard because I managed to turn off something that was being linked in regardless. Kernel bug, obviously, but very annoying because I knew the config option I had to get back in there, but I couldn't find where in the config the option was.

The trick turned out to be that under Device Drivers there's something about Voltage regulators that you can turn on at which point you can go back to the Power supply option and suddenly you'll find the AXP device available for you. I have *ZERO* interest in that device, but turning both Voltage regulators and Power supply drivers off results in the kernel trying to link with functions that exist in the then-not-present axp device.

Anyhoo, relative to the sun7i_defconfig kernel with as only modification the gmac instead of emac thing, I now have a kernel that's about a third less big (from 4.55 MB to 3.18 MB). I've made a number of directories that get written to quite regularly but whose contents I don't much care about at the moment a tmpfs mount (/var/log for instance - don't do this on a production machine!) and I tweaked the kernel boot parameters explicitly telling it to not do anything in terms of video aside from a simple text-oriented console and that console would be served to an HDMI monitor at 1280x720p@50Hz instead of the standard 1920x1080p@60Hz. This gave me an additional 30 megs of memory to play with (that's 3% of the total, for those not keeping track) and because there's less traffic on the memory bus for video there's now more bandwidth available for me there to do actual useful stuff with. Once I'm happy with this box I'm going to keep this kernel as a backup and config out the whole HDMI out.

Stability-wise, this kernel is rocking! One thing I did find though is that when I reboot, there's no /dev/fd to be seen which, on Gentoo at least, causes all sorts of problems during building. The way to remedy this is by updating udev but doing that means updating kernel headers to 3.18 which I explicitly configured that I don't want because I'm not using that kernel. We'll see how that is going to turn out but I was a bit surprised to say the least when I asked it to emerge udev and it started downloading that headers package.

Because /dev/fd should simply point to /proc/self/fd I manually created the link for that, but it's udev's job so we'll see how that's going to turn out.

Regardless, the current situation is a MASSIVE improvement from the previous one, where I could maybe install a package or two and there's be an Oops in the logs. I'm still not convinced all the problems are gone but things are once again looking up.

[cooper]

Package compilation was still being a bit iffy. No more kernel complaints, but I had all sorts of problems from bash, make and even gcc. In case of bash I'd get segfaults or the process stuck at 100% cpu usage doing a whole lot of nothing, for make it was a sudden disagreement about the validity of a Makefile and with gcc the problem was that the process would just up and die on me. Nothing in the logs, nothing to the console, just one minute it's there and gone the next. My current method of resolution entails removing "-j3" from the MAKE_OPTS variable, forcing compilation to occur using a single process. It might not be particularly quick, but certain packages are starting to behave.

I've since rebuilt binutils, make, sandbox (the Gentoo hack to do 'make install' into a separate directory even though the build process thinks it's installing in /) and automake and I'm in the process of building gcc itself. That last one has never succeeded yet, so it's an interesting one.

I got the idea from make complaining about segfaulting due to unexpected jobcontrol problems and the building of the kernel (make -j3 uImage modules_install) seemingly doing a 'make uImage' and a 'make modules_install' simultaneously as separate processes which, obviously, didn't work well since the latter depended on files produced by the former.

If it were to turn out that even this won't allow me to do a full rebuild of Gentoo I'm going to switch the boards around on the assumption that I'm dealing with a faulty device rather than problematic software. But let's hope it doesn't come to that.

Edit: Quick update. After 5 hours worth of non-stop compilation torture using just one job, gcc completed building stage 3 and successfully compared the output of the stage 2 compiler against that of the stage 3 compiler. That means the build now believes I have a working compiler which it will use to create... (wait for it)... my new compiler! So another hour or so of compiling and the monumental (or maybe just mental) task of updating gcc from 4.8.3 to 4.8.4 will have completed with the added bonus that unlike the old compiler the new one will be optimized for the Nano. Next step is to build everything else, which will probably take a few days aswell... Ah, the joys of a Gentoo system!

Edit 2: Morning update. The compiler installed, but the subsequent rebuild of the entire system got stuck. No errors, just something waiting for something that will never happen. Gentoo has the 'emerge -r' command which simply continues from the last succesfully installed package of your last emerge command and this time it did continue beyond that point. I'm actually growing mildly suspicious towards python, which is what the emerge command was written in. Something didn't detect that a program succesfully completed its work so... Stability-wise things are still golden though.

Edited January 19, 2015 by Cooper

[barry99705]

VIA Eden board with a "flash harddisk" which was like a pre-SSD - CompactFlash with an IDE interface. It was dreadfully slow and its total capacity was a whopping 4GB! To be honest, I wasn't even aware that thing was in there anymore.

Heh, these were slow. Compact flash rotating disk hard drives! The white one on it's back is a whopping 340Mb, the other is 4Gb. That's as big as they ever got. Wifi Pineapple microsd card for scale. Pulled the 4giger out of a linux pda and exchanged it with a 16Gb flash based cf card. Huge difference!

Edited January 20, 2015 by barry99705

[cooper]

I remember those. Fascinating bit of technology. You basically had a single-platter harddisk the size of a CF card.

Benefit was that it could be much larger (yes, 340MB total qualified as "much larger" back then) than the flash memory.

I think they were quite a bit more expensive aswell though and all problems of a HD still applied (being not very able to survive a modest drop, say).

[cooper]

Still working on getting the system to fully recompile everything. Discovered that I can do better with the gcc compiler options so I've documented them and will recompile everything once more when I'm done. Maybe even turn on make's jobserver once more to see if that works out once the current full recompile completes. I did verify that the original compiler options are entirely valid for the chip, but the code won't use NEON for parallellism due to a missing flag.

The problem with the system recompilation is still that every so often the process gets stuck on something - the process either stalls waiting for something that never happens, errors out with an 'Illegal instruction' or claims that there is a package conflict because a file is already claimed by a different package.

The python interpreter itself has been rebuilt already so that's not the issue. The times when I see that "Illegal instruction" message, that would suggest something is run that isn't built for this particular CPU. That something issues an instruction the CPU can't cope with and it all falls apart. Nice in theory, but if that were the case I'd expect the problem to appear consistently when in fact it's highly erratic. Sometimes I get such an error three times in a row, but each time at a different stage in the process. Try it once more and the build succeeds without a hitch. Strange. Faulty hardware is still an option, though a less likely one. I would've expected the kernel to still be complaining if that were the case, but that's been going very well. Didn't have to reset the system in I think it's 3 days now.

Out of the 212 packages to rebuild I've got about 100 still to go. The biggies are sorted though, like gcc (yes, again) and glibc (yes, again). I still want to squeeze a bit more crap I don't intend to use out of the kernel but this comes first. Hope to finish up before the weekend so I can test the SATA PMP functionality - what got this particular ball rolling in the first place. If I get that to work, which I fully expect to, I'm going to get this thing mounted in the case as I copy the OS from the main machine's SD card onto the other one. Then as the one compiles samba the other can show it supports my USB 100MBit ethernet adapter.

I've also decided my next project will be a smaller DIY 19" rack to replace my existing, very DIY 19" rack. The existing one is built in such a way that it takes up too much space - it's too high, too deep and assumes that that big fileserver is at the very bottom so the rack mounts start half-way up. The thing's also not particularly straight (insert gay joke here). I'll make a separate topic for that when the time comes.

Quick update: 30 more packages to go and it's all done. Had to restart the rebuild process twice - once because it stalled in the install phase (just sitting there, no error no nothing) and once because it felt there was a package conflict with a previously installed package already having provided a file at a given location. In both cases I just requested it to resume (retry the last package and if successful continue with the rest). At this rate I can try attaching the harddisks tomorrow and get all that goodness going. Really, really looking forward to that.

Edited January 20, 2015 by Cooper

[cooper]

Everything's now been built so on to the harddisks. Oooooh yeeeeaaah!
First I re-attached the harddisks and the 1-to-5 adapter (had the power pulled on both while I was working on the software side of things) and out of the 3 it found only 1. Now, this is with a kernel where SATA and PMP support is compiled in, so clearly the driver doesn't allow me to do PMP with the chip. I ran a quick test to see what kind of performance I'm getting out of this one SATA port with only the one harddisk attached, still going through the 1-to-5 adapter.

localhost ~ # hdparm -tT /dev/sda1
/dev/sda1:
Timing cached reads:   342 MB in  2.01 seconds = 170.25 MB/sec
Timing buffered disk reads: 172 MB in  3.03 seconds =  56.72 MB/sec

That's looking pretty damned good.

So I'm now rebuilding the kernel to include that minute patch to the SATA driver. Once that compile completes and I've rebooted with it I'll report back on the status.

Update: IT WORKS!

[ 5.774444] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 5.793688] ata1.15: Port Multiplier 1.2, 0x197b:0x0325 r0, 5 ports, feat 0x5/0xf
[ 5.827252] ata1.00: hard resetting link
[ 6.304499] ata1.00: link resume succeeded after 1 retries
[ 6.435084] ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 6.453622] ata1.01: hard resetting link
[ 6.904492] ata1.01: link resume succeeded after 1 retries
[ 7.055218] ata1.01: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 7.073896] ata1.02: hard resetting link
[ 7.524490] ata1.02: link resume succeeded after 1 retries
[ 7.655083] ata1.02: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[ 7.694535] ata1.03: hard resetting link
[ 8.784487] ata1.03: failed to resume link (SControl 0)
[ 8.804266] ata1.03: SATA link down (SStatus 0 SControl 0)
[ 8.823345] ata1.04: hard resetting link
[ 9.934535] ata1.04: failed to resume link (SControl 0)
[ 9.953003] ata1.04: SATA link down (SStatus 0 SControl 0)
[ 10.020599] ata1.00: ATA-7: ST3250620NS, 3.AEE, max UDMA/133
[ 10.037858] ata1.00: 488397168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[ 10.095583] ata1.00: configured for UDMA/133
[ 10.159259] ata1.01: ATA-7: ST3250620NS, 3.AEE, max UDMA/133
[ 10.176031] ata1.01: 488397168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[ 10.254515] ata1.01: configured for UDMA/133
[ 10.317785] ata1.02: ATA-7: ST3250620NS, 3.AEE, max UDMA/133
[ 10.334747] ata1.02: 488397168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[ 10.409432] ata1.02: configured for UDMA/133
[ 10.426625] ata1: EH complete
[ 10.444756] scsi 0:0:0:0: Direct-Access ATA ST3250620NS 3.AE PQ: 0 ANSI: 5
[ 10.464495] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 10.483627] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 10.483966] sd 0:0:0:0: [sda] Write Protect is off
[ 10.484011] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 10.484318] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 10.539707] sda: sda1
[ 10.561707] sd 0:0:0:0: [sda] Attached SCSI disk
[ 10.575599] scsi 0:1:0:0: Direct-Access ATA ST3250620NS 3.AE PQ: 0 ANSI: 5
[ 10.577780] sd 0:1:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 10.578453] sd 0:1:0:0: [sdb] Write Protect is off
[ 10.578500] sd 0:1:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 10.578805] sd 0:1:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 10.658308] sd 0:1:0:0: Attached scsi generic sg1 type 0
[ 10.679179] scsi 0:2:0:0: Direct-Access ATA ST3250620NS 3.AE PQ: 0 ANSI: 5
[ 10.682114] sdb: sdb1 sdb2
[ 10.719666] sd 0:2:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[ 10.720098] sd 0:2:0:0: Attached scsi generic sg2 type 0
[ 10.760243] sd 0:2:0:0: [sdc] Write Protect is off
[ 10.762193] sd 0:1:0:0: [sdb] Attached SCSI disk
[ 10.798880] sd 0:2:0:0: [sdc] Mode Sense: 00 3a 00 00
[ 10.799269] sd 0:2:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 10.840670] sdc: sdc1 sdc2
[ 10.863652] sd 0:2:0:0: [sdc] Attached SCSI disk

Maybe I should point out that this is with 3 of my spare 250GB disks. I wasn't going to jeopardize my actual data on this when I didn't have to...

I did a really basic parallel hdparm on the disks with

hdparm -tT /dev/sda1 > sda1.txt &
hdparm -tT /dev/sdb1 > sdb1.txt &
hdparm -tT /dev/sdc1 > sdc1.txt &

which I simply copy-pasted into the terminal. Not a good way to do parallellism, but works for now. The output:

/dev/sda1:
Timing cached reads: 224 MB in 2.01 seconds = 111.66 MB/sec
Timing buffered disk reads: 56 MB in 3.09 seconds = 18.11 MB/sec
/dev/sdb1:
Timing cached reads: 202 MB in 2.01 seconds = 100.30 MB/sec
Timing buffered disk reads: 52 MB in 3.06 seconds = 17.01 MB/sec
/dev/sdc1:
Timing cached reads: 248 MB in 2.01 seconds = 123.61 MB/sec
Timing buffered disk reads: 54 MB in 3.06 seconds = 17.62 MB/sec

So what this shows to me is that this solution still provides pretty decent performance when used in parallel. 60MB/s is probably the limit for what this chip can transfer from the physical device. More devices means it needs to hop around the connections and that ends up slowing things down a bit more than when it's just talking to a single drive. But in all, this is frigging *AWESOME*! I would've been happy with 20 MB/s total.

Update 2: Power consumption measurements.

I placed a kill-a-watt type thing on the power plug and measured a staggering 92 watt maximum power draw - obviously these 3 old harddisks spinning up at once gives my paltry 60W power brick a beating but thankfully not long enough for it to falter. I'm quite amazed by that because when I had 10 of these in that big fileserver I used a 1000W PSU to power things and it still has brown-outs when everything was connected. Anyways, once everything's up and running power consumption drops to just 31.4 watts!

And mind you, this is with the disks still spinning. I subsequently ran

hdparm -S 12 -K 1 /dev/sd{a..c}

which tells your harddisks to spin down and go into low-power mode after 1 minute of idling. Once you do this your harddisk won't spin up again until you access it, so initial latency will be absolute shit. HOWEVER once the disks stop spinning, power draw for the full system (2 PcDuino3 Nano's, one 1-to-5 SATA board, a 12V to 5V buck converter and 3 harddisks in low power mode) is a mere 9.6 watts!

If I spin up 1 drive, power draw goes up to a still very, very modest 16.2 watts.

So what does this mean, money-wise? I used to have a setup that consumed a fairly constant 80W which meant a yearly electricity bill of 160 euro. On the average week I work 5 days and only do something with my machines when I get back home. If I get home early I can maybe watch 2 movies, meaning about 4 hours that I have 1 drive spun up, the rest of the day it's idling. During the weekends I'll be making more use of the system, but in general I'd say I'm accessing files for about 18 hours of the total time in the weekend. That means for an average week 5*(4*16.2+20*9.6)+30*9.6+18*16.2 = 1863.6 KWh which per day is 266Watt and that per hour is just over 11 watts. That means the electricity bill for this setup and my typical use for a whole year is now just 23 euro. I did have to spend a little extra on the heatsinks and I'm also counting the keystone jacks which have yet to arrive so the total amount spent is about 125 euro but relative to the old setup I'm saving 0.375 euro per day so this project will have paid for itself in exactly 320 days. That makes me very happy.

Edited January 21, 2015 by Cooper

[cooper]

Setting things up so I can place my build into the chassis and I run into this:

I've put back in place the original Mini-ITX board as a size reference, since my build is on a piece of plexi cut to Mini-ITX size and I took a spare harddisk to fill out that space. I don't expect to end up using 4 harddisks anytime soon, but when it's possible I want to be able to. To see the problem, here's a close-up of my build I posted earlier

Can you see what the problem is?

In the first image you can see that the distance between the Mini-ITX and the harddisks is rather small. In fact, I measured just 2 cm there. If you take a cheap, straight SATA connector, then the hard bits of the connector that don't plug into the harddisk amount to that. Add to that the fact that the 1-to-5 SATA board was mounted on the ITX with considerable overhang AND there has to be a connector mounted there aswell, you can see I'm well and truly fucked.
So, sadly, I'm going to have to redo the board again, this time allowing the thing to be wider than ITX (I've got plenty of space in that direction) but stay well clear of the edge of the board with my 1-to-5 SATA board. This, in the parlance of our time, sucks. I need to rearrange things such that I don't have to redo the power cables. I'd hate having to buy new ones over this, but it might come to that.

Since I'm going to do this I might address some other issues aswell. First, one of the Nanos was so close to the edge that its ethernet port would be up against where your typical Mobo would have its on-board connectors - right in front of the hole in the chassis I want to close up because I don't want anything that can be put under strain directly touching the boards meaning that in its current location/orientation I can't close the hole in the chassis but at least still reroute the cable to the keystone jack for it that is yet to arrive.

The other, slightly more daunting task is that I'm going to mod my SATA data cables. Yup. I'm going to make them the exact size required, starting with the one going from one Nano to the 1-to-5.
When I look at the connector for the first cheap-ass cable I pulled from my parts box there's an obvious seam on the sides. I ran an exacto knife along that seam and used that to pry it open, showing its utter quality to us all.

I was actually a bit shocked at this because when I made this picture I'd already used a screwdriver to pry the inner cables apart a bit because they were so close. There is nothing to keep the cables from touching each other in the first full centimetre of the plug. The 2 data cable pairs are the only insulated ones (within the red wire, that is) and they could've easily run them, still insulated, all the way up to the connector. You're looking at the bottom of the connector which is why you see those orange-yellowy bits. It's open on the other side and the cables are just soldered onto that. What I'm going to do is shorten this cable, reattach the connector and thus have a perfect fit between the two. The other end is an angled connector, so I don't have to bend it much either. I'll probably do something similar with the other SATA cables aswell, just to keep the cable clutter in there down.

I'm still in the process of recompiling the full distro (compiling 215 packages using a single thread and accounting for having to restart builds every so often because something fell over takes 2-3 days) but when that's done I'm going to attach the real harddisks, see what that will mean for overall power consumption and then I'll get to work repositioning the boards such that it'll also work within the chassis.

Edited January 24, 2015 by Cooper

[cooper]

Sometime yesterday I noticed that I'd omitted the 'neon' use flag. That would've been decidedly stupid except for the simple fact that most packages currently installed don't have this particular flag. It's more reserved for video encoding/decoding type packages and since I'm still mucking about with the base system this hasn't really come into play just yet. So I figured I'd wait until the build completes, add the flag and do a quick 'emerge --sync && emerge -uND world' where the N would cause any package that can make use of the newly-added flag to get slotted for recompilation.

Last night the build completed so I issued those commands and it turns out that between that moment and my last --sync the brand spanking new GCC 4.9 had become available.

Now you must understand that GCC has spent *YEARS* optimizing generated code for the x86 instruction set. Due to the similarly vast installed base of x86-64 machines, a lot of effort has already gone into making GCC create great binaries for that platform aswell. For ARM, things are going well but it's lagged behind relative to those other platforms mostly because, well, not a great many developers (relative, of course) had these machines to play with and, thus, cared. With the Raspberry Pi becoming such a massive success and causing a huge amount of extra interest in that and other ARM devices, people are buying these devices and trying to make the existing tools work really well on these platforms. Because of this, ARM support in GCC has improved significantly over the last few releases and I suspect that, while slowing down somewhat, 4.9 will be no different.

New in GCC 4.9 is support for the 'armv7ve' architecture with is armv7-a with virtualization extensions and as it turns out, the AllWinner A20 has those. You can tell from the output of /proc/cpuinfo where under Features it mentions 'idiva' and 'idivt'. To be honest, I don't think having these additional instructions are going to do me much good, but I'll let the compiler itself be the judge of that. It's worth noting that GCC 4.9 contains a large number of improvements to itself which they exemplify by stating that a debug build of Firefox went from requiring 15 GB of memory to build to just 3.5 and linking the thing went from 1700 seconds to 350. All things considered I once again feel I should just rebuild everything...

Note that since the current system was built with GCC 4.8 this will be Gentoo's default compiler. Use the gcc-config tool to switch to the 4.9 one.

But before I do I'll put the correct harddisks in, measure power draw, work on aligning the various boards such that I can start on the new chunk of plexi that will support everything and then, as I work on making the board the machine can spend its time rebuilding once more.

Edited January 25, 2015 by Cooper

[cooper]

Power draw with the correct harddisks attached:

Max boot: 34W

Idle, spinning drives: 19.6W

Idle, stopped drives: 8.5W

Idle, 1 Samsung drive spinning: 12.5W

Idle, 1 WD drive spinning: 11.4W

That's considerably less than the previously quoted numbers. Let's look at my average week once more, and assume that there's a 50-50 chance the data I want to access is on a samsung drive or on a WD drive meaning the average power used while a disk is spinning is 12.0W. Then my average power use for a week is 5*(4*12.0+20*8.5)+30*8.5+18*12.0 = 1561 KWh which per day is 223Watt and that per hour is 9.3 watts.

Money-wise, this setup for a full year will cost me 18,50 which is quite the improvement over the 160 per year I was spending previously.

It's important to note that the Western Digital "Green" drives interpret the -S parameter to hdparm, used to set the time until the drive motor spins down, differently. I try to go for spin-down after 1 minute of idle, which would imply -S 12 but not on the WD. The WD interprets -S 3 as spin down after 10 minutes. There's little variation in its behaviour when you for instance use a lower value, so let's just stick with 3.

Next up: Repositioning everything so it actually fits within the chassis.

[barry99705]

Heh, I just use my servers in the basement as extra heat....

[cooper]

In my youth I had 4 machines running 24/7 in my bedroom (still with my parents back then). To drown out the monotonous hum I'd leave the radio on at just over the hum noise level. My room was right under the roof so it got stupid hot in my room during the summers. Also, I was running distributed.net back then aswell so those machines were really doing their best to warm up the room. I know there was one day where the mid-day temps hit 39 degrees C. With our climate, that's close to inhuman.

When I got my own place my first order of business was to get a separate room for the machines. Took a while for me to learn to sleep comfortably in silence. I can still sleep straight through armageddon though. Due to the room the machines were in being completely non-ventilated I had to leave the door open during the summer so a lot of the noise made there got out. Eventually I started consolidating and now, I'm down to this.

Imagine that. There used to be 4 machines in that room running full tilt burning god knows how much electricity to do the same thing these 2 puny machines do at a grand total of 10 watts average.

Edited January 25, 2015 by Cooper

[barry99705]

In my youth I had 4 machines running 24/7 in my bedroom (still with my parents back then). To drown out the monotonous hum I'd leave the radio on at just over the hum noise level. My room was right under the roof so it got stupid hot in my room during the summers. Also, I was running distributed.net back then aswell so those machines were really doing their best to warm up the room. I know there was one day where the mid-day temps hit 39 degrees C. With our climate, that's close to inhuman.

When I got my own place my first order of business was to get a separate room for the machines. Took a while for me to learn to sleep comfortably in silence. I can still sleep straight through armageddon though. Due to the room the machines were in being completely non-ventilated I had to leave the door open during the summer so a lot of the noise made there got out. Eventually I started consolidating and now, I'm down to this.

Imagine that. There used to be 4 machines in that room running full tilt burning god knows how much electricity to do the same thing these 2 puny machines do at a grand total of 10 watts average.

Sure, at a much slower pace though. I'm looking at replacing one of the machines in the basement. It's a windows home server box I built years ago. The other machines are a vmware server, the pfsense firewall, and a proxmox test rig that will also go away. Oh, there's also a headless laptop running windows 7 that's my Ubiquity wireless controller box.

[cooper]

Sure, at a much slower pace though.

Slower for sure. Much slower... you'd be surprised. The main point for me is that it's fast enough. All I want these machines to do is to push data across the network and maybe verify a torrent or unpack a rar file every now and then. So long as the throughput is good, I'll be happy.

[barry99705]

Slower for sure. Much slower... you'd be surprised. The main point for me is that it's fast enough. All I want these machines to do is to push data across the network and maybe verify a torrent or unpack a rar file every now and then. So long as the throughput is good, I'll be happy.

Good point!

[cooper]

Back in comment 35 of this thread I mentioned that I was occasionally getting "Illegal instruction" errors every so often which, while annoying, didn't occur often enough for me to be really bothered by it. Well, I got pissed about it anyway in part because this is a dual-core machine and the best way so far of preventing this issue was to only build using a single make job meaning one CPU would sit idly while the other is doing the actual work. Spending 3 days rebuilding everything when you know it could be done in 2... Nope, not gonna happen.

So I got pissed about this issue and decided to ask for help on the #linux-sunxi IRC channel. Turns out that somewhere along the way my bash is doing something that's causing the vminnm.f32 instruction to be executed. This instruction is only available from the armv8 cpu onwards and since this is an armv7 I'm quite curious how it ended up here. Because I compiled all my packages with '-fomit-frame-pointer' the backtrace in gdb doesn't show what chunk of code the program is currently executing - just a large chunk of ASM. Could be bash itself or just as easily any of the libraries it's linked with. I will need to rebuild bash and its dependency tree without the '-fomit-frame-pointer' parameter and with Gentoo's 'FEATURES="nostrip"' option and the 'debug' USE flag to find out where this code is coming from - if it's a hand-crafted optimization that shouldn't be there, or if the compiler is making false assumptions because of possibly too aggressive parameters. I'm leaning towards hand-crafted ASM somewhere because I was seeing very similar problems when everything on my system was built with gcc 4.8 even though I'm using gcc 4.9 now. You'd think that if gcc had a bug that caused it to incorrectly emit this instruction somewhere, it would get found and fixed before the next version of gcc.

To summarize, I'm hot on the heels of some bug on my system that causes an armv8 instruction to be attempted on my armv7 platform. This can be a bug in the program code as well as the toolchain and I won't know for sure until I'm further along in the process.

One alternate cause can be a program bug that's corrupting the stack in a subtle way causing the next instruction to execute be something it's not supposed to be.
I'll keep digging.

Edited January 27, 2015 by Cooper

[barry99705]

Other than the fun you're having building this thing, this would be much easier...

[cooper]

Easier, sure. But quite a lot more expensive in both the near-term as well as the long term. My setup is below the $100 mark not counting the drives (remember, I'm sporting TWO machines of which only 1 will be fileserver) so I'm saving over $300 right there already. In terms of electricity the current-gen Drobo 5N draws 13W idle, 56W typical. I've already posted my power requirements and they're lower. I'm sure the drobo will blow my setup out of the water performance-wise (otherwise it's just another crappy product in a shiny exterior) but my setup is good enough for me. Looking at the Amazon page, people aren't exactly cheering for the company's customer support...

And, like you said, I'm having a TON of fun here. It turns out that my toolchain is fine. The problem I'm seeing has to do with memory corruption rather than incorrect ASM generation. This is to some degree to be expected from such a new board. There are tests one can perform to verify stable operation and back in december the .fex file for the similar cubieboard2 and cubietruck boards received an update to increase various voltages around the board to increase overall system stability. It might also be that the ondemand cpufreq governor is boosting the CPU to frequencies outside of recommended range or something like that.

This is new terrain for me aswell - my Odroid never had any such problems but then, they had kernel trees and setup files explicitly targeting THAT board. The Linux-sunxi people are trying to support basically 'anything with an AllWinner chip on it' without any advanced warning that some new part gets released. If they're lucky someone buys one, grabs the vendor-provided SDK (assuming one exists), finds the appropriate voltage levels, timings and what not and feeds that back to them. On the whole you can tell that it's only slightly off - things mostly only break when the hardware gets significantly pushed. I'm confident we'll work this out in the next few days which will result in a fix that saves anybody coming after me the trouble. Everybody wins.

Talking to these guys I've also found that there's no reason for me to not run the sunxi-next a.k.a. Mainline kernel. It's basically Linus' current tree with a few sunxi-specific patches. I prefer running that, it's just easier in the longer run, but let's get the thing stable first.

Edited January 28, 2015 by Cooper

[cooper]

Figure an update is long overdue because I haven't been sitting still.

First, here's a really shitty picture of the chassis showing off my 1337 electrical taping skillz, putting a USB plug on the end of the cable where that block used to be.

Right...

So, I had to redo the layout again and ended up with this:

Again, sorry for the crappy pic, but I tend to make these at the end of the day and by then light is rather shitty. So, in this pic the front/bottom would be where the harddisks go, with a bit of spacing between them and the plexi so there should be plenty of room overall there. The 1-to-5 SATA port multiplier and the buck converter have both been bolted to the plexi once again and this time to keep the Nanos in position I've once again drilled holes in the plexi and ran nails through them to have something for the boards to slide over, however this time I used thinner nails to have a bit more wiggle room and because the plexi is only like 1-2mm off the isolated(!) bottom of the chassis, I'm just gonna leave it like that. Nothing to gain with fixating the nails. Once the plexi is screwed to the chassis the nails have nowhere to go meaning that as long as I don't hold the chassis upside down the Nanos won't have either.

The SATA situation was a bit of a mess that I managed to get somewhat sorted. Here's a close-up:

As you can hopefully make out, both Nanos are positioned with the same side forward and in the board in the back you can see the empty SATA slot with that inner L thing with the small notch at the top, aimed to the left, so it's an L rotated 180 degrees. If you look at most SATA cables with a 90 degree plug, they're shaped such that the cable would continue to the left, away from the port multiplier. Thankfully a rummage through my big box of spare parts turned up 1 SATA cable with the cable moving in the other direction (thus the ugly blue thing). Everything's lined up beautifully although I did have to trim down the nail that it passes over - it's the currently empty hole in the corner of the board in this pic. All relevant inner cables are exposed and thankfully they're solid-core so they're good to work with. It might look like I've left myself with very little excess on the cable to play with, but it's actually about 7mm too long currently.

That's the good news.

The bad news is that I now have to atach those ends to that SATA connector I opened up a few posts ago and it turns out that unlike what I originally thought, those cables are NOT soldered on. The yellow-orangy bit is in fact a metal clamp that's sort of crimped around the core it's attached to and I need to either find a way to pry that loose or solder onto it directly. This might be a good time to point out that I'm also not a particularly leet solderer. In fact, the amount of things I've soldered you can count on the remaining hand of Mad Bill, the clumsy guy who still works at the sawmill in spite of several trips to the emergency room...

I still need to redo the wiring and I'm also wondering aloud if I should do something to keep the power brick in position - if I do, it should allow the insertion and removal of the power chord, but it's a tight fit (as it should be) and I'm not sure I want it to actually be removable at all. If I put something against the back it'll be something that's pressing against the plexi and I don't think it handles stresses well enough to survive a lot of that either.

Finally, I didn't snap a picture of it just yet but the keystone jacks are in so I can start work on the back plate that would close up the entire rear of the chassis and provide something for the keystone jacks to... jack into. They're all white but since one is intended for the outside line I'm wondering if I should maybe paint it red like the network cable I'll be using for this (all other cables are gray)... Okay, I'll admit that sounds a wee bit gay, but whatever. It's important to be able to distinguish that one from the rest.

[barry99705]

Please swap out the electrical tape with heat shrink tubing! It will save your sanity down the road. The electrical tape will turn into an ungodly mess after a few months with the heat. Also, here's a shot of the inside of my home server, now running windows 8. It was running windows home server 2011, but I knocked it off the shelf it was sitting on and killed the os drive... Thought I'd give windows 8 storage spaces a shot. 8TB of storage drives, haven't done the spaces thing yet, so not sure how much usable space I'll have when done. Have to do it in chunks because I have to move data from the home server drives to the new drives live.

[cooper]

I know what you mean. Eventually under the influence of heat the sticky stuff simply leaks out from the tape.

Problem was that I didn't have any heat shrink tube on hand and now it's too late - I'm not going to rip these cables apart again just for this. It was a far too fiddly a job to do in the first place so it's going to have to wait for me to get annoyed again. And that may take a while since it's not going to get particularly hot in there anyways.

What might happen though, is that once everything's in the chassis I get annoyed by the amount of excess cable this represents and therefore decide to redo it, but that'll become apparent in due time.

Something else I noticed when I was still compiling everything on this board was that when I had 1 drive mounted and a swapfile on there registered (swapon /mnt/drive/swapfile) the system would access the harddisk every few seconds, presumably to see if it's still there. I'm not sure if this is because it was mounted or because it housed the swapfile or even simply because it housed the swapfile and actually needed to access some swap on there. If it has to do with being mounted, it could prevent the harddisks from reaching the deep sleep state so it's something to keep an eye on. I might have to resort to some automounting tricks.