There's always room for Pi.

[cooper]

Long story short....

... got bored with a LOT of things this year, so half januari I ditched the girlie and as luck would have it a former co-worker positively mentioned my name to the recruiter if his current company. I had an interview with them yesterday and I went in somewhat cocky, feeling that I knew my stuff and was good at what I do only to find out during that interview that over the last 6 years I've effectively fallen asleep at my work desk and since I also effectively stopped tinkering and hacking during this time I've actually been standing still resulting in the technological advances flying past me without me even so much as noticing. During the interview certain projects, products and concepts were mentioned that I wasn't familiar with and I spent (way, way too) much of last night diving in with the ferocity of a starving hyena at a captured gnu. (Maybe I should add that even though the interview itself was humbling for me, I did get a callback and we're currently discussing salary) I'm very much aware I'm playing catch-up but very happy this company jolted my synapses and am once again actively scouring the internet looking for everything I've missed so far. And in the course of that...

... I'm looking into the concept of HA and HP clusters. And an easy and by far cheapest way to dabble with this is using a couple Raspberry Pi. Case in point, the 64 node supercomputer at the University of Southampton. Looking at that link and in partucular the pictures at the bottom, I got inspired. So my plan is to (first) get 4 Pi's and play with them, see how that goes and if I like it expand upon that if I feel like doing so.

You see in one of the images how they're powering every single Pi there with its own wall-wart. That struck me as rather ugly. All those plugs, all those cables.

Since the device is powerd over its micro USB port, my first idea/solution to that was to use a powered hub to feed them. Problem is that powered hubs that adhere to the official standard won't supply more than 0.5A per port and the Pi officially requires 1-1.2A. There's a list describing powered USB hubs that a single Pi can run off of, by the general idea that the Pi can use up more amps if the other ports remain vacant or are filled with low-power devices. In my case that won't suffice since if I were to go this route I'd want to get a 4-port hub and connect all 4 Pis to that one hub rather than use a 7-port hub to connect them, just to have sufficient power available.

So I'm now thinking about taking an external power supply/brick I have lying around to power the PicuPSU in a now-dead (capacitor plague) Via Eden Mini-ITX. It's currently set to 12V but there's a little plug you stick into it to change the output voltage and if I didn't toss the alternate plugs (please, please, please) I should be able to switch it to 5v and deliver up to 10 amps (it's rated at 60W) to a power rail. Using some wire, solder and micro USB connector plugs I can wire the lot up from a single power brick.

Should be a simple little hack. With it the full cluster, even when expanded to 7 or 8(*) Pi's, should be able to run off this one power brick, greatly reducing the cable clutter and possibly even raising electrical efficiency. Does this plan sound sensible to you, or is this more an elaborate way to fill my home with the Magic Smoke?

* I'm thinking that the cluster is on its separate network segment using a standard 8-port switch. One of those ports connects the cluster to the main network so I can attach Pi's to the remainder, making a total of 7 Pi's the logical amount. From that I could make one a load-balancer and the other 6 worker nodes. Alternative would be a USB network adapter on the main node / loadbalancer to tie into the main network and all Pi's on the switch which is now dedicated to the cluster... We'll see.

[james92]

That sounds like a brilliant project cant wait to see how it goes a friend of mine told me about this a good while ago http://www-public.it-sudparis.eu/~trahay_f/odroid_supercomputer/

[cooper]

That's AWESOME. I own one of those U2's and they're magnificent. They're EOL now and instead you can now get the U3 for $65 ($60 is you get just one) ex shipping which is pricey since it'll be fedexed from Korea.

The problem with my concept is that I want it cheap, which favors the pi (35 euro vs ~ 65 euro incl shipping), and fast which, by a very, very large margin (700mhz single core vs 1.7 GHz quad core) favors the Odroid and finally, since I'm a professional Java guy, I could really use a good helping of RAM aswell, which is another win for the Odroid which has 2 gigs onboard, relative to the pi's puny 512. The droids have a phone charger type pin connector power plug which might be harder to source as a mere connector. They're also smaller, for whatever that's worth.

Life is full of important choices. Maybe I can get the company to pony up the shipping and taxes as they might find it to be tax deductable. Won't know until I tried...

[GuardMoony]

Ever looked at: http://www.parallella.org/ they got 16 cores at 700Mhz. you also got version with 64 cores. And there made for clustering.

The disadvantage is: You need to compile it on the SDK of that board. and some restrictions in the hardware.

[cooper]

Looks like they have a VERY hard time delivering. They're planning to complete delivering the current pre-orders in august (!!). Also, the FPGA they're using is getting mighty hot in spite of its modest frequency and power draw (5w).

It's an interrsting project, but for my purposes it won't suffice.

I'm a bit miffed right now. I was looking at a switch for my cluster to be and expected to find a really cheap 8-port 100mbit switch with a single gbit uplink port. Turns out it's way cheaper to get a 9 port gbit switch (8+1). Now I think that's a bit of a waste since the boards are all 100mbit.... What to do, what to do...

[GuardMoony]

yeah the delivery got kinda delayed. because its pretty new. ( kickstart project ) But they always been given decent information. And why things are delayed. atm there clearing a 15 000 device backlog. I ordered 1 in november to mess around with. At least they gave every1 the choice for a free t-shirt or a case.

[cooper]

Hehehe. I can see it now:

"I got shafted on a kickstarter project and all I got was this lousy t-shirt"

[james92]

It is a little pricey with the odroids I have a odroid XU i would LOVE to have a cluster of them ha

when you do set up your cluster you gota brute force something ha i really really wanted to see how fast i could brute force a handshake it would be lovly just to know you can do if fast ha :D

And also i was looking at the parallella for ages but doesnt look like they have as good a community as odroid seems like theres alot more support there for idiots like myself ha :D

[cooper]

I'm on the Odroid forums aswell. They're an allright bunch and genuinely interested in getting the most out of their little platform, as am I as time permits.

My schedule for this is to start putting things together at best 3 months from now.

I need to get my place back from the girlie and some time and money to tidy up. After that I'm going for this, most likely with 8 U3s.

[cooper]

Made a quick sort-of mock-up and BOM for this little project. Here's what I chose:

- Gigabyte GA-C1037UN

I wanted a Mini-ITX board with 2 gigabit connections. One to serve the outside world and the other internal to the cluster network from where the commands are given to the cluster. This machine must also act as a fileserver to the cluster, so the board must have at least 2 SATA ports. The Gigabyte was the cheapest board that fit the bill.

- TP-Link TL-SG1016D 16-port gigabit switch

I would've preferred a thinner one, with all 16 ports in one long row like with the Netgear GS116E which is less deep to boot, but the Netgear is 30% more expensive.

- Corsair Builder CX430 Bronze

It's the cheapest (kinda) name-brand PSU, well reviewed and provides sufficient power to the cluster. This PSU is going to power *EVERYTHING* so the 5v rails is actually the most important one in the bunch considering each Odroid can claim 2A. This PSU can deliver 20A on that rail so allowing some spare room for the MoBo and potentially the switch (not sure if it's 5 or 12v) I could manage a cluster of 8 off this one PSU. If I want to go beyond that, I'll need to change the power solution here.

- 6x Odroid U3

Why six? Well, looking at the measurements of all the parts the switch will be defining the total width at just over 28.5cm (29.4 for tplink, 28.8 for netgear). The depth will be defined by either the Mobo (17cm) or the switch (18cm for the tplink). The dimension of the Odroid is 83x48 and unknown height. Due to the placement of the ethernet port on the long side of the board, the alignment of the fins of the heatsink and my desire to have the airflow going from front to back I want that long edge to face forward. If I place the boards on their side I lose the aid of convection and the height of the cluster jumps up by almost 9 cm. So I want them horizontal, wide side front. Given that I can place 3 next to each other before I need a second row. But this would make a single odroid layer just 7 cm deep (need some room behind it to plug into a power solution) which is a bit wasteful considering the total depth is over twice that. So I'm going to have 2 Odroids behind one another. I was going to make sleds anyways to house the device so each of these sleds will now house 2 Odroids, which will of course also mean that when I remove a sled to do something with them I must deactivate and remove 2 odroids which will be something for me to remember... Still, given this setup it makes the ideal amount of odroids to start out with 6, allowing me to grow this at a later time with another layer of 6 still leaving me with 3 unused ports on the switch to hook other stuff into (like a truly dedicated file serving device or a secondary cluster or something though I'm going to have to win the lottery first).

I'm not sure yet if I want the switch above or below the odroids, but the regular Mobo will probably go on top with the PSU next to it and 2 harddisks positioned somewhere nearby, all due to expected heat generation. It's going to be a bit cramped so I might have to rethink that setup (maybe have the harddisks under the MoBo?) but I have several ITX boards, spare HDs and a few old PSUs that I can use for a mock-up.

Total price for all this is roughly 500 euros, with potentially another 100 if the tax man is being a dick.

Update:

I've decided to replace the single TP-Link 16-port switch with 2 Netgear GS108's.

Upon closer inspection I found that the TP-Link runs off of 220V which doesn't match my intent to have everything running off the main PSU so the Netgear GS116 floated to the top of my list seeing that it runs off a 12v wall wart which I'd replace. Problem as I mentioned before is that these things cost quite a bit of money, so I took a closer look and found that getting 2 8-port Netgear GS108's would solve all my problems. The width of the combination of the two is, at just under 32 cm, slightly wider (=good since more space for MoBo and PSU), still runs off of 12v, still as thin as the 16 port Netgear at just 2.7 cm (was 4.4 with the TP-Link), I can put off buying the second one until later and combined they still cost about 40% less than the 16-port one.They're just over 10 cm deep so I have about 7 cm of room behind the switch which can be put to good use for any leftover cabling, the power solution to the Odroids and potentially the additional power brick for the second batch of Odroids.

The two switches combine to 16 ports total but I'll need one from each to make it a single network. There's a slight performance penalty to this, but the network speed isn't the main focus of this project and in the grand scheme of things the impact will be negligable. That leaves 14 available ports, of which 6 on each switch will be claimed by a row of Odroids leaving 2 ports: one for the mobo and one for expansion.

I think I should try to make some properly-sized boxes from cardboard for the various parts and see if I can do an actual lifesize mockup.

Edited April 7, 2014 by Cooper

[james92]

That sounds soooo nice man :D i really wana get a U3 they look sooo nice and 6 would be a pleasure to own ha

i can tell you have a GREAT understanding of all this :D im not sooo nifty and swifty ha

I would love your opinion on this question your definitly the man to ask this question ha

What do you think about a cluster of odroids for mining do you think it would be worth it because my friend who showed me the odroid cluster has been saying for ages that it would be worth it to get 20 or so odroids in a cluster for mining and i could borrow the money to set that up but would it be worth it i see some big asshole companys have MASSIVE clusters of pi's with hubs full of asic usbs sooooooooo much power

[cooper]

I keep reading that for mining and actually making a buck you're pretty much required to get an ASIC.

In this thread someone's exploring the idea of using cgminer on his Odroid U2.

To make the math easier, let's assume each core delivers 500kh/s. At 4 cores that means 2mh/s which according to The BitCoin Calculator means that my 6-droid cluster would pay me $0.01 per month. And to get that I'd be running 24/7 at 6*5v*2A=60w when counting only the Odroids in full swing (although it's probably closer to 1.5A so that's 45w). In a month that means a total power use of just over 10 KW (or rather 7.5KW) and at the current dutch energy prices that's about 0.85 (more like 0.65) euros in power.

So yes, it's a fast system, but, for mining, not a very economic one.

[GuardMoony]

yeah, you may forget mining on this.

You seen the new rasp pi ? its like a plugin pci. Redesign a new base board to plugin like 16 of these and you got your superpi :)

Edited April 7, 2014 by GuardMoony

[cooper]

I didn't see that one yet. The problem with the Pi is that it's a comparatively shit CPU with a decent GPU. The Odroid has a decent GPU aswell but since that's a Mali it's currently a black box you're not allowed to play with whereas Broadcom recently took the lid off of their sandbox and is inviting people to come and play.

If your intent was to make a media player, I'd go for the Pi.

However my intent is to play with java software and maybe some apache webserver and caches and stuff on this and at least for the foreseeable future the computational prowess of that GPU will be out of reach in that context.

Corenumber and frequency wise, you'd have to combine 10 Pi's to get the CPU power of one Odroid. I think I'll be doing allright with these for now.

[GuardMoony]

Hmmm think ill need to look a litle deeper in the Odroid.

[james92]

I dont know anything about hardware so what would you connect to that pi to make it powerful and do you connect these things to the GPIO pins cause my odroid has them aswell and Id love to give that a kick in power :D

Edited April 8, 2014 by james92

[cooper]

The problem with all of these dev boards is that they're a fixed package.

With your regular PC you can buy a separate everything. Motherboard, cpu, memory - it's all separately replaceable/upgradable.

Want more memory in your Raspberry/Odroid/Wandboard/AllWinner/whatever? If you're very lucky you can buy a more expensive version of the same board which has it, otherwise you're SOL.

Want a more potent CPU? Tough. Try overclocking like this guy that got his Pi up to 1.5 GHz

Bottom line:

If you want the highest performance, whip out your wallet and get a potent CPU made by Intel and an equally potent GPU.

If you want decent performance at a modest power draw for a little less money, get one of those AMD boards with an on board GPU.

If you don't care a lot about performance but want a low power draw even if it costs more than the previous option, find an ARM dev board that is capable of doing what you want but be aware that it's a dev board. It means you probably won't be able to get an identical one a year from now. A lot of things will need some work before they'll behave properly and it may turn out that nobody can be arsed to do that work for you so you must either hire someone to do it or do it yourself. If that's too much of a sacrifice for you to make but you don't want a board from one of the other two categories, get a Pi. Now that it's a fully open platform there will be a lot of interested parties willing to invest time and money to take full advantage of the board which will benefit everybody in the long run including you.

[GuardMoony]

Hey cooper. Found a interesting power supply you might can use.

http://www.circuitspecialists.eu/power-supplies/switching-power-supplies-encl?output=27

if you get a switch on 5V. You could run it all of that

[cooper]

Since the build includes a regular off the shelf mini itx celeron (atom, really) board I'll be using a standard ATX powesupply for that. I could try using a PicoPSU to power the board, but I also need to power 2 spinning disk HDs so I'd prefer getting a regular PSU for that. Given the presence of that PSU, running the switches off of one of its 12v rails is trivial.

The 5v power rating of the PSU is also such that I can run my 6 Odroids off of that so initially I'd be done with just that. If (when) I get the next 6 I'll need additional amps on the 5v line and then those PSUs you're suggesting here become mighty interesting. They make a 5v 40A PSU! 200W @ 85% efficiency. That's damned good in my book, though active cooling to get the heat out would give me peace of mind. If I run into trouble running the droids from a spare power brick I have and for which I did locate the conversion plug thingies (didn't check if 5v was amongst them... stupid) these most certainly are contenders. Thanks!

[cooper]

@GuardMooney: you got me thinking. I would prefer to have everything running off of the one PSU. Its power rating should easily suffice.

What I could do is make each slide that holds 2 Odroids draw 12v from the power solution and use two of these to down-convert it to 5v.

It'll cost some but I wouldn't have to include a separate power brick anymore.

Edited April 12, 2014 by Cooper

[GuardMoony]

Yeah. also possible.

I myself am looking for a small BOINC group/cluster. Maybe running 1 of them as GW. And using networkboot to load the others. For switches ill probaly gone use zyxel ES-108A ( dont need 1gbit on them )They run of 5V 0.6A

Edited April 12, 2014 by GuardMoony

[cooper]

For network boot I'm going to equip the Odroids with a tiny local SD card and they'll use tftp to grab their kernel off the main server, assuming I can get this process to wait for / retry until the main server manages to power on and start running the service, otherwise I'll just make them boot off of their SD cards and leave it at that. (they don't have an EEPROM on the network device, obviously, and they also all have the same MAC address at boot so that'll require a bit of tweaking aswell)

That looks like a nice switch with indeed about half the power draw of its gigabit counterpart but from what I can gather here it'll be a hard find since no shop here is stocking it anymore. A good and, at least over here, still very easy to acquire alternative is the TP-Link TL-SF1008D which, like the Zyxel, runs off of 5V 0.6A. I personally also think it's prettier with only ports on the front and the power plug at the back, which should also make it less wide than the Zyxel.

To me the GBit is important as I want the nodes to be able to fill their network link and the server to still be able to provide its data to all of them simultaneously. I do suspect that the power draw will be less than the 12W advertised since only 1 or 2 of the 8 ports will be populated with an actual GBit device.

What are you considering using for computing nodes?

Edited April 12, 2014 by Cooper

[GuardMoony]

As far as look into making computer nodes / clustering. The penalty is to big on the odroid.

Towards the gigabit. Why get a gigabit switch when the odroid got only 100Mbit connections. Ok the processor has a higher switching pass-through. But will you have so much data traffic ? that's the main question

As for the switches. Because of my job i can the zyxel without any problem. the tp-link is a good alternative.

[cooper]

What penalty are you referring to? Purchase price?

I don't know for certain if I need the bandwidth, but the price difference between gbit and 100mbit is less than 5 euros for both switches combined and the power draw isn't a big concern for me. So I see potential benefits to using gbit and very little in terms of drawbacks.

Edited April 13, 2014 by Cooper

[GuardMoony]

http://www-public.it-sudparis.eu/~trahay_f/odroid_supercomputer/

Performance on one node (4 cores):

Odroid: 3.79 GFLOPS

Performance on the whole cluster:
Odroid (6 nodes, 24 cores) : 16.13 GFLOPS.

As I suspected ...

Running flat-out processing tasks individually, 6 ODROID x 3.79 GFLOPS = 22.74GFLOPS
Running 6 nodes: Odroid reaches 16.13 GFLOPS

Nodes lose (22.74 - 16.13) 6.61GFLOPS vs individual ODROIDS running flat-out.

... in fact a lot worse that I thought. Nearly 2 ODROIDS-worth of effort lost.

So from a test that ran last year you can see the results if you would cluster the odroids. It has a big drop in performance compared to

running them standalone. But this was on a test with high traffic if i remember. so maybe its less on pure computing setup

Edited April 13, 2014 by GuardMoony