whitehat Posted November 5, 2012 Share Posted November 5, 2012 I have various VPS and also many comps connected to my home network. If I understand correctly, for my servers to be a "cloud" they should be sharing resources like memory and processing power through cluster computing (?). I really wish I had learned this stuff in school LOL. How do I do that? I know there's something called Cloud Linux... do I need a special OS like that or can I leave them all and somehow run some program that makes them into a giant super-powerful cloud? I could probably pull off automatically sharing storage space easy enough with some shared folders or standard file exchange tools, but is it really possible to share RAM/CPU?? Quote Link to comment Share on other sites More sharing options...
digip Posted November 5, 2012 Share Posted November 5, 2012 (edited) The internet, is the cloud, not to be confused with parallel or clustered processing, but if you need to do something that requires distributing computing power, to crunch data, look into Amazon cloud services. Basically lets you use their network of multiple systems to tie all the data together for clustering, or you could do it at home depending on the type of data you need to crunch, with GPU processing and buying a few CUDA based cards, and compiling whatever it is you are trying to crunch, to work against the GPU vs CPU. These things like the term "cloud" are all bs though. Its marketing. If you need shared resources to have multiple clusters or computers do the work of one machine though, switching over to something like Amazon's web services will let you do this, but it won't speed up your home PC for normal tasks or such if thats what you we're looking to do. Just depends on what it is you are trying to accomplish. If its backup storage and redundancy, there are tools and even services that let you sync up with cloud based storage though, so if you save a file locally, it also saves it online. Edited November 5, 2012 by digip Quote Link to comment Share on other sites More sharing options...
whitehat Posted November 6, 2012 Author Share Posted November 6, 2012 Thanks for the reply. Yea, I also think the term cloud is a metaphor for the non-technical, but I was told (correct me if I'm wrong) that it does refer to cluster computing. I do know of Amazon/AWS but I want to make the computers/servers I own into a cloud just for the educational benefit of doing it. I am a scientist and I do have a lot of data to crunch, but that's not really the motivation. I don't know much about GPU. Is that what I should read about to get caught up and make my own cluster with shared resources? I apprecaite the leads. I don't have time to read about it this second but I will tonight or tomorrow. Any more topics I need to look up to accomplish this? Quote Link to comment Share on other sites More sharing options...
digip Posted November 6, 2012 Share Posted November 6, 2012 (edited) Only reason I mentioned using the GPU vs CPU, is 1, cost of multiple machines if all running at home, most CPU's run idlea half the time and bottlenecks become HDD's and data sent back and forth from disk to CPU, 2, some things that can be compiled to run using OpenCL, run faster on a few high end graphic cards since they have tons more parallel processing power than a CPU does, and often more power then they do on supercomputers and other home based clusters. Now, cluster together a few machines with multiple Cuda based cards, you trump the factor of a normal cluster even more, but thats only if the code you are crunching runs on the GPU vs CPU. As for "cloud" vs "cluster" most people today think of cloud in terms of either 1, storage, or 2, web apps and services that run on servers in the "cloud" or, internet, to do the work for you, so you don't have to build your own data center, which has its pros and cons. Con being, when the cloud is down, so is your data and access to it. Its most often, not redundant enough for mission critical data, nor do any of the cloud service prividers, offer any guarantees. In fact, they state in the fine print, no guarantee of security, or data integrity on nearly all of the companies that offer it. Only people that offer disaster recovery solutions, that mirror your systems and hardware the same as you would at your own data center, have that kind of ability, like SunGuard for instance. SunGuard is not a service I would consider "cloud" based though as much as a backup solution as well as disaster recovery setup. Say you're a bank, and your data center blows up or catches on fire. You can take all your backups or data already migrated to SunGuard, go over, load your data on their systems, and have all your bank branches back up and running usually within 24 hours. Edited November 6, 2012 by digip Quote Link to comment Share on other sites More sharing options...
whitehat Posted November 6, 2012 Author Share Posted November 6, 2012 (edited) OK, so I do have a GPU card in my laptop and servers except for the 4 that are headless. I am not really interested in a commercial cloud, so we don't have to worry about that. I just want to get my personal computers all working together as one cluster/cloud. I see that OpenCL is a big topic... an entire computer language. But what software can I run to make all my computers act as a cluster? I guess I need to try CloudLinux? But I don't want to replace the existing OS's... GPU info on main comp: Name VNC Mirror Driver PNP Device ID ROOT\DISPLAY\0000 Adapter Type Not Available, RealVNC compatible Adapter Description VNC Mirror Driver Name Intel® HD Graphics Family Adapter Type Intel® HD Graphics Family, Intel Corporation compatible Adapter Description Intel® HD Graphics Family Adapter RAM (262,217,728) bytes Name NVIDIA GeForce GT 555M Adapter Type GeForce GT 555M, NVIDIA compatible Adapter Description NVIDIA GeForce GT 555M Adapter RAM (1,073,741,824) bytes Edited November 6, 2012 by whitehat Quote Link to comment Share on other sites More sharing options...
digip Posted November 6, 2012 Share Posted November 6, 2012 (edited) Could setup the main workstations to run VMware, and cluster them that way via cloudlinux and control them from the laptop remotely. Clustering still works in virtual machines, although not as fast or great as on native hardware, many clusters in the "cloud" as we've been talking about, are all VM's anyway. Lots of web hosts use VM's of some sort anyway, whether people know it or not, most of the time their websites are not running on native hardware since the costs to host 1,000's of sites, can be reduced by using virtualization. If you wan to go native, you could dual boot linux and whatever else is on the machines now, just have to shrink the partitions and make space to install some other OS that does clustering, or if its Windows based, find the software to do what you want. Fold at home for example, used distributed computing power, so its depends at what level you want to go with "clustering" to get the work done. Folding at home was just a screensaver, that when your machine was idle and the screensaver was running, it would then run the folding at home software in the background and do tasks it was given for the folding at home project and then send in the results later when you were back online using the machine. Windows server can also do windows clustering too though, not just linux, but linux is the free option. Google clustering in general and check youtube. There are some videos on setting them up using either windows or linux. Most clustering in Windows is for failover services and redundancy though. http://www.mcsr.olemiss.edu/bookshelf/articles/how_to_build_a_cluster.html http://helmer.sfe.se/ Edited November 6, 2012 by digip Quote Link to comment Share on other sites More sharing options...
whitehat Posted November 7, 2012 Author Share Posted November 7, 2012 Run a Cloud OS as a VM -- I LIKE IT! Thanks DigiP. You came through. CloudLinux is apparently very propriatary. I just went to get it, it's a no go. I will be hunting distro watch for a free alternative, please let me know if anyone knows of one. Quote Link to comment Share on other sites More sharing options...
digip Posted November 7, 2012 Share Posted November 7, 2012 I don't think you need a specific distro though. I think it can be done with any linux distro and the right packages to manage the hardware, but I've never tried to build one. Only read and watched stuff on the subject, but check those links above, they might include something of value to point you somewhere to do what you want. Quote Link to comment Share on other sites More sharing options...
whitehat Posted November 8, 2012 Author Share Posted November 8, 2012 (edited) I've been trying PelicanHCP. It's one of two listed by DistroWatch.org for clustering, cloud, and distributed OS when you search on those terms. It's pretty badical, except that there's not much support and there's an issue with the default pw when run as a VM from .iso, but if you make the files yourself you can fix it. The other choice was something called RockyCluster, but it didn't do so hot when I tried it and this is supposed to be the higher performance cluster OS. You just pop this only any machine in your LAN/WLAN and it automatically clusters. It's really pretty hot stuff!! http://pelicanhpc.788819.n4.nabble.com/A-PelicanHPC-Setup-That-Works-for-me-Hardware-and-Software-Configurations-td2069021.html Edited November 8, 2012 by whitehat Quote Link to comment Share on other sites More sharing options...
yukondokne Posted November 9, 2012 Share Posted November 9, 2012 can i ask what your trying to do? i only ask because normal applications (games, movies etc..) wont run over a cluster configuration. if your trying to compile huge data (video rendering, 3d modeling, web service load distribution) then there is applications that are built for clusters that will function on your home cluster. i just don't want to see someone put all the work into building a cluster, and realize it will not so what they thought it would do. Quote Link to comment Share on other sites More sharing options...
digip Posted November 9, 2012 Share Posted November 9, 2012 can i ask what your trying to do? i only ask because normal applications (games, movies etc..) wont run over a cluster configuration. if your trying to compile huge data (video rendering, 3d modeling, web service load distribution) then there is applications that are built for clusters that will function on your home cluster. i just don't want to see someone put all the work into building a cluster, and realize it will not so what they thought it would do. QFE -Yeah, unless its for services like 1 - domain failover, round robin, or redundancy, colocation hosting, or 2, crunching large data computation, password cracking, etc, its not going to be very useful other than a learning experience, but he did mention wanting to do it just for learning, even though he has data to crunch. I do know of Amazon/AWS but I want to make the computers/servers I own into a cloud just for the educational benefit of doing it. I am a scientist and I do have a lot of data to crunch, but that's not really the motivation. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.