visionk1 Posted December 30, 2011 Share Posted December 30, 2011 Hey guys I have read the forums for quite some time now but this is my first time posting. So hear it is I am trying to see what is the best way to run 100s of VM's at a time I am looking at running about 300 vms. My host can be either Ubuntu 11.10, Mint 12, or Windows 7. As far as for my guest OS' I would be running xp or any small Linux distro. The only real functionality that I need is that each guest OS needs to be able to run a web browser with flash and each has to be able to run an instance of Tor. I have a box with an amd phenom ii x6 t1090 overclocked to 4.4Ghz 16Gb Ram, I will be installing a new boot drive some type of raid 0 configuration with 2 ssd's and two 2Tb 7200 rpm hard drives for secondary storage. I figure I will be needing to make a cluster since I will probably not be able to run everything off of this one unit but I was just wondering what are some of you guy's ideas on how to make this work. Can I just use virtual box to run everything or do you guys think that Proxmox is the way to go?? One last thing what type of Internet connection would be best suited for this type of environment? I will be running this from my house so I cant get too crazy but eventually I guess we could migrate to a more enterprise friendly location. As I said any input is greatly appreciated and if you have been doing anything similar share, why are you doing it and how are you deploying it. Visionk1 Quote Link to comment Share on other sites More sharing options...
Sparda Posted December 30, 2011 Share Posted December 30, 2011 For this amount of virtual machines you probably want to look at ESX (ESXi may do, not sure). Quote Link to comment Share on other sites More sharing options...
Infiltrator Posted December 30, 2011 Share Posted December 30, 2011 (edited) For this amount of virtual machines you probably want to look at ESX (ESXi may do, not sure). ESXI is definitely what you need, running that many vm instances will require a few cluster of servers for load balancing and performance purposes. I would suggest you to buy a couple of Dell servers, they are a bit cheap compared to other vendors and they can also be customized to your needs. Check out the Dell's T610 servers, http://www.dell.com/au/business/p/poweredge-t610/pd they are ideal candidates for small to medium server virtualization projects. Since you are running that many VMs, you should maximize the ram capacity in those servers, as well as look into ways for maximizing your server I/O performance, or things will get sluggish. SSDs are very good candidates for maximizing throughput, you might want to look into NAS devices, that supports them. Ensure your servers are using 10Gigabit Ethernet adapters, instead of 1Gigabit or else they will get saturated very quickly, if the servers are under heavy load. And that could affect the performance of other VMs as well Edit: You will find these links quite informative. http://www.vmworld.com/thread/3184 http://www.itworld.com/answers/topic/virtualization/question/how-many-virtual-machines-can-i-run-one-system Edited December 30, 2011 by Infiltrator Quote Link to comment Share on other sites More sharing options...
visionk1 Posted December 31, 2011 Author Share Posted December 31, 2011 Hey guys thank you for your input and for those insightful links. Let me first answer the question of the budget for the project and then I will give a little bit of the background story behind my idea of running this many VM's. So there is no real budget for this project because it is a proof of concept, the idea is to find the cheapest way of being able to run so many machines 24/7 for at least six months. So how did this idea come to be? Well A very good friend of mine was looking into hosting various events such as tournaments for games, motivational speaking, security talks and countless many other events and he thought it would be a good idea to stream them live over one of the so many sites available. So me being me and arriving at a very similar conclusion to Brian Brushwood's, the guest on this week's episode, that if "other people are watching" more and more will follow. Which by the way was so funny when I watched the last episode on Saturday morning. I was like yeah I have been saying this for a while and have tried, on several occasions to prove this theory in a real world scenario. So this is the next evolution of the concept ported over for a technology driven world. Anyways So when my friend and I discussed this he agreed and asked if there was anyway to generate views. Immediately I had the idea about the virtual machines. As previously stated each VM has to be running its own instance of Tor and has to be able to navigate to the URL (I also want to have all of this automated so all one has to do is start the machines). Please feel free to improve on the idea, maybe you have not had the best experience with Tor and want to recommend other free open source products, or maybe you have an idea for what OS to run. Yet still maybe you can come up with the best way to automate these tasks; in short what do you guys think, are there better ways to achieve this, or am I out of my mind? I also would like to incorporate that concept of crowd sourcing using Facebook and other social media to generate momentum but that is for another day. Again thank you for your responses and for allowing me to pick at your brains... Quote Link to comment Share on other sites More sharing options...
Sparda Posted December 31, 2011 Share Posted December 31, 2011 You can control your exit nod with tor, which is really the most important part of this. The use of VMs is largely unnecessary as Chrome (and other browsers) in 'private mode' will appear to be a separate computer to most web service providing it has a different public IP (assuming that matters). Quote Link to comment Share on other sites More sharing options...
telot Posted December 31, 2011 Share Posted December 31, 2011 (edited) Hey guys thank you for your input and for those insightful links. Let me first answer the question of the budget for the project and then I will give a little bit of the background story behind my idea of running this many VM's. So there is no real budget for this project because it is a proof of concept, the idea is to find the cheapest way of being able to run so many machines 24/7 for at least six months. So how did this idea come to be? Well A very good friend of mine was looking into hosting various events such as tournaments for games, motivational speaking, security talks and countless many other events and he thought it would be a good idea to stream them live over one of the so many sites available. So me being me and arriving at a very similar conclusion to Brian Brushwood's, the guest on this week's episode, that if "other people are watching" more and more will follow. Which by the way was so funny when I watched the last episode on Saturday morning. I was like yeah I have been saying this for a while and have tried, on several occasions to prove this theory in a real world scenario. So this is the next evolution of the concept ported over for a technology driven world. Anyways So when my friend and I discussed this he agreed and asked if there was anyway to generate views. Immediately I had the idea about the virtual machines. As previously stated each VM has to be running its own instance of Tor and has to be able to navigate to the URL (I also want to have all of this automated so all one has to do is start the machines). Please feel free to improve on the idea, maybe you have not had the best experience with Tor and want to recommend other free open source products, or maybe you have an idea for what OS to run. Yet still maybe you can come up with the best way to automate these tasks; in short what do you guys think, are there better ways to achieve this, or am I out of my mind? I also would like to incorporate that concept of crowd sourcing using Facebook and other social media to generate momentum but that is for another day. Again thank you for your responses and for allowing me to pick at your brains... So let me get this straight - this server with 100s to 1000s of VMs is going to artificially inflate social presence on a website/forum/blog/whatever in order to increase interest, or "trend" a specific idea to the mass public? If I'm correct, I think this is a great idea and will really expose holes in the social nature of the web (just like that annoying hair guy on this weeks ep, like you said...god I hate his hair). Screw proof of concept, you could easily market this to SEO and other internet marketing sites...captcha would be the only hurdle to automating this process in my mind, but theres a number of workarounds by the looks of it (just did a quick google search, I've never had any experience using the techniques however). As for the chrome private browsing vs tor - I would combine a number of different routes to anonymizing the traffic - that way all the traffic isn't generated by just one type of browser, as 100% chrome usage is not common on the internet and would be a red flag to any admin looking at traffic. So you could have some chrome private browsing, some firefoxing tor and maybe even some i2p to switch it up a bit. Keep us updated on your progress please, I'm very interested to see how far this can go! telot Edited December 31, 2011 by telot Quote Link to comment Share on other sites More sharing options...
Infiltrator Posted December 31, 2011 Share Posted December 31, 2011 Why don't you just download the TOR Browser bundle and have it installed on each VM. Practically, I don't think its possible to use Chrome with Tor unless there is plug written for it. On the other hand, I've been using the TOR browser bundle for quite sometime now and really happy with how it works, they only issue is the browsing speed which can be painfully slow sometimes. Anyway, just my opinion. Quote Link to comment Share on other sites More sharing options...
digip Posted December 31, 2011 Share Posted December 31, 2011 For multiple VM's running at the same time, like 300, you need several things. 1, memory. I've got 16GB in my home desktop, and while I can run many VM's it certainly won't be running 300 at the same time. At lease, not full blown desktops. If you had say, headless servers that ran on 64mb or ram, then maybe, but if you need them all to be able to run TOR and a Browser, then you are talking full blown desktop setups, which at a minimum need at least DamnSmallLinux with the required libs for TOR. If you can get TOR to run in windows 95, then you could use OperaTOR, but I don;t think WIndows 95 has the proper TCP/IP stack and socks capabilities. (Offtopic, OperaUSB by itself, will still run on windows 95 and allow flash, which is one of the things I love about Opera :) ). On top of memory, you need a beefy CPU, or multiple CPU's, like in a 2u rack. And 3rd, you need disks. You can run a few VM's off one drive with not much issue, but 300 running at the same time, would need many independent disks to get decent performance, or a few small SAN's/raids for IO. Lastly, you need a fat pipe. You won't be bridging 300 machines on one gigabit NIC. You would still end up with broadcast storms, collisions and network performance issues, just like you would on a real network, so this rack setup will need some decent network hardware and then switch(es) and router(s) to balance the load. You would pretty much be doing a DoS to yourself before you leave your network if 300 of these machines were trying to leave your home at the same time. Basically, you would want to have what you would in a corporate lan, which in most cases is T1 or better in and out of the network. Quote Link to comment Share on other sites More sharing options...
digip Posted January 1, 2012 Share Posted January 1, 2012 Don't forget 10-20 thousand dollars to make it all happen with. Not necessarily, but it won't be dirt cheap either. You can get most of the server stuff cheap on ebay these days, and cisco stuff is on there as well but will be the most expensive items on the list. If he does this all openSource, then the cost is cut even further with no license fees to worry about and OS purchases. You should be able to do it for under $5000 if you get all used hardware. Used Dell Power Edge severs on ebay can usually be found for around $300 bucks these days. Most expensive parts will be switches and routing equipment, as well as storage. An HP storage works enclosure would be cheap, but the drives will add up in the end. Sometimes you can get them WITH the drives for dirt cheap though, but no telling how many drives work and how many won't. Quote Link to comment Share on other sites More sharing options...
Infiltrator Posted January 1, 2012 Share Posted January 1, 2012 (edited) Unless you buy second hand hardware like Digip mentioned, it won't be cheap to set up a virtualized infrastructure like that. Edited January 1, 2012 by Infiltrator Quote Link to comment Share on other sites More sharing options...
digip Posted January 1, 2012 Share Posted January 1, 2012 Well, if he goes with something like traditional t1/t3/oc3 then he'll pay through the nose. If he did a dedicated cable business line like comcasts new docsis3 at 100 down 10 up, then like 400/month and he would be faster than T1 and T3, but on par with OC3 at a fraction of the costs. I've seen OC3 plans start at $3,000/month and that doesn't include network equipment, so a cable provider would be cheaper for small businesses these days and should suffice 300 nodes. Might be easier, to go cloud services in this case if the monthly budget isn't an issue, hosting that offers virtual machines you setup and maintain yourself though is generally expensive at 300 nodes for cloud hosting, something like $4000/month at least, but would accomplish all of his needs. The other option, hack 300 computers, create your own bot net and take over the world /queue evil Plankton music from SpongeBob Squarepants Quote Link to comment Share on other sites More sharing options...
Infiltrator Posted January 4, 2012 Share Posted January 4, 2012 The other option, hack 300 computers, create your own bot net and take over the world /queue evil Plankton music from SpongeBob Squarepants Or you could definitely do that, it would be a lot cheaper to maintain, with no hardware or costs involved. But it would be extremely illegal thing to do. Quote Link to comment Share on other sites More sharing options...
digip Posted January 4, 2012 Share Posted January 4, 2012 Or you could definitely do that, it would be a lot cheaper to maintain, with no hardware or costs involved. But it would be extremely illegal thing to do. That. was. a. joke....hence the queue Plankton evil theme music from Spngebob... Quote Link to comment Share on other sites More sharing options...
visionk1 Posted January 21, 2012 Author Share Posted January 21, 2012 Hey guys sorry I have not posted anything in a while I have been busy trying to deploy this in a very organized way. So the first thing I had to do was decide if I really wanted to go ahead with this project given the estimates that everyone had given me in addition to my research. The answer was yes so I have been looking to price out some good i5 cpu's in an effort to get a cluster going and see if this would work in the way we want it to. The other thing was deciding what OS to use and I am still not sure on if I want to use Lubuntu because it works well but is rather slow at 256mb, at the moment I am demoing 4 win7 with 512mb and 1 Lubuntu with 256mb and I am using team viewer on all to rdp in. I wanted to do a test run with several machines and see how my one Machine will handle everything and so far with 5 VMs everything is running just fine. In addition to this I was doing some more research and most sites even very popular ones like YouTube and UStream allow for several connections from one given IP add so if we were to get several students at a local college to log in and stay on for streams or look up you-tube videos we posted they would count them all as different views. In light of this new information I concluded that the same amount of impact could be achieved with fewer computers. I am still working everything out but the fact that this can be done with fewer computers means that I am able to give out even 512mb per user and this makes everything work much better. So I know someone talked about bot-nets above and I thought this could be a great idea seeing that this is how they are used in the real world so what about creating my own virtual bot-net what do you guys think and how would you go about this if you have any article please share because I have little experience with this and I would love to get this going. I really like the idea of a command and control center so I can get all the VMs to go to the target site with only one command. again think you everyone for your input and sorry if i typed anything wrong it was done in a hurry... Quote Link to comment Share on other sites More sharing options...
Infiltrator Posted January 22, 2012 Share Posted January 22, 2012 The bot-net idea was definitely a joke and I do not recommend doing it in the real world, as it's extremely illegal and could get you jailed for. When creating a bot-net you are essentially infecting other people's computers with Trojan/Malware and then controlling them remotely from a C&C (command center). Now you can achieve the above, by building a cluster of 2 or more servers and using Vmware ESXI to do all the load balancing work for you. That will help minimize the load and increase the number of VMs you can run in a more efficient way. Quote Link to comment Share on other sites More sharing options...
BattZ Posted January 25, 2012 Share Posted January 25, 2012 Color me intrigued by this topic. I've looked a little into how google page ranks and what-not, but couldn't find a definite answer. If a small little tech firm made something like this, and kept running this virtual botnet and clicking on their link from google searches, their page rank would go up? So soon they would be in the top 5 for whatever their little company does? Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.