Adam Ierymenko Posted July 29, 2014 Share Posted July 29, 2014 I'm new to the forum. A friend of mine suggested that I check it out, and that I post my current project here. So here it is: https://github.com/zerotier/ZeroTierOne https://www.zerotier.com/ It's an open source Ethernet virtualization engine. It works by emulating an Ethernet switch (layer 2) over a peer to peer network. All the core infrastructure code is open and the service is free, though I've got a freemium model in place for networks that you create and manage through the control panel at zerotier.com. (Charging for convenience.) It kicks in with >10 devices or Ethernet bridging enabled. The charges only apply to private networks. You can create public networks of unlimited size for free. There's also a public network called Earth: https://www.zerotier.com/earth.html It's sort of like a global coffee shop WiFi. Obviously make sure your machine is secure before joining. People who want to run their own network configuration servers can do so by digging into the netconf-server/ subfolder in the source and following the directions there. Those directions are rather sparse so far, but it's probably enough for experts to go on. I can answer questions if anyone wants to play with that part. I'm curious to hear this forum's feedback, since it seems full of people who get this kind of thing. So far I've got reports of people running it on Raspberry Pi, using it to play old school games that only support IPX networking (it emulates Ethernet so it doesn't care what protocol you use), etc. Quote Link to comment Share on other sites More sharing options...
cooper Posted July 30, 2014 Share Posted July 30, 2014 Some thoughts: You chose to use Eliptic Curve Cryptography even though trust in ECC has pretty much evaporated thanks to our loyal friends in the NSA. Read this paragraph of the Wikipedia article on ECC, particularly that last section. And example of someone voicing his distrust is this Bruce Schneier post. To your credit, you're using DJB's work rather than the NIST ones, and I think you should make this explicit in your non-technical bit aswell since some of the "ECC is bad! Ooooh!" FOX-level reporting might have stuck with some people. Location of your private key under Linux is /var/lib/zerotier-one/identity.secret but I feel somewhere under /etc is far more appropriate. How do you deal with a key that's expired? And what about when it's become exposed? Can you revoke your key somewhere? Can someone see that a key was revoked? Why hardcode the global supernodes as opposed to using some configuration file for this? Any stats on usage so far? Like what's the highest number of nodes that have been on the network thus far? I still have some problems with the usage scenario. What, exactly, is the problem you're trying to solve? Since you yourself claim your code is pretty secure but shouldn't be considered sufficiently trustworthy for ultra-secret stuff (simply because it hasn't been reviewed enough yet to warrant it) why would I want to prefer this over a regular VPN? The only reason I found mentioned was ease of use. It's an interesting project and you've clearly put a lot of work into it. I hope it pans out. Quote Link to comment Share on other sites More sharing options...
Adam Ierymenko Posted July 30, 2014 Author Share Posted July 30, 2014 Thanks for the responses. All great questions. I'll start with the last. Ease of use and user experience is one of the main goals here. Existing virtual networking (both VPN and enterprise-scale stuff) is inconvenient. It's annoying even for experts, and impossible for non-experts. What I want is to make virtual networking as easy as, say, joining a Skype conference call. It's not there yet but the technology is designed to enable that level of usability. It's not that you can't do what this does with other tools, but doing so takes hours of jiggering with config files and port forwarding and other stuff and is basically impossible for regular people. Usage: right now there's about 400 users online all over the world. It's gotten fairly popular among Chinese business users using it to collaborate with non-Chinese users over the wall. There's a few paying customers but not many. I have not advertised it really heavily yet, since it's still in beta and I want to make sure all the major issues are ironed out. The actual protocol has been very stable for a while... it's been over six months since there's been a significant bug report. But there are still rough spots around OS integration on Mac and Windows and a few other things I want to polish up before inviting in the hordes. On test networks in VMs I've spun up over 100,000 nodes. I'm also doing some refactoring right now to make it easier to test on a 100% emulated pseudo-net, which will allow me to run giant tests with tens or hundreds of millions. My calculations of bandwidth show that the existing supernode architecture can accommodate up to about two million concurrent users. After that a bit of rearchitecting will be required, but I already have a strategy mapped out that doesn't require changes to the client-side protocol. Why supernodes aren't in a config file? Goes back to usability. The idea is that the network is globally flat and unified. Anyone can join any network, etc. Fragmenting the supernodes would fragment the network, since they're the anchor points used for rapid provisioning of p2p links. (They also relay if you can't establish p2p links, which is about 3% of users right now.) If an OSS user wanted to change Defaults.cpp and recompile, they could. In the near future I'm going to make the supernode list hot-upgradeable via a signed configuration that can be pushed out to the network so I don't have to do a software release if the supernodes or their IPs change. A supernode is just a regular node designated as such. They run exactly the same software, so there's no closed-source code there. The only closed-source code in the ecosystem right now is the web control panel at zerotier.com, but the underlying netconf master that actually allows provisioning of virtual networks is open (see netconf-master/ in the source repo). I might open the control panel up too in the future... not sure yet. Depends on what I figure out revenue model wise. The protocol is designed to enable evolution toward a more mesh-like setup with a reduced or even eliminated role for supernodes, but I'm not willing to sacrifice user experience or speed for that. That makes it a really, really hard problem. I have some ideas but right now I'm focused on UX as I said. If I can get a real business under this I'll have resources to spend on that. Eliminating the supernodes is appealing to me both for the decentralized networking geek / cypherpunk factor and the fact that the supernodes cost me money to run. (Not a lot, but something.) On crypto: I'm not sure I agree on ECC. Unless someone can show a real attack, I think it's FUD. It hasn't been around as long as RSA, but it has existed for quite a while. Not only that, but right now there is a monstrous cash bounty on breaking ECC in the form of Bitcoin. ECC is vulnerable to quantum crypto, but so is DH and RSA, and right now anyone who can develop a quantum computer with enough coherent qubits to crack these can jackpot every cryptocurrency and steal a huge proportion of their current collective value. (If Bitcoin suddenly crashes and D-Wave gets very rich... :) I respect Bruce's opinion, but lots of other very well respected cryptographers including DJB do not agree. I did choose DJB's curves over the NIST ones, both for security reasons and because of the relative cleanliness of DJB's 25519 code vs. the hideous crawling mess that is OpenSSL and most other crypto libraries. I wanted a neatly encapsulated portable implementation. But the bottom line is this: the odds of someone breaking ZT1 by breaking ECC, poly1305, or Salsa20 are almost infinitely lower than the odds of someone breaking it by finding a flaw in the protocol or a bug in the implementation. If you look at attacks against SSL, SSH, etc., pretty much all of them are attacks against the implementation not the crypto. The only crypto attack against any cryptographic protocol that I'm aware of is against RC4, which is widely regarded as a weak cipher. Even those are very difficult and require a lot of traffic analysis, compute power, and man-in-the-middle to have a chance of pulling off. But even RC4 is more than good enough to keep all but the most determined and well-funded attackers out. If your adversary is the NSA or another nation-state, I'd suggest defense in depth-- layer your crypto and use multiple implementations and different algorithms. Also use air-gapped networks, disposable read-only OS installs like TAILS, etc., to make sure the attacker can't just backdoor your system and steal your key. That's the most likely way your crypto will get broken. That's the sort of paranoia that would be required to stay secret against a real, well-funded adversary with cryptographers on staff. I'd also stay below the radar. I'm not aware of any algorithm that can stand up to a rubber hose attack. The reason I include a bit of a warning is that ZT1 is a new code base. I tried to use secure coding techniques throughout and think through everything, and I've run it past some security gurus and none of them could find a problem. But it hasn't been around as long as other things. Personally if I were handling very secret data like credit card numbers or pictures of dead aliens, I would always practice defense in depth rather than relying on one implementation's security. We've seen that even tried and true things like SSL can be riddled with bugs for years before anyone finds them... or until anyone publishes that they found them. I wonder how long the NSA knew about heartbleed, CRIME, and other attacks? Quote Link to comment Share on other sites More sharing options...
cooper Posted July 31, 2014 Share Posted July 31, 2014 I'm still wondering if this is a VPN, a DarkNet or both. By setting up a server for this I'm inviting clients to connect to me. The virtual connection is encapsulated in your protocol that includes encryption. Is this 2-sided meaning my VPN server needs to know (and authorize) the cert of the remote client that's trying to connect to it in order to establish the connection? Is this different when the supernodes relay, by which I mean do I need to authorize the supernodes aswell? You still have the problem of cert distribution, revocation and expiration. When you create the self-signed cert, what's the lifetime of it? Quote Link to comment Share on other sites More sharing options...
Adam Ierymenko Posted July 31, 2014 Author Share Posted July 31, 2014 It's a VPN, though that term has become confusing of late. It's not a darknet, since it does not implement onion routing or anything else to strongly conceal your location. It does encrypt though, and the keys are kept private to each peer. The basic way this works is as follows: (1) When you first start it, it generates a public/private key pair that satisfies a proof-of-work criterion and then from that derives a 5-byte identity. (See Identity.cpp -- it's more involved than it sounds, and it's very hard to duplicate an identity). (2) It then sends HELLO to the supernodes, letting them know that you exist and what your public key is. The supernodes now know that address "deadbeef11" has public key X and is at 1.2.3.4 port 6666 (to make up an example). The supernodes remember address->identity mappings, so it's first-come-first-claim for addresses. With 5-byte addresses there can be up to 2^40 or 1,099,511,627,776 devices online. If by a small chance you do happen to collide with another identity, the supernodes send an error and you generate another. (3) When you (as deadbeef11) want to talk to another host -- let's call it feedbeef22 -- you first send a packet to the closest supernode addressed to feedbeef22. The supernode knows where feedbeef22 is, so it forwards the packet. It also sends a message to both deadbeef11 and feedbeef22 telling each where the other is located and how to execute NAT traversal. Both peers will now try to connect directly. How do they encrypt? When the second peer (feedbeef22) gets a message from deadbeef11, it sends WHOIS to its supernode and asks for its full identity including public key. Supernodes are the trusted authority for this, though a second level of identity validation is performed to check and ensure that the identity's public key indeed maps to its address. Once a node has another node's key, it can perform elliptic curve diffie-hellman key agreement. At this point it has a shared secret to communicate. Each message contains a keyed message authentication code (MAC), which allows either end to verify its authenticity. There is presently no mechanism for cert revocation or expiration. A key identifies a device, so it lives forever with that device. Such a mechanism might be added in the future, but honestly it's out of scope. Compromise of a key would mean the device had been hacked, which means you're probably going to want to reinstall it and generate a new one anyway. The goal is easy direct communication and virtual networking, and it's important to stay in scope if you want to ship something. Quote Link to comment Share on other sites More sharing options...
cooper Posted August 1, 2014 Share Posted August 1, 2014 It's a VPN, though that term has become confusing of late. It's not a darknet, since it does not implement onion routing or anything else to strongly conceal your location. It's my understanding that a Darknet is simply an encrypted network on top of an existing network. Tor would be an extreme example of one in that it tries to hide you from others within the Darknet itself aswell. Since in your description you make it clear that a connection request is governed by the supernode and basically anybody who can communicate with a supernode can initiate a connection with anybody else within this private network, I feel it's more a Darknet. It doesn't appear to be possible to initiate the connection 1-on-1, without the involvement of the supernodes. It does encrypt though, and the keys are kept private to each peer. Well, obviously. The basic way this works is as follows: (1) When you first start it, it generates a public/private key pair that satisfies a proof-of-work criterion and then from that derives a 5-byte identity. (See Identity.cpp -- it's more involved than it sounds, and it's very hard to duplicate an identity). Sounds interesting. I'll take a peek in a bit. (2) It then sends HELLO to the supernodes, letting them know that you exist and what your public key is. The supernodes now know that address "deadbeef11" has public key X and is at 1.2.3.4 port 6666 (to make up an example). The supernodes remember address->identity mappings, so it's first-come-first-claim for addresses. With 5-byte addresses there can be up to 2^40 or 1,099,511,627,776 devices online. If by a small chance you do happen to collide with another identity, the supernodes send an error and you generate another. If I want to make a connection with this deadbeef22 machine, what do I need to know about it? The 40-bit identifier itself? Or is some sort of DNS capability available and if so, what does a domain name look like and how do you register one, or if not so, how do I discover the 40-bit identifier of a remote machine? Your address space is 8 bits more than regular IPv4 but once an address has been claimed it's retained for that identity seemingly indefinately. When an identity claims an address, do the old addresses claimed by that same identity get returned to the pool or is there some other way for an unused address to get returned? If not, it's possible to slowly DOS your network over time as new machines will have to use more attempts to acquire an address (forcing them to repeat the proof of work criterion which should mean it may take a while before they finally get an address). (3) When you (as deadbeef11) want to talk to another host -- let's call it feedbeef22 -- you first send a packet to the closest supernode addressed to feedbeef22. The supernode knows where feedbeef22 is, so it forwards the packet. It also sends a message to both deadbeef11 and feedbeef22 telling each where the other is located and how to execute NAT traversal. Both peers will now try to connect directly. Just to clarify the NAT bit, when both clients are behind a NAT, the connection is relayed via a supernode as you described earlier and in all other instances the connection is made from whichever of the two that can connect to the other or does the supernode always relay once one of the clients is behind a NAT? How do they encrypt? When the second peer (feedbeef22) gets a message from deadbeef11, it sends WHOIS to its supernode and asks for its full identity including public key. Supernodes are the trusted authority for this, though a second level of identity validation is performed to check and ensure that the identity's public key indeed maps to its address. This proves to each side that they're talking to who, according to the network, they should be talking to. But if this is the first connection from someone to my network, how should I authorize this connection? Do I retain a keystore with that same public key as provided by the supernode? Can I run a public-within-the-network service that would accept anybody's key? Once a node has another node's key, it can perform elliptic curve diffie-hellman key agreement. At this point it has a shared secret to communicate. Each message contains a keyed message authentication code (MAC), which allows either end to verify its authenticity. There is presently no mechanism for cert revocation or expiration. A key identifies a device, so it lives forever with that device. Such a mechanism might be added in the future, but honestly it's out of scope. Compromise of a key would mean the device had been hacked, which means you're probably going to want to reinstall it and generate a new one anyway. The goal is easy direct communication and virtual networking, and it's important to stay in scope if you want to ship something. The point is that if you do get hacked and your key is compromised, even though you can easily renew your identity with a new keypair, people might still attempt to connect to who they thought was you using the compromised identity. So unless you can communicate to your clients your new identity effectively, there's a problem here. Quote Link to comment Share on other sites More sharing options...
Adam Ierymenko Posted August 1, 2014 Author Share Posted August 1, 2014 All you need to know (from a user point of view) to establish a connection is the 40-bit ZeroTier address. (The system actually works by creating a virtual Ethernet network, so that 40-bit address is mapped to a static 48-bit Ethernet MAC on a per-network basis.) As far as address space DOS, it's presently possible but would be computationally expensive for the attacker. But a big botnet could still do some damage. I've got other ideas for mitigating this in the future, such as allowing reservations to expire if they are not used for a long enough time. As far as NAT goes, it does NAT traversal via UDP hole punching. In all but about 3% of cases (as I've measured), direct connectivity can be made. Finally, it doesn't solve the end-service authentication problem. The goal is to provide flat virtual networks that you can use for any purpose. I explicitly recommend against ever relying on IP addresses (or ZeroTier addresses, or MACs, etc.) alone to authenticate an endpoint. This is pretty much always a bad idea on any kind of network for many different reasons. So you should still use passwords, keys, certs, etc. as you would on any other LAN or WAN. Quote Link to comment Share on other sites More sharing options...
Adam Ierymenko Posted August 1, 2014 Author Share Posted August 1, 2014 Oh... one more addition to the last point: ZeroTier does do some authentication, so you could argue that it helps by adding a second factor. So if you're using password, certificate, and checking the IP address on a ZeroTier virtual network, you've technically got three different authentication factors there. But an address is never sufficient alone unless it's for something you don't care about very much. Quote Link to comment Share on other sites More sharing options...
dingo Posted August 9, 2014 Share Posted August 9, 2014 Quite honestly I think you are better off using N2N as it doesnt require you to login into some account manager to create a network see http://www.ntop.org/products/n2n/ its honestly as simple as #supernode > supernode -l xyw #edge node1> edge -a 10.1.2.1 -c mynetwork -k encryptme -l a.b.c.d:xyw#edge node2> edge -a 10.1.2.2 -c mynetwork -k encryptme -l a.b.c.d:xyw now if you provided me a central way that i could manage the system, and didnt force me to use your "portal" id be much happier. the zerotier client was a simple command..... If you want to test by joining the Earth network, try: sudo ./zerotier-cli join 8056c2e21c000001 though I refuse to use infrastructure outside of my control. let us create our own portal, and it might be more likely to be used. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.