whitenoise

Two weeks later .... I was able to fix the crashes by using the asyncore module around the nfqueue. Internetspeed was significantly increased. I used that script on an old 2 GHz dualcore laptop with 1GB of RAM (but booted from an USB dongle) and got 26 MBits. Works like a charme! Next aspect I'm working on is IPv6. As far as I know, iptables does only work for IPv4 packets but not for IPv6. There is ip6tables which works analog to iptables for IPv6 packets. I'll see if I can send IPv6 traffic into the same nfqueue and write a distinction of cases in the callback function. Does anyone have some ideas about combining those versions via iptables and nfqueue?

Can you please give us some more information? What is the output of "iwconfig" and are you able to enable monitor mode on your wifi device? Does another program block the device (maybe network-manager or so) ? Try: airmon-ng start wlan0 and see what happens. If airmon-ng creates a monitor interface for you, you can use that one for the injection test.

Next update: I found a solution for handeling the crashes. Although I still don't know why it crashes I put the q.try_run() in a while loop like: try: x=0 while x==0: x=1 q.try_run() # Main loop x=0 Which basically restarts the try_run() function after every crash. Fortunatly there is no configuration or blocking problem when restarting the try_run() function. So the crashes are still there but everytime they appear, the function is restarted and at the other end of the connection you don't notice these drops. This might be a quick & dirty workaround but at least I get a stable connection. Here some speed results (testmy.net/dl-5000): normal connection: 26 mbps intercepted connection: 3 mbps I hope this helps ;) Thank you all!

Here comes an update: I tried several setups. I mentioned above that it might be the delay-time of the packet stuck in the queue that may cause the crash but thats not the case. I added a time.sleep(1) and time.sleep(10) into the callback function and it didn't produce an error so there is no issue with the delay time. As I'm parsing packets with scapy I thought maybe there are 'evil' packets for scapys IP() function so I set up a callback function that does nothing but returns the packets back to the nfqueue via accept. That worked like a charme for a q.set_queue_maxlen(10) with a speed of several megabytes per second. As I increased the value to q.set_queue_maxlen(50000) it crashed again. I used the link testmy.net/dl-5000 suggested by i8igmac above. The crash appeard at around 7%. So what do I know from here? It is not a scapy issue and it is not a packet delay issue. It looks more like some kind of overflow and the queue seems to stumble or gets out of clock if it is flooded with too many packets. I'm still working on that one, further input is appreciated ;)

@i8igmac: The problem actually seems to arise from the main()-function: def main(): q = nfqueue.queue() q.set_callback(callback) q.fast_open(0, AF_INET) # 0 is the index of queue q.set_queue_maxlen(3) q.set_mode(nfqueue.NFQNL_COPY_PACKET) try: q.try_run() # Main loop except KeyboardInterrupt: q.unbind(socket.AF_INET) q.close() sys.exit('...termination') You see "q.set_queue_maxlen(3)" is pretty low, this decreases the speed but the CPU is barely hitting the 100% on the pi. I can visit websites like example.com which are loaded in a second but if I switch over to a website with lots of content and pictures i.e. bbc.com it takes a while, BUT it does not crash. If I increase the value for set_queue_maxlen the scripts puts more packets into the queue and on websites like bbc.com there is a higher risk of crashing the script. Unfortunatlely there is no error message, it just flips out of the script back to command line and as the loop-through is broken then, the internet connection is, too. I'm using a Raspberry Pi B by the way, so not the hottest model. Maybe it is some driver issue, I don't know, but it seems to be something lowlevel. I read a bit about this problem and it seems to be that the packet accept delay is too high then. If you send a packet into the nfqueue it seems to be expected coming back in a certain amount of time, if this time is exceeded the script stops working. At least that is what I assume for the moment. I also read that one could analyze and the drop all packets by default but send a copy of each to a raw socket so there is no process that is waiting for the packets and you can choose by yourself how long you want to store the copied packet until you send it. Do you have any information about that? Alternatively one could code all the stuff in C which would run it faster but I'm not familiar with coding in C yet ;) @cooper: Yep, basically you're right but I was too lazy and wanted to have everything in one script ;) the right way would be to seperate the stuff.

Here comes an update! I connected an USB-Ethernet adapter to my Raspberry resulting in having 2 ethernet ports. Now comes some configuration magic: def setupInterfaces(iface1, iface2): # I needed that one for the Raspberry, otherwise I got problems with nfqueue os.system("modprobe br_netfilter") # Setting up interfaces and creating a bridge os.system("ifconfig %s down" %(iface1)) os.system("ifconfig %s down" %(iface2)) os.system("brctl addbr br0") os.system("brctl addif br0 %s" %(iface1)) os.system("brctl addif br0 %s" %(iface2)) os.system("brctl setfd br0 0") os.system("brctl stp br0 off") # Put interfaces up in promiscious mode os.system("ifconfig %s 0.0.0.0 promisc up" %(iface1)) os.system("ifconfig %s 0.0.0.0 promisc up" %(iface2)) os.system("ifconfig br0 up promisc") # Reset iptables os.system("iptables -F") os.system("iptables -X") os.system("iptables -t nat -F") os.system("iptables -t nat -X") os.system("iptables -t mangle -F") os.system("iptables -t mangle -X") # Configure iptables os.system("iptables -P FORWARD ACCEPT") os.system("iptables -P OUTPUT DROP") os.system("iptables -P INPUT DROP") os.system("iptables -A FORWARD -i br0 -p tcp -j NFQUEUE") # Enable ip forwarding os.system('echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward') return This is the Python code for setting up the interfaces and enable forwarding. From there it should be easy to pass the packets through a self programmed filter script based on nfqueue. With python and nfqueue you can filter and alter packets, dpi is also an option as long as the traffic is not encrypted. I should mention that using this script on a Raspberry Pi might result in a slow connection. I also read, that the network plug and the usb ports are using one single bus to the processor and as there is also a second ethernet device connected via usb this bus probably becomes the bottleneck. So it might be useful to switch to another system that has more RAM and CPU power, i.e. an Intel NUC. I did a traceroute through the device and it was completly invisible. I guess the only way to detect it might be via time period analysis as the packets need a little bit longer if the device is in between. Thanks for all the input, hope this helps ;)

Thanks to both of you! I was actually thinking about the Throwing Star, too but I still think I can realize that with my setup described above. What I read so far is that one can bridge both ethernet adapters with brctl (from the bridge-utils package) in a bridge called bridge0, which you then can route through the nfqueue script. I will test this during the next days and see if it works. @barry: for just monitoring this is a nice solution, when I wrote passive I meant that I didn't want it to be discovered on the network, nevertheless it would be cool to activly filter and maybe alter the content of unencrypted traffic. For example I read about the pihole project, so I could easily introduce something similar for blocking ads or other stuff as well.

Hey guys, is it possible to put a Raspberry Pi (having two ethernet ports) in the middle of a network connection monitoring all the traffic but without having an IP address assigned to the Pi? I want it to be there 100% invisible and passive, basicaly I only want to use the forward option in the ip tables and put a loop in between which sends all the packets via nfqueue through a python script which then is able to filter the packets at an optional level up to some kind of dpi. I'm not sure if I can prohibit it getting an IP assigned because I don't want it to be accessible. I guess I need some nice configuration for the /etc/network/interfaces file, the routing can be done with iptables at startup, which should not be a problem. All the other stuff with python and nfqueue also isn't the problem for me.

Basically you are looking for packets. It's true that you will get a lot of noise because of reflecting signals but what you can try is a statistical analysis of the packets. I mean, the packets are sent with a maximum relative signal strength. In a complex environment you will get lots of packets with different relative signal strenght because of the reflections, although nothing is moving yet. Try to collect a certain amount of packets and then look at the highest relative signal strenght and calculate a mean value of them. In theory every reflection should decrease the signal strenght a little bit because of absorption. Finally it also depends on how precise you want to locate your target. At least that would be my approach.

@cooper: I just found another solution ;) Dealing with such big numbers in the method above takes too much memory. However, the problem can be solved with a for-loop by what is called Modular Exponentiation. Python has a built-in function called pow(), which does modular exponentiation and can take up to three arguments (base, exponent and optional a modulo parameter). If you substitute: A=b**x % p by A=pow(b,x,p) A=67106303645408219781140727427280612111960487935201208302504831122113989347565507978724914118463849388439348417955712740781306188779036136994829742272005372145131781890427605537882 That's the way Diffie Hellman is calculated.

Hey, I'm trying to code some crypto stuff. I don't just want to simulate an algorithm but want to use numbers that are used in practice. During the calculations of a Diffie Hellman key exchange some large numbers have to be processed. There is a prime number with >2048 bits and a base with an exponent, where the exponent should have a size of around 160-256 bits. At the moment I'm having problems with finding the right data types for this big numbers. If I just try to calculate something like (p is not a prime here): p=123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789 b=2 x=987654321987654321987654321 A=b**x mod p Here for testing: #!/usr/bin/env python #-*- coding: utf-8 -*- p=123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789123456789 b=2 x=987654321987654321987654321 A=b**x % p print A I run into a memory problem (MemoryError). Converting the int's to long does not help, using Decimal() does not help either. I was trying to use the datatypes of numpy (see here) but I get OverflowError: Python int too large to convert to C long Has anyone experience in handling such big numbers and doing arithmetics with them in python? I mean .. DH key exchange is something my browser does a hundreds time a day and it never told me it is out of memory so I guess there has to be a trick ;) Cheers!

I'm only interested in HTTPS not in HTTP. HTTP sessions are clear to me. It is all about how the encryption keys of a https connection are stored and referenced to the right connection. For me there has to be some kind of table somewhere looking like: ip:port <-> ip:port | key1 ip:port <-> ip:port | key2 ip:port <-> ip:port | key3 ... And each time a new https connection is established with all the client hello and server hello and so on after the negotiation the server and also the browser needs to add that connection to a table to refer encrypted content (incoming payloads) and unencrypted content (outgoing stuff) to a key that is used for the encryption/decryption so both partners speak the same language.

Totally makes sense to me. It's still not clear to me how this finally is working :S Obscured internal processing? huh?

@cooper: Thanks for this nice summary about TCP and HTTP layers working together. How these layers work is clear to me. I conclude that the tuple of src-ip, src-port, dst-ip and dst-port identify to a unique connection and therefore I would say HTTPS keys are stored together with this connection information. However the HTTPS port on most webservers is port 443 so the tabs inside a webbrowser can target the same https website but each outgoing connection uses another port so inside the browser this connection is tied to the keys and is distinct by the ip address of the user and in case of the tabs by the portnumbers. Thanks to both of you for clearing this up to me.

@digininja: Okay, TCP means we have a source-ip, source-port, destination-ip, destination-port, then there are ack-numbers and sequence-numbers for all the data and a bunch of flags (+checksum). However there is not really a session ID in the TCP segment of the packets. There has to be something distinct. So would you say the keys are bound to processes running on the server? Are all TCP connections followed after the TLS handshake (which also includes a TCP segment) kind a child-connections from that TLS handshake TCP connection?