[Version 1] Fun With Base64

[Cerolobo]

Edit (04-27-10): Lots of changes from the proof of concept

http://www.craigwill.com/Downloads/Fun_With_Base64_v2.zip

Summary:

- Apply LZW compression (~30% size reduction)

- LZW decompression done via VBScript

- Automatic Teensy source file generation (IE, it should be easy to use)

- Program and script the script code is now stored in the Flash memory

Current Goals:

- Port Base64 Library to Teensy, so the compressed binary can be stored

- Optimized/Tweak the LZW library

- Linux Build

- Flat binary file, rather then a ton of global variables

Know Issues

- Current LZW compression will increase the size of already compressed files

(via UPX and similar). Compression should be disabled if the file

grows in size.

- Script file is now a lot larger, since it must handle LZW decompression

- LZW compressor is slow. I was mainly targeting correctness and simplicity.

The best this about this update is the automatic code generation. Namely, you only have to drop a executable on the included program and it'll do nearly all the work for you, including the generation of the Teensy code. It will produce a header file and a source file with everything already compiled into it. Simply add the source to your project, call the provided helper function, and your off!

-- Initial Proof of Concept --

Well, I received my Ducky in the mail yesterday, and I decided to play with it a bit.

Instead of having the Ducky download a file from the web, I wanted to focus on executable uploading from the ducky. This was accomplished by converting the executable to base64 and then back again on the target computer.

I've attached to proof of concept that should run on Windows (only tested on Windows XP). It uses a vbs script with Microsoft.XMLDOM to perform the actual conversion. The program that is encoded was written in assembly. It simply hangs around in the background until you hit the 'z' key. In which case it'll pop up a message box and then terminate.

The same concept can be applied to linux if a program like base64 is installed.

Full project with test program, encode/decode scripts, and the teensy project

http://www.craigwill.com/Downloads/Fun_With_Base64.zip

Edited April 28, 2010 by Cerolobo

[DjMaddius]

Nice ;D i love it. We are already making great progress and people just started receiving there usbs. It works perfect :D

EDIT:: Do you think it would be possible to convert the text view of a pre-written .exe to base64, then decode that and write that to another.exe and run? Thus we will be able to do anything.

Edited April 25, 2010 by DjMaddius

[Cerolobo]

Do you think it would be possible to convert the text view of a pre-written .exe to base64, then decode that and write that to another.exe and run? Thus we will be able to do anything.

Definitely; however, that's kind of a roundabout method.

You can actually convert any executable to base64 (assuming it will fit on the chip), so you don't actually have to do the intermediate step.

I did include the script I used to convert the executable to base64 in the zip at the bottom of my original post. Simply just drag and drop a executable over it (Fun_With_Base64/Scripts/Encode.vbs), and it'll produce a text file.

Once you have that, you'll have to convert it into something that the ducky can use. I just ran a bunch of key macros over the text file to stringify the file.

I originally wrote the sample program in assembly since I didn't know how long the actual transfer would take. Base64 will always produce a larger file, so I'm actually thinking about attempting a different encoding scheme, but then it more or less becomes a fight between script size and encoding size.

[magic_smoke]

How would a tiny implementation of LZW or a bespoke compression algorithm compare to base 64 encoding it?

Nice job.

[DjMaddius]

Definitely; however, that's kind of a roundabout method.

You can actually convert any executable to base64 (assuming it will fit on the chip), so you don't actually have to do the intermediate step.

I did include the script I used to convert the executable to base64 in the zip at the bottom of my original post. Simply just drag and drop a executable over it (Fun_With_Base64/Scripts/Encode.vbs), and it'll produce a text file.

Once you have that, you'll have to convert it into something that the ducky can use. I just ran a bunch of key macros over the text file to stringify the file.

I originally wrote the sample program in assembly since I didn't know how long the actual transfer would take. Base64 will always produce a larger file, so I'm actually thinking about attempting a different encoding scheme, but then it more or less becomes a fight between script size and encoding size.

Yea, the fact that it multiples the size of the program by 4 or 5, isnt going to work. Need an encryption method that makes it smaller.

Could possibly compile to .rar or .zip then decompile. This way we can include more with the program also. DLLs it might need, etc.

Edited April 25, 2010 by DjMaddius

[Paul Stoffregen]

I've been working on USB mass storage, so far with the unused space in the chip for media. On Teensy++ it's writable. On Teensy it's read-only. I just got it working on Teensy++ yesterday, though it still fails a couple of the USB-IF compliance tests, but seems to work fine on WinXP, OS-X 10.5 and Ubuntu 9.04. Today I'm writing a utility to package up files into a minimal FAT12 filesystem and merge it into compiler's ELF output before the final HEX file is produced and downloaded, so for Teensy you can have something more interesting than blank, read-only media. Next week, I'm going to add code to support a SD flash card instead of internal memory. Alpha/beta test versions will be appearing soon. Eventually I'm going to add a filesystem layer and arbitration between the Teensy's code and the PC's access. My main goal is enabling data collecting apps on Teensy, which need a lot more functionality than simply automating PC input.

On Teensy with 31.5K memory, the filesystem overhead might be similar to base64, though it should be a lot faster! The USB mass storage code adds about 2K and a minimal FAT12 filesystem has 2.5K overhead (MBR, Vol ID, 2 FATs, root dir).

Of course, if you want to maximize on-chip storage, you would just store the code in binary and convert to base64 on the fly. On AVR chips, to directly use the flash memory, you need to declare the array with PROGMEM and read it with pgm_read_byte(). Yes, that's annoying, but it's the way the avr-gcc authors decided to implement it. The above code actually allocates "program" in RAM, and the C runtime startup code copies it from flash into RAM before the program starts. If it gets much bigger than 2K on Teensy, you'll run out of RAM and crash. Using PROGMEM and pgm_read_byte() will let you use up to 64k flash.

[Cerolobo]

How would a tiny implementation of LZW or a bespoke compression algorithm compare to base 64 encoding it?

Nice job.

Could possibly compile to .rar or .zip then decompile. This way we can include more with the program also. DLLs it might need, etc.

The main issue is the actual character set. Since you are limited to keyboard input, you can only use visible keys. As such, traditional compression algorithms won't work correctly without modification. With that being said, it would potentially be possible to run LZW or similar on a binary file, and then convert the overall output to Base64 for the transfer. The main caveat with this will be the actual script size. I wanted to rely on common built in functionality. Once you move over to a custom compression scheme, the script file will become a lot larger.

A alternative to doing this would be to perform a bootstrapping esk setup. Namely, you have a minimum program stored in base64 which would be used to perform the more complicated decompression steps.

Yea, the fact that it multiples the size of the program by 4 or 5, isnt going to work. Need an encryption method that makes it smaller.

Not true. Base64 uses 6 bits while a byte in 8 bits. This effectively works out to a 33% increase in size. Unfortunately, Microsoft.XMLDOM converts everything to Unicode, which arbitrarily doubles the encoded size. In order to actually transfer the program, you'll have to convert it to ASCII and then reconvert it to Unicode when decoding, which current happens.

Anyway, yes, I do agree that compression would be desirable in this situation.

[H@L0_F00]

UPX, then encode it.

Also, why are you not using base16?

[Cerolobo]

Also, why are you not using base16?

Simply stated, size constraints. Hex would double the size of the encoded program. IE, you won't be able to do nearly as much and it'll take longer to transfer to the computer.

[H@L0_F00]

Simply stated, size constraints. Hex would double the size of the encoded program. IE, you won't be able to do nearly as much and it'll take longer to transfer to the computer.

Yes, you're right. I was thinking base64 encoding would be larger than just representing the program in hex for some reason.

[Paul Stoffregen]

Why not just store it in binary, and convert to hex or base64 or whatever, on the fly as you read it out of memory. Read 1 byte, send 2 (hex). Or read 3, send 4 (base64).

[DjMaddius]

Why not just store it in binary, and convert to hex or base64 or whatever, on the fly as you read it out of memory. Read 1 byte, send 2 (hex). Or read 3, send 4 (base64).

Because now your making it even larger. Each character will be 8 characters now (i believe thats what binary makes it into)

[H@L0_F00]

No, the size on the Teensy would be equal to the size of the executable normally.

[Cerolobo]

As H@L0_F00 stated, it would be a 1-1 mapping when compiled into the Teensy.

The main issue with this method it that you'll still need to convert the binary file into something that can be compiled. I don't know of any method to pull in a binary file, and load it at a specific address. If there is a method, I'd love to know. Currently, the only way I can see to do this would be to convert the binary into a byte array, which would require a fairly trivial program to perform the conversion in a sane manner (output a c source file).

Generally, my initial post was just a proof of concept. I have some free time today, so I plan to do some further experimentation. Namely, I'm currently thinking about applying LZW and then convert to Base64. Assuming I can get the script to a manageable size, I'll post an update in a day or two.

[Cerolobo]

Ok, I did some more work on the executable uploading. It should be fairly easy to actually use at this point.

See my initial post for a list of the changes and such.

Other then that, any feedback or requested would be appreciated.