Check out stuff like PIC chips or the ATMEL chips - they are basically single chip solutions with built in USB, IDE interface, RAM, Flash memory etc so with minimum glue you can interface to ide drives (even pcmcia), keyboards etc.
Problem is now that alot of stuff uses surface mount technology so you can't just buy a breadboard and solder something together, but the flip side is that aloy of the chips have emulators that run under windows/linux and good debugging/programming interfaces to load your software on and embedded OS's ready to go.
IMHO a search for BTNode, PicoWeb or Ethernut is a good way to start as some of the older kits would provide you with a PCB and such to get building your own.