I haven’t seen this issue, but this Apple Support Forums discussion (http://discussions.apple.com/thread.jspa?threadID=2072449&tstart=25 ), discusses a non-obvious troubleshoot step to take if you are experiencing seriously degraded download performance.

1. Quit Safari
2. In your home folder open Library –> Safari
3. Drag your downloads.plist file to the desktop
4. Open Safari
5. Attempt an download, and compare the speed of the download

If you receive improved download speeds, then your downloads.plist file was damaged or corrupted.  Otherwise, you can move your downloads.plist file back to the safari folder, or trash it.  It’s main purpose is to keep track of what files you have downloaded, so that Safari can display the “Downloads” pop up listing.