I was presented with a problem today that I couldn’t find a non hackish way around. I hope maybe you guys can come up with a better solution. I needed to mirror a site that was a hidden service on the Tor network. For those of you that are not all that familiar with tor, this means a couple things.
- The site needs to be accessed with https
- The site is not available through standard DNS
- The site can be accessed by appending .to to the end of the address, but with a network entry page
- The network entry page is not wget friendly
Testing first on Windows (I find it easier to edit config files and test), I found that wget cannot directly use Tor as a proxy. Reading the documentation I guess its because wget expects a proxy to be socks5 only? This left me with the only option of using a proxy to connect to a proxy so wget could connect to Tor? Enter privoxy which today saved my ass. Below are the steps below to help out anyone in the same boat, but I would really like to see some comments turning me on to a better way!
sudo apt-get install -y tor tor-geoipdb privoxy nano /etc/wgetrc http_proxy = http://localhost:8118
save the file and exit
uncomment the line below, save the file and exit
# forward-socks5 / 127.0.0.1:9050 . service tor start;server privoxy start wget -mk -proxy=on <site>
I am certainly not authoritative on this subject, everyone who knows me knows that I hate using Tor. Please comment with a better way, and please explain why it is better. Wget is way to important to just leave unable to connect directly to Tor!