A new proxy lecher is in town
Searching for working, anonymous proxies is currently the most time consuming and frustrating aspect of cracking for me. There are a few programs out there that help you leeching proxy lists and forums, but none of the programs I've tried up to now is clever enough to leech every proxy site I like and to differ ancient forum topics from the newer ones that are really interesting. Usually I end up with quite a few websites I have to harvest by hand and a list of tens of thousands of proxies that has been leeched by programs, takes quite a while to test and leaves only about 100 working, anonymous proxies ...
Eventually, I've decided to code an own proxy leecher that really suits my needs. It's a linux command line application called pleech, I've added a link to the source code to the end of this post. Some of its features are:
- Support of HTTP POST requests. This enables pleech to process HTML forms in a
fully automated manner, for example login forms of message boards.
- The ability to follow an arbitrary number of HTML links from a certain start
page before the application starts searching for host and port pairs. It's
possible to specify a search string for each depth of the search so that
pleech follows a link only if it points to an URL containing that search
string. This is usefull for scanning forums (including attachments) or
link lists.
- The ability to skip searching a website for host and port pairs if it appears
to be older than a given number of days. This is usefull for skipping old
forum topics.
- Support of HTTP cookies. This is usefull for logging into message boards.
- Spoofing HTTP referer and user-agent headers.
- Decoding characters that have been encoded using HTML or (very basic)
javascript functions.
- The ability to recognize (very basic) AJAX doGet requests and to scan the
request target in addition to the original website.
- The resolution of any hostname that has been found to an IP address.
- The application doesn't require any external library and runs completely non
interactive. Pleech can easily be run as a cron job or in scripts, for example
in combination with sort, uniq and proxyknife.
Please feel free to try it, however, there's one big WARNING I have issue first: Although the program seems to work quite well for me, it's still an alpha version, and comes without any warranty. I will not be responsible in case the application reformats your hard disc, causes your monitor to explode or floods your home directory with Gina Wild DVD rips ;)
Hopefully the program is as useful for you as it currently is for me, please let me know how it works for you! You can get the source tarball at:
http://rapidshare.com/files/37000827/pleech.tar.bz2.html
Last edited by Permutant; 06-18-2007 at 05:07 AM..
|