3
\$\begingroup\$

I'm brand new to using rust, so please bear with me. I'm trying to write a rust program which takes two arguments: A URL, and a wordlist. This wordlist has the following contents, and is several hundred thousand lines long:

...
example1.php
example2.php
example3.php
...

I want to read all of these from the wordlist file, then as quickly as possible (asychronously??) send a head request to the URL + the filename in the wordlist (for example if we do rust_fuzzer https://example.com examplewordlist.com, it should send the requests to https://example.com/example1.php...) then check if the status code is a 200. If it is, add it to a string vec, then at the end of the program print the vec using "{:?}" formatter.

So far, I have the following code, which seems to work much faster than the python equivalent I wrote (I'm much more familiar with python), however I do wonder if it could go faster, and if I've done the async stuff right. The code is sort of hacked together from existing stuff I found online and my own (very basic) understanding of the language. I would really appreciate some improvement suggestions (apart from the obvious like error handling which I'll figure out after the main part is done).

use std::fs::File;
use std::io::{self, BufRead};
use std::path::Path;
use std::env;
use reqwest::Client;
use reqwest::Url;

fn read_lines<P>(filename: P) -> io::Result<io::Lines<io::BufReader<File>>>
where P: AsRef<Path>, {
    let file = File::open(filename)?;
    Ok(io::BufReader::new(file).lines())
}

async fn fuzzer(url_string: &String, file_path: &String) -> Vec<String>{

    let Ok(base_url) = Url::parse(url_string) else { todo!() };

    let mut found_resources = Vec::new();

    let client = Client::new();

    if let Ok(lines) = read_lines(file_path) {
        for line in lines {
            if let Ok(resource) = line {
                let Ok(target_url) = base_url.join(&resource) else { todo!() };
                let response = client.head(target_url.as_str()).send().await;
                if !response.is_err() {
                    if response.unwrap().status() == reqwest::StatusCode::OK {
                        found_resources.push(resource);
                }
                }
            }
        }
    }
    else { println!("Something went wrong reading the file.."); }
    return found_resources;
}


#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>>{
    let args: Vec<String> = env::args().collect();

    if args.len() == 3 {
        let found_resources = fuzzer(&args[1], &args[2]).await;
        println!("{:?}", found_resources);
    }
    else {
        println!("USAGE: ./rust_fuzzer <URL> <Wordlist>");
    }

    Ok(())  
}

Essentially, I want this code to get the status codes and add the resources with a 200 code to the String Vec as fast as possible, but I know I may have to consider rate limiting and whatnot

\$\endgroup\$
3
  • 2
    \$\begingroup\$ I'd be happy to play with this a little later today. Do you have a domain and wordlist you'd be comfortable sharing with strangers for testing? \$\endgroup\$ Nov 6, 2023 at 15:47
  • \$\begingroup\$ @ShapeOfMatter Thank you so much! I've just been using the n0kovo wordlists found here and unfortunately cant offer you a domain as I've been using locally hosted sites, but I've been using DVWA or another one I found from a book called scruffybank (I might be able to send you a copy of the .vdi for the virtual machine if that would help!) \$\endgroup\$
    – TmDobbin
    Nov 6, 2023 at 20:24
  • \$\begingroup\$ Just realised I didn't copy all of the code and missed the main function - oops \$\endgroup\$
    – TmDobbin
    Nov 6, 2023 at 21:49

1 Answer 1

4
\$\begingroup\$

I said I'd take a crack at this, and so here we go. It's going to be kinda rough; both because my rust is very shallow, and because I don't have as much time as I'd hoped.

Basically, I don't think you're doing the requests concurrently at all.

I tried to get it to work using Tokio Streams, but that didn't work, so instead this is a straightforward adaptation of the technique suggested here: https://stackoverflow.com/a/51047786/10135377
I didn't read the whole thing, but you definitely should because it's clearly explaining a lot of detail relevant to the kind of optimization you're doing.

use std::fs::File;
use std::io::{self, BufRead};
use std::path::Path;
use std::env;
use reqwest::Client;
use reqwest::Url;
use futures::{stream, StreamExt};
use tokio;

const CONCURRENT_REQUESTS: usize = 2000;

fn read_lines<P>(filename: P) -> io::Result<io::Lines<io::BufReader<File>>>
where P: AsRef<Path>, {
    let file = File::open(filename)?;
    Ok(io::BufReader::new(file).lines())
}

async fn fuzzer(url_string: &String, file_path: &String) -> Vec<String>{

    let Ok(base_url) = Url::parse(url_string) else { todo!() };

    let client = Client::new();

    if let Ok(lines) = read_lines(file_path) {
        stream::iter(lines).map(|line| {
            let Ok(resource) = line else { todo!() };
            let Ok(target_url) = base_url.join(&resource) else { todo!() };
            let client_ = &client;
            async move {
                let result = client_.head(target_url.as_str()).send().await;
                let response = result.ok()?; // trashing some important failures here!!
                if response.status() == reqwest::StatusCode::OK {
                    Some(resource)
                } else {
                    None
                }
            }
        }).buffer_unordered(CONCURRENT_REQUESTS).filter_map(|r| async {r}).collect().await
    }
    else {
        println!("Something went wrong reading the file..");
        Vec::new()
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>>{
    let args: Vec<String> = env::args().collect();

    if args.len() == 3 {
        let found_resources = fuzzer(&args[1], &args[2]).await;
        println!("{:?}", found_resources);
    }
    else {
        println!("USAGE: ./rust_fuzzer <URL> <Wordlist>");
    }

    Ok(())
}

I benchmarked it against a localhost scotty server that just checks if the hash of the path is an even number. I used n0kovo_subdomains_tiny.txt

Orignal:

real 23.81
user 21.52
sys 7.50

Updated:

real 8.30
user 19.10
sys 6.59

So there is some improvement. It's not a ton of improvement, and IDK if it's worth trying to do better.

What in heck are you actually trying to do!?

It seems like you're trying to crawl a website by asking it which of millions of paths are real. Don't do that.

  • There are better ways to crawl a website.
  • If you get rate-limited, your task will never finish.
  • If you don't get rate-limited, there's a risk you'll DOS them.
  • Making someone rate-limit you is a dick move; implementing rate-limiting is a chore that we wouldn't do if badly-written scrapers didn't try to DOS us. A very likely outcome of pointing your tool at a stranger's website is that some hapless admin somewhere spends their Tuesday figuring out what the hell you're doing and then rate-limiting you. (In case you can't tell, that's been me on occasion.)
  • You'll never be able to make your tool faster than the server it's querying.

Sorry this is casual and not super educational; it's late here.

\$\endgroup\$
3
  • 1
    \$\begingroup\$ Thanks so much! Regarding your words on my actual goals - you've got it spot on, it's for a little project I'm doing. If you get a chance, could you elaborate a bit on the better ways of crawling a website, perhaps without the negative side effects? \$\endgroup\$
    – TmDobbin
    Nov 12, 2023 at 9:26
  • \$\begingroup\$ Well first thing to try is to check if the site has a robots.txt file; if they do, try to parse and follow any directives in there. The helpful aspect of that is that it might point to a sitemap. \$\endgroup\$ Nov 13, 2023 at 2:05
  • \$\begingroup\$ Whether any of that works or not, "crawling" is not supposed to be a brute-force process; it's about discovering resources by following links. You start with a small list of urls (maybe stuff you found in robots.txt and/or sitemap.xml, maybe just /, maybe a list like you've been using but pared down to just a few hundred items. Then when you fetch each of those pages, you scan the document (assuming it's html or something else you can easily parse) for new paths you can add to the queue. \$\endgroup\$ Nov 13, 2023 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.