Get HTTP status of many URLs

Question

I'm brand new to using rust, so please bear with me. I'm trying to write a rust program which takes two arguments: A URL, and a wordlist. This wordlist has the following contents, and is several hundred thousand lines long:

...
example1.php
example2.php
example3.php
...

I want to read all of these from the wordlist file, then as quickly as possible (asychronously??) send a head request to the URL + the filename in the wordlist (for example if we do rust_fuzzer https://example.com examplewordlist.com, it should send the requests to https://example.com/example1.php...) then check if the status code is a 200. If it is, add it to a string vec, then at the end of the program print the vec using "{:?}" formatter.

So far, I have the following code, which seems to work much faster than the python equivalent I wrote (I'm much more familiar with python), however I do wonder if it could go faster, and if I've done the async stuff right. The code is sort of hacked together from existing stuff I found online and my own (very basic) understanding of the language. I would really appreciate some improvement suggestions (apart from the obvious like error handling which I'll figure out after the main part is done).

use std::fs::File;
use std::io::{self, BufRead};
use std::path::Path;
use std::env;
use reqwest::Client;
use reqwest::Url;

fn read_lines<P>(filename: P) -> io::Result<io::Lines<io::BufReader<File>>>
where P: AsRef<Path>, {
    let file = File::open(filename)?;
    Ok(io::BufReader::new(file).lines())
}

async fn fuzzer(url_string: &String, file_path: &String) -> Vec<String>{

    let Ok(base_url) = Url::parse(url_string) else { todo!() };

    let mut found_resources = Vec::new();

    let client = Client::new();

    if let Ok(lines) = read_lines(file_path) {
        for line in lines {
            if let Ok(resource) = line {
                let Ok(target_url) = base_url.join(&resource) else { todo!() };
                let response = client.head(target_url.as_str()).send().await;
                if !response.is_err() {
                    if response.unwrap().status() == reqwest::StatusCode::OK {
                        found_resources.push(resource);
                }
                }
            }
        }
    }
    else { println!("Something went wrong reading the file.."); }
    return found_resources;
}


#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>>{
    let args: Vec<String> = env::args().collect();

    if args.len() == 3 {
        let found_resources = fuzzer(&args[1], &args[2]).await;
        println!("{:?}", found_resources);
    }
    else {
        println!("USAGE: ./rust_fuzzer <URL> <Wordlist>");
    }

    Ok(())  
}

Essentially, I want this code to get the status codes and add the resources with a 200 code to the String Vec as fast as possible, but I know I may have to consider rate limiting and whatnot

Answer 1

I said I'd take a crack at this, and so here we go. It's going to be kinda rough; both because my rust is very shallow, and because I don't have as much time as I'd hoped.

Basically, I don't think you're doing the requests concurrently at all.

I tried to get it to work using Tokio Streams, but that didn't work, so instead this is a straightforward adaptation of the technique suggested here: https://stackoverflow.com/a/51047786/10135377
I didn't read the whole thing, but you definitely should because it's clearly explaining a lot of detail relevant to the kind of optimization you're doing.

use std::fs::File;
use std::io::{self, BufRead};
use std::path::Path;
use std::env;
use reqwest::Client;
use reqwest::Url;
use futures::{stream, StreamExt};
use tokio;

const CONCURRENT_REQUESTS: usize = 2000;

fn read_lines<P>(filename: P) -> io::Result<io::Lines<io::BufReader<File>>>
where P: AsRef<Path>, {
    let file = File::open(filename)?;
    Ok(io::BufReader::new(file).lines())
}

async fn fuzzer(url_string: &String, file_path: &String) -> Vec<String>{

    let Ok(base_url) = Url::parse(url_string) else { todo!() };

    let client = Client::new();

    if let Ok(lines) = read_lines(file_path) {
        stream::iter(lines).map(|line| {
            let Ok(resource) = line else { todo!() };
            let Ok(target_url) = base_url.join(&resource) else { todo!() };
            let client_ = &client;
            async move {
                let result = client_.head(target_url.as_str()).send().await;
                let response = result.ok()?; // trashing some important failures here!!
                if response.status() == reqwest::StatusCode::OK {
                    Some(resource)
                } else {
                    None
                }
            }
        }).buffer_unordered(CONCURRENT_REQUESTS).filter_map(|r| async {r}).collect().await
    }
    else {
        println!("Something went wrong reading the file..");
        Vec::new()
    }
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>>{
    let args: Vec<String> = env::args().collect();

    if args.len() == 3 {
        let found_resources = fuzzer(&args[1], &args[2]).await;
        println!("{:?}", found_resources);
    }
    else {
        println!("USAGE: ./rust_fuzzer <URL> <Wordlist>");
    }

    Ok(())
}

I benchmarked it against a localhost scotty server that just checks if the hash of the path is an even number. I used n0kovo_subdomains_tiny.txt

Orignal:

real 23.81
user 21.52
sys 7.50

Updated:

real 8.30
user 19.10
sys 6.59

So there is some improvement. It's not a ton of improvement, and IDK if it's worth trying to do better.

What in heck are you actually trying to do!?

It seems like you're trying to crawl a website by asking it which of millions of paths are real. Don't do that.

• There are better ways to crawl a website.
• If you get rate-limited, your task will never finish.
• If you don't get rate-limited, there's a risk you'll DOS them.
• Making someone rate-limit you is a dick move; implementing rate-limiting is a chore that we wouldn't do if badly-written scrapers didn't try to DOS us. A very likely outcome of pointing your tool at a stranger's website is that some hapless admin somewhere spends their Tuesday figuring out what the hell you're doing and then rate-limiting you. (In case you can't tell, that's been me on occasion.)
• You'll never be able to make your tool faster than the server it's querying.

Sorry this is casual and not super educational; it's late here.