I need a method which fetches all of the text from a URL (generally a sitemap URL), and returns an IEnumerable
of all valid URLs contained in the text returned from the initial address. What I have so far is:
public IEnumerable<Uri> GetSitemapUrls(Uri sitemapUrl)
{
var sitemapText = GetSitemapText(sitemapUrl);
if (string.IsNullOrWhiteSpace(sitemapText))
yield break;
var urls = new List<string>();
var urlRegex = new Regex(@"\b(?:https?://|www\.)[^ \f\n\r\t\v\]]+\b", RegexOptions.Compiled | RegexOptions.IgnoreCase);
foreach (Match m in urlRegex.Matches(sitemapText))
urls.Add(CleanUriString(m.Value));
foreach (var url in urls)
{
var cleanedUriString = CleanUriString(url);
if (Uri.IsWellFormedUriString(cleanedUriString, UriKind.RelativeOrAbsolute))
yield return new Uri(cleanedUriString);
}
}
string GetSitemapText(Uri sitemapUri)
{
var wc = new WebClient
{
Encoding = System.Text.Encoding.UTF8
};
return wc.DownloadString(sitemapUri);
}
string CleanUriString(string dirtyUriString)
{
var legalCharacters = @"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;=".ToCharArray();
var cleanedString = dirtyUriString;
foreach (var character in dirtyUriString)
{
var matchIndex = dirtyUriString.IndexOf(character);
if (!legalCharacters.Any(x => x.Equals(character)) && matchIndex > 0)
cleanedString = dirtyUriString.Substring(0, matchIndex);
}
return cleanedString;
}
It seems to work as intended for actual sitemaps, null/empty responses from the URL it receives, and URLs with URL-illegal characters in them. I have a feeling I'm missing out on potential issues or letting bad URLs through anyway, not to mention I haven't thought about spidering through URLs that the initial sitemap (if it is a sitemap) returns.
Is there anything I can do to improve it?
CleanUriString
works. It looks for achar
fromdirtyUriString
inside the same string. This means thatmatchIndex
will never be negative :-| Could you help me out with that? ;-) Maybe an example? \$\endgroup\$