I tried using __ once and found it to produce more problems than it's worth. Take:
self.__baz = __foo()
NameError: name '_Bar__foo' is not defined
I'd suggest you follow PEP 8 and only use it for classes that will be subclassed and where you need to prevent name collisions. ...
Disclaimer: Don't know Python nor PEP 8.
In __download_robots(self) you could reverse the if condition and return early which saves one level of indentation.
if self.__url is None:
raise InvalidUrlException("\n\nInvalidUrlException: Invalid Url.\n")
..the remaining code
this applies to get_allowed(self), get_disallowed(self) and get_all(self) ...
I've been experimenting with the best data structure for this. Here is what I have for now. It is definitely a work in progress, I hope to keep updating it constantly.
The CSV output was unexpected, ultimately a result of my decision to use a namedtuple. I chose namedtuples because, like you, I thought the data fit that tabular style quite well. You wrote ...