I tried using __ once and found it to produce more problems than it's worth. Take:
self.__baz = __foo()
NameError: name '_Bar__foo' is not defined
I'd suggest you follow PEP 8 and only use it for classes that will be subclassed and where you need to prevent name collisions. ...
Disclaimer: Don't know Python nor PEP 8.
In __download_robots(self) you could reverse the if condition and return early which saves one level of indentation.
if self.__url is None:
raise InvalidUrlException("\n\nInvalidUrlException: Invalid Url.\n")
..the remaining code
this applies to get_allowed(self), get_disallowed(self) and get_all(self) ...
Your escaping for the search term is not sufficient. Other characters than spaces might be in the user entered string, which also need to be escaped. Luckily, requests can do it all for you:
def rechercher_google(search_term, number_results, language_code):
input: Nom de l'entreprise à siretiser, le nombre de page et la langue
output: La ...
I think this part is ok (will still take your feedback :) )
RecipeParser::parseHtml is asynchronous - CookCrawler::getRecipeData() could also be asynchronous as well.
As far as the "factory" method is concerned, a Map or simple Object could be used to store the mappings of domains to the parsers. Then instead of using a switch statement, have the code ...