spiders: Set error status for exceptions generated by spider.

Spider raises exceptions when errors like FileNotFound
are detected. However, these did not set error state
before exiting causing spider to fail silently.

This patch sets the status causing exceptions to exit with
non-zero exit status.

tools/documentation_crawler/documentation_crawler/spiders/common/spiders.py

@@ -121,5 +121,6 @@ class BaseDocumentationSpider(scrapy.Spider):
         elif isinstance(failure.type, IOError):
             self._set_error_state()
         else:
+            self._set_error_state()
             raise Exception(failure.value)
         return None