Microservice for scraping images with celery

Question

I made a project, that scrapes images asynchronously and saves them in container. I have access to them through volume. Scrapy finds images on given web page.

Any tips will be good. But first I would like to focus on docker-compose I would appreciate tips on how to improve it.

files tree

My project looks like that. celery.py for celery config, scrap_images directory for scrapy project. finals images are stored in pictures in one possible directory

.
├── celery_test.py
├── docker-compose.yaml
├── image_collector
│   ├── celery.py
│   ├── ENV_FILE
│   ├── image_collector   # overlaping
│   ├── pictures
│   │   ├── big
│   │   ├── medium
│   │   ├── small
│   │   └── tiny
│   ├── pipeline.log     # scrapy pipeline log
│   ├── scrap_images
│   │   ├── custom_loger.py
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── middlewares.py
│   │   ├── pipelines.py
│   │   ├── settings.py
│   │   └── spiders
│   │       ├── image_spider.py
│   │       └── __init__.py
│   ├── scrapy.cfg
│   ├── spider.log   # scrapy spider log
│   └── tasks.py
├── README.md
├── requirements.txt
└── Worker

docker-compose.yaml

I had problem with path, so in docker-compose in Worker I got 2 volumes that overlap to start celery and scrapy from same workdir. I start celery with docker command, and withing same workdir i start scrapy spiders when task is received

version: '3.3'

services:
  worker:
    build:
      context: .
      dockerfile: Worker
    image: collector_worker:3.8.5
    environment:
      - HOSTNAME=broker
      - PORT=5672
      - DB_ACCES_NAME=postgres_db
    env_file:
      - ./image_collector/ENV_FILE
    volumes:
      - ./image_collector:/app/image_collector
      - ./image_collector:/app
    depends_on:
      - broker
    restart: always
    command: bash -c "mkdir -p pictures && chmod -R 777 pictures && celery -A image_collector worker --loglevel=info --autoscale=5,1"

  broker:
    image: rabbitmq:3.6.6-management
    hostname: broker
    restart: always
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - "5673:5672"
      - "15673:15672"

Worker

Container that runs celery, and starts spiders in current workdir position

FROM python:3.8.5

ENV PYTHONUNBUFFERED 1

COPY requirements.txt /app/requirements.txt

WORKDIR /app

RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen

RUN pip install -r requirements.txt

celery.py

Celery configuration to connect with rabbit broker

from __future__ import absolute_import, unicode_literals
from celery import Celery
import os

user = os.getenv('LOGIN', 'admin')
password = os.getenv('PASSWORD', 'mypass')
hostname = os.getenv('HOSTNAME', 'localhost')
port = os.getenv('PORT', '5673')

broker_url = f'amqp://{user}:{password}@{hostname}:{port}'
app = Celery("tasks", broker=broker_url, namespace="image_celery", include=['image_collector.tasks'])

__all__ = ("app",)

Tasks.py

I start spiders with system command and pass url to scrap that page and process in pipeline

#from scrapy.crawler import CrawlerProcess
#from .scrap_images.spiders.image_spider import ImageSpider
#from scrapy.cmdline import execute
from .celery import app

import sys
import os


@app.task
def start_spider(url):
    """Task for crawling web"""
    os.system(f"python -m scrapy crawl image_spider -a url={url}")


if __name__ == "__main__":
    url = r'https://www.wykop.pl/'
    start_spider(url)

celery_test.py

i send tasks to celery manuall at this moment, nothing special

from image_collector.tasks import start_spider

for pages in range(1000):
    url = f'https://memy.jeja.pl/nowe,0,0,3{2216 - pages:>04}.html'
    start_spider.delay(url=url)

image_spider.py

import scrapy
from ..items import ImageItem
from ..custom_loger import define_logger


class ImageSpider(scrapy.Spider):
    name = 'image_spider'
    my_logger = define_logger("spider")

    # allowed_domains = ['www.wikipedia.pl']

    def start_requests(self):
        url = getattr(self, "url")

        if url:
            self.my_logger.debug(f"Starting image parsing at: {url}")
            yield scrapy.Request(url=url, callback=self.find_images)
        else:
            self.my_logger.error(f"URL is empty")
            yield None

    def parse(self, response):
        print(f"Parsing: {self.url}")

    def find_images(self, response):
        images = response.css("img::attr(src)").extract()
        item = ImageItem()
        item['image_urls'] = []
        item['title'] = response.css("title::text").extract()[0]
        item['title'] = response.xpath("//title/text()").extract()[0]

        self.my_logger.info(f"Found :{len(images)} images")
        for img in images:
            img = str(img)
            im_url = img if img.startswith("http") else "http:" + img

            item['image_urls'].append(im_url)
            self.my_logger.debug(f"Got image: {im_url}")

        yield item

    # def save_images(self, response):
    #     image = ImageItem()
    #     self.my_logger.info(f"Got image: {response.url}")

    def err_hanlder(self, failure):
        url = failure.request.url
        callback = failure.request.callback
        errback = failure.request.errback  # should work same way as callback... ?
        # status = failure.value.response.status
        self.my_logger.error(f"Fail request: @: {url}")
```