3
\$\begingroup\$

As a student, I am working on a project where I need to forward data from an already existing website to a new one, with the data presented in a different manner.

  • In order to do so with my mediocre skill, I learned from the various web resources available and developed a first version. But, upon launch, I apparently killed the source website, too many request at once.

  • I did a second version yet I would like to be sure, also any improvement is welcome as I know my skills are far from good. If you can look at theses lines and tell me if you see problem in the program (or any improvement) I will gladly listen

So now the Back-end code:

  • When someone connect on the homepage I send him the static files needed for the application, it will request a websocket connection to my server that it accept and register in the list of client.

  • Every 60 seconds, I want a JSON String from the source website that contains the data I am looking for. I am using the API of the source website, creating the url for the requests.

That I map with grequests and use the response that I manipulate before I store it in Redis.

  • I then send the data to all of my registered client.

    import os
    import logging
    import redis
    import gevent
    from flask import Flask, render_template
    from flask_sockets import Sockets
    
    import json
    import grequests
    import time
    import random
    
    
    REDIS_URL = os.environ['REDIS_URL']
    REDIS_CHAN = 'chat'
    
    app = Flask(__name__)
    app.debug = 'DEBUG' in os.environ
    
    sockets = Sockets(app)
    redis = redis.from_url(REDIS_URL)
    redis.set('before', time.time())
    redis.set('activity_data', json.dumps([
        {
            "name": "Nom" ,
            "n": i,
            "lastEvent": "A",
            "tmc": 0,
            "acti": ""
        }
        for i in range(111, 116)
    ]))
    
    
    class LiveMonitoringBackend(object):
        """Interface for registering and updating WebSocket clients."""
    
        def __init__(self):
            self.clients = list()
            self.pubsub = redis.pubsub()
            self.pubsub.subscribe(REDIS_CHAN)
    
        def __iter_data(self):
            for message in self.pubsub.listen():
                data = message.get('data')
                if message['type'] == 'message':
                    yield data
    
        def register(self, client):
            """Register a WebSocket connection for Redis updates."""
            self.clients.append(client)
    
        def send(self, client, data):
            """Send given data to the registered client.
            Automatically discards invalid connections."""
            try:
                client.send(data)
    
            except Exception:
                self.clients.remove(client)
    
        def run(self):
            """Listens for new messages in Redis, and sends them to clients."""
            for data in self.__iter_data():
                for client in self.clients:
                    gevent.spawn(self.send, client, data)
    
        def run_time(self, function, interval, *args, **kwargs):
            while True:
                duration = time.time() - float(redis.get('before'))
    
                if duration > interval:
                    print(duration)
                    function(*args, **kwargs)
                    redis.set('before', time.time())
                    print('updated')
                    redis.publish(REDIS_CHAN, redis.get('activity_data'))
                gevent.sleep(interval / 10)
    
        def update(self):
    
            data = json.loads(redis.get('activity_data'))
            urlbase = 'http://sourcewebsite/api/monitoring'
            eventactif = (A','B','C')
            evenement = (A','B','C', 'D','E','F')
    
            cookies = {
            'JSESSIONID': 'cookie'
            }
    
            #Create the request
            rs = (grequests.get(
                urlbase + str(key["n"]),
                cookies=cookies
            )
                for key in data
            )
    
            # Map the request  and store it under
            # answer = (reponse1,reponse2,...,response n)
    
            answer = grequests.map(rs)     
    
            for i,room in enumerate(data):
                ro_n = json.loads(answer[i].text)
    
                # Update last event for each room
                room['lastEvent'] = ro_n['room']['lastEvent']
    
                if room['lastEvent'] in eventactif:
                    room['acti'] += '1'
                else:
                    room['acti'] += '0'
    
                if '00000' in room['acti'] or '1' not in room['acti']:
                    room['tmc'] = 0
                    room['acti'] = ''
                else:
                    room['tmc'] = int(len(room['acti']) / 5) * 5
    
            # Sort room by tmc value
            data = sorted(data, key=lambda room: room['tmc'])[::-1]
    
            redis.set('activity_data', json.dumps(data))
    
            return True
    
        def start(self):
            """Maintains Redis subscription in the background."""
            gevent.spawn(self.run)
            gevent.spawn(self.run_time, self.update, 60)
    
    
    livemonitoring = LiveMonitoringBackend()
    livemonitoring.start()
    
    
    @app.route('/')
    def hello():
        return render_template('index.html')
    
    
    @sockets.route('/receive')
    def outbox(ws):
        livemonitoring.register(ws)
        while not ws.closed:
    
            # Context switch while `LiveMonitoringBackend.start'
            # is running in the background.
            gevent.sleep(0.1)
    

So already two questions:

  • Do you think I will kill the server again by overflowing it with request?

  • How do I get rid of my infinite loop ?

\$\endgroup\$
  • 1
    \$\begingroup\$ Does the code above work exactly as you intend it to? As I'm currently reading it as it does not. Code Review requires code that works exactly as the OP intends, so that we're not flooded with bug-fixing questions. And so if this doesn't work as you intend, please fix it, where we'd then be more than happy to review it. \$\endgroup\$ – Peilonrayz Jun 19 '17 at 12:01
  • \$\begingroup\$ The previous one was the one not working. This one work as intended but that's what I see, I may miss something and I don't want to cause another problem. And sorry for the bug fixing question, I'll close the topic when just the second question is answered. Maybe you can answer the second question, how would I do an update without a infinite loop (while true loop)? \$\endgroup\$ – Loerzt Jun 19 '17 at 12:04
  • \$\begingroup\$ Thank you, my apologies I misread you question. As for the while True in run_time, if you want it to run infinitely, then there's no need to remove it is there? Or is there a point that it should stop running that you're asking us to fix? \$\endgroup\$ – Peilonrayz Jun 19 '17 at 12:11
  • \$\begingroup\$ I thought infinite loop disgracious and unstable and to be avoided at all cost, but I have both asynchronus and syncronus action to make. I don't know how to do without it, so if someone know :) If you have other things that you see that could be improved don't hesitate ! \$\endgroup\$ – Loerzt Jun 19 '17 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.