I am working on an app that analyses the activities of users on some external sites. The contents of the target sites fall into three types:
Forum post: A user can either start a thread or reply to an existing one, the reply will get an unique url;
Blog entry: Any user can post. The blog entry gets an unique url, comments do not.
News article: Not user posted, from a number of news sources. Comments do not have an unique url.
The focus of my app is on user activities, below are some possible questions:
- How often does user A post (with respect to types, daily, weekly, etc.)?
- What topics are the user mostly interested in? (for this, all threads will be tagged based on their original category and scanned keyword from title/content).
- Is user A tend to reply user B's posts more often then to user C's? (obviously this particular question suits more on forum posts and blog entries).
I plan to consolidate a
Thread model to hold all three types of contents, do not address the diffrences in them, for example, theat the news source just as a regular user. The
Thread model will have roughly 10 millions of records.
Below are the models that I came up in very rudimentary form. I have few questions that need your help:
- Is the logic clear in the models?
- Regarding the
last_seenfields in Nick model: if I can look up the Activity model inexpensively in runtime, I would rather not to have them, since the two fields need to be computed each time I insert a new record. What are my better options here?
- There will be about 10 sites included, should I have another model just for sites?
Activitymodel exists only for a forum type, meaning quite a few millions of records (Blogs and News) will not have a value there, is this a good practice?
Thank you all very much! I appreciate any input!
# should i index some fields? class Thread(models.Model): site = models.CharField(max_length=40) thread_type = models.CharField(max_length=20) category = models.CharField(max_length=20) title = models.CharField(max_length=200) pub_date = models.DateTimeField() url = models.URLField() hits = models.IntegerField() votes = models.IntegerField() def __str__(self): return self.title class Nick(models.Model): site = models.CharField(max_length=40) username = models.CharField(max_length=40) first_seen = models.DateTimeField(null=True, blank=True) last_seen = models.DateTimeField(null=True, blank=True) class Meta: unique_together = ("site", "username") def __str__(self): return self.username class Activity(models.Model): usernames = models.ForeignKey(Nick) threads = models.ForeignKey(Thread) post_date = models.DateTimeField(default=datetime.now, blank=True) is_original = models.BooleanField() url = models.URLField() # in case a forum reply and not original def __str__(self): return Nick.username # needs to be more informative
Update: The users of all sites are almost from the same population, they may register on multiple sites, may or may not use the same nickname. One of my research goal (in a later phase) is to explore the correlations between users of different sites.