This is relatively simple solution you have here and it could in theory work, but in my experience in a real word you need something more flexible. Also you dont think about redundancy of the data.
What if a doctor for example works only one day in a week. Most of the columns would then be empty. Or if a doctor can work only in the morning? Then there would be no break_start or end in any of the calendar fields. What if, for example, the doctor works every second week? Then you will not have an easy way how to store this information. Also there would be a lot of redundant information. If you will have each doctor working from 8:00 AM to 4:00 PM, you will have this data stored there redundantly. Much less space is needed, if you can store such information as a relation to some kind of duration record.
If I were you, I would create more generic design where you can store up to 365 records (for each day in the year), for each calendar. Then each day would have a relation to a list of intervals, that could later be of required activity. That way you would end up with more flexible design.
The draft could look something like this:
doctor = models.ForeignKey(Doctor, on_delete=models.CASCADE)
clinic = models.ForeignKey(Clinic, on_delete=models.CASCADE)
date = model.DateField()
calendar = models.ForeignKey(Calendar, on_delete=models.CASCADE)
activity = models.Integer(choices = [(1, 'Working'), (2,'Lunch Break'), (3, 'Afternoon Break') ...])
start = models.TimeField()
end = models.TimeField()
calendar_day = models.ForeignKey(CalendarDay, on_delete=models.CASCADE)
In order to ensure data integrity, then you could add an unique constraint on the combination of date and calendar columns in the CalendarDay table. That way each calendar can have only one record for particular date in the CalendarDay table.