Tell me more ×
Code Review Stack Exchange is a question and answer site for peer programmer code reviews. It's 100% free, no registration required.
up vote 0 down vote favorite

I have an RSS reader project and I need to update my database automatically. I have used cron but I have to set new crons every time I have new links and it also get me stuck because of too many cron jobs must be set. Also I have runtime and I/O problem with each cron.

The second way I found was I added an AJAX in user page so that it runs my serverside script and when it finished it calls the next and on. I dont know what are the real ways?

Meanwhile I have problem with I/O. My server admin argues about the run time and I/O of each URL.

I have a feed table and feeds_items (contains RSS items associated with each feed), so when my script opens a link compile it and then check it if they are not already in the database, it will be inserted but because there are too many items and the size of them are so high I have problem. What can I do?

share|improve this question
1  
This site is strictly for improving working code so your question is off-topic here. You've also got various different questions. Please ask your questions on programmers.se one at a time. – Winston Ewert♦ Jul 26 '12 at 15:00

closed as off topic by Winston Ewert♦ Jul 26 '12 at 15:00

Questions on Code Review Stack Exchange are expected to relate to code review request within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.

1 Answer

up vote 1 down vote

I don't understand the last part of the question but I'd try something like this:

  1. In the feeds table store a lastChecked and a checkFrequency attribute for each URL.
  2. Use only one cron job which runs relatively often (every 2 or 5 minutes, for example) and checks every URL if it has to be downloaded/refreshed or not. (lastChecked + checkFrequency < currentTime)
share|improve this answer
This is very useful and I thank you, but when i use this (also in mentioned and the like) ways, every RSS url has sometimes up to 10 new feeds which should be inserted , so this fast insert in the database cause I/O problem for my server – user539656 Jul 25 '12 at 18:48
@user539656: How many inserts does it mean in a minute? – palacsint Jul 26 '12 at 18:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.