Timeline for determine residuals and outliers
Current License: CC BY-SA 4.0
24 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 5, 2023 at 21:00 | history | tweeted | twitter.com/StackCodeReview/status/1622339290125549570 | ||
| Jan 3, 2023 at 18:47 | answer | added | Sᴀᴍ Onᴇᴌᴀ♦ | timeline score: 2 | |
| Jan 3, 2023 at 18:37 | history | edited | Sᴀᴍ Onᴇᴌᴀ♦ | CC BY-SA 4.0 |
make title describe what code does instead of goals - refer to help center section "Titling your question" on https://codereview.stackexchange.com/help/how-to-ask
|
| Oct 20, 2015 at 9:03 | comment | added | Bas | @Flodel, if you create your comment about the HPC as answer I will accept it, you can't believe how much it helped me :) | |
| Sep 23, 2015 at 6:32 | comment | added | Bas | @flodel Thank you for the feedback, trying to get speedglm working atm and will switch my BLAS to Atlas soon! | |
| Sep 22, 2015 at 7:31 | history | rollback | Bas |
Rollback to Revision 3
|
|
| Sep 22, 2015 at 6:39 | history | edited | Bas | CC BY-SA 3.0 |
edited title
|
| Sep 22, 2015 at 1:11 | comment | added | flodel |
I'll point you to these results avrahamadler.com/2014/04/20/r-3-1-0-openblas-speed-comparisons showing how switching from the default blas shipped with R to OpenBLAS improved this person's qr decomposition (what lm uses) computation times by a factor of ~4 (from 417 to 113 ms). So regardless of whether you choose to try speedglm (my other suggestion), it is definitely worth looking into what blas you are currently using and possibly switching to a better one.
|
|
| Sep 22, 2015 at 0:38 | history | edited | flodel | CC BY-SA 3.0 |
fixed profiler display
|
| Sep 21, 2015 at 23:43 | history | edited | Jamal | CC BY-SA 3.0 |
deleted 304 characters in body
|
| Sep 21, 2015 at 23:08 | comment | added | flodel |
The CRAN High-Performance Tasks Views (cran.r-project.org/web/views/HighPerformanceComputing.html) mention the speedglm package (cran.r-project.org/web/packages/speedglm/index.html). Worth a try. Note how it says "High performances can be obtained especially if R is linked against an optimized BLAS, such as ATLAS". You will find many articles showing you how to do that if you google R blas atlas.
|
|
| Sep 21, 2015 at 22:47 | history | migrated | from stackoverflow.com (revisions) | ||
| Sep 21, 2015 at 16:15 | comment | added | Ben Bolker |
For further speedup you could try RcppArmadillo::fastLm (see here and here)
|
|
| Sep 21, 2015 at 14:52 | comment | added | Bas | @Erasmortg the 'Coalesce' grabs the first of the three columns that contains data. The "to_char" could probably be removed, I took that part from our other query as I didn't knew what coalesce was before. In which way do you think I could index the statement, as I'm not entirely sure what you mean. Also reading more about bind_rows now. Also looking at the Rprof() data in my edited question I think the bottleneck is lm(), and not the query! | |
| Sep 21, 2015 at 14:36 | comment | added | Bas | Thank you @BenBolker, time has already been reduced to 15 minutes using lm | |
| Sep 21, 2015 at 14:36 | comment | added | erasmortg | @Heuer What is your 'coalesce' doing? I can't seem to identify any expression in particular (other than to_char). You could retrieve those and perhaps pass by reference in R? Perhaps using a subquery to index first on your where statement. If you think retrieving pieces first could be faster, try so and then try to bind_rows in R | |
| Sep 21, 2015 at 14:22 | comment | added | Bas | @erasmortg I considered splitting up the query into pieces to reduce memory load, but I need the retrieved data as a whole to calculate the residuals :( | |
| Sep 21, 2015 at 14:20 | comment | added | Bas | @erasmortg I need every little bit of that query. Which transformations dyou think could be improved? :) | |
| Sep 21, 2015 at 14:03 | comment | added | erasmortg | I'd look at optimizing the sql queries. You should (probably?) have some sort of client to explain the query you are trying to run. Start there. The transformations you are doing could potentially be faster in data.table (which you are loading anyway) | |
| Sep 21, 2015 at 14:03 | comment | added | Roland |
or even lm.fit if you only need the residuals
|
|
| Sep 21, 2015 at 13:49 | comment | added | Ben Bolker |
try lm instead of glm ?
|
|
| Sep 21, 2015 at 13:47 | comment | added | Bas | Thank you @Roland ! Will give it a try, I was actually just reading about it :) | |
| Sep 21, 2015 at 13:45 | comment | added | Roland |
There is a technique called "profiling", which you can use to find out which parts of your code are actually consuming most of the time. Learn that technique. See help("Rprof") and several packages for the R part of your code.
|
|
| Sep 21, 2015 at 13:35 | history | asked | Bas | CC BY-SA 3.0 |