Comment by @flodel:

The CRAN High-Performance Tasks Views mention the speedglm package. It is Worth a try. Note how it says "High performances can be obtained especially if R is linked against an optimized BLAS, such as ATLAS". You will find many articles showing you how to do that if you google R blas atlas.

I'll point you to these results showing how switching from the default blas shipped with R to OpenBLAS improved this person's qr decomposition (what lm uses) computation times by a factor of ~4 (from 417 to 113 ms). So regardless of whether you choose to try speedglm, it is definitely worth looking into what blas you are currently using and possibly switching to a better one.