Timeline for Read bunch of files in parallel and populate concurrent map
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 14, 2022 at 7:17 | vote | accept | AndyP | ||
| Mar 21, 2022 at 19:08 | comment | added | AndyP | Yeah my server didn’t come out of startup since last night. We have more than 700k products. The upsert function is O(n**2) where n is the length of the slice I guess. I think that is a problem if n is big which is in my case. Are there any other options I can try to see if it can work for me? If you can suggest some other ideas then I can try it out | |
| Mar 21, 2022 at 13:40 | comment | added | rose | Linear scan may be slower, I didn't test it. The trade offs will depend on how many productIDs there are in the map. | |
| Mar 21, 2022 at 7:33 | comment | added | AndyP | Thank you. Also for some reason the idea of using linear scan to upsert concurrent map is somehow very slow for me when I run with all 60 files. It takes lot of time to populate the map. I am not sure why. Any idea on that or anything that can be improved? | |
| Mar 21, 2022 at 4:01 | comment | added | rose | I fixed the bug which would be causing that panic. I would recommend using int64 instead of string as the map key, this should help the memory usage. | |
| Mar 21, 2022 at 4:00 | history | edited | rose | CC BY-SA 4.0 |
added 12 characters in body
|
| Mar 20, 2022 at 19:35 | comment | added | AndyP |
Also I got an error while using duplicate map logic you added as panic: interface conversion: interface {} is int64, not []int64. Any idea what is wrong here?
|
|
| Mar 20, 2022 at 19:18 | comment | added | AndyP | Yeah I went back to the previous question where you also answered same thing so I noticed you forgot to release semaphore. I am testing these changes and so far looking good. Also wanted to ask is there a difference between keeping any id as int64 vs keeping it as string in a map as key? Will that affect memory usage too? | |
| Mar 20, 2022 at 14:38 | history | edited | rose | CC BY-SA 4.0 |
added 2133 characters in body
|
| Mar 20, 2022 at 14:20 | comment | added | rose | Sorry my original example code forgot to release the semaphore. Fixed. I picked 100 arbitrarily, you can decrease this to trade off memory use/speed. | |
| Mar 20, 2022 at 14:19 | history | edited | rose | CC BY-SA 4.0 |
added 38 characters in body
|
| Mar 20, 2022 at 8:58 | comment | added | AndyP |
I am still using this as it is for now g, ctx := errgroup.WithContext(context.Background()) for testing this out. I hope this isn't causing issues with memory with semaphore changes I did. Also I am noticing very weird thing, if I use 20 instead of 100 in semaphore then my file loading isn't completed at all as I have total 60 files. Not sure what's happening behind the scene. Are we missing anything in semaphore logic?
|
|
| Mar 20, 2022 at 8:21 | comment | added | AndyP | I am going over that presentation now which you shared in the other question. Btw I have total number of files as 60 max so 100 is good number here for semaphore or should we bring it down? Also I just tried this version on aws environment where I replaced my worker pool implementation with semaphore changes and I am noticing memory is almost full on the box even after file read is completed. And my original worker pool build takes only half of memory on what I have on the box. Not sure what's wrong here with semaphore changes. Is it because of 100 or something else you can think of? | |
| Mar 20, 2022 at 8:01 | comment | added | AndyP |
Thanks for your good suggestion. I am working on making changes related to worker pools. So if I insert directly into concurrent map then will there be a problem from main application threads reading all the data from the same map? I mean like any performance issue or any locking issue or anything else? Also can you provide an example on this inserting directly into the concurrent map without constructing a separate map locally using my same concurrent map upsert strategy?
|
|
| Mar 20, 2022 at 4:19 | history | answered | rose | CC BY-SA 4.0 |