Running out of memory during alignment
Answered
Hi Guys
I am trying to align about 20k photos + 1 rig lidar (about 250GB). I used to work with the CLI version of RC over several month. Now I am trying to use the PPI for the same project. The new system would hopefully help me to calculate the costs for the client. So far so good.
Now I am running out of memory during the alignment. I have been able to process all the data with the previous version using the exact hardware setup.
In my understanding, RC has an out of core memory algorithm that shifts the data to a harddrive in a case of low memory, right?
My Hardware: CPU Intel i9 14core with 128GB RAM and a GPU1080Ti. Pictures are on an Optane Drive (500GB). Cache is on a SSD 1TB. (Switched the pics and caches around to different drives a couple of times – no luck)
What am I missing here?
Thank you for your help!
-
I have the same kind of issue. Same kind of powerful workstation. This is a new bug.
The latest update?
-
Hello,
what alignment settings do you use? With the default settings you are able to align approx. 16K images at once on a machine with 128 GB of RAM.
In order to align this number of inputs, you need to lower the Max features per image or use components workflow.
-
Hello Zuzana,
thank you for your response and support. Just today I did reset the application like a clean install. This helped me to register 14k along with a lot of little components.
My first question concerning this problem and so that I can get this straight: Can I even align that many photos (20-25k) with my setup and if yes only in batches? If no... the out of core algorithm would not compensate the lack of RAM?
-
Any thoughts on my last question?
-
Yes, it should be possible with the following technique - Whenever you try to work with such big number of images (larger than what the limit is -16K for your 128 RAM) you should split your images into at least 3 separate components. Once you do this, select all of the images and in the "Selected input" menu go to Feature source option and set it to "Use component features". Also navigate to alignment settings and in the advanced set "Merge components only" to Yes. -
As an option, you can just increase your computers virtual memory aka paging file size on you systems hard drive. I only recommend using with an ssd system drive. I only have 64gb of ram and use 200gb of virtual memory and am processing 30k images at 70000 features per picture. Using virtual memory means not a efficient as using ram only, but it works for me.
-
Thank you Mark—That is very good point. I have just added my entire optane harddrive (450GB) to the virtual memory. How do you see that RC actually takes advantage of the additional virtual memory during the process?
-
You can get a realtime drive checking software to see the usage of it. There will also be other solutions on the web as paging file is quite popular and a real good solution here too, without it if your memory leaks and there is none it can get stuck.
Please sign in to leave a comment.
Comments
8 comments