CUDA error : 2 : out of Memory
Answered
specs
Processor: AMD FX-9590 Eight-Core
Memory: 16.0GB
Graphics Cards
- Geforce GTX 1080 Ti
- Geforce GTX 970
OS: Windows 10 64bit
-----------------------------------------------------------------------------
The Error : CUDA error : 2 : out of Memory
-----------------------------------------------------------------------------
This happens every time I go to reconstruct anything other than the preview reconstruction.
Things I have tried
- https://support.capturingreality.com/hc/en-us/community/posts/360006253079-I-keep-getting-CUDA-error-number-2-WHY-?page=1#community_comment_360000847519
- I have followed the instructions from people on this post and have tried every archived version of nvidia drivers available from 441.28 - 436.02
- I made sure that my cache was set to a drive with plenty of space
- I have restarted the process and created a new scene several times to see if it was a bad file
- in settings I have messed with vertex distance, decimation threshold and force single part mode
- I have changed the reconstruction area anywhere from the whole model to just bits and pieces
Things to note about the setup
- processing 335 images
- total number of points: 1,230,123
- image features range from 60,000 - 90-000
- lens model: EF40mm f/2.8 STM
Reconstruction Settings
- have set both GPUs to use and every combination inbetween
- Normal Model Image downscale: 5
- Mesh calculations: anywhere between 0 - 10
- I have left macimal vertex count per part at: 5,000,000
- Detail decimation factor: anywhere between 0 - 100
I have not changed or messed with anything in the Advanced settings.
I hope this give you enough information about the issue and what I've done to try and make a reconstruction happen. If there is anyone with more information on what I should try next or has a setting I should change I would love to hear from you!
-
Exactly there is no difference if you will use one or two GPU units and support only advice o use one graphic card. Besides all my scans was done with high detail on two GPU units until installation of new version of reality capture :( so definitely it's a software issue in new version
-
Hello,
This is our official statement for this issue.
Two immediate temporary solutions:
1. Physically remove all GPU cards, except one - single card configuration.
or
2. Install old GPU driver. The version 419.67 is working for us.
Reason:
The issue is with the CUDA memory de-allocation function, that has stopped working properly with latest NVIDIA GPU drivers.
More specifically the function CUDAFreeHost() resulted with success code, but the memory was not de-allocated and therefore after some time, the GPU pinned memory was filled up and the SW ended up with the message "CUDA error : 2 : Out of memory".
It seems that this bug causes problems also to another CUDA developers:
https://devtalk.nvidia.com/…/cudafreehost-not-clearing-all…/
We have contacted NVIDIA and we hope that there will be a new driver with fix very soon.
Meanwhile, we are also trying to fix the problem by some workaround in our code and we will try to release hotfix ASAP.
-
Hi, I have this CUDA error popping up once in a while without any clear logic - is sometimes appears, sometimes not -no matter if the project has 40 or 900 images - but when it does, it is usually during the texturing process and it keeps reappearing repeatedly even after canceling the error window many times - so as the result the texturing process could not be finished...
I have tried the newest Nvidia drivers, and the older one recommended above (419.67) but it still keeps appearing, can you help me solve this, please???
PS I do not have SLI - just one older GeForce GTX 750 card...
-
Sorry for posting in this old thread, I just came across the same memory problem and didn't want to open a new one just to post what helped me fix it:
I've got a GTX 1050Ti and a GT 710 that I only use for extra monitors. I've reconstructed parts of the same mesh multiple times before but when I started the full reconstruction in high detail I got the same "low device ... memory" error message. It looks like RC ran out of memory because it was using the 710 instead of the 1050 (which should be the main one).
You can choose the GPU RC uses in:
Reconstruction - Process / Settings - Image depth map calculation / GPUs to use
I set the 710 to "False" and it looks like it's working fine now. :)
Please sign in to leave a comment.
Comments
15 comments