Texture crisis -- a plea for workarounds

Answered

Comments

49 comments

  • Avatar
    Kevin Cain

    My plea above is for any help getting a large scene to compute textures without the 'sanity check' error noted above.

    I don't want to draw attention from that, but below is an addendum as I've tried to understand the circumstances of the problem -- which entails it's own problems:

    Since texturing mostly works with my small scenes (<100 images) and not with my large scene (~4000 images) I'd like to cut the large scene into pieces to see at what point things stop working.  We can't delete images/cameras, so the only way to test that I can see involves stepping through thousands of cameras in the file by hand, one by one, to turn them off for a given reconstruction. 

    We can export cameras and images as .fbx, for instance, but then I'm stuck trying to group thousands of cameras in the file by hand and we can't import .fbx back to RC so that also seems not to work.

    0
    Comment actions Permalink
  • Avatar
    Jennifer Cross

    Hi Kevin - rather than one-at-a-time on the pictures you can shift-click bunches of them and disable them for texturing (the inputs box applies to all the selected images)

    At least that way you could play with the numbers to use anyway. 

    Is there a chance this is related to the licence limit of 2500 photos for some classes of licences?

    Good luck on the test... Jennifer 

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hey Jennifer,

    2500 should be ruled out since he has the CLI, but you never know - would be easy enough to test.

     

    Hey Kevin,

    do you know the grouping menu that is activated by clicking on +Images in the 1D menu? Makes grouping possible in one click, provided you don't have any special requirements. Why do you want to export the cameras? Just to be sure? Have you tried using export Registration and importing that as a component into a new file?

    One more idea, but I guess you already came across this: it seems like intersecting surfaces due to slight misalignments can cause RC to react erratically and in your intricate project I can see dozens of places for something like that to happen...

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Another silly idea - have you tried texturing in a third party software? With the camera export it should be rather straightforward, even MeshLab does that. Although the amount of images could be a challenge...

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Hello and thanks Jennifer and Götz!

    Your mutual points about grouping are good -- but since the camera list is not currently grouped, it would take thousands of manual clicks to group them appropriately now.

    Jennifer, as Götz noted, I'm using the CLI version so theoretically the number of cameras/images is fine.

    Götz, I understand your fear that RC could be acting strangely with intersecting geometry, but I've successfully textured this data set in the past weeks, with no problems.  The only difference is that I've subsequently managed slightly better alignment.  There is nothing novel in the scene after the last alignment round -- identical poly count, identical input images, only slightly better alignment.  So I don't see why RC is now giving texture errors for a fundamentally identical reconstruction. 

    I could throw away my hard-won alignment (hard won because alignment itself crashes RC after hours of computation in this project, as you've seen in a separate thread) to get back to the previous textured version, but that seems silly.

    Instead, I hope is that somebody has figured out when, and why, this particular 'texture sanity check' error arises.

    A hint:  if I strip out geometry to a minimal number of triangles, RC completes the texture.  I don't know why.

    To answer Götz's other question and suggestion:  I have thought of exporting the cameras to Maya to bake a texture from their aggregate projection, but I see in this forum that people seem to have problems getting thousands of cameras to import correctly from RC-->Maya, so I haven't been enthusiastic about that.  Better and simpler I think to keep this in RC, where a core function is texturing.  If people think that ~4K cameras is viable in Mudbox/Maya/et cetera, I'll give that a go but again prefer to stay in RC as the mesh reconstruction is native to RC and it should be able to texture its own simplified mesh.

    0
    Comment actions Permalink
  • Avatar
    Jennifer Cross

    Oh sorry - this wasn't a suggestion about making a group of them...  Instead just group-select a bunch with shift click, then set the selected cameras to not be used for texturing (just to see if fewer cameras gets you over the line).

    Then select all the cameras and return them to service.  No playing with the group id's etc -  just  bulk selects.  you could even lasso select in 3d to disable them by region... 

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Kevin,

    I'm always reluctant to point things out to you because you are clearly playing in a different league.  :-)

    But I don't understand your point about grouping - did you not mean exif grouping? That would be one click for all cameras grouped. You can also select many, like Jennifer has pointed out, and change whatever value you want for all of them.

    About the 2500 - it might be worth a try, maybe there is some little chink in the code somewhere.

    I see your point about the hard won alignment. Although your experiments prove (in my view) that it is somehow tied to the mesh geometry, since it seems to disappear when you simplify it to a high degree. So the intersecting faces do seem like a possibility for the source of the error. Have you tried cutting random parts of the mesh away to narrow the spot down? Maybe downscaling images for texturing helps reduce calculation times for this.

    Sometimes it can be faster to talk about such things in person. If you think I might be able to get you any further, feel free to call me - you should be able to find my details in google with my name...

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Thanks again Jennifer, that's a good suggestion, I don't know why I was making it more complicated in my mind.  ;-)

    I group selected all the cameras and turned their texture contribution off, then picked just a few cameras to turn on as a test.  That worked -- or at least got past the 'sanity check' stage, like my test where I arbitrarily removed 99% of the triangles in the mesh.

    Unfortunately, even with just (3) images and an 8000 triangle count mesh, RC crashed 50% of the way to completion when it ran out the 128GB of memory available!

    I can't begin to wrap my mind around how RC can spend 128GB of RAM to texture an 8000 triangle model with three photos...I must be doing something odd, but I have checked again and am using stock (reset) settings for RC globally.

    All of which suggests that there is a camera/geometry line here, and makes me wonder how I was able to texture models as large before!

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Thanks also Götz, especially for your very kind offer to try to sort through this on the phone.  I hate to abuse your kindness, but given my desperation with this particular project I won't rule it out!

    Let me first do some more homework with re-projecting the corrected images and RC camera positions in other software, to see what kind of reprojection error there might be and whether dealing with thousands of 24MP images is tractable for VFX software pipelines.  While those kinds of demands are nothing special in the MVS community, VFX software is still built around expectations of hand-built geometry and traditionally uv texture atlases, not thousands of images and very large models.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Kevin, no worries - I offered!  :-)  The worst that can happen is that we both wasted a few minutes on the phone...

    My bets are still on the geometry. Better numbers in alignment does not necessarily mean "better" geometry in my experience. So given your project I could imagine some tiny metal object in the background somewhere that causes some inconsistent geometry.

    0
    Comment actions Permalink
  • Avatar
    Wishgranter

    Hi Kevin Cain 

    What settings are you using for UNWRAP ? can you make screenshot just of the  UNWRAP settings  used there ? 

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Hello, @Wishgranter,

    Here are my unwrap settings for this project:

    The settings for the 'Unwrap' tool in Reconstruction:Tools match the above.

    I've tried both the new and 'legacy' simplification and unwrapping.  The new simplification hangs when converting the ~750M triangle source to 3M triangle target, so I've defaulted to 'legacy' in both simplification and unwrapping.

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    I don't know if this relates to textures failing with the 'sanity check' error above, but I notice that RC is again turning off groups of aligned cameras by itself, which I reported earlier and was assigned bug 6983 here.

     Here's an example of a block that was turned off:

     

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    When I explicitly call the unwrap tool before texturing, I sometimes get an unnamed error banner shown below:

    As you can see in the upper right, the last console output is 'Texturing model failed...' with no further details.

    I've seen this with settings for [1 tile at 8K], and [1 tile at 512].

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain
    Here's a summary of the sanity check error in this thread and what I've tried -- all comments welcome!
     
    The 'sanity check' error and steps to reproduce:
     
    Before the problems in this thread, I successfully textured a simplified 3M triangle model from ~4,000 aligned source images -- let's call this reconstruction 'A', via these usual steps:
    • import images, CPs from .csv (previously exported by RC)
    • align (iterative)
    • reconstruction at normal detail with hand placed bounding box
    • simplification to [3M triangles, 1 part] from above reconstruction [751.5M, 500 parts]
    • texture [24 tiles x 16K]
    So far, so good.  Next, reconstruction 'B' shares ~3,900 of the same source images as reconstruction 'A'.  The difference, as noted in this thread, is that RC added ~100 cameras to the prior ~4,000 camera alignment between reconstruction 'A' and 'B'.
     
    While I had no problem texturing reconstruction 'A' up to [24 tiles x 16K], reconstruction 'B' fails even with [1 tile x 8K].  If I explicitly call unwrapping, it consistently gives the unwrapping error shown (at 512 to 8K) but succeeds at 16K.
     
    Even when uv unwapping succeeds, during texturing RC gives the 'Texturing stage 01 sanity check' error that is the subject of this thread.
     
    What works and potential clues:
     
    If I manually turn off texturing for all but a few cameras in the 'Camera poses', RC will complete the texture -- but of course, only with the contribution of those few cameras.
     
    After many trials I've increased the number of cameras with texturing on to a maximum of about 1/4 of the cameras in the scene, ~1000.  Any more than this again gives the 'sanity check' error.
    I've manually winnowed the camera list to see if individual cameras are causing the error, but so far it seems that it doesn't matter which cameras are set to participate in texturing, just the total number of cameras set to participate.
     
    After texturing fails, RC will sometimes turn off groups of cameras in the 'Camera poses', as shown above.  This is not consistent.
    0
    Comment actions Permalink
  • Avatar
    chris

    have you tried more than a max of 10 textures?

    for that many photos, i'd be setting the max to 32 or 64. it will still only use as many as it needs.

    have you also tried photo consistency? it slower but it may help. 

    i would also have a go at 16k textures too. they will be easier to resize smaller less textures anyway.

    I would also just do unwrap stage until you have that looking like its all working ok.

     

    there is probably a small part somewhere in the mesh that's a bit dodgy, it might be worth exporting mesh to see if you can see any parts that could be causing the issue, and then maybe make a new reconstruction region to build a new mesh that dosen't include it.

     

     

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Following Jennifer Cross' suggestion, I manually turned off texturing in groups, and by walking through the whole set of images in this way was able to isolate a few images that were causing the texturing to crash after unwrapping. 

    The lesson (for me, at least) is that images where RC finds very few correspondences can cause crashing during texturing.

    This gave me part of the answer -- at least I have been able to get a rendered result in most cases -- and now I'm stuck with a new problem, strange texture results.  But that is a new topic for another thread.

    Thanks @chris, I agree that it's best to unwrap first, then texture.  Your other suggestions were also well taken, though they didn't relate the crashes in my case.  Thanks again!

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hey Kevin,

    that's a very helpful observation, thank you!

    By correspondences do you mean Tie Points or "partner" images? Is there an easy way to determine the latter?

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Tie points are the crucial numbers to watch for this in the case of this crash -- an image with less than 10 tie points is able to reliably cause crashes in my experience.

    However this is connected to partner images, in that user-supplied control points establish image neighbors.

    If I had to guess, I would say that when RC cannot reconcile user-supplied control points during bundle adjustment, it's left with erroneous points and therefore erroneous tie points.  The default weighting value of 10 isn't enough to drag the tie points from SIFT too far out of alignment, since they are much more numerous, even for relatively featureless images. 

    However, in the 'right' cases. these points are enough to cause RC to establish a few tie points with images whose adjacency doesn't match what it expects from bundle adjustment -- and this disparity leads to the crash.  That is, RC should know better but can't downweight the user control points -- because they are correct (see below), but in locally aligned groups that don't manage to be reconciled into a global alignment.  In my guess, only, as RC is not open source.  ;-)

    I say this because my observation is that in complex scenes with hundreds of control points, even with high quality points (that is, placed < 1 pixel in reprojection) RC sometimes can't find a global alignment that fits all points.  In this case, it reports high errors for control points we can objectively see are well correlated.

    In these cases, for whatever reasons, RC alignment stops in a local minima, stopping short of a BA that fits all points.  This is to some extent unavoidable as in incremental sequential alignment, large scenes accumulate error.  A global SfM approach would be better at closing the loop in these cases, but more brittle and likely would incorporate fewer images of the whole set.

    It seems RC prefers to try to make use of all possible images but is then unable to square the alignment error vis-a-vis user control points in complicated scenes.

    If anyone has any experiences along these lines I'd love to hear.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hmm, I didn't undestand half of what you said again, which is no surprise...  lol

    I fear there aren't too many people out there (or rather in here) who understand the intricacies of the theory behind all this as thoroughly as you! I hope to be proven wrong!

    It is a real problem though and in my opinion it could and should be possible to implement some kind of probability filter to identify the obvious errors, like I suggested for the distortion model once. https://support.capturingreality.com/hc/en-us/community/posts/115001352772-identify-obvious-distortion-errors

    Since I am not sure how closely RC team members follow extended threads like this, I would suggest to create a new bug report or feature request.

    0
    Comment actions Permalink
  • Avatar
    chris

    I've seen this I think.

    I would get lot more crashes as i added control points. it only affected some projects.

    it was probably fighting with the tie points.

    i think adding minimum number of control points needed helps. in case you add errors with each one you place.

    0
    Comment actions Permalink
  • Avatar
    Benjamin von Cramon

    Hello Kevin (and others),

    I've been experiencing similar behavior under somewhat different conditions, believe these problems and ground causes are related, so let's compare notes and explore workarounds. I get the same sanity check red flag when, instead of running textures, I'm running reconstruction at normal. The one difference is, in console I'm seeing the same invalid function call with identical "0x2012\0x2010\0x2000\0x4999\0x5051" as reported in this post, which also features my response.

    Kevin writes,"Tie points are the crucial numbers to watch for this in the case of this crash -- an image with less than 10 tie points is able to reliably cause crashes in my experience."

    In my experience, I get consistent crashing with images having plenty of tie points, but I believe too few where I need them, these regions being in the distance, which only exacerbates re projection errors. I was aware during capture that the subject matter, a transition between a large space in a mining tunnel through a small window high up on a wall leading into an upper chamber (a ventilation branch), would call for great care in how I stack the ordered pairs of images leading into this high up window. Without a jet pack, I was further constrained by precarious stances on ladders set on lose cobble, cantilevering the camera out as far as I could reach at times, really sketchy work conditions. I might have used my extension pole, but believe that wouldn't have delivered much different perspectives. Another factor, dust and fog really high that day, producing back scatter and polluting the subject matter into the distance. Here's 2D in 6-panel view showing some of these images with tied points enabled:



    You see what I mean with a dearth of tie points in the distance. So, might it be more than simply not having enough tie points in the image, but not enough where they're needed? I've wondered why when users have written in questions about capturing objects, that dominate the center of the frame and asking about how the background figures in, whether it can be masked, whether it's preferable to let distant background subject matter go into soft focus or dark, why Wishgranter suggested RC supported such notions, that RC doesn't benefit from that information, that it might even mess with properly modeling the object. I can see how these two scenarios aren't in conflict. If RC keeps its eye on the object, no problem whatever the background wants to do, but if you're capturing the environment, now corner to corner in every frame, that data now has to properly triangulate, very different situation. 

    To Kevin's thought that the problem is born when insufficient tie points create a situation locally, that may pass the test through alignment, and in his case through reconstruction, but not through unwrap/texture, that this localized solution hits a wall when tied into loops at the macro scale, I find that idea most intuitive. Now what? Should the bar be raised on what allows a component to stay together at alignment? BTW, I also run iterative alignment, ratcheting down on max reprojection error, RC gets me to below .5 with a mean around .3. I can live with some minor flaws around a transition, much support keeping the component whole, but then there needs to be some work around.

    My proposed solution is to provide for allowing the user to export either an .rcalign file or a smaller chunk mesh that preserves scale and coordinate system, allowing user to process this smaller section successfully by not requiring it to solve against the global model, to then match it up as close as common scale/coordinate system will satisfy. I can deal with the minor flaws at the seam between the globally modeled set and this smaller problem child mesh. I'm wondering if RC already supports this, haven't worked with ground control points in my work. Can I set a ground control point in the larger set and export that with the smaller reconstruction region used to define reconstruction of the problem child mesh section, and will that provide this common scale and coordinate system? Thanks for other ideas.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Benjy,

    I think the XMP workflow would be ideal for this. That's the one I didn't remember last time we chatted. What it does is very similar to working with a RAW file and the corresponding file with the alterations in it. If you press that with all the cameras selected that you are interested in, it creates XMP files with all geometric data of the cameras in it. If this XMP is present and has the identical name as the image, then RC takes those infos instead of calculating new one. You might have to tick a box somewhere to tell RC that the position should be fixed.

    0
    Comment actions Permalink
  • Avatar
    Benjamin von Cramon

    Nice, didn't realize RC supported exporting scale/coordinate system over XMP metadata. I'll have to hunt for where to "press". I may have found another work around in the meanwhile. Once this clicked about the problem resting in this dramatic transition, with no great way to counteract that with stronger capture, I thought to disable all photos across this divide, RC had no problem reconstructing mesh D, let's call it, since A-C were in place. In the same component I can then reconstruct everything falling on the other side of the divide with a neighboring reconstruction region, now disabling all the photos falling on the A-D side of the window and enabling the previously disabled images. I might need to do a little trimming, and the seam may not be perfect, but I'm confident this approach is optimal, and surely delivers an identical end result as what I'd expect comes from the XMP workflow, yes?

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Benjy,

    awesome that you solved it! Very important insight.

    That really reeks of bug to me though. The software should be able to deal with such situations...

    And yes, I think it would be the same result. The button is "hidden" in Alignment - Export. I'd written it already but then deleted it again for apparently no reason...  ;-)

    0
    Comment actions Permalink
  • Avatar
    Benjamin von Cramon

    Thanks, really curious how close this gets in the seam. I'll post some snips from UE4 to show results. 

    About it being a bug, yes and no. If humans doing the best we can to mitigate the challenge posed by the data deliver acceptable results, then yes, RC should also be able to support a solution to the bug. But, given the issue with either too few tie points in the image or not where you need them to connect two very different/adjoining chambers, it's also fair to say any SFM solution has its limits. I do believe the human solution here may point to a strategy for CR coders. Since RC was able to deal with the challenging data to deliver a mesh in Preview, it should be able to isolate, as I did, which images fall to either side of the divide where the math becomes too challenging, solve separately for each before then producing the combined result. That is, if I'm able to reconstruct each separately while maintaining the same scale and coordinate system from a common component, then bringing these together later, why can't RC do something similar. If it can present other anomalies, like a step in a plane where groups of related cameras aren't tied in with other groups, that's in essence the same behavior I'm suggesting inviting here, bring the two sides of this divide together the best you can, let us deal with cleaning badness in the less than perfect seam.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Well, in my opinion every unexplainable crash (and I mean NOT having to experiment like you guys did for ever) is a bug. Also, it gets much more atention in the specific bug area of the forum. It is no bug anymore if RC stops the process and tells the user why it did so and, ideally, what the user should do.  :-)

    0
    Comment actions Permalink
  • Avatar
    Benjamin von Cramon

    As usual, your logic is ahead. Thanks, RC if you're reading, for addressing this bug.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Says the one who has worked out the clues...  ;-)

    0
    Comment actions Permalink
  • Avatar
    Kevin Cain

    Thanks Benjamin for letting some light into the topic, Chris for saying you're seeing related errors in your world, and as always to Götz for stepping in faster than those of us in the New World can!  ;-)

    To quote Benjamin:

    In my experience, I get consistent crashing with images having plenty of tie points, but I believe too few where I need them, these regions being in the distance, which only exacerbates re projection errors

    You're certainly right -- the tunnel you describe makes it hard (or, maybe, impossible) to get fully supported access in an unbroken chain extending to the furthest reaches.  Based on my recent jaunts with RC, however, I'd be willing to bet that the problem is not the number of tie points.  In my case, I was able to turn off the objects with poor point counts, and the crashing stopped.  But in your case, where you have plenty of points but also registration instability, I'd be willing to bet that the problem is how RC picks the initial views during alignment.

    In other words, yours is a classic failure case for sequential SfM -- not starting the reconstruction with the best initial view, when the order of views matters.  That causes the alignment to fall into local minima before it can converge on the actual best RMSE for the given views.  Tie points establish the network of neighbors, but initial view selection is often independent.

    The good news is that a global SfM approach doesn't have these initialization problems.  Can you test your data set with a (free) system like Simon Fuhrmann's excellent MVE?  (https://www.gcc.tu-darmstadt.de/home/proj/mve/).  Simon's work is very good and he has a nice Windows front end to make things easy, although there are many equally capable systems (Theia, OpenMVG, and others) you could also use.

    Perhaps the best 'production ready' global SfM system is Pierre Moulon's wonderful OpenMVG:

    https://github.com/openMVG/openMVG 

    If you use one of these, just make sure you use the global alignment option, not incremental.

    All of these have the advantage that they're free, and they're open source so you can see what they do and extend them.  They have the disadvantage of not being as well optimized for the GPU as RC, so if you're running on a single PC it will take a lot longer.  However, the final mesh results from my tests demonstrate that while the open solutions are slow compared to RC, they can generate better surfaces than RC:  both cleaner and more accurate to ground truth.  That may be important, or even decisive for you as it seems you're working for a scientific result.

    As Götz and I have discussed in other threads, Global SfM methods solve for all positions at once, so they are much less brittle in cases like yours.  It appears that RC uses a sequential solver, but I've asked for clarification on this and never got a response.  Since RC isn't open source, we can't check for ourselves.  ;-)

    While the backscatter isn't helping the focus, any of those points that get recognized during feature search will be discarded in the next image as the dust moves, so they'll fail the homography test.  If the dust is enough to cause blur, that's a different story -- blur is destructive to reconstructions in exactly the slippery way you describe your current trouble.

     

    0
    Comment actions Permalink

Please sign in to leave a comment.