Workflow Process - Interior Image Alignment

Answered

Comments

47 comments

  • Avatar
    ValerioPaolux

    Hi Mike,

    thanks for sharing. I work on large VR environments (both exteriors and interiors) for Unity. I generally deal with large assets as the tolerance for both polycount and texture count is fairly high when developing for Desktop/Laptop. Generally a model will be between 1 to 5 million polygons, with around 15 to 50 8k texture maps. 

    I usually reconstruct in Normal Detail using the default settings. For finer detail reduce the decimation factor to 0.5 or lower.  

    The built in Simplify tool is a good start, although in my experience will generate holes. Plus, you have no control over the distribution of the polygons (dictated by your tie-points and initial reconstruction). I find that a combination of Autodesk Recap Photo, and ZBrush will produce the best low poly meshes. Recap Photo is phenomenal for filling holes, and is also great at re-baking your textures (Useful when you want to change texture count without re-texturing assets in RC). Zbrush is great for decimating your mesh and hitting your target polycount, and you can customize density using masks. 

    I use Autodesk Maya to manage the various models and sub-models that comprise my final environments. When touch ups to the textures are necessary I go with Photoshop or Mudbox (using UDIM to paint directly in 3D). Finish by packing everything into an .fbx file that goes straight into Unity. 

    Hope this is helpful. If you want more details just ask! 
    Cheers

    2
    Comment actions Permalink
  • Avatar
    Not_Nick

    Excellent thank you! 

    0
    Comment actions Permalink
  • Avatar
    Lucia CR

    hello Mike,

    thanks for sharing, I'm sure lots of RealityCapture users will find this topic very valuable, since scanning interiors is quite a challenge

    here you can see the winner of our competition from the summer 2017: https://blog.sketchfab.com/sketchfab-3d-scanning-challenge-interiors/

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Mike,

    thanks for brining my attention back to the image overlap! I set it to High since I though that achieves exactly what you say Low does. So what's the difference to High then in your opinion?

    0
    Comment actions Permalink
  • Avatar
    Mike Simone

    Hi Gotz,

    I was using a post that wishmaker had 2 years ago:

    He stated

    low-> it try use whole img area = when have small overlap 

    high-> it will use "central" part

    ----------------

    Wishgranter
     
    Hi Götz 

    low-> it try use whole img area = when have small overlap 
    medium-> default settings
    high-> it will use "central" part ( best optical quality) of img = high overlap 

    for the speed "degradation" is minimal ( few % ) so can use it freely 

    But if set LOW try set in alignment higher numbers for detected points and preselector as get larger area used..
     
    -------------------
     
    From my tests, setting the overlap to high makes alignment almost impossible, or you would need a lot more pictures and control points and most of the time would not be necessary. I am already going to be remodeling in a program like Maya/modo/Zbrush for the extra detail and 1000's of more pictures would really not give me what I am looking as I need a lower poly model when moving this into VR or a type of game engine. 
    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Mike,

    I was mainly interested in your experience!  :-)

    Interesting though, my last projects were all on high and I still get 99-100% of all images aligned.

    I do take many though, which could explain that.

    Also, accuracy is very important in my case so that's good for me.

    It would explain a few problems that I had with "outlying" areas - need to test that with medium setting and see if it gets better.

    Thanks again for the insight!

    0
    Comment actions Permalink
  • Avatar
    Mike Simone

    I am 3000km away from where I took my photos :) which made me improvise!. There was lots of rooms which matched up but the link between them was very difficult. Normally 1 room I could align pretty easy on any setting but it was the combination of multiple rooms which were lacking and having RC look at the whole image while increasing max features provided the best results.

    0
    Comment actions Permalink
  • Avatar
    Mike Simone

     Thanks ValerioPaolux!

    Honestly I am creating these environments on my own for just personal projects so its a bit difficult to justify the high prices of some of those software. Especially multiple types! 

    What would be your recommendation for just one that you could clean up your mesh in for a game asset and re-texture?

    I am Trialling Modo right now and am a bit stuck on how re-texturing works so im not getting optimal results. (Here is my post on trying to nail down that workflow.. https://support.capturingreality.com/hc/en-us/community/posts/360000190092 )

    Thanks!

     

    0
    Comment actions Permalink
  • Avatar
    ValerioPaolux

    Hi Mike,

    I believe all the software you need to reach your goal is Reality Capture. Any additional tool can add flexibility but is not required. If I were to choose a single additional tool it would probably be Recap photo, and I would only use it to fill holes.

    I noticed from your screenshot that you have 84 control points in your scene. That is a LOT. Interiors are generally much easier to align than exteriors and generally don't require as many control points. It all comes down to how you move through the environment when shooting. Make sure to choose a path that is as continuous as possible when moving from one room to the next. Avoid jumping from place to place. Also pre-processing your images always helps. 

    Here is my basic workflow to generate low-poly assets for Unity: 

    • Align images into single component 
    • Test reconstruction on critical areas. Reconstruct and Texture small areas to verify that the alignment is sound. Do this before reconstructing the entire environment to save time down the line.
    • Reconstruct in Normal Detail 
    • Filter the multi-million poly mesh. Remove extra geometry like the water tight box generated by default 
    • Simplify to 40k. This is the render upper limit cutoff in RC. This will allow you to see your mesh in Solid or Sweet mode to ensure everything looks good.
    • If your mesh if very noisy you can run a Smooth 
    • Simplify to desired polycount (depending on specs, generally few thousands to few million polys) 
    • Unwrap: i find that "fixed texel size" mode yields the best texture utilization results. I use this mode to unwrap my models. Tweak the texel size value to try and meet your ideal texture count (5-50 8k). This takes a bit of practice to get right. If you use the "Optimal Texel size" value as your "texel size" you will technically use 100% of your photos. This will likely result in more texture maps than you want. Work your way down (bigger texel size) to meet your desired texture count. (I use 8k)
    • Texture your model. I find photo-consistency mode to work best.
    • At this point you have a Unity-ready asset with correct poly and texture count. You can use Modo or just about any 3D software before texturing to improve different areas of your models such as quality of the geometry, UV utilization, textures, and such, but doing so requires a good understanding of traditional 3D. 
    • IF you do clean up your mesh outside of RC, I still recommend creating UVs and re-texturing your assets inside RC. If you want your models to look good they are going to need multiple 8k texture maps. RC does a great job at unwrapping & texturing. Dealing with UVs in a traditional way (Modo, Maya)can be extremely time consuming/challenging. Let RC do the heavy lifting for you. 

      Happy to offer more workflow-related clarifications.
      Cheers,

      Valerio


      Here is my latest file. 2655 images. 12 control points. 

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Mike,

    I think MeshLab is probably the best suited for you then. It's price to gain ratio is phenomenal!  ;-) I think there is a decimation brush included now, meaning that you can customize the areas. It's a bit awquard to use though because it still doesn't recognize edges. My only problem with it is that it has performance limits that are easily overwhelmed by RCs High Reconstructions.

    Other than that, from my trials a while ago Last-letter-of-the-alphabet-Whisk (synonym) is the best you can get for it's intelligent decemation master. It's price is not nothing, but still within affordable range for a hobbyist I would think, Saying that, I haven't afforded it myself yet and I am a professional...  ;-)

    I thought that the program mentioned by Valerio from a well known CAD developer also has some kind of limited free version? Because a good hole-filler is essential due to the holes that Valerio mentioned.

     

    Hi Valerio,

    just out of curiosity, why do you use decimation 0.5 in normal reconstruction?? Isn't that just the same as high detail reconstruction? Does it really change anything? Because I can't see how there could be more vertices than the depth maps provide, which would be implied by a factor lower than 1.

    Another way of getting better-to-handle meshes would be to set the minimal distance of vertices - I haven't tried it myself yet though. Does anyone know wheter this is something that interferes with the calculation before the final mesh or is it just a more elaborate simplification which would then again create holes?

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hey Valerio,

    this is weird, I can only see your last post now, 3 hours after my last one!

    Anyway, I think this is a very good summary! I agree with you entirely (apart from the statement that interiors are easier than exteriors, but that is very much a personal matter and also one of experience I think). Just a shame one can't fix the holes inside RC, but that might change in the future. The only reason for an external unwrap is if people need to work with the texture afterwards, as in changing it, enhancing it artistically etc. Because for that one needs an unwrap that has as few parts as possible where most of the triangles are attached as they are on the mesh. This is the only thing that RC cannot provide at this point because RCs unwrap looks like a section of a sanbox with artefacts in it... :-) Of course that also depends on the object - the more and larger plain faces it has, the less of an issue is that...

    When you say you simplify to 40k Tris, do you really mean that or rather 40m? Also, the amount that RC can handle depends on the memory available, so there can be significant differences...

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    Fabulous wisdom - thanks Mike.

    You say

    "Max Features per image = 80000 - 120000 (This is important if you are using low in image overlap but this has a big impact on RAM requirements, 32GB of RAM will be required if you are using 80k and have 2000 images"

    It may not be necessary to set Max Features per image (and Max Features perMpx) so high - or alternatively may need to be even higher. See

    https://support.capturingreality.com/hc/en-us/community/posts/115003012052-Optimising-Alignment

     

     

    0
    Comment actions Permalink
  • Avatar
    ValerioPaolux

    @Götz
    It looks like my posts are being delayed. I am just learning the ways of the forum, and how to phrase my posts wisely.
    40 millions is indeed what I meant for Solid Mesh display.

    The Texture touch-ups are a key step as no environment is ever captured in its entirety (occlusion).I often have to remove objects, fix floors, or simply fill in areas with missing textures or slight misalignment, but still stick to RC and avoid manual unwraps for 99% of the work.

    I then export models as UDIM and paint textures in 3D instead of 2D and do not  worry about seams. For any ad-hoc work I just detach small areas of the models and unwrap separately into a single additional 8k that can be easily managed in 2D (sometimes you can simply add to an existing tile)


    This of course requires a bit of planning around mesh creation and splitting environments into multiple models (eg. rooms) so as to keep a manageable per-obj texture count.

    I think RC unwraps are actually quite good, and surpassed only by...(same tool mentioned earlier) with some limitations. Texture utilization averaging around 75% is quite amazing for the time required. I need dozens of maps to meet resolution requirements, and Utilization is the most important thing in the end.

    More custom Unwrap options would be great though (eg. ability to paint/preserve seams or groups of faces without having to separate geometry. 

    What is your workflow for dealing with UVs, and touch-ups?

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Valerio,

    it's interesting that you say the unwraps are quite good because so far people were rather critical.

    But that just prooves my point (that I often make when people ask about "the best workflow") that there are so many different workflows that require so many different features that it's totally impossible to bake them all into one software. So where we have similar objects, we still have very different needs and consequently workflows.

    I personally couldn't worry less about the unwrap since I keep everything within RC. I am mostly interested in orthophotos for buildings. They need to be as accurate as possible since I use them for documentation. So I cannot just fiddle around with some parts and make them look nice - which I often wish I could!  :-)

    The only thing I do to the finished orthos is to adjust the color slightly and sometimes touch up some areas that got brighter than the rest because I had to go in closer. All in IPS and 2D, so nothing fancy there at all!

    0
    Comment actions Permalink
  • Avatar
    Bill Isenberger

    Hi Mike,

    You said; "Valve has an amazing guide here but there were a few details that still need to be addressed."

    Is this the guide you were referring to:

    https://developer.valvesoftware.com/wiki/SteamVR/Environments/Photogrammetry

    Thanks for posting!

    0
    Comment actions Permalink
  • Avatar
    Bill Isenberger

    After spending time playing around with RC Unwrap, I agree with ValerioPaolux that the RC unwraps are actually quite good. I also like that RC Unwrap gives complete control of pixel resolution in real world units.

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Tha real size pixels/texels are great indeed!

    If only there was a way to determine the actual distribution of sizes for complex models, like a histogram or something, it would be even better!

    0
    Comment actions Permalink
  • Avatar
    Mike Simone

    Well, for a massive interior, RC really can struggle with UV's and I also am trying to keep texturing efficient so I dont have 50 different textures. 

    Using both a low poly texture for walls and simple objects are the key to a nice finish for a VR product, my problem is how to handle UV's in 3rd party programs then using them back in RC.

    Also I have no idea what baking a mesh means from a high to low poly, Is that for the mesh it self or just the texture? ugh so much to learn, I just need a good course to take which will get me up to speed quickly.

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    "I just need a good course to take which will get me up to speed quickly"

    Amen to that

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hehe, the never ending story...

    I was there too - don't give up!

    Mike, I'm no expert on this but I think what baking means is to basically just transfer information from a high to a low poly model. This can be e.g. texture or also geometry as bump maps (which is depth info stored for each triangle).

    0
    Comment actions Permalink
  • Avatar
    Bill Isenberger

    This YouTube playlist has a series of short videos that explain the high poly to low poly workflow. It should give you a basic understanding of the concept in a short time. You can start with the second video, in the first video he shows how to sculpt the high poly model. RC makes the high poly mesh for you.

    https://www.youtube.com/playlist?list=PLboXykqtm8dyMYptKj-tDeJSA2ljqG4_A

    For RC you would export a mesh to be used as a reference (high poly mesh) in a modeling program. In the modeling program you would create a low poly version of the model with UV's. Then you would import the low poly mesh into RC to texture it.

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    How basic can I get, but what's UVs (like XYs (coordinates) but different?) and what would a modeling programme be - Pointfuse? MeshLab?

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Yes, exactly. UV is the XY of an unwrap.

    No idea about the software though...

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    Thanks kindly. Next silly question - what's unwrap (I have vague idea but would like to hear it)

    0
    Comment actions Permalink
  • Avatar
    ValerioPaolux

    Unwrap faces of a 3D object onto a 2D space (texture map). There is a ton of free documentation/tutorials on the web from basic to super advanced 3D for all kind of software. 

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    Ok, I'd better get looking. Obviously unwarpping millions of triangles of a mesh is a bigger challenge than a cube (not sure why you'd want to).

    Aha - perhaps UV means the coordinates (before unwrapping) of ea vertex, which remain attached to ea vertex after unwrap. Or the opposite.

    0
    Comment actions Permalink
  • Avatar
    Bill Isenberger

    This image shows an RC mesh.

    This image shows a low poly mesh (with UV's) I made in Blender using the RC mesh as a reference.

    This image shows the low poly mesh textured in RC.

    This is an image of the texture showing the Blender UV layout.

     

    Here's a link to the low poly .obj file:

    http://www.digital-mapping.net/forums/CapturingReality/2018/low-poly.zip

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Nice work Bill!

    Could you give us a rough idea of how much additional work it is from the RC model to your nice low-poly?

    Do you really texture it in RC? How do you avoid the ivy ending up all over the place? Use only one image?

    0
    Comment actions Permalink
  • Avatar
    Tom Foster

    Glad you're watching this Gotz, as someone whose output is technically accurate, reliably measurable model and/or orthophoto (am I right?) rather than beautiful seamless visual work-of-art.

    Not having yet gone into 'ton of free documentation/tutorials on the web' as Valerio recommended, I don't yet understand the workflow/images Bill helpfully offered. But I'd wonder whether such super-simplification (single flat image on single flat plane, no ins and outs) would be relevant for technically accurate output?

     

    0
    Comment actions Permalink
  • Avatar
    Götz Echtenacher

    Hi Tom,

    you're right, that's my main objective.

    But I also see the advantages of having simplified models. Bills certainly is an extreme example.

    It always depends on what kind of accuracy you are after - this is always an important question. I am sure Bills model is not far off, otherwise the texturing would not have worked well. It certainly helped that the structure is rather new (thereby straight) and the geometry is generally rather simple. It also depends on how you do it. If Bill really used only one image for texturing and took that from far away and dead centered, then there isn't too much that can go wrong since there is not too much depth in the image.

    Since Bill also deals with archaeological stuff, I am sure he knows very well what he's doing. I think I remember somer really good stuff on his sketchfab protfolio. 3D for research, not just fancy looking playthings.

    0
    Comment actions Permalink

Please sign in to leave a comment.