Terrain Texture Compositing by Blending in the Frame-Buffer (aka "Splatting" Textures)

Nov 2, 2000

by Charles Bloom (cbloom@cbloom.com)

Introduction

Splatting is a technique for texturing a terrain using high resolution localized tiling textures which transition nonlinearly. There are a lot of people who are doing and have done similar things, there is a tradition of "texture layers" which are alpha blended together. Some people understand this basic texturing by saying it's "like photoshop layers". I haven't really invented anything new here, I'm just presenting what I think is a good way to do it, with a novel technique for making the blend non-linear. Splatting is based on original texture synthesis work by Stuart Denman.

Splatting requires (almost) zero CPU work. It heavily uses the graphics card, which is asked to render each triangle many times to combine textures in the frame buffer. This makes such heavy use of fill-rate and triangle-rate that the only existing hardware which can handle it are GeForce2 and the PS2. Splatting is really intended for X-Box, PS2, and future PC titles. In the future (year 2001+) the massive fill rate of graphics hardware will be sitting idle unless you off-load the CPU to the GPU using techniques like splatting.

You can see screenshots of my splatting technical test at :

http://www.cbloom.com/3d/splatting/index.html

Basic Splatting

This produces unique textures tiled over the landscape, with linear alpha fades between them.

You take a terrain, which I'll treat as a heightmap for simplicity, and assign a texture to each element with your level editor or paint program. An "element" of terrain is either one terrain tile (a tile is a quad formed by four vertices) or several tiles, depending on the level of granularity you desire. This is really texture painting just like Ultima or any of the many old over-head 2d map games, like Warcraft or whatever. The difference here is that you don't have to manually paint transition texture tiles like they did, the transitions will be automatically generated by the system (eg. you paint textures for grass and rock, no need to paint textures with grass and rock).

Cut the terrain into chunks. The chunks should be roughly 32x32 tiles, which is 33x33 vertices. For each chunk, find the set of textures that influence it. These are all the textures on the elements of that chunk, and the textures on the immediately neighboring elements around that chunk. You now have a "splat chunk" and the set of "splat textures" for that chunk. For each chunk, construct a vertex buffer of the 33*33 vertices, optimize it, and never touch it again.

For each splat texture (for the current chunk), gather all the elements which have that texture on them, or adjacent to them. Build a triangle list for all those elements. This list is easy to optimize or stripify if you so desire (do it!).

Now, we haven't got the cross-fading between splats working yet, but we've got everything else. You cull the splat chunks based on their bbox. For each chunk, you upload the vertex-buffer associated with it. Then, you set each splat texture into hardware, and render the triangle list for that splat. I'll refer to a "splat" as this triangle list in a chunk. Rendering the chunk consists of laying down several overlapping splats.

So, how do we do the cross-fading? We need to generate alpha values for each splat to make it fade out around the edges (remember that we generated triangles for each splat which include the elements bordering on that texture; this is the region of influence of that texture/splat). In the past, this was done with vertex alpha, but that's very bad for ?GeForce, because it would require modifying the vertex buffer. Instead, I generate a texture containing the alpha values. To do this, for each splat on a chunk, you create a texture which covers the whole chunk, at a resolution of 2x2 per element (so this is a 64x64 texture). For each pixel of the texture, you find the weight of the given splat texture. This weighting is computed by finding the influence of all the textures of the surrounding 9 elements (a 3x3 grid of elements, the center one containing the current pixel). The influence on a given texel falls off by distance from the center of the element containing a texture, starting at 1.0 and falling to 0 at a distance of 1.75 elements. This is mighty unclear, I know, so I'll try to give an example :

  +--+--+--+
  |A |A |C |
  |  |  |  |
  +--+--+--+
  |B |A |C |
  |  | *|  |
  +--+--+--+
  |B |B |B |
  |  |  |  |
  +--+--+--+

So, this is a 3x3 grid of elements with texture assignments (A,B,C). The '*' is the alpha pixel I'm trying to evaluate. If the center of the middle tile is at coordinate (0,0) in element space, then my alpha pixel is at (.25,.25) The upper left element is centered at (-1,-1), etc. The weight for two points like this is :

Weight( pt1 , pt2 ) = 1 - DistanceSquared(pt1,pt2) / (1.75)^2

(clamped at zero, so negative values are zero, eg. out of range of influence). (or any other similar weighting function, this one is just an example). So the weight from my '*' to the lower right tile would be 0.63

You add up all the weights from the A elements to get a total weight for A, B, and C. You then normalize these so that they all sum to one. You can't precompute the normalization into the weighting function, because at boundaries you may not have all your neighbors.

If you're working on the splat for texture 'A' then the weight of texture A is just what you put in the alpha map.

The result is an alpha channel which is all white around the middle of a splat, gray where it blends with another, and black where the splat has no influence (there aren't any triangle there either).

If hardware supported it, you could use an 8-bit only-alpha texture. Hardware doesn't support it, so I use a 16 bit 4444 alpha texture, so each alpha map requires 8k bytes per splat (64*64*2).

Now, to render each splat, you want to render it using the alpha value from the alpha map, and the color from the splat texture (which is tiled on a per-element, or per-two elements period). To do this on ?GeForce, you can use multi-texturing : just set the base texture in Stage 0, and the alpha map in Stage 1. Then you take the color from stage 0 and the alpha from stage 1. The D3D render state for this is [fixed 3-11-02] :

	// stage 0 coloring : get color from texture0*diffuse
	lpDev->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
	lpDev->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
	lpDev->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);

	// stage 0 alpha : nada
	lpDev->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_SELECTARG1);
	lpDev->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_CURRENT);

	// stage 1 coloring : nada
	lpDev->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
	lpDev->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_DIFFUSE);

	// stage 1 alpha : get alpha from texture1
	lpDev->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_SELECTARG1);
	lpDev->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);

Note that the uv's for each splat are different, but I don't want to ever touch the VB again. To fix this, I just load in some regular gridded uv's when I build the VB. Then, for each splat I have a texture matrix which maps that grid into the proper tiling for the texture that I need. In particular the stage 0 (splat texture) uv's should tile per few elements, and the stage 1 (alpha map) uv's should stretch over the entire chunk.

This is all very neat, but how do you handle hardware that can't do nice multi-texturing like this? (such as the PS2). Well, you can use destination alpha. To do that, for each splat, you first render with the alpha map as your texture, and write only the alpha value to the frame buffer (you use a 32-bit frame buffer). Then, you render the splat again with the splat texture and take the alpha value from the frame buffer to use for your blend. This double rendering is very fast since it's just the same chunk of triangles again.

There's one final detail to the basic splatting. If you did it just as I have described here, you'd get little transparencies between your splats, since the first splat would drop down with a 50% blend onto the frame-buffer, and the next would drop down another 50% blend, so that 25% of the result would come from whatever was in the frame buffer to begin with! The easiest way to fix this is to find the most common texture on the terrain and render the whole terrain with that texture before you start any splatting. You then don't build splats for that texture (it's already taken care of by this "base" pass).

So, you're done with basic splatting. The result is very easy for artists to work with, and the blends between textures are totally linear (actually, they're quadratic, but the point is that they're very regular). This implementation that I've described is somewhat new (apparently Jan Svarovsky of Muckyfoot independently came up with something similar for Startopia); it runs very fast on modern hardware, even though individual terrain tiles may be rendered many times, because VB's are never touched and the same VB is rendered with repeatedly. In fact, this is kind of the ideal "unrealistic" scenario in which ?GeForces benchmark so well : rendering the same model over and over. It's quite easy to approach theoretical triangle rates on GeForce2 with splatting. In practice, I found my renderer to be fill-rate limited on GeForce2 GTS.

Note that the texturing you get from this is *very* high detail. Since you only have a few splat source textures for your entire terrain (eg. 16 to 32) and you can use DXTC/S3TC on them, you can let them be quite large (eg. 512x512) which gives you lots of texels per land element (btw you should certainly use trilinear mipping for your splat textures).

Another note : you must find a way to absolutely order your splats, so that they are layed down in a predictable way. I do this by giving my artists a "priority" parameter on each texture. Splats with higher priority get drawn later, making them more dominant. If you don't do this, splats on neighboring chunks may go down in different orders, which will cause visual anomalies.

Enhanced Splatting : 1. The Base Pass and Fade-Out

The first thing that makes us uneasy is the way we fixed the "holes" between splats with the most common texture. There's a much better way to do that, by making a "base texture". As a precomputation pass, I generate this base texture, which has 4x4 texels per land element (four times the texels of the alpha map). The base texture is filled out by taking the splats, and making the weighted sum of the pixels from the splat textures that influence that pixel. That is, combining all the texture around using the weights computed just like we did for the alpha maps. Since the base texture is so low res, I just use the 4x4 mip from the splat textures. Thus, I have a 128x128 base texture for each splat chunk which has a very low resolution version of my terrain texturing in it. Now, we can stop doing the most-common thing. Where splats transition, some of the base texture will show through; that's not so bad, since the base texture has the right color to match what's over it.

Now that we have this nice base texture, another cool possibility opens up. In the far distance, the base texture is good enough, and we don't need the splats at all. Thus, we can make the splats fade out (just like detail textures do), and turn them off beyond a given distance. I use a distance of 100-200 meters, which seems to be sufficient for games where the view is scaled for a humanoid. In order to do the fade out, you do it just like you would for detail textures : use vertex alpha, or put the alpha fade-out in the mips of the alpha map. That is, use trilinear filtering on the splat alpha map, and generate mips which are more and more transparent, so that the 4x4 mip (or whatever mip you find suits the fade out distance you choose) is totally transparent.

This fade-out is only really an advantage when you're drawing out very very far. Note that on the very far splats, you have only to render the base texture, so you have a very efficient renderer indeed. Near the camera, you can think of the splats as a way of doing detail textures which depend on where you're standing and transition nicely.

I should note that the base texture and the alpha maps both have the "usual problem" with chunked textures, that is, there will be seams at the chunk boundaries due to bilinear filtering problems. This can be fixed the usual way, by duplicating the pixels along the boundary and making the uv's start half a pixel into the texture (instead of at 0,0).

The "Linear Fade" screen shots at http://www.cbloom.com/3d/splatting/index.html were made using a splat renderer just like what's described up to here.

Enhanced Splatting : 2. Improving the Alpha Values

There are a couple little things you can do to improve the alpha values that you put in the alpha maps. The first is very simple : normalize the sum of weights to something larger than 1.0 (for example, 1.5 or 2.0 are both reasonable). You must then clamp each alpha at 1.0. The result of this is that where two splats overlap, each one is layed down with a 75% blend (if you normalize to 1.5). The result is mainly that the base texture is hidden a bit more, and higher priority textures dominate a bit more.

The next enhancement is to bias the alphas based on where they are in the sequence of splats for a given element. That is, earlier splats in the sequence should have higher alphas, and later ones should have lower alphas to let the onces under them show through. To do this, for each element, when I'm making the alpha map I look at all my neighbors (which will make up the splats which influence this element) and I see where my texture lies in the sequence. I count the number of splats which will be below me, and the number which will be above, and then I multiply by (number_above + 1) / (number_below + number_above + 1)

Finally, you can improve the look of basic splatting by putting a little random variation in your alphas. This is really easy, I just take the alpha and multiply by a random number between 0.75 and 1.25

Adding Noise

With "noise" splatting texture transitions can be made irregular, and the tiled texture look is almost totally eliminated.

The noise is just an additional alpha channel in each splat source texture. That is, your "grass" texture (or whatever) would be painted with an alpha channel in it for use in the noise pass. During the base splat pass, this alpha channel is ignored.

The noise pass simply renders the splats all over again, but with these alpha channels enabled to take part in the blend. In D3D this is just:

	// stage 0 coloring : get color from texture0*diffuse
	lpDev->SetTextureStageState(0, D3DTSS_COLOROP, D3DTOP_MODULATE);
	lpDev->SetTextureStageState(0, D3DTSS_COLORARG1, D3DTA_TEXTURE);
	lpDev->SetTextureStageState(0, D3DTSS_COLORARG2, D3DTA_DIFFUSE);

	// stage 0 alpha : get alpha from texture0
	lpDev->SetTextureStageState(0, D3DTSS_ALPHAOP, D3DTOP_SELECTARG1);
	lpDev->SetTextureStageState(0, D3DTSS_ALPHAARG1, D3DTA_TEXTURE);

	// stage 1 coloring : pass through current
	lpDev->SetTextureStageState(1, D3DTSS_COLOROP, D3DTOP_SELECTARG1);
	lpDev->SetTextureStageState(1, D3DTSS_COLORARG1, D3DTA_CURRENT);

	// stage 1 alpha : blend in the alpha from texture1
	lpDev->SetTextureStageState(1, D3DTSS_ALPHAOP, D3DTOP_MODULATE);
	lpDev->SetTextureStageState(1, D3DTSS_ALPHAARG1, D3DTA_CURRENT);
	lpDev->SetTextureStageState(1, D3DTSS_ALPHAARG2, D3DTA_TEXTURE);

The noise alpha channel should generally have a maximum around 200 so that it's never totally opaque. Of course, near the center of influence of a splat, the noise pass does nothing at all, it just writes the same pixels again. In the transition areas, it makes the textures have bubbles of influence larger than their uniform fade would allow.

The noise pass slightly less than doubles the triangle count, because the base texture is not rendered again. The noise can fade out in the distance just like the splats. Often (depending on how your artists paint the noise) the noise pass can fade out even sooner than the splats, saving the GPU some work. Just as with the splats, the noise pass is very efficient because you're just rendering from the same VB again. All of the pixel filling and alpha blending does push the fill rate, but this isn't a problem for PS2, and won't be a problem for future NVidia hardware.

The "Non-Linear Fade" screen shots at http://www.cbloom.com/3d/splatting/index.html were made using a splat renderer just like what's described up to here.

Note that the PS2 can't do this noise pass, and you also can't do it with destination alpha. Nonetheless, I'm doing it in Drakan-PS2. I can't go into detail here because of the fascism of the evil Sony empire, but if you are under Sony NDA, email me for info.

Comparison to Quad-Tree Synthetic Texturing

Ok, let's finish up by comparing to other techniques for texturing your terrain. The most common technique currently used is to take a large texture and spread it over all or much of your terrain. This texture contains the gross coloration of various parts of the terrain (perhaps computed from the elevation and other properties of the terrain). Because this texture is so stretched, a detail texture is applied. This method can be very fast, but provides very little real local texture detail on the terrain. It's appropriate when the player can't get too close to the terrain (eg. flight sims and such) and when the terrain doesn't have much variation (eg. the eternal desert of ?TreadMarks).

The only other practical existing technique that can provide texture detail on par with splatting is "quad-tree synthetic texturing". In this method, textures are synthesized from some function. Texels are generated where the detail is needed, eg. with more detail near the viewer, and fewer in the distance. As the viewer moves, higher and lower resolutions of the textures are generated as desired. The textures are layed out in some way on the terrain; when new textures are needed, the synthesizer generates them on the CPU. Synthesizers usually make use of three things : elevation/ecosystem of the point, some source texture blending, a noise function (like Perlin's).

One way of managing this is to split the terrain into chunks (just as you did for splatting). Then each chunk is assigned a resolution of synthetic texture based on the distance to the camera (eg. a mip-level estimation is made). That is, chunks near the camera might have 256x256 textures, while more distant ones would use 16x16 textures. As the camera gets closer to a chunk, the synthesizer must generate higher resolution textures.

Another way is to lay a quad-tree of textures over the map. That is, all the synthesized textures are the same size, but closer ones are layed over a smaller area, while farther ones are layed over larger areas. This has the advantage of not creating a lot of small textures, which are inefficient to upload, but has the disadvantage of re-grouping vertices dynamically, which is bad for vertex buffering.

No matter which one you use, a cache of recently synthesized textures should be wrapped around the synthesizer, so that if a texture is requested soon after it was created, it can be returned immediately from the cache. This is important when the camera moves back and forth across a point which causes the texturing to change.

Typically, this system need about 1 million texels of synthesized texture information (you basically have to fill the screen, and generate mips, and make a bit more texels than screen size, so you need about 800*600*2 texels). The cache should contain about this many texels again, so you need about 4 megs of texture data (assuming 16-bit textures). This technique can provide entire-terrain unique texturing, which is very cool.

You can get visual artifacts at texture boundaries very easily in this system. You need to use a healthy dose of trilinear mipping (a bit difficult to manage with the texture synthesis, but possible) and the old blending pixels across texture boundaries process. This is complicated by the fact that your textures are changing all the time, but it can be handled.

Texture synthesis like this can definitely be a good solution for some target platforms and games. It doesn't push hardware optimally, but that's not always a problem. For example, if your fog is very close, so only a small amount of terrain is visible, then you don't have to worry about your triangle rate too much, and you can just make sure that the few triangles you are showing are textured as well as possible. This method is also very CPU-heavy. That may not be a problem if your game isn't doing much else on the CPU, but it is a problem if you want to run a lot of AI or gameplay logic, in which case you need to offload your rendering to the GPU so that the game can use the CPU.

This technique is definitely the best solution for older graphics cards, eg. anything beforce ?GeForce simply can't handling splatting, so this technique wins by default. Older cards paired with fast CPU's also work best with this system. For example, "business" machines are currently usually sold with a 800+ Mhz CPU and a TNT2 or Rage128, which is a very fast CPU with a slow GPU.

The nail in the coffin of this technique (for me) is the fact that the amount of time it takes is very irregular. That is, when the viewer stands still, this technique is fantastic, you don't have to generate any textures. When the viewer is moving fast, however, you will have to generate a new texture every few frames. When you hit a frame when you have to generate a new texture, the frame rate dives, which causes a hiccup or stutter. That's intolerable, so instead we must estimate the longest time it will ever take to generate a texture, and cap our frame-rate there. For example, we could lock our game at 30 fps. That means we can allocate 20 milliseconds for texture generation per frame (assuming the rest of our engine is quite efficient and never takes more than 13 millis). You might try to stop generating textures at 20 millis and finish the next frame, but that can get you stuck in a bad feedback loop (just like ROAM, see

http://www.bolt-action.com/dl_papers.html

for a discussion by Jonathan Blow on this bad feedback). Generating your textures in 20 millis can definitely be done. It means (roughly) that you can use about 40 CPU clocks per pixel (you need to be able to generate about 512*512 texels on a 500 Mhz machine). That definitely is do-able, especially if you have a nice MMX loop for doing the texture synthesis, but it means you can't get too fancy with your texture synthesis. Of course, these numbers assume you're aiming at 30 fps. If you are more ambitious and aim at 60 fps, then you only have 16 millis for an entire frame, so your texture generation must be roughly twice as fast, and anything but the simplest synthesis is right out.

credits

The original idea for using "noise" to make the blend non-linear is due to Stuart Denman. Most of the hard work of implementing splatting on the PS2 was done by Shaun Leach.