The Grand old Duke Of Rustica, he had 10,000 prims. He marched them up to the top of the sim and he marched them down again.
And when they were up they were up, and when they were down they were down, and when they were only half way up, they were quantised into oblivion.
So goes the tale of the Grand Old Duke of Rustica, the Honourable Maxwell Graf, Esq.
For those not interested in curious, obscure Second Life bugs, go and watch my video instead, just a few minutes of fun watching a large castle be moved (again)
Relocating Rustica - An exploration of terrain export woes in Second Life
It's rare these days for me to write about anything other than Firestorm/viewer stuff. But I thought I'd record a recent adventure in relocation, scripting ingenuity and a smidgen of mathematics (no, no, please stay, I'll keep it easy)
A few years ago, the impressive ancient castle of Rustica found a home attached to the New Babbage estate. Sheltering in our community gave its owner, Maxwell, the community support he needed at that time. The move was non-trivial. New Babbage, to some known for its Steampunk theme and resident-driven community builds, also happens to be one of the few deepwater estates in Second Life. With the sea level set at 100m, we host many a technical scuba diver practising their controlled ascents and decompression stops (yes, really, we do). So what? So.. that meant that the 14000+ meshes/sculpts and prims that combine to form the impressive Rustica Castle and its surrounding land all needed to be lifted, from the shallow grid-default water depth of just 20m, and with high accuracy, ensuring that no cracks appeared and the castle did not crumble.
I recorded that feat as a record for Max and for Liz, whose LSL-wielding powers had allowed this to happen. You can watch it over on my vimeo account.
Poor Max has had a terrible few years. A traumatic brain injury led to a stroke and a bucket list of things you don't want to happen. It has left him broken in several ways, but not in spirit. Doggedly re-learning his skills as an artist, destroyed by the cognitive-loss from the TBI, has shown amazing results, some of his recent creations are some of my favourites, but these things take time and patience, things that don;t sit well with the frenetic pace of running a store in Second Life.
Unable to create new products at the rate and quality that stores today require, but not wishing to give up on Rustica and SecondLife, a place so very dear to him. It was here where, among many other important events in his life, Max met his RL wife so losing SL and Rustica was never really a choice he'd choose to make. Instead, he made the difficult choice to find it a new home where, amongst a dedicated group of role players, his castle can live on without needing the income from his store alone to pay the fees.
So, where do Beq and Liz come into this? Well, naturally, this means...yep, reversing the process. On the one hand, this is easy enough; we have the technology, or at least Liz does, and with the experience from the Rise, we can handle the Fall. However, it has also given us the need to address one of the problems we faced the first time around. Adjusting the terrain.
When we raised the terrain, we encountered issues, but given the stress had been over the lift and shift of the stonework, it was mostly accepted as a small price to pay and reworked by hand.
Why was raising the terrain hard? There is no tool on the viewer or the server to simply add or remove an offset. Arguably, there should be, but we'll come back to that. To adjust the terrain, you need to export the terrain data as a raw file. 99% of Second Life users will never do this because it is a function only available to estate or private region owners.
Terrain files? Raw? wth?
Terrain files are ... "speshal". They are "raw" images, just bundles of bytes. Consisting of 13 channels of 256x256 8-bit values.
Regular images have three channels Red, Green and Blue (RGB). In the Terrain files, we have 13 channels, but in reality, all but three of them are ignored these days, and for practical purposes, only the first two of these are used. Each "pixel" is notionally applied to a 1x1 square metre of the region. The Red and the Green Channels encode the height, while the Blue channel encodes the water level.
The height encoding is where things get interesting. Given that an 8-bit number can only hold whole number (integer) values between 0 and 255, and we need to be able to represent terrain elevations higher than 255 metres, but we also want to be able to represent sub-metre levels (2.3m, 75.7m etc), some magic needs to happen. Two channels are combined.
The most immediately obvious way to do this would be to combine the two channels in a binary 16-bit value, allowing up to 65535 discrete elevations, where an elevation step could be an arbitrary unit (let's say a centimetre), allowing a little over 655m of elevation. Instead, though, the Terrain format is encoded at a variable precision. The value of the First Channel (Red) is treated as representing the number of height "units". Meanwhile, the value of the Second Channel (Green) is used to define the "unit" measured in 128ths of a metre.
So if R is 32 and G is 1, then the resulting height is 32 * 1/128 metres, a quarter of a metre, or better known as 25 centimetres. If we want larger jumps from a shallow shelving beach to a towering cliff, then we switch to a large unit size, 128 gives us steps of 1 metre. If we really want, we can go for 255, units of close to 2m jumps. A red value of 32 here, when coupled with a Green of 255, is not 25cm but close to 64m. So this rather obscure way of encoding is there to allow us fine-grained control over the terrain that we work with. Or it should.
And here we finally come to the problem at hand. The Second Life Server does not seem to dynamically tune the unit size. In an ideal world, you'd look to encode each height in the most accurate representation, this would scale the unit size to be smaller when the target height was lower, allowing small tweaks at lower altitudes and coarser ones higher up.
When Max reached out to Liz and me to "do the drop", he sent me the terrain file, knowing that I would be doing the adjustments to drop the terrain height back to the 20m sea level. However, when I examined the file that the server had exported for him, I immediately saw an issue. The entire region uses a single "unit size" of 140/128ths. That's about 1.1metres. The minimum "step" from one metre to the next is 1.1 metres, which is not going to be able to capture the gentle slopes of the Rustica landscape. In fact, as Max found to his cost, it "Minecraftifies" the whole place. The image below shows how the land looked in Second Life before the export.
A better export is possible
I was then able to write some code that translated that back into height data, but being careful to properly encode it to the most accurate combination of unit size and unit count (height), allowing us to extract an accurate topology.
I have placed my python code and a few of the generated RAW files on github in a dedicated repo I have called "Terrain Tools"
Since the first commit I've added, a little extra code to allow land to be raised or lowered by an arbitrary amount, to create a "zero" empty terrain (which used with the raise-by command creates flat terrain of varying heights.
OK, export "fixed" does it really work?
The same file in Second Life gave a very different result.
Oddly, baking that terrain and exporting it still gave us the smooth ramp. So, whatever it is doing to it, it is doing independently. We naturally tested the Second Life viewer as well as FS, and it was broken in the same way.
