The Minecraft Overviewer

For the last few months, I've been working on a replacement for PIL to use with Overviewer. PIL is nice for simple imaging tasks, but as Overviewer grew we were constantly hitting annoying limitations or performance issues. Over time we've accumulated a bunch of ad-hoc image manipulation code, and we decided to consolidate it all and finally replace the parts of PIL we still used. The result is OIL (Overviewer Imaging Library), which is now very close to being finished.

Along the way, I decided to also rewrite Overviewer's renderer. Currently, Overviewer renders a world by pre-rendering a bunch of block sprites, then pasting those sprites onto the final images. This has worked well, but adding new blocks is an extremely tedious process of manipulating pixels directly to produce something that looks 3D. It also has a major drawback: the view angle and block size is fixed. We've been asked many times to allow configurable block sizes or map rotations, and we haven't been able to do these because our current renderer design is not flexible enough.

The new renderer is based entirely around 3D block meshes. It draws these directly to the final images, meaning it's trivially easy to rotate and scale them exactly how you want. Overall this means more flexibility for you in how you render your maps, and easier to maintain code for us: adding new blocks is as simple as creating a new block mesh.

Now, there are two main concerns I want to address as soon as possible:

• Won't this new renderer be slower? It sounds much more complicated.

  Wonderfully, no! The current experimental renderer currently runs 5% to 15% faster than our old sprite-based renderer. There are still blocks and features to add, but I don't expect it to get much slower from here. At the very least, the new renderer will run in about the same time as the old renderer.

• 3D? I don't like needing a video card just to render maps.

  The new experimental renderer is entirely CPU-based. You don't need a video card, and you don't need a graphical environment. If you were able to run Overviewer before, you'll still be able to. In particular, for Unix users, you will not need X to run Overviewer.

  For those of you who have a video card and would like to use it, OIL has an OpenGL backend that currently renders 15% to 25% faster than the old renderer; but this is an option, not a requirement!

So, that's what's been going on for the last few months. The rest of this post is dedicated to pictures, but as always, if you have any comments or questions we would love to hear from you either in IRC or on GitHub.

Cool Features and Pretty Pictures

I'm going to be putting a lot of images below. You can click on them to get the full-sized versions.

Renderer

First, two shots of the same area, one with the current renderer and one with the experimental renderer:

Please remember that the experimental renderer is still unfinished; there are a lot of missing blocks, and biome coloring hasn't been implemented yet. These are just a preview of what's coming.

The first picture (with the old renderer) took 1.37 seconds, while the second (new renderer) took only 1.14.

Just for fun, here's a picture of the same area, rendered on my GPU with OpenGL:

It's almost identical to the CPU-rendered version, and this one rendered in 1.04 seconds.

Configurable Views

The advantages of the new renderer don't really show themselves in side-by-side comparisons though. How about something the old renderer couldn't do?

What if you wanted to render your map with huge blocks, at a really weird angle? I don't know why you'd want to, but with the new renderer, you can:

Dithering

One benefit we'll be getting from OIL is much more subtle than the new renderer. Consider these two images:

They look about the same, right? Well, would you believe the one on the right is 80% smaller? That's because the one on the right is an indexed image, meaning it uses only 256 total colors instead of the 4 billion colors normal Overviewer images use. Overviewer images look really good as indexed images, because Minecraft doesn't really use a whole lot of distinct colors.

While PIL has support for indexed images, it has no support for transparent indexed images, which made implementing this impossible. OIL has no such restriction.

When will we see it?

While OIL itself is just about done, the new renderer still needs a lot of work before it'll be merged. This blog post is just to let you know that Overviewer isn't dead -- in fact, far from it. There are exciting things in the future.

RamsesA's Crafting Azeroth project completely recreates the World of Warcraft in Minecraft using an elaborate series of custom tools and old-fashioned manual touch-ups. For the last month or so, the Overviewer team has been working with him to render high-resolution maps of these worlds.

The two largest continents, Eastern Kingdoms and Kalimdor, are complete and are ready to view. Other continents may also be made available as they are rendered.

The process of creating these worlds to begin with is difficult and extremely interesting, and you can learn about it in this Minecraft Forum thread. This blog post, though, will focus specifically on what we had to do to create this map.

A Tall Order

The first hurdle to overcome was the world height. Normal Minecraft worlds are 256 blocks high, but Kalimdor is 1408 blocks high. This means that Kalimdor is 5.5 times taller than normal Minecraft worlds, and 11 times taller than old Minecraft worlds (before the Anvil map format). Since normal Minecraft clients can't handle worlds this tall, RamsesA split the world up into 7 normal-height overlapping world layers, and used a Bukkit plugin to automatically teleport between the layers for a nearly seamless experience. This is great for exploring the map, but Overviewer expects all its map data in a single world. So, the first step was to turn these 7 layers back into a single mega-world with all 1408 blocks of height. To do this, I used my own libredstone with python bindings.

Once all the world layers were merged, it was possible to load up the world in Overviewer. However, Overviewer still expected the world to be 256 blocks high, so it only rendered a tiny part of the world. Back after the Anvil change, when worlds doubled in height, brownan had worked out the changes needed to the render code. This change was so similar it was mostly a matter of changing a few constants. Our recent rewrite ensured that very few parts of Overviewer code actually depend on world height, and by the end of this process, it had been reduced to two source lines.

Making it Pretty

So now we were rendering, and boy was it something else. These worlds were huge and beautiful. We worked with RamsesA to come up with a combination of render primitives to best show off his work. Since there were a lot of forests with very large trees, a lot of the ground was too dark with the normal Overviewer lighting, so we settled on smooth lighting at 65% strength.

There were a few details left to work out about the rendering, though. RamsesA's conversion code placed water blocks in ways that Minecraft never does, causing a few minor errors in the render. Fixing these was just a few changes to the C source, and an since Overviewer is an incremental renderer, it was easy to just re-render the parts of the map that needed it. Having a render of the entire world also helped to find minor errors in the world itself, which were quickly corrected and fixed with more incremental renders. All told, it took about 2 - 3 weeks for all these renders to finish, running on the beefy 4-core machine hosting overviewer.org (graciously hosted by nexcess.net).

Final Presentation

Now that the bulk of the rendering was done, it was time for the finishing touches. Eminence and pironic assisted RamsesA in placing markers on the map, which showed location info and photos when clicked. This was mostly a matter of using Overviewer's built in POI code, with a few minor tweaks for using it to display so much information. A lot of the Overviewer team also helped with placing the markers and taking photos in-game to go along with them. Finally, Eminence made a few changes to the map controls to simplify the map interface, and a bunch of smaller ones to make the entire experience feel better.

One of the goals of the rewrite was to make large changes like this easy, and I think the success of this project is a sign that we succeeded. Now that we're done, the improvements and changes made for this project will trickle back into the official Overviewer source as soon as we clean them up and make them configurable. As a bonus, RamsesA found a wonderful optimization in our code that should speed up render times by as much as 15%.

All together, this was a massive and fun project for us. It was great working with RamsesA to stretch the limits of what Overviewer can do, and we hope you like the result.

(Reddit thread)

For the last few months, Overviewer development has slowed. Fear not, the project is not dead! The semester is coming to a close and us core devs will hopefully re-acquire the necessary free time and motivation to handle some of the larger, outstanding, and much-requested features the community has been asking for.

Features like:

• Easy addition of custom/mod blocks such as IC2, Redpower, etc.
• As a superset of the above bullet, a plugin-type system for Overviewer to easily add functionality
• A re-design of some of the config options that I personally think are confusing (like how --check-tiles works)
• Better handling of maps edited with external tools
• More performance improvements
• A re-design of the docs to make it more clear and easier to find relevant information
• Probably more I'm forgetting

I have high hopes that development will pick back up in the next few weeks. We've been keeping up with smaller bug-fixes and features, but these larger features require larger blocks of time.

Incidentally, if anyone out there has any Python experience, or would like to help e.g. with documentation, please see us on our IRC channel (#overviewer on Freenode). The Overviewer is a community-run project and needs volunteers to help keep things moving. That means you could help drive development while us regulars are too busy with school/work/vacation/whatever!

derchris in #overviewer has made for us an updated visualization of the Overviewer commit history. Check it out!

What's interesting is you can pretty clearly see when I left around 0:30, or November 2010, and then when I return around 2:00, or October 2011.

Thanks derchris for the video! Thanks everyone for your contributions!

For an intro to Overviewer in general, see the front page of our documentation site.

Support for Minecraft 1.2 worlds is now merged into the "master" branch of our repository!

Those of you who have repositories with "anvil" or "rewrite" checked out will need to switch to the master branch (git checkout master).

Those of you who have the old "master" branch checked out will need to see This Post to upgrade to the new version. Config files are incompatible with this upgrade, so this is important!

Also to note if you're upgrading from 0.4 (the old master branch): remember that due to changes made to the format you WILL need to re-render your maps from scratch. Trying to update existing maps will NOT work. Also due to major changes to the web assets, you should not simply copy over custom web assets to the new version. In particular, custom overviewerConfig.js files can cause strange problems.

The new version has been arbitrarily named 0.9. Windows builds are now available on our downloads page.

As always, stop in on IRC with any questions or just to say hi!

This is still in beta because there are a few features yet missing, but since Minecraft 1.2 is released, we felt that it was more important to have the master branch compatible with the current Minecraft. Here are the features yet to be implemented:

• Signs, Spawn markers, and other Points of Interest
• Ordering of renders and worlds in the javascript interface Edit: Done! Mar 6 2012
• Overlays