The Minecraft Overviewer

a high-resolution Minecraft world renderer with a LeafletJS interface


On the New optimizeimg

Hi, I'm CounterPillow. If you've ever been in our IRC channel, you've probably seen me. I have been an irregular contributor to Overviewer for some time now, and recently, I've rewritten the way optimizeimg works. I thought that this would be a good opportunity to explain what it is, and how to best use it.

First, let's explain what this is all about. In essence, this increases the time it takes to render a map in favor of smaller images. But how does it do that? optimizeimg allows you to run various image optimizers against each tile image the Overviewer outputs. As of the time of writing, we support optipng, pngcrush and pngnq.

Lossless vs. Lossy Compression

One of the key things to understand is what lossless and lossy compression mean. In lossless compression, the file size is reduced, but the decompressed data is exactly the same. No data is lost in the compression step, hence the name "lossless". Lossy compression, on the other hand, does not preserve all information. An uncompressed output of a lossy compression algorithm is not the same as the input.

As you may already know, PNG uses lossless compression. However, there are a lot of different knobs to tweak when it comes to how the image is compressed, thus reducing file size without altering the image data, and we can even reduce the amount of colors in the image, essentially making the process lossy! I'm not going to go into detail as to how this all works in practice, but I will showcase the tools we currently support to achieve this.

Lossless Image Optimization, and "crushing"

First off, I'll have to admit that I've lied in the previous section. Some of the lossless image recompression methods are not lossless in the sense of "data in is the same as data out decompressed". Some tools, like optipng, discard unused alpha channels (where images store their transparency), to reduce file size. I said unused, which means that the visible output is still pixel-per-pixel the same, thus lossless, but the image data is obviously different. However, for all intents and purposes, we can call this process "lossless", because no visible information has been changed.

Discarding unused alpha channels is one way of making the image smaller, but there are other ways which are truly lossless. Both pngcrush and optipng tweak the previously mentioned knobs of the compression, trying to find the smallest possible output. The two tools do this a bit differently, but the principle is the same: Tweak the encoding parameters. This is what the Overviewer code refers to as a "crusher". If you wish to learn more on how this is done internally, you can read about it on the optipng site, but beware: It's quite technical.

Lossy Image Optimization

With pngnq, we also offer a way of lossy image optimization. You may be wondering how a PNG can be lossily optimized, as the compression method is lossless. The answer is simple: The amount of colors.

PNG supports a ton of different color modes, amongst them is an indexed color mode, or also referred to as "Palette PNG" or "PNG8". In images with indexed color modes, the colors of each pixel aren't determined individually using red, green and blue channels. Instead, it has a palette of colors, and then specifies for each pixel the color for it by assigning it the a color of the palette. This means less colors, in our case 256 at most, and because we can represent those with just one byte, it means less data that needs to be compressed. If we find the 256 colors that can best represent the image, we get pretty good results that are just a fraction of the original size; and since Minecraft doesn't use a lot of colors in the default texture pack, it's barely noticeable. agrif briefly mentioned this in the "Introducing OIL" blog post. In fact, if our input image has less than or exactly 256 colors anyway, the process is completely lossless!

Instead of waiting for OIL, you can use palette PNGs right now, with the pngnq tool. However, please be aware that due to multiple bugs in the imaging library we currently use, PIL (or Pillow, both are buggy), this will break transparency. As stated in the docs, this is not pngnq's fault.

Using Image Optimization In Practice

But how is this used in practice, you ask? It's not that hard. The first step is to install the tools you wish to use and make sure they can be called by Overviewer (i.e., are in your PATH environment variable). Then, add them to your config.

Please be aware that this will increase render times.

An example for everyone

The following example should be satisfactory for most people who wish to use image optimization. Please also be aware that when I say "most people", I always actually mean "most people". If you think you are not "most people", then you're doing so carrying the risk that you may be wasting your time.

from optimizeimages import optipng

renders["foo"] = {
    # ... other stuff here
    "optimizeimg":[optipng()],
}

This will run optipng with the default number of trials (which is the way to go for most people).

Playing around with optimization levels

We can, of course, also specify some parameters for optipng. Please note that you're dangerously venturing out of most people-territory here.

from optimizeimages import optipng

renders["foo"] = {
    # ... other stuff here
    "optimizeimg":[optipng(olevel=3)],
}

This will essentially execute optipng -o3 (instead of -o2), you can see what the exact meaning of that is in the optipng manual, or by typing optipng -h. Please note that anything above, or most likely even including the fifth optimization level, is pointless. The OptiPNG developers said so themselves:

It is important to mention that the achieved compression ratio is less and less likely to improve when higher-level presets (trigerring more trials) are being used. Even if the program is capable of searching automatically over more than 200 configurations (and the advanced users have access to more than 1000 configurations!), a preset that selects around 10 trials should be satisfactory for most users. Furthermore, a preset that selects between 30-40 trials should be satisfactory for all users, for it is very, very unlikely to be beaten significantly by any wider search. The rest of the trial configurations are offered rather as a curiosity (but they were used in the experimentation from which we concluded they are indeed useless!)

Getting crazy

Let's move on to something more hardcore, then. Let's say you don't care if your map looks a bit weird thanks to PIL. Let's say you want to use pngnq.

from optimizeimages import optipng, pngnq

renders["foo"] = {
    # ... other stuff here
    "optimizeimg":[pngnq(), optipng()],
}

Note the order in which the two optimizers are specified. Here, pngnq is run first, then optipng is run on it. This is important: If you feed crushed output into something that is not a crusher (i.e. does not do lossless compression tweaking), all benefits of the crusher will be lost; essentially, you'd be wasting your time. If you choose to use more than one optimization tool, make sure the crusher is at the end. Overviewer will warn you if it isn't.

Don't be silly

Speaking of more than one optimization tool, let's answer a very important question: Will the image be smaller by using multiple crushers chained together? (i.e. optipng after pngcrush)

The short answer: No.

The long answer: Most likely not.

Crushers, by default, exit if they fail to make the image smaller, and just admit that they have not been able to crush it. Crushers may perform differently in different situations, so using two crushers chained after each other gives you essentially the best output of either; however, for most images (and I say "most images" like I say "most people"), this will be optipng in our example. Every optimizeimg configuration that uses both pngcrush and optipng can most likely be reduced to just optipng, with exactly the same results but much faster renders.

Conclusion

As you may have noticed, I often used "most likely" and "most people" in this post. This is because we can't make certain statements without large-scale tests, which I have not done, but other people did. If you're interested in utilizing image optimization, what you should take away from this blog post is that [optipng()] is the way to go. If you're not sure, do tests yourself. We may add support for better image optimizers in the future, including some for JPEG, so frequently checking the documentation to see which are available is a good idea.

(permalink)

Overviewer 0.11 released, supporting Minecraft 1.7

Earlier this week we released an update to Overviewer to support Minecraft 1.7. This includes support for all of the new blocks and biomes!

Head over to github or our downloads page to get a copy! As usual, stop by IRC, or open a github issue if you run into any problems.

We have also recently completed some long-needed work on our build infrastructure, so our Debian and RPM repositories should be updated much more often.  

(permalink)

Minecraft 1.6 Support

This blog post is a bit late, but yes, the latest Overviewer does support the latest Minecraft version 1.6.

As some may know, Minecraft 1.6 adds Resource Packs, and thus the format for texture packs and has changed. Since Overviewer pulls textures straight from the Minecraft client assets, this necessitated a change in Overviewer. And as we're not fans of keeping lots of code around for backwards comparability, this means the latest versions of Overviewer will not support reading textures from Minecraft clients or texture packs from 1.5 and before.

This typically isn't a big deal—as people update their clients and update Overviewer, things will just work. If you do not want to upgrade your Minecraft client or texture packs, then you should not upgrade to the latest Overviewer.

As always, the best place to get help, ask questions, or get involved is on IRC!

(For posterity, the most recent build that supports 1.5 is a147ca4a. Downloads: Win64. Also, to be clear, the newer Overviewer will still render older worlds, you just need the new textures.)

(permalink)

Overviewer 0.10 released, supporting Minecraft 1.5

Today we have just released Overviewer 0.10.0, with support for Minecraft 1.5 (aka "The Redstone Update")!

Head over to github or our downloads page to get a copy!  As usual, stop by IRC, or open a github issue if you run into any problems.

With Minecraft 1.5, the format of texture packs has changed significantly.  Previously, textures were all stored in the same terrain.png file.  Now, each item and block has its own texture file.  You will run into problems if you use Overviewer 0.10.0, but haven't updated Minecraft (or your custom texturepack, if you are using one), so if you don't intend on upgrading Minecraft, you should not upgrade Overviewer.  

(permalink)

Introducing OIL

For the last few months, I've been working on a replacement for PIL to use with Overviewer. PIL is nice for simple imaging tasks, but as Overviewer grew we were constantly hitting annoying limitations or performance issues. Over time we've accumulated a bunch of ad-hoc image manipulation code, and we decided to consolidate it all and finally replace the parts of PIL we still used. The result is OIL (Overviewer Imaging Library), which is now very close to being finished.

Along the way, I decided to also rewrite Overviewer's renderer. Currently, Overviewer renders a world by pre-rendering a bunch of block sprites, then pasting those sprites onto the final images. This has worked well, but adding new blocks is an extremely tedious process of manipulating pixels directly to produce something that looks 3D. It also has a major drawback: the view angle and block size is fixed. We've been asked many times to allow configurable block sizes or map rotations, and we haven't been able to do these because our current renderer design is not flexible enough.

The new renderer is based entirely around 3D block meshes. It draws these directly to the final images, meaning it's trivially easy to rotate and scale them exactly how you want. Overall this means more flexibility for you in how you render your maps, and easier to maintain code for us: adding new blocks is as simple as creating a new block mesh.

Now, there are two main concerns I want to address as soon as possible:

  • Won't this new renderer be slower? It sounds much more complicated.

    Wonderfully, no! The current experimental renderer currently runs 5% to 15% faster than our old sprite-based renderer. There are still blocks and features to add, but I don't expect it to get much slower from here. At the very least, the new renderer will run in about the same time as the old renderer.

  • 3D? I don't like needing a video card just to render maps.

    The new experimental renderer is entirely CPU-based. You don't need a video card, and you don't need a graphical environment. If you were able to run Overviewer before, you'll still be able to. In particular, for Unix users, you will not need X to run Overviewer.

    For those of you who have a video card and would like to use it, OIL has an OpenGL backend that currently renders 15% to 25% faster than the old renderer; but this is an option, not a requirement!

So, that's what's been going on for the last few months. The rest of this post is dedicated to pictures, but as always, if you have any comments or questions we would love to hear from you either in IRC or on GitHub.

Cool Features and Pretty Pictures

I'm going to be putting a lot of images below. You can click on them to get the full-sized versions.

Renderer

First, two shots of the same area, one with the current renderer and one with the experimental renderer:

Please remember that the experimental renderer is still unfinished; there are a lot of missing blocks, and biome coloring hasn't been implemented yet. These are just a preview of what's coming.

The first picture (with the old renderer) took 1.37 seconds, while the second (new renderer) took only 1.14.

Just for fun, here's a picture of the same area, rendered on my GPU with OpenGL:

It's almost identical to the CPU-rendered version, and this one rendered in 1.04 seconds.

Configurable Views

The advantages of the new renderer don't really show themselves in side-by-side comparisons though. How about something the old renderer couldn't do?

What if you wanted to render your map with huge blocks, at a really weird angle? I don't know why you'd want to, but with the new renderer, you can:

Dithering

One benefit we'll be getting from OIL is much more subtle than the new renderer. Consider these two images:

They look about the same, right? Well, would you believe the one on the right is 80% smaller? That's because the one on the right is an indexed image, meaning it uses only 256 total colors instead of the 4 billion colors normal Overviewer images use. Overviewer images look really good as indexed images, because Minecraft doesn't really use a whole lot of distinct colors.

While PIL has support for indexed images, it has no support for transparent indexed images, which made implementing this impossible. OIL has no such restriction.

When will we see it?

While OIL itself is just about done, the new renderer still needs a lot of work before it'll be merged. This blog post is just to let you know that Overviewer isn't dead -- in fact, far from it. There are exciting things in the future.

(permalink)

  • Page 2 of 5
  • 1
  • 2
  • 3
  • 4
  • 5