blog

For a while now, the art-direction use-case have been treated by browser vendors as resolution-switching's imaginary friend.

When talking to people who work for browser vendors about that use-case, I've heard phrases like "that's a really obscure use-case" and "No one is really doing art-direction".

This got me wondering — how big is that use-case? How many Web developers & designers are willing to go the extra mile, optimize their images (from a UI perspective), and adapt them so that they'd be a perfect fit to the layout they're in?

Methodology

With the lack of solid data on the subject, I had to go get some :)

Arguably, one of the easiest ways for Web developers to implement art-direction today is to use picturefill — the framework that polyfills the picture element's syntax. So all I had to do is find sites using picturefill and see which ones use the framework for art-direction rather than simple resolution-switching.

I've used the WebDevData scripts to get a hold of Alexa's top 50K websites' HTML. Then I grepped through those HTML files to find pages that contain "data-picture" (the data attribute used by picturefill), downloaded the images and (manually) went through the results to find which sites art-direct their images. Not very scalable, but it works for a small amount of sites.

Results

The results showed that 24% (7 out of 29) of the sites that use picturefill, use it to implement art-direction. While a larger sample would be better, this is a strong indication that the art-direction use-case is an important use-case for responsive images.

Update

Embedding the Gist with the results:

I just read Jason Grigsby's post, and tried to answer it in the comments, but saw that my response passed the limits of a reasonable comment. So here I am.

This post is a proposal for a file structure that will enable browsers to fetch images encoded using a responsive image format.

But which format?

Regardless of the image format that will eventually be used, a large part of the problem is coming up with a way to download only the required parts of the responsive image, without downloading unneeded image data and without reseting the TCP connection.

In any case, the format itself should be constructed in layers, where the first layer contains the image's lowest resolution, and each further layer adds more detail. An example of such layers are JPEG's progressive mode scans.

Earlier proposals

In a recent discussion, Jason linked me to a proposal for a responsive image format. While I didn't find the proposal practical because of its use of JPEG-XR, I did like the way it suggested to handle fetching of the different layers (for different resolutions). Actually, I liked it more than I liked my own proposal to use ranges.

The main disadvantage of this method is that it may cost up to a full round-trip time (RTT) per layer to fetch an image. If you have more then simple low/high resolution layer, the delay might quickly add up.

Responsive image file structure

• The image will be split into two or more files
• Each one of these files will have its own URL
• The image's header and the first (lowest resolution) layer will be in a single file. This file's URL will be part of the HTML and will trigger fetching of the image.
• Other files may contains one or more layers
• If a file contains more than a single layer, the layers must be in ascending order, from lower resolution to higher one.
• The first layer should contain meta data that includes the number of files, which layers each file contains and the byte offset of each layer inside each file.
• The HTTP response headers of the first layer should contain a list of files to the followup layers.

Image loading process

The browser will fetch the image's first layer file, as part of the page's loading process, using the lookahead pre-parser. That first layer will provide the browser with all the information it needs to further download more layers (which might be in one or more further files) as it sees fit. Fetching more layers will be based on the file structure. Files that only contain needed layers will be fetched in their entirety. For files that also contain unneeded layers, "Range" requests will be used.

Advantages

That file structure will give the author enough flexibility to arrange the image's layers in an optimal way. In case the author knows that its server and front-end cache support the HTTP "Range" header, he can use a single file to serve all the layers beyond the first layer. If this is not the case, the author can serve each layer in a file of its own.

From the browser's perspective, this structure enables it to fetch additional layers as soon as it knows the dimensions of the image to be displayed. Additional layers can be fetched using "Range" (where supported) or using separate HTTP requests. In case that separate HTTP requests are used, the browser can also parallel them, since it has all the URLs for the layers it needs once it got the first layer. The requests for the different layers can also be pipelined in this case.

By definition, the browser needs to wait for the layout phase in order to be absolutely sure it needs to download followup layers. If that would prove to be a performance bottleneck, the browser can heuristically download followup layers before it is certain they are needed (based on viewport size, image dimensions, etc).

Another advantage is that for "non-responsive" images, the browser simply downloads the image itself. There's no need to declare in the markup if an image is responsive or not.

Disadvantages

When compared to simple image fetching, image fetching with the technique described above may suffer up to a single RTT delay, when "Range" is supported. If "Range" is not supported, the delay per image may go up, even though it is not likely that it will reach the maximal "RTT per layer" performance cost. This disadvantage is probably negligable compared to the time savings that will result from fewer bytes passing over the wire.

On the other hand, for retina display devices that download all the image's layers, this delay may be noticeable.

Thoughts?

Summary for the impatient

Lossless compression with current formats can reduce image size on the web by 12.5%.

PNG24 with an Alpha channel comprise 14% of images on the web. We can cut their size by 80% using WebP.

Savings from lossless optimization can save 1.5% of overall Internet traffic!*

Savings from conversion of PNG24 with an alpha channel to WebP can save 2.1% of overall Internet traffic!!!

That's 2.8 Tbps!!! That's over 10 million kitty photos per second**!!! Save bandwidth for the kittens!

How it began

A couple of months ago, I attended (the awesome) Mobilism Conference in Amsterdam, and got introduced to the legendary Steve Souders. We got talking about images and the possibilities in lossless compression. He suggested to send me over the image URLs for the top 200K Alexa sites and that I'll run some lossless compression analysis on them. How can you say no to that?

So, 5.8M image URLs later, I started downloading, analyzing and optimizing images. That took ages, mostly because ls is slow with 5.8M files in a single directory. (I found a solution to that since)

Results

Bytes distribution according to type:

Image format	% from overall images
JPG	65.7%
GIF	11.5%
PNG8	1.3%
PNG24	5.1%
PNG24α	14%
other	2.4%

Lossless optimizations:

Optimization	% savings
JPEG EXIF removal	6.6%
JPEG EXIF removal & optimized Huffman	13.3%
JPEG EXIF removal, optimized Huffman & Convert to progressive	15.1%
PNG8 pngcrush	2.6%
PNG24 pngcrush	11%
PNG24α pngcrush	14.4%

Overall it seems that with these lossless optimization techniques, about 12.5% of image data can be saved.

Notes:

• I used jpegtran for the JPEG optimization and pngcrush for the PNG optimization.
• In order to speed things up, I did the optimization experiments over a sample of 100K random images from each type.

PNG24α

PNG24 images with an alpha channel (PNG color type 6) are the only way to put high quality real life images with an alpha channel on the web today. This is the reason they comprise 14% of overall image traffic on the web. What distinguishes them from other image formats is that in most cases, they are the wrong format for the job. JPEGs represent real life images with significantly smaller byte sizes. The only reason they are used is their alpha channel. That's where WebP fits it.

WebP

WebP is a new(ish) image format from Google. It is a derivative of their VP8 video codec, and provides significant image savings. One of the killer features of their latest release is an alpha channel. It means that PNG24α images can be converted to WebP (in its lossy variant) with minimal quality losses and huge savings.

PNG24α => WebP

I ran that conversion on the set of 100K PNG24α images. What I got was 80% size reduction in average for these images. From looking at Google's latest research, even if they don't say it out loud, they get similar results in their latest study. (0.6 bits per pixel for lossy WebP vs. 3.6 bits per pixel for PNG)

What's the catch?

There are 2 problems with deploying WebP today:

• Browser support
  • WebP's previous version is currently only supported by Chrome, Android & Opera. WebP's current version will probably be supported in Chrome in 3-6 months.
  • FireFox has refused to implement the format in its previous incarnation for various reasons. Let's hope they would reconsider the format in it's current version.
  • Microsoft and Apple have not made any public comments.
• Lack of fallback mechanisms for the <img> element.
  • That means that implementing WebP requires server side logic, and caching that varies according to User-Agent.
  • The proposed <picture> element does not include such a mechanism either. It probably should.

What I did not yet test?

I did not yet test WebP's benefits for lossy images, which Google claim to be around 30%. These savings are likely to make WebP even more attractive.

Conclusions

• Better lossless image compression using current formats by web authors can provide 12.5% savings of images data. Web authors should start using the free tools that do that, and should start doing this today! No excuses!
• WebP in its latest incarnation can provide dramatically higher savings, especially in the use case of real-life alpha channel photos. It would increase potential image data savings to at least 21.7%.
• We need browsers to either support WebP or offer a better alternative. Current file formats are not good enough, especiall for the use case of real-life photo with alpha channel.
• We need a fallback mechanism in HTML that will enable browsers and authors to experiment with new file formats without cache-busting server side hacks.

* Assuming that images comprise 15% of overall Internet traffic, which is a conservative assumption

** Assuming 35KB per kitty photo, similar to this one:

Can't be done?

All along the responsive images debate, there were several people that claimed that the salvation will come in the form of a new image format that will enable images that are automagically responsive.

My response to these claims was always that it can't be done.

It can't be done since the browser needs to download the image in order for it to analyze which parts of the image it needs. Yes, the browser can start to download the image and reset the connection once it has enough data to display the image properly, but that will always download much more than actually neccessary. (not to mention, an extremely ugly solution)

Also, introducing new image formats to the web is less than trivial and extremely slow at best (If you're not convinced, see Mozilla's response to WebP a year ago.)

And don't get me started on the lack of fallback mechanisms for new image formats :)

So, in one of the latest twitter discussions, when the subject came up, I was about to make all the above claims once again. But then I realized I was wrong all along. It can be done, it can be done gracefully, and it can be done with current image formats

HOW?!?!

The web already has a "responsive" format, which is progressive JPEG. The only issue at hand is getting the browsers to download only the neccesary bytes of the progressive JPEG.

Here's how we can do this:

• The author will compress the progressive JPEG with multiple scans
• The browser would download an initial buffer of each image (10-20K), using the "Range" request header
• This initial buffer will contain the image's dimensions and (optionally) a "scan info" JPEG comment that will state the byte breakpoints of each one of the JPEG scans (slightly similar to the MP4 video format meta data)
• If the image is not a progressive JPEG, the browser will download the rest of the image's byte range
• When the scan info comment is present, the browser will download only the byte range that it actaully needs, as soon as it knows the image's presentation size.
• When the scan info comment is not present, the browser can rely on dimension based heuristics and the "Content-Length" header to try and guess how many bytes it needs to really download.

Advantages

• DRY and easy to maintain - no need to sync the URLs with the correct resolution between the image storage and the HTML/CSS. Only a single image must be stored on the server, which will significantly simplify authors' lives.
• The image optimization can be easily automated.
• Any progressive image used in a responsive design (or that its display dimensions are smaller than its real dimensions) can benefit from this, even if the author is not aware of responsive images.

Downsides

• The optimization burden with this approach will lie on the shoulders of browser vendors. Browsers will have to come up with heuristics that correlate between number of bits per scan and the "visually acceptable" output dimensions.
• Two request for every large image, might have a negative effect on the download speed & uplink bandwidth. Browser vendors will have to make sure it won't negatively effect speed. SPDY can resolve the uplink bandwidth concerns.
• It is not certain that savings using the "responsive progressive" method are identical to savings possible using resize methods. If it proves to be an issue, it can probably be optimized in the encoder.

Disclaimers

This proposal does not claim that all the current <picture> tag efforts are not neccessary. They are required to enable "art direction responsiveness" to images, and give authors that need it more control over the actual images delivered to users.

With that said, most authors might not want to be bothered with the markup changes required. A new, complementary image convention (not really a new format) that will provide most of the same benefits, and can be applied using automated tools can have a huge advantage.

It is also worth noting that I did not conduct a full byte size comparison research between the responsive progressive method and full image resizing. See the example below for an anecdotal comparison using a single large image.

Examples

All of the images in the responsive progressive example are a single progressive JPEG that was truncated after several scans.

This is an attempt to simulate what a single progressive JPEG might look like at various resolutions when only a part of its scans are used, and how much the browsers will have to download.

We can see here that the thumbnail image below is significantly larger as responsive progressive than it is as a resized thumbnail, and the largest image is about the same size.

IMO, the responsive progressive images look significantly better than their resized counterparts, so there's probably room for optimization here.

---

The original image is 1920x1280, weighs 217K and can be found here (It is part of Ubuntu's default wallpapers package)

---

240x160 - responsive progressive - 17K

---

240x160 - resize - 5.2K

---

480x320 - responsive progressive - 21K

---

480x320 - resize - 15K

---

960x640 - responsive progressive - 57K

---

960x640 - resize - 59K

---

Update: I just saw a slightly similar proposal here. My main problem with it is that a new format will take too long to implement and deploy, and will have no fallback for older browsers.

TL;DR

Spoiler: If you have inline scripts that must be inside your page, adding an empty <div> before all stylesheet links will avoid a CSS bottleneck!!!

Do we really need Async CSS?

I've stumbled upon a post by Guy Podjarny called Eliminating the CSS bottleneck which talks about using async stylesheets in order to avoid external stylesheets from blocking the download of other resources. What bothered me about the entire post is that according to BrowserScope this issue is non existent in all modern browsers. All modern browsers tested there are not blocking parallel downloads of stylesheets and other resources. Even when an inline script is present between the external stylesheet and other resources, most browsers (IE8 is the only exception) will continue to download the resources.

I've commented on that post, saying that the only point I see for async stylesheet is to avoid Front-end SPOF. Guy responded that a stylesheet followed by an inlined script will still block resources in modern browsers, and provided a sample page that proved his point. I rechecked the BrowserScope tests and was baffled. Now I have two tests that are supposed to test the same thing, but show different results...

Not really

So I did what every reasonable developer would do and started cutting off parts and changing both tests till they were almost identical. The "almost" part was a <p> tag that was written in the BrowserScope test *before* the stylesheet link tag. Once I added a similiar <p> tag to the blocking example, it stopped blocking!!! And an empty <div> had the same effect.

Conclusions

• A simple empty <div> tag before any stylesheet links can save you from performance issues if you must add scripts inside your page's end
• The BrowserScope test is not good enough and should be modified to avoid <p> tag before the stylesheet.
• Here are the blocking vs. non-blocking examples

If anyone has a reasonable explanation as to why a div before the stylesheet links releases them from blocking, please share in the comments.

Update:Guy updated his blog post with an explanation. It seems this trick causes Chrome & Firefox to start the body earlier, and they simply don't block for body stylesheets.

Update 2:Fixed broken links.

Responsive Images and cookies

Jason Grigsby wrote a great post summarizing different approaches to responsive images and what sucks about them. Among other things, he discussed the problem of getting the first page load right. After a short discussion with him on Twitter, I decided to take a deeper look into the Filament group's cookie based method.

Tests

Testing their demo site using Firefox showed that the first browse brings in the smaller "mobile" image. The same test on Chrome downloaded the larger image. I became intrigued.

From their source code, I saw that they are using an external script that modifies the page's domain cookies. From sniffing on my traffic to the demo site, I saw that on Firefox, the image's request was sent without cookies before the external script finished downloading.

Conclusions

After a short research and some asking-around I came up with:

• Chrome's preloader blocks image download until all <head> scripts and stylesheets were downloaded for performance reasons.
• Firefox's preloader does not block for external scripts by design. It does block for inlined scripts though.
• Running this test in IE9 shows that its preloader does not block for neither external nor inlined scripts.

It seems that preloader behavior of each browser is focused on performance rather than sending cookies that may have been set using scripts. Inlined scripts currently block preloading in both Firefox and Chrome, but not on IE9.

Bottom line

Different browsers act differently with regard to which resources they download before/after the <head> scripts are done loading and running. Furthermore, that behavior is not defined in any spec, and may change with every new release. We cannot and should not count on it.

TL;DR

Adding a media attribute that supports queries to the base tag is all that's required to have responsive images with no performance penalty.

The thread

After my last post Nicolas Gallagher pointed me towards a mail thread on html-public mailing list that discusses appropriate solutions to the responsive images problem.^*

There were a few suggested solutions there:

• Each image tag will have child source tags with a media attribute in each
• A new "image" format that will deliver the browser the real image URLs according to dimensions. The browser will then fetch the image it needs according to that info
• Web authors will use a progressive image format and browsers will terminate the connection once they have enough data to properly reconstruct a downsized image
• Allow media attribute on all elements
• Add HTTP headers that will allow content negotiation

In my opinion, only the two solutions that involve the media attribute can resolve the problem with a front-end only solution, where content stays in the HTML (leaving the CSS cacheable independently from content) without any performance problems. The downside of both is that they add a lot of repeating code to the HTML. Each resource will have to be defined several times while adding a media query to each resource. A lot of copy-pasting...

Eureka

That got me thinking of a "conditional comment"-like media query syntax inside the HTML that will enable to define a different base tag according to dimensions. Then I realized that we don't need the fancy new syntax I just made up. All we need is a media attribute in the base tag that supports queries.

A base tag with a media attribute will enable us to set the base for relative URLs according to dimensions, so we would be able to simply have small images in one directory and larger images in another one, without having to specify that on a per-image basis.

Also, adding media attribute only to the base tag will probably be simpler to implement than adding it to all resources.

While that solution won't provide maximal flexibility in determining the different resolution URLs, I believe it is good enough to resolve the responsive images problem in a clean, pure front-end manner.

Thoughts?

^{* I actually saw the initial suggestions there a couple of months ago, but missed the followup responses}

TL;DR

Responsive images are important for mobile performance. Hacks may solve the problem, but they come with their own performance penalty. Browser vendors must step up to create a standard, supported method.

What we have

There have been several techniques published lately that enable responsive images using various hacks:

• Harry Roberts suggested to use background images & media queries to deliver larger images to desktop browsers
• Keith Clark suggested to use JS at the document head to plant cookies that will then send the device dimensions to the server with every image request. The server can then serve different image dimensions on the same image URL
• Yet another approach is that of the filament group which is based on dynamically modifying the base tag according to device dimensions

Not good enough

The reason we need responsive images in the first place is to avoid downloading excessively large images and avoid the performance penalty that such excessive download incurs.

All current techniques avoid some of this performance penalty, but come with new performance issues. Using the same URL for images in different dimensions means that the images can't be cached by intermediary cache servers. Using only/mainly background images means that images download won't start until the CSS was downloaded and parsed. It also means that the CSS that contains content-related image URLs cannot be long term cacheable. Dynamically modifying the base tag is generally frowned upon by browser vendors since it can mess up the preload scanner, that loads external resources before all CSS and JS were downloaded and run.

All in all, since techniques that modify the URL prevent the preload scanner from working properly, and techniques that don't modify the URL prevent caching, I don't see how a responsive images hack can avoid a performance penalty of its own, which kinda misses the point.

What we need

We need browser vendors to step up and propose (i.e. implement ☺) a standard method to do this. A standard method that will be supported by the preload scanner, and therefore, won't delay image download and won't have a performance penalty.

I have made a proposal a couple of months ago for such a method, in response to Nicolas Gallagher's and Robert Nyman's proposals, but basically any method that will keep the URL maintenance burden in the HTML, keep both CSS & images cacheable, and will have no performance penalty will be welcome.

Thoughts?

Proposal flood

In the last few days there have been a lot of proposals regarding ways to load lower resolution images for lower resolution devices. There have been Nicolas Gallagher's Responsive images using CSS3, a proposal on the W3C list, and yesterday came Robert Nyman's proposal.

Obviously, I also have an opinion on the matter :)

My contribution to the flood

While the proposals from Nicolas & Robert are interesting, they throw a huge maintenance burden on the CSS, and make it practically uncacheable in many fast-updating pages. In a CNN-like scenario, where the images are changing every 1-2 hours, the CSS (or at least one CSS file) will have to change along with them, which is bad for performance, and basically means dynamically generated CSS. I think it is bad news.

I think we are much better off with something in the lines of:

img{
    src-prefix: "big_";
}
@media screen and (max-width: 600px){
    img{
        src-prefix: "small_";
    }
}

When an image defined as <img src="img.jpg"/> will be requested, it will be requested as "big_img.jpg" in devices that exceed 600px max-width, as "small_img.jpg" in devices that have lower resolution, and as "img.jpg" in browsers that do not support this new attribute.

To me this is a solution that has very small maintenance, keeps all changing image content in the HTML where it belongs, and has an inherit fallback.

Please leave a comment here or on twitter if you have some feedback on this.

Yoav

Update: As Nicolas Gallagher pointed out in the comments, his proposal does not put maintenance burden on the CSS. Extra URLs are written as data attributes in the HTML. I still think my proposal is better, though :)

I had a twitter discussion today with Robert Nyman regarding how we should treat old IEs (6,7 and 8) when we develop. The trigger was his tweet regarding his post from 2 years back that we should stop developing for IE6. Since twitter is fairly limited for this kind of things, here's my real, chatty opinion on this subject.

Stats below are taken from statcounter and regard EU and North America. Other markets' mileage may vary :)

So, what's the problem?

While we certainly can simply ignore IE6 in most of the world (around 2% market share), IE7 still has 7%-11% in the western world. But then again, this is where IE6 was 2 years ago when Robert wrote his post. In any case IE7 can be ignored or simply nudged to upgrade, since it is by all means an obsolete browser.

The real trouble start with IE8. While IE8 is much better then its older brothers, it is still a piece of crap in comparison to today's browsers and it does not support any of the new APIs we need to make the web awesome. It has a market share of 26-34% in the western world and since IE9 is not for XP, it is not going away anytime before the end of 2014 when XP is *finally* decommissioned. It will probably last a little while longer after that as well.

What can we do about it?

There are a few approaches that web developers can use in order to drive people away from the old IEs into the modern web:

• Advocate - Campaigns like HTML5forXP are trying to get the users to upgrade through awareness
• Nag - Display in-site messages that notify the user that he would be getting a better experience if he'd upgrade to a modern browser or install Chrome Frame
• Ignore - Stop testing on old IEs and trying to create a similar experience for these users using various polyfills
• Exclude - Block out old IE users from sites until they upgrade

While ignoring is tempting, you probably don't want 40% of your users to have a shitty experience on your site, so my personal favorite is "nag and ignore the none-essential parts" approach. (kinda like twitter with border radius)

On the other hand, I can't help from reflecting on the fact that Macromedia Flash (before it was bought by Adobe) gained 98% market share through Exclusion. "If you want to see this website - you MUST install Flash" was the paradigm that got it there.

The big question is "Who was the first to exclude users without Flash?" (If anyone knows, I'd love to hear about it). A bigger question is "Will one of the big guns on the web today (Google, Yahoo, Facebook, Bing) start excluding services from old IEs?". I know some of them don't support IE6, but who will be the first to not support IE8? I'm only guessing here, but it probably won't be Bing...

That's it for now.

Thoughts?




UPDATE: I found some "Way back machine" stats that indicate that Macromedia Flash made a final market share leap from 90% to 95% in the summer of 2000. Could not find stats before that though...