What is HTTP Chunked Encoding? How and When Is It Used?

What is "Chunked Encoding"?

Introduction

Chunked encoding, which is provided under the HTTP 1.1 specification, involves dividing (cutting) data into smaller "blocks." Crucially, chunks are sent independently of one another, usually through a single persistent connection. The type is specified in the Transfer-Encoding header (in the first block).

In other words, the receiver never sees the entire file (as it might not have been completely available in the first place — some examples being a server reading and sending a large file to a client, or generating a table of results from a database).

One use case for HTTP chunked encoding is in video streaming applications — breaking up video into smaller pieces also allows for near-instant load times for video, compared to a browser/client waiting for (a significant amount of) data before being able to render a single frame. Video streaming also takes advantage of this encoding format to deliver chunks (or “video blocks”) as they arrive to the server itself.

Pausing and Resuming

Chunked encoding is not exclusive to video streaming or file downloads; it is a concept used to transmit any large amount of data to a client. There is one caveat to chunked encoding for such transfers, though: pausing and resuming downloads. Due to the use of persistent connections and lack of known total size (until the end of a download), the previously mentioned features are unavailable.

How does it work?

In a general sense, the client sees the first chunk almost immediately, resulting in a significantly improved user experience. While only video streaming and large file transfers have been used as examples thus far, HTTP Chunked Encoding works with any type of data, such as text:

6\r\n
Hello,\r\n
6\r\n
world!\r\n
\r\n

The 6\r\n lines indicate the length of the content that is to follow, while the \r and \n elements are control characters, signalling the end of a line. Data can also be compressed and chunked at the same time, though the compression algorithm used varies (gzip and deflate are commonly used compression formats).

(NOTE: Servers will NEVER compress blocks individually -- data is first compressed then split up).

Finally, the server (or sender) trasmits a final "empty" block:

The final block consists of two control characters (\r\n), marking the end of a chunked stream.