#format rst #language de ================================================ HTTP: The Protocol Every Web Developer Must Know ================================================ by Pavan Podila, April 2013 .. contents:: .. sectnum:: Part 1 ========= by Pavan Podila, 8 Apr 2013 (`Original link for part 1 `_) HTTP stands for Hypertext Transfer Protocol. It's a stateless, application-layer protocol for communicating between distributed systems, and is the foundation of the modern web. As a web developer, we all must have a strong understanding of this protocol. Let's review this powerful protocol through the lens of a web developer. We'll tackle the topic in two parts. In this first entry, we'll cover the basics and outline the various request and response headers. In the follow-up article, we'll review specific pieces of HTTP - namely caching, connection handling and authentication. Although I'll mention some details related to headers, it's best to instead consult the RFC (`RFC 2616 `_) for in-depth coverage. I will be pointing to specific parts of the RFC throughout the article. HTTP Basics ----------- HTTP allows for communication between a variety of hosts and clients, and supports a mixture of network configurations. *To make this possible, it assumes very little about a particular system, and does not keep state between different message exchanges.* This makes HTTP a **stateless** protocol. The communication usually takes place over TCP/IP, but any reliable transport can be used. The default port for TCP/IP is **80**, but other ports can also be used. .. figure:: http1-request-response.png :align: center *Custom headers can also be created and sent by the client.* Communication between a host and a client occurs, via a **request/response** pair. The client initiates an HTTP request message, which is serviced through a HTTP response message in return. We will look at this fundamental message-pair in the next section. The current version of the protocol is **HTTP/1.1**, which adds a few extra features to the previous 1.0 version. The most important of these, in my opinion, includes *persistent connections*, *chunked transfer-coding* and *fine-grained caching headers*. We'll briefly touch upon these features in this article; in-depth coverage will be provided in part two. **URLs** At the heart of web communications is the request message, which are sent via Uniform Resource Locators (URLs). I'm sure you are already familiar with URLs, but for completeness sake, I'll include it here. URLs have a simple structure that consists of the following components: .. figure:: http1-url-structure.png :scale: 100% :align: center The protocol is typically ``http``, but it can also be ``https`` for secure communications. The default port is ``80``, but one can be set explicitly, as illustrated in the above image. The resource path is the *local path* to the resource on the server. **Verbs** URLs reveal the identity of the particular host with which we want to communicate, but the action that should be performed on the host is specified via HTTP verbs. Of course, there are several actions that a client would like the host to perform. HTTP has formalized on a few that capture the essentials that are universally applicable for all kinds of applications. These request verbs are: * **GET**: *fetch* an existing resource. The URL contains all the necessary information the server needs to locate and return the resource. * **POST**: *create* a new resource. POST requests usually carry a payload that specifies the data for the new resource. * **PUT**: *update* an existing resource. The payload may contain the updated data for the resource. * **DELETE**: *delete* an existing resource. The above four verbs are the most popular, and most tools and frameworks explicitly expose these request verbs. ``PUT`` and ``DELETE`` are sometimes considered specialized versions of the ``POST`` verb, and they may be packaged as ``POST`` requests with the payload containing the exact action: *create*, *update* or *delete*. There are some lesser used verbs that HTTP also supports: * **HEAD**: this is similar to GET, but without the message body. It's used to retrieve the server headers for a particular resource, generally to check if the resource has changed, via timestamps. * **TRACE**: used to retrieve the hops that a request takes to round trip from the server. Each intermediate proxy or gateway would inject its IP or DNS name into the ``Via`` header field. This can be used for diagnostic purposes. * **OPTIONS**: used to retrieve the server capabilities. On the client-side, it can be used to modify the request based on what the server can support. **Status Codes** With URLs and verbs, the client can initiate requests to the server. In return, the server responds with status codes and message payloads. The status code is important and tells the client how to interpret the server response. The HTTP spec defines certain number ranges for specific types of responses: **1xx: Informational Messages** *All HTTP/1.1 clients are required to accept the ``Transfer-Encoding`` header.* This class of codes was introduced in HTTP/1.1 and is purely provisional. The server can send a ``Expect: 100-continue`` message, telling the client to continue sending the remainder of the request, or ignore if it has already sent it. HTTP/1.0 clients are supposed to ignore this header. **2xx: Successful** This tells the client that the request was successfully processed. The most common code is **200 OK**. For a ``GET`` request, the server sends the resource in the message body. There are other less frequently used codes: * **202** Accepted: the request was accepted but may not include the resource in the response. This is useful for async processing on the server side. The server may choose to send information for monitoring. * **204** No Content: there is no message body in the response. * **205** Reset Content: indicates to the client to reset its document view. * **206** Partial Content: indicates that the response only contains partial content. Additional headers indicate the exact range and content expiration information. **3xx: Redirection** *404 indicates that the resource is invalid and does not exist on the server.* This requires the client to take additional action. The most common use-case is to jump to a different URL in order to fetch the resource. * **301** Moved Permanently: the resource is now located at a new URL. * **303** See Other: the resource is temporarily located at a new URL. The ``Location`` response header contains the temporary URL. * **304** Not Modified: the server has determined that the resource has not changed and the client should use its cached copy. This relies on the fact that the client is sending ``ETag`` (Entity Tag) information that is a hash of the content. The server compares this with its own computed ``ETag`` to check for modifications. **4xx: Client Error** These codes are used when the server thinks that the client is at fault, either by requesting an invalid resource or making a bad request. The most popular code in this class is 404 Not Found, which I think everyone will identify with. 404 indicates that the resource is invalid and does not exist on the server. The other codes in this class include: * **400** Bad Request: the request was malformed. * **401** Unauthorized: request requires authentication. The client can repeat the request with the Authorization header. If the client already included the Authorization header, then the credentials were wrong. * **403** Forbidden: server has denied access to the resource. * **405** Method Not Allowed: invalid HTTP verb used in the request line, or the server does not support that verb. * **409** Conflict: the server could not complete the request because the client is trying to modify a resource that is newer than the client's timestamp. Conflicts arise mostly for PUT requests during collaborative edits on a resource. **5xx: Server Error** This class of codes are used to indicate a server failure while processing the request. The most commonly used error code is **500 Internal Server Error**. The others in this class are: * **501** Not Implemented: the server does not yet support the requested functionality. * **503** Service Unavailable: this could happen if an internal system on the server has failed or the server is overloaded. Typically, the server won't even respond and the request will timeout. **Request and Response Message Formats** So far, we've seen that *URLs, verbs and status codes* make up the fundamental pieces of an HTTP request/response pair. .. figure:: http1-req-res-details.png :scale: 100% :align: center Let's now look at the content of these messages. The HTTP specification states that a request or response message has the following generic structure: .. code-block:: text message = *() CRLF [] = Request-Line | Status-Line = Field-Name ':' Field-Value It's mandatory to place a new line between the message headers and body. The message can contain one or more headers, of which are broadly classified into: * general headers: that are applicable for both request and response messages http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.5 * request specific headers http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.3 * response specific headers http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.2 * entity headers http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.1 The message body may contain the complete entity data, or it may be piecemeal if the chunked encoding (``Transfer-Encoding: chunked``) is used. All HTTP/1.1 clients are required to accept the ``Transfer-Encoding`` header. **General Headers** There are a few headers (general headers) that are shared by both request and response messages: .. code-block:: text general-header = Cache-Control | Connection | Date | Pragma | Trailer | Transfer-Encoding | Upgrade | Via | Warning We have already seen some of these headers, specifically ``Via`` and ``Transfer-Encoding``. We will cover ``Cache-Control`` and ``Connection`` in part two. *The status code is important and tells the client how to interpret the server response.* * ``Via`` header is used in a TRACE message and updated by all intermittent proxies and gateways * ``Pragma`` is considered a custom header and may be used to include implementation-specific headers. The most commonly used pragma-directive is ``Pragma: no-cache``, which really is ``Cache-Control: no-cache`` under HTTP/1.1. This will be covered in Part 2 of the article. * The ``Date`` header field is used to timestamp the request/response message * ``Upgrade`` is used to switch protocols and allow a smooth transition to a newer protocol. * ``Transfer-Encoding`` is generally used to break the response into smaller parts with the ``Transfer-Encoding: chunked`` value. This is a new header in HTTP/1.1 and allows for streaming of response to the client instead of one big payload. **Entity headers** Request and Response messages may also include entity headers to provide meta-information about the the content (aka Message Body or Entity). These headers include: .. code-block:: text entity-header = Allow | Content-Encoding | Content-Language | Content-Length | Content-Location | Content-MD5 | Content-Range | Content-Type | Expires | Last-Modified All of the ``Content-`` prefixed headers provide information about the structure, encoding and size of the message body. Some of these headers need to be present if the entity is part of the message. The ``Expires`` header indicates a timestamp of whent he entity expires. Interestingly, a *"never expires* entity is sent with a timestamp of one year into the future. The ``Last-Modified``` header indicates the last modification timestamp for the entity. *Custom headers can also be created and sent by the client; they will be treated as entity headers by the HTTP protocol.* This is really an extension mechanism, and some client-server implementations may choose to communicate specifically over these extension headers. Although HTTP supports custom headers, what it really looks for are the request and response headers, which we cover next. **Request Format** The request message has the same generic structure as above, except for the request line which looks like: .. code-block:: text Request-Line = Method SP URI SP HTTP-Version CRLF Method = "OPTIONS" | "HEAD" | "GET" | "POST" | "PUT" | "DELETE" | "TRACE" ``SP`` is the space separator between the tokens. ``HTTP-Version`` is specified as "HTTP/1.1" and then followed by a new line. Thus, a typical request message might look like: .. code-block:: text GET /articles/http-basics HTTP/1.1 Host: www.articles.com Connection: keep-alive Cache-Control: no-cache Pragma: no-cache Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Note the request line followed by many request headers. The **Host** header is mandatory for HTTP/1.1 clients. **GET** requests do not have a message body, but **POST** requests can contain the post data in the body. The request headers act as modifiers of the request message. The complete list of known request headers is not too long, and is provided below. Unknown headers are treated as entity-header fields. .. code-block:: text request-header = Accept | Accept-Charset | Accept-Encoding | Accept-Language | Authorization | Expect | From | Host | If-Match | If-Modified-Since | If-None-Match | If-Range | If-Unmodified-Since | Max-Forwards | Proxy-Authorization | Range | Referer | TE | User-Agent The ``Accept`` prefixed headers indicate the acceptable media-types, languages and character sets on the client. ``From``, ``Host``, ``Referer`` and ``User-Agent`` identify details about the client that initiated the request. The ``If-`` prefixed headers are used to make a request more conditional, and the server returns the resource only if the condition matches. Otherwise, it returns a ``304 Not Modified``. The condition can be based on a timestamp or an ETag (a hash of the entity). **Response Format** The response format is similar to the request message, except for the status line and headers. The status line has the following structure: ``Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF`` * HTTP-Version is sent as ``HTTP/1.1`` * The Status-Code is one of the many statuses discussed earlier. * The Reason-Phrase is a human-readable version of the status code. A typical status line for a successful response might look like so: ``HTTP/1.1 200 OK`` The response headers are also fairly limited, and the full set is given below: .. code-block:: text response-header = Accept-Ranges | Age | ETag | Location | Proxy-Authenticate | Retry-After | Server | Vary | WWW-Authenticate * ``Age`` is the time in seconds since the message was generated on the server. * ``ETag`` is the MD5 hash of the entity and used to check for modifications. * ``Location`` is used when sending a redirection and contains the new URL. * ``Server`` identifies the server generating the message. It's been a lot of theory upto this point, so I won't blame you for drowsy eyes. In the next sections, we will get more practical and take a survey of the tools, frameworks and libraries. Tools to View HTTP Traffic -------------------------- There are a number of tools available to monitor HTTP communication. Here, we list some of the more popular tools. Undoubtedly, the Chrome/Webkit inspector is a favorite amongst web developers: https://developers.google.com/chrome-developer-tools/ There are also web debugging proxies, like Fiddler on Windows and Charles Proxy for OSX. My colleague, Rey Bango wrote an excellent article on this topic. For the command line, we have utilities like curl, tcpdump and tshark for monitoring HTTP traffic. * https://curl.haxx.se * http://www.tcpdump.org * https://www.wireshark.org Using HTTP in Web Frameworks and Libraries ------------------------------------------ Now that we have looked at the request/response messages, it's time that we learn how libraries and frameworks expose it in the form of an API. We'll use ExpressJS for Node, Ruby on Rails, and jQuery Ajax as our examples. **ExpressJS** If you are building web servers in NodeJS, chances are high that you've considered ExpressJS. ExpressJS was originally inspired by a Ruby Web framework, called Sinatra. As expected, the API is also equally influenced. Because we are dealing with a server-side framework, there are two primary tasks when dealing with HTTP messages: * Read URL fragments and request headers. * Write response headers and body *Understanding HTTP is crucial for having a clean, simple and RESTful interface between two endpoints.* `ExpressJS `_ provides a simple API for doing just that. We won't cover the details of the API. Instead, we will provide links to the detailed documentation on ExpressJS guides. The methods in the API are self-explanatory in most cases. A sampling of the request-related API is below: * `req.body `_: get the request body. * `req.query `_: get the query fragment of the URL. * `req.originalUrl `) * `req.host `_): reads the Host header field. * `req.accepts _`): reads the acceptable MIME-types on the client side. * `req.get OR req.header `_: read any header field passed as argument. On the way out to the client, ExpressJS provides the following response `API `_: * res.status: set an explicit status code. * res.set: set a specific response header. * res.send: send HTML, JSON or an octet-stream. * res.sendFile: transfer a file to the client. * res.render: render an express view template. * res.redirect: redirect to a different route. Express automatically adds the default redirection code of 302. **Ruby on Rails** *The request and response messages are mostly the same, except for the first line and message headers.* In Rails, the `ActionController `_ and `ActionDispatch `_ modules provide the API for handling request and response messages. **ActionController** provides a high level API to read the request URL, render output and redirect to a different end-point. An end-point (aka route) is handled as an action method. Most of the necessary context information inside an action-method is provided via the ``request``, ``response`` and ``params`` objects. * `params `_: gives access to the URL parameters and POST data. * `request `_: contains information about the client, headers and URL. * `response `_: used to set headers and status codes. * `render `_: render views by expanding templates. * `redirect_to `_): redirect to a different action-method or URL. **ActionDispatch** provides fine-grained access to the request/response messages, via the `ActionDispatch::Request `_ and `ActionDispatch::Response classes `_. It exposes a set of query methods to check the type of request (``get?()``, ``post?()``, ``head?()``, ``local?()``). Request headers can be directly accessed via the ``request.headers()`` method. On the response side, it provides methods to set ``cookies()``, ``location=()`` and ``status=()``. If you feel adventurous, you can also set the ``body=()`` and bypass the Rails rendering system. **jQuery Ajax** Because jQuery is primarily a client-side library, its Ajax API provides the opposite of a server-side framework. In other words, it allows you to read response messages and modify request messages. jQuery exposes a simple API via `jQuery.ajax(settings) `_: By passing a ``settings`` object with the ``beforeSend`` callback, we can modify the request headers. The callback receives the jqXHR (jQuery XMLHttpRequest) object that exposes a method, called ``setRequestHeader()`` to set headers. .. code-block:: text $.ajax({ url: 'http://www.articles.com/latest', type: 'GET', beforeSend: function (jqXHR) { jqXHR.setRequestHeader('Accepts-Language', 'en-US,en'); } }); * The jqXHR object can also be used to read the response headers with the ``jqXHR.getResponseHeader()``. * If you want to take specific actions for various status codes, you can use the ``statusCode`` callback: .. code-block:: text $.ajax({ statusCode: { 404: function() { alert("page not found"); } } }); **Python** * HTTP modules, https://docs.python.org/3/library/http.html * Requests: HTTP for Humans, http://python-requests.org (added by H. Högl, Dec 2018) Summary ------- So that sums up our quick tour of the HTTP protocol. *We reviewed URL structure, verbs and status codes: the three pillars of HTTP communication.* The request and response messages are mostly the same, except for the first line and message headers. Finally, we reviewed how you can modify the request and response headers in web frameworks and libraries. Understanding HTTP is crucial for having a clean, simple, and RESTful interface between two endpoints. On a larger scale, it also helps when designing your network infrastructure and providing a great experience to your end users. In part two, we'll review connection handling, authentication and caching! See you then. References ---------- * HTTP specification http://www.w3.org/Protocols/rfc2616/rfc2616.html * HTTP Definitive Guide http://www.amazon.com/HTTP-Definitive-Guide-David-Gourley/dp/1565925092 Part 2 ========= by Pavan Podila, 29 Apr 2013 (`Original link for part 2 `_) In my previous article, we covered some of HTTP's basics, such as the URL scheme, status codes and request/response headers. With that as our foundation, we will look at the finer aspects of HTTP, like connection handling, authentication and HTTP caching. These topics are fairly extensive, but we'll cover the most important bits. **HTTP Connections** A connection must be established between the client and server before they can communicate with each other, and HTTP uses the reliable TCP transport protocol to make this connection. By default, web traffic uses TCP port 80. A TCP stream is broken into IP packets, and it ensures that those packets always arrive in the correct order without fail. HTTP is an application layer protocol over TCP, which is over IP. HTTPS is a secure version of HTTP, inserting an additional layer between HTTP and TCP called TLS or SSL (Transport Layer Security or Secure Sockets Layer, respectively). HTTPS communicates over port 443 by default, and we will look at HTTPS later in this article. .. figure:: http2-http-https.png :scale: 100% :align: center An HTTP connection is identified by and . On a client, an HTTP application is identified by a tuple. Establishing a connection between two endpoints is a multi-step process and involves the following: .. figure:: http2-connect-delays.png :align: center * resolve IP address from host name via DNS * establish a connection with the server * send a request * wait for a response * close connection *The server is responsible for always responding with the correct headers and responses.* In HTTP/1.0, all connections were closed after a single transaction. So, if a client wanted to request three separate images from the same server, it made three separate connections to the remote host. As you can see from the above diagram, this can introduce lot of network delays, resulting in a sub-optimal user experience. To reduce connection-establishment delays, HTTP/1.1 introduced persistent connections, long-lived connections that stay open until the client closes them. Persistent connections are default in HTTP/1.1, and making a single transaction connection requires the client to set the Connection: close request header. This tells the server to close the connection after sending the response. In addition to persistent connections, browsers/clients also employ a technique, called parallel connections, to minimize network delays. The age-old concept of parallel connections involves creating a pool of connections (generally capped at six connections). If there are six assets that the client needs to download from a website, the client makes six parallel connections to download those assets, resulting in a faster turnaround. This is a huge improvement over serial connections where the client only downloads an asset after completing the download for a previous asset. Parallel connections, in combination with persistent connections, is today's answer to minimizing network delays and creating a smooth experience on the client. For an in-depth treatment of HTTP connections, refer to the Connections section of the HTTP spec. **Server-side Connection Handling** The server mostly listens for incoming connections and processes them when it receives a request. The operations involve: * establishing a socket to start listening on port 80 (or some other port) * receiving the request and parsing the message * processing the response * setting response headers * sending the response to the client * close the connection if a ``Connection: close`` request header was found Of course, this is not an exhaustive list of operations. Most applications/websites need to know who makes a request in order to create more customized responses. This is the realm of **identification** and **authentication.** Identification and Authentication --------------------------------- *HTTP is an application layer protocol over TCP, which is over IP.* It is almost mandatory to know who connects to a server for tracking an app's or site's usage and the general interaction patterns of users. The premise of identification is to tailor the response in order to provide a personalized experience; naturally, the server must know who a user is in order to provide that functionality. There are a few different ways a server can collect this information, and most websites use a hybrid of these approaches: * **Request headers**: ``From``, ``Referer``, ``User-Agent`` - We saw these headers in Part 1. * **Client-IP** - the IP address of the client * **Fat Urls** - storing state of the current user by modifying the URL and redirecting to a different URL on each click; each click essentially accumulates state. * **Cookies** - the most popular and non-intrusive approach. Cookies allow the server to attach arbitrary information for outgoing responses via the ``Set-Cookie`` response header. A cookie is set with one or more *name=value* pairs separated by semicolon (;), as in ``Set-Cookie: session-id=12345ABC; username=nettuts`` A server can also restrict the cookies to a specific ``domain`` and ``path``, and it can make them persistent with an ``expires`` value. Cookies are automatically sent by the browser for each request made to a server, and the browser ensures that only the ``domain-`` and ``path``-specific cookies are sent in the request. The request header ``Cookie: name=value [; name2=value2]`` is used to send these cookies to the server. *The best way to identify a user is to require them to sign up and log in, but implementing this feature requires some effort by the developer, as well as the user.* Techniques like OAuth (https://oauth.net) simplify this type of feature, but it still requires user consent in order to work properly. Authentication plays a large role here, and it is probably the only way to identify and verify the user. **Authentication** HTTP does support a rudimentary form of authentication called **Basic Authentication**, as well as the more secure **Digest Authentication**. In Basic Authentication, the server initially denies the client's request with a ``WWW-Authenticate`` response header and a ``401 Unauthorized`` status code. On seeing this header, the browser displays a login dialog, prompting for a username and password. This information is sent in a base-64 encoded format in the ``Authentication`` request header. The server can now validate the request and allow access if the credentials are valid. Some servers might also send an ``Authentication-Info`` header containing additional authentication details. .. figure:: http2-authentication.png :align: center A corollary to Basic-Authentication is **Proxy Authentication**. Instead of a web server, the authetication challenge is requested by an intermediate proxy. The proxy sends a ``Proxy-Authenticate`` header with a ``407 Unauthorized`` status code. In return, the client is supposed to send the credentials via the ``Proxy-Authorization`` request header. Digest Authentication is similar to Basic and uses the same handshake technique with the ``WWW-Authenticate`` and ``Authorization`` headers, but Digest uses a more secure hashing function to encrypt the username and password (commonly with MD5 or KD digest functions). Although Digest Authentication is supposed to be more secure than Basic, websites typically use Basic Authentication because of its simplicty. To mitigate the security concerns, Basic Auth is used in conjunction with SSL. **Secure HTTP** .. figure:: http2-https-addressbar.png :align: center The HTTPS protocol provides a secure connection on the web. The easiest way to know if you are using HTTPS is to check the browser's address bar. HTTPs' secure component involves inserting a layer of encryption/decryption between HTTP and TCP. This is the Secure Sockets Layer (SSL) or the improved Transport Layer Security (TLS). SSL uses a powerful form of encryption using RSA and public-key cryptography. Because secure transactions are so important on the web, a ubiquitous standards-based Public-Key Infrastructure (PKI) effort has been underway for quite sometime. Existing clients/servers do not have to change the way they handle messages because most of the hard work happens in the SSL layer. Thus, you can develop your web application using Basic Authentication and automatially reap the benefits of SSL by switching to the https:// protocol. However, to make the web application work over HTTPS, you need to have a working digital certificate deployed on the server. **Certificates** Just as you need ID cards to show your identity, a web server needs a digital certificate to identify itself. Certificates (or "certs") are issued by a Certificate Authority (CA) and vouch for your identity on the web. The CAs are the guardians of the PKI. The most common form of certificates is the X.509 v3 standard, which contains information, such as: * the certificate issuer * the algorithm used for the certificate * the subject name or organization for whom this cert is created * the public key information for the subject * the Certification Authority Signature, using the specified signing algorithm When a client makes a request over HTTPS, it first tries to locate a certificate on the server. If the cert is found, it attempts to verfiy it against its known list of CAs. If its not one of the listed CAs, it might show a dialog to the user warning about the website's certficate. Once the certificate is verified, the SSL handshake is complete and secure transmission is in effect. HTTP Caching ------------ It is generally agreed that doing the same work twice is wasteful. This is the guiding principle around the concept of HTTP caching, a fundamental pillar of the HTTP Network Infrastructure. Because most of the operations are over a network, a cache helps save time, cost and bandwidth, as well as provide an improved experience on the web. Caches are used at several places in the network infrastructure, from the browser to the origin server. Depending on where it is located, a cache can be categorized as: * **Private**: within a browser, caches usernames, passwords, URLs, browsing history and web content. They are generally small and specific to a user. * **Public**: deployed as caching proxies between the server and client. These are much larger because they serve multiple users. A common practice is to keep multiple caching proxies between the client and the origin-server. This helps to serve frequently accessed content, while still allowing a trip to the server for infrequently needed content. .. figure:: http2-cache-topo.png :align: center **Cache Processing** Regardless of where a cache is located, the process of maintaining a cache is quite similar: * **Receive** request message. * **Parse** the URL and headers. * **Lookup** a local copy; otherwise, fetch and store locally * Do a **freshness check** to determine the age of the content in the cache; make a request to refresh the content only if necessary. * Create the **response** from the cached body and updated headers. * **Send** the response back to client. * Optionally, **log** the transaction. Of course, the server is responsible for always responding with the correct headers and responses. If a document hasn't changed, the server should respond with a 304 Not Modified. If the cached copy has expired, it should generate a new response with updated response headers and return with a 200 OK. If the resource is deleted, it should come back with 404 Not Found. These responses help tune the cache and ensure that stale content is not kept for too long. **Cache Control Headers** *Parallel connections, in combination with persistent connections, is today’s answer to minimizing network delays.* Now that we have a sense of how a cache works, it's time to look at the request and response headers that enable the caching infrastructure. Keeping the content fresh and up-to-date is one of the primary responsibilities of the cache. To keep the cached copy consistent with the server, HTTP provides some simple mechanisms, namely **Document Expiration** and **Server Revalidation**. **Document Expiration** HTTP allows an origin-server to attach an **expiration date** to each document using the ``Cache-Control`` and ``Expires`` response headers. This helps the client and other cache servers know how long a document is valid and fresh. The cache can serve the copy as long as the *age* of the document is within the expiration date. Once a document expires, the cache must check with the server for a newer copy and update its local copy accordingly. ``Expires`` is an older HTTP/1.0 response header that specifies the value as an absolute date. This is only useful if the server clocks are in sync with the client, which is a terrible assumption to make. This header is less useful compared to the newer ``Cache-Control: max-age=`` header introduced in HTTP/1.1. Here, ``max-age`` is a relative age, specified in seconds, from the time the response was created. Thus if a document should expire after one day, the expiration header should be ``Cache-Control: max-age=86400``. **Server Revalidation** Once a cached document expires, the cache must revalidate with the server to check if the document has changed. This is called *server revalidation* and serves as a querying mechanism for the *stale-ness* of a document. Just because a cached copy has expired doesn't mean that the server actually has newer content. Revalidation is just a means of ensuring that the cache stays fresh. Because of the expiration time (as specified in a previous server response), the cache doesn't have to check with the server for every single request, thus saving bandwidth, time and reducing the network traffic. *The combination of document expiration and server revalidation is a very effective mechanism, it and allows distributed systems to maintain copies with an expiration date.* If the content is known to frequently change, the expiration time can be reduced—allowing the systems to re-sync more frequently. The revalidation step can be accomplished with two kinds of request-headers: ``If-Modified-Since`` and ``If-None-Match``. The former is for date-based validation while the latter uses Entity-Tags (ETags), a hash of the content. These headers use date or ETag values obtained from a previous server response. In case of ``If-Modified-Since``, the ``Last-Modified`` response header is used; for ``If-None-Match``, it is the ``ETag`` response header. **Controlling the Cachability** The validity period for a document should be defined by the server generating the document. If it's a newspaper website, the homepage should expire after a day (or sometimes even every hour!). HTTP provides the ``Cache-Control`` and ``Expires`` response headers to set the expiration on documents. As mentioned earlier, ``Expires`` is based on absolute dates and not a reliable solution for controlling cache. The ``Cache-Control`` header is far more useful and has a few different values to constrain how clients should be caching the response: * **Cache-Control: no-cache**: the client is allowed to store the document; however, it must revalidate with the server on every request. There is a HTTP/1.0 compatibility header called Pragma: no-cache, which works the same way. * **Cache-Control: no-store**: this is a stronger directive to the client to not store the document at all. * **Cache-Control: must-revalidate**: this tells the client to bypass its freshness calculation and always revalidate with the server. It is not allowed to serve the cached response in case the server is unavailable. * **Cache-Control: max-age**: this sets a relative expiration time (in seconds) from the time the response is generated. As an aside, if the server does not send any ``Cache-Control`` headers, the client is free to use its own heuristic expiration algorithm to determine freshness. **Constraining Freshness from the Client** Cachability is not just limited to the server. It can also be specified from the client. This allows the client to impose constraints on what it is willing to accept. This is possible via the same Cache-Control header, albeit with a few different values: * **Cache-Control: min-fresh=**: the document must be fresh for at least seconds. * **Cache-Control: max-stale** or **Cache-Control: max-stale=**: the document cannot be served from the cache if it has been stale for longer than seconds. * **Cache-Control: max-age=**: the cache cannot return a document that has been cached longer than seconds. * **Cache-Control: no-cache** or **Pragma: no-cache**: the client will not accept a cached resource unless it has been revalidated. HTTP Caching is actually a very interesting topic, and there are some very sophisticated algorithms to manage cached content. For a deeper look into this topic, refer to the Caching section of the HTTP spec. Summary ------- Our tour of HTTP began with the foundation of URL schemes, status codes and request/response headers. Building upon those concepts, we looked at some of the finer areas of HTTP, such as connection handling, identification and authentication and caching. I am hopeful that this tour has given you a good taste for the breadth of HTTP and enough pointers to further explore this protocol. References ---------- * RFC 2616, HTTP specification (https://www.w3.org/Protocols/rfc2616/rfc2616.html) * Gourley, Totty and others, HTTP Definitive Guide, O'Reilly.