Lawrence Wagerfield Fragmenting The Heap Since 2000

REST Principles

This article composes various findings from technical papers, journals, books and of course Fielding's original dissertation: Architectural Styles and the Design of Network-based Software Architectures.

Some aspects of this article are perhaps a little vague, but should hopefully provide a good starting point to those learning REST. I'd strongly recommend Fielding's dissertation and 'RESTful Web Services' by Richardson & Ruby; the former is essential for anyone serious about the topic.

Introduction

Representational State Transfer (REST) is an architectural style which was introduced in 2000 by Roy Fielding as part of his doctoral dissertation. The paper was written as a description of what made the Web successful, and identifies that the Web's basic technologies generally suffice for distributed computing.

REST advocates a set of ideals for developing resource-oriented architectures (ROAs) and therefore exists as a contrast to concrete remote-procedure-call (RPC) specifications like WS-* and CORBA.

The chief principle of REST is to use the Web's core features and to avoid implementing any custom extensions. The following 8 principles simply reiterate this statement at a more granular level. Hi-REST APIs strictly conform to each of these principles, whereas Lo-REST APIs will only consider a few:

1. Resources
2. Representations
3. Addressability
4. Statelessness
5. Descriptive URIs
6. Hypermedia and HATEOAS
7. Safety and Idempotence
8. Interface Unity

Note: The term 'RESTful' can be used to describe any number of applications and protocols, primarily due to the broad scope of Fielding's REST dissertation. To prevent ambiguity this article will use the term 'REST API' to describe RESTful HTTP web services which are intended primarily, although not explicitly, for machine consumption.

REST Principles: Resources (1 of 8)

REST promotes a resource-oriented-architecture (ROA) whereby resources are defined as "any information that can be named" (Fielding, 2000, p. 88). In a more practical sense, a resource is something that can be represented on a computer as a stream of bits.

All resources must have at least one URI, although the decision as to which entities should be exposed as resources is entirely up to the developer. However, to help improve addressability developers should heed the following advice:

A resource should have an associated URI if another party might reasonably want to create a hypertext link to it, make or refute assertions about it, retrieve or cache a representation of it, include all or part of it by reference into another representation, annotate it, or perform other operations on it. (Berners-Lee, 2004)

Theoretically resources must be unique, although multiple resources may point to the same data. Assuming that the underlying concept of each of these resources is different, then they are still considered as being unique, even though their representations are identical. Consider a software website which exposes a resource for each version of their software (/versions/1.5) in addition to a resource pointing to the latest version (/versions/latest). Two of the website's resources will always share the same data, although their concepts remain different.

A resource may also have multiple URIs, although it is generally recommended that one of them is defined as being canonical. The canonical URI represents the 'master' URI for that resource and will exhibit normal behaviour. Conversely, all other URIs should return a 303 response ("See also") along with the canonical URI, thus hinting at the master location. However, this is generally not advocated due to the obvious maintenance overheads this method imposes. Consequently, it is recommended that resources only have one URI.

REST Principles: Representations (2 of 8)

REST defines resources as having one-or-more representations (Fielding, 2000). Representations provide tangible content for each of the service's resources using specific formats.

A common misconception is that resources are the data. They are not. Resources merely symbolise the concepts and ideas behind how a service's data is split up. The representations are the data, and provide a means for the service to articulate these resources as a series of bytes.

A single resource may expose several representations. Systems typically distinguish between these representations using either of the following two methods:

Multiple URIs (The Wrong Way)

One method for referencing particular representations is to use URIs. This is the simplest approach and is implemented by many popular services, including the Twitter API. The aforementioned API utilises standard file extensions to differentiate between representations. For example: *.json URIs return JSON representations, whereas *.xml URIs return XML.

This approach also provides a familiar interface for the untrained user, since it is similar to the approach taken by operating systems, which determine file types based on extensions.

However, the approach technically breaks the opacity axiom as defined in 'Universal Resource Identifiers - Axioms of Web Architecture':

It has been very tempting from time to time for people to write software in which a client will look at a string such as ".html" on the end of an identifier, and come to a conclusion that it might be hypertext markup file when dereferenced. But these thoughts of breaking of the rule could lead to a broken architecture. (Berners-Lee, 2004)

This method also lends itself to the problems discussed in the post on resources, whereby multiple URIs dilute the value of one another. The 'canonical URI' solution was discussed, although that too introduces new problems with regards to additional maintenance overhead.

Worse yet: this method would technically be classed as tunnelling, since content negotiation (discussed next) is actually a native feature of HTTP. Disregarding this feature in favour of another approach would be violating the chief principle outlined in this article's introduction: Use the Web's core features and avoid implementing any custom extensions.

Content Negotiation (The Right Way)

Fortunately HTTP offers a better solution: content negotiation. When a client makes a request, it specifies a range of representations it is willing to receive via HTTP's Accept header. The header accepts a prioritised list of MIME types which supports the use of wildcards, thus allowing the client to specify fallback ranges if their initial preferences cannot be satisfied.

Fielding also hints at this approach in his dissertation by describing how URIs should point to resources and not representations:

The abstract definition of a resource enables late binding of the reference to a representation, enabling content negotiation to take place based on characteristics of the request. It also allows an author to reference the concept rather than some singular representation of that concept, thus removing the need to change all existing links whenever the representation changes. (Fielding, 2000, p. 89)

The following example helps to illustrate this concept: At the time of writing, the Twitter API uses multiple URIs to differentiate between representations. Assuming the Twitter API was refactored to use content negotiation, requesting a JSON representation might have the following syntax:

GET /statuses/public_timeline HTTP/1.1
Host: api.twitter.com
Accept: application/json, application/xml;q=0.9
...

Notice how the URI contains no information regarding a specific representation of the resource. Also notice how XML representations will also be accepted, but are 10% less favourable than JSON representations. This is specified using a quality value (RFC2616 – section 3.9).

Content negotiation also provides a mechanism for supporting different versions of client software. For example, the service could detect client versions based on their accept headers, e.g. application/vnd.twitter1+xml would imply a 1^st generation client requesting XML, whereas application/vnd.twitter2+json would imply a 2^nd generation client requesting JSON. The service could then alter its response accordingly. Also notice usage of the vnd. or 'vender tree' prefix; this should be used for custom types to prevent MIME pollution (RFC2048 - MIME, 1996).

This is the favourable approach given its recommendations and key benefits.

REST Principles: Addressability (3 of 8)

An addressable service exposes a URI for every resource it might conceivably serve (Richardson & Ruby, 2007). Following this principle allows clients to harness the power of URIs, enabling them to bookmark and reference resources.

The URI concept is often taken for granted in this sense: up until the mid 1990s FTP wasn't addressable via URIs (Berners-Lee, 1994). Consequently people had to reference FTP resources by describing a sequence of events: anonymously logon to example.com, change directory to /www/pages, and then download index.htm.

Unfortunately the deficiency of poor addressability still exists in Big Web Services, and is therefore one of REST's key merits in the Big Web Services vs. REST debate.

REST Principles: Statelessness (4 of 8)

A stateless service processes every request in complete isolation. This is achieved by requiring that each request includes all necessary information for fulfilment, thus moving session-state from the server-side to the client-side.

Fielding (2000, p. 78) argues that all services should be stateless, and the advantages gained by taking this approach usually outweigh the more restrictive development style it promotes. Fielding claims the primary advantage of this approach is improved network efficiency, although other advantages are also apparent:

• Caching is the main advantage outlined in Fielding's dissertation. HTTP caching mechanisms are built-into most Web clients, and are designed to determine whether to cache a resource just by analysing its request. Stateful services typically break these mechanisms, thus providing the main argument for statelessness.
• Load balancing also forms another argument for statelessness. Since requests are independent from one-another, stateless services can easily be deployed across multiple servers. This is also possible with stateful services, although this incurs additional overheads since 'session replication' must also be implemented. Needless to say the stateless approach is a lot simpler.
• Failure conditions are also reduced through statelessness. Client-timeouts are eliminated and services can never 'lose track of' or miscalculate sessions. Consider issuing a 'delete' command on a subfolder when the service has you registered in the root directory; the result would be dire.
• Coincidently, statelessness also promotes addressability. URIs from a stateless service can easily be referenced without requiring the service to be primed before each request.

REST Principles: Descriptive URIs (5 of 8)

URIs should be descriptive and well structured. It is also important to remember that URIs designate resources, not methods. Therefore it is never appropriate to include verbs inside path names. Relationships can be described through hierarchical paths (Berners-Lee, 1996) and querystrings should be avoided since they fail to describe relationships between URI parts (Bone, 2002).

Despite the latter argument, it is generally acceptable to use querystrings to provide additional data at the same level as the resource, providing the feature is not abused (Richardson & Ruby, 2007). Some APIs avoid using querystrings altogether by imbedding variables into path space, using a predefined character to delimit values. This method follows a similar approach to 'matrix URIs'; a concept introduced by Berners-Lee (1996). Unfortunately matrix URIs failed to receive wide-spread adoption by Web clients. Furthermore, the approach also limits the character set available to resource names due to reservations on the semicolon ';'. This limitation alone makes the approach impractical for inclusion within the project, since APIs may wish to expose resources (like files and folders) which could include said reserved character.

As previously hinted in the post on representations, file extensions should be omitted from path names in favour of content negotiation. File extensions technically break the opacity axiom, and are also subject to change. Consider a service switching from PHP to Java; if this approach were taken, the old *.php hyperlinks would break, since the new resources would now have a *.jsp suffix.

URIs should also have a predictable structure. Predictable URIs follow a generative naming style, which implies that given any URI a client should be able to ascertain additional resource locations without having to follow links. Generative naming is also useful for describing information spaces which cannot (realistically) be enumerated by hyperlinks. These conditions are rare, so using generative naming instead of hyperlinks demands a very good reason.

REST Principles: Hypermedia and HATEOAS (6 of 8)

Fielding (2000, p. 82) describes a way of using hypermedia as the engine of application state (HATEOAS). This concept requires services to outline workflows using hyperlinks, and therefore gains the advantage of being able to expose distributed resources. HATEOAS also implies that a client should be able to access every permissible resource in a service given the root URI (or 'bookmark').

Unfortunately this requirement is frequently overlooked by most REST APIs. Services like Twitter and Flickr use external documentation to inform clients about resource locations. The actual representations the services return make little attempt at referencing surrounding resources.

However, if implemented correctly HATEOAS can have considerable benefits on a REST API:

• Client coding errors are reduced by minimising the number of hardcoded URIs to one: the bookmark. This reduces the impact of broken links, which is currently one of the largest problems with online applications (Ingham, Caughey, & Little, 1997). Using a bookmark, additional resources can be located either by generatively producing URIs based on hardcoded relative paths, or by examining hyperlinks in resource representations.
• Invalid state transitions are reduced. Hypermedia systems use links to outline the application workflow, rather than relying upon the client to make (possibly incorrect) choices using hardcoded URIs.
• Increased scalability. The ability to reference infinite resources has obvious scalability implications. Furthermore, providing only new links are added and the old ones remain, updates to the service won't cause existing clients to break.

REST Principles: Safety and Idempotence (7 of 8)

These two concepts are used to define method behaviours: safe methods do not modify state, whereas idempotent methods can, but will not cause additional changes after their initial execution.

Most of HTTP's methods fall into one or both of these groups, therefore allowing developers to design conventional services which expose predictable interfaces to clients.

It is important to note that POST is neither safe nor idempotent. The method is somewhat more flexible, but completely unpredictable. The post on interface unity describes when it is appropriate to use the method.

Safety

GET and HEAD methods are considered safe. These methods should only be used to read data, and must therefore have no side-effect on the server's state. Making a request multiple times should be the same as never making it at all.

Unfortunately many websites violate this concept, which is potentially dangerous. Applications like Google Accelerator rely upon GET requests being safe, and therefore break when poorly designed websites are encountered. Consequently, many reports claim that early versions of Google Accelerator accidentally deleted resources due to its pre-fetching techniques (Lenssen, 2005), thus demonstrating the importance of conformant design.

Idempotence

GET, HEAD, PUT and DELETE methods are considered as idempotent. These methods should perform the same effect if performed once or multiple times. For example, performing a DELETE request on a resource implies that it's gone. Performing the request again implies the state has not changed, and the resource has still gone. Likewise with PUT; uploading a resource 10 times is the same as uploading it once, but is different to not uploading it altogether.

REST Principles: Interface Unity (8 of 8)

Maintaining a consistent interface across the Web grants the transfer of skills between services, since everyone only has to learn one set of rules. This has several advantages, including minimised documentation, reduced learning curves and higher interoperability (Richardson & Ruby, 2007).

Understanding the correct use of a universal interface (in this case, HTTP) is paramount to developing a conventional service. The remainder of this subsection discusses the correct usage of various features within HTTP, drawing close attention to the most frequently misused facets of the protocol.

HTTP verbs

HTTP provides several verbs for specifying resource interaction. The 4 verbs most frequently used by REST APIs include: GET, PUT, POST and DELETE (GPPD). Despite common misconceptions, these methods do not align directly with CRUD operations.

GET and DELETE are the simplest verbs. They have no body and do in fact map onto the 'retrieve' and 'delete' CRUD operations, respectively.

The exact operation of PUT and POST is slightly more controversial. Their intent is often disputed, perhaps due to the lack of an exact mapping between GPPD and CRUD. However, RFC2616 makes the intended operation of the two methods very clear:

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line … The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it. (W3C, 1999)

The PUT method requests the entity be stored under the supplied URI.(W3C, 1999)

Evidently POST requests are far more unpredictable than PUT requests, since the posted entity may be saved anywhere, providing it's a subordinate of the referenced URI. Fortunately HTTP provides a mechanism for tracking the whereabouts of the new resource via the Location header. This header can be returned by services to reference the locations of new resources.

HTTP status codes

HTTP defines various codes for indicating a request's status. The correct use of these codes is often overlooked, with a majority of services tunnelling a request's status through the response body. Consequently many services return 200 ("OK") even when the request was unsuccessful.

This is a trivial issue with regards to the human Web. People are accustomed to determining a request's status by reading the returned document. Scanning for words like 'missing' or 'access denied' provides a good insight into the request's status without relying upon codes.

However, this issue is obviously more severe with regards to the programmable Web. Client applications require predefined codes to accurately understand a request's status. The correct use of HTTP status codes can be used to address this issue by providing a conventional, machine-readable means of determining a request's status.

HTTP status codes are divided into 5 categories: informational, success, redirection, client error and server error. A full description of each status code is impractical given page constraints. However, the W3C provides a complete description of each code in RFC2616 under section 10 ("Status Code Definitions").

HTTP headers

HTTP headers can provide additional scoping information which isn't necessarily appropriate for inclusion within address space (the URI). The section on content negotiation in the representations post demonstrates this by explaining how formats can be selected using the Accept header and a range of MIME types.

HTTP headers can also specify metadata (e.g. Last-Modified), conditional statements for requests (e.g. If-Modified-Since) and authorisation information. The list of HTTP headers within the current protocol specification is extensive.

Services can also utilise custom headers to transfer metadata which is not covered by standard HTTP headers. The common convention is to place an 'X-' before the header name to imply the header is an extension. The Amazon S3 service makes extensive use of these for authentication.