24 February 2011

Versioning and Types in REST/HTTP API Resources

There are a variety of ways to type and version the data with REST services, many of which are used successfully. Because of the living nature of APIs, changing versions and changing data types can lead to API designer headaches that saturate much of their time. I am going to discuss the ways to avoid some of these problems by configuring things properly up front.

Dealing with Types


Lets look at a common way REST calls are made:

===>
GET /customer/123 HTTP/1.1
Accept: application/xml
<===
HTTP/1.1 200 OK
Content-Type: application/xml
<customer>
  <name>Neil Armstrong</name>
</customer>
Ok, we have an API that returns a customer - looks good. The problem here is that the API does not return a customer - but rather a generic XML document. When the designer made this particular API, they didn't bother to specify the type of document being returned. Sure there is an API document somewhere that defines the customer XML being returned, and as soon as you call it you are going to see that it is a customer - what is the big deal? Well, the problem is that the API is never static. And as far as REST/HTTP is concerned, sticking a product in the response is perfectly valid (but totally wrong in concept). Wouldn't it be nice if the client could actually validate the information coming back? Wouldn't the client be more stable and predictable if we knew the format of the data that was going to come back? Wouldn't it be great if the server knew which format the client wanted and could give the client what they ask for? Sure it would, so lets change the call to make sure we are asking for what we want:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer+xml
<customer>
  <name>Neil Armstrong</name>
</customer>
Here I have substituted the generic xml request MIME type with a vendor specific MIME type. Awesome, now we ask for a customer XML formated customer, and we get a customer XML back from the call - which can even be validated on the client! That should solve our data format problem once and for all, right?

Dealing with Versions


BUT WAIT - there is still an issue! Lets say you decide to change the customer XML because you want to add some really cool new stuff to make that million dollar sale. How does that impact your client programs?
Here is the new flow:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer+xml
<customer>
  <firstName>Neil</firstName>
  <lastName>Armstrong</lastName>
  <salutation>Mr.</salutation>
</customer>
This change has just broken all the clients using that resource since they can no longer parse the changed XML properly! Further, there is no way for the client to check the returned version without calling it or asking for a specific version of the API return object - it just always gets the latest format. Sure you could say that your API needs to maintain backward compatibility - but that is not very realistic when you are properly reusing your API across your product line. To demonstrate further, lets say you have 30 applications (and maybe a handful of external companies using the API), all of which are relying on the "customer" REST resource - your choices now are:
  1. Keep it backward compatible (and lose the million dollar sale because you couldn't implement cool feature X)
  2. Change all 30 applications simultaneously to handle the new data (you likely don't have enough resource to do this and deliver on time)
  3. Make the change, breaking the apps you don't have time to upgrade, but get the sale. (of course you will fix the remaining apps in the future, right?)
Basically, you lose no matter which choice you make. So what can we do now to avoid problems with newer versions within the API?

Common Solutions to Versioning


Some common ways of handling this problem are to request a specific version in the call URI:

Method 1 (Put API version in the URI):
http://server:port/api/v2/customer/123

Method 2 (Add a request parameter in the URI):
http://server:port/api/customer/123?version=2

Both of these methods are used to request a specific version of the resource we are looking for. And for many situations, these work fine. But we are munging up the resource identification with the resources representation (breaking an essential REST tenant). We don't want the identifier that we use to find an object to be mingled with the format we are requesting (for reasons discussed in my last post)! But we are smart designers, and this is essentially the same problem we had above, so we immediately recognize that we can solve this with the same solution as above - just make the type requested more specific:

Method 3 (version the request type):
===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer-v1+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer-v1+xml
<customer>
  <name>Neil Armstrong</name>
</customer>
Similarly, the newer clients make a different call since they are aware of the new version:

===>
GET /customer/123 HTTP/1.1
Accept: application/vnd.company.myapp.customer-v2+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.company.myapp.customer-v2+xml
<customer>
  <firstName<Neil>/firstName>
  <lastName<Armstrong>/lastName>
  <salutation>Mr.</salutation>
</customer>
Great! Now you have added the changes and made the sale and not broken any of the clients - you are the company hero! As an added bonus, clients and servers can work together to keep your API stable going forward. By requesting a specific type and version in the request header, the server can decide if it is capable of fulfilling the request and inform the client appropriately (i.e. return a HTTP 415 if it can't fulfill the request, or a 301 if it is changing). This is a much more stable and polite way of dealing with API changes for the client.

Conclusion


We can see that the versioning and the typing of the data are not independent concepts for REST APIs - they are in fact the same, since changing either one results in different data being returned. And since REST is built on a very well tested technology like HTTP, lets take advantage of the capabilities of that technology. It is a simply matter to type the responses being generated and it adheres to 'proper' calling conventions, simplifies the URI and solves our versioning and typing issues cleanly.

18 comments:

  1. Thanks, this was clearly explained.

    ReplyDelete
  2. How do you deal with clients which run as a plugin in a browser (e.g. flash)? They can not set the Accept Header because of security (the browser does the GET for them)

    ReplyDelete
  3. cyberroadie, That is an interesting problem. I feel it is a poor implementation that flash sometimes hijacks/restricts the accept headers (the differing limitations discussed here: http://www-jo.se/f.pfleger/restful-flash). If I designed flash, I would consider it a bug that the client cannot determine the accept headers being used on the requests. (and the linked article does mention a bug(FP-209) with the headers for POSTS). This falls into the same variances you get with many browser implementations, including the HTML specification - there are wild variations causing drastically differing behavior for the same HTML on different browsers. Unfortunately, using versioned vendor specific XML over REST with flash seems to have its limitations. I suppose you would have to make things generic again and have code to check and restrict the XML versions. Seems silly if you watch the requests flying back and forth (do a GET with generic XML - but really its a very specific version of the xml, I just can't specify that).

    ReplyDelete
  4. You're not addressing the fundamental use-case behind url versioning, that is, how do you handle a change to the resource address. All you did was version the return type.

    What if I, the API designer, would like to change request URLs from /customer/123 to /department/customer/123 or to /person/123, or add to or change existing parameters?

    In that case adding a version namespace is by far the most elegant solution. Going from /v1/customer/123 to /v2/department/customer/123 is clean, simple and preserves backward compatibility.

    ReplyDelete
    Replies
    1. To start, yes all that is basically explained here is versioning the return type - which is a critical piece of the REST architecture. It is also why I have other posts talking about the HATEOAS side of REST as well, where this fits into the grander scheme of things.

      The interesting part part of your question to me is that you seem to be relying on the resources to be a specific URI to begin with. Adhering to REST standards, your app should be invariant to changes in the URL by the host. If you are hard-coding your app to a specific address, you are violating REST practices. The client should be discovering the addresses through links from a root address (see http://thereisnorightway.blogspot.com/2012/04/putting-t-in-rest.html). In which case changing the URL has no effect on the client (just like when web pages move something around - you don't change your browser, just the path to the object from the root). In theory your client would behave like a browser, and the data type would give all choices that the client could go.

      But ignoring that, you still have to deal with the versions of the objects themselves. You didn't really make your API work very well for backward compatibility - because lets say you have two apps using the API, AppA and AppB. AppA uses the old URL /v1/customer/123, AppB uses the new URL /v2/department/customer/123. Now you discover a bug in the AppA, v1 code that necessitates changes to the v1 customer object - there is no way to alter the return type of the object without changing the URL in the client (to something like /v1.1/customer/123) or forcing the client to upgrade to v2. But you aren't really solving the problem by doing that, just forcing an upgrade(which could be easy or hard depending on the changes between versions). And if you have dozens of different apps relying on the API, you get yourself into a pickle where the only way to make changes is to upgrade the client to a new version of the API rather than fixing the one issue you have with the one object you have. (there is a lot more discussion to be had with the versioning of the API verses the versioning of individual objects within the API.)

      If you are foregoing REST, then it is probably cleaner to go with a specific URL for a specific version of the API. But then, it is just proprietary API using HTTP and XML. REST is supposed to let the server decide what choices the client can choose from (following the HATEOAS principle) and provide a means for the client to discover new or moved resources. By knowing the URL or the object type on the client ahead of time, you have violated that and are not letting the control rest on the server - you have coupled the client and server together. So you are only slightly better off, and a little more generic, than writing a proprietary protocol over a direct socket. And you shouldn't call it REST because its not. There is nothing wrong with that design, its just not REST and doesn't gain the benefits that REST aims to provide since you can't freely make changes on the server.

      Delete
  5. I have posted my doubts in a comment on another post about the practicality of address discovery for API consumption. And, now it comes up again here. I feel like the urgency to keep versions out of the URL comes down to this need for discoverability for the sake of some hypothetical auto-navigating client that does not exist.

    In RESTful Web Services (O'Reilly 2007), Richardson and Ruby strongly favor versioning in the URL, or as a subdomain (8.12. Service Versioning). I don't recall if at any point they advocate application-specific data types at all. Did they get it wrong, or has there been a shift in RESTful thought regarding versioning since 2007?

    The approach you advocate does seem consistent with that presented in the REST API Design Rulebook (O'Reilly 2011). But I'm trying to think about this pragmatically. It feels to me like this puts undue burden on the client programmer to manage consumption of resources by specific version rather than to simply build to a specific version of the overall API. Also, If I am designing an API, I feel like I might be turning away potential customers who look at my application-specific data types and simply freak out because they have never seen such a thing. I just don't know of any API out there in the wild that does it this way. Can you point to any examples?

    The need for a resource URL to be a strict permalink for all time feels a bit ideological to me, and I have yet to see its proper justification. And as I dive into designing my API, I can't help but wonder, if the API building community can't really come to a consensus on the advantages of strict permalinking, and everybody is versioning their URLs, then in the interest of actually being a part of that community and maximizing appeal to potential consumers, why shouldn't I as well?

    I apologize if this comment comes across as a bit trollish. That is not my intent. I guess I'm not asking any specific question so much as I am thinking out loud at this point as I contemplate how to approach my API design. I do appreciate any further comments on these thoughts, and I am really getting a lot from the blog in general.

    Thanks

    ReplyDelete
    Replies
    1. Jeremy, I should have more carefully read your reply to dsk before posting my comment. You do touch on some of my concerns. Your example of dealing with a bug, resulting in forced upgrades is an interesting one to think about.

      Also, I think you might have the same response to me regarding my "if everybody else is doing it" thoughts -- maybe what I am reaching for here is simply not doing REST.

      I guess if I better understood just how one goes about building a client that does proper discovery, then maybe I would understand this particular aspect of REST better. Do you know of any examples of properly designed REST APIs and clients who properly take advantage of navigation discovery?

      Delete
    2. I was responding to your other comment when you wrote this one - AtomPub attempts to do navigation discovery but only in the context of RSS feeds. So it does it, but it may not be that useful for general APIs. In general, I have found very few examples of proper navigation discovery (other than the web itself). The internet as it exists today, is the best example to use for HATEOS. Its just hard to extend APIs to that realm, because you probably don't want your API to only use HTML.

      Delete
    3. The RESTful Web Services book has some issues in my opinion. I feel quiet strongly that you should not version in the URL. Even if you don't do full (HATEOAS) REST, I still would NOT do versioning in the URL. The HTTP protocol already has a good way to deal with versions and by putting it in the URL you are subverting that architecture. So why would you rework something that is done quite well in the protocol you are using? I understand that versioning the entire API with a single version may have some advantages, but again, there are other ways to do that without corrupting both the protocol and the concept of the URL.

      REST is ideological for many applications - since for many applications there are far simpler ways to do what they need. And I would suspect that if the web didn't exist, REST would be dismissed as theoretical. But since the web does exist and it is RESTful, you have a fantastic example of what REST can provide. The web would not have been a success if the browser was coupled with the server. That fact alone speaks volumes about why we should be focusing on what it means to do REST properly.

      Porting those concepts to an API is the challenge. The reason I started this blog was to work through that idea myself (since there we so little in the way of example out there). Much of what is out there, touted as REST, is clearly not RESTful. My colleagues and I have been bouncing around RESTful ideas for a while and we keep trying to figure out what it would mean for our applications. I stated in my example that making an application that doesn't look like a browser is exceedingly difficult. I have been butting my head against a wall trying to come up with something that doesn't require a ton of work but still makes sense in terms of a proper RESTful app. I can see that being successful in that endeavor can provide huge benefits (since I can just change the server and have a whole new app, theoretically), but actually implementing that concept proves to be difficult.

      But I can see that you could, with much less effort, produce a client that knows how to render specific resources well (say for example, just orders-v2, customers-v1, and turns known link types into buttons), and everything else is ignored. It could still obey navigation (home page gives back say orders link, customers link, and contacts link - but the app doesn't know about contacts, so that link is not rendered). It is not far from realizing that there is probably a representation that would handle the common tasks needed from an API client. Finding that minimal set of objects that maximizes what the client can do would be my goal.

      Delete
  6. Hi Jeremy, do you have any feedback on the impact of using HTTP Request headers to specify version and caches/Content Delivery Networks (eg. Akamai)? My understanding is that an accepts header is ignored by most proxies/caches and that a stale version of this resource could be/will be delivered if you don't embed version (and language/culture in our case) in the URI.

    GET /version/1/locale/en-US/portfolio/12345/transaction/12345

    would have a different result if local fr-FR is used even though the resource is the same resource.

    ReplyDelete
    Replies
    1. Normally the accept headers are ignored in those services for determining cache "freshness." However, the HTTP spec has a header called "Vary" which was designed to fix just this problem - If you specify the Vary header with the items that could change (like language), you are telling the caching service that it needs to include that element when determining if the request is the same (and hence whether it can use the cached version of the resource to satisfy the request). It basically is just a switch to include that header information in the cache key generation (thus making it a component of the object being cached).

      Here is an article I ran across discussing its relation to REST requests: http://www.subbu.org/blog/2007/12/vary-header-for-restful-applications

      Delete
    2. Upon review, I was a little incorrect in my explanation:
      I would expect most content delivery networks (CDNs) to respect the request headers for the purposes of caching. I don't have experience with Akamai directly, but from the quick research, they appear to have a very sophisticated means of handling caching that uses primarily custom headers - so you would have to review their docs since it could have significant implications to the standard headers.

      That being said, the "Vary" header should still apply - essentially telling the caching service that the request header field defined by the vary header should be handled specially - i.e., this field is important and affects output, make sure the cache realizes it varies, so you will need to include it in the cache hash.

      Delete
  7. I know this thread is pretty old, but I too got into loong discussions around putting version (API not resource version!) part of URL.

    Version Management Complexity
    =============================
    The main arguments I heard around the PROs of making API version part of URL is that it removes complexity to consumers to maintain the "x" versions of "y" resources you expose as an API. Say you first app release exposed 10 resources, then next release exposed 2 additional ones, yet next version removes 3 resources (yes you may terminate resource support as you business requirements evolve). So we have x=3 and y(1)=10, y(2)=12 and y(3)=9.

    With consumers having to know which versions they support, they have to remember 10 versions (as part of resource media-types) on first implementation. Then after server release next app, consumer may want to upgrade to latest release (to take advantage of latest resource versions/features). So it needs to know which of the API v1 resources are unchanged, which were added, and which have been upgraded with new features (new versions for these individual resources). All-in-all, consumers have a fairly complex matrix of resources versions to maintain over time.

    Compare that to the same use cases above, but now managing API versions instead of only resource version. Note that I do not imply the API version is part of Header or URL. I am just looking at versioning complexity from consumer standpoint. With previous example, on first server release, consumers just need to know one thing. They use v1. All resources. When server releases new version, a new API versions v2 becomes available. If consumer decides to upgrade to latest features, all it needs to remember, is to use API v2. All resources. Maybe internally, most resource representations (media types) of v1 remain unchanged in v2. But consumer does not need to maintain or even know that.

    (to be followed 1/2)

    ReplyDelete
  8. (followed 2/2)

    Cross Domain Links
    ==================

    Now I talked about maintaining versions of resources vs. versions of APIs (which represent a set of resources available at a given service release). I implied all the resources were part of a same server/domain/API.

    The main argument advocating for CONs is how to handle links - is when resources point to other resources that are *NOT* part of the same server/domain/API. Say I have an app server that exposes "actors". Each actor may appear in one or more movies. Instead of designing a movie resource in the same app server, the API designed a link relation to the http://imdbapi.org/ API (not hosted/maintained by the app server). This is an example, it could be any other app server that is maintained by the same company but maintained on a different release cycle (e.g. another division of the company). There is a inter-resources dependency between the different app servers. Each live within a different API (and thus different API lifecycle).

    Assuming my application wants to be RESTful, and support HATEOS. I expect my "actor" resource, part of my app server API v1, to have links to the other API "movie" resource. Assuming the API was designed to include API version as part of the "movie" URL, how do I address the following use cases:

    - Assign an existing actor to a new movie (PUT on "actor"). Does the consumer sends the link of the "movie" API including version? If so, does the "actor" API server implementation stores the full link in its underlying persistence layer (so that it "keeps" reference of the movies for that actor)? e.g. http://imdbapi.org/v1/movies/123

    - Fetch actor details to retrieve the movies he plays in (GET on "actor"). What should the "actor" API return? The links that were persisted in previous use case as-is? What if the versions (that was stored as part of the link) is not supported any longer by the "movies" API? My links are broken?

    Consider this vs. removing the version from the URI. I would retrieve my movies links from the "actor" API, eg http://imdbapi.org/movies/123. The "actor" API does not need anything else than this link, which is merely a pointer to an existing "movie". The consumer developer however now has the burden to actually now which (API) version of the "movies" API it supports, and request that version. If the consumer was supporting the "v1" version, the consumer app would be broken as well. But if the "actors" were created by different consumers (some using v1, other v2, other v3 of the "movies" API) then here, each consumer has freedom to upgrade or maintain the "representation of movie" it supports, instead of having to deal v1-v3 that may return from the "actors" API.

    I don't know if my example is clearly exposed, feel free to ask if not. My point is making API (or resource) version part of what is persisted on the API side makes the maintenance of consumers app a nightmare.

    Phil

    ReplyDelete
    Replies
    1. Regarding part 1:

      From the consumer point of view, it doesn't really matter whether the version is in the URL or not - the client will still need to know which resource versions the client can handle. If you don't put the version in the URL, you can still ask for a specific (older) version of a resource - even if you have deprecated a resource (you can have side-by-side versions available). You would just make the item return a deprecated or moved status if you want to encourage clients to move away from that particular version/URL. This is NOT really different than having a client written against APIv1 then putting up an APIv2. You generally leave support for the old API for some amount of time before deprecating it completely and then refusing to honor the request - and you can do that whether or not you put the version in the URL.

      The whole situation you have is a little uncomfortable because you have tight coupling between the client and server since you are hitting resources directly from the client - so you are completely not following HATEOS (otherwise you would start at a root and the client would browse to a location and get the URL and type via discovery and it wouldn't matter if it changed).

      Regarding Part 2:
      I agree that it would be difficult to maintain if the version were in the URL (which is why I highly discourage it). It seems flawed to me that you are attempting to return the URL from two independent servers in a request to either server. It would be like a website having a hard-coded link to a different website - then the other website changed the URL of the resource(so you get a bad link). But I can think of two ways to handle the situation (which are really the same, but at first glance look different):
      1) Be like Youtube(and others) where they provide a static reference to a resource (that is not version dependent) and it redirects to the "active" resource path when requested. So they can change the URL, move it around, provide different versions - and the client can discover the versions available if a specific version that it supports is not available. But you always have a fixed resource path that is static.

      2) Provide a third intermediary server/resource which hosts the API representations that dynamical determines the URLs of both a movie and an actor from the two servers and returns it to the caller. So as the URLs change, the independent sides don't have to know how to link the two together - only the intermediary does. The API would have a resource such as "Movie and Actor", and it discovers the two resources from the two independent servers and merges the result to the client.

      It is not a situation I normal get into in my work because our systems do not have the throughput that this scenario is calling for, but you get the idea. Again though, I would recommend just having links to ideas, not specific versions because you are already asking for a versioned representation in the request anyway.

      Delete
  9. Thanks for this post. I am leaning away from your suggestion and more towards putting the version in the URL, but I definitely appreciate your post; it helps me to make a decision for our project with a full understanding of what I might be getting myself into.

    ReplyDelete
  10. Can you explain why vnd.company.myapp.customer+xml-v1 is better than or preferred to vnd.company.myapp.customer; version = 1; format = xml ?

    ReplyDelete
    Replies
    1. If you are saying you put both of those into the accept type, I don't think it matters. What you wrote is really what the ...+xml-v1 is saying, you just spelled it out (so yours is slightly more verbose, but clearer). The reason I put +xml on there to begin with is because its what is stated in the RFC for HTTP (vnd explained in RFC2048, and +xml in RFC3023). You could put the same information in the header of the request in custom headers, or in a request object past to the server, but HTTP has a spot for the "type" and a standard for how it should look so I was using it.

      Delete