In our organization some of our code is stored in a Git repository and I created this document internally to help people understand Git better. So I thought it could be useful for others to use since I try to put the git system in the context that the commands were created (making them more logical for use and easier to remember). This document will explain the basics of the Git repo and how it is used.
Git is a version control system that was original created by Linus Torvalds (Linux creator) to manage the Linux kernel source code. Its core principle is that it is a distributed source control system (meaning the code does not have to be stored in a central server). This allows for each developer contributing code to work entirely isolated and then later mesh those changes with other users in a consistent manner. Each developer has a full copy of the source control for the project on their machine. A program named git was created to perform the actions on the repository locally and the synchronize those changes with others.
Staging Area - index within the git repository that contains what changes you have made to the files in the workspace.
Local Repository - compressed database containing all the files and their history.
Remote Repository - a remote copy of the project that others may use as a reference.
You can think of the local .git directory as the entire source control system. All changes, adds, and deletes are done in the workspace and managed in the local git repository using git commands.
In a traditional source control system, there is a central server containing the source files and users checks out and checks in changes to that server. In git, that all occurs locally against your local git system (.git directory). Those changes can then be shared to a central location with which others can synchronize.
The remote repository is nothing more than a copy of the local git system for others to use as a reference. It serves as a centralized sharing point for everyone to communicate their changes to the others.
Developers make changes in their local projects, link those changes to their local git systems, then share those changes to the central reference copy known as the "remote repository". Others may pull down those changes to their local git system and mesh those changes with their own using git commands.
This creates the local git system for the project and copies down the history and working files while being linked to the shared reference copy.
All developers who are linked to the same remote repository can now share their changes with the same centralized set of changes using git commands.
- pushes changes make to your local repository (linked to a remote repository) to the remote.
- merges changes in your local repository with your working files. Normally changes are meshed by git seamlessly unless two conflicting changes were made (known as a "merge conflict", which must be dealt with manually).
There are several commands to add your workspace changes to the local git source control system. At any point your changes can be undone by either overwriting your local repository with the remote reference, or by undoing changes you have committed to your local repository.
In addition to the history of the current files in the project, git can also manage changes to multiple branches of the same project, each with their own history and common ancestor.
The history of all the branches is stored in the same local repository as before. This allows for distruptive development to occur in parallel with a separate copy of the files. In addition, it can encapsulate a series of changes in a visually separate stream in git visualization tools.
The git branch is not specific to a single file, but to the entire project. So there can be changes to any files within the project in the separate branch and they are all stored in the same repository.
The user may chose to work with any branch in their repository by switching the files in the workspace to the branch of their choosing, or creating a new branch.
Once you have switched branches, all changes made to the project are isolated to that branch and do not affect files in the other branches.
- Switches the current workspace to the <branchname> branch and sets all edits to be on that branch.
Breaking changes will not disrupt the main branch.
Multiple changes can be made slowly over time and tested independently of the main branches changes.
Multiple feature commits can be compressed into a single commit on the main branch.
Next, changes may occur on either the source branch or the feature branch independently. The feature branch can be shared in the remote repository or it can be isolated to a single users workspace.
Once all work has been completed on the feature branch, it is ready to bring back into the master branch. Each of the branch 'tips' will be merged into a single branch.
A merge occurs that combines the two branch tips together in a single commit on the branch where the integration occurs.
This merge represents the simplest flow, but much more complicated branching can occur.
Create a local git system to handle source control for a project.
Share that local git system with others to synchronize.
This take all files and folders in the current directory and creates a .git directory with the necessary git database and other components.
At this point the reference copy of the repository is available for others to clone and to submit changes.
In the last post I talked about the non-RESTfulness of most "REST" api implementations. I will attempt to show a more RESTful implementation and some examples of how this benefits the software. While researching this topic, it became clear (by both lack of examples and poor examples) that this is a tough concept. Before I jump into the example, lets examine why its so hard - because these issues will drive some of the design.
a set of functions, procedures, methods or classes used by computer programs to request services from the operating system, software libraries or any other service providers running on the computer.
And when creating a traditional API, you come up with methods and data formats to tell people how to "talk" to the system. But one of the primary REST tenets as mentioned by Roy Fielding is his blog is that you NOT have "out-of-band information driving interaction instead of hypertext." To put more simply, you should not pre-define the format of the data or the methods to be used, they should be discovered. But isn't that what an API is? How does one define the methods and the data formats but not have to know the methods or data formats? And why should I follow this "silly rule" anyway?
The solution here is that there was a loophole of sorts, baked into the "Rules of REST" - note the last part in the rule: "out-of-band information driving information instead of hypertext." You can tell the client what it is allowed to do, as long as its part of the data (hypertext) being returned. The "media type" can have the information about what is allowed and what format things should be in! This, combined with the specifics of our chosen protocol (HTTP here), should give us all we need. So how do we do that?
NOTE: I am going to ignore security at the beginning because it doesn't help in understanding the example, but I will discuss what could be done at the end.
Client starts with the root URL www.example.com/api/ and a discoverable media type apiIndex-v1+xml.
Here I have defined a set of valid hyperlinks the client can travel to from the root. It is reasonable to assume that the client could get more stuff or less stuff depending on what the business wanted to provide at the root, or based on what the user is authorized to use. This exchange would be the equivalent of a generic "index.html" page for the web. The client could show the descriptions as buttons or a drop down or whatever makes sense for the client (but it doesn't matter at the API level because it is not providing a page to be rendered, just resources to be processed as the client sees fit).
That is the basic REST API example.
Typically, you could do something similar to all resources, but probably using 401 Not Authorized instead of the redirect since you really want the user to go through the index page instead of "knowing" the URL ahead of time. This is just a quick example of security and there are a lot of security vulnerabilities here that you should be careful to avoid. Plus, the client would have to start over at the index page, using the valid token - which is a little awkward. There are dozens of different ways to handle security, like OAuth, BASIC Authentication, etc, so more research is warranted with this topic, but you get the basic idea.
So now let me talk about some of the nuances with the flow.
I am going to parallel these comments with my interpretation of the "rules" that Mr. Fielding presented in his REST blog post.
The only thing the client should know ahead of time is the root URL. It shouldn't know or expect as specific resource to be available. For example, the client can't assume an "order" resource or a "search" resource - those should be provided as choices to the client from the server.
"A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types."
The bulk of what can be done or what is available is meant to be part of the object that is returned. You will notice above that I included a special "resultInformation" block in the product search results. This allows the client to present a "1 of X" link in the app, or make multiple requests to cycle over the results from the server (and limit the bandwidth from the server). Also you will notice that the flow through the API does not jump around, for example, already "knowing" the URL of orders. You don't just jump to /api/orders/123, because the server may have moved it, or deleted it, or you may not be authorized to view it. If you want to give the client the ability to view orders, you create an orderView resource that returns a displayable form of the order and lets the client pick from a list. (server is in control what is available).
"A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of under specified bits of standard protocols, such as HTTP's PATCH method or Link header field."
You will notice above that I presented a use of the HTTP OPTIONS method to assist with the consumption of the API, but tried to stay within the bounds of what is "standard" HTTP. In a normal non-REST API, these method signatures are predefined in a document and both sides code to them. If you do that in REST, you couple the client and server so that neither may change without the other changing. You want the server to decide what is presented and make the client smart enough to handle those choices. Again, it is very difficult to write a client that that is not browser-like and is a little unnatural for an API, but gives great power to the server to evolve.
Again, I have been more long-winded with my post that I would like, but hopefully it gives some value to the developers and REST-ees out there. This was a very difficult post for me to come up with because I had to constantly focus on the REST principles and avoid the classic API mindset. I found it very helpful to discuss with colleagues and to compare what a browser does with what we are trying to get out of the API. There are lots of things that I didn't have space to discuss, like versioning, content-negotiation, internationalization, etc that make this a very rich communication platform to build on. I would appreciate any comments or questions about this flow and REST in general. I know that there are things I can do better, and things that likely need fixing (and that everyone is going to do it differently for their purposes since there is no "right way") so be sure and add a comment.
Our development team is starting to build up some domain objects for the various business objects that we will need and there are a variety of approaches that people seem to take as they start this endeavor. Many of the approaches seem hap-hazard and often capricious with respect to consistency, re-usability, and simplicity. So I decided to talk about my thoughts and maybe solidify my own ideas.
My tack has been to use a "Standards first" domain object design - meaning first look to see if there is a well recognized standard on how something should be represented before I venture out on my own. So for instance, lets say we need to pass a date out of the API, what should the XML for that date look like? I would ask myself, Are there standard ways to represent date in XML? For this example there is a standard XML date representation. I kind of expected everyone to follow this approach when I first started, but it seems this is not the way many developers operate. There are a lot of reasons to look to industry standards first, but there are also some good reasons to burn your own trail.
So what are the advantages of using the standards when possible? Well, for one, it often saves a bunch of time later (often in situations where you wouldn't anticipate it). For instance, In the last project I worked on, I was responsible for formatting the data coming out of the API and I choose to format times using the XML date time standard. I didn't realize at the time that when I went to parse it in our Android application, that android already had an ISO 3339 time/date parser built into the standard libraries - which works with the XML date time format (the XML format is a subset of ISO 3339). So I got automatic time parsing for free by just choosing the right format. I did not know the parser existing when I choose to use standard XML dates with timezones, but it ended up making all the pieces work together much smoother than it would have if I had chosen a "custom" format. These kinds of "happy accidents" seem to follow when standards are used (and it forms the basis for why computer software expands so rapidly - building on things before using components).
Another big advantage to standards first domain objects would be when the time comes to share your API with 3rd parties. The adoption of an API occurs far faster when the formats look like things that developers have worked with before. Not having to explain a format is something that everyone should strive for. Also, the use of the API is better when the developers can use your data using their standard tools (which were developed to standards) with components that are already written.
OK, OK, standards are great blah, blah - why would you choose to NOT use them? The primary disadvantage to them is that the research takes time. It takes a lot of time to look up standards for the data you are trying to use, figure out which one is the "best" or "most common" format. Often there are competing standards, many times the standards bodies want you to pay for the standards documents, sometimes the "standards" are far too complex for the task at hand. I personally like standards that are freely available (like RFCs), because they seem to be more highly vetted and have far more adoption because of the availability. I also like standards that are easy to read - often not needing explanation because they just make sense. So research is necessary and it take a lot of time.
And by this I mean, if I were writing my own software for my own company, and I had 10 APIs to chose from, would this be the one I would go with?
Does it handle my complexity?
Does it handle the nuances that I may not have time to fully flesh out for this project at hand (thus saving me time in the long run). As an example, the time format (ISO 3339) handles time zones and durations and daylight savings time - things that might require me making a few stabs at to get right.
Does it give me room to grow?
Is the structure so strictly defined that I can't build on it, or is it componentized enough that I can use the structure as pieces to my architected objects.
There are a lot of considerations to take into account when designing formats for domain objects and this post merely scratches the surface of a crazy complex issue, but standing on the shoulders of those who came before is definitely in your best interest. And a "standards first" domain object design is a great way to crank out a sustainable and flexible design.
There are a variety of ways to type and version the data with REST services, many of which are used successfully. Because of the living nature of APIs, changing versions and changing data types can lead to API designer headaches that saturate much of their time. I am going to discuss the ways to avoid some of these problems by configuring things properly up front.
Here I have substituted the generic xml request MIME type with a vendor specific MIME type. Awesome, now we ask for a customer XML formated customer, and we get a customer XML back from the call - which can even be validated on the client! That should solve our data format problem once and for all, right?
BUT WAIT - there is still an issue! Lets say you decide to change the customer XML because you want to add some really cool new stuff to make that million dollar sale. How does that impact your client programs?
Basically, you lose no matter which choice you make. So what can we do now to avoid problems with newer versions within the API?
Great! Now you have added the changes and made the sale and not broken any of the clients - you are the company hero! As an added bonus, clients and servers can work together to keep your API stable going forward. By requesting a specific type and version in the request header, the server can decide if it is capable of fulfilling the request and inform the client appropriately (i.e. return a HTTP 415 if it can't fulfill the request, or a 301 if it is changing). This is a much more stable and polite way of dealing with API changes for the client.
We can see that the versioning and the typing of the data are not independent concepts for REST APIs - they are in fact the same, since changing either one results in different data being returned. And since REST is built on a very well tested technology like HTTP, lets take advantage of the capabilities of that technology. It is a simply matter to type the responses being generated and it adheres to 'proper' calling conventions, simplifies the URI and solves our versioning and typing issues cleanly.
We just began switching our existing architecture over to a proper SOA design (yeah!) and decided on using REST (yeah!) for the "API". One of the first discussions we got into was what the REST URI's will look like in the new design. We all agree that proper URI design is a core factor in determining the success of REST so the debate is lively and its good to see everyone passionate about the technology.
One of our architects choose the first for the prototyping we were doing and when I saw it, something didn't sit right about it. So I decided to do a little research -- and it turns out that there are a lot of folks who do it this way, including the Twitter REST API, and the seeming "reference" text on the subject (RESTful Web Services by Leonard Richardson & Sam Ruby).
The first solution has several advantages and it is certainly a quick solution to multiple formats.
Easily testable in a browser - you just type in the format you want.
Simple for most to understand - because you explicitly ask for the format you need.
While this is a sort of a minor problem (you could just say "put it at the end"), it does represent something you will have to think about when writing the calls and something you will have to explain whenever a 3rd party uses your API.
You can certainly do this generic handler, but you end up parsing the type out of the URL which can be tricky if your URI doesn't contain the format or contains the format keywords in the URI.
Provides some confusion with the return type.
What are you really requesting, if we watch the wire while the browser makes this request, you see the browser pass an HTTP GET with an accept header of type "HTML" and the function returns you XML or JSON data. This seems wrong to me. From a browser you won't notice this as much, but if you are actually writing a client you will have to understand that you are not asking for the format in the request header - just the URL.
Can't rely on the REST hosting framework to enforce the accept formats and deny requests the server cannot fulfill without manually checking the formats. If you are going to be doing this, you probably aren't going to be writing the whole REST framework from the socket up - you are going to be depending on .NET or a Java library to "help" you out with the coding and you just want to slap some tags on a function to do handle it. If you stick the return type in the URI, you are not letting the framework handle the format negotiations for you. So you are essentially manually overriding what part of the framework was designed to handle.
This seems to be fighting against something that is already supported in the HTTP header into the URI instead.
The first method has some problems, so lets look at Method 2.
Again, easily testable in a browser - you just type in the format you want.
Again, simple for most to understand - because you explicitly ask for the format you need.
Still have the issue of confusion with the return type.
The format query parameter serves as an override to whatever HTTP header request type you have passed.
This method seems to fix many of the problems mentioned with method 1 and I feel is also preferrable. The cons are relatively minor and it seems workable from a design perspective.
Consistent URIs for all objects regardless of format.
Much more difficult to test than method 1) or 2), but still doable with a plugin in most browsers (Firefox). Since you overriding the browsers default return type (HTML), you must use a plugin to alter the request that gets sent.
More training of the staff to understand how HTTP works and why you have to fill out the HTTP request information properly to get the data you want.
Adhering to the blog title - there is no "right" way to do REST API design. From the analysis, I feel that method 3 is the "best" choice for our architecture and our particular needs. But all 3 have their place and all are used successfully in a variety of architectures. What do you think, what are the pros and cons I didn't think of?
I named this blog "There is No Right Way" to remind myself of that fact when it comes to programming. Often there is a general consensus about which way to go, but with the constant flux of technologies and capabilities, even the sure way will change over time. I will attempt to stir up some discussion about programming ideas and situations as I encounter them in my day-to-day programming.