Twisted Web in 60 seconds: static URL dispatch
Welcome to the third installment of "Twisted Web in 60 seconds". The goal of this installment is to show you how to serve different content at different URLs using APIs from Twisted Web (the first and second installments covered ways in which you might want to generate this content).
Key to understanding how different URLs are handled with the resource APIs in Twisted Web is understanding that any URL can be used to address a node in a tree. Resources in Twisted Web exist in such a tree, and a request for a URL will be responded to by the resource which that URL addresses. The addressing scheme considers only the path segments of the URL. Starting with the root resource (the one used to construct the Site) and the first path segment, a child resource is looked up. As long as there are more path segments, this process is repeated using the result of the previous lookup and the next path segment. For example, to handle a request for /foo/bar, first the root's "foo" child is retrieved, then that resource's "bar" child is retrieved, then that resource is used to create the response.
With that out of the way, let's consider an example that can serve a few different resources at a few different URLs.
First things first: we need to import Site, the factory for HTTP servers, Resource, a convenient base class for custom pages, and reactor, the object which implements the Twisted main loop. We'll also import File to use as the resource at one of the example URLs.
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.web.static import File
Now we create a resource which will correspond to the root of the URL hierarchy: all URLs are children of this resource.
root = Resource()
Here comes the interesting part of this example. I'm now going to create three more resources and attach them to the three URLs /foo, /bar, and /baz:
root.putChild("foo", File("/tmp"))
root.putChild("bar", File("/lost+found"))
root.putChild("baz", File("/opt"))
Last, all that's required is to create a Site with the root resource, associate it with a listening server port, and start the reactor:
factory = Site(root)
reactor.listenTCP(8880, factory)
reactor.run()
With this server running,  will serve a listing of files from /tmp, http://localhost:8880/bar will serve a listing of files from /lost+found, and http://localhost:8880/baz will serve a listing of files from /opt.
Here's the whole example uninterrupted:
There are a variety of options for dealing with the issue when you do need to consider it. Serving static content with another web server is certainly one. At some point, you don't even really care what the web server is, you just want to dump your static data onto a CDN and move on. :) However, it's also possible to implement something like File that makes use of some platform's asynchronous I/O APIs. O_NONBLOCK doesn't really help here - POSIX more or less lets systems pretend that file-based I/O is always non-blocking. O_NONBLOCK is around mostly for FIFOs and for its interaction with fcntl(2). However, Windows does have real asynchronous file I/O (via IOCP) and it's possible that someday Linux and other POSIX platforms will too. Any of these could be used to create a more event-driven-friendly version of File.
Yes they are useful. I hope after this series you start another one "Twisted in 5 minutes" to include some more advanced concepts ;)
jcalderone September 20, 2009 at 11:41 AM
I don't think Twisted Web itself is very deep. I'll go to the end, or so, I think. Suggestions for topics to cover are welcome, of course.
Just starting with twisted. It's a great framework and I'm learning a lot from it. Your posts make it really enjoyable!
I'm waiting for the next one...
ext_207759 September 21, 2009 at 11:16 AM
I have a server with an ad hoc, informally-specified, bug-ridden, slow implementation of half of Twisted, which I want to supplement with an internal webserver. (Preferably on the same port as its other protocol, evil as that may sound...) After looking at Twisted's documentation, I gave up and started looking at other asynchronous frameworks; you've given me hope that I can use it after all.
shimon_lor September 23, 2009 at 4:09 AM
Isn't there AIO for such purposes? Though I don't know if it's stable enough.
Another solution would be a separate worker thread which would take requests out of a queue, serve them and report the results in a loop; the Twisted reactor would then fill the queue and poll for interesting events from that worker.
Don't know if it smells too much like duct tape solution, but at least we would have more or less complete asynchronous support for all I/O bound operations.
jcalderone September 23, 2009 at 9:04 AM
The AIO APIs available for POSIX and on Linux are among the APIs which may someday be good enough to use to solve this problem, yes. :) There seems to be little interest from Linux kernel developers to actually solve this problem, though, so it may take a while.
You're absolutely right that a userspace threadpool could also be used to do this, though. Great point.
Roomate and I are excited. This is illuminating. Thanks a lot I feel indebted :D