Tendril Frame-based Network Connection Tracker
Tendril is a network communication library based on two main features:
it is based on sending and receiving frames, and it tracks the state
of an abstract connection as defined by the application. Tendril is
designed to be easy to use: creating an application requires
subclassing the Application
class and providing an implementation
for the recv_frame() method; then get a TendrilManager
class
instance and start it, and Tendril manages the rest.
Tendril Concepts
Frames
Tendril is based on the concept of passing around frames or packets of data. The fact is, most network protocols are based on sending and receiving frames; for instance, in the SMTP protocol used for sending email, the sender will start off sending the frame "MAIL FROM email@example.com" followed by a line termination sequence (a carriage return followed by a newline). The SMTP server will then respond with another frame acknowledging the "MAIL FROM" frame, and that frame will also end with a line termination sequence. Thus, even though SMTP is defined on top of the TCP protocol, which provides an undivided stream of data between the client and server, a framing boundary is imposed upon it--in this case, the carriage return followed by a newline that terminates each frame.
Tendril includes the concept of framers. A framer is nothing more
than a subclass of Framer
which has one method which extracts a
single frame from the stream of undifferentiated data, and another
method which converts a frame into an appropriate representation. In
the case of the SMTP protocol exchange above, the frameify()
method finds each line terminated by the carriage return-newline pair,
strips off those characters, and returns just the frame. In the same
way, the corresponding streamify()
method takes the frame and
appends a carriage return-newline pair.
For text-based protocols such as SMTP, this may seem like overkill. However, for binary-based protocols, a lot of code is dedicated to determining the boundaries between frames, and in some cases even decoding the frame. Tendril's concept of a framer for a connection enables the framing logic to be isolated from the rest of the application, and even reused: Tendril comes with several pre-built framers, including framers designed to work with a text-based protocol such as SMTP.
Another important advantage of the framer concept is the ability to switch between framers as needed. Taking again the example of the SMTP protocol--the actual email data is transferred to the server by the client first sending a "DATA" frame; the server responds indicating that it is ready to begin receiving the message data, and then the client simply sends the message data, ending it with a line containing only a single period ("."). In this case, an SMTP server application based on Tendril may wish to receive the message data as a single frame; it can do this by creating a framer which buffers stream data until it sees that ending sentinel (the period on a line by itself), then returns the whole message as a single frame. Once the server receives the "DATA" frame from the client, all it has to do is temporarily switch out the framer in use for the receiving side of the connection, then switch it back to the standard line-based framer once it has received the message frame.
Tendril allows for different framers to be used on the receiving side and sending side of the connection. This could be used in a case like the SMTP server example cited above, where the server still wishes to send line-oriented frames to the client, even while buffering a message data frame. In addition, although the provided framers deal with byte data, Tendril itself treats the frames as opaque; applications can use this to build a framer that additionally parses a given frame into a class object that the rest of the application then processes as necessary.
Connection Tracking
Tendril is also based on the concept of tracking connection state. For connection-oriented protocols such as TCP, obviously, this is not a big problem; however, Tendril is also designed to support connectionless protocols such as UDP, where some applications need to manage state information relevant to a given exchange. As an admittedly contrived example, consider DNS, which is based on UDP. A client of the DNS system will send a request to a DNS server over UDP; when a response is received from that DNS server, the connection state information tracked by Tendril can help connect that response with the appropriate request, ensuring that the response goes to the right place.
This connection state tracking is primarily intended to assist applications which desire to be available over both connection-oriented protocols such as TCP and over connectionless protocols such as UDP. Although Tendril does not address reliability or frame ordering, its connection state tracking eases the implementation of an application which utilizes both types of protocols.
Extensibility
Careful readers may have noticed the use of the terms, "such as TCP"
and "such as UDP." Although Tendril only has built-in support for TCP
and UDP connections, it is possible to extend Tendril to support other
protocols. All that is required is to create subclasses of
Tendril
(representing an individual connection) and of
TendrilManager
(which accepts and creates connections and manages
any necessary socket data flows), and to register the
TendrilManager
subclasses as pkg_resources
entry points under
the tendril.manager
namespace. See the setup.py
for Tendril
for an example of how this may be done.
In addition to allowing Tendril to support protocols other than TCP
and UDP, it is also possible to implement new framers by subclassing
the Framer
class. (Note: as Tendril deals with Framer
objects, it is not necessary to register these framers using
pkg_resources
entry points.) Objects of these classes may then
simply be assigned to the appropriate framers
attribute on the
Tendril
instance representing the connection.
Advanced Interfaces
Tendril also provides an advanced interface that allows a given raw socket to be "wrapped." Using this feature, an ordinary TCP socket could be converted into an SSL socket. Other uses for this interface are possible, such as setting socket options for the socket. Tendril also provides an interface to allow multiple of these wrapper functions to be called in a given order.
Standard Usage
The first step in using Tendril is to define an application by
subclassing the Application
class. (Subclassing is not strictly
necessary--Tendril uses Python's standard abc
package for defining
abstract base classes--but using subclassing will pull in a few
helpful and/or required methods.) The subclass need merely implement
the recv_frame() method, which will be called when a frame is
received. The Application
subclass constructor itself can be the
acceptor to be used by Tendril (more on acceptors in a moment).
Once the Application
subclass has been created, the developer then
needs to get a TendrilManager
instance, using the
get_manager()
factory function. The exact call to
get_manager()
depends on the needs; for making outgoing
connections, simply calling get_manager("tcp")
is sufficient. If
listening on a port or making an outgoing connection from a specific
address and/or port is desired, the second argument to
get_manager()
may be a tuple of the desired local IP address and
the port number (i.e., ("127.0.0.1", 80)
).
All managers must be started, and get_manager()
does not start the
manager by itself. Check the manager's running
attribute to see
if the manager is already running, and if it is not, call its
start()
method. To accept connections, pass start()
the
acceptor (usually the Application
subclass). The start()
method also accepts a wrapper, which will be called with the
listening socket when it is created.
If, instead of accepting connections (as a server would do), the
desire is to make outgoing connections, simply call start()
with
no arguments, then call the connect()
method of the manager. This
method takes the target of the connection (i.e., the IP address and
port number, as a tuple) and the acceptor. (It also has an optional
wrapper, which will be called with the outgoing socket just prior to
initiating the connection.)
Acceptors
An acceptor is simply a callable taking a single argument--the
Tendril
instance representing the connection--and returning an
instance of a subclass of Application
, which will be assigned to
the application
attribute of the Tendril
instance. The
acceptor initializes the application; it also has the opportunity to
manipulate that Tendril
, such as setting framers, calling the
Tendril
instance's wrap()
method, or simply closing the
connection.
Although the TendrilManager
does not provide the opportunity to
pass arguments to the acceptor, it is certainly possible to do so.
The standard Python functools.partial()
is one obvious interface,
but Tendril additionally provides its own TendrilPartial
utility;
the advantage of TendrilPartial
is that the positional argument
passed to the acceptor--the Tendril
instance--will be the first
positional argument, rather than the last one, as would be the case
with functools.partial()
.
Wrappers
A wrapper is simply a callable again taking a single argument--in
this case, the socket object--and returning a wrapped version of that
argument; that wrapped version of the socket will then be used in
subsequent network calls. A wrapper which manipulates socket options
can simply return the socket object which was passed in, while one
which performs SSL encapsulation can return the SSL wrapper. Again,
although there is no opportunity to pass arguments to the wrapper in a
manager start()
or connect()
call (or a Tendril
object's
wrap()
call), functools.partial()
or Tendril's
TendrilPartial
utility can be used. In particular, in conjunction
with TendrilPartial
, the ssl.wrap_socket()
call can be used as
a socket wrapper directly, enabling an SSL connection to be set up
easily.
Of course, it may be necessary to perform multiple "wrapping"
activities on a connection, such as setting socket options followed by
wrapping the socket in an SSL connection. For this case, Tendril
provides the WrapperChain
; it can be initialized in the same way
that TendrilPartial
is, but additional wrappers can be added by
calling the chain()
method; when called, the WrapperChain
object will call each wrapper in the order defined, returning the
final wrapped socket in the end.