We’ve recently discussed our UI and decided that it would be best to unify all the functionality of Concert into a single interface, so that is what we are headed towards. Here is our revised architecture document that is based on the single page scheme: https://github.com/Concert/Concert/wiki/Frontend-architecture.
We will post some screenshots / design mockups when they are available.
My problem was that I wanted to incorporate fields from Django’s user profile semantics into a model resource which was based off of the Django `User` model. I didn’t want to have to modify my client-side models to have a `User` and a `UserProfile` model just because that is what needs to be done in Django. Here is how I dealt with it for anyone who runs into similar issues:
https://gist.github.com/915948
Pretty ghetto, but works for now. Also, this wouldn’t work in the opposite direction (hydrate) currently.
Turns out there is a bug that forces me to do things this way. https://github.com/toastdriven/django-tastypie/issues/120
This past week I spent a bit of time on Concert’s documentation. We are utilizing a few different technologies for documentation generation:
You can see on our Documentation Documentation wiki page what we need to do to get pretty JavaScript documentation :)
I also documented the UI architecture a bit on our Frontend Architecture wiki page. It is missing a few diagrams and such, but an overview of the architecture ideas are expressed. We welcome your feedback!
(Cross posted from http://blog.adamgeorgiou.com)
Background
So I spent 6pm Saturday night, till 9:30am Sunday morning in a hackathon where I worked on, you guessed it, tastypie; specifically, on work concerning an open source project I’m working on, Concert.
Long story short, Concert had an issue. It’s a very javascript heavy app, which handles a lot of state information client-side. Because of this, we’re using a MVC-like javascript framework called Backbone.js, which abstracts away a lot of the interaction between our server-side django app and the client-side code.
Traditionally, django apps end up being their own flavored instances of the Model/View/Controller paradigm, but there’s an issue with such an approach when you’re dealing with the situation described above; state-manipulating client side code and state-manipulating server side code have no enforced methods of communication when said communication is handled by django views - at least not out of the box. Luckily, this exact situation is perfectly handled if we choose to stick to a RESTful style architecture, which is where tasty-pie comes into play.
(*Note: there’s nothing saying that you can’t implement a RESTfully compliant interface using nothing but django views - i.e., implementing RESTful interfaces on a per case basis for all the django model classes your app has - but this has to be all done manually for each class instance, and there’s nothing forcing you to stick to the standard’s definition. It’s also, in my opinion, a waste of time when there are so many comprehensive options out there - e.g., django-tastypie, django-piston, etc.)
The Problem
The specific issue we ran into when getting tastypie to work was how to deal with many-to-many model relationships RESTfully and intelligently (blog post).
Say you’ve got a couple of related models that look like this:
and say we’ve got two tastypie resources that map to each model.
Out of the box, tastypie will give you a handful of URLs and all the basic CRUD operations you’d expect.
A url like this “/api/audiosegment/<optional_primary_key>” can be sent POST, PUT, GET, and DELETE requests, each of which does exactly what it’s supposed to: create, update, retrieve, and remove states (objects), respectively. So let’s say we send a PUT request to “/api/tag/1” with the intention of updating what’s in the segments attribute of the already instantiated tag with primary key 1.
The PUT request itself is going to have to contain, as an arguments, the new state of tag #1; it’s going to send a name, creator, collection, and a new list of segments, including all the segments previously contained by that tag and the new one.
Where this becomes an issue for us is when we try to determine what a specific request is doing, in full. During the time the request is made, dealt with, and responded to, all any component of our app knows is that a model is being updated - whether or not a relation was was added, removed, (or any combination) from our model instance is a mystery (e.g., whether or not a segment was added or removed from Tag #1 is unknown). Not usually a big deal, unless you have actions that are dependent on knowing such information, like if you have an notification system that informs users when audio segments get tagged. Not knowing when relationships are being generated - in detail - means not having anywhere to tell your notification methods to hook into.
The Solution
The way I went around solving this was by cooking up my own recipe for nested resources. The tastypie documentation mentions them here (Tastypie Cookbook), but the specifics are left kind of vague, which is why I’m writing this. Perhaps I ended up reinventing the wheel, but at the time I couldn’t find anything that was (very) helpful.
What I wanted (and got) was the ability to hit resource urls that looked something like this:
/api/<nested_name>/<nested_primary_key>/<resource_name>/<resource_pk>
or in terms of my project:
/api/audio_segment/1/tag/
POSTing to the above resource URL creates a tag, and then relates it to the specified audio_segment. This gives me a hook for my notification creation methods, and solves all my headaches.
I also ended up implementing a slightly less restful shortcut which allowed you to POST to /api/audio_segment/1/tag/1/ with no data. This would expect for both resources to already exist and then simply create the relation between the two.
DELETE works in a similar way.
GET and PUT were left unimplemented since in those cases you could just manipulate the non-nested resources and deal with them as you wish.
The code is generic, so you can include it and have your nested resources inherit from it.
It’s all GPL, as it’s being released under the original project’s name. You can look at the in use version here: GitHub.
A slightly more reusable version is embedded bellow.
The primary functionality of Concert is centered around navigating and creating segments of uploaded audio files within a Collection. On selection of an audio segment or file, a Detail Waveform Widget and Overview Waveform Widget are populated with the waveform representation of the audio file parent of the selected segment. The Overview Waveform Widget’s purpose is to display the entirety of the selected audio file, while the Detail Waveform Widget displays the waveform scaled to a particular resolution, allowing the user to scroll left and right to view the entire file. For now, we just have one resolution level, though we hope to implement zooming in the future.
Mockup Design by Craig: https://docs.google.com/leaf?id=0By64YSWDH2DYZmNjNzZjNDktZjk4MC00NjIwLTkwN2UtNmY5YzRlNjBiMDkw&hl=en&authkey=CKX4-_UG
Since the primary use of Concert is centered around these waveform widgets, it is important to create consistent behaviors associated with these waveforms. A waveform widget should be capable of being played, paused, navigated by time, navigated by segment selection, and segmented. All these associated behaviors should function at a ‘global scope’.
It is important to grant maximum flexibility to audio segment creation. Regardless of what segment or file is selected or where the playhead is, the user should be able to create a segment (on either the overview or detail waveforms) by dragging across that waveform. Given the lack of granularity in the overview waveform widget, the ends of a segment should be readjustable after segment creation, but the natural action for readjusting the ends of a segment - dragging the endpoint - is in conflict with the drag behavior that creates a new segment - “regardless of what segment or file is selected”.
Given that dragging creates the segment, it makes sense to be able to ‘pick up’ the endpoints and drag them to a new location. A modal solution doesn’t seem reasonable - you shouldn’t have to go into ‘create mode’ to create a segment or into ‘readjustment mode’ to readjust an endpoint of a segment. For now, a partially modal solution seems to be the best - a segment’s endpoints can only be readjusted if that segment is selected (‘readjustment mode’) by dragging from a small area surrounding the segment’s endpoints. This ‘readjustment mode’ is consistent with the behavior for making other changes in a segment’s data - a name or tag, for example, can only be modified when the segment is selected. Though the implications of this behavior on segment creation behavior is that, when a segment is selected, a new segment cannot be created by starting to drag in this region surrounding the selected segment’s endpoints. A new segment can be created, however, by starting to drag anywhere else in the waveform widgets.
Another implication of drag behavior indicating segment creation (or endpoint readjustment in the specific case) is on the playhead’s movement - the tool permitting the waveform to be navigable by time. While it is common practice in audio software for a playhead to be draggable, there isn’t a particular need for that to be the case. A click on the waveform at the desired location could move the playhead to that location. We could also create yet another caveat to our drag behavior and make the playhead another location on the waveform that segment creation cannot begin at. A third solution is to extend the playhead outside of the waveform and make the extended area draggable.
If segment creation is a top priority in waveform behavior, browsing and selecting segments so as to comment on and tag them is not far behind. While the segments are listed by name in a list that can be filtered over by tag and/or search results, these segments also have associated time data: beginning and end points as well as duration - information useful to display visually. Since the overview waveform is already being used as a way to view an overview of the entire audio file, it makes sense to also display an overview of the segments associated with that audio file on or around the overview waveform widget. A visual display, however, is only so meaningful - ideally, the user could select the segment from this visual representation of it.
The problem becomes how to readily browse and select (potentially nested and overlapping) segments from a visual representation of them that doesn’t add too much visual clutter to the waveform itself (especially at high numbers of audio segments) and without interfering with the drag behavior used to create a new segment. At our disposal are both visual and behavioral elements. It might be best to just discuss ideas we’ve had and the potential problems associated with them.
Immediately, a CSS filter on the waveform (utilizing opacity to indicate nesting and overlapping segments) seemed like a good idea. While opacity levels could indicate nesting and overlapping segments though, it didn’t address segment selection of these nested and overlapping segments. A few ideas for segment selection with this particular visual representation was a click through such that the smallest segment would be selected first, a second click at the same location would select the next smallest segment, a third click the next, and so forth. This interferes with some of our options for playhead mobility (the click behavior) though. Another idea for segment selection with this visual was to display on hover a horizontal menu below the waveform listing the names of the segments the mouse was currently hovering over - selecting a segment by clicking its name in this menu. This visual representation doesn’t scale to high numbers of audio segments particularly well though, and could potentially cover the waveform in possibly nearly opaque CSS filters. Additionally, differentiation would be needed between a CSS filter denoting the selected segment and the CSS filter denoting other segments.
Another idea for the visual representation was a series of horizontal bars a few pixels tall displayed underneath the waveform and stacked to indicate overlap and nesting. These horizontal bars could be selected, and on hover, a filter could display just that one segment on the overview waveform to more clearly demonstrate where the segment was. Here though, no names are associated with browsing the segments on the overview waveform widget until one segment is selected. There is also some question as to how easy it would be to select a horizontal bar only a few pixels tall. To solve this problem, the click through idea came up again, though this time displaying the CSS filter only on the segments which exist at whatever point the mouse is at on the waveform. Another idea that came up was to make the horizontal bars expandable on hover of that piece of waveform to display the name of the segment(s) and to make them more easily selected. Both the click-through and the expandable horizontal bars seem a bit convoluted to me though.
For Concert, we’ve decided to implement a REST API so that our frontend is as decoupled from the backend as possible. Although this allows us to write less code on the backend and gives things a bit better organization, there are a few issues to deal with, primarily because we want to keep track of “events” on the system.
Our application has something called an “Event”. Much like an event on Facebook, it contains information about an action taken by a user, like tagging an audio segment for example. There are many of these events, probably just as many as there are actions on the system, so we need to ensure that the corresponding event object will get created when the actual event takes place.
There are only a few cases to consider when an actual event occurs on the system, either a model instance will be created, deleted, or updated. Since we are using the REST API, there is no Django view where model instances are created or modified but instead the API contains general code that exposes the models in a standard way.
This case is easy enough, we can just override the save/delete method of the object, and create the corresponding event there. We can also override methods of the API resource in order to create events, but we figured this is better suited to be part of the model. Here is an example: http://bit.ly/hmOnwb.
This is a bit more difficult, but still manageable. For example, in Concert, in order to begin organizing audio with a group of people, a user must first request to join that collection. The reason for this is that once a user is a member of a collection, she/he can have pretty much free reign on the material, and could delete everything if she/he wanted to be malicious.
So, when a user first requests to join a group, her/his “Request” object is created with a status of “Pending”. Once an administrator accepts or denies this request, the status is set to “Accepted” or “Denied”, and corresponding actions must happen. If the request is accepted, the user who made the request must be added to the group. To handle all of this with a single HTTP request from the client to the server we check to see how the status attribute is changing in the API code. Here is the code to illustrate:
The API code that determines how the request object has changed: http://bit.ly/h3qk0B.
The methods on the model instance that are called (depending on how it has changed): http://bit.ly/e4UOio.
And the JS that makes it happen is my favorite part :) http://bit.ly/f5mste. As you can see here, the save method is only being called on the request object, not on the collection object. That is because we can be sure that when the request is saved, the backend will know to add the user to the collection, and we don’t need an extra HTTP request to tell the backend to do this. (The closure that is sent with the save method as error_callback simply undoes the action, incase the server errors.)
Now the hard part…
This is what I’ve been struggling with for a bit and I hope someone has some insight that I don’t have. I have been discussing with the folks on #tastypie, and I’m not quite sure that anyone has a particularly good answer for me, so here is what I’ve decided is best for now.
First, an example of where we get screwed. In Concert, there are “Audio Segments” which have a list of “Tags” associated with them. To tag an audio segment, we simply add the tag object to the segment’s “tags” attribute. Here is where we run into trouble. A REST client doesn’t have any way to denote the “addition” of an attribute in a relationship. It just sends the entire array of objects for that attribute. Because we want to create a “SegmentTagged” event, in the API code we would have to go through all of the segment’s old tags, compare them to the new tags, and determine what had happened. This is bad and I don’t think anyone wants to see this occur.
So what could we do? Instead of depending on some fancy ManyToManyField, why don’t we just create a new object called “SegmentTag”, which is just a weak entity that is being created each time (which is probably what is happening under the ManyToManyField anyway). This “SegmentTag” object would have an attribute for the Segment, and an attribute for the Tag.
Well…This is annoying too. I really don’t like creating these objects to represent weak entities that have no semantic value, but instead are just extra work to overcome our technological barriers.
But wait…I already have a model with a “segment” attribute and “tag” attribute, it is called “SegmentTaggedEvent”. Therefore, my solution: Smart Events. These events, when instantiated, will check to see if the actual action associated with the event has occurred. If it has not, the event will make it happen. To illustrate, consider the segment tagging action described above. Instead of worrying about how this will happen, on the client I will just create my “SegmentTaggedEvent” object, and save it to the server. That object will check its corresponding segment, see if it has the tag, if it does not the tag will be added. This behavior can be generalized, and these smart events can be used whenever this problem would occur.
Of course there is a problem with this solution too, and it is that developers will have to be conscious of which actions need a smart event and which ones can happen directly. For now this is just how it will be, unless someone has a better suggestion for me, which I will be more than happy to consider. If I find something better, I’ll surely post it here.
Thanks for reading.
This semester we will be focusing primarily on the functionality of the user interface of Concert. Thanks to the efforts of last semester, we now have a solid foundation on which to build this interface.
Concert is built using the following technologies:
This decoupling of the UI and backend allows for future scalability and sanity.
If you are interested in the development of Concert, or helping out this semester, we will be listing our goals and tasks here (for now).
This semester, we developed much of the infrastructure for Concert in a robust, modular way. Adam and I (Colin) will discuss the various aspects of the application that we have been developing since our last presentation at RCOS.
You can see our slides here: https://docs.google.com/present/edit?id=0AVuQWmnCzsgiZGRoeHhqMzRfMjNneGhyMmpmeA&hl=en&authkey=CMa725UD
And watch our presentation here: http://www.youtube.com/watch?v=0-oGVCp5MPA
As always, we strive to maintain consistency and modularity throughout the application. This goal is especially important on the client-side code which will undoubtedly change dramatically in the future.
An example of where this modularity is important is something like loading notifications. If a panel on the user interface needs to have a loading notification, this clearly must not be coded for each panel, but instead just once and re-used for every panel.
Another example where modularity is useful is when initializing the functionality on a page. Not only can the code for the UI elements on the page be object-oriented, but the page itself can be as well. This allows us to send a string from the server which informs the JavaScript of which page we are on, and from there we can instantiate the proper Page object to initialize all functionality on the page.
We have not done too much styling work this semester, as we are waiting until we are happy with a design iteration. We are fairly certain how the interface will be laid out in general (see mockups here), however, so the styling that we have been doing has been to match the layout to these mockups.
The CSS3 box layout model is quite intuitive, and makes many layout configurations extremely easy. The box layout is based on a “flex” model where you can define the ratio between the width/height of a box and its container, and all of the boxes will expand to fit the container based on these ratios automatically. This often eliminates the need to use the float: property, and in turn eliminates the need for clear: both; everywhere. For more information, here is a great resource from Mozilla: http://mzl.la/ijPwMj
We have commented the client-side code to conform to the JSDoc toolkit, which is a JavaScript documentation generator (among other things). Some features of the code have not been recognized automatically, such as class inheritance, but we will keep these documentation standards in mind when developing the client-side code.
The client-side functionality has been developing somewhat slowly as the framework is being set up, but the pace is beginning to increase. Setting up such a robust framework ensures that adding new features and functionality in the user interface will be seamless in the future.
In Concert, a user can join a “Collection” which is a group of users and audio files/segments. This forms the basis of Concert and is the concept that the user will first grasp when beginning to use the application. It is for this reason that the experiences associated with requesting to join a collection or creating a collection must be as smooth as possible.
I have been reading about Backbone.js for some time, the buzz about this framework has been difficult to ignore. Recently I recognized the benefit of using Backbone.js (http://bit.ly/hNzNO7), and decided that now is the proper time to integrate it into our application. It is for that reason that I have been re-factoring our code a bit in order to make the functionality that we have implemented thus far work well with Backbone.
Backbone is an extremely powerful framework and provides a few abstractions that allow robust organization of client-side code. I will discuss two of these abstractions, one is the Backbone.Model, and the other is Backbone.View.
Backbone’s “Model” class allows object-oriented access to the data that is represented on your user interface. Much like an ORM which abstracts away the database interaction from your model, Backbone.Model abstracts away the interaction with the server entirely. This allows for code such as the following:
var book = new Backbone.Model({
title: "The Rough Riders",
author: "Theodore Roosevelt"
});
book.save();
When Book.save(); is executed, the server is informed of the new Book object, and there is no need to keep track of where in the JavaScript objects are changing.
Backbone’s “View” abstraction allows a complete decoupling between the data that is represented on the user interface, and the actual data itself. Consider an application where you have a set of data, and it must be displayed in two different ways on the UI. Under a typical architecture, when this data is changed, we would need to go to both of the places where the data is represented and change the representation. With Backbone.View, this is entirely abstracted away and the “view” object, which is any sort of UI component that depends on some data, simply watches the collection of data that may change. When this data is changed, the view elements update themselves based on their render() method, which is ultimately the only thing that needs to be coded.
A single data collection, that is being watched by two different visualizations of the same data.
Consider in our application, where we have a set of “Collections” that the user is a member of. These collections must be displayed on the UI in both the “Manage Collections Panel”, as well as the “Organize Selector”, which allows the user to select a collection to organize.
In our case, we have two views, Manage Collections Panel and an Organize Selector that are watching this CollectionSet.
When something like the following code is run to load the CollectionSet, the two components of the UI will look as follows:
var userCollections = new CollectionSet([
{
name: "[ ]",
userCount: 2
}
]);
userCollections.save();
The Manage Collections Panel when one collection is added to model.
The Organize Selector when one collection is added to the model.
Using Backbone, both of these “views” will be watching the userCollections “CollectionSet”, and when this data is updated, both representations of this data on the UI will automatically be updated based on their render() method:
userCollections.add({
name: "colin group",
userCount: 1
}).save();
The Manage Collections Panel when another collection is added to the set.
The Organize Selector when another collection is added to the set.
This is not all Backbone.js has to offer, and I have yet to look into the other aspects of the framework. It is obvious that utilizing Backbone.js will take a bit of time to set up, but once it is up and running in our application it will save an incredible amount of time.
In Concert we have the concept of an “event” which is an object which tracks all the various actions taken by members of the application. Whether it be tagging a segment, joining a collection, uploading audio, or anything else, Concert’s back end automatically creates an event object for the corresponding action taken. The point of this process is to allow for a feed of all actions taken - within a specific collection - to be read and followed by other members of the collection.
The concept isn’t exactly novel, as it’s pretty much the same idea as a Facebook notification - people want to know what their friends and members of their communities are up to.
The purpose in Concert is pretty much the same. Events notify users when things they might be interested in happen within the application. Say, for example, my band-mate uploads an audio file that I’ve been waiting for. I can log into Concert and whether or not they uploaded the file will be immediately apparent to me.
One reason that things are moving somewhat slowly on this project is that we are dedicated to implementing things the cleanest, most modular way. Another reason is that we are RPI students, and have to do other stuff.
With Django, you can implement templates in such a way that you eliminate any repetition in the templates, even if you need most of templateX in templateY.
The point of templates is so that you DO NOT repeat markup code, even if you need this markup code more than once. Here is a simple example where a “partial” template is being created in Django, so that if we need to change how a “large_icon_button” on the UI looks, we only need to change the markup in one place, and all of the “large_icon_buttons” will change.
e8804b73b41f6f787e9bd16ec7fa75102f1700f1#L7R1
Here is where it is used:
e8804b73b41f6f787e9bd16ec7fa75102f1700f1#L5R19
As you can see, I have removed all redundant html, and consolidated it into one template for this type of button.
Now for the really good stuff: template inheritance.
In Concert, we have a concept of a “Panel” on the UI. A panel will contain “widgets” and “buttons” that do different things, and represent different types of audio/social data. Most panels will have a similar base functionality, such as the ability to display a “loading” notification. Therefore, it makes perfect sense to design the templates using inheritance. Here we have the “panel” template, with its associated “partial controller”:
db1ea9961d5292a82ad91632916fd3caff1f6e9e#L2R1
It is used on this line:
db1ea9961d5292a82ad91632916fd3caff1f6e9e#L0R7
The great thing about template inheritance is that now we don’t need to create an entirely new template for our “audiolist_panel”, just because it needs a different title:
db1ea9961d5292a82ad91632916fd3caff1f6e9e#L1R1
Now, when we need to add a per-panel loading notification, it will be extremely easy to add the required structure to every panel. Our JS classes are mirroring this inheritance structure, and everything makes sense.
This type of thing is especially helpful for us because we are developing software at the same time as the UI is being designed. It is very difficult to know exactly how the markup for the UI will be structured, but by making design decisions like this, we can at least ensure that it will be very easy to implement the structure once it is determined.
Personally, I do not like how I need to create a “parital controller” function like this:
db1ea9961d5292a82ad91632916fd3caff1f6e9e#L3R5
For every “partial” template that I need. Especially since we don’t need any functionality outside the template that needs to be executed in this python “partial controller”. Other than that, things are pretty smooth.
EDIT:
Adam has fixed this worry by simply passing in the template url to the parent “partial controller”, so the panel controller takes an argument, which defines which child template to use. Simple and easy:
a93d168174a0fe682a5c3fdfb5a23a00ecde5bfe/concertapp/templatetags/panels.py
So Colin and I (Adam) have been spending most of our free time in the library coding as a pair. I’ve never been one for prescribing to any one project methodology, but I gotta say, this entire “Extreme Programming” thing has got something going for it. Granted we’re not on a single terminal, and most of the time we’re working on pretty independent parts of Concert, but when there is a conundrum to contemplate, it’s nice having another guy there to look over your shoulder and point out the lacking of quotes in some trivial piece of code.
Anyway, as far as updates on the project are concerned… Colin and I have been working on pretty much a giant cleanup. Concert v1 - what we showed in our first presentation - was mostly an introduction to django for the former dev team, and even though we wanted to build something worth building, a lot of the focus was on figuring out django semantics and best practices. Moving out of the old and into the new, a lot of functionality… scratch that, most of the functionality of the site has been either deprecated or completely re-written.
Following our initial plan, I’ve so far…
Colin has been working on some extremely robust UI architecture and development, ensuring that everything is entirely object oriented and modular. There have been developments in client-side templating as well, so modifications like internationalization will be easy in the future.
Chris has been banging out some mockups which he’s currently working on marking up in HTML and SASS.
