Almost two years ago I was in a rather unlikely situation in that I was running a software engineering department containing both a C# team and a Python team. The Python group was focused on building scientific computing and NLP-type applications, whereas the C# team was focused on building web applications.

A few of us Python folks in the department had already started playing around with Django–building internal web applications and projects outside of work. It did not take long for us to realize the power of Django and how quickly we were able to produce high-quality applications with little effort. This was my (strong) impression, but in order to propose a corporate platform shift I was going to need some data to support my claims.

It slowly dawned on me that I had a perfect test bed. Here we had two teams using different technology stacks within the same department. The same department. That means they shared the same development processes, project management tools, quality control measures, defect management processes. Everything was the same between these groups except for the technologies. Perfect! So like any good manager I turned my teams into unwitting guinea pigs.

The Hypothesis

We can accomplish more with Python + Django than with C# + ASP.NET given the same amount of time without sacrificing quality

Measuring Productivity

For the sake of this study, I defined productivity as a normalized team velocity: how many story points were completed / developer / week. I record the normalized team velocity for each team’s sprint for later analysis.

For those of you unfamiliar with the concept story points I highly recommend Mike Cohn’s Agile Estimation and Planning.

WAIT! You can’t compare story points between teams!

I hear this a lot. Yes, you can. The problem is that most people do not bother creating a common scale or continually calibrate their estimations (within or between groups). Generally, it’s way more work than most groups need to deal with and it doesn’t deliver much utility to most groups so it isn’t often discussed or practiced.

The methods described below should outline the additional calibration work that was performed to ensure a common estimation scale between the two teams.

Methods

Both teams continued business as usual working on projects in parallel. Each sprint consisted of 3-4 developers. It is worth noting that Team ASP.NET did not make use of MS MVC Framework, but they did use Linq-to-SQL for its ORMy powers.

Special care was taken to maintain linkage between the two team’s effort estimates. During sprint planning, each team would use a common story point calibration reference when making estimates. In order to detect any potential deviations in calibration, during several planning poker sessions I included stories that had already been estimated during previous sprints or by the other team; no significant deviations were found.

At the end of each sprint I would calculate the normalized developer velocity ( # of completed story points / developer / week ). These values were recorded for both teams. It should be noted that only Django-based sprints were used in analysis for Team Python.

I recorded results for approximately 6 months.

Results

Normalized Developer Velocities: C# + ASP.NET and Python + Django

The above histogram shows the distribution of normalized velocities associated with each completed sprint. The table below summarizes the distribution of velocities associated each team.

The distribution of velocities between the two samples are similarly shaped, but have clear differences in their mean. The average velocity of a  C#/ASP.NET developer was found to be 5.8 story points/week. A Python/Django developer has an average velocity of 11.6 story points/week. Independent t-tests reveal these differences as being statistically significant (t(15) = 4.19, p<7.8e-4).

Discussions and Conclusion

Given our development processes we found the average productivity of a single Django developer to be equivalent to the output generated by two C# ASP.NET developers. Given equal-sized teams, Django allowed our developers to be twice as productive as our ASP.NET team.

I suspect these results may actually reflect a lower bound of the productivity differences. It should be noted that about half of the Team Python developers, while fluent in Python, had not used Django before. They quickly learned Django, but it is possible this fluency disparity may have caused an unintended bias in results–handicapping overall Django velocity.

Epilogue

The productivity differences quantified by our findings were then included as part of an overall rationale to shift web-based development platforms. Along with overall velocity differences, the costs associated with maintaining each environment were considered: OS licensing and database licensing for development and production environments, as well as costs associated with development tools. I’m happy to say we are now a Python and Django shop.

Updated:

Several good questions over at Hacker News

Protosprout ( http://protosprout.com ) is a NC-based company founded by Justis Peters that is looking to bring local entrepreneurs together and foster an energized, start-up community. I think it was a successful first meeting. Justis pulled folks together and lead some interesting and informal discussions including how the community could work together to provide support and value to one another. Ideas included things like skill and service bartering, community forums, general networking, official Protosprout mentorship and startup programs. A local skill bartering system would be great for a lot of startups ( ala programmer meets designer ) as would a tighter knit community for bouncing ideas off of or even testing your prototypes.

Part of the evening included time for pitching your startups and Q&A. Even Wayne Sutton got up and pitched an idea. I’ll take a moment to mention one really cool company that pitched: Durham TechShop ( http://www.durham.techshop.ws )

The TechShop will be a a DIY workshop with a slew of tools and machines for any number of tasks ( electronics, woodworking, plastics, machining, welding, 3d printing, even blacksmithing! ). They’ll offer classes so that you can get certified to use the various tools and then you’re off to create whatever you want. I am really excited to see this in action, so all you folks with checkbooks contact Scott Saxon over at the TechShop and get this puppy funded!

I met a lot of interesting people that night and look forward to attending the next Protosprout meeting. You should check it out over at http://protosprout.com . I think Justis is doing a great job getting this organized.

You can also follow Protosprout on Twitter  ( http://twitter.com/protosprout )

Something Big is a Brewing

I think NC ALTA has really started a powerful ball rolling by creating a common focus on which a number of industries and research disciplines are beginning to converge. To demonstrate this let me list areas represented by the some of the people I encountered at the conference:

• Game engine developers
• Serious game developers
• Developers of virtual worlds including frameworks like Second Life or Croquet. This also includes the corporate facet aimed at facilitating remote work and collaboration via a virtualized workplace.
• Researchers dealing with immersive visualizations and virtual experiences. This includes some of the cool immersive environments hosted by RENCI.
• Second (or third?) generation of e-learning companies and their interest in assessment, monitoring, and integration into larger learning management systems.
• Educators interested in getting these technologies out in the schools and in the hands of students. As well as educators interested in seeing proper learning theory incorporated into program design.

Now imagine the intersection of all (or some) of these groups.

Parallel Tinkering and Research

I attended one birds-of-a-feather discussion that dealt with educational games and simulations in higher ed. I was surprised at the number of professors and students who have already begun building, deploying, and testing their educational games within their universities. Some folks met up after the round table and shared war stories, common hang-ups, and I think even identified some future collaborations. Exciting stuff!

One thing that came up a number of times during that session was the fact that we did have a number of independent development efforts going on. Essentially, each project required the development of a slew of management frameworks, authentication, integration with LMSs, middleware, etc. Everyone needed these same components and so everyone wrote their own.

Libby Evans of UNC and several others identified the need for some common, modular solutions to these problems. Establishing a framework of common solutions would allow researchers to focus on the interesting problems and it would encourage compatibility and collaboration. So it seems like a nice next step would be to look at the number of developed solutions and start distilling out some design patterns.

The Efficacy of Games in Education

Do games actually work in education? can they be used as a tool to teach or explore? I am particularly interested in this topic and have begun looking at this in terms of math games. Marrilea Mayo of the Kauffman Foundation was kind enough to share a tremendous amount of her own findings from reviewing the disparate literature.

Many of us have heard a number of positive anecdotal (or underpowered, small n) successes, positive pilots, but few full-blown psychometric studies on the efficacy of games in learning. In fact, I have not found any commercially available shrink-wrapped math games with associated efficacy studies. I have seen products claiming they are “scientifically-based”, because the product is designed around a number of accepted practices not that their actual efficacies were tested.

While I have personally been focused on the efficacy in K-12 and higher ed, there were also a number of discussions dealing with the same concepts in corporate space. How do we measure the success of our corporate training programs? ROI? So, it seems like efficacy and the ability to measure and monitor success is a common theme and something we need to start incorporating into our designs. Luckily, I think this is solvable once we begin embedding the means to track psychometrically-valid metrics.

So, I walked away from the ALT Summit pretty excited that we had begun a conversation that I think will develop into something revolutionary down the road. I encourage you to keep tuned in and check out NC ALTA.