Python + Django vs. C# + ASP.NET: Productivity Showdown

People are often asking me how and why my department shifted from an ASP.NET environment to Django. I’ve finally gotten around to writing about the process leading up to our decision. I hope people out there find it useful in their own development groups and discussions.

Almost two years ago I was in a rather unlikely situation in that I was running a software engineering department containing both a C# team and a Python team. The Python group was focused on building scientific computing and NLP-type applications, whereas the C# team was focused on building web applications.

A few of us Python folks in the department had already started playing around with Django–building internal web applications and projects outside of work. It did not take long for us to realize the power of Django and how quickly we were able to produce high-quality applications with little effort. This was my (strong) impression, but in order to propose a corporate platform shift I was going to need some data to support my claims.

It slowly dawned on me that I had a perfect test bed. Here we had two teams using different technology stacks within the same department. The same department. That means they shared the same development processes, project management tools, quality control measures, defect management processes. Everything was the same between these groups except for the technologies. Perfect! So like any good manager I turned my teams into unwitting guinea pigs.

The Hypothesis

We can accomplish more with Python + Django than with C# + ASP.NET given the same amount of time without sacrificing quality

Measuring Productivity

For the sake of this study, I defined productivity as a normalized team velocity: how many story points were completed / developer / week. I record the normalized team velocity for each team’s sprint for later analysis.

For those of you unfamiliar with the concept story points I highly recommend Mike Cohn’s Agile Estimation and Planning.

WAIT! You can’t compare story points between teams!

I hear this a lot. Yes, you can. The problem is that most people do not bother creating a common scale or continually calibrate their estimations (within or between groups). Generally, it’s way more work than most groups need to deal with and it doesn’t deliver much utility to most groups so it isn’t often discussed or practiced.

The methods described below should outline the additional calibration work that was performed to ensure a common estimation scale between the two teams.

Methods

Both teams continued business as usual working on projects in parallel. Each sprint consisted of 3-4 developers. It is worth noting that Team ASP.NET did not make use of MS MVC Framework, but they did use Linq-to-SQL for its ORMy powers.

Special care was taken to maintain linkage between the two team’s effort estimates. During sprint planning, each team would use a common story point calibration reference when making estimates. In order to detect any potential deviations in calibration, during several planning poker sessions I included stories that had already been estimated during previous sprints or by the other team; no significant deviations were found.

At the end of each sprint I would calculate the normalized developer velocity ( # of completed story points / developer / week ). These values were recorded for both teams. It should be noted that only Django-based sprints were used in analysis for Team Python.

I recorded results for approximately 6 months.

Results

Django ASP Histo

Normalized Developer Velocities: C# + ASP.NET and Python + Django

The above histogram shows the distribution of normalized velocities associated with each completed sprint. The table below summarizes the distribution of velocities associated each team.

Summary statistics of each team’s normalized developer velocities

Units: story points / developer / weekC#/ASP.NETPython/Django
mean5.811.6
stdev2.92.7
min.38.5
max9.315.8

The distribution of velocities between the two samples are similarly shaped, but have clear differences in their mean. The average velocity of a C#/ASP.NET developer was found to be 5.8 story points/week. A Python/Django developer has an average velocity of 11.6 story points/week. Independent t-tests reveal these differences as being statistically significant (t(15) = 4.19, p<7.8e-4).

Discussions and Conclusion

Given our development processes we found the average productivity of a single Django developer to be equivalent to the output generated by two C# ASP.NET developers. Given equal-sized teams, Django allowed our developers to be twice as productive as our ASP.NET team.

I suspect these results may actually reflect a lower bound of the productivity differences. It should be noted that about half of the Team Python developers, while fluent in Python, had not used Django before. They quickly learned Django, but it is possible this fluency disparity may have caused an unintended bias in results–handicapping overall Django velocity.

Epilogue

The productivity differences quantified by our findings were then included as part of an overall rationale to shift web-based development platforms. Along with overall velocity differences, the costs associated with maintaining each environment were considered: OS licensing and database licensing for development and production environments, as well as costs associated with development tools. I’m happy to say we are now a Python and Django shop.

Updated:

Several good questions over at Hacker News

14 responses to “Python + Django vs. C# + ASP.NET: Productivity Showdown

  1. Nice story — good to have some definite stats rather than anecdotals. Still — I misread the headline as “… Productivity SLOWdown” !

  2. Still the biggest pro IMO for Django is that it runs on the vastly cheaper linux based hosting solutions as opposed to getting M$ vendor lock-in syndrome.

  3. Still the biggest pro IMO for Django is that it runs on the vastly cheaper linux based hosting solutions as opposed to getting M$ vendor lock-in syndrome. Good to see some real hard data on team velocity as well. Good job for your quantitative efforts to make an unbiased decision. Thanks for the read.

  4. Have you considered writing this up as a paper and submitting it to a software engineering conference like ICSE (which has a Software Engineering in Practice track) or ESEM?

  5. Is there any way you can control for the types of developers who do .Net and the types who do Django? There may be some systemic difference between the capabalities of these two communites which might account for the results. Just a thought.

  6. What about maintenance? Code is written once and rewritten one thousand times. I program in Java and Python and I’m a big evangelist of Python but… having something like static typing and a good IDE like Eclipse let you refactor and redesign at full speed being in a team of more than one person.Every comparative I see about Python vs. X is always talking about implementation (first implementation), but there is really few information about really big Python codebase and its maintenance.Do you have some info about that?

  7. @lorin Thanks for the encouragement. It’s something I have been meaning to do. I had considered ICSE, but hadn’t heard of ESEM. I’ll check it out.@jeremy Great point. Generally the developer and the technology are confounded into a sort of developer-tool construct that is hard to tease apart. So maybe there are some inherent personality traits that draw each group of developers to their preferred toolset that in turn affects productivity? I would not be surprised if there was something to this. But, for practical purposes I’m not sure if it matters since a “Java developer” is most likely going to be hired to do Java work.That being said, I did have a couple of switch hitters that made an appearance in both Team .NET and Team Django. There isn’t enough data to make any statistical claims, but it seemed like the technology was still the best predictor of productivity.@José We’ve had teams developing in Python for over eight years now on many sizeable projects. The natural readability of Python has been a important factor in promoting the longevity and maintainability of our codebases. I would also say that comprehensive test coverage is a much more significant factor in predicting the maintainability of a codebase regardless of the language you’ve chosen.As far as tools go I agree they are important, but I don’t think that is necessarily an issue. You may prefer Eclipse, others might feel more at home with emacs and rope, and others WingIDE. I will say that I did not control for editor choice in the original study. Everyone used whatever editor stack they preferred. Team .Net: Visual Studio; Team Django: vim, emacs, and Wing IDE.I don’t have any longer term parallel statistics, but I would also be very interested in seeing figures on maintenance. It would probably make sense to shift the metrics from velocity to things like defect rate, time to patch, etc..

  8. @kurt Don’t mistake my words, I’m an «emacs poweruser» (if there is something like that). But I always use a test for maintenance: «change under stress».A language, framework, ide or environment that let you make safe changes under stress (a customer calling and shouting, machines that fail, people running) is a big win in a company.In our my java projects we use unit testing( TestNG) and test coverage tools as we do with Python.The strange thing is that I feel as productive with Java+Eclipse+TestNG+iBatis+Wicket+Jetty as in Python (but as you can see I’ve to compose my own framework in Java). I remember reading something Guido said about the performance of Python programmers vs C++ programmers in Google, he said that it was surprise to him that the C++ programmers were as productive as the Python ones, so he think that it’s the person, not the language, who is productive.I hate Java complexity (that you can reduce in base of the tools you use). And, please, don’t confuse my words, I love Python. In fact I write Python articles for a magazine since 4 years. It’s just that I always remember the phrase: «there is no thing like a free lunch».What do we trade for when we use Python instead of Java or C# or any other system?People is always praising the good things of Python, but in Software it’s impossible to find people without biases that can tell you about hard data on the problems of every language or framework. Every company try to hides this data because it’s a competitive advantage to let other use the wrong tools, and I think that in fact many companies try to flood the web with disinformation and FUDs.

  9. @Ilya “It is worth noting that Team ASP.NET did not make use of MS MVC Framework, but they did use Linq-to-SQL for its ORMy powers.” I wouldn’t call MVC a direct analog, but certainly the closest. At the time of the study the MVC framework was not mature enough for our consideration, but ORMy functionality was available to Team ASP.NET.@José I think we’re in agreement on a number of your points.I would absolutely agree with you that the programmer is more important than the tool. If you have a star programmer they’ll make things work in any environment. This study was out to determine…all things being the same, which toolset granted greater productivity. Both teams shared comparable experience, same hiring managers, similar projects, same development processes, etc. The major difference was the toolset.This study was not interested in the feelings or perceived productivity of developers by the developers themselves or management. A driving goal was to gather empirical data for the purpose of planning and refining our department’s capabilities — evidence-based software engineering. This resulted in one piece of data gathered under a known set of conditions and constraints. Serving recommendation: use multiple datapoints before making any big decisions.I agree that finding good, public, real-world metrics associated topics in software engineering are tough to come by. I hope this study serves to encourage others to gather and publish similar metrics.Thanks for the thoughts and discussion!-Kurt

  10. No estoy del todo acuerdo….Tal vez no eh probado Python + Django En su totalidad.Pero si C# + Asp.Net, asi como Ruby, etc.Y bueno… en proyectos grandes.. Resulta eficaz. Ademas. lo que se ah hablado es una parte del Mundo de Ing. de Software. y bueno.Tendré que investigar / analizar y probar Django.Saludos..

  11. Comparison of ASP.NET MVC vs ASP.NET will bring even bigger productivity boost, because of your team’s existing C# skills and much better development tools.But you have probably already switched your teams back to C#

Leave a Reply

Your email address will not be published. Required fields are marked *