Your Trusty SVD Tool

SVD is one of those tools that every machine learning practitioner and computational geek will pull out at some time or another. It’s a powerful matrix factorization technique that allows you to get at the matrix’s eigenvectors and eigenvalues. One of reason it tends to be used so often is the fact that it can be used on those pesky M x N matrices, which us data junkies tend to generate.

For most small problems I can just use scipy and numpy’s svd and never give it a second thought. LAPACK’s suite of SVD routines power the svd functions of scipy, numpy, and MATLAB among others. It is developed for dense matrices and processes them in their entirety. What happens when you start dealing with problems in high-dimensional space? Those dense representations and full processing are expensive. So, when your problem space is better suited for sparse matrices you tend to run into not enough memory, non-convergence…no SVD.

At the time I was considering a problem that would be well-suited for incremental training, meaning I did not want to have to rerun the entire dataset through SVD after adding a small set of new data; GHA lets you avoid that sort of inconvenience and approximates the same outcome (as far as my problem was concerned).

GHAPACK

GHAPACK is written by Genevieve Gorrell and based on her work using GHAs to perform Latent Semantic Analysis (LSA).

“Offline” Calculations

My first order of business upon joining the project was to get the offline training working. This allows you to compute a pseudo-SVD based on a massive matrix without having to load the whole thing into memory.  No more out-of-memory segfaults. Now, you’re just limited by the resiliency of your hardware. This is now working.

Memory Management

I addressed a few memory leaks, but will likely do some restructuring to optimize memory management.

Resource Library

I would like the core of the GHA magic to be extracted into a library that others could embed in their own projects. So, I intend to move core functionality into a library and restructure the existing apps into commandline tools that utilize those libs.

Performance

GHA, off-the-bat, is not known for its speed compared to some other eigen decomposition approaches. Besides that, there is room for some major gains in performance. Let’s see what we can squeeze out of GHAPACK and perhaps lean on things like BLAS.

Testing Framework

A testing framework that objectively keeps track of performance gains, while ensuring computational integrity through unit testing always makes refactoring work that much less stressful.

Lots of work to do and a hearty thanks to Professor Gorrell for letting me join her efforts.

Other SVD Resources

There are other SVD libraries out there that will carry you farther if SVD is what you really want and not necessarily the means to an end.

ScaLAPACK has parallel SVD code, which creams LAPACK’s performance when you have access to multiple cores and/or MPI. ARPACK and SVDPACK both offer Lanczos-based SVD solutions for sparse matrices with ARPACK being well-suited for parallel processing.

Ken Schwaber replied to Jason’s article:

When Scrum is mis-used, the results are iterative, incrementa​l death marches. This is simply waterfall done more often without any of the risk mitigation techniques or people and engineerin​g practices needed to make Scrum work.

The article also sparked Bob Martin to write an essay “Dirty Rotten ScrumDrels” in response to some of the Scrum scapegoating that has been going on recently. Check out the comments for some Uncle Bob-Shore dialog.

Things like self-reflection and process and work reviews are all practices that people naturally adopt to improve themselves. I think successful developers tend to do this anyways in order to stay ahead of the curve. When you have mediocre teams flailing about who are looking for a silver bullet and turn to things like Scrum who don’t already do this sort of thing, it is easy for them to brush that off as a triviality; if they haven’t already seen the value of it, it just gets lost in the noise.

I share Mr. Shore’s frustration in seeing Agile and Scrum being blamed for the shortcomings of teams and improper implementations, when you’ve seen the real thing work over and over.

It was not until a couple of weeks ago that I actually set up a trial account. I signed up and began configuring FogBugz to manage a small project that would likely run its full course over three to four 2 week iterations. Let me start by saying that FogBugz has been a pleasure to work with.

Simple and Intuitive

The folks over at Fog Creek have done a good job minimizing the amount of clicks required to accomplish some basic tasks like creating new cases/features/bugs. Lists of bugs and features can be viewed by various characteristics through filters or by easily accessible reports. It might seem like these features would be difficult to get excited about in a world filled with things like Django and Rails, but FogBugz managed to get its factors right making the process “flow” really well.

Oh, The Integration

Wiki, bug tracker, project management/prioritization view, customer email (support), and source control all talking to one another… Again, while other applications might offer these same tools and integrate some subset of them, FogBugz goes a step further. The integration is tight and using these utilities feels like you are just dealing with another dimension of the same data as opposed to using two independent applications that are similarly branded and share some ability to markup references.

That being said, I still have not yet figured out how to integrate source control. You can download a plugin for Visual Studio if that’s your tool of choice, but my experiments have been taking place in a Linux environment. There are a pair of shell scripts available for download that should solve my problem, but it was not immediately clear to me how to use them. That’s my next mission.

Evidence-based Planning

I am a huge proponent of evidence-based planning and it looks like they’ve done a good job bringing this capability to those who might not be familiar with the process. The gist of it is that by capturing both project estimates (at the feature level) and actual time taken you can build a model of your estimate accuracy, development velocity and then in turn generate release forecasts based on historical data. So with a click of a button (and disciplined engineers who enter their estimates and elapsed times) you can have nice reports and charts showing when you are most likely to deliver your releases and with what features — all based on historical data. It is a powerful concept and well executed here.

FogBugz goes as far as to model the estimation error of individual developers. This sounds great and is a good story, but I wonder how useful this actually is in reality where developers learn from past experiences and self-correct; without some Bayesian filtering I could see these values become noninformative fairly quickly.

I am also still playing around with the best way of capturing my existing process where we use team-based story point estimation. It seems like it will be straightforward, but I am waiting for a few gotchyas lurking in the darkness.

Things That Make Me Go “Hmmm”

I have struggled a bit to fit my Scrum-based backlog management mindset into FogBugz. The bugtracker allows features and bugs to be prioritized. Those priority levels are totally configurable and could be defined for whatever level of detail you need. One thing I am missing is the ability to deal with micro sprint-level priorities (i.e. what order should this set of features be implemented in to minimize overall project risk?). I realize I could define as many priority levels as I need, but it would start getting a bit tedious defining that many sublevels. Fortunately, the project I am currently using FogBugz for is small enough that I can manage without that functionality. It does however become important when you start increasing the team size and dealing with projects where you really do have a rank ordering of feature priorities. It could be that ScrumWorks by Danube Technologies has just spoiled me with their Scrum-based backlog system.

Another thing that has detracted from my experience is its response latency, but let me say that it is not bad. I think it might be the case that the application’s simplicity and overall experience allows me to forget that it is a remotely hosted application. I wonder how responsive the server license version is, where you host the code and the server is just working with your projects and perhaps on-site. Maybe I will find out one day.

Conclusion

My preliminary run through FogBugz has been overwhelmingly positive. It is a powerfully usable application that offers tight integration amongst core tools used in not only the development, but also the support of software applications. Check out the official highlighted feature list.
If you are looking around for some project management software then definitely check FogBugz out and see how it fits with your current process. I am still waiting to see how FogBugz will fit into our overall process once it has had a chance to run the full cycle — so far, it is looking good.