Productivity
November 24th, 2007 | Published in dynamic languages, organizations, productivity | 25 Comments | Bookmark on Pinboard.in
Speaking of Reg Braithwaite, in this post he touches on team size and productivity. There are so many gems there, I could comment on just about every sentence, but I think this is my favorite:
If we know that bug per line of code remains amazingly constant, why do we try to scale code out in verbosity rather than up in abstraction?
I’ve very often wondered the same thing.
When I’ve given talks about dynamic languages and productivity, I’ve sometimes brought up the “bugs per line of code” constant. With dynamic languages, I generally write far fewer lines to solve the problem than I would in Java or C++. While that alone doesn’t guarantee correctness, of course, in my experience it sure seems to lead to solutions that are completed faster and with fewer bugs. Yet in those presentations, I’ve always gotten pushback on that idea from someone in the audience, typically based on the “static typing will save you” argument, which I’ve never fully understood despite having written, read, and fixed tons and tons of statically-typed code over the course of my career. First, the notion that you’re generally likely to have more bugs if you have more lines of code just seems like common sense, since there are just more chances to be wrong. Second, static typing does nothing to help ensure that you’re applying suitable algorithms and are correctly solving the problem, yet that’s where the bugs tend to be.
But it goes beyond bug counts — sometimes it even comes down to getting a solution vs. not getting one at all. The fact that you can quickly visualize and produce a solution in Python or Ruby, for example, means you’re a lot more willing to actually sit down and crank out that solution in just a few days, rather than agree to take on a multi-month multi-person task to try to drive that nail with a Java or C++ hammer. Not to abuse another cliché, but I truly have been there and done that, believe me.
Postings like these often seem to attract comments of the form, “Well, dynamic languages are just too dangerous; we have several hundred programmers and we can’t watch everything that every one of them does.” I think that’s precisely Reg’s point! But this also gets to another related issue, which is that if you build teams that you can’t or won’t trust, you’re simply creating a self-fulfilling prophecy. You’re essentially expecting your teams to fail, so they will. In that type of environment, you’re just not going to be able to attract or keep anyone that you actually can trust. Furthermore, if you’re that type of manager or leader, not only is the likelihood that you’ll be able to scale your organization quite low, but you have nobody but yourself to blame for the eventual failures you’re sure to encounter. Issues related to dynamic languages and static typing are the least of your worries. Rather than tying your developers’ hands, it’s far better to devote continual effort to instilling common values and culture across the organization so that your teams can work toward common goals without requiring constant centralized oversight. Read Peter Senge’s The Fifth Discipline for great insight into this area.
For me, dynamic languages mean increased productivity. It’s as simple as that. It’s because the programming languages I use actually can do this, and because succinctness is power. And if you’ve never actually tried developing real software in such languages, that’s too bad, because that power — that productivity — really is quite amazing.
November 24th, 2007 at 10:05 am (#)
It’s tiresome to see the same old false dichotomy played out again. Pick particularly compact, pleasant dynamic languages. Draw a few verbose and ugly static languages. Draw generalisations about each entire category from these poorly chosen examples.
Let me remind you that BASIC is a dynamic language. I’m not talking about VB.NET, which is basically C# gussied up and jammed into a BASIC corset. But plain old BASIC can’t get no respect, and for good reason. Similarly some of the heinously awful “business 4GLs” like Progress.
On the other side, Haskell and OCaml are static languages with power-to-weight ratios that vastly better C++ and Java.
When you talk about “static” and “dynamic”, you’re picking straw men on *both* sides, and contributing to the mire of confusion rather than enlightenment. Reg got it right when he talked about expressiveness and abstraction rather than language features that are orthogonal. I’m not saying that the choice of a type system isn’t important, but it doesn’t give a language the properties that you seem to think it does.
November 24th, 2007 at 10:48 am (#)
Hi Bryan: keep in mind that I’m speaking from pure experience, not theory. In general terms, what you say is accurate, and I’m well aware of the static typing systems of languages like Haskell and OCaml. But if you’re speaking to a certain class of developer from a particular technical area who doesn’t have your background and and experience, what you’re saying won’t have any impact whatsoever.
In the world I come from, which is enterprise middleware and integration, the things I’ve written about are the very arguments I get when I suggest that using dynamic languages would be beneficial. In the world I come from, the majority of developers use Java and C++, period, and that’s why I choose them. In the world I come from, nobody uses Haskell and OCaml — in fact, few there have even heard of them — so basing arguments on them would be rather pointless. It’s why I mention Ruby and Python, because they’ve at least heard of them.
Yes, the term “dynamic languages” is broad, but it’s the name of a class of languages that’s much easier to write than listing only a set of specific languages every time I want to discuss that class.
Does the concept of writing fewer lines of code to solve a problem not imply expressiveness and abstraction?
November 24th, 2007 at 3:17 pm (#)
If all your audience knows really is C++, Java, Orbix, and J2EE, then I think that notions like expressiveness and abstraction could indeed be every bit as alien as Haskell.
Still, arguing from unsound examples isn’t the strongest way to buttress your point, even if you’re not trying to hold a rarefied conversation.
November 24th, 2007 at 11:00 pm (#)
Bryan: please enlighten me as to how I’m “arguing from unsound examples.” Let’s go over what I said:
Which part is unsound?
November 25th, 2007 at 12:50 pm (#)
Haskell/ML programmers say: “it’s point 3”. Well, there are even powerful type systems (dependent types) where _every_ bug is a type error, basically because the types encode the specification of the program. But I guess the people from point 2 didn’t mean those or Haskell :-)
Oh, and according to your own arguments, you should already be programming in Haskell :-) Really, those “dynamic” (argl, this word makes me nervous due to its emotional load “cool”) languages are no match for the power of functional programming. (Hm, I guess that “functional” can be perceived as emotionally loaded, too ;-)
November 25th, 2007 at 1:45 pm (#)
At risk of repeating Bryan’s comments:
– saying “dynamic languages” is a red-herring, your “bugs per line of code” argument is obviously based on how many lines of code there are and not on differences in type-checking.
– if you tell the audience that your shorter programs are less buggy than your equivalent Java and C++ programs, and they respond with “static typing will save you” then they simply don’t believe your claims that your shorter programs are less buggy. Are you showing them bug counts for various programs you’ve written over the last 5 years? iow do you /demonstrate/ to them that bugs per line of code remains amazingly constant?
– “Does the concept of writing fewer lines of code to solve a problem not imply expressiveness and abstraction?”
No more than it implies that we have a domain relevant library of tested code.
– “static typing does nothing to help ensure that you’re applying suitable algorithms and are correctly solving the problem, yet that’s where the bugs tend to be”
That’s just as bad as “static typing will save you”.
Maybe you just passively suffer static typing checking instead of trying to actively use it to force algorithm errors to show up as type checking errors?
November 25th, 2007 at 6:19 pm (#)
Or perhaps, because of having had to fix tons of statically-typed code over the course of your career… ;) (depending, I guess, on what types of fixes it needed).
November 25th, 2007 at 6:21 pm (#)
Haskell B. Pepper: I agree the term “dynamic language” is not a good one. I just don’t know a better one.
And you’re correct that I should already be programming in Haskell. I’m using Erlang a lot at the moment, and would very much like to get deep into Haskell, and OCaml too.
Isaac: as far as the term “dynamic languages” goes, see my comment above.
Regarding demonstrating the bugs per line of code constant, I believe Fred Brooks already did that.
Regarding the library of domain-relevant code, yes, I agree that such a library can help shorten programs. But then the issue just shifts to the library.
And regarding your final comments around static typing, I don’t really have an issue with static typing. I’ve been using C++ for 19 years, still write a fair bit of it, and over my career have probably written more lines of code in that language than in any other. I think C++ templates can sometimes support the active use of the type system that you mention. However, unlike a number of developers I talk to, I don’t see the C++ type system, or the Java type system for that matter, as something that simply I can’t live without. These days I prefer to develop in Erlang or Python rather than C++, but generally try to use the best language that I know for the task.
Thanks to both of you for your constructive comments.
I think Haskell B. Pepper has it right: I need to become fluent in Haskell. If anyone can share recommendations for where to start, that would be great, but I figure I can’t go wrong by visiting http://haskell.org/.
November 25th, 2007 at 6:28 pm (#)
I would suggest that the biggest, most mainstream statically-typed languages out there don’t offer a sufficiently powerful type system to make that a practical use of it. Haskell, ML, perhaps… my limited knowledge of them suggests so. But Java? not so much. All this talk about Haskell and OCaml sounds great, but those of us living in the “awful place where people make money with software” would like something we can get away with using at work.
November 25th, 2007 at 6:49 pm (#)
Hi Chuck: that “awful place” is exactly the kind of place I was targeting with my comments about team trust and developer control. Too many shops force the use of the wrong languages for their tasks, based mostly on hearsay and perceived popularity of languages, instead of actual technical considerations. One could argue that it’s too hard to hire people fluent in those other languages, but Reg Braithwaite’s point is that you probably don’t need as many developers if you’re using such languages. I would add that it’s a chicken-and-egg problem, and I believe significant competitive and economic advantage can be had by those shops willing to show some leadership here instead of just following the herd.
As I mentioned in the conclusion of my most recent column, I look back at all the CORBA software I wrote in C++ and I now know I could have written it faster, easier, and with better results in Erlang, but I’m also completely certain that management would have nixed the idea. Too bad, because had we done so, everyone would have won — the customer, the company, and the developers.
November 25th, 2007 at 11:31 pm (#)
> “dynamic languages”
If you mean expressive say expressive.
> “Regarding demonstrating the bugs per line of code constant, I believe Fred Brooks already did that.”
I think that means you did not demonstrate bugs per line of code remains amazingly constant for Ruby and Python and C++ and Java.
Did Ruby or Python or C++ or Java exist in 1975 when Fred Brooks wrote The Mythical Man Month?
Did most programmers have code-completion IDEs with fully automated refactoring in 1975?
It’s hard to introduce bugs when you have fully automated refactoring.
November 25th, 2007 at 11:40 pm (#)
Umm, Isaac: the whole point about the bugs per line of code constant is that it’s language independent.
You can argue with me all you want on the bugs-per-line-of-code-constant thing, but you’re just wasting your time. I don’t especially feel the need to redo others’ research just to prove their point all over again. If you don’t believe it, why don’t you do the work to prove that it’s wrong?
And surely you jest when you say that it’s hard to introduce bugs when you have IDEs that do automated refactoring. That’s just a joke, right? I mean, if it isn’t a joke, wouldn’t that mean that all Java programs developed with Eclipse, for example, would be bug-free?
November 25th, 2007 at 11:51 pm (#)
chuck wrote
> All this talk about Haskell and OCaml sounds great, but those of us living in the “awful place where people make money with software” would like something we can get away with using at work.
David Pollock was a Ruby programmer for 18 months – now he chooses to use a statically type checked JVM language, Scala: compiles to JVM bytecode, works with Java libraries.
5 Things a Ruby developer needs to know about Scala
November 26th, 2007 at 7:14 am (#)
steve:I agree the term “dynamic language” is not a good one. I just don’t know a better one.
I tend to say “scripting language” but that doesn’t fit either.
I need to become fluent in Haskell. If anyone can share recommendations for where to start, that would be great, but I figure I can’t go wrong by visiting http://haskell.org/.
Good idea :-) But I think that learning Haskell from the web doesn’t work, you’ll have a much easier time with one or two books made of paper. I recommend
* Graham Hutton: Programming in Haskell
* Richard Bird. Introduction to Functional Programming using Haskell, 2nd edition.
* Paul Hudak. Haskell School of Expression.
Hutton’s book is generally recommended for beginners, but I don’t know it myself and it seems a bit dry. Bird’s is a must-read as soon as you know a bit of Haskell, but I don’t know whether it works for starting from zero. Same for Hudak’s marvelous book about graphics and music.
See http://haskell.org/haskellwiki/Books_and_tutorials#Textbooks for a full list of textbooks. Maybe your local library has one of those, so you can “try without buy” ;-)
November 26th, 2007 at 8:48 am (#)
Moving up in abstraction does not reduce the number of lines of code, it just moves them around. It is still a very good thing to do, because the most important thing to do with complexity is to manage it hierarchally; but you cannot say, my program is more abstract, therefore it is more concise, therefore it has fewer bugs, because the real work is still being done somewhere and that code is still just as bulky and still has most of the bugs. They may be a little easier to find if complexity is optimally managed.
November 26th, 2007 at 10:13 am (#)
And therein, I think, lies some of the problem. The perception is that programmers are just technicians of a narrow skill-set, where programming languages are the “skills” in that set, and you can’t realistically expect them to learn anything new after they’re out of school. This is not only wrong, it’s destructive, and becomes a self-fulfilling prophecy that has given rise to the phenomenon of “JavaSchools.” Further, it doesn’t comport with my experience: I am currently still with the first company I was hired by out of school, and I haven’t used any of the same programming languages here that I had even touched beforehand — what I use here, language-wise, I learned on the job, and I have improved vastly in overall technique and design during the same period. What I learned before, in school, was the theory and techniques that made it possible for me to to do that. Don’t look to hire people fluent in programming language X; hire people who are good, or at least potentially good, programmers.
Great Isaac, but that still doesn’t address my concern. That it runs on such-and-such platform is not quite the first concern of most companies with respect to a programming language, but rather that it is a language that their existing team, and every programmer they will ever hire in the future, already knows — due largely to the misconception I just pointed out. It’s not hard to see how this leads to inertia…
Another dimension of the problem may be that the communities behind the kind of languages we wish we could be using don’t seem to know how, or even want to know how, to really promote their technology to industry. It almost seems as if there’s a barely-concealed disdain for the “enterprisey” world among the more academically-oriented communities that give rise to these technologies, such that they feel it is beneath them to try to make their language popular. I’ve seen a similar
snobberyattitude in the “indie” music world before. Way to advance the state of the art, there.Remember, though, Ruby wasn’t always as mainstream a language as it is today either. The Web 2.0 startup culture, in spite of what else you can say about it, has had a lot to do with that, because if you’re a one- or two-man company, you can use whatever language you want, and then blog like crazy about how well it’s working out for you.
November 26th, 2007 at 1:52 pm (#)
chuck wrote November 26th, 2007 at 10:13 am
> Remember, though, Ruby wasn’t always as mainstream a language as it is today either.
You think Ruby is mainstream!
November 26th, 2007 at 2:35 pm (#)
Note the use of the word “as.” It was a relative statement. But perhaps that concept eludes you.
November 26th, 2007 at 3:23 pm (#)
> rather that it is a language that their existing team, and every programmer they will ever hire in the future, already knows
And that erects a barrier that neither Java nor C++ would have overcome.
If you genuinely “would like something we can get away with using at work” then I’ve suggested something that people /do/ get away with using at work.
November 26th, 2007 at 3:38 pm (#)
[…] typing because they heard that it keeps programmers from making catastrophic mistakes — “static typing will save you,” they say. They heard that some pointy-headed academic said so, and since those guys are supposed to […]
November 26th, 2007 at 4:31 pm (#)
steve vinoski wrote November 25th, 2007 at 11:40 pm
> why don’t you do the work to prove that it’s wrong?
I didn’t write a blog article that assumes it’s true.
I can suggest why we shouldn’t just assume it’s still true after 30 years of technology change.
Just 10 years ago, for programmers using a statically checked language, simply changing the name of a type could be a major effort that could introduce bugs – that is no longer true.
Just 10 years ago, it was understood that encapsulating fields avoided the rework and bugs that would follow renaming – the cost and risk from field renaming has disappeared.
How much boiler-plate code does your IDE generate? Wouldn’t that change any bugs per line relationship?
The whole point about the bugs per line of code constant is that it’s a dated empirical observation and not an immutable law of nature.
December 2nd, 2007 at 10:31 pm (#)
Isaac, I agree with you 100%. The problem with Steve is that he’s still stuck in the 20th century coding with vi, emacs, and refactoring with grep and sed. Now that he’s coding in Erlang and Ruby, he’ll never know the productivity we have in IDE land. Seriously, I don’t understand how somebody could make any of these dynamic language arguments without the experience of using one of these IDEs in a medium to large project.
The thing is, these languages with zero type safety make it nearly impossible for an IDE to reliably refactor simple things like moving a method to a different class, renaming a method, class, or property, changing a method signature. Sure, they could make incomplete guesses, but then in that case, I’m better off using grep/sed and living in the 20th century.
December 2nd, 2007 at 11:16 pm (#)
Bill: funny, I don’t seem to have any problem whatsoever with productivity or meeting schedules. In fact, I’m usually ahead of schedule, and typically deliver a few extra niceties in every iteration. What do I need with a crutch/IDE? The main differences between an IDE and the tools I use (which BTW are not grep and sed) is that the IDE takes up way more memory, is slower, and is far less extensible.
Don’t underestimate the power of brevity. When you can choose the right language, you can often write a more functional application in an order of magnitude fewer lines, and the need for an IDE is significantly reduced.
If you think the languages you mention have “zero type safety,” then you have some studying and learning to do. And if you think IDEs can’t handle such languages, think again. I would expect a self-proclaimed IDE expert like you to know better.
And finally, the last time I checked, Bill, your single hammer for all nails, Java, is from the 20th century…
December 10th, 2007 at 10:28 am (#)
Steve, I think you should reread my comment. I didn’t say that a Ruby IDE couldn’t do refactoring, just that such refactorings wouldn’t be reliable and that the IDE would be guessing when it tried to refactor.
With an IDE with reliable refactoring, I’m more than an order of magnitude more productive anyways, so any increase in initial productivity I might get with Ruby et al. is negated.
December 10th, 2007 at 10:34 am (#)
Bill: negated? I hardly think so. Refactoring is only one of several areas related to productivity.