Steve Vinoski's Blog

Eric Newcomer gives us his take on my recent articles (1, 2, 3, 4) and blog postings (1, 2, 3, 4) about RPC, REST, and programming languages:

Anyway, after carefully reading the article and blog entries, I believe Steve is not against RPC per se. He wants people to think before just automatically using it because it’s convenient.

Exactly!

Also spot on are the following postings:

• Bill de hÓra: Patterns of Web Architecture
• Three postings from Stu Charlton:
  • Convenience, Correctness, and Network Interactions
  • Can technology progress?
  • On changing industry mindsets

The beauty common to all these postings is the breadth, depth, and variety of thinking and reasoning they present. There’s a lot to read, but if you’re interested in critical thinking about the design and construction of distributed systems I encourage you to read them all the way through, including the comments, and to follow the links they offer as well.

Mark Pilgrim tells us why Protocol Buffers are so nice. Notice, though, that everything he writes focuses entirely on their form and structure as messages. If you focus only on that perspective, then sure, they’re better than what many could come up with if they were rolling their own. In fact, if Google had stopped there, I think Protocol Buffers could be a superb little package.

But they didn’t stop there. No, they had to include the fundamentally flawed remote procedure call. Well, sort of, anyway:

The Protocol Buffer library does not include an RPC implementation. However, it includes all of the tools you need to hook up a generated service class to any arbitrary RPC implementation of your choice. You need only provide implementations of RpcChannel and RpcController.

Why ruin a perfectly good messaging format by throwing this RPC junk into the package? What if I want to send these messages via some other means, such as message queuing, for example? Do I have to pay for this RPC code if I don’t need it? If my messages don’t include service definitions, do I avoid all that RPC machinery?

In my previous post I talked about the message tunneling problem, where data that doesn’t fit the distributed type system are forced through the system by packing them into a type such as string or sequence of octets. Since Protocol Buffers require you to “hook up a generated service class to any arbitrary RPC implementation of your choice,” it’s likely that you’re going to run into this tunneling problem. For example, if you want to send this stuff over IIOP, you’re likely going to send the marshaled protobufs as Common Data Representation (CDR) sequences of octet. You’re thus unavoidably paying for marshaling twice: once at the protobuf level for the protobuf itself, and then again at the CDR level to marshal the sequence of octet containing the protobuf. Any worthwhile IIOP/CDR implementation will be very fast at marshaling sequences of octet, but still, overhead is overhead.

But there are other problems too. What about errors? If something goes wrong with the RPC call, how do I figure that out? The answer appears to be that you call the RpcController to see if there was a failure, and if so, call it again to get a string indicating what the failure was. A string? This implies that I not only have to write code to convert exceptions or status codes from the underlying RPC implementation into strings, but also write code to convert them back again into some form of exception, assuming my RPC-calling code wants to throw exceptions to indicate problems to the code that calls it.

What about idempotency? If something goes wrong, how do I know how far the call got? Did it fail before it ever got out of my process, or off my host? Did it make it to the remote host? Did it make it into the remote process, but failed before it reached the service implementation? Or did it fail sometime after the service processed it, as the response made its way back to me? If the call I’m making is not idempotent, and I want to try it again if I hit a failure, then I absolutely need to know this sort of information. Unfortunately, Protocol Buffers supplies nothing whatsoever to help with this problem, instead apparently punting to the underlying RPC implementation.

Still more problems: the RpcController offers methods for canceling remote calls. What if the underlying RPC package doesn’t support this? Over the years I’ve seen many that don’t. Note that this capability impacts the idempotency problem as well.

Another question: what about service references? As far as I can see, the protobuf language doesn’t support such things. How can one service return a message that contains a reference to another service? I suspect the answer is, once again, data tunneling — you would encode your service reference using a form supported by the underlying RPC implementation, and then pass that back as a string or sequence of bytes. For example, if you were using CORBA underneath, you might represent the other service using a stringified object reference and return that as a string. Weak.

All in all, the Protocol Buffers service abstraction is very leaky. It doesn’t give us exceptions or any ways of dealing with failure except a human-readable string. It doesn’t give us service references, so we have no way to let one service refer to another within a protobuf message. We are thus forced to work in our code simultaneously at both the Protocol Buffers level and also at the underlying RPC implementation level if we have any hope of dealing with these very-real-world issues.

My advice to Google, then, is to just drop all the service and RPC stuff. Seriously. It causes way more problems than it’s worth, it sends people down a fundamentally flawed distributed computing path, and it takes away from what is otherwise a nice message format and structure. If Google can’t or won’t drop it, then they should either remove focus from this aspect by relegating this stuff to an appendix in the documentation, or if they choose to keep it all at the current level of focus, then they should clarify all the questions of the sort I’ve raised here, potentially modifying their APIs to address the issues.

I’ve gotten a few emails asking me to blog my opinion of Google’s Protocol Buffers. Well, I guess I pretty much share Stefan’s opinion. I certainly don’t see this stuff providing anything tremendously innovative, so as with Cisco Etch, it seems to me to be just another case of NIH.

Ted Neward already wrote a pretty thorough analysis — it’s almost 0.85 Yegges in length! — so I’ll just refer you to him. There are at least two important points he made that bear repeating, though:

Which, by the way, brings up another problem, the same one that plagues CORBA, COM/DCOM, WSDL-based services, and anything that relies on a shared definition file that is used for code-generation purposes, what I often call The Myth of the One True Schema.

Indeed. Usually when one points this out, those who disagree with you come back with, “Oh, that doesn’t matter — you can just send whatever data you want as a string or as an array of bytes!” Having been forced to do just that numerous times back in my CORBA days, I know that’s not a good answer. You have a bunch of distributed infrastructure trying to enforce your One True Schema Type System, yet you go the extra mile to tunnel other types that don’t fit that schema through it all, and the extra layers can end up being especially complicated and slow.

The second point I’ll quote from Ted says it all:

Don’t lose sight of the technical advantages or disadvantages of each of those solutions just because something has the Google name on it.

Most excellent advice.

There’s one last thing I’ll quote. This one’s not from Ted, but directly from the Protocol Buffers documentation:

For example, you might implement an RpcChannel which serializes the message and sends it to a server via HTTP.

Sigh.

[Update: more opinions — and some questions — in my next post.]

As an author I don’t expect everyone to like what I write or agree with everything I write. I’m always up for receiving constructive criticism, as it helps me improve my writing, and when carried out with a positive mindset I think it helps everyone involved in the dialogue learn something.

Unfortunately, not all reviews are constructive, and those that aren’t can be hard to handle. I don’t know why some people have to resort to name calling and harsh words if they have a technical disagreement with you; perhaps those folks lack emotional fortitude and feel big when they hide behind their blogs or email, or maybe they don’t realize how foolish their own words make them look. If you’re an author faced with such a review, do you sink to the level of the reviewer and fire back at them with harsh words of your own to try to put them in their place? Probably not, since that just makes you too look like a jackass. Do you just ignore them? Sometimes that’s the way to go, as you don’t want to encourage your new-found stalker to continue stalking you and getting off on having succeeded at trolling to gain attention. (I use the term “stalker” here because there are definitely people out there who seem to just sit and wait for you to publish something, anything, so they can immediately “review” it to prove to the world just what an idiot you are.) But sometimes ignoring the review is wrong, too, as it might make it seem as though you have no answers for the reviewer’s criticisms.

Steve Jones just published such a gray-area review of my “Convenience Over Correctness” column. I don’t know him but he’s apparently a dyed-in-the-wool SOA fan, so it’s totally unsurprising that he disagrees with me. I started to go through his review paragraph by paragraph and respond to each point, but I found that it wasn’t very productive, mainly due to the personal insults and attacks he decided to throw in, oh, every fourth line or so. I’ll therefore just look at three of his criticisms in response, and leave it at that.

First, Steve chides me for pointing out what he deems to be obvious:

So far these problems [with RPC-based systems] have been detailed [in the column] as

• Remote calls have more issues than local ones
• Remote transaction processing is a bitch

There are no other issues raised and both of these points fall into the “well duh” school of pointing out the obvious.

But then he says:

I’ve built distributed systems and I’ve had to manage teams who delivered the architectures I created and I’ll say that

• 60% of the people didn’t understand the challenges and wouldn’t have understood Waldo
• 30% would have read it and got it wrong
• 6% Understand the challenges and can make a decent crack at it with minor problems
• 4% actually understand what it takes

These two sections seem contradictory. How can Steve fault me for “pointing out the obvious” when by his own estimate only 4% of my audience actually understands the issues?

I know for a fact from my columns and my conference presentations that there is a great desire for this sort of information, and that not everyone truly understands the hard issues of distributed computing, so at least Steve and I agree on that. My preference, though, is to help provide that information and help raise awareness, whereas Steve’s seems to be to just assume everyone else is a “muppet,” thus enabling him and his other 4% friends to do all the heavy lifting and spoon-feed everyone else with simple frameworks they might, just might mind you, be able to understand.

This is elitism, pure and simple, and it’s an expensive and non-scalable model. It puts the self-proclaimed 4% experts in control and wastes the vast skills and talents of the majority.

Coincidentally, my September/October column is going to touch on this. It’s already partially written and is due in a few days, and while it’s not at all a response to Steve’s review, it will explain in part why elitist systems simply cannot, and do not, last. I’m sure Steve will completely hate it.

Then there’s this:

What a load of crap. Seriously this is an unmitigated pile of tripe in what it means to write distributed systems. It makes two basic errors

1. That the architecture and design of a system is focused on a programming language
2. See number 1

Ignoring the foul language and such, how can anyone claim to be an expert in real-world distributed systems development like Steve does, yet apparently be unaware of the various Java and C# systems out there, for example, that use special meta-language annotations to export and expose language features directly as distributed system features? There are many out there who think you just throw some annotations on a class and it magically becomes distributed — they think only within the confines of their language, and magic frameworks provided by the 4% experts like Steve make all the distribution work under the covers. (Steve claims this approach is necessary because it’s all that the other 96% are capable of understanding, but IMO it’s really just one of the ways the big vendors and big consultants can continue to relieve uninformed enterprisey companies of their money.) Criticizing the column on this basis simply shows that Steve, a self-proclaimed expert, is unaware of the language-specific distribution frameworks out there, which is odd given that they’ve been proliferating for years.

The only other particular issue I’ll remark on in Steve’s review is his idea that he could quickly cobble together some code in his blog to “prove” that RESTful systems can fit in a programming language, and thus also suffer from the “convenience over correctness” problem. The column already states that people are trying to do just that, but that from what I’ve seen they invariably run into problems with various REST constraints such as the hypermedia constraint. If it were so easy, Steve, we’d already be overrun with RESTful language frameworks. Thankfully, though, we’re not, because the two simply don’t mix conveniently, which is why I’ll continue to stand by what I wrote.

For many years I lived in the same enterprisey SOA world as Steve, and so the only positive thing I got out of his review was a reminder of how glad I am to be away from it. I met a lot of bright people there, don’t get me wrong, but I miss neither the elitism nor the “that will neither work nor scale within the enterprise, you complete muppet!” attitude that was doled out when anyone dared make a suggestion that might actually improve things or threaten the control held by those elite 4%. Thankfully, though, that culture can’t last forever.

But then again, what do I know? I’m just one of the 96% muppets.

Stefan Tilkov‘s latest InfoQ article, REST Anti-Patterns, is simply outstanding. It covers all the bases. While it’s a great read for anyone doing RESTful service development, it’s especially suitable for developers who’ve already studied the fundamentals of REST but are still struggling to put them into practice, and also for developers who’ve done a lot of browser-focused web development but not much application-to-application web development.