Programming

Solving “accidents” and “essences” of programming with better languages

Saturday, January 27th, 2007

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be. … The computer resembles the magic of legend in this respect, too. If one character, one pause, of the incantation is not strictly in proper form, the magic doesn’t work. Human beings are not accustomed to being perfect, an few areas of human activity demand it. Adjusting to the requirement for perfection is, I think, the most difficult part of learning to program.

Frederick Brooks, author of the The Mythical Man-Month, disinguished between two problems in software development: the accidental problems and the essential ones. (Read Brooks’ watershed article, “The Silver Bullet“.) The industry likes to argue that we have made huge leaps in terms of the accidental in recent years. Programmer productivity is better with the help of better software revision systems, but more importantly there are RAD tools like GUI designers, powerful IDEs like Eclipse/VS.NET. Do I think we’re headed in the right direction?

In a sense. I think those tools are useful, but only because I think all we can do is shrink the accidental problem as small as possible, and then do our best to tackle the essential. The major win in terms of the accidental and even the essential is high-level programming, and I mean really high level, like Python. Why is this important? Well, Brooks once talked about how PL/I was a great software engineering language because the statements written to the compiler are “pretty close” to the “thought stuff” the programmer is working with in his head. But PL/I is nowhere near there (neither is C++, or even Java or C#). However, Python heads in that direction.

In my head, unless I really am immersed in C code all the time, I don’t think of things like types, memory management, or, God, pointer arithmetic. When we solve problems, we solve it in something computer scientists have used for years: pseudocode. Pseudocode is nothing more than code that expresses an algorithm or approach without worrying about the gory details of the underlying hardware.

Python isn’t quite pseudocode, but it’s closer than anything else I’ve seen. And that’s a huge help to productivity not just from the point of view of the single developer, but, I’d argue, also from the point of view of a team of developers. Code readability is hugely important in team environments. C and Perl can be very unreadable, which makes them ill-suited to team development. Java and C++ are better, but even they suffer from some readability problems (where the actual solutions can be masked by “best practices” and strange ways of exploiting constructs of the language). Python can be hacked too, but it tends not to be, and it tends to have very high readability.

But what about the tools mentioned above (Eclipse/VS.NET)? I often wondered about all the *nix programmers (including many of my Computer Science professors) who still load up X11 and dump into vim or emacs for their programming needs. Wouldn’t they benefit from the latest and greatest in IntelliSense, Refactoring, CallTips, SaneOnlineDocumentation, and any other CamelCase ideas I can think of for a development environment? Most *nix users would say that stuff is unnecessary–and in many senses, they are right. The essential problem remains, no matter how fancy your IDE is.

A lot of *nix developers shrink their accidental problem on a “as-needed” basis, by coding plugins or complex configurations for their highly-extensible text editors. But nowadays, productivity is starting to be a concern even in the OSS world, where things have traditionally stuck with the old “mortar and pestle,” or, should I say, “gcc and gdb”. That’s why we see projects like Mono gaining a lot of popularity.

But Python offers a nice, alternative path, I think. The accidental problem is worth shrinking, and I think it can be done simply by one major refocus:

(1) make the language usable, not the tool. (IMO, already done with Python)

and

(2) create source code analyzing tools that integrate with development environments to make the language-bearer more productive.

You can see enormous success in (2) with the Eclipse IDE project. My main problem: Eclipse’s focus is on a language that isn’t very usable. Java is certainly better than C, but still, it has significant problems that stop it from connecting me (the programmer) with my problem-solving ideas (the thought-stuff of programming).

At my new job, I work with Java eight hours a day, but my code just doesn’t read and work like my Python code, which is much closer to the underlying algorithms and approaches. Java certainly pushed forward OO on the masses of computer programmers, but it did so without learning non-OO but very handy scripting language concepts, like having Lists and Hashes be built-in types, with tons of syntactic sugar, for crying out loud. Most problems break down to List and Hash problems, after all. What’s more, OO on its own isn’t a panacea: I’ve seen many convoluted OO designs that I wish were written procedurally, and which might have been more performant if they had been. The software engineering community thinks that there will be a silver bullet (OOP! AOP! XP! MDE!), but I think just getting programmers closer to the “thought-stuff” will be an effort with better rewards.

The Three-Way Division of Web Development Labor

Tuesday, December 12th, 2006

There is a great article posted by a developer of the open source FreeMarker project on a proposed “third” role for web development teams — something between a backend application developer and a frontend web designer.

Check it out.

I have to say, so much of this article rang true for me. I just finished a web application in which we had four developers but two main rules: backend developer and frontend designer/developer. Due to the type of project it was (and the skillsets of the people involved), backend work was done by 3 of the 4 team members, and frontend work was done by a sole designer.

The issues we had were that:

  • The backend developers could only make guarantees about what kind of object to pass to the view. But often, in order to get access to other “global” objects, the frontend designer would have to hack controller code as well. (Luckily, data services were tiered off out of the controllers, so at least the designer didn’t have to muck about with SQL.)
  • The designer hated the idea of pushing some logic “back” into application code. So, though we as backend developers proposed a layer for configuring things like which menus to display on each page, etc., the frontend designer preferred to engage in the copy-paste-modify antipattern instead, so that it could just “get done.” I can see why: it’s a pain to context switch from JSPs, HTML and CSS to Java POJOs that then will have to be checked using JSTL anyway!

If, early on, we had decided to use FreeMarker instead of JSTL, perhaps the frontend designer could have had access to more powerful macros, and wouldn’t have done the copy/pasting. But I think overall, the two tiers are too few; I agree that you need someone to sit in between and think about how to cleanly integrate backend logic and frontend design.

Great tool for LaTeX users

Tuesday, September 12th, 2006

Check it out. It’s called rubber, written in Python.

Mark Zuckerberg: Luckiest Man Alive

Sunday, April 30th, 2006

Sak and I were recently discussing how upset (read: envious, depressed about our own lives) we were about The Facebook seeking $2 billion.

The main reason we’re depressed is because, though both Sak and I like the Facebook and are users, we can’t help noticing one thing:

It’s not that complicated to build a website like that.

In fact, it’s downright easy. If I weren’t so busy with computer science classes, I probably could have threw something like it together myself.

Now, we know that business opportunities don’t have to be complicated to make money. They just have to be Right, that is in the Right place, with the Right look, taking advantage of the Right fad, etc.

But doesn’t it seem to you that a straightforward PHP/MySQL application just isn’t worth $2 billion? I mean, that’s $2,000 million. That’s $2,000,000,000.

Yet, I can’t say I’m entirely unhappy about it. Mark Zuckerberg, here’s to you, man. You’re my age, and you did exactly what I wish I had done. Built some crappy website, and made out like a bandit with sacks of cash. Kudos. You’re honestly my fucking hero.

The Flight of Computer Science majors

Sunday, April 30th, 2006

I read this response to an article at eWeek on Bill Gates’ views about Computer Science research, graduates, spending, and company strategies.

I think it’s pretty clear that despite the disagreements I have with Mr. Gates over a subject known as “business ethics” (if such a subject truly exists!), he does seem to be a genuinely patriotic guy who loves technology. I mean, what good is it for more Americans to get into CS, if other countries are diving in and filling whatever knowledge gap may exist? Can’t Bill just hire those workers, and what’s more, for less money per hour?

Well, I think Mr. Gates really wants innovation in computer software to remain “America’s Great Industry.”

I was very intrigued by this response to the article:

Everyone knows that Open Source is taking over the software development industry. And according to the Open Source philosophy; developers should be enslaved, source code should be free. No, no, that’s not politically correct, let me try again. Developers should give their work away because code needs to be free (as in speech) and the needs of the code is more important than the needs of the people who create it. Well, that doesn’t sound quite right either but in any case, it doesn’t really matter to me because my kids won’t be studying computer science.

This is a very interesting post. True, it will be seen as a troll by some, since open source philosophy definitely doesn’t say anything about programmer enslavement. But his point is real and felt in the industry. That is, if you aren’t selling software, how are software developers to make money from it?

I think the response to this was best-articulated by Eric Raymond, when he pointed out that of programmers, only about 1-2% make their cash from off-the-shelf software sales. Instead, most make their money from “in-house” or “custom” software solutions. In other words, the majority of developers aren’t working on the Adobe Photoshop team, they’re working on Acme Inc.’s payroll or issue tracking system.

I kind of love this sort of propaganda, though. Because it is all good news for me.

When I first decided to do CS, I considered the possible effect of outsourcing and other factors on my employment possibilities. I thought, what if there are no jobs when I get out of college? But I stuck with it.

Well, it turns out, everyone had a hunch similar to mine, but they were more wooed by it than I was. So everyone fled CS. And now I’m the only one left. (An exaggeration, but you get what I mean — my computer science classes are nearly empty, whereas they were packed during registration only a few years ago).

It turns out, firms are hiring more than ever before. Why? Because the dotcom bubble is over, and green-eyed imposters are getting flushed out of the industry. But the demand is still there. Software is pervasive. Everyone needs software development done. There simply isn’t anything under the sun that can’t benefit from a little software developer finesse.

You can’t have all this work done in India and China because, it turns out, people want software developers to work with customers (big surprise). They want applications which meet their sensibilities, and they want them changes when the environment changes.

I liked that Mr. Gates said the #1 thing he’s looking for is project management IT types. Funny, it’s the #1 thing I’m looking for, too. Software developers are a dime a dozen. Find me a software developer who doesn’t get nervous when you ask him a tough question, or ask him to write, in plain English, a high-level overview of the system you’re asking him to create, and you’ve got yourself someone who’s valuable.

Changing the tools you use

Wednesday, April 26th, 2006

Mark Shuttleworth has written a nice little blog post about the tools we learn through life and how we discard old tools and learn new ones.

I personally find this to be very true in my life.

When I was in high school, I prided myself (from the point of view of “tools”) as knowing graphic design (Photoshop/Illustrator), web development, and print/page layout. Handy tools to know for (1) making money and (2) working on a high school newspaper. The only real programming languages I knew back then were Actionscript (for Flash), JavaScript, and (eegads) Perl. Then I got to college and armed myself with algorithms, data structures, and systems, and started picking up Java and C on my own. Now I consider myself well-versed in those, and this past summer learned Python and used that on a lot of different projects. Then this semester I got interested in C++ and used that a lot. Nowadays, when I look at problems, I look at them in terms of my tools. Text parsing problem? Wow, Python’s re (regular expressions) module could handle that pretty easily. Big engineering project? Wow, using templates and OO features in C++ may lead to a nice design. Database-driven web application? Well, Java/JSP may fit you nicely. (I know, I know, what am I doing not knowing Ruby on Rails!)

I think Mark’s onto something. Changing toolsets often is definitely useful. Even though I couldn’t write full programs for you in Perl nowadays, what I do know about it (its limitations, capabilities) is definitely good enough to see when it may be the best choice for the job.

As for academic tools — very true. A lot of techniques I learned in e.g. Discrete Math, Linear Algebra were in one ear and out the other. Alas, I think the main point is to learn them once and then be able to Wikipedia them later, when needed 😉

That said, stuff I learned in my algorithms and data structures and operating systems courses have stayed with me. I think some of that stuff is just essential.

Using C++ instead of GObject/C

Tuesday, April 4th, 2006

There’s a great post on why to use C++ instead of GObject/C over at BMP CodeBlog, the blog for developers of beep media player (my preferred MP3 player on Linux). I have to agree with most of what the author says. Using GObject/C is quite cumbersome, I’m almost surprised so many GTK+ developers do it. Especially considering the fact that C++ doesn’t incur overhead unless it has to, nicely expressed here:

The ultimate feature is that all of these are optional. We incur no overhead when we do not use a language feature (with the exception of… exceptions). In fact, cxx_conversion is currently in perfectly valid C++ and uses none of the features or C++ libraries mentioned above yet. The C++ language follows the philosophy of ‘pay for you use’ (better rephrased as ‘don’t pay for what you don’t use’). As it is right now, cxx_conversion is no less (or more) CPU or memory efficient than its C original.

GTKMM keeps getting more and more mature, so now that I’m finally learning C++ properly, I think moving over to GTKMM will be quite nice for a future project that needs a cross-platform UI.

UNIX Zen

Monday, March 6th, 2006

You know you’re in UNIX zen when you are looking at pictures for an apartment you are interested in, and you think to yourself, “Man, I’d really just like to download all 8 photos of this apartment at once.” So you realize they end in numbers 1-8, and you open up a terminal and write:

for i in `seq 1 8`; do 
  wget http://www.livinginbaires.com/imgs/departamentos/46-$i.jpg; 
done

And that’s it, you hit enter and *vroooooom!* You see progress bars fly across your terminal as wget snappily fetches your photos, and suddenly, all eight photos are downloaded, and you get back to work. That’s empowering.

I don’t often quote Bill Gates, but…

Saturday, March 4th, 2006

Early in the history of Microsoft, our view was, if you were very smart then you could learn how to manage people, how to do business, how to do marketing.

It turns out that talent isn’t that fungible. Somebody who is great at doing software in many ways is often not the right person to manage people. I think the thing to recognize is these different kinds of ability, and to think through, as you build a team, “How do you get the right mix?”

Although I probably wouldn’t have used the word “fungible” myself, I agree with Bill’s sentiment. Programmers are not developers. And developers are not managers. But, I would add something else to this:

A manager who doesn’t know what it’s like to be a programmer or a developer cannot be a manager of programmers or developers! Management is a special combination of leadership ability and empathy. Your manager needs to be able to stick his neck out for you, because he knows you. And he needs to be able to provide wisdom — not because he’s smarter, or because he’s more mature, but because that’s now his role. Managers are the guys who protect you from everyone not on your team, who make sure you have everything you need to innovate, and who provide wisdom because their job description includes, “must be able to dispense timely wisdom at all the critical moments.”

Management also includes vision. Some software developers become managers and their vision alone leads their team. Where that kind of energy isn’t possible, lots of empathy, planning, and wisdom are necessary.

Getting the troops mobilized

Saturday, February 25th, 2006

I gave a talk last Tuesday called “Open Source Development: A Rapid Introduction.”

Here’s the blurb I sent out when I advertised the talk:

Have you wanted to work on open source projects, but just don’t know how to get started? This talk will provide the basics you need to start working on open source software the next time you sit down at your computer.

In particular, this talk will cover:

(1) A brief overview of open source development in the industry and press.

(2) The UNIX development platform. A brief and whimsical overview of the UNIX shell, its surrounding tools, and the power of shell scripting. (Useful to anyone wanting to learn more about UNIX tools.) Learn how to do in a few lines of shell script what you only thought was possible with a big, extravagant hundred- or thousand- line program, and learn why so many of the world’s best hackers hack on a *nix system.

(3) The basics you need in order to hack on open source project: how mailing lists, wikis, bugzillas, source code revision systems all come together to form an organic code management process, and how to get started using those tools and others to learn about a project and what parts need development work done. This will include a brief introduction to CVS.

(4) The last part of the talk will involve actually watching open source development in action. In particular, the speaker will checkout some code from a source repository, make a change to it, create a patchfile from that change, then track down the mailing list or bugzilla related to the project and submit the patch to the maintainer. You will actually get to see open source “in action,” and will want to do it right when you get home!

If you’re a Windows or Mac OS developer who has always wanted to learn more about *nix systems, or if you’re a developer who wants to either take his own project open source or work on existing open source projects, this talk is for you.

Of course, if you’re someone who is just interested in the concept of open source, this talk will give you an inside look at “how things get done” in this community.

This talk was at least partially inspired by Nat Friedman’s blog post, check it out here:

“How to be a Hacker”

The talk was a general success, I think. About 12 people attended. You can see the talk in PDF or ODP formats, and you can also download the patches I wrote specifically for the talk to illustrate “open source development in action.” The patches are pretty stupid, but do illustrate the point, at least. Plus, each of the three patches served one of my own goals (hacking my CPU frequency scaler, fixing a gnome-terminal bug, and hacking galeon “for fun”), so that’s that. I think it’d be cool to give this talk again (maybe a little refined to include less basic UNIX tools and more hacking stuff) at a later date. We’ll see.