PyCon US 2013 is over! It was a lot of fun — and super informative.

The People

For me, it was great to finally meet in person such friends and collaborators as
@__get__, @nvie, @jessejiryudavis, and @japerk.

It was of course a pleasure to see again such Python super-stars as
@adrianholivaty, @wesmckinn, @dabeaz, @raymondh, @brandon_rhodes, @alex_gaynor, and @fperez_org.

(Want to follow them all? I made a Twitter list.)

I also met a whole lot of other Python developers from across the US and even the world, and the entire conference had a great energy. The discussions over beers ranged from how to use Tornado effectively to how to hack a Python shell into your vim editor to how to scale a Python-based software team to how to grow the community around an open source project.

In stark contrast to the events I’ve been typically going to in the last year (namely: ‘trade conferences’ and ‘startup events’), PyCon is unbelievably pure in its purpose and feel. This is where a community of bright, talented developers who share a common framework and language can push their collective skills to new heights.

And push them, we did.

Read the rest of this entry »

My friend Jennifer Anyaegbunam (@JenniferAdaeze) has published a new piece on HuffingtonPost about the role of humanities in medical education.

Matriculating into medical school, we were proud of our humanities roots and felt it made us uniquely poised to become great clinicians. Yet, we have often found that we have had to defend our educational choices to interviewers, advisors and even our peers– something science majors rarely, if ever, have to do. This is because the medical humanities is often regarded as a “second tier” or an extracurricular interest and not something that is fundamental to the practice of medicine.

She finds that the humanities are derided in a classroom setting, as well:

Courses on ethics and social science are few and far between. To make matters worse, students often do not take these exercises seriously, and these courses are often the ones with the poorest attendance, for example

Here, I’ll offer a parallel from a different field: computer science.

As a computer science major at NYU, I too encountered hostility and a dismissive attitude toward the humanities and other “softer” fields from my peers.

A traditional computer science curriculum consists of mathematics, algorithms, and theory. These are important areas of academic interest, and provide a good foundation for thinking about the deepest problems surrounding computation. But the vast majority of computer science majors don’t go on to research computation. They go on to practice it — by becoming software professionals (programmers), writing applications used by real people.

It turns out that to be a successful software professional, you need much more than a computer science background. Indeed, many of the world’s most successful programmers have no computer science background at all. My father was a software professional, but when he graduated from college, computer science did not even exist as a field of study!

You need software design skills, which are often not taught except in a trivial way in traditional curriculums. It is considered “vocational”. You need communication, management, and product design skills. These are too “soft” to be taken seriously.

The industry suffers from a widespread lack of these skills.

Read the rest of this entry »

On my LinkedIn profile, I list one of my skills as “thought-driven development”. This is a little tongue-in-cheek; software engineering over the last few years has developed a lot of “XDDs,” such as test-driven development, behavior-driven development, model-driven development, etc. etc.

“Thought-driven development” doesn’t actually exist, but by it, I simply mean: perhaps we should think about what we’re doing, rather than reaching for a nearby methodology du jour.

In my last job, a colleague of mine used to also joke about “design-driven design” — perhaps the ultimate play on the XDDs since it is also a strange loop.

All this is not to say the XDDs aren’t useful — they definitely are. A lot of them have spawned entire groups of cross-platform open source projects. I am all for anything that makes the adoption of XYZ best practice easier for my team. But these techniques often require some lateral thinking to get to any real benefit.

When evaluating technologies like this, you have to take each little community with a grain of salt. Almost every programming framework / methodology / etc. that exists portends to offer some order of magnitude increase in software reliability / developer productivity / whatever else. And almost all, if not all, fail to do so, in practice.

Here is one anecdote to illustrate the point. From 2006-2008 at Morgan Stanley, the entire corporation was obsessed with Java’s Spring framework and its core “architectural pattern”, Inversion of Control. I can’t even begin to explain to you the number of man-hours that were wasted re-architecting existing, working software to meet this chimerical conception of component decoupling. I even contributed to this, urged by the zealots and their blind faithful. All of the reasons seemed great: decoupling code, using more interfaces, allowing for easier unit testing, being able to “rewire dependencies” and use fancy technologies like “aspect-oriented programming”.

Even Google got swept up in the madness and developed their own, competing framework called Guice. And in the end — after all that work — my diagnosis is that IoC is basically a non-starter, a complete waste of time.

A complicated framework that morphed into a programming methodology, developed exclusively to work around some annoying limitations of the Java language. Since it was applied without thinking, now everyone’s Java code has to suffer, and you can hardly pick up a Java web application today without being crushed by the weight of its IoC container’s XML configuration files. (Nevermind that most other communities, such as Python’s and Ruby’s, have hardly a clue what IoC is all about — a good enough indication that it is a waste of time.)

Every framework and approach should be judged on its true merits, that is the true cost/benefit analysis of applying that particular technology. Will it save us time? Will it simplify — not complicate — our code? Will it make our code more flexible / adaptable? Will it let us serve our users and customers better?

I regularly go back to old classics like the Mythical Man-Month to remember that nothing we do in software is truly new. I highly recommend you read it, and also its most famous essay, “No Silver Bullet”.

tl;dr stay healthily skeptical, don’t drink the kool-aid.

Me: Thanks so much for your fix to my issue. My friend, who majors in business, once told me that I should no longer major in Computer Science because “programming is like banging your head against the wall repeatedly, but with less reward.” I find that to be a rather rash dramatization, but I know in dealing with bugs as subtle as these it may feel that way. I hope at least the end-result is rewarding for you.

Programmer: Are you a Computer Science major? If so, don’t let your friend discourage you. Just ask him about “head banging” when those business majors find that their product development and marketing efforts fail to work after spending millions of dollars.

Me: Yes, I’m a CS major. And you’re right — the reward is great in software, and the cost of building an useful product is relatively minimal. That is one of the reasons I chose this path. It’s why I love helping out honest, intelligent developers such as yourself in any way I can. I have found that hardworking CS majors who are not only better programmers, but more often than not better thinkers and better managers — if you’d give them the chance. I am happy in my decision, and still have that naivete that perhaps I can change the industry a bit, shake things up, come up with an idea that changes everything, innovate in whatever way I can. Big aspirations; we’ll see what happens. For now, I’ll just keep respecting the good software I find in the world, such as yours.

It’s hard to find me gushing more unapologetically than when I talk about the virtues of my favorite programming language, Python.

Indeed, my life for the last 3 years has been dominated by the language. In many ways, pursuing a startup and enduring the associated financial hardship was partially because I had become frustrated with using Java in my full-time work and wanted to convert hobby projects I was building outside of work hours into full-fledged projects.

Read the rest of this entry »

And, sadly, our top engineering graduates don’t always become engineers. They move into finance or management consulting — both of which pay far higher salaries than engineering. I have seen the dilemma that my engineering students at at Duke University have faced. Do they take a job in civil engineering that pays $70,000, or join big Wall Street financial firm and make $120,000? With the hefty student loans that hang over their heads, most have made the financially sensible decision. In some years, half of our graduates have ended up taking jobs outside of engineering. Instead of developing new types of medical devices, renewable energy sources and ways to sustain the environment, my most brilliant students are designing new ways to help our investment banks engineer the financial system.

[...] We also need to make the engineering profession “cool” again, with the same sense of excitement and urgency in engineering and science that we saw during the Sputnik days. Back then, engineering was considered essential to the nation’s survival. Engineers and scientists were national heroes. It’s not that we don’t have problems to solve. The economy is in dire straits. Natural resources such as food, water, and crude oil are becoming scarce. Drug-resistant bacteria threaten us with doomsday plagues. But we’re not offering our best minds incentive to solve them.

From Mr. President, there is no engineer shortage.

Luckily this is happening already in high tech in NYC, thanks to awesome programs like HackNY and collabraCode (both of which my startup Parse.ly formally supported). As much as it pains me to say it, I also think The Social Network may be seen as a cultural catalyst for software engineers becoming “cool”.

But high tech is only a small piece of the puzzle — we need the same active marketing for students’ minds in biotech, education, medical research, civil engineering, etc.

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be. … The computer resembles the magic of legend in this respect, too. If one character, one pause, of the incantation is not strictly in proper form, the magic doesn’t work. Human beings are not accustomed to being perfect, an few areas of human activity demand it. Adjusting to the requirement for perfection is, I think, the most difficult part of learning to program.

Frederick Brooks, author of the The Mythical Man-Month, disinguished between two problems in software development: the accidental problems and the essential ones. (Read Brooks’ watershed article, “The Silver Bullet“.) The industry likes to argue that we have made huge leaps in terms of the accidental in recent years. Programmer productivity is better with the help of better software revision systems, but more importantly there are RAD tools like GUI designers, powerful IDEs like Eclipse/VS.NET. Do I think we’re headed in the right direction?

In a sense. I think those tools are useful, but only because I think all we can do is shrink the accidental problem as small as possible, and then do our best to tackle the essential. The major win in terms of the accidental and even the essential is high-level programming, and I mean really high level, like Python. Why is this important? Well, Brooks once talked about how PL/I was a great software engineering language because the statements written to the compiler are “pretty close” to the “thought stuff” the programmer is working with in his head. But PL/I is nowhere near there (neither is C++, or even Java or C#). However, Python heads in that direction.

In my head, unless I really am immersed in C code all the time, I don’t think of things like types, memory management, or, God, pointer arithmetic. When we solve problems, we solve it in something computer scientists have used for years: pseudocode. Pseudocode is nothing more than code that expresses an algorithm or approach without worrying about the gory details of the underlying hardware.

Python isn’t quite pseudocode, but it’s closer than anything else I’ve seen. And that’s a huge help to productivity not just from the point of view of the single developer, but, I’d argue, also from the point of view of a team of developers. Code readability is hugely important in team environments. C and Perl can be very unreadable, which makes them ill-suited to team development. Java and C++ are better, but even they suffer from some readability problems (where the actual solutions can be masked by “best practices” and strange ways of exploiting constructs of the language). Python can be hacked too, but it tends not to be, and it tends to have very high readability.

But what about the tools mentioned above (Eclipse/VS.NET)? I often wondered about all the *nix programmers (including many of my Computer Science professors) who still load up X11 and dump into vim or emacs for their programming needs. Wouldn’t they benefit from the latest and greatest in IntelliSense, Refactoring, CallTips, SaneOnlineDocumentation, and any other CamelCase ideas I can think of for a development environment? Most *nix users would say that stuff is unnecessary–and in many senses, they are right. The essential problem remains, no matter how fancy your IDE is.

A lot of *nix developers shrink their accidental problem on a “as-needed” basis, by coding plugins or complex configurations for their highly-extensible text editors. But nowadays, productivity is starting to be a concern even in the OSS world, where things have traditionally stuck with the old “mortar and pestle,” or, should I say, “gcc and gdb”. That’s why we see projects like Mono gaining a lot of popularity.

But Python offers a nice, alternative path, I think. The accidental problem is worth shrinking, and I think it can be done simply by one major refocus:

(1) make the language usable, not the tool. (IMO, already done with Python)

and

(2) create source code analyzing tools that integrate with development environments to make the language-bearer more productive.

You can see enormous success in (2) with the Eclipse IDE project. My main problem: Eclipse’s focus is on a language that isn’t very usable. Java is certainly better than C, but still, it has significant problems that stop it from connecting me (the programmer) with my problem-solving ideas (the thought-stuff of programming).

At my new job, I work with Java eight hours a day, but my code just doesn’t read and work like my Python code, which is much closer to the underlying algorithms and approaches. Java certainly pushed forward OO on the masses of computer programmers, but it did so without learning non-OO but very handy scripting language concepts, like having Lists and Hashes be built-in types, with tons of syntactic sugar, for crying out loud. Most problems break down to List and Hash problems, after all. What’s more, OO on its own isn’t a panacea: I’ve seen many convoluted OO designs that I wish were written procedurally, and which might have been more performant if they had been. The software engineering community thinks that there will be a silver bullet (OOP! AOP! XP! MDE!), but I think just getting programmers closer to the “thought-stuff” will be an effort with better rewards.

Sak and I were recently discussing how upset (read: envious, depressed about our own lives) we were about The Facebook seeking $2 billion.

The main reason we’re depressed is because, though both Sak and I like the Facebook and are users, we can’t help noticing one thing:

It’s not that complicated to build a website like that.

In fact, it’s downright easy. If I weren’t so busy with computer science classes, I probably could have threw something like it together myself.

Now, we know that business opportunities don’t have to be complicated to make money. They just have to be Right, that is in the Right place, with the Right look, taking advantage of the Right fad, etc.

But doesn’t it seem to you that a straightforward PHP/MySQL application just isn’t worth $2 billion? I mean, that’s $2,000 million. That’s $2,000,000,000.

Yet, I can’t say I’m entirely unhappy about it. Mark Zuckerberg, here’s to you, man. You’re my age, and you did exactly what I wish I had done. Built some crappy website, and made out like a bandit with sacks of cash. Kudos. You’re honestly my fucking hero.

I read this response to an article at eWeek on Bill Gates’ views about Computer Science research, graduates, spending, and company strategies.

I think it’s pretty clear that despite the disagreements I have with Mr. Gates over a subject known as “business ethics” (if such a subject truly exists!), he does seem to be a genuinely patriotic guy who loves technology. I mean, what good is it for more Americans to get into CS, if other countries are diving in and filling whatever knowledge gap may exist? Can’t Bill just hire those workers, and what’s more, for less money per hour?

Well, I think Mr. Gates really wants innovation in computer software to remain “America’s Great Industry.”

I was very intrigued by this response to the article:

Everyone knows that Open Source is taking over the software development industry. And according to the Open Source philosophy; developers should be enslaved, source code should be free. No, no, that’s not politically correct, let me try again. Developers should give their work away because code needs to be free (as in speech) and the needs of the code is more important than the needs of the people who create it. Well, that doesn’t sound quite right either but in any case, it doesn’t really matter to me because my kids won’t be studying computer science.

This is a very interesting post. True, it will be seen as a troll by some, since open source philosophy definitely doesn’t say anything about programmer enslavement. But his point is real and felt in the industry. That is, if you aren’t selling software, how are software developers to make money from it?

I think the response to this was best-articulated by Eric Raymond, when he pointed out that of programmers, only about 1-2% make their cash from off-the-shelf software sales. Instead, most make their money from “in-house” or “custom” software solutions. In other words, the majority of developers aren’t working on the Adobe Photoshop team, they’re working on Acme Inc.’s payroll or issue tracking system.

I kind of love this sort of propaganda, though. Because it is all good news for me.

When I first decided to do CS, I considered the possible effect of outsourcing and other factors on my employment possibilities. I thought, what if there are no jobs when I get out of college? But I stuck with it.

Well, it turns out, everyone had a hunch similar to mine, but they were more wooed by it than I was. So everyone fled CS. And now I’m the only one left. (An exaggeration, but you get what I mean — my computer science classes are nearly empty, whereas they were packed during registration only a few years ago).

It turns out, firms are hiring more than ever before. Why? Because the dotcom bubble is over, and green-eyed imposters are getting flushed out of the industry. But the demand is still there. Software is pervasive. Everyone needs software development done. There simply isn’t anything under the sun that can’t benefit from a little software developer finesse.

You can’t have all this work done in India and China because, it turns out, people want software developers to work with customers (big surprise). They want applications which meet their sensibilities, and they want them changes when the environment changes.

I liked that Mr. Gates said the #1 thing he’s looking for is project management IT types. Funny, it’s the #1 thing I’m looking for, too. Software developers are a dime a dozen. Find me a software developer who doesn’t get nervous when you ask him a tough question, or ask him to write, in plain English, a high-level overview of the system you’re asking him to create, and you’ve got yourself someone who’s valuable.

Mark Shuttleworth has written a nice little blog post about the tools we learn through life and how we discard old tools and learn new ones.

I personally find this to be very true in my life.

When I was in high school, I prided myself (from the point of view of “tools”) as knowing graphic design (Photoshop/Illustrator), web development, and print/page layout. Handy tools to know for (1) making money and (2) working on a high school newspaper. The only real programming languages I knew back then were Actionscript (for Flash), JavaScript, and (eegads) Perl. Then I got to college and armed myself with algorithms, data structures, and systems, and started picking up Java and C on my own. Now I consider myself well-versed in those, and this past summer learned Python and used that on a lot of different projects. Then this semester I got interested in C++ and used that a lot. Nowadays, when I look at problems, I look at them in terms of my tools. Text parsing problem? Wow, Python’s re (regular expressions) module could handle that pretty easily. Big engineering project? Wow, using templates and OO features in C++ may lead to a nice design. Database-driven web application? Well, Java/JSP may fit you nicely. (I know, I know, what am I doing not knowing Ruby on Rails!)

I think Mark’s onto something. Changing toolsets often is definitely useful. Even though I couldn’t write full programs for you in Perl nowadays, what I do know about it (its limitations, capabilities) is definitely good enough to see when it may be the best choice for the job.

As for academic tools — very true. A lot of techniques I learned in e.g. Discrete Math, Linear Algebra were in one ear and out the other. Alas, I think the main point is to learn them once and then be able to Wikipedia them later, when needed

That said, stuff I learned in my algorithms and data structures and operating systems courses have stayed with me. I think some of that stuff is just essential.