Me: Thanks so much for your fix to my issue. My friend, who majors in business, once told me that I should no longer major in Computer Science because “programming is like banging your head against the wall repeatedly, but with less reward.” I find that to be a rather rash dramatization, but I know in dealing with bugs as subtle as these it may feel that way. I hope at least the end-result is rewarding for you.

Programmer: Are you a Computer Science major? If so, don’t let your friend discourage you. Just ask him about “head banging” when those business majors find that their product development and marketing efforts fail to work after spending millions of dollars.

Me: Yes, I’m a CS major. And you’re right — the reward is great in software, and the cost of building an useful product is relatively minimal. That is one of the reasons I chose this path. It’s why I love helping out honest, intelligent developers such as yourself in any way I can. I have found that hardworking CS majors who are not only better programmers, but more often than not better thinkers and better managers — if you’d give them the chance. I am happy in my decision, and still have that naivete that perhaps I can change the industry a bit, shake things up, come up with an idea that changes everything, innovate in whatever way I can. Big aspirations; we’ll see what happens. For now, I’ll just keep respecting the good software I find in the world, such as yours.

Swig grew a fully compatible C++ preprocessor that fully supported macros. A complete C++ type system was implemented including support for namespaces, templates, and even such things as template partial specialization. Swig evolved into a multi-pass compiler that was doing all sorts of global analysis of the interface. Just to give you an idea, Swig would do things such as automatically detect/wrap C++ smart pointers.It could wrap overloaded C++ methods/function. Also, if you had a C++ class with virtual methods, it would only make one Python wrapper function and then reuse across all wrapped subclasses.

Under the covers of all of this, the implementation basically evolved into a sophisticated macro preprocessor coupled with a pattern matching engine built on top of the C++ type system [...] This whole pattern matching approach had a huge power if you knew what you were doing [...]

In hindsight however, I think the complexity of Swig has exceeded anyone’s ability to fully understand it (including my own). For example, to even make sense of what’s happening, you have to have a pretty solid grasp of the C/C++ type system (easier said than done). Couple that with all sorts of crazy pattern matching, low-level code fragments, and a ton of macro definitions, your head will literally explode if you try to figure out what’s happening. So far as I know, recent versions of Swig have even combined all of this type-pattern matching with regular expressions. I can’t even fathom it.

Sadly, my involvement was Swig was an unfortunate casualty of my academic career biting the dust. By 2005, I was so burned out of working on it and so sick of what I was doing, I quite literally put all of my computer stuff aside to go play in a band for a few years. After a few years, I came back to programming (obviously), but not to keep working on the same stuff. In particularly, I will die quite happy if I never have to look at another line of C++ code ever again. No, I would much rather fling my toddlers, ride my bike, play piano, or do just about anything than ever do that again.

From this Swig mailing list entry.

Indeed, my life for the last 3 years has been dominated by the language. In many ways, pursuing a startup and enduring the associated financial hardship was partially because I had become frustrated with using Java in my full-time work and wanted to convert hobby projects I was building outside of work hours into full-fledged projects.

Something else I noticed in the last three years is that my programming life has become very zen-like. I now rarely discuss or debate things like programming language features, strange constructs like generics, or which framework to use or ignore. Instead, I spend most of my time building a product that people love. My colleagues and I communicate with code. And what better language to communicate in than arguably the world’s most readable? What better language to deliver value in than one that simply gets out of your way?

I therefore get a great amount of joy of showing other people what the Zen of Python can mean in their lives. Last year, I gave a training course to a 20-person team of government employees who were using dated languages like Fortran and COBOL to build important government systems. A bright manager in the organization realized how much more productive their team could be if they stopped worrying about compiler versions and IDEs and started thinking in code. But the key was to understand the value of Python, not necessarily as a language with a certain set of features, but as a way of doing things, as a cultural influence. This is a culture that says, “the language should fade away”, similarly to how Edward Tufte argues that the chrome and administrative debris should fade away when displaying content.

Since then, I have used these slides countless times to espouse the virtues of my favorite interpreter. This has included giving short seminars in NYC, training Parse.ly interns and new hires (here’s a shot of me and two Parse.ly engineers, Michael and Zach), holding Python office hours at HackNY’s hackathon (indeed, here’s an action shot on Flickr to prove it!), and introducing my friends to the joy of building software.

Code and Slides: The Zen of Python in 3 days

It’s therefore with elation that today I am able to provide the slides and materials for this talk free to the world on my Github account. Simply click here:

https://github.com/Parsely/python-adv-slides

Or, if you’re not interested in the code behind the slides (because the slides are actually created with the help of Python itself), you can go here to simply view them.

http://pixelmonkey.org/pub/python-training/

(note, modern web browser like Chrome or Firefox recommended for speed and smoothness)

This may be my first “open source presentation”, in that the slide presentation itself is provided as code that is freely available, and reproducing the slide presentation is a matter of running a code generator.

A touch of NLP with Python

I mentioned that one of the reasons I love Python is because it gets out of your way. Nowhere is this more evident than Python’s ability to prototype algorithms to natural language processing and corpus linguistics problems, something I do everyday. The Natural Language Toolkit (NLTK) provides an excellent starting point from learning about this aspect of computer science.

Recently, Parse.ly has gotten some new engineers on the team who are great Python programmers but without any background in NLP. To give them a taste of how well-suited Python is for this task, I gave a seminar entitled, “Just Enough NLP with Python”. Just as with the prior slides, you can view these online on Github:

https://github.com/Parsely/python-nlp-slides

And you can view the compiled slides here:

http://pixelmonkey.org/pub/nlp-training/

Between these two pieces, you can learn the Zen of Python (PEP 20) very rapidly, and incorporate it into your everyday software life.

If you have any comments or feedback, feel free to reach out to me on Twitter. I’d love to hear what you think. And if you, too, are bitten by the Python bug (as one of our Parse.ly interns recently was in a big, almost romantic way) then you should reach out to us, since we’re hiring.

Among them:

• Lifehacking through standing desks. I have created a standing desk setup for my home office, and my investors in Parse.ly have created a healthy startup office in NYC. I have some thoughts about this as it applies engineering and hacking to the thing many of us do most: sit in a desk chair all day.
• 2-day Python Training Course. I have created a 2-day Python training course that helps existing programmers learn Python by comparing it very directly to languages they may already know, like C and Java. I gave this course to a group of government employees a few months ago. It has some interesting characteristics: I designed the whole course using ReStructured Text (ReST) and compiled it into a web-based presentation. This means that the entire course has the potential to be “open source” — exercises, slides, and all. I plan to release this to the public. For now, I am just clearing a few of the images I used in the course to make sure I don’t inadvertently infringe copyright. After that, I will open to the public.
• Groovy/JavaScript articles go public domain. Some articles I wrote for GroovyMag and JSMag last year are now able to be published in the public domain. These include one about RESTful services with Groovy, one that talks about functional programming with JavaScript, and finally one that discusses a design for “metagrids” in ExtJS 3.x. I will put all three articles up on this blog once they are reformatted.

Stay tuned!

Now, don’t get me wrong — Python also has batteries included, and usually, when I think that I’m missing a great module I used to use in Java, it already exists in a much more powerful form in Python’s Standard Library or the wealth of modules on PyPI, GitHub, and Bitbucket. However, I believe in not reinventing the wheel, and so if a great open source tool exists in Java, I will want to interact with it.

One of these modules which we use extensively at Parse.ly is Apache Solr, and its surrounding Lucene project modules. Lucene is an extremely mature framework for document indexing, and Solr is a powerful server-ization of that technology that fits well into complex, mixed language distributed systems. I know there are efforts — like Whoosh — to build fast search engines atop the Python language. And I applaud these efforts — more projects means more competition, and more competition means better products. However, I still believe that you go with the best of breed tools available for production software, and you try not to let religious arguments about programming language get in the way.

Lately, I have come across more and more Java open source projects that have no equivalent in Python, and which I would like to access. Knowing that I wanted to feel comfortable incorporating Java open source projects — beyond Solr, which was already nicely wrapped as a web service — I, at first, thought that I’d be forced to still live among the weeds of complex class and interface definitions, cumbersome Java IDEs, XML configuration files, and (IMO) time-wasting rabbit holes like dependency injection, configuration management, and classpath hell. And then I found Groovy.

The most under-rated language ever

Groovy was released as one of the earliest dynamic languages written atop the JVM, back in 2003. I remember hearing about it back then and not “taking it seriously”, as we programmers often do with important, new technologies. I had the same ill feelings toward JavaScript until as late as 2007, but now consider myself more a JavaScript programmer than a Java one!

In truth, had I read James Strachan’s blog post discussing why he created Groovy, I would have probably paid more attention:

So I’ve been musing a little while if its time the Java platform had its own dynamic language designed from the ground up to work real nice with existing code; creating/extending objects normal Java can use and vice versa. Python/Jython’s a pretty good base – add the nice stuff from Ruby and maybe sprinkle on some AOP features and we could have a really Groovy new language for scripting Java objects, writing test cases and who knows, even doing real development in it.

I had thoughts like these back in my Java programming days, too. I would see features that were standard in other languages — concise list/map syntax, “bare” functions (that could live outside of class definitions), optional/dynamic typing, metaprogramming, a proper REPL, first-class functions — and think to myself, “Aren’t we missing a whole lot by lacking these features in Java?” The more I started to code Python and Java side-by-side, the more I started to realize that many “core” Java technologies were basically created to cope with inadequacies of the language itself.

For example, unit testing “frameworks” and formal IDE/debuggers are more popular in Java than other languages because of the lack of the REPL — dynamic language programmers tend to test their code from an interactive shell as a regular part of programming. Utility libraries like Apache Common Utils were created to cope with syntactic deficiencies around core types like lists and maps. And on and on.

Groovy is a “post-modern language” dynamic language. By that, I mean it is clearly inspired by and learns from the lessons of both Python and Ruby. It is also the world’s first programming language that was written to target an existing community of programmers. Let me explain. Python was not written “to target” C programmers or Visual Basic programmers. It was written as a new language altogether, not targeted at anyone in particular. However, Groovy was written “to target” Java programmers — it is the dynamic language Sun should have built for the JVM as dynamic languages gained popularity and the Java language stagnated.

Groovy is respectful of and cooperative with Java itself. One of its primary design goals is to live alongside existing Java code, even while Groovy’s syntax far surpasses that of Java. In this respect, Groovy plays a very similar role in the Java ecosystem that Python plays in the C ecosystem. However, unlike Python and C’s relationship, Groovy’s relationship with Java is bidirectional. What that means is that one can easily write Java libraries that execute Groovy code (provided the Groovy Language is on the classpath as a JAR), and Groovy libraries can easily execute Java code. In fact, you can even have complex relationships among these two languages, such as Groovy classes serving as base classes for Java classes and vice versa.

However, what’s perhaps most astonishing about Groovy is that it truly got a whole lot of things right. That is, the language simply is extremely well designed, and specifically greased for programmer productivity. It learned from the mistakes of existing dynamic languages, and made a dynamic language that is truly the “best of both worlds”. When I introduced the language to an engineer on my team at Parse.ly, he uttered some words that really stuck with me: “This language was clearly designed by very, very lazy programmers.”

So true. Typing is a chore — let’s do less of it to code without sacrificing readability. That’s the Groovy way.

Over the last several years, it has matured to the point where I feel extremely comfortable writing production code in Groovy, especially thanks to the surrounding ecosystem of the Grails project.

Python is still my muse, but Groovy is my Winston Wolf

A famous character in the movie Pulp Fiction is Harvey Keitel’s “Winston Wolf”. He’s the underworld problem solver — a guy who gets things done with a no-nonsense attitude.

Winston Wolf: If I’m curt with you, it’s because time is a factor. I think fast, I talk fast, and I need you two guys to act fast if you want to get out of this.

Python is a beautiful language and will continue to be my language of choice for production code. However, I simply can’t ignore the enormous wealth of production-quality Java code that has been written for which there is no good Python equivalent. I also can’t ignore the unbelievable engineering effort that went into making the JVM a powerful, scalable, and stable platform for production software. I thought I’d have to throw this community away (for myself and my companies) due to the deficiencies of the Java language. However, I have come to realize that Groovy has proven itself the worthy successor to Java. (Yes, successor — as in, you would be a fool to write your code directly in Java these days.) And personally, I will never work on a Java project again without Groovy. Java is a mess, but Groovy is my clean-up man.

I have also observed that the “ported” JVM languages have failed. This was a surprise to me as it was happening, but in retrospect, is not as surprising. Jython and JRuby are simply not viable long-term open source projects. The target community is only those programmers who know both Python and Java or both Ruby and Java. And there was no great impetus to truly “learn” those environments, because Python programmers felt that they already “knew” Jython, since the language was the same. Is the benefit of having access to Java libraries worth the downside of using a non-standard VM for my primary programming language? Usually not.

Now, while I wasn’t even paying attention — slowly learning Groovy in my spare time and using it for small projects dependent on Java — it seems that the world of independent dynamic JVM languages has taken off. Groovy has not had nearly as much popularity as languages like Scala (used at Twitter and Foursquare) or Clojure. And Groovy lacks “sex appeal”, possibly because no significant large-scale projects have been documented atop Groovy and because the language is an incremental improvement on successful mixed-paradigm languages.

However, unlike Clojure and Scala, the Groovy language can be learned in a weekend by an existing Java programmer with dynamic language experience, and can be learned in a couple weeks by anyone else. And unlike those other languages, it does not try to be a fundamentally new programming language with new programming models. Instead, it simply dramatically improves the Java language itself using a syntax that becomes quickly familiar to dynamic programmers.

When you consider the cost/benefit analysis of learning Groovy — for a couple weeks of my time, I gain access to being able to easily script — and, with Grails, wrap as web services — an almost limitless supply of Java open source code, you realize you’d be a fool not to learn the language.

I like new stuff as much as the next guy, but Java is a language that is still used by millions of professional programmers across the world, and has proven its ability to be the basis for large projects. And Groovy is a 2X or 3X productivity improvement over Java, with full bi-directional compatibility with existing Java code? I’ll take that over a new-fangled “revolutionary” language any day!

So, what’s so groovy about Groovy?

Oh, nothing… just things like this:

["Rob", "Christopher", "Joe", "John"].findAll{ 
  it.size() < = 4 
}.each { 
  println it 
}
// ==> prints "Rob", "Joe" and "John"

Yes, we’re really not in Java anymore. Let’s walk through the pain that is Java step by step to arrive at our Groovy salvation.

First, Java loves boilerplate. It really is an affliction of the language and the community. A great example of this is Java’s take on module imports.

Even though Java has a great standard library that is extremely well-designed and complete (for the most part), it fails to make this standard library easily accessible to you. That’s because all of Java’s best libraries are hidden in obtuse, deeply-nested library locations, and one can never remember where they are. IDEs solve this problem by providing an “Organize Imports” facility, which expert Java programmers use instead of managing imports themselves. This problem, alone, puts Java IDEs on significantly more productive footing than simple text editors like vim.

Groovy says, “this is nonsense — stop the madness”. In Groovy, a slew of commonly-used Java modules are simply imported automatically. If you run into a namespace conflict, that’s OK — we’re all adults here.

// these are all automatic in every Groovy module you write
import java.lang.*;
import java.util.*;
import java.net.*;
import java.io.*;
import java.math.BigInteger;
import java.math.BigDecimal;
import groovy.lang.*;
import groovy.util.*;

Note that groovy.lang and groovy.util add significant utilities above and beyond Java.

Next up are the major data structures used by any modern programmer — lists and maps (aka dictionaries). If you haven’t been doing Java in awhile, you won’t miss this kind of code.

List<Integer> someItems = Arrays.asList(new Integer[] {1, 2, 3, 4});
for (Integer item : someItems) {
    System.out.println(item);
}
Map<String, String> someMapping = new HashMap<String , String>() {{
    put("ST", "started);
    put("IP", "in progress");
    put("DN", "done");
}}
for (Map.Entry<String, String> entry : someMapping.entrySet()) {
    String key = entry.getKey();
    String value = entry.getValue();
    System.out.println(key + " => " + value);
}

Note that the above is about the most concise possible code you can write for these operations using standard Java facilities. In fact, I have even tapped into some obscure features, using Arrays.asList to initialize the ArrayList (leveraging the fact that arrays, but not lists, have a concise initialization syntax) and leveraging double curly brace initialization for the Map, which is an extremely non-standard and perhaps even overly inefficient technique. (The alternative, though, is to write out someMapping.put(key, val) for each entry in the Map — ick).

Let’s look at this same syntax, but in Groovy.

someItems = [1, 2, 3, 4]
someItems.each {
  println it
}
someMapping = ["ST": "started", "IP": "in progress", "DN": "done"]
someMapping.each { key, val ->
  println "${key} => ${val}"
}

Not only is this code more concise, it is also much more readable. The signal-to-noise ratio is much better.

Groovy has great support for Functional Programming and Domain-Specific Languages.

// first-class functions/methods and DSL syntax
clean = { text ->
  text.replaceAll("[^A-Za-z]*", "")
}
clean "my-text-*foo"
remove = someItems.&remove
remove 3
someItems

It retains support for loved Java features like switch statements, but these become much more powerful and concise, e.g. in this factory function that actually returns cleaner functions depending on the passed-in type.

// proper closure, switches and variable interpolation
cleanerFactory = { type -> 
  switch (type) {
    case "text":
        return clean
    case "numbers":
        return { text ->
            text.replaceAll("[^0-9]*", "")
        }
    default:
        throw new IllegalArgumentException("invalid type ${type}")
  }
}

Using the cleanerFactory is a breeze, simply call it and you get back a function which you may then call. Yes, this is basically impossible in Java. To get something close, you’d have to make the factory return “Runnable” instances and then call .run() on the results.

// try/catch and bean syntax shorthand
test = "my-text-*foo-1234"
println cleanerFactory("text")(test)
println cleanerFactory("numbers")(test)
try {
  println cleanerFactory("foo")(test)
} catch (IllegalArgumentException ex) {
  println "cleanerFactory returned exception: ${ex.class.simpleName}, ${ex.message}"
}

Just like Java, Groovy supports classes. But it improves them by adding a concise syntax for declaring proper JavaBeans. The following class has getters, setters, and a keyword constructor autogenerated for you.

// classes and types, if you need them
class Article {
  String title
  String summary
  String content
  Date date
}
article = new Article(
    title: "Parse.ly uses Groovy, oh noes!", 
    summary: "Didier resigns in protest", 
    content: "Keith goes back to .Net, Toms returns to Ireland.  Ahhhh!",
    date: new Date(2011, 3, 7)
)
article.properties.each{println it}

Also notice the special “.properties” property, which allows you to iterate over a JavaBean’s property without using Java’s obtuse reflection API.

// inheritance if you need it
class WSJArticle extends Article {
    List topics
}
wsj = new WSJArticle(topics: ["US", "Finance", "World News"])
wsj.topics.each{ println it }

Inheritance works exactly as expected, and follows Java rules. However, inheritance gets even better in Groovy because it loosens stupid rules that Java has, such as “final” classes.

// and sure, even call and inherit from Java code (even final classes!)
class BetterDate extends Date {
    def getTimeString = {
        return "{this.time}"
    }
}
println new BetterDate().timeString

It’s simply not possible to subclass Date in Java, but Groovy has no problem with that. We’re all adults here.

// and sure, use Groovy scripting constructs with existing Java APIs!
lotsaItems = [1, 1, 2, 2, 2, 3, 3, 3, 3, 3]
println Collections.frequency(lotsaItems, 3)
Collections.shuffle(lotsaItems)
println lotsaItems

The above is an example of using existing Java APIs. Collections is surely a nice static class. Groovy lets you use it to your heart’s content.

The API for the Collections.binarySearch method is written as:

static <T> int binarySearch(List<? extends T> list, T key, Comparator<? super T> c)

Yes, that is “documentation” So basically, I need to create a class that implements the Comparator interface, and then I need to pass that comparator to this method along with my list. That will require a lot of angle brackets to get the generics right. Hmm, who cares, let’s just do it the easy way: in Groovy.

// even existing, complex APIs work fine in Groovy, and with less code!
comparator = [
  compare: { a, b -> a.equals(b) ? 0 : Math.abs(a) < Math.abs(b) ? -1 : 1 }
] as Comparator
println Collections.binarySearch([0, 2, 5, 7, 8], 5, comparator)

This shows off the ability of Groovy to easily coerce values using some very intelligent rules. Comparator is an interface that expects a “compare” method — therefore, we can coerce a map with a “compare” key and a function value into an implementation of that interface.

// convenience features abound, like heredocs and safe pointer access
myDocument = """
  This is a long document...
"""
foo = [myDoc: null]
println foo.myDoc?.size() // returns null rather than an exception

If I had a dime for every time I noticed Java programmers using StringBuilder to work around its lack of multi-line strings, or adding extra methods just to do null checks... well, I'd be very rich indeed Groovy says -- no need to complicate your life with that noise.

// and type coercion that actually works!
set = [1, 2, 3, 4, 5] as Set
5 in set
// ==> true!

Yes, the in keyword. Yes, lists can be converted to sets. Because that makes sense!

// this is a feature I wish Python had -- JavaScript-like map syntax
map = [:]
map.item1 = "foo"
map.item2 = "bar"
map.each { println it }
// ==> "item1"="foo", "item2"="bar"
println "done"

Groovy also learns a little from JavaScript -- maps are accessible using either key index or dot syntax. I personally like this, I know some Python people won't.

// nice range and slice syntax
items = [1, 2, 3, 4, 5]
println items[0..2]
for (i in 0..5) { println i }

Similar to Python's slice syntax, Groovy supports a "Range" type using two dots. It works where you expect it to: array indexing, for loops.

Using Grails, the standard web framework for Groovy, you can write simple RESTful, JSON web services in just tens of lines of code. e.g., here's a Grails controller that renders the result of a function as JSON.

package service
import com.thirdparty.library.ContentExtractor
import grails.converters.JSON
 
class ParseController {
  def contentExtractor = new ContentExtractor()
  def index = {
    def article = contentExtractor.extractContent(params.url)
    render article as JSON
  }
}

This will respond to a service like http://service:8080/parse?url=http://google.com with a JSON document containing the Java properties from the article object.

Seriously, how much easier can it get to wrap up a Java library as a JSON web service? Not much.

// in short, a pretty language worth looking at
println "DONE."

Download Groovy and Grails to start playing around!

This is highly subjective, but can be easily understood by Pythonistas who have been with the language for awhile.

Here’s some un-Pythonic code:

def xform(item):
    data = {}
    data["title"] = item["title"].encode("utf-8", "ignore")
    data["summary"] = item["summary"].encode("utf-8", "ignore")
    return data

This code is both un-Pythonic and unidiomatic. There’s some code duplication which can very easily be factored out. The programmer hasn’t used concise, readability-enhancing facilities that are available to him by the language. Even lazy programmers will recognize this code’s clear downsides.

Here’s another version that is more idiomatic but is nonetheless still un-Pythonic:

class ItemTransformer(object):
    def __init__(self, item):
        self.item = item
 
    def encode(self, key):
        return self.item[key].encode("utf-8", "ignore")
 
    def transform(self):
        return dict(
            title=self.encode("title"), 
            summary=self.encode("summary"))

Nothing about this code is particularly unidiomatic. I might even see code like this in many popular open source projects. But it’s in poor taste. It’s un-Pythonic.

What is the code doing? It’s just taking an incoming dictionary, encoding its values using utf-8, and returning a new dictionary with those encoded values. There is no need to introduce an ItemTransformer object — it’s an extra abstraction and is just making the signal-to-noise ratio poorer. People coming from Java often write un-Pythonic code because Java is a language that does not reward good taste. The Pythonic view is: programming is hard enough — let’s not make it harder for ourselves.

Here’s a more Pythonic version:

def encode_news_item(item):
    def encoded(*keys):
        for key in keys: 
            yield (key, item[key].encode("utf-8", "ignore"))
    return dict(encoded("title", "summary"))

This code shows comfort with Python’s features, but does not abuse this comfort by obfuscating the code with mind-bending constructions. The programmer has reduced the problem to two comprehensible subproblems: creating a stream of tuples (key, encoded_value) and constructing the new dictionary from that stream. This leverages the elegant fact that in Python, dictionaries can be easily constructed from (key, value) tuples.

This version avoids code duplication while also making the last line (the return statement) a rough description of the entire function. “return a dictionary of the encoded values for the keys title and summary” Idiomatic, yes. But also tasteful, and thus Pythonic.

Even Pythonic code can be improved:

def encode_news_item(item):
    encoded = lambda key: item[key].encode("utf-8", "ignore")
    keys = ("title", "summary")
    return dict((key, encoded(key)) for key in keys)

And though Pythonic code is often smaller than its un-Pythonic counterparts, the experienced Pythonista knows the road to hell is paved with good intentions:

def encode_news_item(item):
    keys = ("title", "summary")
    return dict(zip(keys, map(lambda key: item[key].encode("utf-8", "ignore"), keys)))

Ick… time to hg revert this idiomatic Python code to the more Pythonic version

For those of us who start companies, we see the company as a vehicle to creating something valuable and lasting for society, while also advancing our personal goals. This isn’t usually hubris or ego, though sometimes it may be. Instead, it’s usually an attempt to make your time worthwhile: to yourself, to those close to you, and — if you’re lucky and persistent enough — to the entire world.

The problem with social mobility is that not every individual begins at the same starting line. In fact, the range is huge. Those who start with an influential family or significant capital resources have a much easier time getting to the top. For those who don’t have this head start, things are a lot harder.

Though America is not entirely merit-based, it can reward individuals for hard work. I’ve experienced the benefit both of an advantageous starting point and hard work in my 26 years on this earth. I also believe that with each step and milestone in my life, my potential to create enduring value for society has increased significantly.

Beginnings

The inspiration for this essay was a comment I read online about a successful young businessman who was the son of a successful businessman. “I’d [like to] read a story about a 25-yo [who] made good on the same scale[,] who went to a state college, had screwed up parents who were too busy fighting with each other or gettiing drunk to even have a clue what he was doing, isn’t childhood friends with a celebrity… Just happened to be smart and hardworking and optimistic even despite all those factors. That would actually be an interesting story.”

My parents weren’t screwed up, but they did fight a lot — my Mom and Dad separated when I was in elementary school and divorced shortly after that. I’ve not been childhood friends with a celebrity and I don’t have a trust fund.

I’m not saying I came from a disadvantaged upbringing — in fact, quite the opposite. I went to public high school in New York. To a New Yorker, that may not sound like a huge step up in the world, but I recognize that public school in New York represents one of the top educations you can get.

I grew up in a nice house in a quiet suburban neighborhood. I had good, encouraging teachers. My parents were liberal and a positive influence. I didn’t have a silver spoon in my mouth, but I also didn’t have any serious handicap in my upbringing. Probably my biggest step up in the world, given my current trajectory, was that when I was 10 or 11 years old, my Dad noticed an interest I had in computers. And so, he bought me a PC (running DOS / Windows 3.1) and set it up for me on Christmas Day. From that point forward, I was enchanted by the machine. And once I got a 28K modem and dial-up access to the web (on one of the first ISPs, The Pipeline), I became a citizen of the world before I had even hit puberty.

This I do know — though I had a head start, I also worked hard. I was a geek — as I got older, I built my own servers in my basement, taught myself to program, and discovered Linux and Free Software. But I also kept ahead in “the real world”. I did feel a little disconnected from my peers in my private pursuits; reflecting on my childhood, I realize I “grew up” a little more quickly than my peers.

My First “Startups”

At 15, I had built and launched somewhat popular websites for a couple of personal projects. Nothing you would remember — each one was related in some way to an interest I had in multiplayer online video games, an interest I aged out of a year later. Each website had between a few hundred and several thousand visitors a week.

But the sites looked nice. When I told my family friends and teachers in my local suburb that “I built websites”, they were all interested in having me work for them. Recognizing the opportunity, I turned this into a business.

By 2010 standards, the websites I built were boring and simple — static designs, gaudy flash “intros”, contact forms and quickly-outdated information portals. Yet in 1999, this was cutting-edge. So cutting-edge that people were willing to pay to be on the web. I printed up business cards and called my company a “digital online identity firm”. Might seem like an obvious thing now, but I felt proud to deliver that value to my early-adopter customers. This was my first “B2B startup”. During this period, my friends would ask me to hang out on the weekends, but I’d be working on HTML/CSS, JavaScript and Macromedia Flash, building up my online portfolio.

At 16, I was approached by the editors of a technology publishing company called Friends of ED to write a couple chapters for a book of theirs called Flash 5 Dynamic Content Studio. They found me on the web — my first “digital introduction”, before the days of LinkedIn. On the weekends, I regularly answered people’s Actionscript programming questions in a listserv called flashpro. They liked that I had both a Flash and programming background, so they contacted me to contribute to their book — one of the first that discussed building backend systems and tying them to Flash frontends.

I corresponded with them entirely via e-mail. I wrote the chapters and edited about 80% of the book during my junior year of high school, while also prepping for SATs and acting as editor of my school newspaper. I probably got 3 hours of sleep a night. My friends thought I was crazy.

When the book finally went to press and started showing up on Amazon.com and in Barnes & Noble stores, it was one of my proudest moments in my early adulthood. I just thought about how the work I did was being seen and read by thousands of people across the world. Each of those individuals would go on and use that knowledge to build web applications for other individuals. There was seemingly no limit to the impact of my work. I may have been naive, but I thought, what could be better than this?

I still remember when the editors in the UK found out I was only 16 (it was while we were negotiating payment). They were shocked, they actually didn’t believe me. They had to check with lawyers to see if they could even pay me. Then they paid me the bunch of money they owed me for my hourly editing rate and IP rights, and I had all the savings I needed to live throughout my first couple college years without asking my parents for support.

By 16, I had some unique experiences. I created a market in my skills. I built working software for companies small and large. I contributed to a project that had global impact. I made some real money. And, in my junior and senior years of high school, I got a taste of the competing demands of personal and work life.

College: Creating Value for Myself

When I got into NYU on scholarship, I was thrilled. My grades in high school let me go to a good college without breaking the bank. I would be able to attend a top-tier school without the financial stress typically associated with private schools. I’d get to stay in New York, one of my true loves. I had also been rejected by just about every other top-tier university in the country. Come to think of it, the ups and downs of the college application process was good training for startup life — especially the rejections.

I worked my butt off in college. I really got into it. Not just the coursework, but also just pursuing my passions within the student body. I held talks that tried to stoke up my peers about technology, economics and open source software, all passions of mine. My friends rarely saw me because I was often working on these and other spare time projects.

College was interesting. I worked hard, but rarely got paid for my work. In some sense, working hard in college is good preparation for a startup lifestyle. You work hard because you believe in yourself and in your own potential, not because you have a paycheck coming.

It was also in college where I came to an important realization: I am a software engineer and computer scientist. Though there are many ways for me to add value, my preparation and passions make me especially well-suited to building software. And so, that’s where I should have an impact.

Internship: Creating Value for Others

I took an internship in summer of my sophomore year where I got paid a flat $3000 to build a web application for a NYC-based non-profit, The Unemployment Action Center of NY. I probably spent 750 hours on the thing (60 hours a week), not making it a very profitable proposition. My friends again thought I was crazy — lots of them had taken well-paying and prestigious internships at software companies and banks, but I didn’t even apply to those.

This was my first experience with building a real product that people actually used. I worked entirely from home. I used my own development environment and hosting environment. I gathered requirements directly from the end-users. I made all the technical decisions and implemented all the code. That was good experience for startup life. Built useful stuff, solved real problems. Made lots of little decisions.

The case management application I built is still running today, relatively bug-free, and has helped thousands of unemployed people get legal representation for free, so that they can get justice for wrongful termination and other cases. I’m really proud of that application. I didn’t care that I got paid $4/hour for it. Looking back, I would have done it for free. What a phenomenal organization and what a great cause to apply technology toward.

Post-College: Creating Value for Money

The rest of my story is told pretty well by this NY Observer article.

I worked at Morgan Stanley for a couple of years while living at home on Long Island to save money. Working at the bank was a huge change from the rest of my life. The primary focus of my job became my paycheck. Technology decisions were made for me, by external committees. I was expected to follow all sorts of processes and procedures. I was expected to become a conventional programmer. I found myself building products that had an unclear end-user.

I did learn things there, too. I learned what it was like to work with engineers on a daily basis. I learned a lot from my managers and colleagues, many of whom were just astoundingly intelligent individuals. I learned how big companies operated. But mostly, I learned a lot about myself.

I learned that work meant more to me than a paycheck. Work should be about solving problems, helping people, and creating enduring value. Money is important, but it wasn’t what attracted me to technology in the first place. And that’s why I knew I had to leave that firm.

Startup Life: Creating Lasting Value for Myself, for Others, and for Money

I accepted this compromise: I would work at the bank to save the money I needed to build my own company. I was “realistic”, and this seemed a fair compromise.

I anticipated that I’d need at least a $20K buffer to start my startup (it ended up being more than that). And in March of last year, I quit Morgan Stanley to embark on this path. My awesome, like-minded friend from NYU, Sachin Kamdar, quit his job too. Together, we tried to build something of value, even though we didn’t know what that would be initially. We worked out of cafes, brainstorming ideas and building prototypes. We consulted on the side — I, through the boutique software engineering firm I founded, Aleph Point. Early on, my girlfriend prophetically referred to my financial strategy in all this as “putting a bandaid on a hemorrhaging femoral artery”.

Our friends thought we were crazy. Chuck a steady paycheck to work on an ill-defined “startup”? For some reason, all we could think was, hell yes we would.

Our first stroke of luck was getting into DreamIt Ventures. NYC didn’t have seed programs like Tech Stars or Seed Start yet. We moved to Philadelphia for the summer, leaving our NYC apartment leases and girlfriends behind. They thought we were crazy. The $20K from DreamIt basically covered our living expenses in Philly, but it did lift some financial pressure for a few months. We came up with a $4/person/day diet — which we called “the startup diet” — that involved eating beans, soup, romaine lettuce, and veggie burgers basically every day. We lived in a “startup house” with the founders of SeatGeek and Tidal Labs. We worked in an office with the founders of Postling, NoteHall, and other fledgling companies. We started hacking on what would eventually become Parse.ly. And for the first time in our lives, we were surrounded by individuals who also felt that what one does is more important than how one benefits.

Our second stroke of luck was finding our exceptionally talented lead engineer, Didier Deshommes, who has been with us ever since we incubated Parse.ly in Philadelphia. Ever since DreamIt, we operate on this maxim: we got our share of luck — now, let’s make our own.

Creating Your Own Luck

Perhaps our startup founding story isn’t remarkable. But lately, reflecting on the last year — where we bootstrapped and self-financed this startup, overworking ourselves and straining our personal relationships — I’ve started to feel like it is pretty remarkable.

We made our own luck. And we’ve brought all our skills and experience to bear. I’ve been preparing for running Parse.ly since the age of 15. I’ve been preparing, basically, my whole life. And I’m so glad we did this together. With all the ups and downs, if I had done this on my own, I would have given up. But we just kept pushing each other. Every time we hit a roadblock, we just said, “we can still do this.” And we did.

I’m really proud of this company and our team. We’ve built a real, valuable technology of lasting value. We’ve learned so much. And we’ve done it all against the odds. We are a small NYC startup with two no-name founders. Neither of us have “prior exits” or “a solid track record”. Neither of us have built and launched startups before. But none of that matters. We discovered that people want their content filtered and prioritized and that online publishers want their content optimized, and we used elbow grease, hustle, and dedicated hacking to make the rest happen.

Our startup is three hard-working and optimistic guys who believe that we can change the world. We put it all on the line to create something from nothing. Our work is still in progress, and there are still challenges ahead. There are easier things we could be doing, but we don’t care. You may think we’re crazy, and you may be right.

But that’s the final lesson of startups, isn’t it? You have to be a little crazy.

—

Thanks to Sachin Kamdar for editing this essay. And thanks to our awesome friends, significant others, and families for still cheering us on even as they rightfully thought we were crazy.

What’s the value preposition of Flavors.me? It’s to unify your various “online identities” into a single, dynamic, automatically-updated, and elegant website.

What do I mean by that? OK — so, like most people on the web, you spread public information about yourself in multiple places. You might run one or two blogs (personal and work?). You might have a Facebook account, a Twitter account. You may share your favorite books at GoodReads, your favorite movies at Netflix, and your favorite music on Last.fm.

Flavors.me lets you take all that information and put it together in a single website to serve as your “online identity”. All your publicly shared information, aggregated in one place, and displayed beautifully.

I’ve been running a Flavors.me site for some time that you can see here: http://flavors.me/pixelmonkey

Now, that’s the end product. All the content gets pulled dynamically from your various online feeds. The real magic with Flavors.me is how easy it is to get there. You can drastically change the look and feel of this site using a dynamic, “WYSIWYG” interface. You can do one or two clicks to add a service, reorder it, rename it. Another couple of clicks and you change font sizes, colors, and even the overall layout.

As I was writing this post, I noticed that Flavors.me had added LinkedIn support, which wasn’t available earlier in the beta. So I went ahead and added it. I simply logged in and pulled up the design panel.

From there, I could navigate over to “Services”, click the “LinkedIn” logo, and Flavors.me would guide me through the authorization process to make LinkedIn data available to them for display on my page. You simply get redirected to LinkedIn.com, log in there (if not already logged in), and then get redirected back to the Flavors control panel.

Next, I could simply drag-and-drop the service box in the panel in order to reorder it on the page. Yay, drag-and-drop! There is also in-line editing, and editing the title of the LinkedIn “popout” dynamically updates my profile page in real-time! Nice touch.

Finally, I can go to the “Design” panel to tweak fonts, layout, sizes, etc. Look at how Flavors.me displays fonts. It’s amazing — this is the web, but it’s easier to change my Flavors.me web design font styles than it is to change font styles in a Microsoft Word document!

Now, depending on where, exactly, you store the bits that make up your online identity, you may find yourself disappointed by a lack of this-or-that service. They already support quite a few, and they are adding more in the future, I hear. The one I sorely missed was Delicious, so I could share my bookmarks with the world.

But wait! For those services that still don’t have first-class support, Flavors.me gracefully supports RSS feeds of any variety. I simply popped the RSS feed for my Delicious bookmarks into Flavors.me services panel, and, voila! — my bookmarks are now publicly shared!

It’s this kind of simplicity, design sense, and user-centric approach that makes me love the web as a place to develop, deploy, and use software. So, what are you waiting for? Sign up for Flavors.me today: it’s free!

I also found it very interesting to hear Crockford wax poetic about the virtue of simplicity in all forms of software design. The following passage concludes the book.

When I started thinking about this[...], I wanted to take the subset idea further, to show how to take an existing [product] and make significant improvements to it by making no changes except to exclude the low-value features.

We see a lot of feature-driven product design in which the cost of features is not properly accounted. Features can have a negative value to consumers because they make the products more difficult to understand and use. We are finding that people like products that just work. It turns out that designs that just work are much harder to produce than designs that assemble long lists of features.

Features have a specification cost, a design cost, and a development cost. There is a testing cost and a reliability cost. The more features there are, the more likely one will develop problems or will interact badly with another. In software systems, there is a storage cost, which was becoming negligible, but in mobile applications is becoming significant again. There are ascending performance costs because Moore’s Law doesn’t apply to batteries.

Features have a documentation cost. Every feature adds pages to the manual, increasing training costs. Features that offer value to a minority of users impose a cost on all users. So, in designing products[...], we want to get the core features—the good parts—right because that is where we create most of the value.

We all find the good parts in the products that we use. We value simplicity, and when simplicity isn’t offered to us, we make it ourselves. My microwave oven has tons of features, but the only ones I use are cook and the clock. And setting the clock is a struggle. We cope with the complexity of feature-driven design by finding and sticking with the good parts.

It would be nice if products[...] were designed to have only good parts.

I removed direct references to the main subject of Crockford’s discussion — namely, the JavaScript language itself. The truth is, this advice is much more valuable for the design of all software products. Perhaps one day someone will write the much needed book, Startups: The Good Parts.

I’ll show why this claim doesn’t matter later, but for now, I’ll assure you that it’s true.

How many of you out there use Dropbox? If you don’t, you should — it’s an excellent tool. In its free version, it provides you with 2GB of storage “in the cloud”, using a new kind of folder called a “Dropbox”. What distinguishes a Dropbox from other folders on your computer? The following:

• Every file put in your Dropbox is automatically (and securely) uploaded to Dropbox’s servers, ensuring you have an offsite backup of all data therein.
• Multiple computers can gain access to a Dropbox, ensuring files are automatically synchronized across computers without having to use complication version control systems.
• All files in your Dropbox are versioned, ensuring you can always recover an older version of a file in case you accidentally overwrite a good version.

Dropbox is supported on Windows, Mac OS X, and Linux, and now even has mobile applications, as well. Further, I have a special place in my heart for this service because I started using it almost 2 years ago, and it has acted as a file sharing and project management tool for my own startup’s internal operations at Parse.ly. I was therefore more than ecstatic to discover that this excellent tool and its smart founders had also made it through all of the hurdles necessary to get an early-stage company the financing it needs: they’ve raised over $7 million in financing and have over 3 million users.

But there is another reason I absolutely love Dropbox: because it was my idea. I invented it.

In the summer of 2004, I was really itching to get into Google’s Summer of Code competition. This was the summer I had taken a job working from home as the lead web developer at the Unemployment Action Center of NY. Though the job was great experience — letting me build my first full web application for a real client — I was itching to work on a technically juicy problem, something that affected me in my daily computer use.

And so, I sat down for a day and wrote up a Google Summer of Code proposal for a new system I had invented called Persistent Folders. It wasn’t exactly like Dropbox, but damn close. Even the implementation is close: Dropbox and my system both sync files using rsync, and both use a Python daemon process. The main difference is that since my system was meant to be open source, it did not require the use of a company-maintained service; instead, I proposed that users piggyback existing storage they have via web hosting providers.

Unfortunately, my project wasn’t selected.

Why am I posting this? I recently had a discussion with another engineer after I had discussed some of the technology behind Parse.ly with him. He was surprised at how liberal I was with explaining our internal implementation, architecture, and algorithms. He asked me, “Aren’t you worried that I could steal your idea?”

I responded, “You can steal it all you want; I dare you to try and implement it!” I then explained that to me, ideas don’t matter. I had the idea for a hundred startups that now exist before they started. I know from talking to users and customers of Parse.ly that they had our idea before we implemented it. What matters in software is not an idea, but execution of that idea. Ideas are a dime a dozen.

I began this post with the statement, I invented Dropbox. And now I’m here to tell you that it doesn’t matter one bit, because I never implemented Dropbox. And you can’t own ideas…

If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Its peculiar character, too, is that no one possesses the less, because every other possesses the whole of it. He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me.
-Thomas Jefferson

Imagine if when I had come up with this idea, I had patented it. Would it really have been fair to force Dropbox to license my technology? To sue them for “infringing” on my patent?

No way — these guys deserve the success they have. How stifling it would have been for me to put any roadblocks in their way because “I had the idea first”! The world is better because Dropbox exists. And, these guys have had — in my opinion — near-flawless execution. So, kudos to them.

Consider the following matrix:

	Mediocre Idea	Great Idea
No implementation
Mediocre Implementation
Great Implementation

A mediocre idea with a good implementation is worth infinitely more than a carefully-guarded, good idea with no implementation. Of course, the best products are both great ideas and great implementations. And I think my proposal for “Persistent Folders” — written three years before Dropbox even started — proves this to me in a very personal way.

Now, I’m certain the Dropbox guys never read my little proposal, because it was only sent to Google and otherwise sat on my hard drive for years, not viewed by anyone. I had a great idea, but no implementation. And the Dropbox guys took a great idea (one they arrived at on their own, I’m sure), and gave it the implementation it deserved.

For those of you who guard your ideas carefully, I’d suggest you stop wasting your time, get off your butt, and focus on actually building stuff. Because if you don’t, someone else will!

For the curious, below is my proposal to Google Summer of Code 2004, unaltered from its original draft. This is just a relic; for now, I’m glad to keep hacking on my own little idea, hoping one day I can look back and say I executed as well as Dropbox did…

PERSISTENT FOLDERS: A New Metaphor for Data Synchronization
May 13, 2004

The Problem

Computer users are more and more finding themselves with a serious problem: the fragmentation of personal data across multiple physical machines, and even multiple operating systems. Users who find it most comfortable to have a desktop machine at home and a mobile laptop computer “on-the-go” (for business or trips) have to deal with chaotic and often frustrating manual methods to copy that data to the needed places. Some users carry USB memory sticks — which hold their important “working set” of files — and either work directly off those disks (suffering reduced speeds) or make copies of the folders therein on their actual machines, as they become needed, spreading yet more copies and compounding the problem of synchronization. Others abuse ubiquitous technologies for accessing important files, by either e-mailing the files to themselves or by uploading them temporarily to web or FTP servers.

The end result of any of these methods is fragmentation of personal data, with error-prone manual processes for replication, and the inability for a consistent way to search, backup or even just keep track of all the important files and folders spread among the PCs in question.

What is needed to solve this problem, or at least make the problem more manageable, is an intelligent, user-friendly, customizable cross-platform program that allows for transparent synchronization of personal data across multiple computers and systems, either via a LAN or via the Internet.

Introducing the Persistent Folder

Up to this point, most users are used to understanding folders as existing solely in one location. The only way to get a folder’s data onto another machine is to copy that folder, thus making a duplicate. The goal of this project will be to introduce a new high-level data storage mechanism known as a Persistent Folder. Persistent Folders differ from regular folders in that they are meant to be transparently persistent, or synchronized, across different computers.

To explore a hypothetical example, imagine user Joe sits down to do some personal accounting work on Friday. He creates a folder on his desktop called “Personal Accounting,” and begins working on some OpenOffice spreadsheets in that folder. A few minutes later, Joe realizes that some of his accounting work will have to be done while he is away during the weekend, on his laptop computer.

To enable this, he simply right-clicks the folder on his desktop and says “Make this folder persistent across multiple computers.” When he does this, a dialog comes up asking him to type in a description of the persistent folder, and to select computers on which he wishes to make this folder persistent. Joe is presented with a dialog of computers currently available on the LAN. He sees his laptop, “MobileJoe”, and selects it. He then presses OK.

Within seconds, Joe sees the folder “Personal Accounting” appear on his mobile computer’s desktop, even though he hasn’t even touched his mobile computer yet. When he enters that folder, he sees the same spreadsheets that are available on his main PC’s desktop.

These two folders are now treated by Joe as “one persistent folder available across two computers.” He can add files to one folder and they will automatically be propagated to the other. He can modify files and the new versions will then exist in the other. Joe no longer has to worry about pushing his data back and forth across the computers.

Internet Synchronization by way of Ubiquitous Services

One of the major questions one may ask at this point, “that is all well and good for persistence of a folder on a LAN, but what about when I leave my home/office, and need to synchronize over the Internet?”

One approach would be to have a special server application with which all computers that wish to share persistent folders could synchronize. But then the user needs his own server, and needs to install my little server daemon on it. And not many users have their own servers on which they can just install any old application. So for normal users, this, in fact, is no approach at all.

Although most users don’t run their own servers, many modern users do have _server access_. That is, many users do have Web/FTP hosting providers to whom they pay subscriptions, and these servers power their blogs, personal photo stores, etc. The goal of Persistent Folders would be to allow users to piggyback their existing web services to synchronize their files, essentially turning a folder on that server into a Persistent Folder Repository.

A dialog in the properties of a persistent folder would include a checkbox that allows the user to “make this folder available over the Internet.” It would then allow a user to choose a method for making this folder available via the Internet, asking the user to provide a “persistent folder repository” via FTP by default (due to commonality), but equally possible would be scp, rsync, NFS, or even something like cvs/svn (the goal, of course, would be to make the design modular enough that it could support any server type with basic file system operations).

Then, every other visible machine on the LAN who shares that persistent folder would have option enabled automatically. If the machines are offline, the option can be entered manually by the same method explained above.

From that point on, the folders become persistently available and transparently usable, just as before. If the other computers are available on the LAN, then the program utilizes LAN speeds and synchronizes directly. Otherwise, it synchronizes with the Internet Repository.

Lucky Joe is now able to work on his files in the office, leave his laptop there (“the darn thing is so heavy to carry around!”), return home, and continue right on working on those files which are now found in the “same folder” on his desktop machine at home.

Implementation Ideas

I plan on implementing this idea in Python, since one of the major goals of the project is to allow Persistent Folders to exist not just across computers, but also across operating systems (so that a Windows desktop could have a persistent folder that also exists on a Linux laptop, for example). For a user interface, I plan to use PyGTK, since that’s what I know, and since it is also cross-platform. I need something powerful like GTK since there will be times when user intervention will be necessary, but since it is a goal of this project to reduce the number of times the user must intervene, I want to make sure that when he does, he is presented with sane, human-readable, and user-friendly dialogs.

I don’t plan to reinvent the wheel. Most of the magic of Persistent Folders is just making existing synchronization tools work relatively silently and in a way that makes sense with a user’s workflow. The main tool I am thinking of using is rsync, the relatively-ubiquitous UNIX utility for incremental synchronization of directory trees. I considered using the unison, as recommended by Ubuntu’s Wiki entry, but I saw two major problems: (1) unison is no longer under active development and (2) it is written in OCaml, a relatively obscure language which I don’t know. Therefore, any bugs I discover in unison would not be fixed in a timely fashion by its developers, and any features I’d like to add to support my idea would be quite difficult to implement.

As for monitoring folders for changes, I imagine the most elegant solution would be to take advantage of inotify under Linux (like Beagle does) and perhaps handle ChangeNotify events under Windows. I’d really like to avoid polling, since polling is just plain evil for something as potentially neat as this. Regardless, I’ll probably need to code Yet Another Daemon (or Windows service) to keep track of Persistent Folders on the local machine and their equivalents across the network.

Finally, for secure synchronization of files, I plan to use SSH tunneling wherever possible, which is available under Windows under the OpenSSH for Windows Sourceforge project, http://sshwindows.sf.net. Linux distros like Ubuntu, of course, have all the ssh support one needs.

Conclusion

This project aims to introduce a new metaphor users may utilize to share important files across multiple computers: the Persistent Folder. A Persistent Folder is not a shared a folder; rather, it is seen as a single folder that exists locally across multiple computers, and can be treated by the user as such. Changes at any one folder rapidly propagate to the others. Properly implemented, this may provide a better way for users to manage important data that might otherwise be scattered, fragmented, and even lost through the daily shuffle of file transfers across networked PCs.

Appendix A: Project Roadmap

o June 24: Begin Work, with ideas now fully developed in the form of documents. Post these documents to Ubuntu’s Wiki to encourage ideas from community.

o July 5: Have a console synchronization wrapper and some network discovery stuff in Python written, and have UI concepts designed in Glade.

o July 20: Make 0.1 (GUI and basic features working) release, so that Google can show it off at OSCON?

o August 1: Have other great features, like the Internet synchronization and multi-protocol support, in 0.2 release.

o August 10: Consider working on features to allow transparent backup and versioning, to include with bug fixes in a 0.3 release.

o August 20: Make simultaneous 0.4 releases on Linux and
Windows, hammering out as many cross-platform issues as possible. Add inotify/ChangeNotify support if not already there.

o August 30: The big 0.5 release, finished for Google’s deadline. Let Google/Ubuntu make the decision if it’s worthy of being renamed a “1.0″ release.

o September 1: Live a less stressful life, since my files are now neatly synchronized among my PCs! But I’ll keep making it better.

Appendix B: Hold On, Isn’t Samba Good Enough?

One common response to this project may be, “Aren’t Samba and SMB shared folders good enough?” Though Samba is good, I do not believe it is good enough. Here’s why:

(1) Samba does not aim to present the user with a metaphor of a folder existing in multiple locations at the same time. I believe this metaphor would be appreciated as powerful by longtime computer users and subconsciously acknowledged as highly usable by novices.

(2) Samba only allows users to share folders on their drives. Other Samba users may then mount those folders via the LAN. This two-step, asymmetric process already seems complicated and convoluted to end users. But more importantly, synchronization is left up to the user. In theory, users could avoid synchronization altogether and work on the files directly, via the share. In practice, LAN speeds are not adequate, and other issues are raised (such as file locking and write conflicts). This forces users to home-brew their own synchronization protocol to make sure duplicate files at different versions doesn’t result in an accidental loss of data.

(3) Samba does not provide any easy method to access personal files once one leaves the LAN and enters the WAN, short of opening up a bunch of ports on your router and trying to connect in from outside (a very slow and insecure method).

(4) Samba provides no recourse when a computer is no longer available. Persistent Folders, on the other hand, make data available “offline” by design.

(5) Samba does not know whether two duplicate files exist across the network. Therefore, Samba cannot be immediately utilized as a transparent form of backup.

Is the Persistent Folder meant to replace Samba? Of course not. I believe Samba has specific purposes: to allow for easy, fast, one-time transfers of files across a LAN, and to allow networked printer and device sharing. I do not believe Samba is an adequate solution to allow a user to treat a folder as if it existed on multiple computers at once.

Appendix C: Versioning’s the Thing

Upon further contemplation of this project, I realized I had left a question unanswered in this document. What is the right way to deal with version conflicts during the silent synchronization phase among Persistent Folders?

Imagine user Joe works on a file in a Persistent Folder which exists on his laptop and desktop computers. The first version (let’s call it version 0.1) was created on his desktop, and Joe had the laptop on the same network, allowing automatic synchronization to occur. But then Joe’s laptop was disconnected from his desktop, as Joe was without network or even Internet access. He works on the document a bit, which can now be called version 0.2. But then Joe forgets that he worked on the document on his laptop, and when he gets home, he works on his desktop computer, creating a version 0.2 there too. Which version 0.2 should be synchronized?

Normally, the answer would be to pick the most recent one. But we don’t want Joe’s laptop work to be lost, just because Joe forgot about his prior work, do we? Well, what needs to happen is a bit of smart conflict resolution with no data loss.

I propose that Persistent Folders should also be versioned to a sane degree. Perhaps by default 3-5 versions of files are always kept, with the most recent versions visible directly in the persistent folder, and other versions buried (in .dotfiles or hidden folders) behind there. Then, the Persistent Folder monitor should inform Joe (“whisper” to Joe via the notification area/systray) that there was a conflict, and the newest file was chosen. But at any point, Joe can choose to revert a file back to an older version. The interface should be such that Joe can even see where the file was edited, for example:

Personal Accounts.xls – Prior Versions
o version 0.2: edited on MobileJoe
o version 0.2: edited on DesktopJoe
o version 0.1: edited on DesktopJoe

Of course, versioning could be disabled (at risk of data loss to the user), or could be enabled as “smart versioning” to only version files when conflicts occur during synchronization. Some may say that versioning is dangerous because it can drastically increase disk usage, but as I mentioned above, the other benefit is that the user gets redundant network backup of files for free. In higher versions of Persistent Folders, things could get more sophisticated by allowing users to turn off versioning even at the file-level, but I don’t think that’d be necessary in the first few releases.

So, for those not keeping score, Dropbox supports:

• Windows, Linux and Mac OS X syncing, using a scheme similar to that described above, except with a managed set of servers (Dropbox’s) acting as the “repository”
• sync over LAN (in latest beta releases) when PCs are local
• file-level versioning
• Collaboration by sharing folders with other users — a feature I didn’t discuss in my proposal, but that would be a clear next step for a managed service

Go Dropbox!

960.gs is a CSS grid framework, similar in spirit to Blueprint CSS and YUI Grid. However, 960.gs is at once more minimalist than these approaches, and more thorough.

The author has a detailed blog post explaining his motivations for working on 960.gs, so I won’t rehash each of those. Instead, I’ll just dive into what I liked about it.

1. Very simple and comprehensible name choices for CSS classes: You can see their demo page (view its source) for examples, but I can even summarize the approach in a few short sentences. You begin a grid container with a container_XX class, where XX is the number of total grid columns in the container. Every child element has a grid_XX class, which is the number of columns (aka colspan) for that element. All grid_XX elements should add up to the XX amount in the container if you are using all the grid elements available. To use larger gutter widths, you can use prefix_XX and suffix_XX classes. Finally, for any child grid elements, you need to mark the first one with an alpha class and the last one with an omega class — this is necessary to get rid of left/right margins on starting and ending elements.
2. Excellent CSS generator: this online CSS generator tool lets you generate 960.gs CSS that is just right for your website.
3. jQuery-based bookmarklet for double-checking grid layouts: another online tool is a bookmarklet which will “inject” a customizable grid overlay on any existing site, to help with alignment.
4. Templates for every major design tool: I saved the best for last. Nathan’s focus in designing 960.gs was to allow better collaboration between trained graphic designers (whose tools are typically Adobe Photoshop, Fireworks, Illustrator, etc.) and trained web designers (whose tools are things like Rails, Grails, Django). To ease this collaboration, the author provides templates for every major graphic design tool that has “grid overlays” compatible with the 960.gs framework.

Finally, the entire project is hosted on bitbucket and developed in the open. What more could you ask for? This has really simplified my approach to standard CSS designs.

I immediately called Chase Customer Service. In order to confirm the details about my account, the representative needed me to identify the fraudulent charges, but also identify charges that were actually valid. For this latter bit, I needed to identify the time/place of a specific transaction. This card was mostly used for online auto bill payments, so this turned out to be impossible for any of my last 20 valid payments. Yet the customer service rep insisted that I name a time and place. I told her, “The time and place was whenever the server for this system decided to automatically bill my account. I don’t know where their server is, I don’t know what time their cron jobs run.”

“Cron jobs?” she said.

Right, I had been hanging around techies at DreamIt Ventures for too long. “Listen, the transaction didn’t take place physically, it took place digitally. I can identify one transaction, which is about a month old, where I actually used the card in-person to buy something.” She finally understood and let me move on.

Burak from Trendsta said he felt bad for me, for how patient I had to be with this person. But that was the least of it. This little technical misunderstanding was nothing compared to what followed.

I was told that in order to get a credit back from my account, they had to collect from me a signed affidavit indicating the charges were fraudulent. This affadavit would be “securely shared” with me via e-mail. OK, “sounds good” I said. I waited around for the e-mail to come in.

Finally, two e-mails arrived in my inbox. The important bits are in red. First:

Message from Chase Customer Claims Secure Document Exchange

From: chase_customer_claims@secure-dx.com

Welcome to the Chase Customer Claims Secure Document Exchange. You recently contacted Chase regarding your claim number XXXX. Your documents are available for your review.

Per our telephone conversation, you will need to register to our secure website.

Your initial password is: password

Your initial user name has been sent to you in a separate email.

On your first log in, you will be required to select a new password.

Thank you for using Chase Customer Claims Secure Document Exchange.

To contact Chase for claim related questions or to withdraw your claim, please call 1-866-564-2262.

Any geek reading this will immediately identify some key things wrong with this e-mail that make it look like a total phishing expedition. Namely:

1. The e-mail address, rather than being from a chase.com domain, was from a strange domain named “secure-dx.com”.
2. Rather than sending a cryptographically secure, expiring activation link, a default password was sent in plain text.
3. To make matters worse, the password is the same for all users, and thus anyone who can guess my e-mail address can easily impersonate me on this “secure document” website.
4. The default password is “password”. WTF?! I mean, c’mon?

I didn’t quite understand why I needed a “second e-mail” now, but I opened it up. Here it is, excerpted:

Your Chase Customer Claims Secure Document Exchange Electronic Package is available online

From: chase_customer_claims@secure-dx.com

ANDREW MONTALENTI,

Welcome to the Chase Customer Claims Secure Document Exchange.You recently contacted Chase regarding your claim number XXXX. Your documents are available for your review.

Per our telephone conversation, you will need to register to our secure website by clicking on the link below or copy and paste the link into your browser’s address bar.

https://chase.secure-dx.com/consumerdcx-chase_atm

Your user name is my.email@hidden.com

Your initial password has been sent to you in a separate email

On your first log in, you will be required to select a new password. NOTE: This site is different from Chase.com and passwords are not related. Updating your password on Chase Customer Claims Secure Document Exchange will have no impact on established Chase.com passwords.

Once registered, you will be able to access your customer correspondence on our secure website. You may be offered the option to complete and sign the form online if you wish to do so. [...]

To say I was confused would be a major understatement. I was downright depressed.

My guess is that the engineers at Chase thought that by separating the “password e-mail” from the “user e-mail”, that somehow made the whole communication more secure. Two e-mails are better than one, right?

The most important thing to point to is the link. The link where this secure communication will happen is not at the chase.com domain Instead, it is at https://chase.secure-dx.com/consumerdcx-chase_atm. There is no way, NO WAY this is a real Chase site, I think.

I click on the link and in Firefox, I see this:

At this point, my paranoid self turns on. Curious, I click through the link anyway. And I see this:

Now I’m really paranoid. Links off secure-dx.com pointing back to chase.com’s privacy policy. A username and password box and a sort of hokey imitation of the Chase.com web design. I realize, holy shit, I’m being duped! Not just small-time credit card fraud, but someone has managed to really take over my life!

Why am I freaking out? The customer service person I talked to, I realize what must have happened. That wasn’t Chase. Someone stole my credit card information and then set up a call forwarding on my cell phone, somehow, to point Chase’s customer service number to some fraudulent interceptor. This person then diligently took my claim only to send me an e-mail that would get yet more information out of me and take me for even more money. I freaked!

Immediately, I double-checked my call logs and compared them to Chase.com customer service numbers. I made sure to change my DNS server to OpenDNS to make sure no one was somehow intercepting that. Finally, I realized I could look at the number written on the back of my Chase credit cards. It all checked out — the number was good. So I switched phone. I called Chase customer service on both my phone and Olivia’s. I made sure the messages were exactly the same. From Olivia’s phone, I called back Chase again to speak to someone there about this. But then I got even more paranoid — how big could this be? — so I decided to hang up. Instead, I called my local Chase branch in my neighborhood.

With my local branch’s help, I got transferred via a branch office line to the actual Chase customer service. Finally on a secure line, I thought to myself. When they picked up, I was expecting to uncover the scam of the century. I felt like an investigative journalist right on the tail of something truly big.

But then I spoke to the Chase representative, on the secure line, and she explained to me that this is just the normal procedure. secure-dx.com is the website they use for “securely” sharing documents.

I was livid. I explained everything wrong with this setup. I demanded to speak to a supervisor. I spoke to a supervisor. He said he did not know why the system was the way it was. He wasn’t a software guy. He just knew that “with the way the business is changing lately, a lot of systems are in flux.” I said this flux was unacceptable. “I’m a software engineer,” I said. “How can I possibly trust Chase to manage my financial accounts if something as simple as sharing a PDF document is done in the least secure way possible?” What other skeletons might they have in the closet?

I wanted to be forwarded to the department responsible for that. After my explanation to him of what was wrong, he fully understood the problem. To his credit, he admitted it was wrong the way it was set up. He actually tried to track down a supervisor. But there was none that could field IT and software requests.

They promised to call me once they could track someone down to talk about this. No call yet.

My excitement came down a couple of notches. I was not the investigative journalist undercovering an elaborate scam any longer. Instead, I was a software engineer. And some members of my profession have let me down. Big time.

In the meanwhile, I did the research and found the vendor who provided this service to Chase. They are Wolters Kluwer, a “financial services and banking compliance solutions provider”. The product page for “SDX”, Secure Document Exchange, is completely ludicrous. They claim this product includes “industry-leading security, including PKI encryption and multi-level user authentication, to keep communications safe at every step of the process.”

Right, so the password was sent in plain text. The default password is “password”. And, rather than having a chase.com subdomain which points at Wolters Kluwer’s server (e.g. secure-dx.chase.com) and sharing a secure chase.com certificate with them, they decide to host the whole thing outside of the chase.com domain, so that as a user, I have no way of confirming this actually is an e-mail or system originating from Chase. Users are so confused by this that they have already reported it as a phishing scam, even though it is not one.

That’s industry-leading? That’s “safe communication”?

No, that’s a joke. Chase should be ashamed.

1. A Whirlwind Excursion through Writing a C Extension. This talk by Ned Batchelder (author of coverage.py and cog) shows that you can write a Python C extension module in under 20 minutes. This is my top talk because I never thought that my skills in C would be so directly useful in writing Python applications. Considering how damn easy it is to write a basic C extension module, I wouldn’t be surprised if the only reason I ever write C code again is to implement some Python functions or types in C. Truly the best of both worlds!
2. Reinteract: a better way to interact with Python. Owen Taylor (of GNOME/GTK+ fame) has spent some time over the last few months building a better Python shell. Specifically, it’s a lightweight shell that is meant to be a prototyping or “worksheet” environment a la Matlab, Mathematica, or Maple. Except, you’re running and re-evaluating Python code. It even supports things like in-line graph plotting, but I’ve already used it to experiment with Python web services API. Any Python programmer who has been frustrated with IPython before should check out Reinteract.
3. Easy AI with Python. This talk might have gotten the #1 slot for most interesting, but not the #1 slot overall because it seems like this talk has been given at a lot of conferences (not just PyCon) over the last few years. This talk introduces some complex AI topics in a very short time frame, and in a very intuitive way. For me, the neural networks example with Jets and Sharks was particularly impressive. Raymond Hettinger is a great presenter, and if you have some time you should definitely check out his recipes on ActiveState’s Python Cookbook and his How-to Guide for Descriptors.
4. Abstraction as Leverage. A talk by one of my favorite Python authors, Alex Martelli (who wrote the best book on Python on the market, Python in a Nutshell), this talk isn’t so much about Python as it is about software engineering overall. But it’s thought-provoking as his talks usually are.
5. Class Decorators: Radically Simple. The presenter is the author of the Class Decorators PEP, Jack Diederich. If you like decorators and you are curious about metaclasses, you’ll love class decorators.

Feel free to share your favorites!

I highly recommend the entire book. It is not so much a book about beautiful code as about passionate and opinionated programmers and their tastes. But this is a good thing. It was one of the few books about software that I have read in the last decade or so that actually gave me entirely positive feelings about my profession. There is so much raw creativity and thought captured in these few essays. What Brian Kernighan finds beautiful is entirely different from what Matz or simonpj find beautiful. And that’s the thing about a fundamentally creative craft like software. You put five software engineers in a room with a piece of code, and you’re lucky if you come out with only six different opinions about it. It’s like art, or writing. Taste matters.

I don’t recommend people read Beautiful Code to try to imitate some of the code described therein. Instead, I recommend you read it as a sociological or psychological study of what makes proud and bright software engineers tick. For example, for Kernighan it is the simplicity and minimalism that is embodied in UNIX. For Matz, it is the notion that the programming language should be as syntactically flexible as our real languages are. For simonpj, it’s that complicated can be made easy, given the right abstractions. And for Jon Bentley, in one of the more thought-provoking essays in the book, beauty and elegance was only perceived as the size of his code shrank.

The essay about the SVN Delta Editor not only illuminates the internals of SVN, but also illustrates the social dimension to software engineering and design. It is a story about programmers, debating an API, producing it, and then putting it into practice. It is about give and take, and an unteachable skill in problem size and complexity reduction. All this in C! There was a period of time in university where I actually programmed in C full-time, so I have a lot of respect for the elegance with which they crafted this powerful API. C gives you few tools (like OO or explicit interfaces) for doing this kind of work; they had to work in spite of the language’s features and plan carefully.

I have about twenty students in my class, so I was going to print up one copy and get it copied and stapled at a local print shop. (See note on copyright below.) I opened up my ebook PDF of Beautiful Code with acroread on UNIX. I navigated to the right chapter and realized that I wanted to print just that single chapter. I always remember being annoyed whenever I had to do this, for a number of reasons.

1. PDF ebooks sometimes lack the proper “bookmark” information to navigate to the right section to print
2. Since ebooks were once print copies, they tend to have page numbers at the bottom of each page. But since the ebook itself has a different page numbering scheme, all sorts of psychic dissonance occurs. You navigate to page 30 (in the print copy) but have to note that it’s actually page 42 in the ebook. You then navigate to page 45 (in the print copy) but have to note that it’s actually 57 in the ebook.
3. OK, now I know what I need to print… I think. So now I have to enter one of those print ranges in the “Print” dialog. Is it 30-42? No, wait, it’s 42-45… I mean, 42-57 — that’s it! Is that inclusive or exclusive? Oh, my…

It’s really not that bad, and it’s only an occasional annoyance, but it’s always there. I’m sure you know what I’m talking about.

I had recently upgraded to acroread and noticed that the UI was all spiffed up. And I noticed that this ebook had the right metadata for the bookmarks. I thought to myself, “Wouldn’t it be nice if acroread supported printing a chapter?” I right-clicked on the first entry in the chapter bookmark and was astonished. Lo and behold, my feature existed! (See the image to the right.) I clicked the “Print Pages…” button with a bit of discomfort. I don’t trust software too often, and am always suspicious when I find a feature I didn’t expect to be there. It’s like my inner programmer is saying, “Yea, right — too good to be true.”

A few minutes later, my chapter was printed. I looked it over, and brought it with my other materials to the local print shop. One hour later, I picked up my copies and brought them home to look them over.

I noticed something very strange. Instead of my copies containing pages 42-57, they contained pages 42, 43, 46, 51, 55, and 57. Damn it. There didn’t seem to be much of a rhyme or reason to the pages that were selected. What kind of sequence was this? I felt that there must be some pattern, some fibonacci-like, non-obvious sequence that applied to these pages. I suspected the first, and obvious, culprit: that the printer had made a mistake. Maybe it’s a human error. But then I looked over my original and indeed, the original only had those pages. Not a human error. I thought to myself, “How is this possible?”

Of course, I’ve probably given you enough information that you’ve already figured it out. Especially if you’re a programmer. We’re just wired to think this way. But in case you haven’t figured it out, I’ll indulge you.

When I went back into acroread, tracing back my steps, I noticed something about that menu item I clicked. It didn’t say Print chapter. Instead, it said, Print pages. Now, conceptually that seems like a small distinction, but I picked up on it.

I started to think like a programmer, rather than a user. This function with a for loop emerged from the program and hovered above it, almost magically. It said:

def print_pages(self, selected):
    to_print = []
    for bookmark in selected.self_and_bookmarked_children():
        to_print.append(bookmark.page())
    PrintSubsystem.queue_job(to_print)

Then I realized the pattern in the pages it picked. There was no pattern. This was a beautiful little bug. A butterfly.

You see, within the narrow world of this Print Pages function, the “feature” works as expected. But from a user’s perspective, it makes absolutely no sense. Rather than printing everything from that bookmark to the next bookmark at the same level (that is, rather than printing a chapter), it printed each individual page that happened to be physically bookmarked (or ‘sub-bookmarked’) in the PDF, at or below that level. This resulted in a bunch of pages being printed that happened to be the pages on which subsections began. But this left out most of the chapter, somewhat randomly.

The worst traits of our profession come out when it is at its least social. I have no doubt that this function that prints these pages was written by a single programmer in a windowless room, without any peer review, pair programming, or other check on his logic. I am sure that he was given the narrow and ill-defined requirement to enable an action to “print bookmark pages”. He needed to think, but instead, he decided to code. And coding got “it” done, for some very weird value of “it”. He was probably under time pressure. But one thing is certain to me: he was alone. No two programmers, debating the design and implementation of this feature, would let each other make this mistake.

The behavior it exhibited truly caught me by surprise. Strange as it sounds, I admired how easily I had been duped by this feature. The human error — the anti-social error — made by that programmer exhibited an odd and enigmatic computer behavior. A human inelegance created a strange sort of cruel machine elegance.

I found it ironic that in trying to print a chapter about beautiful design from a book called Beautiful Code, I came across this beautiful bug. I call the bug beautiful because it managed to fool me, to get me to suffer its wrath while thinking I was getting some convenience. It exhibited behavior that challenged me to identify a pattern, where there was none. It was so clever, it even cost me money (the printing charges). And even though I was a discerning programmer — skeptical of the feature, and so unsure of the software’s operation that I checked the output, albeit too briefly — this little bug managed to outsmart me.

My students will have to live without the chapter, or read it online on their own. I’m not upset about it. There can be beauty, even in failure.

A note about copyright: some readers on reddit and on my comments section suggested that I might be ignoring copyright issues by thinking that I could just photocopy a chapter from this book to distribute to my students. Trust me, I know about copyright. The content of this chapter happens to be available online for free under Creative Commons Attribution 3.0. Karl Fogel, the author of the article, has even given his informal blessing in my comments section. And finally, by most people’s interpretation of the rules of Fair Use, it was OK for me to copy a chapter for my classroom. I’m surprised no one suggested that this bug might be beautiful in another way: that it saved me from a copyright disaster. I don’t think it was that good… If you are really interested in seeing a debate about this, you can read this thread on reddit. Warning, somewhat painful and longwinded. (Also, if you clicked the anchor link within the article to get to this note, simply click “back” in your browser to return and continue reading.)

This trip emerged as a major convenience for me. My brother was heading out to California for the Real Estate Connect conference, since his company is intimately involved with Real Estate technologies. But he didn’t want to go alone, so he paid for my plane ticket and hotel stay the first few nights to convince me to come out. (Thanks, Alex!)

I got in touch with some people I knew out there, but many of them weren’t going to be around during my stay, due to summer vacations, etc.

Joseph Sofaer, a friend of mine from NYU, was gracious enough to delay a trip to Europe to show me around the area. I was even lucky enough that the first first day I was there, he invited me to a party at the Facebook headquarters in Palo Alto, CA, where I briefly met Mark Zuckerberg, saw Robert Scoble walking around, and met about twenty other software developers in a room of 300, who worked for companies like Yahoo, Pandora, del.icio.us, and, of course, Facebook itself.

The train ride home, I was riding with the creator of the MillionDollarHomepage, probably the best website idea of all time.

Palo Alto and Software is Sexy
—
The vibe in Palo Alto was enviable, especially for a native New Yorker. When I pull out my laptop on the train or in a cafe in NY, screens full of code or running with full-screen vterms, I get stares and strange looks. (Including right now, as I write this blog post on the LIRR…) In Palo Alto, my brother and I walked by two or three cafes, and in each I saw young guys and gals at their laptops, coding away, screens full of GNU make output scroling by, emacs and vim buffers splitting off. I overheard conversation snippets like, “we’re hoping to release our next version in a couple weeks”, “you have got to try metaclass programming with Python”, and “did you see that article on /.?”

In New York, people aren’t that proud to be a geek. You go to a typical gathering and say, “I’m a software developer” and the typical response is, “Oh…”, followed by silence, followed by, “I think I need another drink…”. Sexy in New York, programming isn’t. If you want sex appeal, go into finance, law, or politics.

Opposite in Palo Alto, of course. Which makes you feel great as a software developer. What are you into? If you answer, “Stocks, bonds, and the market” you’ll get the treatment I described above. You answer, “Ruby on Rails” and you’ll be having a 2-hour, heated conversation.

BTW, the Silicon Valley excitement about Ruby on Rails is palpable. Everyone wants a Rails developer.

A quick anecdote to illustrate the NY disdain for geeks: I was out with a friend once who had attend culinary school and was trying her way up the restaurant ladder, to become a chef. She complained, however, that the money in these entry level jobs (food preparer, etc.) in the kitchen was way too little, that it was difficult to get by. So, someone suggested, “Why don’t you waitress on the side?”

Her response, “Waitress? I’m trying to become a chef. That’s like saying to someone who wants to run a company, ‘Why don’t you work as an IT guy?’”

Something tells me that joke wouldn’t work that well in Palo Alto.

San Francisco and Good Coffee
—
It was great to be able to get a little sense of the Silicon Valley vibe on my first day in Northern California. But by the end of the day, my head was spinning. I was ready to head back to an environment that was less technobubble, something a bit more real.

My second day there, I walked around some neighborhoods on my own, particularly Union Square (which is a bit of a tourist center), the Mission (which reminds me of Astoria, but with a different set of ethnic groups), and Noe Valley (which reminds me of the Upper East Side, but with houses instead of apartment buildings).

In the Mission, I ended up stopping by a coffee place named Philz. I asked for an iced coffee. I then watched as the barrista (who I think was Phil!) took a scoop of coffee beans, ground them in a grinder, put them in a coffee machine, and made a fresh cup of coffee. Some ice went in with frothed milk atop it. I kid you not, the BEST iced coffee I have ever had.

Later in the day, when I met up with Joey at his apartment, I mentioned that I had just stumbled upon the best iced coffee of my life a couple hours ago. And he said, “Oh, man… if you want good coffee, you have to try this place Philz.”

I laughed and said “THAT’s exactly where I was!” and he responded, “Man, you had ‘just stumbled upon’ possibly the best coffee in the world.” So that was a bit of good luck. We ended up going back to Philz later that day

Joey had to leave the next day for Europe, but we spent the rest of the afternoon chatting about software ideas, in particular details about his latest idea, a Blackboard replacement application hosted on the Facebook Platform. If you’re on the Facebook, check out the application called “Courses”. It’s quite cool, even if it is in the early stages.

The Weekend
—
My brother’s real estate conference was winding down, so he snuck me in so I could catch the last couple presenters. Unfortunately, I had missed the presentation by Craig Newmark (of craiglist fame), which was apparently pretty good. Instead, I caught a panel by some head honchos of real estate technology at Google, Yahoo, Microsoft and Realtor.com.This panel was mighty uninteresting. Despite the fact that these guys were all supposed to be team leads and head-techies-in-charge, they all gave tired lines and made me question whether they actually knew what they were talking about.

Nonetheless, it was interesting to see a conference of this kind — lots of buzz, lots of software developers all gathered together, a lot of excitement about an emerging market.

That afternoon, we took a bus tour of San Francisco, which was well worth it due to the hilarious tour guide named “Silvio.”

We spent that night at a Vietnamese Restaurant named “La Colonial“. Excellent.

Saturday morning, we walked around the city, starting out in Union Square, moving through to Civic Center. We stopped at the Asian Art Museum, where I saw an exhibit on Manga and Japanese Wood Block art, both very good, among others. Finally, we ended up in the Lower Haight and Haight Ashbury, where we spent the afternoon lounging and exploring.

At night, we headed to North Beach and ate dinner there. We ended the night at the bar at the top of the Sir Francis Drake hotel, which my brother said reminded him a bit of Windows on the World, when that was still around.

Conclusion
—

For a short trip, I got to see a good chunk of the city, and even got to take that important visit outside the city. Overall, I had a great time.

As a technologist based out of New York, I have to say that San Francisco is quite appealing. The weather, the laid back attitude, the independent shops, cafes and restaurants, the technology buzz. But moving there would certainly seem like selling out. Sure, we may not be able to reproduce Silicon Valley but we can do our own thing. New York is too strong a city to be left in the hands of financiers and lawyers.

Dilbert’s pointy-headed managers are everywhere. In my current Fortune 100 company, virtually none of the managers with any authority have ANY IT development background. They manage entirely by cost and project plan – ignoring any and all input from those developers who actually have a successful track record.

I can’t say this is true in my team, at my company, but I have certainly heard it from a lot of my IT friends. Also, I have an acquaintence who is an IT Project Manager who thinks that development is “easy work” and that most software developers are just “lazy”, which is why projects end up behind schedule. I think many innumerate IT managers share this opinion, and this can lead to problems, low morale, and resentment.

Today, I did some digging on the Metacity bugzilla to see if it was fixed, and found this bug.

Over the course of 3.5 years, this bug has sat on the bugzilla, and still isn’t satisfactorily resolved. There is now a reduced_resources flag in gconf, but this flag only disables the minimize animation at the expense of forcing you to use an ugly wireframe window dragging animation. (Complete, utter insanity.)

I don’t understand how after two years of debate, and nearly 3.5 years down th road from when this bug was first opened, the final fix was something that throws the baby out with the bathwater.

I don’t want the minimize animation, but I don’t want wireframe windows, either. Why can I only opt out of them as a couple?

Havoc, you asked for rationale and not rant, and lloyd had provided it:

the window minimization animation’s sole purpose is to indicate to an absolute GUI neophyte “where the window goes” when it’s minimized. for a huge number of users – basically anyone who’s ever used a GUI with a taskbar before, which is in excess of 98% of current desktop computer users – they are never useful because they already know this.

for the remainder, window animations are useful exactly once in their lifetimes – the first time they minimize a window and wonder where their application window “went”.

thereafter, window animations are a distracting waste of time, and to me and many others, an annoyance. it’s roughly equivalent to never being able to turn off Clippy.

I wholeheartedly agree with this analysis. Minimize animations are useful to newbies, not useful to anyone else.

The lack of an ability to disable this animation (alone, not in a catch-all flag that enables wireframe drawing, for crying out loud!) is completely absurd. On my Core Duo 1.6 Ghz machine running an Nvidia graphics card, this animation still feels slow and clunky, and is completely superfluous.

In the future, maybe we’ll be able to use compiz and get nice fade in and fade out that works smoothly and nicely. Even then, I’ll still want to be able to disable it, because it serves no purpose for me, and isn’t central to metacity’s operation.

In the GNOME community, we’ve heard mention of “Nat’s Pendulum”, referring to Nat Friedman’s insistence that the GNOME “usability pendulum” has swung too far in the other direction. The 3.5 years spent blocking on the inclusion of this simple gconf flag, was, I think, a perfect illustration of what Nat meant. The fact that the flag still isn’t there in the form suggested may mean that the damage may just be permanent in this community surrounding metacity development.

Why must gconf be “usable” too? Don’t give me a reduced_resources catch-all flag, but give me separate flags to enable/disable certain animations/effects. Is this really such a usability disaster, or are you just being a Metacity Nazi:”no fine-grained configuration for you!”

As Jeff Waugh has mentioned, religious usability insistence like that expressed in this thread was enough to send many would-be GNOME users and contributors packing for projects like KDE, XFCE, or even to proprietary platforms like OS X. The damage is deep and felt. I lost a lot of faith in GNOME personally because of these issues, and am only slowly regaining it.

Has anyone already written a patch which removes the reduced_resources flag and adds this fine-grained control of the minimize and wireframe animations? If not, I’ll write it and attach it to this issue.

I will not open a new bugzilla bug for this, because I think history is important so that we learn from our past mistakes.

Asking me to get off my a$$ and code drivers for this baby is what I consider elitist and a very unreasonable demand on the end user. It’s one thing that gives GNU/Linux zealots a very bad name in the real world.

I have sneaking suspicion that you get this response from the Linux community because we feel you’re placing unreasonable demands on us. Your points are valid, but your energy is misdirected. Unfortunately for those who don’t like to code, that’s how software is created and improved. We invite you to participate in our projects in a variety of capacities including but not limited to programming, but of course participation is not required.

I think that Linux “zealots” get a bad name because much of the “real world” believes in a culture of entitlement. Look at everybody living life with a chip on their shoulder, blaming everyone else for their problems and scoffing at the notion that they take responsibility for their own situation. Somewhere along the line we stopped believing in opportunity as a means of realizing our dreams and began to foster the idea that we’re entitled to our expectations. In “Linux land,” we believe that the opportunity to participate in our information society is fundamental to our inherent desire as human beings to better our situation and control our own destiny.

Of course, money can make just about any dream come true. Mark Shuttleworth, for [example], invested $10 million to help make the Ubuntu project a reality. But years ago my great-grandmother told me the story of how my family came to America with nothing but the promise that here they would find a land of opportunity. This is the same promise we make with free software. This isn’t elitist, this is egalitarian.

I have to say, this is part of what makes me love Free Software.  It’s this idea of widespread opportunity.  Sure, F/OSS has power structures and means of coercion/control built into certain parts of it, but for the most part, it’s based upon a very simple, powerful, and egalitarian idea.  “Anyone can improve this, anyone can make it better.”  It’s that kernel of an idea that makes any process — whether software production, book editing, encyclopedia editing, or even beer brewing, more enjoyable to those involved, and, as a side effect, better for the general public.

I have been perpetually delaying a review of Capitalism 3.0 and Dreaming in Code, both of whom deserve it. But I promise one soon. I use Hofstadter’s Rule of Thumb lately for estimating time: however long you think it’s gonna take, double it and add a unit of time. So if you think it’ll take two hours, it’ll really take four days. If you think it’ll take five days, it’ll really take 10 weeks. And so on.

In the meanwhile, I’ve been busy at work — actually working on some cool stuff from a technology standpoint, mainly in the realm of hacking with pieces of the Eclipse Modeling Framework, and its related projects like GMF, RCP, Eclipse Core, etc.

On my commute, I’ve been enjoying reading Making Globalization Work by Stiglitz. Although one of my friends mentioned to me that this book would be quite boring, and for the most part he was right. Not the lofty stuff of Barnes in Capitalism 3.0; but perhaps Stiglitz’s recommendations are much more practical for ways to improve the current system.

The other book I started recently is a long, written interview with John Kenneth Galbraith (much in the style of Socrates) which is entitled, Almost Everyone’s Guide to Economics. What’s amazing is to see Galbraith, this towering (literally) Keynesian economic thinker, speaking in the 70s of the growth of corporate power, the undermining of labor, and the insidious nature of market fundamentalism. And yet, here we are, 30 years later, heeding none of his warnings, and entering into the new “global age” of “The World is Flat”.

Oh yes indeed, I do need to write some reviews very soon.

I’d post a longer review, but I’m headed down to New Orleans today. Will post a longer review when I get back, hopefully also of Capitalism 3.0, whose ideas have been swimming in my head the last few days of commute.  I think they really deserve to be summarized and presented here.

In the meanwhile, I’ve started reading Making Globalization Work by Joseph Stiglitz. This book, in particular, has been a kind of catharsis for most of my armchair ideas in economics, at least so far.  It’s a very strange feeling to read the ex-Chief Economist of the World Bank explaining his own ideas about overcoming the zealousness of “market fundamentalism” prevalent in economic circles, while I, who never studied economics formally, think, “Why would anyone trained in this discipline actually believe that markets are a magic force that work on their own?”  But I guess ideology always trumps rationality.

On February 13, they’re going to have the creator of VIM, Bram Moolenaar, giving a talk on VIM7.  What’s funny is that Bram, who has been working full-time on VIM for the past few years (living entirely on donations and money votes for VIM features) has now been hired by Google.  Smart move.

Here’s what I wrote:

Hi Miguel,

It looks like a new set of “Halloween Documents” have come out, thanks to a case in Iowa, Comes et. al. v. Microsoft (http://iowaconsumercase.org/index.html). I’m wondering if you have any comments on this document in particular, which suggests that Microsoft management knew full well they were “stealing Java” to intentionally marginalize the cross-platform language issue.

A select quote from the document, “Screw Sun, cross-platform will never work. Let’s move on and steal the Java language.”
Here’s the e-mail archives, which was submitted into evidence:

http://www.iowaconsumercase.org/011107/PX_2768.pdf

I’m wondering, given these thoughts from within Microsoft management, and given the recent news of Sun open sourcing the Java language under GPL terms, how is it that you can still push for the Mono project on Linux? Aren’t we always going to be fighting an uphill battle against a monopoly company protecting its biggest cash cow: the Windows platform?

Although my question was more “devil’s advocate” and meant to rile him up, Miguel provided some of the strongest and most cogent arguments for Mono that I’ve seen on record.

I just want to say great work to Miguel and the Mono team, and that if you ever doubted your raison d’etre, all it would take is reading this thread to be convinced! You’ve certainly convinced me!

Update: it’s really this kind of dependency on Windows I’m worried about in .NET. I think it’s just that the culture of the Java runtime is one of platform independence, whereas .NET from Microsoft is one of “platform dominance,” and Mono is some sort of stepping stone between Microsoft’s single-platform vision and those of us who want to write cross-platform apps using .NET.

There are three options you have when developing software:

1. Quality

2. Lots of features.

3. Low cost

Pick any two.

When you actually go through the combinations, it’s quite funny, and poignant.

Also, check out this excerpt from Rosenberg’s book, called “Words fail us.”  I’m currently working on a software project and we’re doing lots of new and innovative stuff, and I can’t even express how much I connect with this article.  Choosing the right vocabulary may be the single toughest thing in working on software team projects today.

Frederick Brooks, author of the The Mythical Man-Month, disinguished between two problems in software development: the accidental problems and the essential ones. (Read Brooks’ watershed article, “The Silver Bullet“.) The industry likes to argue that we have made huge leaps in terms of the accidental in recent years. Programmer productivity is better with the help of better software revision systems, but more importantly there are RAD tools like GUI designers, powerful IDEs like Eclipse/VS.NET. Do I think we’re headed in the right direction?

In a sense. I think those tools are useful, but only because I think all we can do is shrink the accidental problem as small as possible, and then do our best to tackle the essential. The major win in terms of the accidental and even the essential is high-level programming, and I mean really high level, like Python. Why is this important? Well, Brooks once talked about how PL/I was a great software engineering language because the statements written to the compiler are “pretty close” to the “thought stuff” the programmer is working with in his head. But PL/I is nowhere near there (neither is C++, or even Java or C#). However, Python heads in that direction.

In my head, unless I really am immersed in C code all the time, I don’t think of things like types, memory management, or, God, pointer arithmetic. When we solve problems, we solve it in something computer scientists have used for years: pseudocode. Pseudocode is nothing more than code that expresses an algorithm or approach without worrying about the gory details of the underlying hardware.

Python isn’t quite pseudocode, but it’s closer than anything else I’ve seen. And that’s a huge help to productivity not just from the point of view of the single developer, but, I’d argue, also from the point of view of a team of developers. Code readability is hugely important in team environments. C and Perl can be very unreadable, which makes them ill-suited to team development. Java and C++ are better, but even they suffer from some readability problems (where the actual solutions can be masked by “best practices” and strange ways of exploiting constructs of the language). Python can be hacked too, but it tends not to be, and it tends to have very high readability.

But what about the tools mentioned above (Eclipse/VS.NET)? I often wondered about all the *nix programmers (including many of my Computer Science professors) who still load up X11 and dump into vim or emacs for their programming needs. Wouldn’t they benefit from the latest and greatest in IntelliSense, Refactoring, CallTips, SaneOnlineDocumentation, and any other CamelCase ideas I can think of for a development environment? Most *nix users would say that stuff is unnecessary–and in many senses, they are right. The essential problem remains, no matter how fancy your IDE is.

A lot of *nix developers shrink their accidental problem on a “as-needed” basis, by coding plugins or complex configurations for their highly-extensible text editors. But nowadays, productivity is starting to be a concern even in the OSS world, where things have traditionally stuck with the old “mortar and pestle,” or, should I say, “gcc and gdb”. That’s why we see projects like Mono gaining a lot of popularity.

But Python offers a nice, alternative path, I think. The accidental problem is worth shrinking, and I think it can be done simply by one major refocus:

(1) make the language usable, not the tool. (IMO, already done with Python)

and

(2) create source code analyzing tools that integrate with development environments to make the language-bearer more productive.

You can see enormous success in (2) with the Eclipse IDE project. My main problem: Eclipse’s focus is on a language that isn’t very usable. Java is certainly better than C, but still, it has significant problems that stop it from connecting me (the programmer) with my problem-solving ideas (the thought-stuff of programming).

At my new job, I work with Java eight hours a day, but my code just doesn’t read and work like my Python code, which is much closer to the underlying algorithms and approaches. Java certainly pushed forward OO on the masses of computer programmers, but it did so without learning non-OO but very handy scripting language concepts, like having Lists and Hashes be built-in types, with tons of syntactic sugar, for crying out loud. Most problems break down to List and Hash problems, after all. What’s more, OO on its own isn’t a panacea: I’ve seen many convoluted OO designs that I wish were written procedurally, and which might have been more performant if they had been. The software engineering community thinks that there will be a silver bullet (OOP! AOP! XP! MDE!), but I think just getting programmers closer to the “thought-stuff” will be an effort with better rewards.

Check it out.

I have to say, so much of this article rang true for me. I just finished a web application in which we had four developers but two main rules: backend developer and frontend designer/developer. Due to the type of project it was (and the skillsets of the people involved), backend work was done by 3 of the 4 team members, and frontend work was done by a sole designer.

The issues we had were that:

• The backend developers could only make guarantees about what kind of object to pass to the view. But often, in order to get access to other “global” objects, the frontend designer would have to hack controller code as well. (Luckily, data services were tiered off out of the controllers, so at least the designer didn’t have to muck about with SQL.)
• The designer hated the idea of pushing some logic “back” into application code. So, though we as backend developers proposed a layer for configuring things like which menus to display on each page, etc., the frontend designer preferred to engage in the copy-paste-modify antipattern instead, so that it could just “get done.” I can see why: it’s a pain to context switch from JSPs, HTML and CSS to Java POJOs that then will have to be checked using JSTL anyway!

If, early on, we had decided to use FreeMarker instead of JSTL, perhaps the frontend designer could have had access to more powerful macros, and wouldn’t have done the copy/pasting. But I think overall, the two tiers are too few; I agree that you need someone to sit in between and think about how to cleanly integrate backend logic and frontend design.

The main reason we’re depressed is because, though both Sak and I like the Facebook and are users, we can’t help noticing one thing:

It’s not that complicated to build a website like that.

In fact, it’s downright easy. If I weren’t so busy with computer science classes, I probably could have threw something like it together myself.

Now, we know that business opportunities don’t have to be complicated to make money. They just have to be Right, that is in the Right place, with the Right look, taking advantage of the Right fad, etc.

But doesn’t it seem to you that a straightforward PHP/MySQL application just isn’t worth $2 billion? I mean, that’s $2,000 million. That’s $2,000,000,000.

Yet, I can’t say I’m entirely unhappy about it. Mark Zuckerberg, here’s to you, man. You’re my age, and you did exactly what I wish I had done. Built some crappy website, and made out like a bandit with sacks of cash. Kudos. You’re honestly my fucking hero.

I think it’s pretty clear that despite the disagreements I have with Mr. Gates over a subject known as “business ethics” (if such a subject truly exists!), he does seem to be a genuinely patriotic guy who loves technology. I mean, what good is it for more Americans to get into CS, if other countries are diving in and filling whatever knowledge gap may exist? Can’t Bill just hire those workers, and what’s more, for less money per hour?

Well, I think Mr. Gates really wants innovation in computer software to remain “America’s Great Industry.”

I was very intrigued by this response to the article:

Everyone knows that Open Source is taking over the software development industry. And according to the Open Source philosophy; developers should be enslaved, source code should be free. No, no, that’s not politically correct, let me try again. Developers should give their work away because code needs to be free (as in speech) and the needs of the code is more important than the needs of the people who create it. Well, that doesn’t sound quite right either but in any case, it doesn’t really matter to me because my kids won’t be studying computer science.

This is a very interesting post. True, it will be seen as a troll by some, since open source philosophy definitely doesn’t say anything about programmer enslavement. But his point is real and felt in the industry. That is, if you aren’t selling software, how are software developers to make money from it?

I think the response to this was best-articulated by Eric Raymond, when he pointed out that of programmers, only about 1-2% make their cash from off-the-shelf software sales. Instead, most make their money from “in-house” or “custom” software solutions. In other words, the majority of developers aren’t working on the Adobe Photoshop team, they’re working on Acme Inc.’s payroll or issue tracking system.

I kind of love this sort of propaganda, though. Because it is all good news for me.

When I first decided to do CS, I considered the possible effect of outsourcing and other factors on my employment possibilities. I thought, what if there are no jobs when I get out of college? But I stuck with it.

Well, it turns out, everyone had a hunch similar to mine, but they were more wooed by it than I was. So everyone fled CS. And now I’m the only one left. (An exaggeration, but you get what I mean — my computer science classes are nearly empty, whereas they were packed during registration only a few years ago).

It turns out, firms are hiring more than ever before. Why? Because the dotcom bubble is over, and green-eyed imposters are getting flushed out of the industry. But the demand is still there. Software is pervasive. Everyone needs software development done. There simply isn’t anything under the sun that can’t benefit from a little software developer finesse.

You can’t have all this work done in India and China because, it turns out, people want software developers to work with customers (big surprise). They want applications which meet their sensibilities, and they want them changes when the environment changes.

I liked that Mr. Gates said the #1 thing he’s looking for is project management IT types. Funny, it’s the #1 thing I’m looking for, too. Software developers are a dime a dozen. Find me a software developer who doesn’t get nervous when you ask him a tough question, or ask him to write, in plain English, a high-level overview of the system you’re asking him to create, and you’ve got yourself someone who’s valuable.

I personally find this to be very true in my life.

When I was in high school, I prided myself (from the point of view of “tools”) as knowing graphic design (Photoshop/Illustrator), web development, and print/page layout. Handy tools to know for (1) making money and (2) working on a high school newspaper. The only real programming languages I knew back then were Actionscript (for Flash), JavaScript, and (eegads) Perl. Then I got to college and armed myself with algorithms, data structures, and systems, and started picking up Java and C on my own. Now I consider myself well-versed in those, and this past summer learned Python and used that on a lot of different projects. Then this semester I got interested in C++ and used that a lot. Nowadays, when I look at problems, I look at them in terms of my tools. Text parsing problem? Wow, Python’s re (regular expressions) module could handle that pretty easily. Big engineering project? Wow, using templates and OO features in C++ may lead to a nice design. Database-driven web application? Well, Java/JSP may fit you nicely. (I know, I know, what am I doing not knowing Ruby on Rails!)

I think Mark’s onto something. Changing toolsets often is definitely useful. Even though I couldn’t write full programs for you in Perl nowadays, what I do know about it (its limitations, capabilities) is definitely good enough to see when it may be the best choice for the job.

As for academic tools — very true. A lot of techniques I learned in e.g. Discrete Math, Linear Algebra were in one ear and out the other. Alas, I think the main point is to learn them once and then be able to Wikipedia them later, when needed

That said, stuff I learned in my algorithms and data structures and operating systems courses have stayed with me. I think some of that stuff is just essential.