Indeed, my life for the last 3 years has been dominated by the language. In many ways, pursuing a startup and enduring the associated financial hardship was partially because I had become frustrated with using Java in my full-time work and wanted to convert hobby projects I was building outside of work hours into full-fledged projects.

Something else I noticed in the last three years is that my programming life has become very zen-like. I now rarely discuss or debate things like programming language features, strange constructs like generics, or which framework to use or ignore. Instead, I spend most of my time building a product that people love. My colleagues and I communicate with code. And what better language to communicate in than arguably the world’s most readable? What better language to deliver value in than one that simply gets out of your way?

I therefore get a great amount of joy of showing other people what the Zen of Python can mean in their lives. Last year, I gave a training course to a 20-person team of government employees who were using dated languages like Fortran and COBOL to build important government systems. A bright manager in the organization realized how much more productive their team could be if they stopped worrying about compiler versions and IDEs and started thinking in code. But the key was to understand the value of Python, not necessarily as a language with a certain set of features, but as a way of doing things, as a cultural influence. This is a culture that says, “the language should fade away”, similarly to how Edward Tufte argues that the chrome and administrative debris should fade away when displaying content.

Since then, I have used these slides countless times to espouse the virtues of my favorite interpreter. This has included giving short seminars in NYC, training Parse.ly interns and new hires (here’s a shot of me and two Parse.ly engineers, Michael and Zach), holding Python office hours at HackNY’s hackathon (indeed, here’s an action shot on Flickr to prove it!), and introducing my friends to the joy of building software.

Code and Slides: The Zen of Python in 3 days

It’s therefore with elation that today I am able to provide the slides and materials for this talk free to the world on my Github account. Simply click here:

https://github.com/Parsely/python-adv-slides

Or, if you’re not interested in the code behind the slides (because the slides are actually created with the help of Python itself), you can go here to simply view them.

http://pixelmonkey.org/pub/python-training/

(note, modern web browser like Chrome or Firefox recommended for speed and smoothness)

This may be my first “open source presentation”, in that the slide presentation itself is provided as code that is freely available, and reproducing the slide presentation is a matter of running a code generator.

A touch of NLP with Python

I mentioned that one of the reasons I love Python is because it gets out of your way. Nowhere is this more evident than Python’s ability to prototype algorithms to natural language processing and corpus linguistics problems, something I do everyday. The Natural Language Toolkit (NLTK) provides an excellent starting point from learning about this aspect of computer science.

Recently, Parse.ly has gotten some new engineers on the team who are great Python programmers but without any background in NLP. To give them a taste of how well-suited Python is for this task, I gave a seminar entitled, “Just Enough NLP with Python”. Just as with the prior slides, you can view these online on Github:

https://github.com/Parsely/python-nlp-slides

And you can view the compiled slides here:

http://pixelmonkey.org/pub/nlp-training/

Between these two pieces, you can learn the Zen of Python (PEP 20) very rapidly, and incorporate it into your everyday software life.

If you have any comments or feedback, feel free to reach out to me on Twitter. I’d love to hear what you think. And if you, too, are bitten by the Python bug (as one of our Parse.ly interns recently was in a big, almost romantic way) then you should reach out to us, since we’re hiring.

Now, don’t get me wrong — Python also has batteries included, and usually, when I think that I’m missing a great module I used to use in Java, it already exists in a much more powerful form in Python’s Standard Library or the wealth of modules on PyPI, GitHub, and Bitbucket. However, I believe in not reinventing the wheel, and so if a great open source tool exists in Java, I will want to interact with it.

One of these modules which we use extensively at Parse.ly is Apache Solr, and its surrounding Lucene project modules. Lucene is an extremely mature framework for document indexing, and Solr is a powerful server-ization of that technology that fits well into complex, mixed language distributed systems. I know there are efforts — like Whoosh — to build fast search engines atop the Python language. And I applaud these efforts — more projects means more competition, and more competition means better products. However, I still believe that you go with the best of breed tools available for production software, and you try not to let religious arguments about programming language get in the way.

Lately, I have come across more and more Java open source projects that have no equivalent in Python, and which I would like to access. Knowing that I wanted to feel comfortable incorporating Java open source projects — beyond Solr, which was already nicely wrapped as a web service — I, at first, thought that I’d be forced to still live among the weeds of complex class and interface definitions, cumbersome Java IDEs, XML configuration files, and (IMO) time-wasting rabbit holes like dependency injection, configuration management, and classpath hell. And then I found Groovy.

The most under-rated language ever

Groovy was released as one of the earliest dynamic languages written atop the JVM, back in 2003. I remember hearing about it back then and not “taking it seriously”, as we programmers often do with important, new technologies. I had the same ill feelings toward JavaScript until as late as 2007, but now consider myself more a JavaScript programmer than a Java one!

In truth, had I read James Strachan’s blog post discussing why he created Groovy, I would have probably paid more attention:

So I’ve been musing a little while if its time the Java platform had its own dynamic language designed from the ground up to work real nice with existing code; creating/extending objects normal Java can use and vice versa. Python/Jython’s a pretty good base – add the nice stuff from Ruby and maybe sprinkle on some AOP features and we could have a really Groovy new language for scripting Java objects, writing test cases and who knows, even doing real development in it.

I had thoughts like these back in my Java programming days, too. I would see features that were standard in other languages — concise list/map syntax, “bare” functions (that could live outside of class definitions), optional/dynamic typing, metaprogramming, a proper REPL, first-class functions — and think to myself, “Aren’t we missing a whole lot by lacking these features in Java?” The more I started to code Python and Java side-by-side, the more I started to realize that many “core” Java technologies were basically created to cope with inadequacies of the language itself.

For example, unit testing “frameworks” and formal IDE/debuggers are more popular in Java than other languages because of the lack of the REPL — dynamic language programmers tend to test their code from an interactive shell as a regular part of programming. Utility libraries like Apache Common Utils were created to cope with syntactic deficiencies around core types like lists and maps. And on and on.

Groovy is a “post-modern language” dynamic language. By that, I mean it is clearly inspired by and learns from the lessons of both Python and Ruby. It is also the world’s first programming language that was written to target an existing community of programmers. Let me explain. Python was not written “to target” C programmers or Visual Basic programmers. It was written as a new language altogether, not targeted at anyone in particular. However, Groovy was written “to target” Java programmers — it is the dynamic language Sun should have built for the JVM as dynamic languages gained popularity and the Java language stagnated.

Groovy is respectful of and cooperative with Java itself. One of its primary design goals is to live alongside existing Java code, even while Groovy’s syntax far surpasses that of Java. In this respect, Groovy plays a very similar role in the Java ecosystem that Python plays in the C ecosystem. However, unlike Python and C’s relationship, Groovy’s relationship with Java is bidirectional. What that means is that one can easily write Java libraries that execute Groovy code (provided the Groovy Language is on the classpath as a JAR), and Groovy libraries can easily execute Java code. In fact, you can even have complex relationships among these two languages, such as Groovy classes serving as base classes for Java classes and vice versa.

However, what’s perhaps most astonishing about Groovy is that it truly got a whole lot of things right. That is, the language simply is extremely well designed, and specifically greased for programmer productivity. It learned from the mistakes of existing dynamic languages, and made a dynamic language that is truly the “best of both worlds”. When I introduced the language to an engineer on my team at Parse.ly, he uttered some words that really stuck with me: “This language was clearly designed by very, very lazy programmers.”

So true. Typing is a chore — let’s do less of it to code without sacrificing readability. That’s the Groovy way.

Over the last several years, it has matured to the point where I feel extremely comfortable writing production code in Groovy, especially thanks to the surrounding ecosystem of the Grails project.

Python is still my muse, but Groovy is my Winston Wolf

A famous character in the movie Pulp Fiction is Harvey Keitel’s “Winston Wolf”. He’s the underworld problem solver — a guy who gets things done with a no-nonsense attitude.

Winston Wolf: If I’m curt with you, it’s because time is a factor. I think fast, I talk fast, and I need you two guys to act fast if you want to get out of this.

Python is a beautiful language and will continue to be my language of choice for production code. However, I simply can’t ignore the enormous wealth of production-quality Java code that has been written for which there is no good Python equivalent. I also can’t ignore the unbelievable engineering effort that went into making the JVM a powerful, scalable, and stable platform for production software. I thought I’d have to throw this community away (for myself and my companies) due to the deficiencies of the Java language. However, I have come to realize that Groovy has proven itself the worthy successor to Java. (Yes, successor — as in, you would be a fool to write your code directly in Java these days.) And personally, I will never work on a Java project again without Groovy. Java is a mess, but Groovy is my clean-up man.

I have also observed that the “ported” JVM languages have failed. This was a surprise to me as it was happening, but in retrospect, is not as surprising. Jython and JRuby are simply not viable long-term open source projects. The target community is only those programmers who know both Python and Java or both Ruby and Java. And there was no great impetus to truly “learn” those environments, because Python programmers felt that they already “knew” Jython, since the language was the same. Is the benefit of having access to Java libraries worth the downside of using a non-standard VM for my primary programming language? Usually not.

Now, while I wasn’t even paying attention — slowly learning Groovy in my spare time and using it for small projects dependent on Java — it seems that the world of independent dynamic JVM languages has taken off. Groovy has not had nearly as much popularity as languages like Scala (used at Twitter and Foursquare) or Clojure. And Groovy lacks “sex appeal”, possibly because no significant large-scale projects have been documented atop Groovy and because the language is an incremental improvement on successful mixed-paradigm languages.

However, unlike Clojure and Scala, the Groovy language can be learned in a weekend by an existing Java programmer with dynamic language experience, and can be learned in a couple weeks by anyone else. And unlike those other languages, it does not try to be a fundamentally new programming language with new programming models. Instead, it simply dramatically improves the Java language itself using a syntax that becomes quickly familiar to dynamic programmers.

When you consider the cost/benefit analysis of learning Groovy — for a couple weeks of my time, I gain access to being able to easily script — and, with Grails, wrap as web services — an almost limitless supply of Java open source code, you realize you’d be a fool not to learn the language.

I like new stuff as much as the next guy, but Java is a language that is still used by millions of professional programmers across the world, and has proven its ability to be the basis for large projects. And Groovy is a 2X or 3X productivity improvement over Java, with full bi-directional compatibility with existing Java code? I’ll take that over a new-fangled “revolutionary” language any day!

So, what’s so groovy about Groovy?

Oh, nothing… just things like this:

["Rob", "Christopher", "Joe", "John"].findAll{ 
  it.size() < = 4 
}.each { 
  println it 
}
// ==> prints "Rob", "Joe" and "John"

Yes, we’re really not in Java anymore. Let’s walk through the pain that is Java step by step to arrive at our Groovy salvation.

First, Java loves boilerplate. It really is an affliction of the language and the community. A great example of this is Java’s take on module imports.

Even though Java has a great standard library that is extremely well-designed and complete (for the most part), it fails to make this standard library easily accessible to you. That’s because all of Java’s best libraries are hidden in obtuse, deeply-nested library locations, and one can never remember where they are. IDEs solve this problem by providing an “Organize Imports” facility, which expert Java programmers use instead of managing imports themselves. This problem, alone, puts Java IDEs on significantly more productive footing than simple text editors like vim.

Groovy says, “this is nonsense — stop the madness”. In Groovy, a slew of commonly-used Java modules are simply imported automatically. If you run into a namespace conflict, that’s OK — we’re all adults here.

// these are all automatic in every Groovy module you write
import java.lang.*;
import java.util.*;
import java.net.*;
import java.io.*;
import java.math.BigInteger;
import java.math.BigDecimal;
import groovy.lang.*;
import groovy.util.*;

Note that groovy.lang and groovy.util add significant utilities above and beyond Java.

Next up are the major data structures used by any modern programmer — lists and maps (aka dictionaries). If you haven’t been doing Java in awhile, you won’t miss this kind of code.

List<Integer> someItems = Arrays.asList(new Integer[] {1, 2, 3, 4});
for (Integer item : someItems) {
    System.out.println(item);
}
Map<String, String> someMapping = new HashMap<String , String>() {{
    put("ST", "started);
    put("IP", "in progress");
    put("DN", "done");
}}
for (Map.Entry<String, String> entry : someMapping.entrySet()) {
    String key = entry.getKey();
    String value = entry.getValue();
    System.out.println(key + " => " + value);
}

Note that the above is about the most concise possible code you can write for these operations using standard Java facilities. In fact, I have even tapped into some obscure features, using Arrays.asList to initialize the ArrayList (leveraging the fact that arrays, but not lists, have a concise initialization syntax) and leveraging double curly brace initialization for the Map, which is an extremely non-standard and perhaps even overly inefficient technique. (The alternative, though, is to write out someMapping.put(key, val) for each entry in the Map — ick).

Let’s look at this same syntax, but in Groovy.

someItems = [1, 2, 3, 4]
someItems.each {
  println it
}
someMapping = ["ST": "started", "IP": "in progress", "DN": "done"]
someMapping.each { key, val ->
  println "${key} => ${val}"
}

Not only is this code more concise, it is also much more readable. The signal-to-noise ratio is much better.

Groovy has great support for Functional Programming and Domain-Specific Languages.

// first-class functions/methods and DSL syntax
clean = { text ->
  text.replaceAll("[^A-Za-z]*", "")
}
clean "my-text-*foo"
remove = someItems.&remove
remove 3
someItems

It retains support for loved Java features like switch statements, but these become much more powerful and concise, e.g. in this factory function that actually returns cleaner functions depending on the passed-in type.

// proper closure, switches and variable interpolation
cleanerFactory = { type -> 
  switch (type) {
    case "text":
        return clean
    case "numbers":
        return { text ->
            text.replaceAll("[^0-9]*", "")
        }
    default:
        throw new IllegalArgumentException("invalid type ${type}")
  }
}

Using the cleanerFactory is a breeze, simply call it and you get back a function which you may then call. Yes, this is basically impossible in Java. To get something close, you’d have to make the factory return “Runnable” instances and then call .run() on the results.

// try/catch and bean syntax shorthand
test = "my-text-*foo-1234"
println cleanerFactory("text")(test)
println cleanerFactory("numbers")(test)
try {
  println cleanerFactory("foo")(test)
} catch (IllegalArgumentException ex) {
  println "cleanerFactory returned exception: ${ex.class.simpleName}, ${ex.message}"
}

Just like Java, Groovy supports classes. But it improves them by adding a concise syntax for declaring proper JavaBeans. The following class has getters, setters, and a keyword constructor autogenerated for you.

// classes and types, if you need them
class Article {
  String title
  String summary
  String content
  Date date
}
article = new Article(
    title: "Parse.ly uses Groovy, oh noes!", 
    summary: "Didier resigns in protest", 
    content: "Keith goes back to .Net, Toms returns to Ireland.  Ahhhh!",
    date: new Date(2011, 3, 7)
)
article.properties.each{println it}

Also notice the special “.properties” property, which allows you to iterate over a JavaBean’s property without using Java’s obtuse reflection API.

// inheritance if you need it
class WSJArticle extends Article {
    List topics
}
wsj = new WSJArticle(topics: ["US", "Finance", "World News"])
wsj.topics.each{ println it }

Inheritance works exactly as expected, and follows Java rules. However, inheritance gets even better in Groovy because it loosens stupid rules that Java has, such as “final” classes.

// and sure, even call and inherit from Java code (even final classes!)
class BetterDate extends Date {
    def getTimeString = {
        return "{this.time}"
    }
}
println new BetterDate().timeString

It’s simply not possible to subclass Date in Java, but Groovy has no problem with that. We’re all adults here.

// and sure, use Groovy scripting constructs with existing Java APIs!
lotsaItems = [1, 1, 2, 2, 2, 3, 3, 3, 3, 3]
println Collections.frequency(lotsaItems, 3)
Collections.shuffle(lotsaItems)
println lotsaItems

The above is an example of using existing Java APIs. Collections is surely a nice static class. Groovy lets you use it to your heart’s content.

The API for the Collections.binarySearch method is written as:

static <T> int binarySearch(List<? extends T> list, T key, Comparator<? super T> c)

Yes, that is “documentation” So basically, I need to create a class that implements the Comparator interface, and then I need to pass that comparator to this method along with my list. That will require a lot of angle brackets to get the generics right. Hmm, who cares, let’s just do it the easy way: in Groovy.

// even existing, complex APIs work fine in Groovy, and with less code!
comparator = [
  compare: { a, b -> a.equals(b) ? 0 : Math.abs(a) < Math.abs(b) ? -1 : 1 }
] as Comparator
println Collections.binarySearch([0, 2, 5, 7, 8], 5, comparator)

This shows off the ability of Groovy to easily coerce values using some very intelligent rules. Comparator is an interface that expects a “compare” method — therefore, we can coerce a map with a “compare” key and a function value into an implementation of that interface.

// convenience features abound, like heredocs and safe pointer access
myDocument = """
  This is a long document...
"""
foo = [myDoc: null]
println foo.myDoc?.size() // returns null rather than an exception

If I had a dime for every time I noticed Java programmers using StringBuilder to work around its lack of multi-line strings, or adding extra methods just to do null checks... well, I'd be very rich indeed Groovy says -- no need to complicate your life with that noise.

// and type coercion that actually works!
set = [1, 2, 3, 4, 5] as Set
5 in set
// ==> true!

Yes, the in keyword. Yes, lists can be converted to sets. Because that makes sense!

// this is a feature I wish Python had -- JavaScript-like map syntax
map = [:]
map.item1 = "foo"
map.item2 = "bar"
map.each { println it }
// ==> "item1"="foo", "item2"="bar"
println "done"

Groovy also learns a little from JavaScript -- maps are accessible using either key index or dot syntax. I personally like this, I know some Python people won't.

// nice range and slice syntax
items = [1, 2, 3, 4, 5]
println items[0..2]
for (i in 0..5) { println i }

Similar to Python's slice syntax, Groovy supports a "Range" type using two dots. It works where you expect it to: array indexing, for loops.

Using Grails, the standard web framework for Groovy, you can write simple RESTful, JSON web services in just tens of lines of code. e.g., here's a Grails controller that renders the result of a function as JSON.

package service
import com.thirdparty.library.ContentExtractor
import grails.converters.JSON
 
class ParseController {
  def contentExtractor = new ContentExtractor()
  def index = {
    def article = contentExtractor.extractContent(params.url)
    render article as JSON
  }
}

This will respond to a service like http://service:8080/parse?url=http://google.com with a JSON document containing the Java properties from the article object.

Seriously, how much easier can it get to wrap up a Java library as a JSON web service? Not much.

// in short, a pretty language worth looking at
println "DONE."

Download Groovy and Grails to start playing around!

I’ll show why this claim doesn’t matter later, but for now, I’ll assure you that it’s true.

How many of you out there use Dropbox? If you don’t, you should — it’s an excellent tool. In its free version, it provides you with 2GB of storage “in the cloud”, using a new kind of folder called a “Dropbox”. What distinguishes a Dropbox from other folders on your computer? The following:

• Every file put in your Dropbox is automatically (and securely) uploaded to Dropbox’s servers, ensuring you have an offsite backup of all data therein.
• Multiple computers can gain access to a Dropbox, ensuring files are automatically synchronized across computers without having to use complication version control systems.
• All files in your Dropbox are versioned, ensuring you can always recover an older version of a file in case you accidentally overwrite a good version.

Dropbox is supported on Windows, Mac OS X, and Linux, and now even has mobile applications, as well. Further, I have a special place in my heart for this service because I started using it almost 2 years ago, and it has acted as a file sharing and project management tool for my own startup’s internal operations at Parse.ly. I was therefore more than ecstatic to discover that this excellent tool and its smart founders had also made it through all of the hurdles necessary to get an early-stage company the financing it needs: they’ve raised over $7 million in financing and have over 3 million users.

But there is another reason I absolutely love Dropbox: because it was my idea. I invented it.

In the summer of 2004, I was really itching to get into Google’s Summer of Code competition. This was the summer I had taken a job working from home as the lead web developer at the Unemployment Action Center of NY. Though the job was great experience — letting me build my first full web application for a real client — I was itching to work on a technically juicy problem, something that affected me in my daily computer use.

And so, I sat down for a day and wrote up a Google Summer of Code proposal for a new system I had invented called Persistent Folders. It wasn’t exactly like Dropbox, but damn close. Even the implementation is close: Dropbox and my system both sync files using rsync, and both use a Python daemon process. The main difference is that since my system was meant to be open source, it did not require the use of a company-maintained service; instead, I proposed that users piggyback existing storage they have via web hosting providers.

Unfortunately, my project wasn’t selected.

Why am I posting this? I recently had a discussion with another engineer after I had discussed some of the technology behind Parse.ly with him. He was surprised at how liberal I was with explaining our internal implementation, architecture, and algorithms. He asked me, “Aren’t you worried that I could steal your idea?”

I responded, “You can steal it all you want; I dare you to try and implement it!” I then explained that to me, ideas don’t matter. I had the idea for a hundred startups that now exist before they started. I know from talking to users and customers of Parse.ly that they had our idea before we implemented it. What matters in software is not an idea, but execution of that idea. Ideas are a dime a dozen.

I began this post with the statement, I invented Dropbox. And now I’m here to tell you that it doesn’t matter one bit, because I never implemented Dropbox. And you can’t own ideas…

If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it. Its peculiar character, too, is that no one possesses the less, because every other possesses the whole of it. He who receives an idea from me, receives instruction himself without lessening mine; as he who lights his taper at mine, receives light without darkening me.
-Thomas Jefferson

Imagine if when I had come up with this idea, I had patented it. Would it really have been fair to force Dropbox to license my technology? To sue them for “infringing” on my patent?

No way — these guys deserve the success they have. How stifling it would have been for me to put any roadblocks in their way because “I had the idea first”! The world is better because Dropbox exists. And, these guys have had — in my opinion — near-flawless execution. So, kudos to them.

Consider the following matrix:

	Mediocre Idea	Great Idea
No implementation
Mediocre Implementation
Great Implementation

A mediocre idea with a good implementation is worth infinitely more than a carefully-guarded, good idea with no implementation. Of course, the best products are both great ideas and great implementations. And I think my proposal for “Persistent Folders” — written three years before Dropbox even started — proves this to me in a very personal way.

Now, I’m certain the Dropbox guys never read my little proposal, because it was only sent to Google and otherwise sat on my hard drive for years, not viewed by anyone. I had a great idea, but no implementation. And the Dropbox guys took a great idea (one they arrived at on their own, I’m sure), and gave it the implementation it deserved.

For those of you who guard your ideas carefully, I’d suggest you stop wasting your time, get off your butt, and focus on actually building stuff. Because if you don’t, someone else will!

For the curious, below is my proposal to Google Summer of Code 2004, unaltered from its original draft. This is just a relic; for now, I’m glad to keep hacking on my own little idea, hoping one day I can look back and say I executed as well as Dropbox did…

PERSISTENT FOLDERS: A New Metaphor for Data Synchronization
May 13, 2004

The Problem

Computer users are more and more finding themselves with a serious problem: the fragmentation of personal data across multiple physical machines, and even multiple operating systems. Users who find it most comfortable to have a desktop machine at home and a mobile laptop computer “on-the-go” (for business or trips) have to deal with chaotic and often frustrating manual methods to copy that data to the needed places. Some users carry USB memory sticks — which hold their important “working set” of files — and either work directly off those disks (suffering reduced speeds) or make copies of the folders therein on their actual machines, as they become needed, spreading yet more copies and compounding the problem of synchronization. Others abuse ubiquitous technologies for accessing important files, by either e-mailing the files to themselves or by uploading them temporarily to web or FTP servers.

The end result of any of these methods is fragmentation of personal data, with error-prone manual processes for replication, and the inability for a consistent way to search, backup or even just keep track of all the important files and folders spread among the PCs in question.

What is needed to solve this problem, or at least make the problem more manageable, is an intelligent, user-friendly, customizable cross-platform program that allows for transparent synchronization of personal data across multiple computers and systems, either via a LAN or via the Internet.

Introducing the Persistent Folder

Up to this point, most users are used to understanding folders as existing solely in one location. The only way to get a folder’s data onto another machine is to copy that folder, thus making a duplicate. The goal of this project will be to introduce a new high-level data storage mechanism known as a Persistent Folder. Persistent Folders differ from regular folders in that they are meant to be transparently persistent, or synchronized, across different computers.

To explore a hypothetical example, imagine user Joe sits down to do some personal accounting work on Friday. He creates a folder on his desktop called “Personal Accounting,” and begins working on some OpenOffice spreadsheets in that folder. A few minutes later, Joe realizes that some of his accounting work will have to be done while he is away during the weekend, on his laptop computer.

To enable this, he simply right-clicks the folder on his desktop and says “Make this folder persistent across multiple computers.” When he does this, a dialog comes up asking him to type in a description of the persistent folder, and to select computers on which he wishes to make this folder persistent. Joe is presented with a dialog of computers currently available on the LAN. He sees his laptop, “MobileJoe”, and selects it. He then presses OK.

Within seconds, Joe sees the folder “Personal Accounting” appear on his mobile computer’s desktop, even though he hasn’t even touched his mobile computer yet. When he enters that folder, he sees the same spreadsheets that are available on his main PC’s desktop.

These two folders are now treated by Joe as “one persistent folder available across two computers.” He can add files to one folder and they will automatically be propagated to the other. He can modify files and the new versions will then exist in the other. Joe no longer has to worry about pushing his data back and forth across the computers.

Internet Synchronization by way of Ubiquitous Services

One of the major questions one may ask at this point, “that is all well and good for persistence of a folder on a LAN, but what about when I leave my home/office, and need to synchronize over the Internet?”

One approach would be to have a special server application with which all computers that wish to share persistent folders could synchronize. But then the user needs his own server, and needs to install my little server daemon on it. And not many users have their own servers on which they can just install any old application. So for normal users, this, in fact, is no approach at all.

Although most users don’t run their own servers, many modern users do have _server access_. That is, many users do have Web/FTP hosting providers to whom they pay subscriptions, and these servers power their blogs, personal photo stores, etc. The goal of Persistent Folders would be to allow users to piggyback their existing web services to synchronize their files, essentially turning a folder on that server into a Persistent Folder Repository.

A dialog in the properties of a persistent folder would include a checkbox that allows the user to “make this folder available over the Internet.” It would then allow a user to choose a method for making this folder available via the Internet, asking the user to provide a “persistent folder repository” via FTP by default (due to commonality), but equally possible would be scp, rsync, NFS, or even something like cvs/svn (the goal, of course, would be to make the design modular enough that it could support any server type with basic file system operations).

Then, every other visible machine on the LAN who shares that persistent folder would have option enabled automatically. If the machines are offline, the option can be entered manually by the same method explained above.

From that point on, the folders become persistently available and transparently usable, just as before. If the other computers are available on the LAN, then the program utilizes LAN speeds and synchronizes directly. Otherwise, it synchronizes with the Internet Repository.

Lucky Joe is now able to work on his files in the office, leave his laptop there (“the darn thing is so heavy to carry around!”), return home, and continue right on working on those files which are now found in the “same folder” on his desktop machine at home.

Implementation Ideas

I plan on implementing this idea in Python, since one of the major goals of the project is to allow Persistent Folders to exist not just across computers, but also across operating systems (so that a Windows desktop could have a persistent folder that also exists on a Linux laptop, for example). For a user interface, I plan to use PyGTK, since that’s what I know, and since it is also cross-platform. I need something powerful like GTK since there will be times when user intervention will be necessary, but since it is a goal of this project to reduce the number of times the user must intervene, I want to make sure that when he does, he is presented with sane, human-readable, and user-friendly dialogs.

I don’t plan to reinvent the wheel. Most of the magic of Persistent Folders is just making existing synchronization tools work relatively silently and in a way that makes sense with a user’s workflow. The main tool I am thinking of using is rsync, the relatively-ubiquitous UNIX utility for incremental synchronization of directory trees. I considered using the unison, as recommended by Ubuntu’s Wiki entry, but I saw two major problems: (1) unison is no longer under active development and (2) it is written in OCaml, a relatively obscure language which I don’t know. Therefore, any bugs I discover in unison would not be fixed in a timely fashion by its developers, and any features I’d like to add to support my idea would be quite difficult to implement.

As for monitoring folders for changes, I imagine the most elegant solution would be to take advantage of inotify under Linux (like Beagle does) and perhaps handle ChangeNotify events under Windows. I’d really like to avoid polling, since polling is just plain evil for something as potentially neat as this. Regardless, I’ll probably need to code Yet Another Daemon (or Windows service) to keep track of Persistent Folders on the local machine and their equivalents across the network.

Finally, for secure synchronization of files, I plan to use SSH tunneling wherever possible, which is available under Windows under the OpenSSH for Windows Sourceforge project, http://sshwindows.sf.net. Linux distros like Ubuntu, of course, have all the ssh support one needs.

Conclusion

This project aims to introduce a new metaphor users may utilize to share important files across multiple computers: the Persistent Folder. A Persistent Folder is not a shared a folder; rather, it is seen as a single folder that exists locally across multiple computers, and can be treated by the user as such. Changes at any one folder rapidly propagate to the others. Properly implemented, this may provide a better way for users to manage important data that might otherwise be scattered, fragmented, and even lost through the daily shuffle of file transfers across networked PCs.

Appendix A: Project Roadmap

o June 24: Begin Work, with ideas now fully developed in the form of documents. Post these documents to Ubuntu’s Wiki to encourage ideas from community.

o July 5: Have a console synchronization wrapper and some network discovery stuff in Python written, and have UI concepts designed in Glade.

o July 20: Make 0.1 (GUI and basic features working) release, so that Google can show it off at OSCON?

o August 1: Have other great features, like the Internet synchronization and multi-protocol support, in 0.2 release.

o August 10: Consider working on features to allow transparent backup and versioning, to include with bug fixes in a 0.3 release.

o August 20: Make simultaneous 0.4 releases on Linux and
Windows, hammering out as many cross-platform issues as possible. Add inotify/ChangeNotify support if not already there.

o August 30: The big 0.5 release, finished for Google’s deadline. Let Google/Ubuntu make the decision if it’s worthy of being renamed a “1.0″ release.

o September 1: Live a less stressful life, since my files are now neatly synchronized among my PCs! But I’ll keep making it better.

Appendix B: Hold On, Isn’t Samba Good Enough?

One common response to this project may be, “Aren’t Samba and SMB shared folders good enough?” Though Samba is good, I do not believe it is good enough. Here’s why:

(1) Samba does not aim to present the user with a metaphor of a folder existing in multiple locations at the same time. I believe this metaphor would be appreciated as powerful by longtime computer users and subconsciously acknowledged as highly usable by novices.

(2) Samba only allows users to share folders on their drives. Other Samba users may then mount those folders via the LAN. This two-step, asymmetric process already seems complicated and convoluted to end users. But more importantly, synchronization is left up to the user. In theory, users could avoid synchronization altogether and work on the files directly, via the share. In practice, LAN speeds are not adequate, and other issues are raised (such as file locking and write conflicts). This forces users to home-brew their own synchronization protocol to make sure duplicate files at different versions doesn’t result in an accidental loss of data.

(3) Samba does not provide any easy method to access personal files once one leaves the LAN and enters the WAN, short of opening up a bunch of ports on your router and trying to connect in from outside (a very slow and insecure method).

(4) Samba provides no recourse when a computer is no longer available. Persistent Folders, on the other hand, make data available “offline” by design.

(5) Samba does not know whether two duplicate files exist across the network. Therefore, Samba cannot be immediately utilized as a transparent form of backup.

Is the Persistent Folder meant to replace Samba? Of course not. I believe Samba has specific purposes: to allow for easy, fast, one-time transfers of files across a LAN, and to allow networked printer and device sharing. I do not believe Samba is an adequate solution to allow a user to treat a folder as if it existed on multiple computers at once.

Appendix C: Versioning’s the Thing

Upon further contemplation of this project, I realized I had left a question unanswered in this document. What is the right way to deal with version conflicts during the silent synchronization phase among Persistent Folders?

Imagine user Joe works on a file in a Persistent Folder which exists on his laptop and desktop computers. The first version (let’s call it version 0.1) was created on his desktop, and Joe had the laptop on the same network, allowing automatic synchronization to occur. But then Joe’s laptop was disconnected from his desktop, as Joe was without network or even Internet access. He works on the document a bit, which can now be called version 0.2. But then Joe forgets that he worked on the document on his laptop, and when he gets home, he works on his desktop computer, creating a version 0.2 there too. Which version 0.2 should be synchronized?

Normally, the answer would be to pick the most recent one. But we don’t want Joe’s laptop work to be lost, just because Joe forgot about his prior work, do we? Well, what needs to happen is a bit of smart conflict resolution with no data loss.

I propose that Persistent Folders should also be versioned to a sane degree. Perhaps by default 3-5 versions of files are always kept, with the most recent versions visible directly in the persistent folder, and other versions buried (in .dotfiles or hidden folders) behind there. Then, the Persistent Folder monitor should inform Joe (“whisper” to Joe via the notification area/systray) that there was a conflict, and the newest file was chosen. But at any point, Joe can choose to revert a file back to an older version. The interface should be such that Joe can even see where the file was edited, for example:

Personal Accounts.xls – Prior Versions
o version 0.2: edited on MobileJoe
o version 0.2: edited on DesktopJoe
o version 0.1: edited on DesktopJoe

Of course, versioning could be disabled (at risk of data loss to the user), or could be enabled as “smart versioning” to only version files when conflicts occur during synchronization. Some may say that versioning is dangerous because it can drastically increase disk usage, but as I mentioned above, the other benefit is that the user gets redundant network backup of files for free. In higher versions of Persistent Folders, things could get more sophisticated by allowing users to turn off versioning even at the file-level, but I don’t think that’d be necessary in the first few releases.

So, for those not keeping score, Dropbox supports:

• Windows, Linux and Mac OS X syncing, using a scheme similar to that described above, except with a managed set of servers (Dropbox’s) acting as the “repository”
• sync over LAN (in latest beta releases) when PCs are local
• file-level versioning
• Collaboration by sharing folders with other users — a feature I didn’t discuss in my proposal, but that would be a clear next step for a managed service

Go Dropbox!

960.gs is a CSS grid framework, similar in spirit to Blueprint CSS and YUI Grid. However, 960.gs is at once more minimalist than these approaches, and more thorough.

The author has a detailed blog post explaining his motivations for working on 960.gs, so I won’t rehash each of those. Instead, I’ll just dive into what I liked about it.

1. Very simple and comprehensible name choices for CSS classes: You can see their demo page (view its source) for examples, but I can even summarize the approach in a few short sentences. You begin a grid container with a container_XX class, where XX is the number of total grid columns in the container. Every child element has a grid_XX class, which is the number of columns (aka colspan) for that element. All grid_XX elements should add up to the XX amount in the container if you are using all the grid elements available. To use larger gutter widths, you can use prefix_XX and suffix_XX classes. Finally, for any child grid elements, you need to mark the first one with an alpha class and the last one with an omega class — this is necessary to get rid of left/right margins on starting and ending elements.
2. Excellent CSS generator: this online CSS generator tool lets you generate 960.gs CSS that is just right for your website.
3. jQuery-based bookmarklet for double-checking grid layouts: another online tool is a bookmarklet which will “inject” a customizable grid overlay on any existing site, to help with alignment.
4. Templates for every major design tool: I saved the best for last. Nathan’s focus in designing 960.gs was to allow better collaboration between trained graphic designers (whose tools are typically Adobe Photoshop, Fireworks, Illustrator, etc.) and trained web designers (whose tools are things like Rails, Grails, Django). To ease this collaboration, the author provides templates for every major graphic design tool that has “grid overlays” compatible with the 960.gs framework.

Finally, the entire project is hosted on bitbucket and developed in the open. What more could you ask for? This has really simplified my approach to standard CSS designs.

The process took some time, which is why I saved it for a weekend. To backup my hard drive took about 2 hours, and doing the partitioning operations took about 3 hours. I went out in the nice weather and picked up groceries while it was loading.

When I got back and could kick off the installation process, I was pleasantly surprised by the installation wizard UI. It easily guided me through the partition setup process. Even though in my case I had to make use of the “Advanced” editor, it easily visualized what was going on in my hard drive, and even detected the operating systems I had on there (WinXP and Intrepid).

I set up my new ext3 partitions (after deciding ext4 too unstable for my taste), and got started. I was pleasantly surprised when instead of asking me to reboot my computer, it just started right up. I still had access to a functioning computer while it was installing! Nice. That allowed me to jump on my blog and start on this post I even connected my MP3 player and have some tunes playing!

I was considering doing an upgrade of my system from Intrepid->Jaunty, but decided to give a clean installation a try. I get the feeling that there is some “drag” in my Linux installation which has been running on my machine for almost 3 years now. (Wow, has it been that long since I got this laptop?) I went through multiple releases of Ubuntu via upgrades, and I simply feel my requirements for my system have shrunk so significantly that a clean install was best to ensure my system is configured well and cleanly.

What do I mean by “shrunk” requirements? Well, when I profile the usage of my computer, nowadays 90% of what I do personally happens within Firefox. The remaining 10% are all handed by newer software. Among things that don’t include Firefox are browsing photos and listening to MP3s. Even some of these tasks are moving to the web platform.

For my work on Cog Tree, I really only have 3 development tools I lean on directly: vim, WingIDE (Python), and Eclipse IDE (Java). Javascript development and debugging happens inside a browser. I still lean on VMWare to give me some high-quality creative professional tools from the Windows world, e.g. Photoshop and Topstyle (for CSS). Aside from these, I don’t really need nor want much other software on my system. Any other development tools can be installed on-demand using Synaptic.

Jaunty’s installation percentage is about 50% right now. We’ll see how the system runs once it boots directly off the hard drive. I’m pleasantly surprised that most of my hardware seems to be working out of the box. Even my volume buttons, brightness buttons and media buttons on my laptop now work, which is a nice touch. My sound quality is still a little poor due to a chipset detection problem that still seems to be present in the snd_hda_intel driver. But I’m pretty sure by setting some options in /etc/modprobe.d/alsa-base I’ll be able to get it working better.

People who know me know that I’m very skeptical about my computer and about Linux. I regularly complain about all the little silly regressions that Linux has suffered over the years. I’m also particularly upset about how certain beautiful and essential pieces of software never end up making it into the Linux mainstream, e.g. TuxOnIce. But hopefully, Jaunty will capture my heart this time, and gain some love from this Linux cynic…

1. A Whirlwind Excursion through Writing a C Extension. This talk by Ned Batchelder (author of coverage.py and cog) shows that you can write a Python C extension module in under 20 minutes. This is my top talk because I never thought that my skills in C would be so directly useful in writing Python applications. Considering how damn easy it is to write a basic C extension module, I wouldn’t be surprised if the only reason I ever write C code again is to implement some Python functions or types in C. Truly the best of both worlds!
2. Reinteract: a better way to interact with Python. Owen Taylor (of GNOME/GTK+ fame) has spent some time over the last few months building a better Python shell. Specifically, it’s a lightweight shell that is meant to be a prototyping or “worksheet” environment a la Matlab, Mathematica, or Maple. Except, you’re running and re-evaluating Python code. It even supports things like in-line graph plotting, but I’ve already used it to experiment with Python web services API. Any Python programmer who has been frustrated with IPython before should check out Reinteract.
3. Easy AI with Python. This talk might have gotten the #1 slot for most interesting, but not the #1 slot overall because it seems like this talk has been given at a lot of conferences (not just PyCon) over the last few years. This talk introduces some complex AI topics in a very short time frame, and in a very intuitive way. For me, the neural networks example with Jets and Sharks was particularly impressive. Raymond Hettinger is a great presenter, and if you have some time you should definitely check out his recipes on ActiveState’s Python Cookbook and his How-to Guide for Descriptors.
4. Abstraction as Leverage. A talk by one of my favorite Python authors, Alex Martelli (who wrote the best book on Python on the market, Python in a Nutshell), this talk isn’t so much about Python as it is about software engineering overall. But it’s thought-provoking as his talks usually are.
5. Class Decorators: Radically Simple. The presenter is the author of the Class Decorators PEP, Jack Diederich. If you like decorators and you are curious about metaclasses, you’ll love class decorators.

Feel free to share your favorites!

Today, I did some digging on the Metacity bugzilla to see if it was fixed, and found this bug.

Over the course of 3.5 years, this bug has sat on the bugzilla, and still isn’t satisfactorily resolved. There is now a reduced_resources flag in gconf, but this flag only disables the minimize animation at the expense of forcing you to use an ugly wireframe window dragging animation. (Complete, utter insanity.)

I don’t understand how after two years of debate, and nearly 3.5 years down th road from when this bug was first opened, the final fix was something that throws the baby out with the bathwater.

I don’t want the minimize animation, but I don’t want wireframe windows, either. Why can I only opt out of them as a couple?

Havoc, you asked for rationale and not rant, and lloyd had provided it:

the window minimization animation’s sole purpose is to indicate to an absolute GUI neophyte “where the window goes” when it’s minimized. for a huge number of users – basically anyone who’s ever used a GUI with a taskbar before, which is in excess of 98% of current desktop computer users – they are never useful because they already know this.

for the remainder, window animations are useful exactly once in their lifetimes – the first time they minimize a window and wonder where their application window “went”.

thereafter, window animations are a distracting waste of time, and to me and many others, an annoyance. it’s roughly equivalent to never being able to turn off Clippy.

I wholeheartedly agree with this analysis. Minimize animations are useful to newbies, not useful to anyone else.

The lack of an ability to disable this animation (alone, not in a catch-all flag that enables wireframe drawing, for crying out loud!) is completely absurd. On my Core Duo 1.6 Ghz machine running an Nvidia graphics card, this animation still feels slow and clunky, and is completely superfluous.

In the future, maybe we’ll be able to use compiz and get nice fade in and fade out that works smoothly and nicely. Even then, I’ll still want to be able to disable it, because it serves no purpose for me, and isn’t central to metacity’s operation.

In the GNOME community, we’ve heard mention of “Nat’s Pendulum”, referring to Nat Friedman’s insistence that the GNOME “usability pendulum” has swung too far in the other direction. The 3.5 years spent blocking on the inclusion of this simple gconf flag, was, I think, a perfect illustration of what Nat meant. The fact that the flag still isn’t there in the form suggested may mean that the damage may just be permanent in this community surrounding metacity development.

Why must gconf be “usable” too? Don’t give me a reduced_resources catch-all flag, but give me separate flags to enable/disable certain animations/effects. Is this really such a usability disaster, or are you just being a Metacity Nazi:”no fine-grained configuration for you!”

As Jeff Waugh has mentioned, religious usability insistence like that expressed in this thread was enough to send many would-be GNOME users and contributors packing for projects like KDE, XFCE, or even to proprietary platforms like OS X. The damage is deep and felt. I lost a lot of faith in GNOME personally because of these issues, and am only slowly regaining it.

Has anyone already written a patch which removes the reduced_resources flag and adds this fine-grained control of the minimize and wireframe animations? If not, I’ll write it and attach it to this issue.

I will not open a new bugzilla bug for this, because I think history is important so that we learn from our past mistakes.

I’ve been in contact with an ALSA hacker, Tobin Davis, over a series of patches that provide support for the sound chipset (intel-hda) that is part of the HP DV2000 laptop.

His patches over ALSA 1.0.14rc3 enable the following new features:

• The headphone port now works, and the speakers automute when the headphones are plugged in.
• The microphone port now works, with great sound quality.
• The built-in mic on the monitor now works, though obviously with worse sound quality due to ambient noise.

The new patches aren’t perfect. I’m noticing some sound quality issues at high volumes, and in order to get it to work Tobin had to essentially enable two PCM channels (PCM and PCM-2), which have a very strange behavior. The first one controls the volume directly from the sound system. The second one controls the volume only between the sound system and the speakers (and thus, will have no effect on the sound when the headphones are plugged in). The master volume control effectively modulates both of these. Tobin has told me that the chipset produced by Conexant is particularly weird, which is why he had to this. I find that it’s not so bad, as long as I keep a launcher to gnome-volume-control set up so that I can control it, knowing these rules.

That said, it’s a huge improvement over out of the box sound support for Ubuntu (which is ALSA 1.0.11). Inside this post you’ll find further instructions, which are adapted from a text document Tobin sent his tester group via e-mail. These are step by step instructions to set up 1.0.14rc3 ALSA drivers plus Tobin’s latest patch.

1. Backup your current drivers

tar -zcvf original-drivers.tgz /lib/modules/`uname -r`/kernel/sound

2. Next, get the latest release and bring it current.

wget ftp://ftp.alsa-project.org/pub/driver/alsa-driver-1.0.14rc3.tar.bz2

wget http://members.dsl-only.net/~tdavis/alsa-patches/conexant-latest-rc3.patch

tar -jxf alsa-driver-1.0.14rc3.tar.bz2

cd alsa-driver-1.0.14rc3/alsa-kernel

patch -p1 < ../../conexant-latest-rc3.patch

cd ../../

3. Build the code.

cd alsa-driver-1.0.14rc3

./configure --with-debug=detect --with-cards=hda-intel

make

sudo make install

sudo depmod -a

sudo dmesg -c >/dev/null

4. Stop all audio applications and remove all audio drivers

On my system, the following works:

modprobe -r snd-usb-audio

modprobe -r snd-hda-intel snd-hda-codec snd-pcm-oss snd-mixer-oss snd-pcm snd-timer snd snd-page-alloc

5. Reload the new driver

Simply run modprobe snd-hda-intel, look at the dmesg output and test it out.

Using alsamixer will show you all available channels. In gnome-volume-control, you can also enable additional channels with Edit->Preferences. You’ll need at least Master, PCM, and PCM-2 enabled to get proper volume working on the headphones and speakers. You’ll need Capture enabled in the Capture tab to get mic input working.

If nothing works, or one function doesn’t work, try reloading the driver with “model=test” as a parameter. On Ubuntu, this can be done by doing sudo vim /etc/modprobe.d/alsa-base and adding a line to the end that says This should enable everything.

If you have issues, you can try to contact Tobin Davis or other hackers at the ALSA team. Or you could even post a comment here and I’ll pass it along. Please include the system make/model, the subsystem line from “lspci -s 0:1b -vn” (Ex: Subsystem: 103c:30b2), the dmesg output from above, and the output from “cat /proc/asound/card0/codec#0″.

Update: Modified instructions based on Henrique’s input below. (Patchlevel argument for patch command was incorrect.)

Asking me to get off my a$$ and code drivers for this baby is what I consider elitist and a very unreasonable demand on the end user. It’s one thing that gives GNU/Linux zealots a very bad name in the real world.

I have sneaking suspicion that you get this response from the Linux community because we feel you’re placing unreasonable demands on us. Your points are valid, but your energy is misdirected. Unfortunately for those who don’t like to code, that’s how software is created and improved. We invite you to participate in our projects in a variety of capacities including but not limited to programming, but of course participation is not required.

I think that Linux “zealots” get a bad name because much of the “real world” believes in a culture of entitlement. Look at everybody living life with a chip on their shoulder, blaming everyone else for their problems and scoffing at the notion that they take responsibility for their own situation. Somewhere along the line we stopped believing in opportunity as a means of realizing our dreams and began to foster the idea that we’re entitled to our expectations. In “Linux land,” we believe that the opportunity to participate in our information society is fundamental to our inherent desire as human beings to better our situation and control our own destiny.

Of course, money can make just about any dream come true. Mark Shuttleworth, for [example], invested $10 million to help make the Ubuntu project a reality. But years ago my great-grandmother told me the story of how my family came to America with nothing but the promise that here they would find a land of opportunity. This is the same promise we make with free software. This isn’t elitist, this is egalitarian.

I have to say, this is part of what makes me love Free Software.  It’s this idea of widespread opportunity.  Sure, F/OSS has power structures and means of coercion/control built into certain parts of it, but for the most part, it’s based upon a very simple, powerful, and egalitarian idea.  “Anyone can improve this, anyone can make it better.”  It’s that kernel of an idea that makes any process — whether software production, book editing, encyclopedia editing, or even beer brewing, more enjoyable to those involved, and, as a side effect, better for the general public.

A poster under the name “jerryleecooper” says,

Vista is far more powerful than windows XP, and runs twice as fast. It is also much harder to pirate, and this point more than anything else has the Linux crowd in a panic.

and, in a follow-up post,

Are you saying that this linux can run on a computer without windows underneath it, at all? As in, without a boot disk, without any drivers, and without any services?  That sounds preposterous to me.

Is this serious, or satire?  If it’s serious, it’s absurd.  If it’s satire, it’s genius.

I have been perpetually delaying a review of Capitalism 3.0 and Dreaming in Code, both of whom deserve it. But I promise one soon. I use Hofstadter’s Rule of Thumb lately for estimating time: however long you think it’s gonna take, double it and add a unit of time. So if you think it’ll take two hours, it’ll really take four days. If you think it’ll take five days, it’ll really take 10 weeks. And so on.

In the meanwhile, I’ve been busy at work — actually working on some cool stuff from a technology standpoint, mainly in the realm of hacking with pieces of the Eclipse Modeling Framework, and its related projects like GMF, RCP, Eclipse Core, etc.

On my commute, I’ve been enjoying reading Making Globalization Work by Stiglitz. Although one of my friends mentioned to me that this book would be quite boring, and for the most part he was right. Not the lofty stuff of Barnes in Capitalism 3.0; but perhaps Stiglitz’s recommendations are much more practical for ways to improve the current system.

The other book I started recently is a long, written interview with John Kenneth Galbraith (much in the style of Socrates) which is entitled, Almost Everyone’s Guide to Economics. What’s amazing is to see Galbraith, this towering (literally) Keynesian economic thinker, speaking in the 70s of the growth of corporate power, the undermining of labor, and the insidious nature of market fundamentalism. And yet, here we are, 30 years later, heeding none of his warnings, and entering into the new “global age” of “The World is Flat”.

Oh yes indeed, I do need to write some reviews very soon.

I’d post a longer review, but I’m headed down to New Orleans today. Will post a longer review when I get back, hopefully also of Capitalism 3.0, whose ideas have been swimming in my head the last few days of commute.  I think they really deserve to be summarized and presented here.

In the meanwhile, I’ve started reading Making Globalization Work by Joseph Stiglitz. This book, in particular, has been a kind of catharsis for most of my armchair ideas in economics, at least so far.  It’s a very strange feeling to read the ex-Chief Economist of the World Bank explaining his own ideas about overcoming the zealousness of “market fundamentalism” prevalent in economic circles, while I, who never studied economics formally, think, “Why would anyone trained in this discipline actually believe that markets are a magic force that work on their own?”  But I guess ideology always trumps rationality.

The website provides a way for Dell customers to vote on the issues which are most important to them. The number 1 issue right now is making Linux available as an install option on their Dell PCs. There are a slew of issues below that related to gaining driver support for Linux, and offering Windows with OOo and Firefox, and even offering OS-less computers to avoid the Microsoft tax.

It’d be interesting to see how this pans out, but in the meanwhile, go ahead and contribute a vote!

On February 13, they’re going to have the creator of VIM, Bram Moolenaar, giving a talk on VIM7.  What’s funny is that Bram, who has been working full-time on VIM for the past few years (living entirely on donations and money votes for VIM features) has now been hired by Google.  Smart move.

Here’s what I wrote:

Hi Miguel,

It looks like a new set of “Halloween Documents” have come out, thanks to a case in Iowa, Comes et. al. v. Microsoft (http://iowaconsumercase.org/index.html). I’m wondering if you have any comments on this document in particular, which suggests that Microsoft management knew full well they were “stealing Java” to intentionally marginalize the cross-platform language issue.

A select quote from the document, “Screw Sun, cross-platform will never work. Let’s move on and steal the Java language.”
Here’s the e-mail archives, which was submitted into evidence:

http://www.iowaconsumercase.org/011107/PX_2768.pdf

I’m wondering, given these thoughts from within Microsoft management, and given the recent news of Sun open sourcing the Java language under GPL terms, how is it that you can still push for the Mono project on Linux? Aren’t we always going to be fighting an uphill battle against a monopoly company protecting its biggest cash cow: the Windows platform?

Although my question was more “devil’s advocate” and meant to rile him up, Miguel provided some of the strongest and most cogent arguments for Mono that I’ve seen on record.

I just want to say great work to Miguel and the Mono team, and that if you ever doubted your raison d’etre, all it would take is reading this thread to be convinced! You’ve certainly convinced me!

Update: it’s really this kind of dependency on Windows I’m worried about in .NET. I think it’s just that the culture of the Java runtime is one of platform independence, whereas .NET from Microsoft is one of “platform dominance,” and Mono is some sort of stepping stone between Microsoft’s single-platform vision and those of us who want to write cross-platform apps using .NET.

I’m not going to say I agree, yet. But I think the author may be right.

The topic is on the common criticisms of Linux, and I think it’s time for me to finally post my thoughts, given the upcoming release of Windows Vista and related products.

Awhile back, /. posted an article entitled “Is It Wrong to Love Microsoft?” The article is horribly written, but does have one point to make: why is it that on /. and other technology news sites like OSNews and Kuro5hin, everyone loves Linux almost unconditionally, and hates Windows unequivocally? I personally am unfond of blind faith as much as I am offended by blind hatred. I’ve thought long and hard about the pros and cons of Linux and Windows, and think I can come up with a pretty balanced evaluation of the two.

Oftentimes the criticisms thrown at Linux are hard to parse, mainly because the authors of these criticisms don’t know much about system design themselves. Having studied it at University, and having been a 10-year user of Windows and Linux (and a 6-year user of Mac OS since version 9), I figured I should chime in.

First, as often happens when evaluating arguments of any kind, one needs to identify the different kinds of critique, and break them down into the smallest possible groups so that they can be evaluated fairly. Broad statements like “Linux is broken” or “Linux is impossible to use” do not identify specific problems, but merely express sentiments, probably acquired through firsthand experience with a particular set of conditions.

Futhermore, attacks mounted at Linux aren’t mounted at Linux “the kernel” but at the whole set of Free (and sometimes, commercial) Software that runs on Linux, which is quite broad and diverse (and not accountable to any one development team, not even distributors). As a result of this, the attacks often seem focused on the whole system, when in fact concern only a small part of the system, and quite often that small part is already in the process of being “fixed”. So, in order to evaluate these arguments, I will begin by pointing out the general complaint lunged at Linux, and then break it down into the proper categories and evaluate each of those in turn, placing whatever context I can to help understand the situation better.

Note for FSF/GNU folks: yes, to your dismay, I will refer to the entire system — kernel, shell, command line utilities, graphical desktop, etc. — as Linux. The GNU folks (especially Richard M. Stallman, whose persistence on this issue is just plain childish at this point) will insist I call it GNU/Linux. Fine, call it GNU/Linux when you refer to it. But truthfully, on my desktop at least, I find GNOME/Linux more appropriate, and on my webserver I find Apache/MySQL/Tomcat/Linux more appropriate, so this petty argument really has no substance. Let’s please set it aside for now.

With that out of the way, let’s continue to the major complaints.

Complaint #1: It’s Hard to Use

Perhaps the broadest and most common critique of Linux is that it’s just plain hard to use. “You’d have to be a geek or a programmer to use Linux.” This complaint has some merit, but often it places too much blame on a particular group (often it is either a big distributor, like Redhat or Ubuntu or SuSE, or the only “widely-known” developer group, e.g. GNOME hackers). The complaint refers to a lot of often related but very specific problems with Linux, so I’ll break it down.

Complaint #1.1: Hardware Doesn’t Work

A user installs Fedora Core or Ubuntu on their laptop, and then discovers the wireless adapter doesn’t “just work.” Maybe their driver isn’t supported, or is only supported through ndiswrapper. If they need ndiswrapper, they might need to go compile the project’s source from source (or worse, CVS) and then find a copy of a Windows driver for ndiswrapper to use.

But maybe the Windows driver comes from the manufacturer’s website packaged as an EXE, which they can’t run on Linux, and so they need to boot into Windows, extract the files, come back to Linux, drop into the command line, place the files in the proper location where ndiswrapper expects them, and then run ndiswrapper’s installation script. Even if the user makes it past this all, he may be at a loss for how to configure his wireless network. He asks on IRC and the forums and people tell him to use iwconfig, iwlist, ifconfig and dhcpcd, all command-line tools, and the user isn’t used to the command line, so this seems very complicated.

I actually quite sympathisize with this criticism. But in this case what you’re criticising isn’t all of Linux, but just one particular part of Linux’s otherwise massive and extensive hardware support. So now, let’s take a look at the full story: in reality, Linux has the most hardware support of any operating system on the market. How’s that you say? Well, Microsoft may have the most hardware support all things considered, but in terms of hardware supported provided by Microsoft as opposed to hardware support included with the Linux kernel, Linux wins hands down. At OSCON 3.3, Greg Kroah-Hartman reported on the status of the Linux kernel and pointed out that “Linux supports more hardware on more platforms than any other operating system in existence, including NetBSD,” and that is why a lot of other operating systems are aiming to simply support Linux drivers so they don’t have to reinvent the wheel. What makes Microsoft “seem” to win is that every hardware manufacturer provides third-party drivers for Windows, and many users are even shielded from this fact because OEMs pre-install 3rd party drivers on computers. There are hardly any Linux OEMs, and even if there were, third party manufacturers tend not to package drivers for Linux. That said, Linux hackers should be proud: they have more built-in hardware support than any OS in existence.

This means that if it weren’t for Microsoft’s anti-competititive practices, third-party hardware manufacturers would have a much easier time getting their hardware to run and supported into perpetuity, since the task of driver development/maintainence would be taken care of by kernel hackers. OEMs would have an easier time shipping Linux systems because driver support would be taken care of for most hardware (not to mention preloading software like CD burning applications and office suites would be free and taken care of by Linux distros). Users, most importantly of all, would be much better off. They know their old hardware will be supported even if the platform evolves, since the source code will always be open.

That said, Linux does have some hardware support issues. But many of these may even be blamed on the users themselves.

This may be hard to swallow. Why are we at fault for the lack of Linux drivers for some desktop peripherals?

In short, because we don’t complain.

A hardware manufacturer will never produce drivers for an operating system with a tiny market share unless the users clamor for that driver to be written. But even more important than drivers, we don’t complain for open specifications. Oftentimes, the biggest roadblock to Linux driver development is that no one even knows how the hardware works. The driver developers have to practice reverse engineering to get a working driver, which is a time-sucking process.

Writing open drivers used to be easier because in the early days of computer hardware, before computers became so ubiquitous, hardware manufacturers often shipped thick manuals with the peripherals they sold. But these weren’t just “use” manuals, they were also specifications of the hardware itself: if the hardware included circuitry, it outlined the functions of the registers, or perhaps the significance of certain pins on a parallel port connector, etc. (Admittedly, I was very young in those days, so I can’t say this firsthand, but both my father and a Computer Science professor have told me about these old manuals, and my professor even once showed me the manual that came of a sound card he bought in the 80s which included full technical specs, which he then used to write a 4.3BSD driver).

Nowadays, with computer use no longer restricted to computer hobbyists, this would be impractical: what’s a typical Windows user need a big thick technical reference for? However, what would be totally practical is for documents like these to be provided on manufacturer websites in PDF form. These documents must exist, since the Windows driver developers needed something to work off: that is perhaps the most frustrating thing. Even more frustrating is that something resembling a specification would exist if any third-party manufacturers open sourced their Windows drivers, but since the culture of free software simply doesn’t exist in the industry (these are still “trade secrets”), the manufacturers don’t even think about it.

To provide some anecdotal evidence against this claim against Linux’s ease of use from a hardware point of view, let me tell you my experience getting support for my AMS E-Cube. That is the system I use as a desktop, which uses a Intel chipset with on-board Realtek ethernet, Intel graphics and Realtek ALC650 audio. I installed both Linux and Windows on this machine. When I booted up with Libranet Linux a couple years ago (before Ubuntu existed, it was the best Debian-based distro), I had to do no hardware configuration. I really mean zero. Every piece of hardware was supported immediately by the Linux kernel. Sound, LAN, 3D graphics, USB devices, everything was working.

Contrast this to Windows. In order to get the hardware fully supported there, I had to go to the hardware manufacturer’s driver page, and install the following drivers: Intel Chipset software (needed for full ACPI support), Realtek ALC650 audio driver (audio had “chirping” noise without this driver), Realtek 8100 LAN driver (LAN wasn’t detected by Windows), ICH5 USB 2.0 driver (otherwise, my system would only work at 1.1), and Intel accelerated graphics (needed for full 3D support). Moreover, every driver came as an EXE which I had to extract, run a setup program, and then reboot my computer. Admittedly, this is because my computer isn’t from a top-shelf OEM, but it does expose the Windows (e.g., Microsoft-led effort) hardware compatibility myth.

Perhaps the most unfortunate thing about Windows hardware support is the inability to easily migrate from one piece of hardware to the next. I had Windows installed on a previous AMS G-Box, and was upgrading to the E-Cube. But because the hardware was so drastically different, simply “swapping the hard disk” caused Windows XP to really crash and burn: blue screen of death, etc. I tried all the directions online for “cleaning out your registry” and “resetting the hardware database”, but even once I got the machine booting it was slow, and unstable. So, like most Windows users, I had to “wipe” and “install fresh.” (This wouldn’t be so bad except that when you wipe a Windows machine, you really have your work cut out for you: you have to reinstall each piece of software again, since there is no package management system; more on this later).

Libranet, OTOH, “just booted” fine. When Libranet came up, I had no kernel panics or anything. All my new hardware was detected, and the drivers for the old hardware didn’t load up anymore.

But I’ve been used to this problem since the Windows 98 days. What’s funny about XP/2000 is that it’s supposed to be able to handle these hardware migrations, as it uses Windows NT’s “hardware abstraction layer”. And even stranger is that I think on some hardware it does handle these migrations. But on my hardware, no dice.

Why did I bother with this anecodte? I think it actually points out one of the big misconceptiions about Windows. See, in my opinion, the only reason Windows hardware support is good is because hardware manufacturers tend to sell computers with Windows preloaded on them. As a result, the hardware manufacturer has to make Windows work with that machine. When you buy a laptop from Dell, it would be unreasonable for Dell to expect you to install drivers for the laptop yourself. To make your laptop work, Dell didn’t just put together the hardware and then install Windows XP on it. No no, they installed Dell’s special driver for this, VIA’s special driver for that, and a handful of other third-party utilities. And then hardware manufacturers provide other functionality missing the “basic Windows install” by licensing third-party programs: To play DVDs they bought a license from PowerDVD, to burn CDs they bought a license from Roxio, to archive files they bought a license from Winzip, to protect from viruses they bought a license from Symantec, etc. See, in the Windows world, functionality of hardware is aggregated only by backroom dealing, with the costs hidden from the consumer, so no one ever knows about it. This probably affects your price tag, something Linux users have often labeled the “Windows tax“. The software certainly isn’t free, so someone’s paying the licensing cost.

I remember I once called Dell about Linux support on my friend’s laptop, and the guy responded with disbelief: “but you’re not supposed to run Linux on our laptops. Our laptops only support Windows.” Indeed. But running Linux on a laptop isn’t like other kinds of “unsupported uses” of a given product. Take a lighter: a lighter manufacturer only guarantees me their lighter will light my cigarette; if I buy a lighter and the spark doesn’t work or it’s clogged and no gas comes out, I have a good reason to return it for another. But if I use my lighter to open my beer bottle, and instead I end up with a bloody hand and a broken lighter, I can’t really complain. That’s a totally unsupported use.

Computer manufacturers aren’t supposed to make computers that “just run Windows,” since users should be allowed to run whatever damn operating system they want on their computers. Some may respond saying “Why? Why should hardware manufacturers care about other operating systems?” Because using Linux on a Dell isn’t like using a lighter as a beer bottle opener. Forcing users to run Windows since it is the only supported operating system is like forcing users of your cigarette lighter to smoke Marlboros because it is the only supported cigarette. “But I’d like to smoke Camels or Carlton Light too, they’re all cigarettes, after all, and all your lighter has to do is light them up” the user says. “Too bad, we only support Marlboro.” Do you think that would be a fair thing for a lighter manufacturer to restrict?

When evaluating Linux’s hardware support, you have to sort of forget about the stuff that is the fault of hardware manufacturers. That is, you have to forget about it when judging Linux on its merits, rather than judging it on someone else’s faults. Look at the good: Linux has open source, working drivers for most of your hardware. But we also need to face up to reality: Linux will never have the “just works” support for all your hardware that Windows has until hardware manufacturers start either producing well-functioning Linux drivers for their hardware, start open sourcing their Windows drivers, or start releasing open specificatioins for their hardware. And that’ll never happen until either we, the users, speak up, and tell hardware manufacturers we want to smoke whatever brand of cigarette we damn well please when using their lighters.

Complaint #1.1: The Desktop Software Experience is Bad

So a user plugs in his iPod to a Linux machine and… nothing happens. We often hear a complaint of this kind, where the iPod can be anything (a USB memory stick, a scanner, a DV camera, whatever). This particular complaint is valid, and has been taken head-on by the desktop development community. Both the GNOME desktop and KDE have been working hard to provide solutions for these problems, and often they have been doing it together through the FreeDesktop.org (fd.o) project. But we have been slowed by the fact that lower-level userspace programs that run in the background (called daemons) weren’t built from the point of view of running Linux on a fully-graphical desktop. So a lot of hackers are at work redesigning some stuff that is under the GNOME/KDE hood, like hal, udev and d-bus. The idea is emit messages when certain events important to a user occur, and provide nice suggestions as to what to do (Copy photos off a camera, open a nautilus window for your memory stick, open up the GIMP for your scanner, whatever). A lot of this work has already been implemented in the latest versions of both desktop environments. It’s certainly not perfect yet, but it’s getting closer and closer everyday.

As for other desktop experiences, I think most of these arguments against Linux desktop software are outdated. I can only speak from a GNOME perspective with authority, but nowadays I think we have good software for doing day-to-day tasks. I personally do document writing in Abiword, which I find much more enjoyable to use compared to the clunky, slow, and featuritis-plagued MS Word. I do browsing in Firefox 2, which I find vastly superior to IE. I do e-mail, contact, calendaring and task management with Evolution, which I find vastly superior to Outlook XP and slightly superior to Outlook 2003; in Microsoft’s latest version, Outlook really got much nicer, but is still catching up to Evolution. Since this claim about Evolution will no doubt be met with skepticism and trolling, I’ll point out that the features I “can’t live without”, and then send you to this article which gives a nice overview of Evolution 2.x: Smarter searching based on “rules”, saved searches, VFolders, powerful filtering, spamassassin-based junk filtering, full iCal support, and integration with the GNOME date/time applet.

So there’s browsing, e-mail and word processing, the Big Three tasks of the average desktop user. What else do I use? There are almost too many choices for music players. I am an old-style guy who likes beep-media-player (which is basically “WinAmp” for Linux) as I’m not really into “music management and categorization” a la iTunes, but there are lots of tools that do that too. File management is made simple with Nautilus, which is quite comparable to Windows explorer and continues to get improvements which make it surpass Windows Explorer (and start the catch-up game with Finder). Check Davyd’s overview of some of the new features from a prior release of GNOME.

All the basics are there, with the one exception of well-done file searching which is being quite rapidly developed by the Beagle guys. When I wrote this article, Beagle was still in its infancy stages, and Deskbar Applet didn’t exist. Now that they both do, file searching as a problem has been solved better than it has been in Windows, even with the latest “Windows Live Search.” I suspect it may be better than Vista’s search support, at least when it comes to support for various backends and its extensibility.

Now you want other applications, like image editors, sound editors, programming tools, document layout programs, and more? Linux has got them, in one form or another, all Free. (More on this later, but just check out e.g. gnomefiles.org if you don’t believe me.)

Complaint #1.3: I have to use the shell (command line), and that’s hard to use

A lot of users complain that the main problem with Linux is that “too much stuff happens at the command line.” For example, I shouldn’t have to drop into the shell to mirror a website with httrack, I should have a website mirroring tool like Teleport Pro.

This goes all against the general UNIX culture. The main reason the command line tool is preferred by UNIX users to the graphical application is that the command line tool is often more flexible. For example, wget, being a command-line tool that generally does fast retrievals from a variety of Internet sources, can be used from any other program that needs to do that task. Furthermore, it can be used by the UNIX user in scripts or scheduled tasks (cron jobs). I’m pretty sure a lot of graphical Linux download managers actually use wget behind the scenes. Because, really, why reinvent the wheel? Why design a whole new application to do spidering and mirroring of websites when you have a high-quality, debugged, speedy tool that does the job.

(Technical aside: It’s true that ideally, re-usable components of software shouldn’t be binary applications. One common problem in UNIX world is that scripts that spawn off lots of separate processes are often a very rapid means of development, but usually aren’t very performant. For example, the UNIX wc utility can be used to count the number of characters in a document, but if you were to code a text editor that ran wc -c on your text document every time a new character was created, it would slow things down quite a bit. Honestly, if the UNIX system were built on a more structured programming environment than C (like Python), we’d probably be better off; that said, the basic contract that UNIX utilities follow (as to standard input and standard output) enables a level of code re-use that’s rarely found in other systems.)

Other command-line tools (generally a part of GNU) have lots of really practical uses, like grep and sed. The tools may have strange syntax, especially for someone coming from the Windows world, but these are very “usable” tools. All you need to do is spend a couple hours learning how the UNIX shell works (including commands like man and info for getting help) and you can find it very natural and productive to use the whole offering of UNIX command line tools. Indeed, recently I attended a two-week long UNIX course in which most students were newcomers to Linux (mostly Java and C# developers). I found that in this two-week course, these students knew more about the UNIX system than I even did. At the end of the day, the “usability” of the UNIX system is quite good, it just has a bit more of a learning curve than “point and click.”

The command line, like windowing systems, is a kind of interface to the user. When I had to train my Mom to use Windows years ago, nothing seemed natural to her. She couldn’t understand concepts like “Windows” and “buttons” and “links” without having me explain things to her very carefully. She couldn’t tell the difference between double-clicking and single-clicking (so she just double-clicked everything). It was hard for her to understand things like “clicking and holding down on the titlebar lets you move the window around.” What’s the titlebar? Why do I have to click there? The task bar at the bottom where each button “represents” a Window and allows you to switch between them. And before she learned the concepts well enough, she always wrote everything down as tasks: “HOW TO OPEN A WEBSITE (1) Click Blue IE button on desktop (2) Click on address bar (3) Type website address and press enter.” Etc. Etc.

When I reflected on that training, that took her a few days to get to be productive in a windowing system, I realized that the command-line’s lack of usability is simply a myth. The command line has a very clear interface: you type in the name of a program, followed by options for what you want it to do, and it will do it. The command line isn’t necessarily a worse interface, it’s just different. And people, when faced with strange and unknown things, do not know what to do with them. Windowing systems may seem highly usable to you, but that’s only because you’re, well, used to them. So you download files by “clicking on a blue IE icon, typing in a url, and clicking save as” and I download files by typing “wget “. That’s only because you understand that process in terms of “Internet Explorer is a web browser which can download files, so I need to pull up a browser window which is ran by double-clicking that browser’s icon, typing an address in the field that application has for web addresses, and using the action the program makes available to save that file.” Meanwhile, I understand my process as “wget is a program that retrieves files, and it takes URLs as its argument, so all I need to do is run wget with the URL as its argument from my shell.” In the end, one can’t really say one process is any better than the other, they’re just different. They have different strengths and weaknesses (yours has a lot of pretty graphics involved with it, mine doesn’t; mine is quicker to do, yours is slower). I don’t want to do web browsing or image editing from the command-line, because the command line isn’t suited for those tasks. But if I need to rename a bunch of files, search through a bunch of text files, mirror websites, or convert thirty jpegs to png files, the command-line is my friend, and should be yours too.

Linux on the desktop is getting to be wonderful because you need the command-line less and less for day-to-day tasks, but it’s still there to give you the proper and needed bit of automation or time-saving functionality. Microsoft realizes this, which is why it’s trying to play catch-up with Windows Services for Unix and Windows Powershell. But due to significant cultural differences, I don’t think either of these efforts will ever lead to the full-featured and inter-connected UNIX system that us UNIX users enjoy.

End of part 1 of this series, “Common Criticisms of Linux, parsed and analyzed.” Part 2 will deal with a second major complaint, “Until Linux runs X, I won’t even look at it…” where X is MS Office, Adobe Photoshop, Macromedia Dreamweaver, and a whole slew of other commercial applications people have grown attached to on Windows.

LINUX AND FREE/OPEN SOURCE SOFTWARE

The State of the F/OSS World Update
with talk/demo by Andrew Montalenti

December 5, 2006 @ 7pm
Room 813, Warren Weaver Hall

Open source software is now mainstream. Whether it’s the nearly ubiquitous Mozilla Firefox browser, the Azureus peer-to-peer client, the Eclipse IDE, or the Linux kernel, almost everything in the computer world has been touched by free / open source software developers collaborating across the globe.

Judging by the state of the community, this movement doesn’t seem to be losing steam. With Microsoft Windows Vista around the corner offering a potentially bloated and hardware-requirements-heavy experience, desktop Linux operating systems are taking aim at the big giant, with big support around Ubuntu, Fedora (Redhat), and SuSE (Novell), among others.

So, what’s next for the free / open source world? That’s what this talk is meant to help you find out. After explaining a bit of the history of free and open source software, and the history of recent community and corporate efforts to make it widely available, this talk will show off some of the new, cool technologies coming out of the open source community, such as 3D desktop effects, productivity tools, enhanced multimedia support, better support for laptops, and a full suite of industry-grade development tools. The talk will also discuss some of the legal and intellectual property issues facing the open source community, with a particular focus on the recent news coming from the Novell / Microsoft deal and Sun’s decision to open source Java.

Who is this talk meant for? Anyone who hasn’t tried out Linux on their desktop, or anyone who is at least mildly interested in the current and future state of the computer industry. Free / Open Source software has completely rocked the industry, changing every aspect of it from top to bottom, and this wave is only growing bigger every day. NYU students interested in copyright issues surrounding open source may also find this talk valuable.

In any event, this won’t be a boring lecture — it’s meant to be interactive and fun!

The speaker will be bringing free CDs of Ubuntu Linux, a community-driven desktop Linux operating system which you can install on almost any home PC! Come for the free CDs, stay for the revolution.

It went very well, with about 10 people in the audience. You can download the slides in OpenOffice or PDF. Admittedly, the slides aren’t as cool without the live demo of Beryl I did at the talk itself. Yay 3D desktop effects.

I am trying to use Evolution with SpamAssassin, and finding horrible slowness and bugs in the bayes_* databases to be the norm.

If you are attempting this setup, I recommend the following hacks:

1. Do not use bayesian autolearning in spam assassin, as this is a broken way to update your spam filters.
2. Instead, create a JUNK and HAM directory in Evolution, and move your junk mails and ham mails there.
3. If you have already tagged mails as Junk using Evo’s “Mark as Junk” feature, you need to hack these mails out of that folder and move them into a regular Evolution folder. “Mark as Junk” is really not moving mail anywhere, but just “tagging” it, and then the Junk folder is like a vfolder which searches for all mail tagged as junk. This is cool, but sucks if you want to run sa-learn on your mbox in order to train spamassassin. The way to get around it is to create a mail filter in Evo which says: “if mail is marked as junk, then mark it as not junk AND move it it to folder named JUNK.” Then you apply the filter to your Junk directory and Evo will copy them out. (Note: if you try to merely “copy” the mail messages to another folder, they will simply not move — again, Junk is a “tag” in evo, not a location).
4. Now from the shell, go to .evolution/mail/local and find your mbox for your mail. You can use sa-learn –spam –progress –mbox to learn it as a spam, and change –spam to –ham for ham.

This is much better than using the built-in Evo stuff. Do this every few weeks until you don’t have to anymore.

More and more advances are going to inevitably lead to Linux “eye candy.” Here’s a great post on Slashdot regarding good versus bad eye candy.

Here’s a great example: pull-down menus in Mac OS X vs the same in Windows XP. On the Mac, pulldowns appear instantly, and fade away once something is selected; this is correct behaviour, as you asked for a menu – there should be no delay. Fading away is fine because the selection has been made, and you have moved on. In XP, the menus fade up, and vanish instantly – totally backwards. That is bad eye candy.

You know, I never thought about that, but I always turned off menu fading on Windows XP because I could “feel” the delay, even on my Nvidia card on my desktop. I never realized how much more sense it would make to display the menu instantly, and fade it away upon selection.

I hope this conservative approach is taken by Linux devs as they implement eye candy. Functional and beautiful, please.

for i in `seq 1 8`; do
  wget http://www.livinginbaires.com/imgs/departamentos/46-$i.jpg;
done

And that’s it, you hit enter and *vroooooom!* You see progress bars fly across your terminal as wget snappily fetches your photos, and suddenly, all eight photos are downloaded, and you get back to work. That’s empowering.

Here’s the blurb I sent out when I advertised the talk:

Have you wanted to work on open source projects, but just don’t know how to get started? This talk will provide the basics you need to start working on open source software the next time you sit down at your computer.

In particular, this talk will cover:

(1) A brief overview of open source development in the industry and press.

(2) The UNIX development platform. A brief and whimsical overview of the UNIX shell, its surrounding tools, and the power of shell scripting. (Useful to anyone wanting to learn more about UNIX tools.) Learn how to do in a few lines of shell script what you only thought was possible with a big, extravagant hundred- or thousand- line program, and learn why so many of the world’s best hackers hack on a *nix system.

(3) The basics you need in order to hack on open source project: how mailing lists, wikis, bugzillas, source code revision systems all come together to form an organic code management process, and how to get started using those tools and others to learn about a project and what parts need development work done. This will include a brief introduction to CVS.

(4) The last part of the talk will involve actually watching open source development in action. In particular, the speaker will checkout some code from a source repository, make a change to it, create a patchfile from that change, then track down the mailing list or bugzilla related to the project and submit the patch to the maintainer. You will actually get to see open source “in action,” and will want to do it right when you get home!

If you’re a Windows or Mac OS developer who has always wanted to learn more about *nix systems, or if you’re a developer who wants to either take his own project open source or work on existing open source projects, this talk is for you.

Of course, if you’re someone who is just interested in the concept of open source, this talk will give you an inside look at “how things get done” in this community.

This talk was at least partially inspired by Nat Friedman’s blog post, check it out here:

“How to be a Hacker”

The talk was a general success, I think. About 12 people attended. You can see the talk in PDF or ODP formats, and you can also download the patches I wrote specifically for the talk to illustrate “open source development in action.” The patches are pretty stupid, but do illustrate the point, at least. Plus, each of the three patches served one of my own goals (hacking my CPU frequency scaler, fixing a gnome-terminal bug, and hacking galeon “for fun”), so that’s that. I think it’d be cool to give this talk again (maybe a little refined to include less basic UNIX tools and more hacking stuff) at a later date. We’ll see.

I always wonder about these Linux puff articles — what motivates the writers to pimp OSs that any experienced person knows will be nosebleeds for 90%+ of the novices trying to do a real, complete migration from XP?

Linux never has been, nor will it ever be, an effortless turn-key OS for all-purpose desktop (let alone laptop!). Ubuntu is an improvement, but it’s still Linux — which, if you consider the costs of replacing unsupported hardware and your time struggling with setup at about two cents an hour, is the most lavishly expensive OS ever devised.

I’ve been screwing around with Ubuntu 5.04 for about five months, all but living on the official support forum, which provides — at best! — solutions for only about a quarter of the hardware and software problems I’ve encountered, and there are tons of them.

When 5.10 was released, the forum exploded with bug reports from those who had upgraded from 5.04 and found they had major-to-fatal problems.

I’ve tried to install 5.10 on my new laptop, but the VIA video is not supported, nor is the RaLink wireless, nor WPA authentication…etc., etc., etc.

If you want to spend the rest of your life talking with geeks (and in the process learn that OS advocacy is best understood as a psychiatric symptom above all else) and trying out an infinite number of kludges, you can get a lot of these problems worked out (more or less), but what you normally wind up doing is succumbing to what I call “Linux disfunction drift,” eliminating tasks that you used to do in Windows because your Linux distro won’t do them, or support the hardware you needed to do them.

There’s also a larger problem with this approach – it sucks for small companies trying to become bigger.

If you are only able to profit off of service contracts, you can’t ‘write once, reach many’ like you can with COTS software. Moreover, companies like IBM and Novell which have large established sales and service teams will win all the larger contracts.

If you write a great peice of software, and then have to sell, educate the customer AND hire/train all the workforce, how much time are you going to have to devote to Rev. 2 of your world beating product?

Whenever folks talk about OSS in the context of markets, I think it should be with a jaundiced eye towards our “helpmates” at IBM, Novell, SAP/MySQL and Sun.

Ultimately, IBM et al are about making money for shareholders, if they didn’t see that as the likley outcome, they would not be out there pimping OSS.

I think a world where software is only ‘sold’ in the context of a service contract is bad for the next great idea. OSS is great in its place, but to preclude software for sale isn’t the answer.

The truth hurts for Free Software zealots, but it’s the truth.

Free Software isn’t about eliminating proprietary software, at least it shouldn’t be. It should be about developing a free system for development, learning, and sharing, because we can.

I think Mac OS/OS X handles the MDI plague best by simply grouping all application windows under a single application class, with a single menubar. But usability experts have debated whether that makes the most sense, since the menubar can change, for example, depending on what window is in focus. I think users get used to that, and it also allows the menubar to be as long as necessary while the window can remain as small as necessary. That’s a nice win.

However, given our current model on *nix/Windows of menubars for every reasonably complex window, and given the lack of the MDI hack in GTK+, we do have a mess for applications that need more than one window to operate properly, i.e. Glade and The Gimp.

What I’ve been doing is giving these programs their own workspace as a workaround. That seems quite greedy of them, and indeed it is. What’s more, however, is that it’s unusable. Even when I switch to my Glade workspace, I see 4 windows in my taskbar, each with the same icon and with the following names: “Glade: h2h”, “Properties: image95″, “Widget Tree”, “h2h”. The first three are actually part of the Glade window class, and if I enable Metacity’s taskbar grouping, I see them as part of the same group. The last one, however, is just my actual window, and so is separate.

This is good–but I only get some form of usability when I actually enable window grouping. The thing is, in Metacity you can’t enable window grouping on a per-application basis. It’s all, sometimes, or nothing. Sometimes means metacity only groups windows when I’m running out of space. Otherwise it’s either on or off.

The thing is, grouping isn’t just about space saving. It’s about being able to perform window manager operations on a group of windows, i.e minimize all and maximize all.

This doesn’t even solve all problems: alt+tab still shows me all 4 glade windows, which can be quite confusing since only one comes into focus at a time. But that’s a separate issue, separate debate.

Wouldn’t it be nice if instead of Metacity just “figuring out” when to group my windows together, it let me just press a hotkey “Group all X windows on this workspace”, where X is the application I’m currently in?

I’ve decided this feature is so valuable, I may just hack metacity to add it. It will at least provide a path for solving the MDI nonsense.

Update: check out these screenies of a “different” approach to MDI written in GTK. It’s called GTK ADI.

We spent about 3 hours there, just talking about Runar’s project, “sqlstring”, my ideas about inferred typing and static source code analysis in Python, Python’s niceness in general, user interface toolkits, AJAX being a big, nasty hack, and web application frameworks in Java and Python. Our discussion really degrenerated into praise of vim once we discovered that we were both happy users. Text editors really bring people together.

Runar kind of convinced me that trying to infer all the types of objects is very “unpythonic,” which I guess is true since it discourages the crazy stuff you can do with Python. Maybe the best thing to do is judiciously eval code, as was my original impulse for getting nice completion out of Python? Not sure.

Or maybe I should just give up the idea and accept the fact that vim plus ipython is just about as good as it gets. That seems like a cop-out, though.

Regardless, Runar seemed somewhat willing (only half-willing) perhaps to give a small talk for Free Coders on Python, I’ll see if I can convince him that it’ll be fun. I suppose I could give the talk myself, but I already do all the talkin’.