Programming

Development under Windows: why so painful?

Friday, October 14th, 2005

It’s really weird. Lately, I’ve been doing so much development in a *nix environment, that doing the development in Windows is really painful for me. I don’t have any of my good old UNIX tools, I don’t have hotkey-optimized user interfaces, I don’t have speed and control. But more than anything else, I don’t feel like I know what’s going on under the hood.

Today, to take a break from reading Philosophy, I decided to work a bit on this little Java Servlet project I’ve been hacking on. (Will be “released” later.) At some point this past summer, I decided to remove Linux from my main desktop machine and just consolidate all my Linux data onto one machine–this made my life easier so I didn’t have three total (one Windows, two Linuxes) places where my shit could be. But the sacrifice is that my laptop screen is small, so sometimes I want to develop with a big screen and thus want to use my desktop.

Web development, especially, makes sense for me under Windows, since I’m comfortable with the major graphic and web design tools (Photoshop, Dreamweaver, Illustrator) and don’t think the Linux “equivalents” (GIMP, Bluefish, Inkscape) are good enough.

But I decided–may as well have the code open on Windows too, since it’s not C hacking I’m doing, but Java. So I installed Eclipse, and the J2EE, and got cracking.

But under Windows, there are all sorts of gotchas. When my UNIX tool craving gets really bad, I need to drop into cygwin, which isn’t so bad. But without good workspace switching (I have VirtuaWin, but it kinda sucks), and without a customizable window manager, I am really much slower. But here’s the other weird thing I ran into. After awhile of coding, I realized that Eclipse wasn’t reading my JavaDoc information for the JDK (no cool descriptions in my autocomplete tooltips). So I go snooping around the preferences file and can’t find anything, I enable a billion options but no luck. But then, eventually, I realize that it’s very possible Eclipse is using a different JDK. In fact, I look in the dialog, and Eclipse is using some J2SE environment that some other application installed, not the J2EE I installed right before Eclipse. And that J2SE is missing the Java API source code and JavaDoc comments.

The reason this seemed so non-obvious to me is because I’m not used to systems which are completely fucking disorganized. Say what you will about Linux not being user friendly, but, by God, you won’t find it likely to find two different JDKs installed on my machine, and even if you do, only one will be getting used (thanks to Debian’s “alternatives” system). Every application on Windows statically compiles, includes its own libraries, and spews its shit all over the file system and registry. No database tracks it, so your system is a fucking nightmare.

I couldn’t even do a reasonable search to find the JDK I needed, either. It turns out it was in C:\Program Files\Sun\j2sdk1.4_02, which may not sound so bad, but considering on Linux I just think, “Where are libraries stored? /usr/lib” and then in there I think, “What is what I’m looking for called? j2sdk” I quickly find any Java environments in /usr/lib/j2sdk1.x-sun.

On my Linux system, which has not that much installed, I just ran a du -hs /usr/lib/ and got 1.7GB. That means on my relatively lightly-loaded Linux system, 1.7GB of raw 0s and 1s are sitting there waiting to be used as SHARED libraries. Meanwhile, on Windows, there could be any amount of duplication of the equivalent libraries, floating around in various Program Files directories.

I can’t believe there are acutally some Linux critics that believe we should be going in this direction, eliminating things like emerge, apt, and rpm and instead just have statically-compiled binaries that come with their own binary libraries and have users duplicating this stuff across their system. Not only is it insane from the point of view of giving control to the user, but it’s also just plain wasteful.

Just For Fun: The Story of Linus Torvalds

Thursday, September 22nd, 2005

For the last couple of weeks, my bedside reading has been this half-biography, half-autobiography on Linus Torvalds. I have to say, however, that the book is like two books mixed into one. Chapters alternate between Linus talking about his life and about big moments in Linux’s history to David Diamond describing modern-day Linus with a kind of forced wonder. Truthfully, Diamond comes off as a sycophant who could care less about Linus’s flaws and positive characteristics, and cares more about molding some kind of “image” of Linus as containing a humility and genius simultaneously. Near the end, I started only skimming the chapters not written by Linus. Diamond’s really not a good writer, either. (Sorry Dave.)

Truthfully, the book kind of pops the lid off Linux and makes you understand it as much less glamorous than say Wired Magazine described it to the public. Linus really just talks about not having a social life, sitting in his room with curtains covering his window, coding all day. Not exactly the ideal role model, I think. Don’t get me wrong, I love the Linux kernel (as much as one can love imperfect software), and Linus made a great contribution toward keeping the UNIX world and UNIX principles alive, but it’s just that I like to think of open source developers as something other than the stereotypical, introverted geek. In fact, much of Linus’s chapters is devoted to his apprehension about giving a public talk about Linux. When I think about the fact that I’ve given three or four of them to date, and enjoy it more every time, I see how different I am from this kind of stereotypical geek.

It also kind of made me dislike Linus. When I saw Revolution OS (a DVD on the rise of open source), the movie kind of endeared me to Linus’s practical nature as opposed to Richard Stallman’s religious idealism. I like idealism, but Stallman is really religious about it. And he’s bitter. Linus, on the other hand, has that great Northern European, “I’m just gonna go with the flow” attitude.

But this book made me realize that Linus is religious is his own sort of way. Included in the book is Linus’s flame war with Andy Tanenbaum on monolithic versus microkernel designs. Truthfully, I’ve studied operating systems and I’m not even sure which design is best, and Linus makes a decent argument of why microkernels end up being just as complex, or more complex than monolithic ones. But what I didn’t like is that in the flamefest, Tanenbaum said that deficiencies in MINIX were due to it being a hobby, and that he had duties as a professor. Linus responded, “Re 2: your job is being a professor and researcher: That’s one hell of a good excuse for some of the brain-damages of minix. I can only hope (and assume) that Amoeba [Tanenbaum’s future OS project] doesn’t suck like minix does.”

This just shows me that Linus really is an asshole sometimes. He states this outright in his book. So now, truthfully, I may like the open source movement, but I think I “at least dislike” two of its most major players (Torvalds and Stallman).

Finally, I think a clip from Tanenbaum’s website points out a nice principal in OS design:

Also, Linus and I are not “enemies” or anything like that. I met him once and he seemed like a nice friendly, smart guy. My only regret is that he didn’t develop Linux based on the microkernel technology of MINIX. With all the security problems Windows has now, it is increasingly obvious to everyone that tiny microkernels, like that of MINIX, are a better base for operating systems than huge monolithic systems. Linux has been the victim of fewer attacks than Windows because (1) it actually is more secure, but also (2) most attackers think hitting Windows offers a bigger bang for the buck so Windows simply gets attacked more. As I did 20 years ago, I still fervently believe that the only way to make software secure, reliable, and fast is to make it small. Fight Features.

I agree. But does a microkernel design actually reduce the overall size of the operating system, or does it just reduce the size of whatever you consider to be the “microkernel”? That is, just because a file system is implemented as a file system daemon talking to a driver subsystem through message passing doesn’t necessarily mean the file system, or driver subsystem, are secure. Insecurity could exist even at the boundaries, no? Not to mention instability.

I think Linus and Tanenbaum have to agree that this debate isn’t an open and shut case. The best kernel is probably one that mixes modularity, a strong kernel/userspace boundary, and some of the fancier features of a microkernel approach, while not sacrificing elegance of design or performance.

Free Coders at NYU

Wednesday, September 21st, 2005

I’m organizing a group of people interested in hacking open source software in a team environment. Right now I’m calling it Free Coders at NYU, and have already set up a wiki and mailing list. This could end up being very cool. Next meeting is hopefully this coming Tuesday.

I set up a mailing list with GNU Mailman (link above), which was decently painless under Debian Sarge. The only annoying thing was utilizing my virtual e-mail address mappings which are stored in MySQL, but I figured out a trick for that.

I’ve already spoken, via e-mail, with an open source developer who works on gstreamer among other projects, Ronald S. Bultje. He has already tentatively agreed to do a talk for us sometime this year.

N-way parallel mail retrieval with getmail and bash

Friday, August 5th, 2005

I wrote a pretty sweet script tonight. It parallelizes the getmail retrieval process, while still printing prefixes so I know which accounts download which messages. This means that instead of my mail fetching process taking sum(i1,…,in), where i is the length of time for a given mail retrieval, my fetching process now takes max(i1,…,in).


#!/bin/sh
GETMAIL='python2.3 -Wignore /usr/bin/getmail'
unwanted() {
  grep -E -v '(Copyright|getmail|Simple)';
}

echo "N-WAY GETMAIL RETRIEVER SCRIPT:"
$GETMAIL \
  --rcfile=/etc/getmail/account1 \
  2>&1 | sed -e \
  's/.*/account1................: &/g' \
  | unwanted &

$GETMAIL \
  --rcfile=/etc/getmail/account2 \
  2>&1 | sed -e \
  's/.*/account2................: &/g' \
  | unwanted &

...

$GETMAIL \
  --rcfile=/etc/getmail/accountN \
  2>&1 | sed -e \
  's/.*/accountN................: &/g' \
  | unwanted &

wait

“Hello World” in every language

Wednesday, August 3rd, 2005

This is why I love Wikipedia.

PIDA: Python Integrated Development Application

Wednesday, August 3rd, 2005

PIDA 0.2.2 was released recently. This is truly a novel development in the Python/OSS world. What PIDA provides is a nice plugin system and the “makings” of an IDE. So, in a nice IDE you have a class browser, an integrated debugger, a profiler, maybe even a RAD-like GUI builder, an interpreter console, etc. The one piece that tends to be most controversial in every IDE is the text editor. This is one-part because UNIX people are really crazy about their text editors, but two-parts because text editors are very important programmer tools, and no one wants to learn a different text editor for every language one uses.

vim happens to be awesome for C programming, which is probably why a lot of UNIX hackers use it. But for Python, more advanced support would be nice. PIDA can run and connect to a vim server instance in order to allow you to have an “add-on IDE” for vim.

But even more interesting to me is the culebra plugin, which provides a code-completion-savvy GtkSourceView inheritor, which has the initial support for fancy Intellisense-like features.

I’ve already spoken to the developers of PIDA, and they said they would very much be interested in seeing Python Intellisense features brought to VIM. When I started thinking about different approaches to doing this, I realized that the whole OSS community could benefit from a general Python module that enhances the Python introspection features (and perhaps combines them with source code parsing) to make available nice productivity-enhancing features. I was thinking of calling this beast “Pyductivity.”

More on that later. For now, check out PIDA.

Exa: a new architecture for Xorg

Tuesday, August 2nd, 2005

This is exciting news. A Trolltech developer has modified KAA, the acceleration architecture used in Keith Packard’s experimental “kdrive” Xserver, to work with the traditional Xorg tree. He announced this new development in an e-mail that makes it clear it is extremely easy to get drivers to use Exa to gain Apple/Windows-like graphics performance.

I know that the unichrome project generally doesn’t bother itself with these very desktop-oriented features (their focuses are more on MediaPCs, etc.), but I think this may be an excellent way for me to begin hacking the new modular Xorg tree I mentioned last time. If I added Exa support to the unichrome driver, would that mean transparency and full-on graphics acceleration for my X desktop, what I’ve long been waiting for? We’ll see.

Xorg goes modular, is now approachable

Tuesday, August 2nd, 2005

It seems that Xorg, as of the 7.0 release, has been split into monolithic and modular development trees. The modular model allows you to compile individual components related to Xorg separately from the whole X server, so that you don’t need to do a two-hour compilation just to work on this or that driver or this or that library.

This is good news for me. My biggest craving lately was to put my C skills to use by diving into a big project that has effects on the Linux desktop, and Xorg is certainly the biggest in that sense. However, in the past I was always put off by the huge amount of groking one needs to do just to understand the Xorg compilation process. After class is over, I’m gonna start diving in.

Amdahl’s Law

Thursday, July 28th, 2005

Gene Amdahl once applied the law of diminishing returns to computation. He pointed out that when optimizing part of a computer program or computer system, one must take into account what percent of the overall task at hand that optimization affects.

I recently read some articles comparing the speed of Python to Java, most of which concluded that about the only place that Java beats Python is in actual interpreter speed (i.e. how fast statements and parsed and executed), and that since Python opts to provide thin wrappers for standard C libraries, Python performance ends up being really good.

A good comparison of the language features between Java and Python can be found online, along with a nice comparison of code simplicity and efficiency.

I think I agree with the first author: Python is a better high-level language, and should thus be used for higher-level tasks, and especially for one-offs. What’s interesting is that a lot of people look at Python, say “Man, Python is slow, I could do this better in C,” but then forget about Amdahl’s Law. If your program is accessing the network, the disk, or any other non-CPU/non-memory resource, no amount of optimization through lower-level languages will save you an order of magnitude on performance. So why waste the programmer time, when it can be done in a few lines of Python?

(I think one forgotten benefit of Python in both these articles is SWIG: if you’re truly a performance-oriented engineer, you can always profile your code, find the bottleneck algortihm/code fragment, rewrite it in C, and wrap it with SWIG so that you can access it as an object in Python. Not hard to do, and potentially huge performance gains. OTOH, you can even write Python-accessible code directly in C, using the same abstractions the Python interpreter uses.)

Joel Spolsky on programmer creativity

Tuesday, July 26th, 2005

Joel has a new article entitled “Hitting the High Notes”, which probably deserves a link.

I happen to fully agree with Joel, that good programming is about finding that “clever solution,” not just a solution that “barely works.” These little epiphanies may not just be the best product of good programming, but also the reward–another striking similarity to conventional artists. I started to realize this similarity myself a few years ago, when I scribbled in one of my notebooks that programming actually seems a lot like sculpting, in that the task is both highly creative and highly technical, where the goals are simultaneously to solve a complex problem and produce something “beautiful” (or, in programmerspeak, one might say “elegant”).

This ideas of mine didn’t find themselves blooming until I came across Frederick Brooks (who, in fairness, Joel mentions), in whose wonderful collection of essays is a very useful distinction between two components to programming. The more important one, says Brooks, is the essential component, which should be contrasted to the accidental. The essential problem to programming is that it is fundamentally a creative act, and as such it requires long periods of uninterrupted thought and even a spark of inspiration to result in amazing products. The accidental component to programming is everything else that incurs cost in time: dealing with the programming languages, the text editors, the compilers, software revision systems, documentation, and all the other software development tools. Brooks’s theory (which, in my opinion, held) is that there was “no silver bullet” for programmer productivity; that even if major leaps are made to reduce the effect of the accidental problem, the essential problem would remain, and without an improvement in the essential, there would be no order of magnitude improvement in productivity.

In other words (and metaphors), even though word processing is a lot easier than typing–which was itself a lot easier than writing by hand–it doesn’t seem that any of these advancements in “text editing” made the creation of great novels much easier, since it is the content of a novel that matters–presumably–and that comes only from within the author.

But Joel is mostly right: to fight the challenges inherent to the essential problem, one must have:

  • A great work environment
  • Talented colleagues
  • Challenging problems
  • Visionary direction
  • Lots of alone time for each developer
  • Lots of brainstorm time among developers

There are probably a couple of others that deserve to be on there, but that’s a pretty good list in itself.