Hey Scott:
What is your opinion on DLP TVs and 1080p Hi Def, if you have one? I know that the only way to get 1080p today is via HD-DVD or BlueRay, as nobody is currently broadcasting at that resolution. Any tech info or opinions?

Bob

Well, Bob, some say 1080p vs 720p is hard to tell apart. It depends a lot on viewing distance.

You might have the tv for 5-10 years, so get a good one. The tv should be able to upsample well from any lower res.

The prices have come down a ton - a year ago I’d have said wait but now I think the prices are more realistic. I’d think the really high quality tvs, good brand, are really worth it. They do use better components and have better software doing the processing, from what I understand.

When you see a price that is too good to be true, don’t go for it. All of this is relatively new technology, so when problems crop up in a particular model, you can expect to see a big discount.

If you’re shopping at a big box retailer, check the model number, it will often be unique to that chain of stores (Costco, for example). Why would they would make a separate model number for that discount store? Is everything exactly the same? Can you tell? What does a manufacturer do with its “grade C” components? I don’t know, but you must compare apples to apples.

Make sure the TV you want has everything the videophiles tell you it shoud have. TVs have been evolving very quickly, and standards have been evolving. There are sets from a few years ago that will likely not be able to play some content at all, ever, due to some standards and DRM changes. HDCP compliance is one of those things. I believe most of this is settled now, but it’s no longer the 1950’s where the FCC dictates everything so that every TV is always compatible. There are probably still TVs out there on the market that lack things that you actually will need.

You might jump at an under-$2000 price without agonizing over every detail. But you need to consider the pain of switching out a TV, trying to fit it into your built-in cabinet, hooking up wires, and teaching your family how to use the new remote control. You don’t want to have to go through that again in a month, so research the TV like you’d research a car.

More importantly, you’ll probably be looking at this equipment every day for a few years. Will you get a warm fuzzy feeling when you sit down to relax with your favorite show, or will you be thinking “I should have spent a few hundred more bucks!”

But yeah, my gut says 720p is for chumps, and that there’s no excuse for 1080i. Get yourself the best 1080p that fits in your budget. Bigger is not better - get the right size for the room.

Oh, and I like DLP if the thickness works for you.
Thanks for writing,

Scott

At Costco today, I was struck by two things. The first was how cheap the huge, flat-panel HD TVs have become (a gigantic Mitsubishi LCD was $2000). The second was how horrible the picture quality was.

The incumbent video delivery industries are focused on moving to “HD” standards. In the short term, this means “highly compressed with lots of artifacts” - see, for example, Comcast HD. In the very long term, the compression problems may go away as equipment gets better and as more bandwidth is dedicated to HD services. But “HD” standard screen resolutions are pretty well set in stone and aren’t going to change for a long, long time.

On the other hand, in the very short term, display technology will allow much higher resolution displays at ever decreasing cost. Similarly, video recording and digital editing equipment will allow higher and higher resolutions at consumer-level costs. As Internet bandwidth increases and drops in price, it can easily address the video quality problem on all fronts - less compression needed as a full connection is dedicated to one or two video signals, quicker adoption of new, better compression technology as software upgrades are immediately available and expected, and higher resolutions to match cheap, high-quality displays (by leveraging increased bandwidth and improved compression methods). By comparison, “standard” equipment with set resolutions and set video codecs will age very quickly and not very gracefully.

Those who are used to standards staying the same for decades will stick with “HD” and think it’s great. Meanwhile, a disruption should occur - consumer-grade video quality will outstrip the capabilities of “pro” equipment, and that high quality video will only be viewable on “non-standard” systems - ultra-high resolution monitors connected to the Internet, instead of a standard “high def” TV connected to cable or satellite.

Yes, this means that in the future we’ll have ultra-high-visual-quality videos with bad lighting and horrible content a-la YouTube. But it also means that the cable TV and satellite TV services need to move quickly to video over IP to avoid disruption. These days, a data service that is dedicated to providing only video makes about as much sense as running another cable to your house to provide MapQuest service.

I’m stating the obvious - the cable and satellite companies surely see video over IP on the horizon - but hey, stating the obvious is what blogs are for. I’m just doing my part.

Software Development is Sometimes Sorta Like Driving to the Airport

Two people, Alice and Bob, unbeknownst to each other, needed to be at the airport Friday morning to catch a 10AM flight from Chicago to Oregon. Alice and Bob each have nice vacation homes in the Oregon hills near the ocean.

The night before the flight, Alice and Bob both packed their bags and made sure they had their plane tickets. They even both double-checked that they had toothpaste, since it is something they have both left behind on various business trips.

Alice spent ten more minutes thinking about her trip. Bob spent ten minutes reading the paper - he knew he had forgotten toothpaste before, but he figured if he forgot anything else, he could buy it after he lands.

Alice tried to make a list of things she needed for her trip. She started writing things down, and realized that she had already packed everything on her list. She kept writing, and after 10 minutes or so, it was easy for her to remember all of the stuff she had already packed, but she wrote it down anyway. She wrote down “keys to the vacation home”, “toothpaste”, “sexy nightgown”, “my book”, then did a double-take and realized the keys to the vacation home were in the kitchen drawer. She checked the rest of the list, marking everything but the keys as packed, then grabbed the keys out of the kitchen drawer and put them in her suitcase.

Bob finished reading the paper, glanced at his suitcase, noticed that he did, in fact, have plenty of underpants, and went to bed.

We now know what is going to happen to Bob. At some point, he, too, will realize that he is missing the keys to his vacation home. Even if he had made a packing list, he might have still forgotten to pack his keys. Bob doesn’t get much vacation, so it’s not surprising that he would forget.

What is important for Bob now is to understand how time gets more expensive as time goes on.

Bob put his bag in his car and started driving. It normally takes 30 minutes for him to drive to the airport. He left at 8:30 to give himself an hour to park and get through airport security. At 8:45, half way to the airport, he realized he had forgotten his keys. He thought about turning back, but it was another 5 miles to the next exit. He finally decided to continue to the airport - he had an unrefundable ticket that couldn’t be rescheduled, and if he missed his flight he would be out $200, plus there wasn’t another flight until Saturday.

He got to the airport early, at 8:50, and found out that his flight had been delayed a half hour to 11AM. Lucky break. He could drive home to get his keys, but he would have to get his car out of long-term parking (10 minute tram ride), pay for a day’s worth of parking ($14), drive home and back (30 minutes), park again (another 10 minute tram ride), and go back through security (another 10 minutes). There was time, but it would have been a huge pain, plus an extra $14 for the parking that just seemed unfair.

He could have called his neighbor, Carol, and asked her to drive the keys out to the airport for him, but he’d still have to go outside of the security gates, and he kind of liked Carol and would hate to ask her to go out of her way, and plus it would have been a bit embarassing to tell her that he forgot something.

Bob bought a magazine and sat down next to a charming young lady. He looked at the tags on her bag and noticed that her name was Alice. A nice name. As he read his magazine, he noticed Alice’s perfume. He liked it. He relaxed a bit and thought about his pending vacation, and wondered if Alice would be on his flight.

As Bob waited for his delayed flight, another announcement was made - another 40 minute delay! Not too surprising, this airline was cheap, but they were also usually late. He thought again about calling Carol, but Alice’s perfume kept distracting him. He decided he’d rather stare at Alice’s bare ankles while he pretended to read his magazine rather than call Carol to ask for his keys.

Finally, good news came over the public address system - the flight would be departing at 11:40, and boarding would begin immediately. Alice put her romance novel in her oversized purse and glanced fetchingly at Bob - “Guess this is us? Too bad I’m in row 39, I hope it’s not lonely back there.” Bob glanced at his boarding pass - seat 39C. He tried to act cool. “Yep, this is us! And it is a long flight, I hope you’re not too lonely back there.”

“We’ll see…”, says Alice with a smile, and Bob watched her walk slowly to the boarding line.

Bob headed back to Row 39 and saw Alice sitting in 39B. She was watching him walk back, as if she already knew he’d be sitting next to her. He sat down. “Well what do you know,” said Alice with a smile.

Suddenly, a stewardess made an announcement to the plane. “The engine that we repaired earlier has caught fire once again and it will be 45 minutes until the flames are out. You may return to the airport if you like, we will board again in 35 minutes.” Bob sat and chatted with Alice, and the 45 minute delay zipped by quickly. He couldn’t believe his good luck - Row 39 never looked better.

The plane finally pushed back from the gate, taxied to the runway, and was airborne. Goodbye Chicago! Hello Oregon! Bob and Alice talked the whole way. Alice laughed at all of Bob’s jokes, and stared deeply into his eyes when he talked about his hobbes, his work, and his dreams. When the plane touched down in Oregon, Bob was a bit disappointed that the flight had gone by so quickly.

As Bob waited for his suitcase at baggage claim, he wondered where Alice had gone. He also thought about his keys and how he would get into his vacation home. He didn’t have any extra keys in Oregon. He started making plans to call a locksmith. The vacation home was a two hour drive from town, so he resigned himself to spending 4 or 5 hours waiting for a locksmith to meet him up there. At least he could call from the airport to get the ball rolling.

He went to a payphone and dialed information. He got the number for two locksmiths. It was still early on Friday, so he didn’t anticipate any problems. The first locksmith he called didn’t pick up the phone. The second one was answered by a recording: “Thank you for calling Bumpers Locksmith Service. The Oregon Locksmith Union strike negotiations are still underway, though we don’t expect a resolution to our problems until the end of the month. Until then, our services are unavailable.”

“Locksmith Union?? Who are they striking against? What the hell is going on?,” Bob wondered to himself. That option was definitely out. He started thinking about how he could break in to his own home, but he knew he had meticulously secured his house against intruders the last time he was here, he knew he would be away for a long time and everything was locked down tight.

He called Carol’s cell phone. She could FedEx the keys, and he could have them tomorrow. He could spend the night in a hotel room, maybe get a nice deluxe room and order room service, that wouldn’t be too bad, then he’d have the keys tomorrow morning at 10AM. Carol picked up.

“Carol, this is Bob. I’m in Oregon.”

“Bob, how nice. It must be lonely there. You should think of bringing a guest the next time you go…”

“Carol, I need you to do be a favor.”

“Hm… too bad you didn’t think of me until after you left… but, anything you want, Bob, just tell me what it is. You need me to fly out there?”

“Carol, no. I need you to go in my kitchen and get my vacation home keys, put them in a FedEx envelope and ship them to me Saturday delivery. I left the keys there and I need them. The key to my house is under the door mat.”

“You want me to ship your keys to you? That’s it?”

“Carol, come on, I’ll owe you one…”

“Hm, Bob, I don’t know how to break this to you, but I’m on a bit of a vacation myself, I’m driving to see my parents and I’m already 5 hours out of town. By the time I got back to your place, it would be too late to send your keys…”

“Ah, nuts! Oh well. Talk to you later.” Bob hung up.

Carol eased back on to her couch, grabbed her copy of People, and looked out her side window at Bob’s house. “Am I being too cruel?,” she wondered. “Nah…”

An hour later, Bob had his bags and his rental car, and he hit the road to his vacation home. There had to be a way in. He would break a window if he had to… but he knew all of the first floor windows were boarded up for the winter.

Four hours later he arrived at his front door. He knew right away that all was lost. There was no way in, not without a crowbar, and by the time he got back to town to buy a crowbar, all of the shops would be closed. Hell, the Crowbar Sellers Union was probably on strike, too.

He noticed a light on in the home down the street, about a quarter of a mile. It was dusk, the sky was beautiful. The blueberry bushes along the road beckoned with ripe fruit. Bumble bees buzzed peacefully from flower to flower. He could see the ocean a mile off, could smell it, could hear it. DAMN KEYS!

He was about to knock at the neighbor’s door when it opened. His jaw dropped. It was Alice. Her hair was wet, and she was wearing only a towel. He stole a glance at her naked ankles.

“Well, what a surprise,” said Alice, sounding not surprised at all. Her cabin looked warm, and candles here and there flickered a bit in the breeze from the open door.

Bob was shocked, but still upset over his keys. “ALICE! Thank God! I need your help! I don’t have my keys! To my cabin! I can’t have them shipped! I’m screwed!”

Alice looked a bit hurt, as if she expected Bob to give her a handful of freshly picked wildflowers and a bottle of cold Champagne. Instead she saw a raging lunatic who couldn’t even remember to bring his own keys.

“Um, I’m sorry to hear that Bob. Good luck.” The door closed. Bob was left out in the dark.

As he sat on the side of the dirt road weeping, he noticed a car pull up to the cabin on the other side of Alice’s place. The California license plate read simply “DAVE.” Bob watched who he presumed was Dave get out of his Lexus, walk briskly up to the door, turn a key in the lock, and go inside. Bob shook his fists at the sky… “WHY??????????”

Bob walked slowly back to his car. As he walked with his head hanging low, he decided he would just go back to the airport and wait for the next flight home in the morning. There would always be next year. He decided he didn’t want a vacation anyway. As he drove down the dirt road, his headlights flashed on Dave. Dave was walking toward Alice’s cabin, a bouquet of wildflowers in one hand, and a bottle of Champagne in the other.

]]> http://www.red-mercury.com/blog/software-development-is-sometimes-sorta-like/software-development-is-sometimes-sorta-like-driving-to-the-airport/feed/ http://www.red-mercury.com/blog/software-development-is-sometimes-sorta-like/software-development-is-sometimes-sorta-like-building-a-house/ http://www.red-mercury.com/blog/software-development-is-sometimes-sorta-like/software-development-is-sometimes-sorta-like-building-a-house/#comments Thu, 28 Sep 2006 16:50:31 +0000 scott http://www.red-mercury.com/blog/software-development-is-sometimes-sorta-like/software-development-is-sometimes-sorta-like-building-a-house/ This is the first in a series. Software development is software development, it has its own unique challenges and it is not exactly like building a house or any of the other things that I will be comparing it to. But there are analogies that can be drawn, and I think they can be constructive, or at the very least, entertaining. So here we go.

Software Development is Sometimes Sorta Like Building a House

• If you put the drywall up before the plumbing is in, you will tear down the drywall to put the plumbing in.
• It is hard to envision what the final house will be like until the drywall is up.
• You may be eager to put up the drywall to feel what the final product will be like.
• This desire will be strongest if this is your first house.
• If this is your 500th house, you will fire anybody who puts up drywall before the plumbing is in.
• Maybe you could attach the drywall temporarily with easy-remove attachments. The drywall will get damaged, and the easy-to-remove attachments will be in the final house. The ultimate owner of the house will not be happy when his ceiling falls on his head while he is reading the Sunday paper.
• Every house is unique. Every house is the same.
• If you are in charge of building a house, there have been milliions built just like it.
• The house you are building is unique. It may even be a one of a kind design envisioned by a famous architecture and design firm.
• No matter how unique your house is, there are expectations for it
• The house will not fall down
• All doors will close tightly and reliably
• The electrical system will be sufficient and meet local standards and codes
• The house will not be in danger of catching fire during normal use
• The house will be visually appealing
• The front door will be on the front
• The garage door will open automatically
• If you keep enumerating the requirements for your home, you will find that this construction project is almost exactly like every home ever built.
• There are companies that build homes like this every day.
• Some of them are wildly successful, profitable, and have highly satisfied customers.
• Some of them go bankrupt, get sued, and have dissatisfied customers.
• The successful companies do things differently than the unsuccessful companies.
• What the successful companies do is usually no secret.
• It is sometimes hard to believe that what they do leads to their success.
• It is even harder to have the discipline to do these things that lead to repeatable success, unless you’ve done it before and have experienced the rewards.
• The people who run bankrupt companies with dissatisfied customers that sue will tell you that the discipline followed by the successful companies is a load of garbage.
• There may be successful companies that tell you what their successful competitors do is a wasteful load of garbage. Go work for these naysayers and you will find that they are in fact quite disciplined, follow many of the same techniques, but call those techniques by different names or have integrated them so deeply into their culture that they don’t see them as methodology at all.
• Building a house is like building a house
• Building a house is not like building a skyscraper
• Building a house is not like building a back yard shed
• Building a house is not like building a diorama for your 4th grade daughter
• You need blueprints to build a house
• You can build a diorama for your 4th grade daughter in an afternoon with just an idea in your head
• You can build a back yard shed in two afternoons if you follow very detailed instructions very carefully, if all of the pre-fabricated parts have been drilled exactly right, if the bolts to attach them together are all in the package, and if there are no mistakes in the instructions. If there are problems, you face at least a week delay (to wait for your next weekend of free time) and possibly 6 to 8 weeks delay (to wait for the manufacturer to ship the correct parts).
• To build a skyscraper, you will need about 10 years. Of course you can’t build a skyscraper without blueprints. But you need much more.
• It will take 2-3 years minimum to secure a location desireable enough for your skyscraper
• It will take another year or two to work through the political process to allow your skyscraper to be built there
• You may never secure the site, or the permission to build there, so your project might not ever get off the ground
• You can do some things in parallel
• The design of the building will take 2-3 years to finalize
• The architecture and detailed blueprints will take a year or two or three
• After 3 to 10 years you can start construction
• Construction will take 2-3 years if all goes well
• Construction will take much longer, or never complete, or be completed with major, expensive-to-fix errors if any of the up-front planning was insufficient.
• Somebody very experienced needs to be in charge of this phase. One supply, labor, design, or implementation glitch can delay the entire project for months.
• If this is your first house, and you realize it is a skyscraper, leave the building and go find a house to work on. You don’t want this skyscraper to fall over because of you. More likely it would just never get built, in which case you will have wasted 5 years of your life on nothing.
• If anything major goes wrong, the investors will lose all of their profit and some or all of their capital, even if the construction finally completes and the building appears to be a success.
• You can build a house in a year if you know what you’re doing
• If you have built this kind of house before, you will have detailed blueprints with corrections
• You will have reliable suppliers for the components you need
• You will have a trusted labor force that can follow the blueprints exactly and on time
• You will know your customers’ rational and irrational expectations and meet or exceed those expectations
• If you’ve been at this a while, you can build 10 or 100 houses in a year, because you will have many corrected blueprints, many reliable suppliers, and a large trusted labor force, and every house is almost exactly the same as long as you are willing to say “no” to the customer when they ask for the front door to be on the roof.
• You can’t build a house in a year without help if you’ve never done it before
• You may find yourself tearing the drywall down to put in plumbing
• Then tearing it down again to put in an electrical system
• Then putting in space heaters and window-unit air conditioners because you forgot to put in a centrail air system and you don’t want to tear down the drywall again
• Then calling in the professionals when the house sinks halfway into mud and starts itself on fire

If you are building a house without blueprints, if this is the first house you are building and you are expected to build it in one year and exceed your customer’s expectations and you are just starting to recruit workers and find suppliers, if your customer requests the front door to be on the roof, if it seems like you are expected to build a skyscraper with the budget of a house, if you are expected to assemble a shed in a weekend with no pre-fab parts, no bolts, and no foolproof step-by-step instructions… you are in trouble, and it’s obvious that you are in trouble and that you will not be able to achieve the goal as expected. You would say “no, I can’t, and you’re an idiot for asking” if presented with any of these challenges.

Software development is sometimes sorta like building a house.

The process starts with a list of bullet points. This is similar to a “requirements list” but less formal, you can do it as bullets in Word or even individual lines of text in a simple text editor. Each bullet point should be 75 characters or less (a few characters will be added later as you will see).

No matter where you are in the project (early conceptual stages, initial planning stages, or even if development is fully ramped up and in progress) sit down with a blank list and start making single line entries, 75 characters or less, that describe the things that need to happen for the project to be completed.

Don’t limit it to product features, include mundane things like “buy new computer to replace the one that exploded”, and “hire new developers and get cubicles allocated”. As you can see, some of the items will be very specific (”buy new computer”) and some will be very broad (”hire new developers”). Try to keep each line focused on one thing - split “hire new developers and get cubicles allocated” into two items, “hire new developers” and “get cubicles allocated”.

After a while you’ll run out of things to put on the bullet list, or the list will get so long that you’ll begin to shake and cry uncontrollably. When either of these things happen, it’s time to stop.

Now, count the number of bullets. This is the number of days you’ll need until the next phase. When translating this to calendar days, make sure you leave out weekends. You will need the weekends anyway, but don’t include them in this number.

Let’s say you had 30 bullet points. This translates to exactly six weeks. For the next six weeks, work on the items on the bullet point list. When you complete one, put a “+” to the left of it. If you decide not to do one, put a “-” one next to it. For all of the other bullet points, leave them as they are - those are the ones that need to be worked on next. Note that even though we allocated one day per bullet point, this does not mean that each bullet point can or will be done in one day. The actual amount of time it takes to complete each bullet point depends entirely on the resources you have at your disposal. Do the best you can. Some bullet points won’t get done. Don’t worry, we’ve got that covered, as we’ll see in a bit.

Often, you will come across a bullet point that is really more than one task. Let’s say you had a bullet point that originally said “Implement user interface”. When you come upon that one, you won’t be able to just do it as-is. You’ll want to expand it out into some more bullet points. For example, “Implement user interface” might expand out to 50 more bullet points, itemizing the widgets that are needed, the various screens and menus that need to be designed, the visual styles that need to be determined, etc.

When you reach the end of the first phase, you will still have your bullet point list, but it will have been transformed. You need to clean it up at the end of each phase. Go through the list and remove all of the lines with “+” or “-” at the beginning of the line. Those have been addressed and you don’t need to ever think about them again until testing time. There will be quite a few bullet points from the start of Phase 1 that have not yet been addressed or thought about at all. That is ok, leave those on the list.

After you have cleaned out all of the addressed items, scan through the list including the newly expanded bullet points. No doubt some things will occur to you that weren’t on the first bullet point list, and are still not there. Add those to the list. Also, you may see some bullet points that can now be expanded out to more detail. Go ahead and expand those out now. As with Phase 1, keep doing this until you run out of ideas to add to the list, or until you begin to shake and cry uncontrollably.

Now it is time to start Phase 2. Count the number of bullet points that are on your cleaned up list and allocate one day per bullet point again. Let’s say you have 120 items on your list now. That is 24 weeks. Start into the new bullet point list exactly as you did with Phase 1. When an item is completed, mark it with a “+”. If you decide an item is not needed or will not be done, mark it with a “-”. If you find a bullet point that needs added detail, expand it into a new list of more detailed bullet points.

As you can probably tell, Phase 2 follows the exact same methodology as Phase 1. In fact, every phase follows this same approach. That is the beauty of this technique. Once you know the basic procedure, you know the whole methodology, and no special software is needed. In fact, this entire methodology can be implemented with a pad of paper and a pencil.

Eventually, the product will need to ship. The forces that determine the ship date of a product are typically beyond anybody’s control, no matter how powerful any individual person may be. You will typically have some early warning of when this date is, and you will need to allocate some time to fix any problems that exist with the software. Do the best you can. No doubt you’ll still be trying to implement items off your bullet point list during this time, so technically any testing you do will be invalid. Plan for some sort of automatic online updating system for the software after it ships.

And that’s about it! Unfortunately, this method doesn’t tell you the full schedule of the project on day 1. All you can say on Day 1 is when Phase 2 will start, and due to the unpretictability of each bullet point (some happen quickly, but most take longer than expected, and almost all of them expand into more bullet points), knowing when Phase 2 starts isn’t really much help. But it’s the best you can do, and it’s about as good as the best professionals can do. In fact, it’s better than a schedule that sets very specific dates in stone, because those kinds of schedules just make everyone shake and cry uncontrollably when they realize there is no way those dates can be met. With the bullet point approach, the shake-and-cry periods are planned out ahead of time and are expected, and so they are easier to deal with.

Am I serious? I started writing this partially to be funny, and partially to put down in words the actual approach that I use for software development. There are projects that I can schedule very well, but those projects are basically repeats of other projects I have done, using well understood technology and doing the exact same tasks as a very similar project that has already been successfully completed.

As the world of software matures, the number of repeat projects increases - this is the software industry’s version of building strip malls. If you’ve been doing it for a while, you know how long it takes to lay the grade, roll the pavement, and slap together the strip of shops. But often in software, there is some innovation required, so the project is more like building a spaceship that can also drive at highway speeds and travel waterways like a motorboat, with an outer skin that can display high resolution full color images and a life support system that includes the capability to manufacture cotton candy from sugar synthesized from molecules formed by shooting lasers at particles being accelerated in a 5-mile radius loop. These types of projects tend to be difficult to schedule, but the bullet-point list will get you as close as you can get to something reliable.

Oh, and we’re gonna need a killer stereo system in that thing. Add it to the list.

It’s dark, it’s quiet, the black Windows XP loading screen glows dimly on my screen…

“XP” … it sounds so natural now. I run “XP”. Most of my machines have “XP”, a few have “2000″. These are brands, everybody knows them. But why XP?

It rolls off the tongue easily enough, it sounds professional, but as trademarks go, it’s not so great. A two-letter non-word isn’t the most defensible trademark. That doesn’t matter so much, obviously the XP brand has power…

Why those letters?

The loading bar zooms left to right, left to right, never giving off a hint of how much is left to go, just letting me know it is still working on it…

Did it really stand for “Windows Experience?” The 2001 Microsoft press release claims this is so, but that sounds pretty weak, way too meaningless… almost like a convenient answer to satiate those who are curious enough to ask…

What else… around 2001, “Extreme Programming” had quite a buzz going in the programming world, and that was always referred to as “XP”. Perhaps some engineers at Microsoft liked what XP stood for and thought that it represented what Windows XP had under the hood?

But that doesn’t make much sense either. Extreme Programming was something happening outside of Microsoft, and it seems unlikely that they would take on the nickname of someone else’s methodology as their product name…

The loading bar continues.. left to right… left to right… “I’m loading… don’t you worry about how much is left to go, I’m on it…” XP… XP… what could it be…

Or, as on the loading screen, lowercase xp… xp… p is the greek letter “rho” … hm… oh yeah and of course, x is chi… “c.r.” does that stand for anything? hmm… chi, rho… chi, rho…

CAIRO!!!!

Of course, Cairo!! A famous Microsoft codename originally used for Windows NT 4.0, but also used to describe a set of technologies that didn’t all make it into 4.0. Cairo was envisioned as the ultimate operating system, and an ambitious feature set was announced, much of which took longer than expected to complete. The remaining technologies that were originally targeted for Cairo were eventually added bit by bit to all of the versions of windows between Windows NT 4.0 and Windows XP… xp being the real Cairo.

All of those technologies are now in the ultimate operating system, XP… except, of course, for the object-based file system WinFS, originally annoucned for Cairo, and recently cancelled yet again for Vista. But that’s ok, I prefer plain old file systems that store and retrieve hunks of bytes, that has always worked well for me. Perhaps whoever chose the name XP decided that Windows XP was finally close enough to the Cairo vision to be called Chi-rho.

Ah, my machine has booted finally… my Windows chi-rho, cairo, xp machine… now, time to get to work. But, on second thought, I guess I have time to type up a blog entry before really digging in…

It happened to me again today, and I thought I’d put a little post here about it to perhaps save somebody some frustration.

Today’s perl Out of memory error happened while I was upgrading SpamAssassin on the Red Mercury mail server. You wouldn’t think that something as simple as upgrading a few perl modules would be enough to run out of memory, particularly since this machine had nothing else running on it at the time, and the machine has a gig or so of RAM backed by a couple gigs of swap space.

The answer, in my case anyway, always turns out to be “ulimit”. What is “ulimit”? If you live inside a unix world, you probably know all about ulimit. For mere mortals like myself, it’s something that I only care about when it prevents me from doing what I need to do.

“ulimit” is short for “user limit” - it limits the amount of resources that a particular user can consume. This is of course very useful as it prevents any particular user from hogging all of the memory or CPU time on a system. If you run a process from the command line that goes absolutely crazy, ulimit can be the sanity check that keeps the rest of the box humming along nicely.

“ulimit” is usually a built-in shell command, so it varies from shell to shell. I happen to be using zshell on an OpenBSD box, but even if what I describe here doesn’t match exactly what you have on your machine, it should get you headed in the right direction (in fact, just saying “ulimit” may be enough to get you headed in the right direction). It could be that the stock ulimit settings on OpenBSD are smaller than on other default installs, but when I first had this problem, I saw a LOT of people asking this question.
So first, I type “ulimit” to see what’s up:

[ultrabox]: ulimit
unlimited

Ok no problem, right? But as it turns out, with no command line arguments, my ulimit will default to showing me my limit on file size, and that is unlimited. Let’s find out what all of the limits are:

[ultrabox]: ulimit -a
cpu time (seconds) ulimited
file size (blocks) unlimited
data seg size (kbytes) 65536
stack size (kbytes) 4096
resident set size (kbytes) 1446704
…(and some others)…

Now we’re seeing some interesting things. The stack size is 4MB. That might be small. The “resident set size” is set to almost the size of the RAM in the machine (well over a gigabyte) and it is maxed out and can’t go any higher. The “data seg size” is 64MB (65536 kbytes above) - that should be enough for anything, right?

It has been my experience that, when working with perl, a 64MB data seg size is too small. I have to make it larger before running most of my reasonably-complex data processing scripts. And, today, I had to make it larger before doing something simple like upgrading SpamAssassin via CPAN. Even though I ran the upgrade with “sudo”, I still had to raise the data seg size ulimit in my shell, since it was limiting the data seg size for everything run from that shell.

So, Greg Graffin, I HAVE THE ANSWER!!!

Simply boost the size of your data seg and make that stupid perl Out of Memory! error go away:

[ultrabox]: ulimit -d 200000

[ultrabox]: perl irsProcessAllUSTaxReturns.pl

Processing… 130,728,360 tax returns… Finished!
[ultrabox]:

It worked!

For scripts that I run often, or from a cron job, I make a little shell script that first sets the “ulimit -d 200000″ and then runs the script. Other possibilities would be to set the needed ulimit settings in a login script, or to bump up the default system-wide (though the method for doing this system-wide varies by system and you might not have the required permissions to do it anyway).

Please note the number of zeros there in the ulimit setting - five zeros - this sets the ulimit to close to 200MB up from the 64MB it was set at previously. You may need more. You may need less! On the less side it doesn’t matter so much - this is just a top limit, it doesn’t mean everything will use 200MB of data segment space, it just allows things to use that much space.

Now, as for the bigger question of WHY perl needs over 64MB of data segment space to do something like install spamassassin… well, in the words of a Gary Sommer, guy I used to work with, “that’s beyond my attention span.”

I don’t like my opinion of City Lights. Part of me says I should cut the movie some slack, because it’s a silent film. Or maybe I should look at in context, with the other films of that era, and judge it against those works. It could be that I’m just ruined by modern attention-grabbing techniques and am incapable of slowing down to appreciate a fine work from a simpler era. Maybe I’m getting old, and my mind is closing.

Or maybe I’m just more honest with myself these days.

This film is a cautionary lesson in technological progress. It was made in 1931, four years after the technology for movies with audio tracks (”talkies”) became available. While watching City Lights, it is painfully obvious that sound is missing (there is, of course, a musical score that punctuates some of the events in the film, but watching the actors’ lips move with no accompanying voice is creepy). When the original silent films were made, it was simply a technological issue. It wasn’t feasible to record and play back the audio in sync with the film, so the filmmakers did the best they could, and made silent films. They didn’t let the lack of sound stop them.

But Chaplin made a lot of silent films. So many, in fact, that he stopped seeing the lack of sound as a problem, and started seeing it as a necessity. Once the technological limitation was lifted, he could not embrace it, and instead saw it as a threat to his way of working, and as a threat to every film he had ever made.

This is a problem that we all face throughout our lives. When we’re young and haven’t yet learned the limits of ourselves or the world around us, we look at things and wonder why they aren’t the way we would like them to be. As we get older, we run in to those limits, and we learn that things are the way they are for a reason. We accept those limitations, and we adapt to them. We may make great advances by pushing hard against these limitations.

Then one day, we wake up, and those limitations are gone. Every great push against those limitations now looks ridiculous. Where there once was a brick wall, there is now a wide open field that stretches for as far as the eye can see. We look down at the patch of ground where we once were proud to have struggled to move that brick wall an inch, and see that someone else has effortlessly moved the wall hundreds of miles while we were resting. Our once-acclaimed lifelong achievements have been reduced to a silly, historical curiosity.

When this happens, when the brick wall disappears, we look to the horizon and we see these innovators disappearing into the mist, charging ahead into lands that to us were once a dream, but that we now consider to be forbidden. Those charging ahead look to us like trespassers, and we try to convince ourselves that they are foolish, that they will fall off the edge of the flat earth, leaving us to move ahead inch by inch again as we have always done.

Soon, however, the reports come back that these adventurers who have removed our wall are continuing to make progress, and are in fact accelerating, and we think back to our younger years and realize that these innovators are living our dream. We look down again at our life’s work in anguish. What is the point of taking another step? One step would take us farther than we ever imagined, but as the rest of the world accelerates thousands of miles away from us, we relegate ourselves to being washed up, retired, finished.

Of course, these days, changes like this happen all the time. They happen quickly, but it takes us all a while to see their importance. By the time we look to the horizon, it feels like we have been left behind. Fortunately, the new horizon is never as far away as it looks. Once we take that first step, we find ourselves on the tail of those innovators, and with our experience and insight, we find that keeping pace and pulling ahead again is really not so hard. Soon we look back over our shoulders and see many others standing where that wall used to be, looking to us, and cheering us on.

And the moment we step past that wall and start charging toward the horizon, it’s fun all over again.

Proponents of the law argue that any job should pay enough to cover the bare necessities of life, such as rent and food, and that large retailers can afford to pay more. Opponents argue that an artificial floor on the hourly wage will decrease the number of jobs, and that large retailers will choose to locate outside of the city.

Who is right? I think a more interesting question is, how do you determine that $9.25 is the perfect hourly wage to put into the law? Among the supporters of the law, this precise amount must have been the topic of much debate.

There were no doubt the scentists who calculated a rent payment, plus cost of food and other necessities for survival, and divided it by an average number of expected work hours to come up with an acceptable pay rate. Perhaps there were others who argued for just a dollar more, pointing to the huge revenues of these retailers as evidence of their ability to pay it. No doubt comparisons were made to jobs where the rates are heavily influenced by unions.

It is undeniably difficult to decide the correct hourly rate to set into law. But I would like to help, in case this debate comes up again in another city.

I have come up with a diabolical plan to enforce the perfect hourly rate, almost down to the penny. It is an approach that the supporters of the big-box wage law would like, because it is designed to extract the most possible jobs at the highest possible hourly rate from those huge, wealthy corporations.

It works like this. I start by pretending to be hands-off, letting the retailer come into town, and letting them set their own hourly rate for all of the jobs they would like to offer. I make sure that they tell the potential employees in the area how much they intend to pay, and that they are crystal clear about their benefits package, or lack thereof. Luckily, they will do this part of the job for me, since they have big human resource departments who are trained to do this kind of thing.

Now, I have them right where I want them. If they have set the compensation way too low, nobody will even apply for the jobs, and they will be FORCED to raise their hourly rates. Of course, they are cunning, and would never make this mistake.

They might set the hourly rate ALMOST high enough but not quite, such that they get enough people hired to do the work. But my plan works here, too - if the hourly rate is just not quite high enough, these new employees will stop showing up to work, or they’ll quit, or they might even go get a job at a higher paying big-box competitor down the street. This causes turnover, which is expensive, and we all know that the big-box retailers hate spending money, so they will be forced to raise their hourly rate in this case as well.

Now, this is where my plan shines. Once they have raised their hourly rate high enough to not only attract employees but keep them coming day after day to do their jobs, I have succeeded in milking these greedy corporations for as much as I possibly can. The highest number of people will be employed, and they will each be getting paid the perfect hourly rate, with the perfect amount of benefits required to keep them each coming in to work at the big-box retailer every day.

With this plan, I’ve got the retailers locked in forever - if they ever slip up and lower their pay too much, the employees will stop showing up to work, or start working somewhere else, forcing the big-box to up the pay rate again.

And the best part? My plan can’t be overturned for being unconstitutional.

I would be able to use Subversion to compare these local versions, so “svn diff” would show me the difference between the latest locally committed version and my working copy. I could specify a specific local revision to “svn diff” and get the differences with that revision, all the way back to the head revision that I started with when I disconnected in the first place. For example, “svn diff myfile.c -r541.23″ would compare my working copy of myfile.c with the 24th locally committed version of myfile.c. Existing commands would still work normally, for example “svn diff myfile.c -r541″ would compare myfile.c with version 541, the latest version I got from the actual repository. However, “svn diff myfile.c” could automatically compare with my latest locally committed revision, or alternatively, a “–local” flag could be added in cases like this to indicate when we want to pay attention to local commits.

Even if I’m not disconnected from the repository, these local commits allow me to work incrementally and implement huge, earth-shattering changes, one small step at a time, without bothering anybody else who is using the main repository until I’m done. While working on these large scale changes, I still get the advantage of versioning in small increments by doing local commits.

Once I am ready to commit these local changes to the repository, I would do a “svn commit” like normal. Subversion would automatically do a sequence of commits, starting from my first local revision and continuing through each locally committed revision up to and including my local working copy, applying the commit comments that I had supplied with each local commit. My local incremental changes and comments would then be commited to the main repository and be given real revision numbers by the repository - the local “541.23″ version numbers would go away, but the sequence of revisions would still be preserved. After the commit to the repository, the versions that had been locally committed now appear as individual revisions in the central repository, as if I had been committing directly to the repository all along. All of these commits would be atomic. Multiple local commits on multiple files could be committed in one atomic operation.

I have only been heavily using Subversion for a short time, but I believe adding “local commit” would fit very nicely into the way Subversion currently works. Whether it would be technically challenging to add it, I don’t know, but that is less important than the way the idea of local commits would interact with today’s Subversion. I think local commits could be added in a way that would not change the way Subversion is used at all, and it may even be possible to add it in a way that works well with existing tools that interact with Subversion.

With “local commit” support, Subversion would go a long way towards solving the types of problems that projects like Bazaar-NG are trying to solve. There are a handful of “distributed” version control solutions in the works, including Bazaar-NG, that are trying to create solutions where there is no central repository. Allowing full-featured disconnected work is a major advantage of such systems. However, the advantage of having a central repository can’t be ignored - it defines a central place for all revisions to be integrated. With distributed systems, it is easy to imagine small clusters of developers in sync with each other, but wildly out of sync from cluster to cluster. Subversion plus local commit allows full incremental commits while disconnected, and loses nothing when the actual commit to the central repository is done.

So… will it happen? I suppose it’s my itch, so in the tradition of open source, I should scratch it. But if I’m lucky, someone hotshot with lots of Subversion development experience will have the same itch after reading this and implement it in 10 minutes… if you do, let me know