Steve Hannah: This week

I recently ran into a problem with a customer’s Java environment where some of the classes that PDF OCR X requires were being overridden by out-of-date classes. The error wasn’t exactly intuitive:


java.lang.AbstractMethodError: java.lang.AbstractMethodError: javax.xml.parsers.DocumentBuilderFactory.setFeature(Ljava/lang/String;Z)V


This was happening when I was trying to parse an XML document, and it only happened on this one customer’s machine. The problem was that he had an old version of Xerces installed in one of his Java extensions directory. He had probably installed it there years ago (or a program did it for him) as a shortcut to make sure that Xerces was always available without having to add it to the classpath for each project. Whatever the reason, it was causing problems for me in present day.

I actually include my own copy of Xerces as part of the application and it is included in the classpath. So shouldn’t that override any jars in the system extensions directory?

Apparently, No.

Java will look first in the bootstrap files, then in the extensions directory for classes. And finally, it will look in your classpath only if it can’t find the class in one of the first two places.

This means that it is a really bad idea to add libraries to the Java extensions directories unless you are willing to keep them up to date. You may have problems down the road, and they might not be easy to solve.

This reminds me of a current discussion on the Mac Java development mailing list regarding the pros and cons of bundling your own Java runtime when you distribute applications. If you do decide to rely on the system’s runtime, you are at the mercy of the user’s environment. They can break your application in a thousand different ways by installing old, broken, and dirty libraries into their extensions directories.

Best practice: If Oracle or Apple doesn’t put something in the extensions directory, then you shouldn’t either. Period.

I have been battling with some performance issues on one of my Xataface applications of late and I think I just found the cause of periodic slow-downs:

SHOW TABLE STATUS

The output of this command includes useful information about all of the tables in the database including such things as creation time, update time, average row length, and number of rows. In Xataface I primarily use this command to:

1. Find the modification time of tables so that I can perform smarter caching operations.
2. Determine if a view exists.

When working exclusively on MyISAM tables, this command is very fast as all of the information returned is cached all the time. However, when we start to throw InnoDB tables (and possibly views .. haven’t looked into it yet) into the mix this command becomes quite slow because much of the data returned needs to be calculated (e.g. the number of rows). I was facing an issue where this command could take upwards of 10 seconds to return when the application hasn’t been used in a while. It would also periodically hang even when the application was in frequent use. Presumably this is because MySQL does *some* caching of the values in this command, but the cache doesn’t last long.

In addition, InnoDB doesn’t keep track of modification times so despite the fact that it is performing calculations in this command, it still returns NULL for table update times. Which renders the function altogether useless for InnoDB tables.

Xataface has long had a back-up strategy for keeping modification times in InnoDB tables. It keeps its own table of modification times. This table is updated whenever a record is updated from within Xataface. It doesn’t work for updates performed outside of the application. In most cases this is good enough. Even with this back-up solution, the primary method of retrieving table modification times was still the “SHOW TABLE STATUS” mysql command.

Solution #1: Use the Information Schema

My first attempt to rectify the involved a direct query of the INFORMATION_SCHEMA. You can obtain the modification times of all tables in the database with the following query:

select TABLE_NAME as Name, UPDATE_TIME as Update_time from information_schema.tables where TABLE_SCHEMA='my_database_name'


Unfortunately, it turns out that this query is also quite slow (though a bit faster than show table status). My initial tests showed that on a database with about 60 tables it would take about 0.2 seconds to return. Far to slow for an operation that doesn’t contribute directly to the building of the page. What I need is something that returns in less than 0.01 seconds so that it is effectively negligible.

Solution #2: Use SHOW TABLES

SHOW TABLES simply returns a list of the tables in the current database. It doesn’t include any stats or information about those tables other than the table names. It is also very fast (generally returns in 0.00 seconds …. i.e. too small to matter). This is enough information to build my own modification times or check for the existence of a table/view.

References:

- How can i tell a mysql table was last updated (Stack Overflow) - Includes an example of using the INFORMATION_SCHEMA to get the update time for a table.
- About slow SHOW TABLE STATUS - An issue tracker thread for another project that is talking about the slow performance of SHOW TABLE STATUS when used on InnoDB tables.
- When Does InnoDB Update Table Statistics? (And When It Can Bite) - Talks about InnoDB’s statistics updating. It also mentions the innodb_stats_auto_update flag introduced in MySQL 5.1.17 that can be used to turn off the auto stats calculations for innodb tables.

In my previous post, I described a situation where my wife was overcharged for transferring from Fido to Rogers in 2009. I spoke to both Rogers and Fido customer support at multiple levels and was told that because the overcharge was too long ago they were unwilling to refund it. I filed a complaint with the Rogers Presidents office and was told that they would not refund the money because it was Fido who had originally charged it and it was too long ago. Finally I filed a complaint with the CCTS but their investigation found that it was outside of their mandate because the overcharge was too long ago.

I cancelled our Rogers accounts and moved to a different provider, swearing to never deal with Rogers or any of their companies again.

UPDATE

Today we received a refund cheque from Rogers for $560. There was no explanation of the refund with the cheque, but I phoned customer support and they confirmed that it was a refund for the 2009 overcharge.

I am pleased that in the end the company did the right thing. It is unfortunate that they took so long to come to this conclusion, and in the mean time I was forced to cancel my account. However, with this gesture, they have redeemed themselves to the extent that I am willing to lift my personal ban on dealing with Rogers in the future. The next time I’m looking for services in which Rogers is one of the providers, I will be willing to consider becoming their client again.

My wife and I have recently been forced to cancel our cell phone accounts with Rogers and we will never be returning to either Fido or Rogers for any service in the future. Over the past 5 years we have spent over $12,000 for our cell phone and data services and it is likely that we would be spending more than that amount in the next 5 years. I also have a growing business that will be requiring cell and data services in the coming years. Rogers and Fido will be receiving none of this business.

Why We Are Cancelling Our Service

In September 2009 my wife and I were both using Fido for our Cell services. She was a little over 1 year into a 3 year contract and I was not under a contract. I was looking to upgrade to a smart phone so that I could receive email on my phone so I went into the local Wireless Wave to learn about my options. The salesman informed me that he could save us money by switching to Rogers from Fido. He told me that because Fido was owned by Rogers, they had a migration program that meant that we wouldn’t have to pay a full penalty breaking my wife’s Fido contract. The penalty, he said, would be only $100.

Based on these numbers, I decided to switch to Rogers per the salesman’s advice. Unfortunately, the final Fido bill went to my wife and it was set up for automatic payment on her credit card. On this final bill the cancellation fee charged to her was $500. That is $400 more than the promised price. She thought this was high but she wasn’t aware of what the sales man had told me (about the $100 fee) so we didn’t pursue it.

Fast forward to January 2011.

My wife mentions how expensive switching from Fido to Rogers was and I am shocked to find out that we had been charged $500. I was certain that it was supposed to have been $100. There must be a mistake.

After phoning Fido customer service they verified that we had been charged $500 for cancellation and no refund was ever made. They also confirmed that they do have a migration program and that she should have qualified for the $100 cancellation fee.

However …

Since the mistake was made more than 90 days ago, they were not willing to correct the mistake in any way. I spoke with 4 separate people at Fido. They all gave me the same line. They suggested that since we no longer had an account with them, there was nothing they could do and that we should check with Rogers to see if they would help us out - since we were still customers of Rogers.

Unfortunately Rogers follows the same play book as Fido. I spoke with 3 people at Rogers customer service. They all quoted me the same thing: Any billing mistakes are assumed to be accepted by the client if they don’t object within 90 days.

Fair enough. We could have showed more diligence in monitoring our bills. We failed in this respect. However, we are still operating under the same 3 year contract for which the overcharge was made. The mistake was theirs - even if I made the mistake of not “catching” their mistake, I expect them to rectify it.

The Result …

1. I cancelled both of our services with Rogers and paid the penalty for the remainder of our contract. The value of this portion of the contract would have been approximately $2000. Instead they will receive a mere $450 penalty payment.

2. We signed with Koodo who did not require us to sign any contract. As it turns out Koodo will save us approximately $100 per month to get even more service than we were receiving through Rogers. (Please be advised that we never even would have looked for an alternative option had we not been mistreated by Rogers).

3. We will never again do any business of any kind with Rogers or any of its companies, including Fido. None of my businesses will ever again to any business with Rogers or any of its companies, including Fido.

The Balance Sheet ….

1. Fido (owned by Rogers) is up $400 for the money that they overcharged us initially.
2. Rogers is up $450 for the cancellation fees we paid to get out of our contracts.
3. Rogers is down $2000 over the next 9 months from the lost revenue they would have received had they rectified the problem.
4. Rogers is down at least $2500 per year in perpetuity from the forgone revenue from my patronage due to their failure to rectify this problem.

Forecast:

Over 9 months: Rogers is down $1150
Over 21 months: Rogers is down $3650
Over 33 months: Rogers is down $6150
…

Seeing how your arrogant policies hit your balance sheet, perhaps you will reconsider your ways for the future.

Best regards

Steve Hannah

UPDATE:

Rogers ended up refunding the overcharge. Read about it here.

A Bug Hunter’s Diary is written as a journal of a “Bug Hunter”. It takes us through seven bugs that the author (Tobias Klein) has discovered in the past five years, giving the reader a glimpse of his process and an awareness of how easily bugs can creep into even the most carefully written software.

All of the bugs discussed in this book are related to memory errors that lead to a proofs of exploit that involve overwriting the program counter (and thus hijack subsequent program execution with arbitrary code). This narrow focus sets the stage for a book that is very heavy on Assembly code, debuggers, and a few easily compromisable C functions.

Some of the bugs involve open source software so we have source code to step through, but others provide nothing more than the finished binary to step through with a debugger and a disassembly of the machine code. For someone who doesn’t do much C programming anymore and hasn’t done Assembly in over 10 years, such as myself, the journey can be quite daunting. The guide on this journey stops occasionally to explain his intuition for certain decisions, but we’re largely left to just follow him along without really knowing how he knew to check for certain things and not others. After following him on 7 missions, however, some patterns do become evident. This repetition enables the reader to feel like more of a partner in the later journeys to contrast his bystander role in the first few.

Being primarily a Java, PHP, and Javascript developer of late, the tools and languages used in this book (Assembly and Debuggers) really stretched my ability to follow along in real-time. It reminded me a little of the first few computer programming books I picked up as a newby when I would be reading pages and pages of code and waiting (hoping) for the eureka moment when all the code would make sense. Like listening to an old song from my past, this feeling of having to follow the master and just have faith that things would make sense if I just stared a little longer, opened a flood of memories of my first days in programming. I’m thankful to the author for that.

If you can think of my train of thought during the reading of this book as a process, then this process had a few different threads running in parallel. In one thread, I was just trying to follow the Bug Hunter and understand the lessons as they were presented. In another thread I was trying to identify analogs in these lessons to my own domain of programming (e.g. web programming, and database programming). There were also a thread that was periodically assessing the author’s style and the the book’s relevance, to various target audiences, and its importance to software.

Thread 1: Just trying to keep up

If I had read this book as a tutorial where I actually do as the author says, it would have taken me a lot longer to read. At a certain point I just settled for getting the gist of what he was doing and focused more mental cycles to Thread 2.

Thread 2: Applying the Lessons to My Domain

The process that Klein uses to track down bugs serves as a forceful reminder of how bugs could slip into my own software if I’m not careful. In trying to exploit memory errors, Klein would focus on the interfaces between user space and kernel space as logically any exploit must pass through one of these bridges at some point. There is an analog in any programming environment. In web development you are looking for the entry points where user input is passed to the application. This book is a reminder that you must always be conscious of where data could possibly have come from before working with it as it is often a combination of small glitches that result in a large exploit down the road.

Thread 3: The Author’s Style

Klein writes much of this book as a sort of diary of his actions. As such it is more like a day in the life of a bug hunter than it is a tutorial on how to become a bug hunter. This style works well for imparting an authentic view of what a bug hunter really does - and as such might serve as inspiration for those who want to follow this craft.

At times (more early on than later) I struggled to grapple with the direction Klein was taking us. Is he just reciting his exploits as a way of bragging, or is he recounting his steps and building up to an eventual purpose. Throughout the second and third bugs I was bogged down a little with thoughts like “this is way too dense in Assembly for the average person”, but these would be countered with rebuttle thoughts like “but this isn’t intended for the average person…”. As I settled in, though, these thoughts were replaced by less critical internal commentary like “wow, it would be really easy to make that mistake” or “that’s really neat!”

By the time I reached the sixth and seventh bugs and I had seen the same types of bugs appear over and over, I had become a true believer. Not only was this book interesting; It was essential! Despite the lip service that is paid to “good programming practice” in computing science programs, I have never been made so acutely aware of some of the simple programming mistakes that can result in dreadful consequences. As I finished the seventh bug, the one clear capstone thought was that this book should be required reading to anyone who wants to write any software. Just as engineers are required to take an ethics course, computer scientists should be *required* to take a “Bug Hunters” course - if for no other reason than to raise their awareness of the nature of bugs.

When I think about the “buggy” code that I have read from new University grads it makes me shiver. Much of this buggy code is in production around the world and quite often the develop has no idea that the bugs are there. This book highlights a few specific functions to handle with great care in C, but every language has its own set of traps that all new programmers fall into. (E.g. in PHP registering global variables, unvalidated input to SQL queries leading to SQL injection, etc…). Computer books very rarely spend much time highlighting these traps, and as a result young programmers regularly fall into them. Unfortunately with software these traps often don’t give immediate feedback. Instead they linger for years before they are exploited - and sometimes to great disaster.

Summary

I meandered a bit in this review, but to summarize. If you can slog through the dense debugger readouts and assembly code and latch onto the core message in this book you will become a better software developer. Even if you don’t learn anything new in this book, it will serve to raise your awareness and prevent a bug.

And one more thing: This book should be required reading for new software developers.

I have just released Survey Builder on Sourceforge. Survey Builder is an application that allows you to design and host surveys using pure HTML. It is the first application built with the new Xataface 2.0 look and feel.

Check out the sourceforge project at https://sourceforge.net/projects/surveybuilder/
Check out the documentation at http://www.fcat.sfu.ca/websurvey/doc_output/html
Check out a sample survey at http://www.fcat.sfu.ca/websurvey/sample

I just went to start up a new sourceforge project for the Survey Builder application I just created. I guess it’s been a while since I created a project because the system is total different than it used to be. Much more powerful it seems. They now give you a blog, issue tracker, and wiki (as well as version control if you want it) which is all that I generally need. A long time ago I decided to move most of my project stuff off of source forge (except for the download hosting) because their facilities were clunky and slow.

There’s still time for me to reform a negative opinion, but for now it all looks good. I may not even set up a separate site for this project at all; just do it all on sourceforge.

I’ve been putting a lot of work into Xataface getting it ready for release. There are still some details to take care of, but I think it’s safe to say that this release will be a drastic improvement over any of the previous releases.

My Dad recently upgraded to Lion only to discover that Logic Pro 8 no longer works on it. When he opened the application it just pops up with message saying that Logic was not compatible with this version of OS X, and promptly quits.

I saw some workarounds on you tube and other sites that involved starting Logic up on the command line by calling the internal binary inside the Logic app package directly - but this “fix” was not clean as it requires the terminal window to stay open and it also allows you to open logic twice or three times by accident and have it running it multiple processes (not a good thing). It also prevented Logic from working nicely with the rest of the OS because you still couldn’t open the app directly the “normal” way.

So I found a fix that will actually allow Logic 8 to work normally on Lion (i.e. you can just open it using the regular application icon and there’s not command line stuff involved).

Steps:

1. Right click (or control-click) the Logic Pro 8 application icon and select “Show Package Contents” from the contextual menu that appears. This will open a folder with the contents of the application package.
2. Open the “Contents” folder.
3. Open the Info.plist file in Text Edit. (You should be able to just double click it). This will open a big XML file with a whole bunch of directives.
4. Look for the text:

<key>LSRequiresCarbon</key>
<true/>

Delete this text.

5. Save the file.

You should now be able to open Logic Pro 8 normally on OS X Lion.

Special Note: That directive says that Logic requires the Carbon libraries to be installed. Carbon is old and it doesn’t look like Logic actually uses it at all - this is just a legacy restriction - I think. That said, it is possible that there are some features that still rely on Carbon to be there and may not work correctly on Lion. I’m sure my dad will let me know if he runs into any problems. So far everything appears to be working properly.

Edit: Apparently updating Lion to 10.7.2 also fixes this issue so you don’t need to perform this workaround.

I just finished reading the new Walter Isaacson biography of Steve Jobs. It is a good read, especially if you have an interest in Apple, Jobs, or computers in general.

This is a story of two Jobs. One is a self-centered jerk of near mythical proportion; the likes of which I’m certain I’ve never known personally. The other is a goal-driven perfectionist obsessed with design and simplicity, … and perfection. The story of this second Jobs is truly inspiring and it is this Jobs that I will choose to commit to memory.

Jobs had a keen eye for design and he would not allow anything less than the best to earn his stamp of approval. He oversaw all aspects of product development, from the design/development of the raw materials (such as the metal used in the Powerbooks or the glass used in the walls of the Apple Store), to the packaging that the finished product was delivered in. If something wasn’t perfect it was “shit” to him - and he certainly wasn’t polite with his criticisms of imperfections. However when a product did become “perfect” he also wasn’t shy about declaring that it was the “greatest thing ever” to the world.

In my own personal work I have been inspired by the story of Steve Jobs to push myself further towards perfection. I am prone to pragmatism or the “that’s good enough” bug that aids my productivity but sacrifices perfection. I doubt I’ve ever produced anything in my life that would stand the test of Steve Jobs’ criticism - and I suspect that most things would have received the prestigious Steve Jobs certification as “shit”. In the past few days I have found myself asking the question “what would Steve say?” when working on different features and designs for my products. Is there any way that this could be simplified? And I have surprised myself by continually making things better; Probably still “shitty” by Jobs standard, but still getting better.

I really think the world would be a better place if we all had some “ass hole” like Steve Jobs looking over our shoulder declaring “it’s shit, get rid of it!” whenever we were about to turn in a sub-standard piece of work. I think the world is better for Steve and his intense passion for perfection. I think the world is worse for his departure.

Rest in peace, Steve Jobs.