Life, Technology and Management

Cheating on interview tests ...

2009-04-06T16:20:00.006+08:00

I never like rejecting resumes or people. I wish I could truly give everyone a chance to prove themselves. Unfortunately, the limited resources of a for-profit organization dictate that that some people get rejected without a thorough assessment of their abilities. We have to use a lot of different reasons to whittle down the huge list of applicants to a number we can actually interview.

Today I rejected 11 people (out of 43 finalists) for cheating stupidly.

We just finished a "programming party" 2 weekends ago. This is a recruitment event where we bring something like 110 students in over the weekend, run them through automated programming tests then hold a small party afterwards. The point is to give students a chance to see where we work and talk to our developers in an informal environment to find out what life is like in Exoweb.

We ran the tests using our older Exobench system, so the programming problems had been used for a while. Therefore I wasn't really that surprised that some of the candidates knew of the problems in advance. What I was surprised at was that so many of them blindly copied answers they obtained elsewhere without thinking it through. If they had, they would have found the errors that made it so easy to disqualify them.

The most glaring example is one of the test problems called Dominator. Given an array of numbers, the problem is to find out if there is a single number that occurs in more than 50% of the array. If it does, return it. It's a simple enough a problem and yet very useful for testing since it has both an easy (exponential complexity) and an ideal (linear complexity) answer. This gives us lots to talk about in the interview.

Apparently someone posted the code for the easy solution, except with one enormously glaring error - the code only searched half of the array, stopping its search at length/2. If you're looking for a number that takes up half the array, you can't evaluate only half the array since you don't know which half of the array this number might be in! This mistake showed up the most and no doubt there were some that honestly made this mistake. However, the cheaters were obvious because there were also other small mistakes that made their code identical.

An even more obvious copy was an ideal solution for another problem. Getting an ideal solution is rare and statistically, we see the ideal solution for this problem once every 400 tests or so. This time, we got 8 out of 110. That's almost 30 times what we would normally expect. However, 7 of the solutions had one glaring error - they forgot to consider an array of all negative numbers. In every single other case, if the programmer was smart enough to come up with the ideal solution, they would always consider the all-negative array boundary case. Somehow we got 7 people who suddenly made the same mistake at once ...

4 out of the candidates made both of these mistakes together (candidates do 4 random problems at a time). The rest made either one or the other. All have been rejected. I have been told that this is standard practice in local universities when a batch of identical exam answers are received.

There are still a few others that have suspicious answers (identical answers, including mistakes, etc) but they're not conclusive enough to reject. We'll take the chance with interviewing them. If they really did cheat, then we'll find out soon enough in the interviews.

I'm bummed out because I recognized a couple of the people rejected. They made very positive impressions while chatting during the party and we were looking forward to interviewing them. Finding their names in the list of bad cheaters was really sad.

Such is life.

2009-01-26T14:32:00.004+08:00

First post! :)

Swapping my blog over to here from the old iosn.net address. For a variety of reasons, using the old address wasn't working out. Will be putting up all my old posts here shortly.

Yeah, it will look weird - the "First Post" will appear in the middle of the blog, not as the first entry. All the older posts migrated from the old blog address will have their original time stamps.

Logging SQL Commands of Selected Users in PostgreSQL

2008-03-16T18:41:00.001+08:00

Recently looked a bit into logging PostgreSQL for one of our production setups. The reason was that we wanted to find a secure way to log the SQL commands run by ordinary users but ignore the commands run by the server users. This was for security and auditing purposes. We tried logging all SQL commands once - was enough to bring the disks to a total halt due to the traffic.

After some investigation, turns out the trick is using the ROLES settings to turn on logging for the users you are interested in. As long as the user is not a superuser, this works fine. Even with superusers, you should at least see the command that turns off logging. A snippet of the configuration:

CREATE ROLE viewer;
CREATE ROLE updater;
CREATE ROLE server;

CREATE ROLE admin CREATEDB CREATEROLE;
ALTER ROLE viewer, updater, admin SET log_statement TO 'all';

CREATE ROLE ken LOGIN INHERIT;
GRANT updater TO ken;

This means that all users with viewer, updater or admin role will have all their statements logged. Of course, you still have to set logging to a secure location.

Burnt Out Light Bulbs vs Burnt Out Managers

2008-03-16T18:38:00.002+08:00

Encountered a great way to describe the management mindset during a weekly chat recently. Say there's a burnt out lightbulb in the office. You're from the maintenance department. What you perceive as the problem and how you go about resolving it is very different depending on whether you're normal staff or manager/team leader:

Staff: Problem - burnt out lightbulb. Solution - Replace bulb.
Manager/Team Leader: Problem - team has not replaced lightbulb. Solution - examine team and processes to improve them and ensure that all burnt out bulbs will be dealt with in a timely fashion in the future.

This highlights the major shift in thinking anytime someone begins moving up the management ranks. Too often, I've seen recently promoted team leaders/managers try to take all the burden of work onto themselves. Anytime they see something wrong, they try to fix it themselves. That is not their job and in the long run, it only burns them out. It is a short term solution that merely masks the underlying problems.

Management isn't about doing the task. It's about training, enabling and motivating your team to do the tasks. The focus is no longer on what you as a person can do but about what you can enable your team to do. You must clean out the obstacles in their way, provide missing training and organize resources so that the team functions at peak productivity.

Another example is watching a team that is struggling because of insufficient teammates. Because there aren't enough people around to do the job, the manager often rolls up his/her sleeves and jumps into the tasks, trying to add 1 more person's productivity to the team. That is fine if and only if the manager has already done everything in his/her power to increase team size. Has every avenue to expand the team been thoroughly explored? Are there no other resources the manager can beg/borrow/steal? In many cases, it is better to let team productivity suffer in the short term while management resources are devoted to building for the future.

Being a manager isn't about what you can do. It is all about what your team can do. It is a very different way of thinking.

Slave or Free Citizen?

2008-02-12T18:36:00.002+08:00

My attitudes towards work has changed a lot over the years. At graduation, I had the usual, "Get into big corporation and work hard" mentality. Over the years, that has changed and mellowed a lot as I have worked with and in several large and dysfunctional organizations. I have come to figure out what I wanted out of life and what type of organization would best suite my needs. I have also learned to laugh at the misconceptions that I had at graduation.

That's why articles like this annoy me. Entitled "4 Things You Should Never Say to the CEO", it gives readers the impression that they must work in fear of the single mistake that will destroy their careers and ruin their lives forever. It suggests that CEOs/bosses are like Greek gods, capricious and unpredictable. Employees, even Vice-Presidents, must tremble in fear.

In the current recession fearing climate, this article and many others like it give the impression that the employee is little more than a slave. They live in fear for their jobs, work only because they have no other choice and are at the mercy of their bosses. Unfortunately enough, too many believe this. There are too many people out there working under miserable conditions and accepting abuse that they should not. The term wage slave exists for a reason. Nothing could be further from the truth.

We are all free beings and we should act like it. In my opinion, there are a few things that distinguish a free being from a wage slave:

Attitude
Financial Independence
Ability

ATTITUDE

This is the most critical part of being free - you must believe that you are. No matter what you encounter, you always have a choice and you _choose_ to be where you are. No matter what, there is always another option. It might not be any more pleasant than where you currently are at, but you chose to take this option rather than one of the many alternatives

This is liberating in a few ways. Firstly, since the choice is yours to make, you are free to unmake your choice or to choose something better if things change. If your boss or co-worker is an unbearable pain, then do something about it! You have many options, including switching jobs. You are never truly trapped.

Second, by realizing that you chose this option out of many others, you keep your eyes open for other potentially better choices that come by. Too many people fail to pay attention and miss the many other better opportunities out there.

Third, since it was a choice, you know your reasons for staying in this job. You know what benefits and drawbacks of this choice as opposed to the others you could have chosen. This reminds you of the positive reasons why you are here, to balance any potentially negative reasons.

Finally, knowing that its your choice makes it your responsibility. For many, this means that instead of passively accepting, they are constantly looking for ways to improve their situation.

FINANCIAL INDEPENDENCE

Many people say that they must stay at their jobs because they feel they are dependent on the pay check. They cannot quit because the loss of income would be disastrous to them.

Financial independence is the solution to that. It's unfortunately far too large a subject to be properly covered here. However, you do not need full independence to be free. You merely need a sufficient buffer to cover you until you have secured another stream of revenue. Often a reserve cash buffer of six months salary is more than enough for this.

Building this reserve takes some effort of course but for most, it is quite achievable. It requires discipline, sacrifice and living within one's means. However, the feeling of liberation and freedom when you have achieved it is worth it. The idea that you can quit the next time someone attempts to mistreat you at your job gives an immense feeling of satisfaction and confidence.

ABILITY

Finally, being really good at what you do opens up a lot more options. Even during a recession, all the competent people I knew were in demand and able to find employment with ease. The market for good people is always tight, no matter how tough it may be for others.

Once again, volumes can be written about the need for constantly improving one's skills and ensuring they are relevant. Just do it. Keep improving and your ability to earn will increase too.

This has a side benefit of reducing your need for financial independence. The better you are, the less financial reserve you need. It is also easier for you to earn money and save it.

FINAL THOUGHTS

Everyone's starting point in life is different. Not everyone has the same array of choices and some are more fortunate than others. That said, it is still our choice and still in our power to make a difference in our lives. We do not have to take the abuse that dysfunctional organizations dish out.

This blog post started as a rant against the poor way many organizations/managers treat their people - as slaves, not as free humans. However, this condition will last for as long as we allow it. As long as enough humans accept abuse and slave-like treatment from their employers and managers, this situation will not change.

If you are unhappy with the way you are being treated, you have to demand better treatment for yourself. However, you can only do this after making sure you are in a position to make these demands. Attitude, financial independence and ability. Get to work collecting them :)

Getting promoted to the level you are performing at ...

2007-12-18T18:29:00.001+08:00

Some of the stuff I write about aren't my own thoughts but thoughts from fellow front-line managers. In particular, today's thought comes from Cindy:

You do not get promoted because you are doing your job well. You get promoted because you have been performing at the next level long enough for people to notice and reward you.

This came out of a discussion about how some people (fortunately not from Exoweb) were thinking that just because they had just started doing well at their jobs, they were due a promotion/raise. Coming from the other side of the trenches, this doesn't seem really realistic to me.

First - you never promote lightly. For the most part, you cannot demote someone. You end up having to terminate that person if you make a mistake. So until you are absolutely sure this person can perform at the higher level, you take your time with the promotion.

Second - humans have highly variable performance. Someone who is highly motivated today may be very demotivated tomorrow for no apparent reason. If what you are seeing is one of those peaks, promoting the person would be a mistake. One week later, the person could very be performing at a much lower level again. You need proven, sustained performance above the minimum level for the position to be promoted to.

Finally, a person doing well for the current position does not merit a promotion. Not unless you want the Peter Principle throughout the organization (people getting promoted to their level of incompetence.) A person doing well in their current position may get raises and bonuses, but never a promotion. Proof that one does well at the current position is rarely proof that the person would do better at a higher level. The skills are often different. The skills required for a software developer are different from that of an architect, project manager, etc.

Year End Thoughts ...

2007-12-13T14:07:00.002+08:00

The year draws to a close and it is time for the yearly retrospective again. Time really does pass quickly:

2007 RESOLUTIONS

So how did I do? From the 2006 year end retrospective, I had the following two goals:

Make myself (almost) redundant
Teach, not do

The first part is pretty much complete and in some ways through no real effort on my part. What happened in 2007 was that I stood aside and let my team leaders take on more and more responsibility. They all rose to the challenge and the team functions pretty well without me. These days, my role is mostly HR management with a small bit of project management on the side. The PM work isn't even because it is really necessary, but more because it is much faster for me to do it due to language and experience issues. My team leaders are a bit overworked at times so I jump in here and there to balance the load.

The second part turned out to be quite a bit different from what I expected. Rather than having long lectures and training sessions on how to do the work, it turned out to be a lot more effective to just let people do the work and give feedback where necessary. My team members are good enough that, for the most part, you can just throw them in and yell, "have fun swimming!" Now and then, feedback from gathered from peers or my personal experiences helped several people to improve what they were doing even better. However, it would be quite laughable to claim that I really taught them anything. For the most part, they figured out how to do things themselves and often came up with new ways that were far superior.

So in conclusion, both goals were achieved but in ways quite different from what I envisioned. I suppose this is one of the benefits of working with really good people :)

2007 LESSONS

The following are some of my major lessons for 2007. All these topics will probably be covered in more detail in later blogs:

Timely feedback
Groups also grow and mature
Really learning from your team mates

If 2007 is anything to go by, the greatest value a manager in Exoweb provides is in giving timely feedback - both positive and negative. As mentioned earlier, I found that I did not really have to teach much. However, I did have to tell people how far along they were towards achieving the goals set for them. Without clear feedback, people are either demoralized by the uncertainty of their performance or wrongly assume they are doing well/badly. In retrospect, some of the people that did not work out in 2004-2006 may well have done a lot better given the correct feedback and corrective actions early. Things have gone a lot smoother in 2007 as the management team as a whole has gotten better at this.

Another realization was that it is not just people that grow and mature. Groups somehow do too. They take a life and culture of their own, often picking up best practices that make the group far superior than merely adding the individual members together. One clear example was when we threw a group of new people into a project. Although everyone individually was quite competent, they were all new and unfamiliar with best practices the rest of Exoweb has gained through bitter experience. As a result, the team repeated all the same mistakes and was on the verge of turning dysfunctional. Ultimately we disbanded the project and team, merged everyone into mature teams and things work out much better after that.

2007 was also the year I learned the most from my teammates. Partly because they were getting comfortable and confident in their teams/roles, partly because there was much more communication this year. The many subtleties of working with people, what works, what doesn't - all these I have learned from talking to my colleagues. Some give me feedback on what they have seen me do, giving me a chance to tweak methods better. Others try new things that have obviously good results. These I adapt into my own daily routine. Overall, 2007 was the year I learned the most about managing and working with people - even more than my days in evil MBA school.

There of course is a lot more I have learned over the course of the year. In many ways, working in Exoweb is like being in school - you are always learning new things every day. May this always continue.

2008 RESOLUTIONS

Strangely enough, I cannot think of anything that I would make as a new year's resolution. Sure, I can see many areas that need improvement/fixing, but these are things that will naturally fix themselves over the course of time. Maybe I'm getting complacent, but I honestly think that Exoweb has built itself a self-sustaining culture that will continuously improve itself as time goes on.

If anything, my resolution for 2008 is to ensure that 2008 is even better than 2007. 2007 was a really good year, but one should always try to strive for more. I would like to make things even better next year, from all aspects. What specifically, we'll figure it out as we go. However, Exoweb in 2008 should be an even greater place to work in than the years before ...

Language learning on FOSS vs OSX

2007-09-23T18:27:00.001+08:00

My primary desktop is OSX and for the most part, it suits my purpose well. 99% of the time, everything just works, there are few hassles and things behave the way you expect them to. The Ubuntu releases are close but not quite there yet for me.

As great a desktop as OSX is though, it only shines in the most commonly used functions. In many niche areas, FOSS can provide superior solutions by virtue of the huge variety of applications, developers and freedom to develop on it. One such area that comes to mind these days is language learning.

I'm barely able to read in Chinese and trying to work my way through a website or document is pretty much impossible without a dictionary. I particularly need a dictionary that can immediately translate words I highlight on screen. On OSX, the options are pretty limited. Most solutions are shareware or paid software. Right now I'm using TranslateIt! which works fairly well. I highlight the word(s) I want translated, hit a keyboard combination and TranslateIt! pops up, hopefully with the translations. It is one extra step though. The paid version for TranslateIt! includes the functionality to get immediate translation after I highlight it.

My colleagues on FOSS desktops don't need to pay for this. StarDict comes with most distributions and does the highlight/translate thing right out of the box. It's invaluable in a mixed Chinese/English environment like Exoweb. Besides the software, the power of FOSS shows up in StarDict's dictionaries, which are varied and extremely useful. So much so that TranslateIt! actually uses StarDict dictionaries and all the translations I am using are StarDict's. Without StarDict, TranslateIt! would actually be useless to me.

My favorite StarDict dictionaries include (all zh_CN -> en):

cedict-gb dictionary (has pinyin and tone marks. Must have)
langdao-ce-gb (a much larger vocabulary but translations sometimes not precise. No pinyin)
Chinese idiom dictionary (a dictionary translating chinese idioms. Unfortunately the translations are chinese->chinese and some colleagues have said its translations are suspect)

Two are under the GPL while the cedict library is under its own license similar to cc-by-nc.

While I'm sticking to TranslateIt! until such time StarDict works natively under OSX, I would simply be unable to read anything at all in Chinese without StarDict's dictionaries.

We Make It Up As We Go

2007-09-23T18:25:00.001+08:00

Every now and then I find myself repeating the same thing in weekly chats, so I try to note them down in a blog post so the Exowebers that actually read the Exoweb Planet have a chance to see it. This particular one has to do with how we developed our work processes and best practices. There is no mystical method, no profound MBA insights or deep pondering. Quite simply, we make it up as we go.

That's not to say all our current processes are arbitrary. Every process or practice we have, we evolved to overcome a perceived problem. We experiment with different processes and those that work, we kept. Those that failed we learned from and moved on. The entire organization is a work in process, continuously trying to improve itself.

There are a few core values we do have, which very much reflect those of agile methodology: People over process, working code over documentation, etc. We also believe very much in making this a great place to work. Since you spend a huge portion of your waking hours at work, why wouldn't you want to make it as pleasant, as fun, as challenging a place as possible?

From these core values we simply figure things out as we go. For example, our current process of code reviews was triggered by the realization that our code quality wasn't good enough, that too much bad code and bugs were leaking into production systems. We tried the NASA style group code reviews but found that much too heavy. We then tried having a couple of core code reviewers doing all the work, but found that it did not scale and that the benefits of code review actually were disproportionately accumulating with the core reviewers. Our current method is a lightweight team review process that seems to combine the benefits of code review while reducing the cost. We are likely to make more tweaks and changes in the future, depending on future needs and ideas.

What does this mean for Exowebers? Most important of all, it means that it is the past ideas of all of us that have created the great environment we have. It means it is your future ideas that will make us an even better place to work. You need to pay attention to your environment, be willing to question our processes and methodologies, and contribute new ideas when they come to mind. No process is sacred. Given good enough reason, visible enough benefits, anything can be changed.

You can contribute ideas openly, by floating RFCs by email, or you can quietly suggest them to your team leaders in weekly chats. Whatever method you choose, it is important that the ideas are communicated and considered. Only then can we as a company improve. Only then can we make this place an even better place to work.

Yes, this includes higher salaries too. If you have ideas on how to make us more profitable, we can all share in the profits in the form of higher salaries :)

What Happens When You Turn Fsync Off On Postgres

2007-09-23T18:23:00.000+08:00

We use the PostgreSQL database extensively to handle a fairly large amount of data. Our largest single database is over 25G in size, with a fair amount of transactions going through it daily. As such, we've had to do a lot of optimization over time. One of our experiments was turning fsync off on one of our non-critical databases. In retrospect, this probably was not that great an idea ...

This database was a non-critical but fairly write intensive database. It logged a lot of information, largely in the form of inserts. Inserts in postgres can be a bit slow sometimes since a insertions tends to lock the same section of the index until the insert is complete, forcing all inserts to go in sequence. Updates are usually a lot faster if you're updating different rows since they don't all rely on the same section of the index and can often be done simultaneously.

The fsync option slows this down even further, since postgres then waits for the data to be flushed to disk successfully before continuing on with the next operation. Not a problem for low traffic databases but if you attempt to insert hundreds of transactions a second, the milliseconds spent waiting for the disk to write the data completely really hurts. fsync ensures data integrity but at the price of speed, especially in the case of unexpected power failure.

Since this was a non-critical database and losing data wasn't really a problem (we could either recreate it or live without it), we turned fsync off on this database. All went well for months, until we actually did suffer a power failure. During the busiest period possible. Good old Murphy.

At any rate, once we brought everything back up, things seemed to work as usual ... for about 30 minutes. Then we realized our servers were frequently losing connection to this particular database. Investigations revealed that the postgres processes were terminating themselves with messages like "Error: out of memory" or complaints about data inconsistency. Yep, we got our first corrupted Postgres database. The first one I've encountered in over 7 years of using this database.

I have to admit, I had very little clue on how to recover a corrupted database and each database was corrupted slightly differently. Initially it appeared only the indexes were damaged and a reindex removed most of the problems. Later we found that there was some damage to the tables themselves (took a long time to find that) and we attempted to restore through a backup. The Write Ahead Log (WAL) backup proved to be useless. Those were corrupted or inconsistent. Strangely enough, the database could still do a pg_dump, so we just dumped out all the data and reloaded it back in the database. This ultimately fixed everything.

Morale of story - don't turn fsync off unless you really know what you're doing, including how to detect database corruption and fix it. Our biggest problem was that postgres, unlike MySQL, does not scream "Table/database corruption!" immediately. It took us a while to determine what the problem was. Then again, unless you turn off fsync, it is probably something that almost never happens on postgres. I've had tons of corrupted MySQL databases. This is my first corrupted postgres database.

Three Questions ...

2007-07-21T18:17:00.001+08:00

This is the English version of Cindy's "3 Questions" blog post, written from a different viewpoint. As a bit of background, Cindy and I are both managers in Exoweb and one of our responsibilities is to carry out weekly chats with the people in our teams. Both of us have been asked these questions before and we figured it would be of interest to Exoweb people in general. I'm writing this without having actually read her post, so it should be interesting to see how our viewpoints differ.

The questions are:

Does Exoweb give team members a chance to switch career paths?
Are personnel evaluations transparent?
If an employee makes a mistake, how will the company handle things?

Does Exoweb Give Team Members a Chance to Switch Career Paths?

Yes. We definitely prefer to use existing, proven and trusted team members to fill open positions, rather than taking a risk with hiring an unknown person from the outside. As long as the person appears to be competent and capable of taking the task, we are quite willing to take a chance.

We have had a few examples of these career changes so far, mostly people switching to project management positions. Two of our admin staff have switched over to project management roles in our software development teams. The admin team handles everything in Exoweb but actual software development, so is a lot more challenging than the name suggests. After battling with unreliable suppliers, organizing events for 40-50+ people and solving the problems that crop up in daily office life, they have proven that they are more than capable of getting the job done. That actually makes them very good project managers as they have the "get things done" attitude.

We had a software developer also try his hand at this a while back. He did a pretty good job at it, but after 6 months, found that it was really not something that he wanted to do and switched back to pure technology. This is also a learning experience that we are quite happy to allow our people to have within Exoweb - the chance to try new things out and discover if a certain path is for them. If it doesn't work out, such is life. You never know until you try and the experience typically makes you appreciate your chosen role more.

Are Personnel Evaluations Transparent?

Depends. We do our best to make obvious to the team member how they are perceived, both by team leaders and fellow team members. Depending on the team lead's management style and workload, we try to schedule chats between once a week to once every couple of months. During these chats, past accomplishments, challenges and issues are discussed, along with feedback on how the team member is doing. These chats are mostly informational and problem solving - for both sides to quickly identify and resolve issues that crop up. They also usually give team members a fairly clear idea of how they are doing.

Of course, not every team lead does things in the same way and there are large variances between each team. As a general rule we try to provide feedback, both positive and negative, as soon as possible and resolve problems before the grow too big.

However, we try to keep these evaluations as private discussions between team member and team lead. In this sense, personnel evaluations are very opaque. We usually do not make it obvious to others when someone is underperforming, even if the under performer knows very clearly that he/she is in trouble. Making this kind of information public while trying to resolve the problem is rarely helpful. We do share evaluations between team leads though, both for redundancy (in case one team lead has to take over for another) and also as knowledge sharing (e.g. how to handle situation x).

If An Employee Makes a Mistake, How Will the Company Handle Things?

In one of the management books I read many years ago, I came across a great story about making mistakes. Unfortunately, I no longer recall where it comes from (if anyone recognizes, please let me know!), but it goes something like this (heavily paraphrased):

There was a young man who had just joined Mega Corp, a huge organization with many capable and talented people. Being very eager to prove himself, the young man throws himself into his tasks and pushes himself to the limits of his abilities. Unfortunately, his eagerness results in him making a huge mistake, one that ultimately ends up costing the company USD10 million dollars.
This young man then walks into his manager's office, confesses to his mistake and finally ends with, "I guess you want my resignation now?"
His manager looks at the young man with a incredulous expression and replies, "are you crazy? We just spent 10 million dollars training you! How can you leave now? Now get back out there and apply your newly gained knowledge to earn us our 10 million dollars back!"

Everyone makes mistakes. As long as one honestly did their best and learns from it, we recognize that this is only natural and move on. We work together with each other to try to minimize mistakes and compensate for each other. We spend less time figuring out who is to blame and more on how to ensure it will never happen again.

For the most part, almost all first-time mistakes are forgiven. The rare exceptions we've encountered are when someone makes a mistake so bad that trust is irrevocably broken. As a highly trust based organization, once someone proves themselves untrustworthy, it is simply not possible to continue working in Exoweb, no matter how much we like the person. Fortunately, it is very hard to make this kind of mistake. It has only happened once in my memory.

Things I Would Like to See More In My Teams

2007-06-28T18:20:00.001+08:00

Overall, I'm working with a great bunch of people, so don't let the topic mislead you - I'm really happy with the people I work with. However, nothing's ever perfect and there's always room for improvement. The following are things that have recently made me stop and think, "I wish I saw more of that ..."

Curiosity
Sharing of information
Dissatisfaction

Curiosity

There is an immense amount of information that we don't know. In the IT industry particularly, it is critically important to keep learning and improving ourselves. A little curiosity goes a long way on making this happen.

We all know that the technologies we learn today are often out of date within five years. Even within the same language, the change can be enormous over time. Java users five years ago would not recognize or be able to read some of the code being produced today. Constant learning is critical to a long term career in IT. For that matter, it's possible that the famous Chinese stereotype that people over 30 cannot code may just be because too many fail to learn and by age thirty, their skills are almost useless.

Keeping up with the information flood out there is of course a skill. There is no way to learn it all and a challenge to find those really worth our time. However, a little effort such as by keeping up with a few good magazines and blogs goes a long way.

Sharing of Information

Everyone has different strengths and experiences. We all have learned different things and it is much easier to learn from someone who has already learned something than to figure out things from scratch. Because of this, learning from each other is a quicker path to improving ourselves than struggling on our own.

Yet too few try to learn from each other. This comes in two different forms - some don't like to share. They do not talk about what they have learned, do not mention it to others, etc. They do not blog, nor do they present in ExoForums (Exoweb's weekly information sharing session) nor do they even mention it in conversation. Anything they learn stays with them.

The other problem is actually receiving this information. This is related to the curiosity issue mentioned earlier. Exoweb has a planet listing all the blogs of colleagues, yet there are a fair number of Exowebers who do not even bother to read this. If one will not learn from their colleagues, who's knowledge and experience is most directly related to their work, what chance is there that this person will learn from anywhere else?

Dissatisfaction

Believe it or not, being dissatisfied can be a good thing if channeled properly. Dissatisfaction indicates that there is something that can be improved. By seizing upon this dissatisfaction and making the effort to ensure that improvement happens, one makes one's environment and self better.

It is too easy to notice a problem and shrug it off, accepting that it is a problem that one has to live with. Quite often this is not the case. If we invest a little of our time each day in removing the most irritating, most unpleasant annoyances in our lives, we find that bit by bit, our lives get better and better. Time and resources are limited so we can never fix everything that bothers us. However, as long as we take the time and effort to fix things, each day gets more and more enjoyable. Ultimately life is short. Why make it miserable too?

Most cultures with a strong work ethic believe in enduring hard, boring work to succeed. However, there's nothing in that ethic that says we can't find ways of doing things better. Just because it is hard or boring doesn't mean it always has to be that way. Sometimes it is far too costly to change. Sometimes it is not. Until we have aggressively investigated the problem, we will never know for sure.

Spammers And Gated Communities On The Web

2007-05-27T18:14:00.002+08:00

YAR - Yet Another Rant ...

From my experiences living in various locations around the world, I have always loved the locations where there are very few gated communities. You can enter the different areas very freely - there are few security checkpoints, very few guards and movement is very unrestricted. These are usually the very safe areas and there's a lot of interaction between people in the different areas. These are usually friendlier, livelier and richer communities, especially with inter-community mingling.

On the other hand, the areas with lots of tightly gated communities bother me. This usually implies that the area is not very safe and you need a variety of fences and security personnel to keep out undesirable and potentially dangerous elements. These communities live in fear and are very closed. While it is peaceful inside these communities, it is still a prison of sorts.

Lately, I have begun feeling that the web is turning more and more into a group of gated communities. Mind you, it's not just linked-chain fences and rent-a-cops. It feels like many areas are starting to turn into fortresses - high walls, constant patrolling of the boundaries by guards with very deadly weapons and constant challenges while you wander through. The problem - spammers.

It used to be that spammers only sent email to your mailbox. Today, they find anything and everything they can get to make you click on their links. Within days of IOSN turning on trackbacks, many of the blog posts were filled with spam trackbacks that were just trying to sell their latest crap. Comment spam became common a couple of years ago, which is why most sites don't allow anonymous comments.

A few months ago, Exoweb set up a test server for the Beijing Linux User's Group. This was a default content management software setup (I believe it was plone) so anyone could register anonymously. It was not publicized and only a small handful of people should have known about it. Next thing we knew, some virus-writer had (obviously automatically) created a ton of accounts and hosted a huge number of trojan binaries or spam pages on the site and spammed people to click onto it. We weren't the only ones hit, but it was quite frustrating when we found out and had to shut down all logins. Till today, you can probably find some remnants of it if you google for Exoweb. I don't even want to get started on how frequently our client sites are probed by bots daily.

These low-lives who send their bots out looking for unsecured sites are the equivalent of thugs roaming your streets, trying every single door they come across. How safe would you feel if you saw criminals brazenly looking for unlocked doors in broad daylight in your neighbourhood? Because of this, people are trying to reduce access to their communities/sites, making sure only verified people are allowed to participate.

One side effect of this is that it gets more and more inconvenient to join communities or participate. Most sites will not allow participation without registration. Registration involves more hoops to jump through (captchas, email confirmations, etc). Gone are the days when you could just swing by a friend's blog and just drop a comment anonymously. Participation is now deliberate - you must really want to say something and have the patience to jump through hoops to make it. Gone are the days when you could just fire off an off-the-cuff post or comment.

This keeps people in small little gated communities. Once they have created accounts in a particular site, they are less likely to bother to register in other sites. Communities get more insular with less intermingling. Diversity, cross-pollination and knowledge is reduced ...

All this, because some people have found it perfectly acceptable to violate all netiquette for personal gain and everyone else lacks the will to do something about it. Is is sad. How quickly the promise of the internet - incredibly convenient and cheap communication and access to information - is sullied and tainted by greedy human nature.

Tallinn Thoughts

2007-03-09T17:56:00.000+08:00

Yet another quickie post with some random thoughts about Tallinn, Estonia. Hopefully useful for my colleagues coming over shortly and anyone else who might be visiting soon. I won't touch anything that should be easily accessible from a tourism website.

First, a couple of thoughts about traveling here - the no-liquids rule and small planes on the final hop here really make it much easier to check in luggage rather than relying on carry-on. Now that flying in EU has the same restrictions on carrying liquids as flying in the US, you pretty much cannot carry any creams, gels or even toothpaste. If you try to stick to carry-on luggage to avoid baggage claim the way I do, you will have to buy everything again when you hit Tallinn. Also, every time I have flown to Tallinn (three times on two different airlines) the final hop uses a very small turbo-propeller driven plane of Russian construction. The problem with these planes is that the overhead compartment is tiny and cannot hold much more than a backpack or a standard briefcase. None of those large carry-ons.

Flying FinAir from China stops in Helsinki but you can stay in the international section of the airport. No EU (Schengen) visa required. Flying Air China requires changing planes/airlines so you will need an EU visa. Much more convenient to fly FinAir even if the EU visa isn't a hassle.

Tallinn appears to be pretty safe. There are people walking alone late at night, so violent crime doesn't appear to be a problem for most folk. The city is small enough that you can walk to most places. Bring a comfortable pair of shoes. It also has a lot of beautiful old europe architecture so bring a camera if you are into snapping pictures.

Lots of decent western food places here, but other than Indian and Japanese, not a lot of Asian food. Prices are about double the price for western food in both Malaysia or Beijing and many times the price of a standard meal in either country. Still quite cheap for an european country though.

A very nice place overall. I could be convinced to come here a lot more often ...

The Writing On The Wall ...

2007-02-24T17:51:00.000+08:00

Just a quickie on a thought that struck me while responding to a friend's email - too many students are not aware of the quiet FOSS wave that is happening under the surface, particularly here in China. They are still focusing their learning efforts on Java, C# and Windows without a thought to picking up basic FOSS skills.

Yet if you pay attention, you can see the trend. From today's news, HP made $25 million in sales directly related to Debian GNU/Linux. This is just one Linux distribution mind you, and one with no commercial company or advertising budget behind it. Exoweb is slightly responsible for that as we (and our clients) bought more than EUR 15k of servers from them last year. :)

In our daily work integrating with third party providers, we are seeing more and more FOSS based products. Not all are fully FOSS based - many are WIMP (Windows, IIS, MySQL and PHP). This FOSS trend is especially noticeable from startup companies or companies with limited legacy infrastructure. We love this of course, since they are typically easier for us to integrate, host and maintain. Yes, we (or our clients) do pay a fair amount for these products so they are making a lot of money while being FOSS based.

My point? For anyone who really wants to be in the tech industry, FOSS skills, no matter how basic, will give you a valuable advantage. Otherwise you will be locking yourself out of a rapidly growing portion of the industry. Besides, as a craftsman, wouldn't you want to know every tool available? Otherwise how can you use the right tool for the job?

The Three Aspects of Management

2007-02-09T18:08:00.003+08:00

Found myself explaining for the third time in a day yesterday about the three different aspects of management that I deal with in my work in Exoweb. Thought it best to blog about them, to put down in writing what I have had to repeat multiple times last week.

These are my personal opinions/divisions of course and have no grounding in scientific research. They are merely based on personal experience. However, all managers in Exoweb perform at least one aspect and some can do all three.

In my daily work, the three aspects I end up spending my management time on are:

Tech Management
Project Management
Human Resource Management

Technical Management

This deals with the actual work that needs doing and varies by department. It is difficult, if not impossible, to do this type of management without solid knowledge and experience in doing this task yourself. e.g. An Accounting manager must understand and be able to balance his own books before attempting to supervise a team. Likewise a senior software designer/architect needs to solidly understand software (object oriented or otherwise) design patterns and the abilities/limitations of the technologies used to be effective.

This aspect of management typically involves setting and maintaining general technical quality in a team. Properly done, this also involves mentoring and training to ensure that team members are constantly improving and requiring less and less management. This is not that frequently done in practice though.

This aspect appears to require the least social skills of the three.

Project Management

To me this has two separate components - client management and juggling tasks and resources to complete the project in time. In theory, they could be handled by different people but in practice this is so inefficient that it is not practical.

Client management requires the most communication skills of the two as it involves managing client expectations, negotiating resources and dealing with the inevitable problems that arise. Strong social skills are especially useful with the class of clients that requires you to wine and dine them.

The second component requires mostly a logical mind to figure out how to achieve the maximum value for the client given available resources. This requires not just understanding client priorities but also the best way to utilize the team to achieve these priorities. e.g. Can the manager arrange the tasks so that dependencies between them are minimized and the team functions like a well oiled machine? Or will tasks clash, resulting in periods of inactivity followed by frantic crunch periods? What are the deadlines coming up and can the team make them?

Some understanding of how the work is done is useful but not to the depth of the tech manager. We have a few project managers that lack a technical background but are still quite capable of carrying out this component of management. Strangely enough, gamers who enjoy strategy or RPG games often do well at this. Ultimately, they involve the same skills - maximizing results by carefully allocating scarce resources.

Human Resource Management

This is often the most neglected aspect of management - the understanding, combination, development and motivation of team members. It also cannot be delegated to the HR department. Both First Break All The Rules and Peopleware are full of studies that show it is the first line manager that has the largest impact on employee morale, productivity and development. All the HR processes and motivational posters in the world will achieve nothing if the immediate manager of a team member sucks at this.

It is easy to neglect this because it this type of management is a long term investment and the symptoms of neglect take so long to show up. Morale is rarely measured and there are many possible reasons for why productivity is suffering. It is easy to blame/focus on something else, especially when there are so many other tasks that obviously need doing. When clients are screaming about late projects, it is very hard to invest in long-term training, even when it is clear that the lack of certain skills among team members is the reason why projects are late.

In my career, the best managers I remember - the ones I was happiest and most productive working for - were the ones that knew exactly how to motivate the team. They may have been merely competent at the other aspects (in some cases minimally competent) but they were far superior to their peers in understanding what made their people tick. They knew how to combine people together so their strengths built on each other and their shortcomings were compensated for by other team members or processes. Working for them was a joy and it was amazing what a gelled team could achieve. It could easily multiply the productivity of a team.

Doing this is hard, especially for the stereotypically anti-social techie. It requires more than just social skills. The manager also needs to really understand the unique strengths and characteristics of each team member. Many company processes/managers treat team members as interchangeable machines that perform the same as each other, rather than the unique individuals they are, with individual needs. They are forced to do this either because they cannot recognize the individuality of their people or are unable to find managers who can manage in this way. This results in work processes and practices designed for the lowest common denominator - meant to protect us from our weaknesses. How much more enriching would it be to work in a place that recognized and allowed us to utilize our strengths and achieve our fullest potential?

Final Thoughts

There are extremely few people who can do all three roles well. Even finding people who can do two (e.g. Tech and Project Management) roles very well is very hard. We are experimenting within Exoweb with having specialists in each of the three roles. Although not nearly as efficient as having all the roles combined in one person (communication overhead is much higher) it has been proven to work with the first two roles. The jury is still out on the third role - we don't know for sure if it can be done by someone who isn't already doing at least one of the other roles. It is also such a sensitive function that we are reluctant to experiment too much with it.

There are so many different ways to manage and we're only beginning to figure out what works for Exoweb's culture and people. Ultimately though, I would stick to the philosophy best summarized in this quote: "that's what management is - it's not about how much you can do yourself. It's about how much you can help others do"

2006 Year End Thoughts

2006-12-31T17:24:00.003+08:00

Can't believe another year has come and gone. It seems like only yesterday that I wrote the 2005 Year End Thoughts, and only a few days ago that I came back to China and started my career in Exoweb. Time really does fly when you are having fun, and I certainly have been having a great deal of fun in the last few years. Sometimes stressful and sleepless fun, but fun all the same :)

General summary of thoughts:

Things that I am happy about:

Exoweb
- Teams improving, growing
- Culture
Personal
- Facing the different challenges over the last year
- Learning to delegate

Things that need to be improved:

Exoweb
- Giving back to FOSS
- Creating a continuous learning culture

Personal
- Working with people

Goals for 2007:

Make self (almost) redundant
Teach, not do

Things that I am happy about - Exoweb

Teams improving and growing - This last year has really seen a dramatic growth in the quality of the people within Exoweb. While continuously improving our HR process has helped us get some really good people, the major change has been the steady improvement of people already within Exoweb. There are various reasons for this but ultimately, I believe it is because we have finally passed a critical mass of smart people and everyone is now learning from each other. Processes/practices such as code reviews, ExoForums and blogging have helped encourage the sharing of information and it appears to be self-perpetuating. People are demanding high standards from each other and everyone brings in their own knowledge and skills. My primary concern is no longer raising standards but knowing when to step aside and let people more knowledgable than me figure things out.

Culture - building a good company culture isn't easy and I'll be one of the first to admit that I haven't really got the faintest clue how to go about it. Culture building is all about people and people skills aren't my greatest strength. Perhaps it is because I'm so clueless about culture building that I'm amazed that Exoweb has somehow produced (possibly by accident) a relaxed, open culture that many people find really attractive and unique. It's not perfect by a long shot and there's still so much more we can improve. Yet it is quite gratifying to hear people tell me that they find the culture of Exoweb one of the key selling points of the company and that they have never worked in a better place. Not everyone feels that way of course, but enough do. Personally, I've never worked with a better bunch of people, in a more comfortable environment. I am naturally biased though.

Things that I am happy about - Personal

Facing and Surviving the Challenges of the 2006 - 2004 and 2005 had different, lower level challenges for me. I joined Exoweb in 2004 as a senior team lead, so my main concern then was ensuring the success of a project with a small (5 devs total) team. The challenges then were a lot simpler! 2005 brought the challenge of maintaining quality in a team that was growing too large for me to personally review all code. 2006 was quite different - Exoweb continued to grow and the challenges that came up daily kept changing. The early part of 2006 was a battle to figure out how to scale technically. Or rather, how to ensure that the things I used to do still were being done when it was becoming obvious that one person could not possibly do it all.

The technical aspect of that problem was solved in early to mid 2006, among other things by the growing abilities of the team and instituting a more scalable version of code reviews (we created a cool trac plugin for this). This had the most fortuitous side effect of promoting learning even more from each other. With a great team that is continuously learning, most of my earlier technical challenges faded away.

Late 2006 was more an issue of scaling Project and Human Resources (HR) management responsibilities. The funny part was that as my team got better (and larger), the seniors within the team came to the conclusion that they did not want to do either PM or HR work, pushing all that to me. As a result, the workload in this aspect grew a lot faster than the team did in 2006. Surviving the PM aspect was done by the fine art of delegation (more about that later) while the HR aspect is still a serious work in progress.

Ultimately, I'm happy that I survived all the challenges of 2006. Ken in 2005, looking at these challenges, would have been quite intimidated. It was a good idea that I went into 2006 blind to the challenges ahead :). Looking back, I can certainly see many areas I could have done things far better but at least I can say that I haven't made an absolute mess of things.

Learning to Delegate - Every book about management talks about how one needs to delegate. Yet they tend to gloss over how to delegate. One thing I quickly learned long ago was that if you just pass a task to someone and hope that they will do it right, 90% of the time things turn out badly. It took some experimentation and trial and error, but in the end, my great lesson in 2006 was realizing that it all boiled down to figuring out who I could delegate what to. Everyone is different, with different strengths and weaknesses. The challenge was to find someone (or combination of people) that had the right strengths to do the task on hand. Not a skill I am strong at (more later about this), so it was harder than it should have been. But as I write this today, I have a good team that functions very well together, with most of the critical tasks covered and working well together.

Yes, I realize this may seem blindingly obvious to some people but it wasn't that obvious to me.

Things that need to be improved - Exoweb

Giving back to FOSS - Despite being strong believers and users of Free/Open Source Software, we don't contribute back nearly enough. Sure, some of us personally have done some work in FOSS advocacy or have code contributions here and there. However, as a company, I am still quite dissatisfied with what we have contributed back. Besides contributing server space (python.cn, Beijing LUG, etc), software usage, bug reports and a few patches here and there, we have given very little back to the ecosystem that makes our business model possible. Despite having people who profess to genuinely believe in FOSS, despite having a contribute back policy and allocating a percentage of developers time to such activities, too little is contributed back. This is something that we will have to focus more on in 2007.

Creating a continuous learning culture - Possibly because most of the seniors of Exoweb possess either the Learner or Input talents (see First Break All The Rules), we tend to expect that everyone will be like us - given the opportunity, will always try to learn and improve themselves. Unfortunately, that is not really the case. Some really talented software developers don't seek out knowledge for the sake of learning but are satisfied with learning only what is required to complete their task. Or despite the best intentions, they need a little external pressure. So despite having a 10% time self-improvement/contribute back policy in Exoweb, too many people do not take advantage of this. Yet continuous learning is a vital aspect of continuously improving the abilities of the organization.

Things that need to be improved - Personal

Working with People - I've touched on this previously, but I'm basically much better at computers and hardware than people. To me, computers seem so predictable - given a fixed input, they typically produce a fixed output. Humans are so much more variable, with too many factors to consider. Yet management is about people, not about computers. According to the Peter Principle, I'm quickly rising to my level of incompetence :)

However, the level of understanding of people I'm looking for might be a bit higher than most. The ability to figure out a person's strengths and weaknesses and combine them with other people/processes that complement their strengths and compensate for their weaknesses is a very rare talent. If you look at most management practices today, they are built to solve this talent shortage. Most large organizations have processes that cater to the lowest common denominator - they allow the organization to survive mistakes made by less competent people, but they get in the way of the truly talented hitting their full potential. We sometimes call that bureaucracy.

Most managers are either poor at or unwilling to manage individuals. It is hard to manage individuals - you have to really understand your team and know how to combine them to achieve maximum results. Most prefer to assume that every human being is interchangeable, that one person can be swapped for another without problems. This only works if you are using people at the lowest of their abilities, so that the job can be done by almost anyone. It does not work when you are trying to make the most out of everyone's unique combination of strengths and talents.

This will probably be my primary challenge in 2007 - to become competent at managing individuals.

Goals for 2007

Make myself (almost) redundant - I have delegated a large amount of my work to others already. I hope to finish this in 2007. Might be a while before I can delegate all the HR management aspects, but all the technical and project management aspects should be possible within 2007. I certainly hope to organize things so that I can go on a month long vacation and no one would notice :)

Teach, not do - One thing that is really hard for me - delegating tasks to someone else instead of just rolling up my sleeves and getting it done in a couple of hours. Only problem is that there are only so few hours in a day and so many problems that need to get resolved. Making myself redundant requires that I restrain myself from digging into problems and instead focus on teaching/guiding others to take over from me. Teaching is a large investment of time - it is always faster to do it yourself than to teach. But without this investment, the organization will never scale.

2007 looks like it will be bringing quite some challenges, many in areas where my strengths do not lie. Still, I wouldn't have it any other way. Life isn't fun without challenges and I can't think of a better team of people to face the unknown with than the crew of Exoweb. Happy New Year everyone!

Time To Turn In My Geek License

2006-12-28T17:16:00.001+08:00

Apparently, I'm spending too much time doing management and turning into a PHB. No one seems to think I'm technically competent anymore. Some recent conversations:

While training a junior project manager:

Me: "So remember, with new tickets, ask a technical senior to help you estimate how many hours are required to complete the task ..."
A few days later, during the usual morning meeting discussing what tickets to create, prioritize, etc:
Me: "... so I think we should put this task in Milestone x, priority critical, estimated time 8 hours"

Jr: "Ok, sounds good. I'll go get an estimate from senior x and take care of it."

Me: "Wha ... didn't I just give you an estimate?"

Jr: "Yes, but you said I needed an estimate from a senior ..."

While showing a relatively new developer the rankings of everyone in Exoweb:

Me: "... and here we have the mid-levels devs, split in 3 sub levels. Finally, these are the seniors ..."

Dev: "Wait, you're considered a senior?"

After relating the above two incidents to yet another developer:

Me: "... so apparently they don't think I'm a senior anymore ..."

Dev: "Well, if you don't tell them, it's not obvious ..."

If anyone wants me, I'll be out getting a haircut, buying some suits and gaining a lot of weight ...

KISS (or why MS CS students have a bad time in interviews ...)

2006-12-05T17:13:00.002+08:00

It's that time of the year when Master's students are hunting for jobs in China and we are flooded with resumes from students with Masters of Science in Computer Science (MScCS). We've spent the last couple of weekends interviewing the candidates that passed the front interviews and it has not been pretty. In fact, it has been pretty sad.

The biggest problem we encounter with MScCS grads is made very apparent in how they approach one of the our typical programming problems. This particular problem is fairly simple and with a little bit of thought, can be made into a linear (computer performs x more calculations for every element added to n, where x is a constant number) or O(n) type of equation. Most competent people will come up with a O(n^2) algorithm (computer performs n operations for every additional element added to n, resulting in rapidly increasing number of calculations per element added) to it at first, then after a little thought, realize that there are a lot of duplicated operations, refactor that out and come up with the linear solution.

MScCS are a little different - almost all the ones we have interviewed to date encountered this problem, probably came up with the O(n^2) equation ... then went off the deep end. They would inevitably come up with complex, fancy algorithms that utterly failed to solve the problem. These fancy algorithms would often handle the common case but fail on the boundary cases. They were fragile, easily tricked or prone to failure. When these flaws were pointed out, these candidates would come up with even more complex algorithms or add a lot more if/else checks ... resulting in even more brittle and unreliable code. None of them, if unaided, could come up with the ideal solution.

Well, that's not really true. One particular candidate, after coming up with 4 complex, unworkable algorithms, finally said, "well, since you have given me such a tight time limit, I have no choice but to brute force it." He then proceeded to give the ideal solution. Linear time, handles all boundary cases. But he would rather give 4 non-working solutions rather than the working, "inelegant" solution.

What I find strange and rather disturbing about these interviews is that it is somehow related to the mindset of those who are doing their MScCS. Bachelor level grads or people with several years of working experience rarely make this mistake. They either do it right or not at all. Yet for some strange reason, the MScCS students seem to value fancy algorithms over _working_ algorithms.

Maybe it's the MScCS curriculum here in China which tries to focus on fancy algorithms. Maybe it's just that the MScCS feel the need to prove that their abilities are above the norm for CS graduates. Maybe we just have incredibly bad luck with our candidates. Whatever the reason, extremely few MScCS fresh grads have passed our interviews. As you can imagine, in a production environment, we highly value working code and eschew the fancy algorithms, especially algorithms that needlessly complex. Simplicity and correctness is far more important in our craft.

The famous quote attributed to Brian Kernigham comes to mind:

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

When writing algorithms, another quote from Agile development comes to mind - "do the simplest thing that can possibly work." Keep It Simple ... er, let's call it Keep It Stupidly Simple (KISS) and avoid insulting anyone :)

Need your MacBook repaired? Wait a while ...

2006-09-13T17:11:00.001+08:00

So after all the fun I had with the MacBook, I eventually found that my MacBook fell victim to the usual MacBook discoloration and random shutdown problems. So did the other MacBook in Exoweb that was purchased at the same time. Last week, we sent the first of them in for repairs with Apple. Guess what the company sysadmin got back as a response from Apple?

"Apple said there's no hardware in the entire world to fix your laptop---they have to manufacture it first. "

So if you're like me and need all these issues fixed, I guess you're going to have to wait a while for the spare parts. At least for me, the random shutdown is limited to once per day, when I work the computer too hard after it has been idle for too long (either in suspend mode or just sitting there unused for several hours.) Once it has shut down once, pressing and holding down the power button for 5 seconds restarts the computer just fine and as long as I keep it busy the rest of the day, the random restarts go away.

Macbook Day 3

2006-06-03T17:08:00.001+08:00

8 am saturday morning and I am up, messing around with development, writing code and tests. Big deadline coming up? No. Urgent bugfixes? No. Just testing out the MacBook as a development environment. So far, I must say I'm quite happy.

Since my projects all run on debian stable in production, my development needs are quite simple - a Unix-like environment and the usual array of FOSS tools. With a little bit of effort (about 4 hours, most spent just waiting for things to download), I managed to duplicate my production environment via darwin ports on OSX. The only thing that didn't match was the operating system - OSX does work a little differently from Debian linux.

This worked out decently. Writing code, especially when you're used to using emacs in a terminal, is the same everywhere. Running tests is usually the biggest problem. The problem I had with the old powerbook was that it took 90 minutes to 2 hours to run the full suite of the tests at the time, a bit too long for the write-test-commit cycle. On our server class machines (3 Ghz Xeons), these tests take 20 minute to run. Still too long (refactoring coming up shortly) but acceptable.

With the MacBook, tests take 30 minutes to run natively. Not quite a match for server class machines, but comparable to and in some cases, faster than our development desktops. If we refactored the tests to run more in parallel, results would be better as current tests only use one of the two cpus in the MacBook.

However, there is one problem with running all this on OSX - it's not exactly our production environment and we have been bitten before with slight differences between behaviour on different systems. What works fine on Gentoo crashes and burns on Debian ... etc.

In comes the Parallels virtualization product, a virtual machine solution targeted for OSX on Intel. A quick download of the software, a few clicks and 30 minutes later, I had a minimal Debian stable install running. Another 30 minutes later, I had a full development environment installed and running inside the virtual machine. The virtual machine was fast, responsive and fully functional. The full suite of tests took 45 minutes to run - not zippy, but still far better than the powerbook and usable until the great test refactoring happens.

I'm really pleased with this laptop. All my needs are more than adequately met by this system for the forseeable future. I just might need to upgrade the hard drive in a few months, as the 60G I'm starting with is a bit small. All these virtual machine images will eventually add up.

MacBook Day 1

2006-06-01T17:06:00.001+08:00

I read PlanetPython a lot, and one of the interesting things I've noticed is the number of Mac posts I've seen there over time. Various Python personalities keep getting their latest MacBook (Pro) and talking about their great experiences with them.

Well, I gave in (really did not take much convincing) and joined the crowd. Got myself the lowest end MacBook, the 1.83 GHz model, with the lowest end specs. Then immediately maxed out the RAM to 2G. I figured I could live with the 60G hard drive (I don't actually pack very much) but I might one day need to run the parallels virtualization software and more memory never hurts when you're trying to run multiple OSes at the same time.

Day one has been pretty good. It really didn't take all that long to get up and fully functional on the new system. All OSX systems include a nice utility to migrate your data over, but I lacked a firewire cable so I had to manually scp my home directory from my Powerbook over to the MacBook. After installing all the other apps I used regularly (Colloquy, Adium, Firefox, OpenOffice.org, darwin ports), everything else just worked like the old Powerbook, just a lot faster. It's nice how OSX puts almost all your user settings into your home directory, making moving everything really easy.

It's funny how the lowest end MacBook beats the crap out of an 8 month old Powerbook, but I guess that's just the power of the new Intel chips. Things are responding much faster now and I believe I can actually use this new system for serious development, rather than ssh'ing to servers to run CPU/IO intensive tasks. We'll see though. Ports is still building the dependencies needed to run my main project applications.

I suppose lots of other people have given their impressions on the MacBook, but my own impressions have all been really positive so far. The screen is much brighter and crisper than the old PB. I like the much larger trackpad and the new keyboard is decent, though it requires a little more force than the old PB. My fingers must have gotten weak.

Heat really doesn't appear to be that much of a problem. I tried running both CPUs at 100% for about 15 minutes and while it got warm, it was usable. In regular usage (darwin ports compiling stuff in one window, me writing in another), it gets a little warmer than the PB, but still usable on bare skin lap. Would be a lot nicer in winter than summer though.

I plan to use the system extensively for development, especially while traveling around Europe later this month. This laptop is going to get a much more intensive workout than the PB. After I found that the PB took up to 3 times longer than my regular desktop to run a full suite of tests on one of my projects, I stopped using it for actual development (was great as an emailing/document generating machine). We'll see how things turn out after darwin ports finishes compiling all the dependencies and I run the full suite of tests for the first time.

Full Emacs Keybindings in OSX

2006-05-16T17:05:00.001+08:00

One fun thing about reading blogs - you learn little gems that you wouldn't encounter otherwise, sometimes even when you search for it. When I got my Mac in August last year, I was thrilled to find that it supported emacs style key bindings in most Cocoa applications. However, it wasn't complete support - while all the control key bindings worked (^f, ^b, ^p, ^n, etc), the ALT/option button bindings did not. So I missed useful keys like page up (Alt-v) or forward one word (Alt-f).

I did spend a couple of hours searching on this, particularly in the help documents and knowledge base contained in the Mac and on google. No luck. I eventually resigned myself to working without the Alt keys and chugged along mostly happy. That is, until today, when I read Erica Sadun's blog on the Mac DevCenter RSS feed.

Turns out that all I really needed was this Apple Developer article on Key Bindings in OSX, including Emacs examples. It was as simple as adding my own custom definitions in ~/Library/KeyBindings/DefaultKeyBinding.dict.

The OSX key binding capability is actually quite impressive. You can even do multi-keystroke bindings, such as ^x^f (if you really miss Emacs that much).

Another day, another great functionality to rave about in OSX :)

Interview Programming Problems

2006-03-29T16:58:00.003+08:00

Another Saturday done, another interviewing round finished. Thought I would put down into words what I look for when reviewing the programming problems done by candidates. I don't really care if candidates know what I look for - if they can do it in an interview, they can do it in their daily work. Especially so when code reviewers are likely to be at least as watchful as I am in an interview.

As an illustration, I'm using a problem we previously used in our written tests. We replaced it recently because everyone answered it in almost the same way, making it useless as a differentiator between candidates. The problem is as follows:

Implement a function "intersection". The function takes two ASCII strings and returns an ASCII string containing only those characters, that are simultaneously present in both arguments. The result should be as short as possible. For example:
intersection("abde", "bexy") may return "be" or "eb"
intersection("exoweb", "candidate") should return "e"

Almost all solutions (that work) are variants of the same form. Below was the minimum acceptable code to pass first round screening in Exoweb, in a prettied up python format:

def intersection(a, b):
   intersections = ""
   for char1 in a:
       for char2 in b:
           if char1 == char2:
               intersections = intersections + char1
   return intersections

(trivia - usage of the "in" keyword is up to 3 times faster than a str.find() call on my laptop)

(trivia #2 - something like 50% of candidates who make it to the written test are unable to write even the above snippet. After this test, 90% of all candidates who have submitted their resume have been eliminated)

There are two problems with the code above, one pretty obvious (not reading the requirements) and the other a lot less so (performance problem). The first is repeated characters in the return statement and the second is an algorithm that does not run in linear time.

The first problem is easy. Given "aaabbb" and "bbbccc", the algorithm above returns "bbbbbbbbb". The problem specification says "The result should be as short as possible." Failure to read the spec or forgetting to check for this is bad, but not fatal as long as one spots this quickly.

The second problem is one that less than 1% of the candidates manage to avoid - algorithmic complexity. If strings a and b were of length n, the double for loops in the algorithm result in a O(n^2) algorithm. For every character added to the length of string n, the computer can end up doing up to n+1 times more computations. This quickly becomes impossible.

On my laptop, with a data set tweaked for the worst case scenario, I get the following execution times:

(n=1,000) 0.00372 seconds
(n=10,000) 0.29497 seconds
(n=100,000) 30.20992 seconds

For every time I increase n by an order of magnitude (*10), the execution time increases by roughly two orders of magnitude (*100). Following this progression, a value of n=1,000,000 would take around 3,000 seconds or 50 minutes!

This problem is relatively easily solved and there are multiple solutions. For those languages without rich libraries, one solution is to build a 128 char length array (the problem specifies ASCII, which is only 128 values) and to run through each string once, putting a value into the array to specify that the character was found. Once complete, it's a matter of scanning all 128 elements to see what was found in both strings. All these operations are in linear time. This has also the advantage of ensuring that the returned result has no duplicates.

For languages with richer libraries or built ins, you can also use hashed containers or even set data types. We disallow using Sets in python because it would simply be too trivial. In Python 2.4, Set data types are built in and the code would look like this:

def intersection(a, b):
   return ''.join(set(a).intersection(set(b)))

Sets in Java aren't quite so feature rich, lacking the intersection() method, so we still allow it in Java. A non-set method in Python, using just the standard built-in data types might look like this:

def intersection(a, b):
   char_seen = {}
   intersections = {}
   for char in a:
       char_seen[char] = True

   for char in b:
       if char_seen.has_key(char)
           intersections[char] = True

   return ''.join(intersections.keys())

With a n=1,000,000 string size, this takes 1.7 seconds on my laptop, much faster than the expected 50 minutes required by the inefficient, O(n^2) algorithm. With the n = 100,000 string size, the algorithm takes 0.18 seconds, the expected roughly linear decrease in time.

The algorithm above can certainly be optimized further, for different focus areas. Using two dictionaries does waste a bit of memory, and there are probably faster ways of doing this. There are probably readability tweaks too.

In our interviews, it does not matter if the code has flaws on the first try (it must work though), as long as the interviewee can understand the problem when pointed out and fix them. No one is perfect and mistakes are to be expected. We just try to minimize them and fix them as soon as possible.

Teaching Software Engineering

2006-03-18T16:50:00.003+08:00

Heh. Having spent a not insignificant proportion of the last 1.5 years doing HR work, I feel a great deal of sympathy when reading of the plight of others when doing HR. Some amusement too, as I recognize the problems and issues faced.

Today's fun article comes courtesy of planetpython.org, regarding teaching the Waterfall method in schools. I wince in sympathy because almost all of the people I interviewed, if they knew anything about development, knew only this method. Yet it is a method we (as in Exoweb) know doesn't work very well for us.

It's a nice, sunny Saturday afternoon so I'm too lazy to ruminate on why schools put too much emphasis on the Waterfall method and SEI methodologies, but I have been recently rambling to colleagues about a few complaints I had with my own college experience in software engineering:

Overly simplified
Short term projects
No Challenge

Overly Simplified

This is related to the Waterfall issue. I realize that colleges first try to teach us the basics, then try to teach us the more complex stuff. But sometimes, the basics are so overly simplified that we learn the wrong things. e.g. the Waterfall method. To me, the failure of the Waterfall method is the assumption that it is possible to get perfect requirements and that they will never change. Working life has taught me that no plan survives first contact with reality. That lesson was most painfully learned.

What is sad is that too many people I meet still stubbornly stick to what they were taught in college. I still see too many people/organizations spending months trying to gather all the requirements while competitors gain a head start by producing an imperfect but workable product. I see man-years of developer time spent haggling over little requirement details, only to find the client or market has changed requirements in the time it took for them to sort out the exact details.

Yes, requirements are important and it is the cheapest stage in the software development process to make changes. Cowboy hacking just as frequently leads to disasters. However, there is a point of diminishing returns and most people following the Waterfall process go way past this point. Agile Development offers the best middle ground that I have found to date.

So, to wrap up this section of the rant, if schools would quit simplifying stuff too much, the tragedy of the 1 year requirements gathering phase would not occur.

Short Term Projects

Almost all college projects are for the duration of a single class - a single semester of a few months in length. This means that a student typically spends an entire semester building a system that works, then forgets about it afterwards.

The problem with this approach is that, like construction, it is much easier to build a small shack than it is to build a skyscraper. If you are just slapping a few pieces of wood together to cover some random stuff in your backyard, you really aren't concerned about how good the foundation is or if the darned thing collapses a few months later. It's not that hard to rebuild it. On the other hand, screw up the foundation of a skyscraper and very horrible things happen. Like software, those screw ups become apparently very late, when the cost of changing things (or failure) is very high. Yet the one semester projects mostly teach us the habits required to build small shacks.

Challenge

There is a quote from Peopleware that I enjoy about good builders:

"The minimum that will satisfy them is more or less the best quality they have achieved in the past."

This seems to be true for myself (not that I consider myself a great builder) and for many great developers that I respect. I cannot be sure that this applies to everyone, but it seems true enough for most.

The problem is that most schools don't really hold their students up to high standards or even show them that it exists. If the "best" that they've done is code that doesn't even compile (I know quite a few professors don't even bother to check this), then they will always be satisfied producing crap because they don't know any better.

I see this in some fresh grads that I interview - they are, in theory, some of the smartest kids graduating that year from their college. They have the highest grades, they've achieved more than their peers ... they think they are the king of the world. The only problem is that compared to the truly best in the world, they are crap. They don't automatically strive to improve their code, they use suboptimal algorithms, miss various corner cases, etc.

I have had classmates that have graduated after 2 years of courses taught in C++, yet still not know what a pointer is. I have interviewed candidates who graduated with a bachelor's degree in computer science, but have never written a line of code in their life. These schools do a great disservice to our profession and society in general (i.e. think of the cost of all that crappy code out there).

I know this has been suggested before by others, but perhaps one thing that would make things better would be a minimum competency exam, administered by a certification board. Professions such as law, medicine and accounting all have professional organizations that set minimum standards and administer an exam that all practicing members of that profession must pass in order to practice being a lawyer, doctor or certified public accountant. Perhaps we are approaching a time when software developers too must meet a minimum competency before being allowed to work on things like nuclear power plant controls or medical equipment. I know I would sleep better at night knowing my pointer-incompetent classmate was not writing the code for medical equipment that would one day be used on me.

HR at Exoweb

2006-03-01T16:41:00.001+08:00

Greg and I got curious this morning about what our interviewees were writing about their experience on the web and decided to do a bit of searching. This ended up in me getting curious about a batch of HR related matters. Final result is a bunch of weird trivia:

Interview stats:

Distinct resumes received in February: 1308
Called for pre-screening interviews: 186
Passed pre-screening: 19
Job offers given: 3

Ouch. We have a huge attrition rate (0.2% get offers). Will write in more detail about the transition from stages 1-2 and 2-3 in later blogs and what typically kills a candidate.

Other fun tidbits we found from scanning bbs posts:

"Those guys must be poor! They're sharing offices with another company! Don't work there!"

Heh. When we moved to this office in 2004, Exoweb was all of 12 people, but we found this large space to renovate into a great loft. We ended up inviting 2 other companies owned by good friends (and fellow FOSS users) to join us. Since then, all of us have at least doubled in size, filling up the entire loft space and overflowing. Although it doesn't look like it, we actually take up the entire top floor of our building, except for one stubborn company that refuses to move out and give us total control of the floor.

"They have an all you can drink policy! Bunch of drunkards!"

We have an all you can drink soft drinks benefit. But I guess it doesn't help that some of the pictures of our office posted on the web have included pictures of "herb liquor tasting party" or "empty bottles after christmas party".

Circular Dependencies When Upgrading Debian Testing (Etch)

2006-02-20T22:19:00.001+08:00

With an office of 30+ users who run debian testing on their desktops, it's not a big surprise that any problems with debian testing can really come and bite us. Recently, a few developers who had been particularly slow with their upgrading hit a really bad circular dependency bug that basically stopped their upgrade cold in the water and prevented them from going any further. The bug in question is the initramfs-tools, kernel 2.6 and udev circular dependency.

The main problem is that udev requires a _running_ >= kernel 2.6.12 (soon to be >= kernel 2.6.15) to even be installed. It is not enough that you are just about to install the kernel. You must be running the latest kernel, which means the kernel must already be installed. The kernels on the other hand, depend on initramfs-tools .. which depends on udev. So udev will not install until you are running a kernel >= 2.6.12 but you cannot install those kernels unless udev is installed ... ouch.

Those who upgraded frequently enough hit that sweet spot when the latest debian kernel was 2.6.12 but did not require udev, so it could all be installed just fine. It did require a reboot after installing the kernel to install udev, as documented in the notes, but it was possible to continue. Those who took too long, or fixed their kernel to a particular version for various reasons eventually hit this bug when they did upgrade.

In the end, a few of the developers that were not quite so familiar with debian ended up reinstalling their system from scratch (debian testing install CD drops a >= 2.6.12 kernel in right away, avoiding the problem). There is a way to break this circular dependency without reinstalling though.

The 2.6.15 kernel (and possibly earlier versions as well. Did not check) does not absolutely require initramfs-tools. It is only the default option. Running dpkg -I on a kernel package shows:

Package: linux-image-2.6.15-1-k7
Version: 2.6.15-4
Section: base
Priority: optional
Architecture: i386
Depends: module-init-tools (>= 0.9.13), initramfs-tools | yaird | linux-initramfs-tool

linux-initramfs-tools is a virtual package, so useless for us there. However, yaird is also an acceptable dependency. The solution then is to install yaird first, removing initramfs-tools, then install the rest of the mess (linux-image-2.6.15-1-x, udev).

Users of kernels that are too old may still be out of luck though, as hints given in the debian bug report suggests that even yaird requires a not too old 2.6 kernel.

Ah well. It is an unstable time again in debian testing, after the relative calm while sarge was being prepared for debian stable. There are quite a few circular dependencies now and people are reporting problems upgrading. In some cases, those upgrading from rather old debian sarge systems to the latest testing report that their desktop environments have become flaky (gnome and kde both). Switching to the other desktop, or purging/reinstalling those desktops seems to fix things.

Amazingly though, KDE 3.5.1 has made it into debian testing a mere 20 days (or less, I only noticed it today) after its official release. Certainly not the slow debian days anymore.

Computer Science vs Software Engineering

2006-01-15T22:15:00.000+08:00

This article entitled Software Engineering, Not Computer Science (PDF), is probably the clearest definition of the difference between the two fields that I have seen to date. It also provides very interesting food for thought because many yearn to do computer science, yet most of us are employed doing software engineering.

In a way, it is a pity that there are not more computer science jobs available. I have encountered a few really smart people who love the discipline and would no doubt advance the field of computer science if they were given the chance. They just made absolutely horrible software engineers as they were not really interested in producing products. They were only interested in creating new things, no matter how unrelated or inapplicable to their task at hand.

The Paranoid Programmer: From Junior to Mid

2006-01-11T22:09:00.003+08:00

While chatting with a fellow developer, the question was asked: "How does one go about raising one's skills?" The answer to this sort of question is different for every person - every person has different talents and weaknesses and develops in different ways. At this moment in time, looking at the current Exoweb team, there are a few areas that I would particularly emphasize:

Paranoia
Mapper vs Packer
Quality Plateau
Knowledge Portfolio Investing

This particular entry is written mostly for Exoweb developers, but any feedback, comments or suggestions are welcome. Update 2006-01-15: Changed the title. Besides the fact that I've previously written something on what makes a senior, what I've written here will only get someone up to a mid level developer in Exoweb. There are a lot more things I left out on what makes a senior, like the soft skills.

Paranoia

Paranoia is good in a developer. Or perhaps some would prefer to refer to it as boundary checking. At any rate, it is always good for a developer to consider that Murphy's Law (anything that can go wrong, _will_ go wrong) is something we encounter far too often in our daily life. Once code is written and being inspected for improvements (you do go through your code again and see if you can improve it, right?), it helps a lot if the developer considers what can go wrong and how one can safeguard one's code against this.

As an example, one area that developers typically forget in web programming is url encoding. For instance, some insert usernames into the url as a variable. e.g. /user/john/details or /user_details?username=john. However, they forget that usernames can often include characters that are not legal in urls, such as spaces, &, ? or others. Worse, they may not even be ascii. In our global environment, it is no longer uncommon to encounter a lot of unicode characters. This of course leads to much pain later. Competent developers learn the first time they make this mistake and never repeat it. The superstars never make this mistake in the first place.

Paranoia can only go so far - you will miss something. Fortunately, that's what code reviews, pair programming, even more paranoid seniors and users^H^H^H^H^H beta testers are for - helping you catch your errors. But it helps the user experience (and your career) a lot if you catch as many of the bugs as possible before anyone else sees them.

Mapper vs Packer

The terminology comes from the Programmer's Stone, and it refers to a mindset. Are you a memorizer (pack information into your brain) or one who figures out the fundamental principles (maps connections between data points)? Packers have a tough time making senior in Exoweb because seniors are the ones that handle the most unusual, newest problems. For that, a packer has to find and pack the appropriate response. That can be rather hard to do. Instead, we need people who are adaptable to new situations and can figure out solutions to problems. Nothing is more annoying than a person constantly bombarding you with questions that could easily be answered with a little thought.

The Quality Plateau

Yet another term from the Programmer's Stone, this time in Day 2 of the website (I consider the first two days the most valuable). I wish they had a HTML tag to that particular section so I could link directly to it, instead of telling people to search for the heading. At any rate, that site shows how even code that is considered well written can be improved and made more readable. You may or may not agree with the style or the different methods used, but it is an eye opening experience - 26 lines of code reduced to 11 much more readable lines of code.

The Quality Plateau is not about reducing unnecessary variables, cramming things into as small a space as possible or holy wars about coding conventions. It is primarily about looking at your code and constantly finding ways to improve it. This mindset may seem expensive at first as you spend time looking over already functional code, but the long run benefits are enormous. Each time you see a way to improve your code, you learn something new for the next bit of code you write. Over time, you get better and better and start producing top notch code without much effort.

Knowledge Portfolio Investing

If the Quality Plateau is about constantly improving your code, then Knowledge Portfolio Investing is about improving yourself. This particular phrase comes from The Pragmatic Programmer, a book I highly recommend. As knowledge workers whose tools are only our intelligence and a computer, our greatest value lies in what is in our heads. If we do not constantly invest in increasing that asset, we will one day find ourselves penniless in our job - we will not have the value left to justify the high salaries that we believe we deserve.

It is hard to take time out to invest in ourselves, to learn something new every day. Work is tiring, our personal lives often seem more interesting and something always seems to come up. But none of us would have gotten this far if we hadn't invested time and effort in improving ourselves. No one in Exoweb has ever studied only the bare minimum required of them in school. Work should be no exception.

This is one of the reasons why Exoweb allocates 10% of work hours to employee improvement and tries to minimize overtime - to give every one of our developers time to continue developing themselves. This is a win-win situation for all parties involved as Exoweb's time and resource investment results in more competent and skilled developers. However, this all depends on the developers actually taking advantage of this time.

Final Thoughts

There are plenty of other things and more will be added over time as the situation change and other needs become more obvious. Of all of the above, the area that is ultimately most important is #4 - Knowledge Portfolio Investment. In the end, if a person is constantly trying to develop themselves, they will learn all the other areas.

The future, our industry and our targets are always moving. We can never be satisfied with hitting all the goals we set today, because by the time we hit them, needs will have changed and new goals are needed. However, if we are at least moving in the right direction, there will be a much shorter distance to travel to the new target after we hit the old one.

Public Key Missing in Apt

2006-01-08T22:07:00.002+08:00

Just a quick blog about some key wierdness in the debian testing apt-get setup. Not sure where the exact problem stems from, but since the new year started, all debian updates are signed with the 2006 gpg key, which my testing systems did not seem to have. So you would end up with this error after doing an update:

Get:1 http://box.exoweb.net testing Release.gpg [378B]
...
Fetched 2810kB in 24s (114kB/s)
Reading package lists... Done
W: GPG error: http://box.exoweb.net testing Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 010908312D230C5F

The problem being that the public key at the end is not recognized. Looking at the key management utility for apt (apt-key) didn't show any simple way for it to download the correct key from the debian keyring, so I ended up having to use a bit of a kludge. These were the commands I had to run (as root):

gpg --keyserver keyring.debian.org --recv-key 2D230C5F
gpg --armor --export 2D230C5F | apt-key add -

The first line downloads the public key and adds it to the root user's list of public keys. The command exports this from the root user's keylist to apt-key. The cleanest way to do this would probably be to use wget to get the actual key from its appropriate location, then pipe it to apt-key (it would be a one liner too). However, that is clunkier to do since one has to look up the appropriate location of the key, etc. In the end, adding just one public key to root's keyring was no real deal.

Ah well, back to your regularly scheduled hacking ...

Year End Thoughts

2005-12-31T20:00:00.001+08:00

With 4 hours left to the end of 2005, it's time to reflect on how the year has gone and prepare for the coming of the new year. Overall, it has been a good year for me, though it has felt like I've been hanging on for dear life on to a runaway train. For FOSS, it has been a great year it continues to grow by leaps and bounds.

The greatest challenge in 2005 was for Exoweb to find and integrate good people into the team as we ramped up to satisfy client demands. Exoweb grew from an 8 man outfit when I joined in 2004 to a 34 person company today, and is still growing rapidly. We would actually be larger if we had not had such a tough time hiring good people. In the last year, we have streamlined our HR process, allowing us to screen 10 times the number of candidates we could previously, with minimal impact to the daily operations of Exoweb.

Beyond just increasing bodies though, Exoweb feels so much better a place to work in now. It was a nice place before that, but in the last year, we have managed to strengthen the company culture, added a bunch of really smart people and started processes to ensure that we are constantly improving. I can honestly say that the current Exoweb team is the smartest, most competent tech team I have had the pleasure of working with. It is both a joy and a challenge to work with intelligent people who are far more knowledgeable than you in their areas of expertise. We may or may not have the stellar brain matter that Google is reputed to have, but the current team can definitely give any other team a real challenge.

2006 will bring its own challenges, no doubt. The company culture is relatively young and it will be challenged quite a bit as it tries to accommodate the changing desires, needs and eccentricities of our growing and maturing developers.

Another thing that I am proud of is that Exoweb's contributions back to FOSS projects are growing. While we mostly filed bug reports in in 2003 and before, we started contributing code to small projects in 2004 and that trend has only accelerated in 2005. Since we became active users of django, several patches have been accepted into the main trunk and we should hopefully be contributing even greater functionality soon. We have patches in various other small projects such as EaseXML (formerly XMLObject), and identified performance improvements in projects such as PostgreSQL. We recently instituted a contribute-back policy, where developers can spend up to 10% of their working hours on FOSS projects, just contributing back to the community that makes our business possible.

Incidentally, the usage of FOSS is on the rise in China, even if the community does not appear to be that visible as yet. A growing percentage of the candidates going through our HR process are listing FOSS skills and projects on their resumes. More and more companies are using FOSS technologies in their daily work. We are also getting more business inquiries specifically seeking our FOSS skill set.

Finally, every competent FOSS person I know is fully employed and in huge demand. I know, because I tried to poach every single one not working in Exoweb :). For those who kept asking, "how can you find a job with FOSS skills?" a year or two ago ... HAH! Everyone I know has options - if they were not happy where they are right now, they could find a new job so very easily.

It has been a good, busy year. I am really looking forward to 2006 - more challenges, more growth and hopefully a bit more free time to relax and really play with technology again.

What Makes a Senior?

2005-09-27T15:26:00.001+08:00

It is time for the semi-annual performance reviews again and a few questions raised during the reviews made me do some thinking. As a young, growing company, we find ourselves defining many things on the spot. Many things which may be well defined in larger, established companies, we do not find appropriate for Exoweb. One of these is the definition of what is a senior.

First, a little background - Exoweb has roughly 3 broad levels of software developers - junior, mid and senior. There are finer distinctions within each broad category, but those suffice for now. In general, what defines whether a person is either a junior or mid level is pretty clear - it is based on their ability to complete a task. Juniors normally require supervision and guidance, including being pointed in the right direction and perhaps some tutoring, in order to complete their tasks according to Exoweb standards. Mids are relatively independent, able to complete a task independently. It is defining what separates a mid from a senior that is tricky.

The problem is that our seniors have varied skill sets and roles. Although Exoweb knows very clearly who are the seniors within it, it is not always easy to articulate what makes them seniors. We do not look at education, expertise in any technology or years of work experience. There has never been a need to define the separation, perhaps because every senior that we have is clearly above and beyond the mids that there was never a need to define the role. We just knew.

However, people normally aspire to improve themselves and advance their careers. In order for Exoweb's mids to advance and join our critically important pool of seniors, it becomes more and more important to define what it is so people can work towards achieving this goal. In my mind, it comes down to one critical factor - trust.

We have to be able to trust our seniors to achieve the job. Exoweb defines its job (right now) as "delivering projects to our clients and ensuring they are happy with them." Simple enough. With that definition, the job of a senior is simple enough to define - we have to be confident that a senior, given a task and often a team, will achieve the task. That is all.

It does get a lot more complicated though, as you look at all the different aspects of delivering a project to our clients. There are multiple different aspects:

Problem solving - This is just a general catch all for anything not covered below. The ability to see problems and ensure that they are resolved is another key quality we look for in a senior. They do not have to resolve it themselves, if it is not their field of expertise. However, they do have to spot problems before they negatively impact performance and ensure that they are resolved, one way or another. If a user interface is clunky and cumbersome, either resolve the issue or find the in-house usability expert to consult on the problem.
Quality - We take great pride in our work. It hurts us when we are forced to rush out shoddy, buggy software. Even though it does happen, it should be something to be avoided where at all possible. The enforcers of excellence, when it is lacking among team members, are the seniors. They are the final, last chance to catch any defects before they hit our customers. This particular aspect of being a senior is often a thankless and tedious task. Yet someone must do it.
Technology - The senior must understand enough of the technology to oversee the team and ensure that they are not going off a steep cliff. Besides knowing what technologies work well together, they must be able to spot and stop common pitfalls, bottlenecks and problems normally produced by less experienced team members. Or in some cases, depending on where their strengths lie, the senior must know where their limits lie and be intelligent enough to call for the assistance of another senior when they are out of their depth. This happens often as no one person can know every single technology.
Project management - Someone has to manage the project and this typically falls to one of the seniors in the team. The person must be able to scope out the work, make estimates, allocate resources and monitor progress. Should any problems arise, the problems must either be resolved internally or escalated to more senior staff. Preferably the former.
Leadership - Management is more than just looking at tasks and watching numbers. It is also about motivating people. Motivating intelligent, opinionated and confident team members is no easy task. People are far more unpredictable than computers. Yet someone has to do this. Leadership is a crucial resource in ensuring that a team of people are far more effective together than spinning their wheels individually. Leadership in this area can be either or both technology leadership and team cohesiveness.

Most seniors do not have every single aspect mentioned above. Some seniors have limited technology skills. Others are hopeless at motivating fellow team members. But part of being a senior is also knowing where one's weaknesses lie and compensating for them, either by teaming up with someone who has complementary skills, or putting in processes that compensate for this shortage.

In summary, seniors give Exoweb peace of mind. It is the knowledge that when something is passed to them, it will be done without problems. Attention can then be turned to other areas, such as growing the company and ensuring that Exoweb achieves its goal of becoming the workplace of choice for everyone here.

Goodbye apt-get

2005-08-14T14:55:00.002+08:00

So it's finally time to leave apt-get, at least on my desktop machine, and go down the path of bsd and ports. I gave up on fink for my software packaging needs and switched to OpenDarwin instead. The main reason for the choice was that OpenDarwn has a larger and more updated package list, so it requires less manual downloading and package dependency resolving. The small price of learning a slightly different controlling program (port vs apt-get) was more than outweighed by the ease of typing 'sudo port install x' and just walking off and letting it do all the work. The lack of a central repository (OpenDarwin downloads the tarballs directly from the original site, not from a central repository) makes it difficult to use a repository cache, but I guess that's irrelevant since I am the only one in the office that uses OpenDarwin anyway.

Also took care of the clunky OpenOffice issue by installing NeoOfficeJ. My colleages have not had good experiences with the older versions but the latest version does not seem too bad. Fires up relatively quickly and seems stable, but I must admit that I have not used it too extensively. Will know more later.

One minor annoyance with OSX are the emacs keybindings, or the lack of two of them. Almost all the commonly used ones are here, even the obscure ones. However, there are two that I use very frequently that are not found in the text editor programs - Meta-f and Meta-b. These go forward and back one word, respectively. I have ctrl-f and ctrl-b, which move one letter at a time, but I have gotten very used to the one word movements that I seriously feel the lack without them. In terminal, all the Meta-x keys are done via option-x, and in text editors, some of this is there. option-backspace is backspace one whole char, the same as Meta-backspace in emacs. However, the option-f and option-b keys produce some strange characters, instead of replicating their Meta-x behaviours. Ah well. At least option-arrow key still moves on word at a time, but I'm too lazy to move my entire hand the few cm required to hit them :)

One other note - never use transparencies on anything you want a fast response time from. I had a slight transparency going on my terminals and was wondering why text-mode emacs was moving dog-slow. Once the transparency was shut off, it was back the good old speeds you have come to expect and love from text terminals. Even with the high-end graphics cards that all these machines come with, transparency is still very expensive on the CPU

Interviewing Thoughts

2005-08-11T14:51:00.002+08:00

Just some thoughts to trends/things that some people say when I am doing interviews. Some I strongly disagree with, others ... I am curious what other peoples thoughts and experiences are with them.

You can't keep up with the young 'uns
You have to change jobs every year, or you will never advance
If you learn on your own, without 'real' experience, it is worthless
To really become an expert, you must focus on one specific field and ignore the others
Details are not important. The compiler will catch it

1. You can't keep up with the young 'uns

This was first related to me by an applicant in her late twenties, with only a little working experience but lots of degrees. In the interview, she said that her goal was to move into project management because, "as you get older, your thought processes slow and you cannot code as fast as those young kids."

The sad part was, all the senior applicants I interviewed that day shared exactly the same viewpoint. All wanted to move out of coding and into project management. They wanted to get into management not because they enjoyed it or because it had to be done, but because they felt it was their only escape from obsolescence.

This is silly. This ignores all the experience and knowledge that older, more experienced coders have gained through painful experience. This assumes that a person's mental faculties begin deteriorating at such a young age (not true!) and that quantity of code equates to productivity. I know I certainly produced a lot more volume of code in my early programming years, but experience has taught me to produce more effective code. It takes fewer lines, has far fewer defects and is far easier to maintain later. If anything, the years have taught me that quality is far more critical than quantity. And woe to anyone that even thinks I'm close to being an old fart ...

2. You have to change jobs at least once a year, or you will never advance

This particular one really annoys me. It feels like the get-rich-quick days of the dotcom boom again. Demand for good technical people is high in China, especially so for those with FOSS skills. Because of this, opportunities abound and everyone is jumping from one job to another to get more money.

I suppose it does give a person more experience and probably a higher salary in the short run. But do you truly gain any skills when you jump around this quickly? By moving so much, you miss the big picture. Most major projects have lifecycles longer than a year.

An eye opening experience I find for most people is to be still around during the maintenance phase of a project. It is then that you realize that those fancy coding hacks you did, or that /* This will never happen */ section of code, will come back and bite you. You get to see the long-term price of taking shorts cuts and skimping on quality. It gives you an appreciation for doing things _right_, especially if you have been through two project lifecycles, where you have seen on where it went right and one where it went wrong.

3. If you learn on your own, without 'real' experience, it is worthless

One thing we always make a point to ask for is whether a person tinkers with technologies on their own, outside of what is required for work or class. More than one applicant has justified not doing so by saying, "if it was not done for work, it is not real world experience. It is worthless."

Is it? The best technical people I know are always voraciously learning. Some deliberately learn technologies far removed from their daily work, to stretch their brain to new ways of thinking. The diverse knowledge base they draw on seems to serve them well.

Yes, "real world" experience is valuable. You never truly appreciate the value of caches until you have been slashdotted, or when you max out all the CPUs on your server farm serving dynamic content. It is hard to really appreciate the value of distributed databases until you have so many transactions that the fastest hard disks in existence cannot write the transactions to disk as fast as they pour in.

However, they are not everything. There are many people with "real world" experience who still have not really learned the best practices and techniques. Unless you are really lucky, you cannot learn this unless you are constantly looking for it. The chances of you working with the best in your field are extremely limited, unless you're lucky enough to be working for someplace like google.

4. To really become an expert, you must focus on one specific field and ignore the others

Partially related to #3. I meet a lot of people with only Java experience. They know every single Java library, framework and technique there is. But nothing else. They have never touched another language, framework or system. They sneer at all others as inferior, but have never tried them.

Depth is valuable, but so is breadth. I have watched Java experts sink their systems as they tried to employ lower performance Java web caches, when squid outperforms the faster available system by an order of magnitude. I have seen people try to squeeze J2ME into embedded systems when an ncurses interface is the only thing that would have given a halfway decent response time.

As they say, when you only have a hammer, everything in the world looks like a nail. If you don't know anything else, you will always use the tools you know, even if they are not the most appropriate for the system. You miss the best of breed tools.

5. Details are not important. The compiler will catch it

Final peeve. Some applicants make careless and copious mistakes while answering their coding question. They explain this away as, "oh, the compiler will catch it when I run this on a computer."

The compiler is not your nanny. It is not your QA department. Even if you had a QA staff to back you up, the most qualified person to find errors in your code is you. There are insidious logic errors that no compiler will catch and even good code reviews and QA people may miss. The best and easiest time to catch errors is right after you have written the code, when all the variables, requirements and interfaces are fresh in your head. Spend the time to review your code and catch everything you can.

Attention to detail is really important. If you lack it, it comes back to haunt you. If you have it, your users may not be able to explain why, but they will like your work a lot more. It is like the Apple products which I rave about - the attention to detail and usability have given their products an incredible competitive advantage. Even with commodity, even sometimes inferior hardware, they can still command per unit profit margins that are the envy of other PC manufacturers.

Powerbook day 5

2005-08-10T14:43:00.000+08:00

(Originally posted 20050810)

So it has been 5 days with this nice little bit of hardware and I am liking it greatly. It occurs to me though, that my blogs do not reflect this. I am more likely to point out its shortcomings and difficulties than all the things I love about it. So I thought I'd spend this blog writing about it. I'll try to gloss over the stuff that most other people rave about and focus on what is special about it to me as a full-time developer.

Ok, the usual stuff people rave about:

Beautiful, well thought out design
It Just Works!
Consistent UI
Sleeps and recovers in less than 4 seconds
High quality hardware

The other stuff that makes me use it as my primary workstation, despite its currently lower performance:

1. Power users welcome

While Apple may have been one of the first to properly implement a GUI and has one of the slickest interfaces around, the mouse is almost totally optional! People like me who like to work primarily through the keyboard interface can actually navigate around purely with the keyboard, using the mouse very rarely. For me, this is a significant speed increase as keyboard shortcuts are far faster than using the mouse. Additionally, because the key bindings are very similar to emacs (my main development environment), everything is comfortable and familiar.

One other cool thing about emacs key bindings is the very limited hand movement. You can reach just about all the keys without moving your hand more than a centimeter from standard rest position. You do not even have to move your hands the ~10 cm required to hit the arrow keys (3-4 cm on a laptop), thereby not disrupting your typing speed. Yes, I am anal about these things. But it does make a serious productivity difference to me.

2. *nix base

While the flashy GUI protects new users from having to deal with the guts and internals of the OS, you can still pop open the hood as necessary. Fire up the terminal program and you can see which processes are the memory/cpu hogs, what crazy programs are running riot, etc. You can fire up your trusty shell scripts to perform routine tasks, run all the *nix applications which we know and love, etc. Get down to the OS level and figure out exactly why that peripheral is not working, if you can.

Granted, this is no different from any *nix environment, but it is a *nix environment with few of the warts. In many ways, it feels like a *nix desktop done _right_.

3. It Just Works!

Ok, so I ran out of other points. But this is a point I cannot emphasize enough. Almost everything works without problems, in exactly the way you would expect it to in a perfect world. One click/keypress often produces the results you are looking for. Things run effortlessly. You waste so little fighting the GUI/OS and so much time just being productive. All the little annoyances that bugged you, be they in the Linux, BSD or Windows world, seem to be magically gone here. No need to spend days or weeks tweaking/configuring it.

Ultimately, the powerbook and osx is what I consider the ultimate combination - the simplicity that Windows promised, the power and security of *nix, all combined in one very well designed package. It is a testament to what attention to detail and usability can do. The hardware, while of high quality, is commodity. Any other company can manufacture materials of comparable quality. However, no one can put it together, hardware and software, the way Apple has done with their computers or iPod.

Powerbook Day 4

2005-08-09T14:41:00.001+08:00

(Originally posted 20050809)

It is closing in to my 96th hour with the powerbook and it has been a fun experience so far. The first 48 hours was when I had to do the bulk of my learning - getting used to the "Mac way", installing applications and waiting for stuff to download. After that, it has been mostly smooth sailing.

The only issues I've encountered so far is the relative slowness of the ppc chip and OpenOffice.org clunkiness. The OpenOffice.org clunkiness is just because the standard installation uses X11 and is not well integrated with the rest of the desktop at all. Things like one click in email to open a document just do not work. It is a cumbersome multi step process. I'm currently downloading NeoOfficeJ (based on OOo) but the slow download speeds in China really takes a while. Still, not a big deal.

The more serious issue is the performance of the system for number crunching. After I had set up my development environment, I was surprised to find that CPU intensive unit tests in my projects took twice as long as they did on my old laptop. Functional tests (fire up all the servers, do black box testing using simulated browsers) took 4-5 times longer. This makes the powerbook substandard as a development station.

A simple test of this is to run code like this:

a = 1
for i in range(1,1000000):
    a = (a + i)/i

On my laptop and workstation (1.5Ghz AMD Athlon, underpowered by today's standards), this takes 0.8 seconds to run. On the default python program on OS X, this takes 1.3 seconds! Using the version of python obtained from fink, this drops to 1.1 seconds, but still far slower than the Athlon systems. This is all using python 2.3.5.

Have not looked into detail for the reasons behind this, but I suspect that it is in part because OS X is tuned to be a desktop, not a server. Responsiveness of the GUI is paramount.

At any rate, this is disappointing, but not a deal breaker for me. While it means I have to run the intensive functional tests on a separate, development machine, I can still use the powerbook for most of my daily work. I always have a workstation somewhere within ssh distance of me anyway, and sadly enough, most of my daily work does not involve CPU intensive activity anymore. Things such as responsive email, web browsing and document editing are more important.

It's Powerbook Time!

2005-08-07T14:37:00.000+08:00

(Originally posted 20050807)

I finally gave in to the dark side this saturday. Could not resist it any more. I finally broke down and ... got me a Mac. A powerbook to be exact. Retired my old x86 laptop and went down the path of OS X and the PPC (temporarily). It has been an interesting 48 hours as I tinker with my machine, getting it ready for duty as my development machine on Monday.

There are quite a few reasons for getting the powerbook. They range from the superior quality of the hardware itself, to the "It Just Works" experience that my colleagues were experiencing with their Apple laptops, to sheer lust for all the nifty features that came fully functional on a standard OS X install. All these great features and I would not be giving up the *nix environment that I have grown so comfortable on after all these years on Linux desktops.

My greatest challenges in the last 24 hours have been adjusting myself to the "Mac" way of doing things and configuring the laptop for development. I have until Monday, as there will be certainly no time for me to do any fiddling with my development system once the workday begins.

Navigating around an OS X system is relatively easy using a mouse, but for a variety of reasons (speed, looming RSI and the unwillingness to carry an extra mouse peripheral), I prefer to use the keyboard for most of my functions. The average linux (KDE/Gnome) desktop allows you to do this rather well, but with a rather different set of key bindings. The biggest thing to get used to is the Alt/Command difference. Coming from an emacs background is both a blessing and a curse. Most of the OS X apps support emacs bindings with the control key, especially cursor movement (^f, ^b, ^p, ^n, etc). Unfortunately, they do not support the Alt keys, as the equivalent of the Alt on OS X is the Command key and keys like Cmd-b and Cmd-f are already used, being Bold and Find, respectively. These are keys I use a lot in navigation (move forward one word, move back one word), so I find myself having to use a lot more keystrokes than I used to. I'm sure that I'll get used to this in time though.

After spending a few hours learning the system and adjusting it to my preferences, I realized that I would have to repartition the system. I wanted to reserve HD space to put a linux ppc distribution on it later, just in case something came along that OS X could not do. Unfortunately, repartitioning a HFS filesystem is a destructive process at present, so I was forced to wipe the system and reinstall. Fortunately, this process is relatively painless. You pop in the disk, select a few options (less than even an ubuntu install) and you can almost leave it alone till it is finish about 30 minutes later. It was the easiest system install I have ever done in my life.

After that, the next challenge was simply setting up a development environment on OS X. OS X is meant to be a consumer desktop, not a development workstation, despite its BSD/*nix roots. It is quite a bit more effort to install the necessary applications I need to develop on. Fortunately, I could draw upon the experience of several of my colleages, particularly Adam. Following his advice, I went with fink to provide most of the resources I needed. I suppose I could have tried OpenDarwin instead, but fink gave me the familiar apt-get tools from Debian.

After downloading the necessary .dmg file from the site, installation and usage of most applications was relatively easy. The major applications such as python2.3 and postgresql 7.4.x was installed with "apt-get install x". The smaller, lesser known applications were more of a pain though. Things such as psycopg, twisted and clearsilver required source installs as they were not in the repository. I even played with DarwinPorts for a while as it had most of those packages. Unfortunately, the DarwinPorts versions were newer than what I needed, so it was back to source installs.

Finally, a little under 48 hours after I got the laptop, my development station was set up, all systems were running and I was ready to develop. It has been an interesting experience so far and the laptop has been a joy to work with.

HR Issues: Interviewing, Trainees and McDonalds

2005-03-27T14:34:00.000+08:00

(Originally posted 20050327)

Just finished the regular Sunday interviewing session. An unusually good interview round, with 3 job offers going out after 10 interviews. Our average is 1 out of 10. Strangely enough, I don't feel so elated. Not sure why. Perhaps it was because the most impressive candidate on paper (all the right open source buzzwords) failed miserably at all the non-programming problems we gave him. Also tried a new tack learned from a recent Motorola acquaintance - trying to encourage interviewees and give them hints when they struggled with a question. Unfortunately, it was all the more depressing because the ones who _needed_ the hints typically couldn't do anything even when given the hints.

On the other hand, the last 2-3 days has been very good as I follow the progress of one of our trainees. Fresh out of college with no working experience, this trainee was added to one of our projects a month ago. For the longest time, relatively simple tasks took 1-2 weeks to complete. I wasn't disappointed since I expected the trainee to take quite some time to learn the technologies we used, but I was hoping that it might be a bit faster.

Like a dam bursting, that breakthrough happened 3 days ago. From 2 weeks to complete a task to finishing 2 tasks a day – I was amazed. Not too often that I get patches faster than they can be reviewed from any developer. Not only that, but the trainee was refactoring crappy code in our code base, producing high quality code I would have expected from someone with far more experience. Ok, so I did casually suggest those refactorings, but all the same, it was a very encouraging development.

That leads me to another tangent – some of our best people were underemployed before we hired them, due to that horrible 'working experience' requirement. Our best junior developer was working at McDonald's before being hired here! Too many could not find a job fresh out of school due to lack of working experience. We have reaped good benefits from hiring fresh graduates - eating the cost of training them for the first few months and reaping the rewards of hard-working developers with the right 'can-do' attitude. But if we hadn't hired them, their skills would have gone to waste. What a waste that would have been ...

Of course, we have had to eat the cost of losing some of these trainees who left for greener pastures the moment they became minimally functional. That is a challenge that every company has to face though. Hiring and retaining good talent is hard and we have only begun to build a corporate culture that is conducive to this.