What Happens When You Turn Fsync Off On Postgres

We use the PostgreSQL database extensively to handle a fairly large amount of data. Our largest single database is over 25G in size, with a fair amount of transactions going through it daily. As such, we've had to do a lot of optimization over time. One of our experiments was turning fsync off on one of our non-critical databases. In retrospect, this probably was not that great an idea ...

This database was a non-critical but fairly write intensive database. It logged a lot of information, largely in the form of inserts. Inserts in postgres can be a bit slow sometimes since a insertions tends to lock the same section of the index until the insert is complete, forcing all inserts to go in sequence. Updates are usually a lot faster if you're updating different rows since they don't all rely on the same section of the index and can often be done simultaneously.

The fsync option slows this down even further, since postgres then waits for the data to be flushed to disk successfully before continuing on with the next operation. Not a problem for low traffic databases but if you attempt to insert hundreds of transactions a second, the milliseconds spent waiting for the disk to write the data completely really hurts. fsync ensures data integrity but at the price of speed, especially in the case of unexpected power failure.

Since this was a non-critical database and losing data wasn't really a problem (we could either recreate it or live without it), we turned fsync off on this database. All went well for months, until we actually did suffer a power failure. During the busiest period possible. Good old Murphy.

At any rate, once we brought everything back up, things seemed to work as usual ... for about 30 minutes. Then we realized our servers were frequently losing connection to this particular database. Investigations revealed that the postgres processes were terminating themselves with messages like "Error: out of memory" or complaints about data inconsistency. Yep, we got our first corrupted Postgres database. The first one I've encountered in over 7 years of using this database.

I have to admit, I had very little clue on how to recover a corrupted database and each database was corrupted slightly differently. Initially it appeared only the indexes were damaged and a reindex removed most of the problems. Later we found that there was some damage to the tables themselves (took a long time to find that) and we attempted to restore through a backup. The Write Ahead Log (WAL) backup proved to be useless. Those were corrupted or inconsistent. Strangely enough, the database could still do a pg_dump, so we just dumped out all the data and reloaded it back in the database. This ultimately fixed everything.

Morale of story - don't turn fsync off unless you really know what you're doing, including how to detect database corruption and fix it. Our biggest problem was that postgres, unlike MySQL, does not scream "Table/database corruption!" immediately. It took us a while to determine what the problem was. Then again, unless you turn off fsync, it is probably something that almost never happens on postgres. I've had tons of corrupted MySQL databases. This is my first corrupted postgres database.

Three Questions ...

This is the English version of Cindy's "3 Questions" blog post, written from a different viewpoint. As a bit of background, Cindy and I are both managers in Exoweb and one of our responsibilities is to carry out weekly chats with the people in our teams. Both of us have been asked these questions before and we figured it would be of interest to Exoweb people in general. I'm writing this without having actually read her post, so it should be interesting to see how our viewpoints differ.

The questions are:

• Does Exoweb give team members a chance to switch career paths?
• Are personnel evaluations transparent?
• If an employee makes a mistake, how will the company handle things?

Does Exoweb Give Team Members a Chance to Switch Career Paths?

Yes. We definitely prefer to use existing, proven and trusted team members to fill open positions, rather than taking a risk with hiring an unknown person from the outside. As long as the person appears to be competent and capable of taking the task, we are quite willing to take a chance.

We have had a few examples of these career changes so far, mostly people switching to project management positions. Two of our admin staff have switched over to project management roles in our software development teams. The admin team handles everything in Exoweb but actual software development, so is a lot more challenging than the name suggests. After battling with unreliable suppliers, organizing events for 40-50+ people and solving the problems that crop up in daily office life, they have proven that they are more than capable of getting the job done. That actually makes them very good project managers as they have the "get things done" attitude.

We had a software developer also try his hand at this a while back. He did a pretty good job at it, but after 6 months, found that it was really not something that he wanted to do and switched back to pure technology. This is also a learning experience that we are quite happy to allow our people to have within Exoweb - the chance to try new things out and discover if a certain path is for them. If it doesn't work out, such is life. You never know until you try and the experience typically makes you appreciate your chosen role more.

Are Personnel Evaluations Transparent?

Depends. We do our best to make obvious to the team member how they are perceived, both by team leaders and fellow team members. Depending on the team lead's management style and workload, we try to schedule chats between once a week to once every couple of months. During these chats, past accomplishments, challenges and issues are discussed, along with feedback on how the team member is doing. These chats are mostly informational and problem solving - for both sides to quickly identify and resolve issues that crop up. They also usually give team members a fairly clear idea of how they are doing.

Of course, not every team lead does things in the same way and there are large variances between each team. As a general rule we try to provide feedback, both positive and negative, as soon as possible and resolve problems before the grow too big.

However, we try to keep these evaluations as private discussions between team member and team lead. In this sense, personnel evaluations are very opaque. We usually do not make it obvious to others when someone is underperforming, even if the under performer knows very clearly that he/she is in trouble. Making this kind of information public while trying to resolve the problem is rarely helpful. We do share evaluations between team leads though, both for redundancy (in case one team lead has to take over for another) and also as knowledge sharing (e.g. how to handle situation x).

If An Employee Makes a Mistake, How Will the Company Handle Things?

In one of the management books I read many years ago, I came across a great story about making mistakes. Unfortunately, I no longer recall where it comes from (if anyone recognizes, please let me know!), but it goes something like this (heavily paraphrased):

There was a young man who had just joined Mega Corp, a huge organization with many capable and talented people. Being very eager to prove himself, the young man throws himself into his tasks and pushes himself to the limits of his abilities. Unfortunately, his eagerness results in him making a huge mistake, one that ultimately ends up costing the company USD10 million dollars.

This young man then walks into his manager's office, confesses to his mistake and finally ends with, "I guess you want my resignation now?"

His manager looks at the young man with a incredulous expression and replies, "are you crazy? We just spent 10 million dollars training you! How can you leave now? Now get back out there and apply your newly gained knowledge to earn us our 10 million dollars back!"

Everyone makes mistakes. As long as one honestly did their best and learns from it, we recognize that this is only natural and move on. We work together with each other to try to minimize mistakes and compensate for each other. We spend less time figuring out who is to blame and more on how to ensure it will never happen again.

For the most part, almost all first-time mistakes are forgiven. The rare exceptions we've encountered are when someone makes a mistake so bad that trust is irrevocably broken. As a highly trust based organization, once someone proves themselves untrustworthy, it is simply not possible to continue working in Exoweb, no matter how much we like the person. Fortunately, it is very hard to make this kind of mistake. It has only happened once in my memory.

Things I Would Like to See More In My Teams

Overall, I'm working with a great bunch of people, so don't let the topic mislead you - I'm really happy with the people I work with. However, nothing's ever perfect and there's always room for improvement. The following are things that have recently made me stop and think, "I wish I saw more of that ..."

• Curiosity
• Sharing of information
• Dissatisfaction

Curiosity

There is an immense amount of information that we don't know. In the IT industry particularly, it is critically important to keep learning and improving ourselves. A little curiosity goes a long way on making this happen.

We all know that the technologies we learn today are often out of date within five years. Even within the same language, the change can be enormous over time. Java users five years ago would not recognize or be able to read some of the code being produced today. Constant learning is critical to a long term career in IT. For that matter, it's possible that the famous Chinese stereotype that people over 30 cannot code may just be because too many fail to learn and by age thirty, their skills are almost useless.

Keeping up with the information flood out there is of course a skill. There is no way to learn it all and a challenge to find those really worth our time. However, a little effort such as by keeping up with a few good magazines and blogs goes a long way.

Sharing of Information

Everyone has different strengths and experiences. We all have learned different things and it is much easier to learn from someone who has already learned something than to figure out things from scratch. Because of this, learning from each other is a quicker path to improving ourselves than struggling on our own.

Yet too few try to learn from each other. This comes in two different forms - some don't like to share. They do not talk about what they have learned, do not mention it to others, etc. They do not blog, nor do they present in ExoForums (Exoweb's weekly information sharing session) nor do they even mention it in conversation. Anything they learn stays with them.

The other problem is actually receiving this information. This is related to the curiosity issue mentioned earlier. Exoweb has a planet listing all the blogs of colleagues, yet there are a fair number of Exowebers who do not even bother to read this. If one will not learn from their colleagues, who's knowledge and experience is most directly related to their work, what chance is there that this person will learn from anywhere else?

Dissatisfaction

Believe it or not, being dissatisfied can be a good thing if channeled properly. Dissatisfaction indicates that there is something that can be improved. By seizing upon this dissatisfaction and making the effort to ensure that improvement happens, one makes one's environment and self better.

It is too easy to notice a problem and shrug it off, accepting that it is a problem that one has to live with. Quite often this is not the case. If we invest a little of our time each day in removing the most irritating, most unpleasant annoyances in our lives, we find that bit by bit, our lives get better and better. Time and resources are limited so we can never fix everything that bothers us. However, as long as we take the time and effort to fix things, each day gets more and more enjoyable. Ultimately life is short. Why make it miserable too?

Most cultures with a strong work ethic believe in enduring hard, boring work to succeed. However, there's nothing in that ethic that says we can't find ways of doing things better. Just because it is hard or boring doesn't mean it always has to be that way. Sometimes it is far too costly to change. Sometimes it is not. Until we have aggressively investigated the problem, we will never know for sure.

Spammers And Gated Communities On The Web

YAR - Yet Another Rant ...

From my experiences living in various locations around the world, I have always loved the locations where there are very few gated communities. You can enter the different areas very freely - there are few security checkpoints, very few guards and movement is very unrestricted. These are usually the very safe areas and there's a lot of interaction between people in the different areas. These are usually friendlier, livelier and richer communities, especially with inter-community mingling.

On the other hand, the areas with lots of tightly gated communities bother me. This usually implies that the area is not very safe and you need a variety of fences and security personnel to keep out undesirable and potentially dangerous elements. These communities live in fear and are very closed. While it is peaceful inside these communities, it is still a prison of sorts.

Lately, I have begun feeling that the web is turning more and more into a group of gated communities. Mind you, it's not just linked-chain fences and rent-a-cops. It feels like many areas are starting to turn into fortresses - high walls, constant patrolling of the boundaries by guards with very deadly weapons and constant challenges while you wander through. The problem - spammers.

It used to be that spammers only sent email to your mailbox. Today, they find anything and everything they can get to make you click on their links. Within days of IOSN turning on trackbacks, many of the blog posts were filled with spam trackbacks that were just trying to sell their latest crap. Comment spam became common a couple of years ago, which is why most sites don't allow anonymous comments.

A few months ago, Exoweb set up a test server for the Beijing Linux User's Group. This was a default content management software setup (I believe it was plone) so anyone could register anonymously. It was not publicized and only a small handful of people should have known about it. Next thing we knew, some virus-writer had (obviously automatically) created a ton of accounts and hosted a huge number of trojan binaries or spam pages on the site and spammed people to click onto it. We weren't the only ones hit, but it was quite frustrating when we found out and had to shut down all logins. Till today, you can probably find some remnants of it if you google for Exoweb. I don't even want to get started on how frequently our client sites are probed by bots daily.

These low-lives who send their bots out looking for unsecured sites are the equivalent of thugs roaming your streets, trying every single door they come across. How safe would you feel if you saw criminals brazenly looking for unlocked doors in broad daylight in your neighbourhood? Because of this, people are trying to reduce access to their communities/sites, making sure only verified people are allowed to participate.

One side effect of this is that it gets more and more inconvenient to join communities or participate. Most sites will not allow participation without registration. Registration involves more hoops to jump through (captchas, email confirmations, etc). Gone are the days when you could just swing by a friend's blog and just drop a comment anonymously. Participation is now deliberate - you must really want to say something and have the patience to jump through hoops to make it. Gone are the days when you could just fire off an off-the-cuff post or comment.

This keeps people in small little gated communities. Once they have created accounts in a particular site, they are less likely to bother to register in other sites. Communities get more insular with less intermingling. Diversity, cross-pollination and knowledge is reduced ...

All this, because some people have found it perfectly acceptable to violate all netiquette for personal gain and everyone else lacks the will to do something about it. Is is sad. How quickly the promise of the internet - incredibly convenient and cheap communication and access to information - is sullied and tainted by greedy human nature.

Tallinn Thoughts

Yet another quickie post with some random thoughts about Tallinn, Estonia. Hopefully useful for my colleagues coming over shortly and anyone else who might be visiting soon. I won't touch anything that should be easily accessible from a tourism website.

First, a couple of thoughts about traveling here - the no-liquids rule and small planes on the final hop here really make it much easier to check in luggage rather than relying on carry-on. Now that flying in EU has the same restrictions on carrying liquids as flying in the US, you pretty much cannot carry any creams, gels or even toothpaste. If you try to stick to carry-on luggage to avoid baggage claim the way I do, you will have to buy everything again when you hit Tallinn. Also, every time I have flown to Tallinn (three times on two different airlines) the final hop uses a very small turbo-propeller driven plane of Russian construction. The problem with these planes is that the overhead compartment is tiny and cannot hold much more than a backpack or a standard briefcase. None of those large carry-ons.

Flying FinAir from China stops in Helsinki but you can stay in the international section of the airport. No EU (Schengen) visa required. Flying Air China requires changing planes/airlines so you will need an EU visa. Much more convenient to fly FinAir even if the EU visa isn't a hassle.

Tallinn appears to be pretty safe. There are people walking alone late at night, so violent crime doesn't appear to be a problem for most folk. The city is small enough that you can walk to most places. Bring a comfortable pair of shoes. It also has a lot of beautiful old europe architecture so bring a camera if you are into snapping pictures.

Lots of decent western food places here, but other than Indian and Japanese, not a lot of Asian food. Prices are about double the price for western food in both Malaysia or Beijing and many times the price of a standard meal in either country. Still quite cheap for an european country though.

A very nice place overall. I could be convinced to come here a lot more often ...

The Writing On The Wall ...

Just a quickie on a thought that struck me while responding to a friend's email - too many students are not aware of the quiet FOSS wave that is happening under the surface, particularly here in China. They are still focusing their learning efforts on Java, C# and Windows without a thought to picking up basic FOSS skills.

Yet if you pay attention, you can see the trend. From today's news, HP made $25 million in sales directly related to Debian GNU/Linux. This is just one Linux distribution mind you, and one with no commercial company or advertising budget behind it. Exoweb is slightly responsible for that as we (and our clients) bought more than EUR 15k of servers from them last year. :)

In our daily work integrating with third party providers, we are seeing more and more FOSS based products. Not all are fully FOSS based - many are WIMP (Windows, IIS, MySQL and PHP). This FOSS trend is especially noticeable from startup companies or companies with limited legacy infrastructure. We love this of course, since they are typically easier for us to integrate, host and maintain. Yes, we (or our clients) do pay a fair amount for these products so they are making a lot of money while being FOSS based.

My point? For anyone who really wants to be in the tech industry, FOSS skills, no matter how basic, will give you a valuable advantage. Otherwise you will be locking yourself out of a rapidly growing portion of the industry. Besides, as a craftsman, wouldn't you want to know every tool available? Otherwise how can you use the right tool for the job?

The Three Aspects of Management

Found myself explaining for the third time in a day yesterday about the three different aspects of management that I deal with in my work in Exoweb. Thought it best to blog about them, to put down in writing what I have had to repeat multiple times last week.

These are my personal opinions/divisions of course and have no grounding in scientific research. They are merely based on personal experience. However, all managers in Exoweb perform at least one aspect and some can do all three.

In my daily work, the three aspects I end up spending my management time on are:

1. Tech Management
2. Project Management
3. Human Resource Management

Technical Management

This deals with the actual work that needs doing and varies by department. It is difficult, if not impossible, to do this type of management without solid knowledge and experience in doing this task yourself. e.g. An Accounting manager must understand and be able to balance his own books before attempting to supervise a team. Likewise a senior software designer/architect needs to solidly understand software (object oriented or otherwise) design patterns and the abilities/limitations of the technologies used to be effective.

This aspect of management typically involves setting and maintaining general technical quality in a team. Properly done, this also involves mentoring and training to ensure that team members are constantly improving and requiring less and less management. This is not that frequently done in practice though.

This aspect appears to require the least social skills of the three.

Project Management

To me this has two separate components - client management and juggling tasks and resources to complete the project in time. In theory, they could be handled by different people but in practice this is so inefficient that it is not practical.

Client management requires the most communication skills of the two as it involves managing client expectations, negotiating resources and dealing with the inevitable problems that arise. Strong social skills are especially useful with the class of clients that requires you to wine and dine them.

The second component requires mostly a logical mind to figure out how to achieve the maximum value for the client given available resources. This requires not just understanding client priorities but also the best way to utilize the team to achieve these priorities. e.g. Can the manager arrange the tasks so that dependencies between them are minimized and the team functions like a well oiled machine? Or will tasks clash, resulting in periods of inactivity followed by frantic crunch periods? What are the deadlines coming up and can the team make them?

Some understanding of how the work is done is useful but not to the depth of the tech manager. We have a few project managers that lack a technical background but are still quite capable of carrying out this component of management. Strangely enough, gamers who enjoy strategy or RPG games often do well at this. Ultimately, they involve the same skills - maximizing results by carefully allocating scarce resources.

Human Resource Management

This is often the most neglected aspect of management - the understanding, combination, development and motivation of team members. It also cannot be delegated to the HR department. Both First Break All The Rules and Peopleware are full of studies that show it is the first line manager that has the largest impact on employee morale, productivity and development. All the HR processes and motivational posters in the world will achieve nothing if the immediate manager of a team member sucks at this.

It is easy to neglect this because it this type of management is a long term investment and the symptoms of neglect take so long to show up. Morale is rarely measured and there are many possible reasons for why productivity is suffering. It is easy to blame/focus on something else, especially when there are so many other tasks that obviously need doing. When clients are screaming about late projects, it is very hard to invest in long-term training, even when it is clear that the lack of certain skills among team members is the reason why projects are late.

In my career, the best managers I remember - the ones I was happiest and most productive working for - were the ones that knew exactly how to motivate the team. They may have been merely competent at the other aspects (in some cases minimally competent) but they were far superior to their peers in understanding what made their people tick. They knew how to combine people together so their strengths built on each other and their shortcomings were compensated for by other team members or processes. Working for them was a joy and it was amazing what a gelled team could achieve. It could easily multiply the productivity of a team.

Doing this is hard, especially for the stereotypically anti-social techie. It requires more than just social skills. The manager also needs to really understand the unique strengths and characteristics of each team member. Many company processes/managers treat team members as interchangeable machines that perform the same as each other, rather than the unique individuals they are, with individual needs. They are forced to do this either because they cannot recognize the individuality of their people or are unable to find managers who can manage in this way. This results in work processes and practices designed for the lowest common denominator - meant to protect us from our weaknesses. How much more enriching would it be to work in a place that recognized and allowed us to utilize our strengths and achieve our fullest potential?

Final Thoughts

There are extremely few people who can do all three roles well. Even finding people who can do two (e.g. Tech and Project Management) roles very well is very hard. We are experimenting within Exoweb with having specialists in each of the three roles. Although not nearly as efficient as having all the roles combined in one person (communication overhead is much higher) it has been proven to work with the first two roles. The jury is still out on the third role - we don't know for sure if it can be done by someone who isn't already doing at least one of the other roles. It is also such a sensitive function that we are reluctant to experiment too much with it.

There are so many different ways to manage and we're only beginning to figure out what works for Exoweb's culture and people. Ultimately though, I would stick to the philosophy best summarized in this quote: "that's what management is - it's not about how much you can do yourself. It's about how much you can help others do"

2006 Year End Thoughts

Can't believe another year has come and gone. It seems like only yesterday that I wrote the 2005 Year End Thoughts, and only a few days ago that I came back to China and started my career in Exoweb. Time really does fly when you are having fun, and I certainly have been having a great deal of fun in the last few years. Sometimes stressful and sleepless fun, but fun all the same :)

General summary of thoughts:

Things that I am happy about:

• Exoweb
  
  • Teams improving, growing
  • Culture
• Personal
  
  • Facing the different challenges over the last year
  • Learning to delegate

Things that need to be improved:

• Exoweb
  
  • Giving back to FOSS
  • Creating a continuous learning culture


• Personal
  
  • Working with people

Goals for 2007:

• Make self (almost) redundant
• Teach, not do

Things that I am happy about - Exoweb

Teams improving and growing - This last year has really seen a dramatic growth in the quality of the people within Exoweb. While continuously improving our HR process has helped us get some really good people, the major change has been the steady improvement of people already within Exoweb. There are various reasons for this but ultimately, I believe it is because we have finally passed a critical mass of smart people and everyone is now learning from each other. Processes/practices such as code reviews, ExoForums and blogging have helped encourage the sharing of information and it appears to be self-perpetuating. People are demanding high standards from each other and everyone brings in their own knowledge and skills. My primary concern is no longer raising standards but knowing when to step aside and let people more knowledgable than me figure things out.

Culture - building a good company culture isn't easy and I'll be one of the first to admit that I haven't really got the faintest clue how to go about it. Culture building is all about people and people skills aren't my greatest strength. Perhaps it is because I'm so clueless about culture building that I'm amazed that Exoweb has somehow produced (possibly by accident) a relaxed, open culture that many people find really attractive and unique. It's not perfect by a long shot and there's still so much more we can improve. Yet it is quite gratifying to hear people tell me that they find the culture of Exoweb one of the key selling points of the company and that they have never worked in a better place. Not everyone feels that way of course, but enough do. Personally, I've never worked with a better bunch of people, in a more comfortable environment. I am naturally biased though.

Things that I am happy about - Personal

Facing and Surviving the Challenges of the 2006 - 2004 and 2005 had different, lower level challenges for me. I joined Exoweb in 2004 as a senior team lead, so my main concern then was ensuring the success of a project with a small (5 devs total) team. The challenges then were a lot simpler! 2005 brought the challenge of maintaining quality in a team that was growing too large for me to personally review all code. 2006 was quite different - Exoweb continued to grow and the challenges that came up daily kept changing. The early part of 2006 was a battle to figure out how to scale technically. Or rather, how to ensure that the things I used to do still were being done when it was becoming obvious that one person could not possibly do it all.

The technical aspect of that problem was solved in early to mid 2006, among other things by the growing abilities of the team and instituting a more scalable version of code reviews (we created a cool trac plugin for this). This had the most fortuitous side effect of promoting learning even more from each other. With a great team that is continuously learning, most of my earlier technical challenges faded away.

Late 2006 was more an issue of scaling Project and Human Resources (HR) management responsibilities. The funny part was that as my team got better (and larger), the seniors within the team came to the conclusion that they did not want to do either PM or HR work, pushing all that to me. As a result, the workload in this aspect grew a lot faster than the team did in 2006. Surviving the PM aspect was done by the fine art of delegation (more about that later) while the HR aspect is still a serious work in progress.

Ultimately, I'm happy that I survived all the challenges of 2006. Ken in 2005, looking at these challenges, would have been quite intimidated. It was a good idea that I went into 2006 blind to the challenges ahead :). Looking back, I can certainly see many areas I could have done things far better but at least I can say that I haven't made an absolute mess of things.

Learning to Delegate - Every book about management talks about how one needs to delegate. Yet they tend to gloss over how to delegate. One thing I quickly learned long ago was that if you just pass a task to someone and hope that they will do it right, 90% of the time things turn out badly. It took some experimentation and trial and error, but in the end, my great lesson in 2006 was realizing that it all boiled down to figuring out who I could delegate what to. Everyone is different, with different strengths and weaknesses. The challenge was to find someone (or combination of people) that had the right strengths to do the task on hand. Not a skill I am strong at (more later about this), so it was harder than it should have been. But as I write this today, I have a good team that functions very well together, with most of the critical tasks covered and working well together.

Yes, I realize this may seem blindingly obvious to some people but it wasn't that obvious to me.

Things that need to be improved - Exoweb

Giving back to FOSS - Despite being strong believers and users of Free/Open Source Software, we don't contribute back nearly enough. Sure, some of us personally have done some work in FOSS advocacy or have code contributions here and there. However, as a company, I am still quite dissatisfied with what we have contributed back. Besides contributing server space (python.cn, Beijing LUG, etc), software usage, bug reports and a few patches here and there, we have given very little back to the ecosystem that makes our business model possible. Despite having people who profess to genuinely believe in FOSS, despite having a contribute back policy and allocating a percentage of developers time to such activities, too little is contributed back. This is something that we will have to focus more on in 2007.

Creating a continuous learning culture - Possibly because most of the seniors of Exoweb possess either the Learner or Input talents (see First Break All The Rules), we tend to expect that everyone will be like us - given the opportunity, will always try to learn and improve themselves. Unfortunately, that is not really the case. Some really talented software developers don't seek out knowledge for the sake of learning but are satisfied with learning only what is required to complete their task. Or despite the best intentions, they need a little external pressure. So despite having a 10% time self-improvement/contribute back policy in Exoweb, too many people do not take advantage of this. Yet continuous learning is a vital aspect of continuously improving the abilities of the organization.

Things that need to be improved - Personal

Working with People - I've touched on this previously, but I'm basically much better at computers and hardware than people. To me, computers seem so predictable - given a fixed input, they typically produce a fixed output. Humans are so much more variable, with too many factors to consider. Yet management is about people, not about computers. According to the Peter Principle, I'm quickly rising to my level of incompetence :)

However, the level of understanding of people I'm looking for might be a bit higher than most. The ability to figure out a person's strengths and weaknesses and combine them with other people/processes that complement their strengths and compensate for their weaknesses is a very rare talent. If you look at most management practices today, they are built to solve this talent shortage. Most large organizations have processes that cater to the lowest common denominator - they allow the organization to survive mistakes made by less competent people, but they get in the way of the truly talented hitting their full potential. We sometimes call that bureaucracy.

Most managers are either poor at or unwilling to manage individuals. It is hard to manage individuals - you have to really understand your team and know how to combine them to achieve maximum results. Most prefer to assume that every human being is interchangeable, that one person can be swapped for another without problems. This only works if you are using people at the lowest of their abilities, so that the job can be done by almost anyone. It does not work when you are trying to make the most out of everyone's unique combination of strengths and talents.

This will probably be my primary challenge in 2007 - to become competent at managing individuals.

Goals for 2007

Make myself (almost) redundant - I have delegated a large amount of my work to others already. I hope to finish this in 2007. Might be a while before I can delegate all the HR management aspects, but all the technical and project management aspects should be possible within 2007. I certainly hope to organize things so that I can go on a month long vacation and no one would notice :)

Teach, not do - One thing that is really hard for me - delegating tasks to someone else instead of just rolling up my sleeves and getting it done in a couple of hours. Only problem is that there are only so few hours in a day and so many problems that need to get resolved. Making myself redundant requires that I restrain myself from digging into problems and instead focus on teaching/guiding others to take over from me. Teaching is a large investment of time - it is always faster to do it yourself than to teach. But without this investment, the organization will never scale.

2007 looks like it will be bringing quite some challenges, many in areas where my strengths do not lie. Still, I wouldn't have it any other way. Life isn't fun without challenges and I can't think of a better team of people to face the unknown with than the crew of Exoweb. Happy New Year everyone!

Time To Turn In My Geek License

Apparently, I'm spending too much time doing management and turning into a PHB. No one seems to think I'm technically competent anymore. Some recent conversations:

While training a junior project manager:

Me: "So remember, with new tickets, ask a technical senior to help you estimate how many hours are required to complete the task ..."

A few days later, during the usual morning meeting discussing what tickets to create, prioritize, etc:

Me: "... so I think we should put this task in Milestone x, priority critical, estimated time 8 hours"

Jr: "Ok, sounds good. I'll go get an estimate from senior x and take care of it."

Me: "Wha ... didn't I just give you an estimate?"

Jr: "Yes, but you said I needed an estimate from a senior ..."

While showing a relatively new developer the rankings of everyone in Exoweb:

Me: "... and here we have the mid-levels devs, split in 3 sub levels. Finally, these are the seniors ..."

Dev: "Wait, you're considered a senior?"

After relating the above two incidents to yet another developer:

Me: "... so apparently they don't think I'm a senior anymore ..."

Dev: "Well, if you don't tell them, it's not obvious ..."

If anyone wants me, I'll be out getting a haircut, buying some suits and gaining a lot of weight ...

KISS (or why MS CS students have a bad time in interviews ...)

It's that time of the year when Master's students are hunting for jobs in China and we are flooded with resumes from students with Masters of Science in Computer Science (MScCS). We've spent the last couple of weekends interviewing the candidates that passed the front interviews and it has not been pretty. In fact, it has been pretty sad.

The biggest problem we encounter with MScCS grads is made very apparent in how they approach one of the our typical programming problems. This particular problem is fairly simple and with a little bit of thought, can be made into a linear (computer performs x more calculations for every element added to n, where x is a constant number) or O(n) type of equation. Most competent people will come up with a O(n^2) algorithm (computer performs n operations for every additional element added to n, resulting in rapidly increasing number of calculations per element added) to it at first, then after a little thought, realize that there are a lot of duplicated operations, refactor that out and come up with the linear solution.

MScCS are a little different - almost all the ones we have interviewed to date encountered this problem, probably came up with the O(n^2) equation ... then went off the deep end. They would inevitably come up with complex, fancy algorithms that utterly failed to solve the problem. These fancy algorithms would often handle the common case but fail on the boundary cases. They were fragile, easily tricked or prone to failure. When these flaws were pointed out, these candidates would come up with even more complex algorithms or add a lot more if/else checks ... resulting in even more brittle and unreliable code. None of them, if unaided, could come up with the ideal solution.

Well, that's not really true. One particular candidate, after coming up with 4 complex, unworkable algorithms, finally said, "well, since you have given me such a tight time limit, I have no choice but to brute force it." He then proceeded to give the ideal solution. Linear time, handles all boundary cases. But he would rather give 4 non-working solutions rather than the working, "inelegant" solution.

What I find strange and rather disturbing about these interviews is that it is somehow related to the mindset of those who are doing their MScCS. Bachelor level grads or people with several years of working experience rarely make this mistake. They either do it right or not at all. Yet for some strange reason, the MScCS students seem to value fancy algorithms over _working_ algorithms.

Maybe it's the MScCS curriculum here in China which tries to focus on fancy algorithms. Maybe it's just that the MScCS feel the need to prove that their abilities are above the norm for CS graduates. Maybe we just have incredibly bad luck with our candidates. Whatever the reason, extremely few MScCS fresh grads have passed our interviews. As you can imagine, in a production environment, we highly value working code and eschew the fancy algorithms, especially algorithms that needlessly complex. Simplicity and correctness is far more important in our craft.

The famous quote attributed to Brian Kernigham comes to mind:

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"

When writing algorithms, another quote from Agile development comes to mind - "do the simplest thing that can possibly work." Keep It Simple ... er, let's call it Keep It Stupidly Simple (KISS) and avoid insulting anyone :)