The die has been cast, many strokes have fallen on the keyboard and the moment of truth is here. I’m abandoning ship here and moving to my own domain at Fatalerror.

It has been wonderful being here at WordPress.com, but I needed more bells and whistles (including advertisements) than what the set up would allow me to.

So long Automattic, and thanks for all the good times!

To understand why Google does things differently, you need to first understand how goes Google work differently as a company.

At its core, Google is one massive computing infrastructure. What the company excels is in building, and maintaining applications on top of this infrastructure, only parts of which are known to us as Big Table, Google File System and Map Reduce. Almost every application (yes, this is speculation, sue me) is built atop this infrastructure, giving Google the ability to have consistency across storage, classification, categorization of any data that comes into its system. Other companies, like Yahoo! and Microsoft, have years and years of legacy sitting on different frameworks and infrastructure, giving Google amazing leverage over them.

For Google, the only real product is user experience and the value the user derives from using Google’s products. This, in turn, helps further refine and better offerings across the plate for Google, creating an endlessly iterative and self-improving product ecosystem. And the products by themselves are a means to bettering the end-product of user experience.

For instance, not many would have much to say about Google’s “web history”, but not many know that the same is used to do drive recommendations in Google Reader, which also uses geographical data (I was recommended feeds related to Trivandrum after being there for a week) which Google collects in conjunction with ISPs (driven by Google Analytics) to further refine these recommendations.

In a similar manner, Google already tracks the clicks that originate from Gmail and I would not be surprised if they are already tracking and indexing the thousands of billions of messages that flow across Google Talk to better know and predict which link you are likely to click more on Tuesdays and Wednesdays (match data from the messages and your web history), compared to Friday or Sunday. And that is very much in line with their mission statement of being more useful to you, in a manner that borders on the eerie quite a few times.

And that is where the greatest challenge lies for companies that aim to compete with Google. Learning systems that improve itself iteratively with time and usage are hardest to beat, because it improves by using you against yourself (something like going against your best time in a racing game than against a pre-programmed computer run) and since Google has been around for such a long time, the amount of data it has about you is something that the competition can’t match unless a vast majority of Google’s users switch overnight to the competing services.

Which brings us back to the non-search problem. Google really does not need to be the number one in other areas (other than the silly acquisitions like Jaiku). It does not cost Google much to create new products (many Google projects like Reader and News were started as 20% time projects) and it does not cost them anything to run those either (they are written with the same framework that is maintained for their core offerings). So, even if all of them were to fail, it would not make a dent on Google, while the fact is that a lot of them don’t.

Now, add Google App Engine to the mix, which opens up the same infrastructure (leveraging the same Google Accounts identity system) to the wider web. With the App Engine, for the tiny cost of supporting the bootstrap process for free, Google now gets even more focussed and specific data regarding usage(in the hierarchy of usage quality, context is king. Apps would have a context that is locked-down taking out the guesswork for Google and the data that is stored in such contexts would also be in a format that Google natively understands).

It would really be stupid to assume that all these processes and data collection is not already being  used to improve the advertising business, which is from where they earn their bread.

p.s: This post has been edited for clarity and a couple of grammatical snafus from its first version.

It is a huge boost for Nginx, which has in any case been growing at a rapid pace in terms of adoption in the recent years, especially as a reverse proxying solutin for the Ruby On Rails crowd. What is quite interesting is that WordPress.com is running the development version of the software (0.6.29) than the stable one (0.5.35). There is, though, no clarity if Nginx is being used purely as a reverse proxying solution for WordPress.com, or if it is actually serving PHP too though the FCGI route.

According to Netcraft, the switchover was made on 11th of April.

1. As the newspaper industry in India is a growing one, do you think there is a need for integration, as in the West it is mostly done to prevent the death of newspapers.

I am not quite sure what exactly does “integration” stand for here. If it means cross-publication integration, as seen in the case of Times of India and Hindustan Times, there is an absolute need for it. Granted, that traditional media is still growing at a rapid pace in India, mainly because we are still a fair bit away from market saturation (more readers/viewers being added to the group and a growing number of English-speaking and newly literate junta) and also because we are about five to seven years behind the curve in the internet penetration game in relation to the West. But it is a given that the scenario won’t be the same forever.

The closer you get to such a situation, the more pressure publications and their managements would feel to increase efficiencies and cut costs. And one of the ways to get something like that done is to integrate across organizations, especially in emerging market segments and niches that don’t justify the cost or the expense of two or more players going at each other just because the other one is getting into that market. I also think we would soon see publications pooling resources on commonplace reporting and coverage, like press conferences and industry events, to cut out the agencies in the longer run.

Within organizations, there is already an increased thrust on integration within themselves. For instance, one of the interesting things we had done with CNN-IBN was to have a deeply-integrated online operation with the channel and while we have been successful, arguably, more than anyone else, there is still a huge distance to go for us. For newspapers, the primary issue is technology, followed by mindsets. It is easier for a television channel to invest in a decent online publishing system than it is for a print publication. There are decent automated publishing solutions available for print publications to push out online content without much additional effort from the existing teams, but they cost a hell of a lot of cash and often involves upgrades to existing print software, thus scaring off a lot of cost-conscious organizations.

In any case, being online, by itself cannot save the print organizations. There are decent revenues coming in from the online operations, but not enough by any means to cover the existing cost structures of print media. There are also issues with scaling. After a certain point, it costs a whole lot more of money to support huge levels of traffic on these websites and the support personnel to keep them going. Thus, the need to integrate is very much there, but how it should be done is a question to which the answer is not clear yet.

2. You stated that HT as an organisation is one of the least change-friendly places in Indian media. Do you think integration in HT  will be successful?

HT has a reputation for being a very “babu” joint. But it is a problem that is not limited just to HT, it is probably visible to a much greater extent there than anywhere else. But they are trying to turn a new leaf, with their new initiatives, though the degree of success they have achieved is something that is not very visible right now. Given enough incentive, initiative and clarity, almost anyone can be successful, but most organisations don’t back up the intent with the seriousness, dedication or resources (financial, infrastructural and human) that is required to pull something like that off.

3. Will integration result in downsizing of employees who are not competent in handling multi-media?

If you ask me, I come from the school of thought that skills are portable and that tools are just things that you use to exercise and leverage those skills. Any good professional will need to be adequately tooled to adapt to his/her working environment, if the skills are to be used right and compensated for in the right manner. A god sub-edtior is a good sub-editor, irrespective of whether he/she is in a print, television or an online environment.

So, the whole issue of competency is a non-starter if you don’t hold on to the thought that the tool is much more important the skill that it brings out.

Those who don’t adapt will lose out.

4. Is integration an attempt to increase the paper’s readership and retain the brand image of the paper, as today, many get their news from the television and internet?

Most of the integration are attempts that fall under the “let us do something!” domain. Most attempts don’t have a clear cut idea of what is the eventual objective than an almost unquestioned worshipping of buzz words and catch phrases. But, I guess, the more important part is that things are being attempted right now, than being totally ignored. A large part of those attempts are driven by desperation, seeing the events elsewhere and trying to keep the wolf out of the door at a future date. The others are business-led initiatives, where companies want to maximise the revenue from the content they have already produced and published. There are very few editorially driven initiatives in the domain.

I do not think there is a reverse flow of audiences from online to newspapers purely as a derivative of their online presence. One of the fallacies of cross-media promotions is that there is a direct correlation between sustained audience growth as a result of context switching (television, print and online being the contexts). Yes, it does have an impact in terms of branding, marketing and recall, but I have not seen anything yet that drives sustained growth in usage.

5. What is the future of print media?

That is a tough one to answer and the answer it itself probably lies in the fact that kids who will be born 10 years from now won’t have a primary instinct to write on or read anything from a piece of paper. Their primary instincts would be driven towards handheld devices or to a keyboard on a computing device. The current usage of print media is sustained by generations who have the primary instinct to write on and read off a paper. Maybe, some twenty or thirty years from now, that generation won’t be the ones who hold the purse strings in terms of spending. From that point of view, the future is very bleak for print media.

That said, print also has been one of the oldest forms that have been around. It has survived radio and television till date and I don’t see why it would not survive the onslaught of the internet. The internet is more of a tectonic change that has consequences much beyond the little sphere of media, so it would be unnatural to assume that print would be left untouched by it. I think what will save print would eventually be technology. It may well not survive as the print we know of now — of being printed on deadwood — and probably move into a form that is only similar in shape, but in the end it may just surprise us all by surviving and doing well 40 years from now.

It is quite a long article, but is well worth the time you spend on it, if numbers, markets, arbitrage, learning (artificial and natural), behaviour and systems design are things which make you salivate more than blondes and brunettes. In a somewhat-related topic we have a fascinating entry on search algorithms that also mentions Pareto in the same breath. Most of the math in it flies like a supersonic above my head (confession time, I absolutely suck at math, go figure!), but do stick with him till the point where explains why there is no “best” search algorithm.

On to Google App Engine (finally!). If Google can bless this with the levels of reliability that they are known for, it will have the same effect that Ruby on Rails has had on start ups, by making bootstrapping of products so easy that it becomes absolutely irritating. David Recordon believes that the App Engine will provide apps that use it with a shared sense of a user, which is one of the major problems that face every socially-enabled product these days. Krzysztof Kowalczyk adds to the existing commentary and says it is the first, true internet operating system. But I do wonder about one thing. Everyone is very bullish on EC2 and GAE from the entry barrier point of view, the thing that remains to be seen is the exit barrier and how difficult it would be to leave such a framework, both in terms of cost and effort.

The last word on the GAE launch has to go to Michael Arrington, who can’t ever be expected to sit out a slugfest, especially one that draws traffic to Techcrunch on what is not really its strong ground – technology. He makes a post on the website about how Google has pulled down one of the first apps, HuddleChat, built by one of the Google employees showcasing the technology from a product perspective. He calls it “censorship” and the easily-inflamed community sets itself alight (rather predictably) over it, while the simple reason behind the move has no more logic behind it than avoidance of bad PR karma, which any company would want to avoid during such a major product launch. The move, by itself, does not make or break the world. Get over it (and yourselves, too) guys.

Meanwhile, all is not well in DTH land in India. The two leading players — Dish TV and Tata Sky — are said to be raking in losses to the tune of Rs 1400 crores (combined) in their quest to do a market land grab first and aim for profitability later. The current cost per user is Rs 1600 – Rs 2300 for each new subscription and the newer  MPEG 4 set top boxes that will hit the market soon are expected to increase the costs and losses even more.

Interestingly, Dish TV seems to think the tipping point where the ARPU will start going up, instead of down, is at the 7-8 million subscription mark. Which would mean that with 3 million subscribers, DIsh TV itself has to double its market penetration before margins start working in the opposite direction for them. That could easily see them doubling current losses in the coming years and that alongside other costs could see their current Rs 300 crore loss going up to 700 crores. In short, this won’t be a fun competition to be in, if you wind up being second-best.

In one of the last links for the day, we have David Manners deviating from his usual domain of semiconductors posting a note on how bad T5 at Heathrow is, which should be sent to every person who is fond of doing the customary India-bashing bits under the pretext “this just would not happen it the west.” The price quote from the post is the captain saying: “We’ve landed at Heathrow which is in chaos”.

Lastly, for today’s silly Twitter apps update. Grouptweet is a Twitter application that allows you to send tweets to a group of people. Even better is the blog Twitterholics, which will allow you to track such inane products without having to leave the comfort of your browser tab.

p.s: Oh yes, Indiatimes has launched their social networking website (finally!). Looks like TIL now has two schools of thought: the IIS/.Net based in-house products and the LAMP-stack based outsourced products. Then there is the Java stack that powers the e-commerce offering, the entirely outsourced email offering. Oh well, this is Indiatimes after all.

That said, it is a very clean implementation and if you want to make friends with half of the staff at iWorld Gurgaon, this is the place to be at! Product-wise this looks like something that was put together after cobbling together everything they could find on other products. And for those who are wondering about the email part in it, it looks like a re-implementation of the current whitebox email solution provided by Indiatimes.

I guess the thinking is that there are way too many inactive/spam accounts on the main Indiatimes email framework, this could be a clean/fresh start towards having a better user base that can be sold for more to the advertisers. Let us file this one away in the “social media will buy me lunch (dinner and next day’s breakfast too!) department.” (hat tip: Contentsutra).

Key takeaways include: 70%-75% market share and 4 million users for Zapak. No mention of the usage frequency, it looks more like an aggregate. Should have been interesting number, if he should have mentioned the daily/regular users. They are looking to build up capacity in their gameplexes up to 10,000 by the end of 2008. Subscription revenues via MMOG titles to start in May. Zero in-house development of games.

Advertising segment action continues with Federated Media being now rumoured to have taken on $50 million in a round-B funding fishing expedition. Valuations currently are very much like a LSD trip, everyone sees a different colour from another, led mainly by the fact that there are fewer exits that are showing up now, and as you can imagine not all exits are going to be profitable or successful, so it eventually ends up being a cozy little circle in which the VCs and investment bankers are going to increasingly push up valuation to get better exits. How will all that work out? You guess is as good as mine. We just don’t know yet.

And all is certainly not well in VC-land. The wise men now agree that earlier puff pieces about an increase in the size of funds making a beeline for India may NOT hold true if the US continues to be as volatile as it has been of late. This maybe actually good news, because that should at least prevent the 60,000th ‘social-networking-site-with-a-difference’ from getting funded as a result.

Jeremy Liew, meanwhile, posts a good short one on Andrew Chen’s take on forecasting sales for web start ups. Both are good reads irrespective of whether you are a start up or not.

Meanwhile, in related advertising news, there is a rumour going around that Tribalfusion is up for sale. There are very few independent players left now in the agency market. Considering how almost of all of them got bought out, you should have opted to make a career running an online advertising agency when your parents asked you much earlier in life, “son, what do you want to do in life?”

Now for today’s ZOMG! section. Apparently (hold your breath, bring on the drum roll and other theatrics!), it is friendship that is driving social networking in India! I am shocked. No, more like SHOCKED! Oh, the horror, the awe! I would not be too surprised if that is the case everywhere. Pointer to the pundits: Robert Scoble and Jason Calacanis are not your typical consumers of social networking.

In music-related news, rapper Jay-Z too bails out from the sinking-on-steroids ship called the record label and signs up with Live Nation for his future releases much like Madge and U2 did a while ago. If they were to have had put some kind of tracking on the memo sent to the labels, that running after piracy and using litigation as a means to profit does not actually get you sustained profits, it probably would show that it is still stuck at the sender’s end. Hint of the day: CD sales are dropping like the stock price of Bear Sterns did on the last couple of days before being picked up by JP Morgan, while concert attendances and profits are booming.

Speaking of Bear Sterns, Henry Blodget gets Goldman Sachs to comment on the Bear Sterns episode that they did not knowingly press the forward button on the tape labeled “How to wipe out an investment bank in a week: the Bear Sterns edition” with their version of the story.

Lastly, how can we have a day pass without having yet another Twitter app being released into the wild? Latest on the list is Twitter Local, which allows you to track tweets by their location via an RSS feed. Now, what could be an interesting use case for this? Oh, someone could pick up the feed and automatically publish that feed via a Twitter account. I mean, whodda thunk that?

Effectively, now you can track all Tweets to within a “1 mile” range of the location specified by you. Imagine the progress that humanity will make as a result of this. Now, not only can I hear my neighbour sneeze, I can also track that sneeze on Twitter when he tweets it. It just absolutely warms the cockles of my sometimes non-existent heart.

Venky Harinarayan, co-founder of Kosmix, brings yet another perspective into the “Funding: why, when and how much?” question on Alt Search Engines. It is a nice read with ample stress on the key issues of valuation (internal, than the overhyped external projections), dilution and timing.

This one is a bit old, but what Theo Schlossnagle has written about his job profile is what a lot of people like me around the world will readily identify with. It is hard to give it a name, but it is a job, a mission, a career: all without a path or a name. And speaking of jobs, Yahoo! Music VP, Ian Rogers has left the company (as expected), to join a start up.

I honestly think the end of the old time internet corporations are very near due to their sheer bulk and lack of agility. They will surely exist in the B2B space, but as far as consumer-facing products are concerned, don’t look at them for the next big idea. Of course, M&A is always an option for them, but in 9/10 cases, being merged into one of those ends up killing the strengths that a smaller company brings to the table.

Parting shot: Teaching a six-year-old about RDF triples.

1) It has no unread count: Almost every application that has hit us since the advent of email has made a point to remind us constantly how far behind are we lagging on information accumulation, processing and categorization. Everything these days tends to tell you that you suck if you are 1) not overwhelmed by information overload and 2) not trying out the zany fancy ways to kill the overload.

2) Simple is the new complicated: There have been numerous attempts made till date to find a complicated explanation for why Twitter works, while the fact is that it works because it is quite simple. All information is presented in flat structure, in a hassle-free manner that saves you from having to tag/organise, categorize information. In Twitter there are no labels, folders or color coding. You can dive in and swim out of conversations at will and also pick your ideal rate/degree of involvement.

In a weird way, Twitter is exactly what you want it to be. It can be a social network, a meme tracker, time-lapse instant messaging or even email lite, which is why everyone has a hard time trying to define it. It means different things to different people.

3) 140 or bust!: Since there is a soft limit of 140 characters per message, Twitter, by virtue of its form, forces users to condense the matter into concise little capsules. This automatically means that value per message per follower or message is considerably higher than what you get from subscribing to an RSS feed. The form itself ensures filtering of the content, rather than having to rely on social categorization or machine categorization.

4) Single window system: The best thing about Twitter is that it does not enforce the use of any particular software or website to participate in the conversations, or just listen in. You can do all the activities specified in (2) using any of the numerous ways that are available to interact with the framework.

The lifecycle of social networking activities on individual websites can be mapped under the following heads:

Insertion: Social networks are very much like nightclubs and if you have ever been a regular clubber, you would know that not all nightclubs are created equally.

The newer and fancier clubs are where most of the cool crowd will always head for, while the less exciting ones go away pretty much unnoticed or end up serving niches that never scales up either in terms of traction or in terms of revenue.

More often than not the case for being at a new club is to be one of the first ones to get in, granting those who manage it a feeling of exclusivity. And at least in the early days insertion is much more important than the stages that follow after it.

You can see the same behavior in the case of social networks. Both Orkut and Facebook have benefited enormously from the same exclusivity as a result of their restricted entry policies in the early days.

Replication: Once an individual joins a social network, he/she wanders about replicating connections they already have in real life. There are exceptions to this behaviour, especially in the dating, younger demographic. Quite a bit of the initial spurt of activity on social networks is just this – replication of your existing social graph.

Discovery: While almost every online social network is similar to the others, they also have their own little unique ways of doing things. For instance, you “scrap” on Orkut, “poke” and “banter” on Facebook. Everyone has their own little unique way of doing things and these days, with the introduction of various platforms, there are also truckloads of applications, alongside new users to discover on the social networking sites.

The determinants

Degree of participation: Each of the above three points have degree of participation numbers attached to it. Growth on social networks slows down primarily when the number of people being inserted (new registrations, invites) decline. Secondarily, growth also slows down when the replication is mostly done with as everyone you know is already on the network by then.

To counter this, social networks may try and induce more participation from users by rewarding more participation (like a more frequently updated social stream). This can end up being a counter-productive approach, with high risk of alienating the less-frequent users.

Degree of fulfillment: All three factors can also be measured in terms of fulfillment a user gets from them. When you join, (the insertion stage) has a high degree of fulfillment attached to it, which declines over time when it is not that cool anymore, it is not that new anymore.

Replication also has high fulfillment in the initial days, getting more people on to the same platform etc. But it comes at a price. After a while, everyone you know is on the same network.

Moreover, everyone being there also deprives you of privacy. With time, you need increasing degrees of effort to maintain your profile. It is not uncommon to see users withdrawing more and more from doing things which is reflected in the public activity streams.

Gradually, everything moves to the inbox and private messages, which is a need that is already excellently served by email.

Discovery also has a high degree of initial fulfillment with users finding their way around the new websites, exploring new applications, features and people. Eventually, users get bored of using the applications and they have already added most of the people they have wanted to discover and add, resulting in falling rates of fulfillment as time progresses.

The Eventual Failure

As demonstrated above, there is little use case for sustained high levels of usage on online social networks. Over time, it is hard to battle inactivity and increasing levels of boredom for existing users.

To offset this churn, and also to prop up their stellar growth numbers, it is imperative that these websites keep adding a steady or an increasing number of new users all the time. But that number is a finite figure, determined by the number of people who use the internet and not all of them are going to sign up with social networks.

Unlike a Digg, Gmail or a news website, the value addition accrued from sustained usage of social networks is comparitively low and the need that it addresses is fairly artificial.

Another major issue of privacy and it is an issue without having to bring something like Beacon into the equation. If you do not fine-grain access control on your social stream, it is hard to figure out who all are getting to see what all parts of your life.

And if you do fine-grain access controls on social streams, it is either too much of work or it ends up being a better deal to use specialized services for it (email for communication, Flickr/Smugmug/Picasa for photo sharing, WordPress.com for blogging and so on).

Lastly, advertising inventory on social networks has till date been a major failure. Google tripped on the expectations it had from the Myspace.com inventory, advertising on Facebook or any other social network has not taken off much and the click through rates have been pretty poor on them. Unlike search or news, users don’t get on social networks to find ancillary information related to their activities. You don’t have to try too hard to imagine why there is not much context to one person poking another. It is, well, just a poke at the end of the day.

Eventually, even nightclubs need to reposition and redefine themselves every couple of years to stay in the game. Unfortunately, that is not an option that online social networks get to have and that is what will kill them

Persai, according to the founders, is an ad-supported content recommendation system. Over time, the guys have crawled a truckload of RSS feeds(there used to be a blog entry which said as much, but is not there on the bog anymore, but Sam Ruby has the list here), indexed and classified them and this in turn powers the recommendation system. You can subscribe to “interests” (known as keywords for the rest of humanity) and get sources thrown at you which the system thinks are relevant to you. While you can’t do much else with the sources, since Persai does not have a built-in feed reader, you can reject sources. And that is all there is to see about Persai. Well, at least for now.

The problems

Use Case: Recommendation systems have not traditionally fared too well on the internet. Previous players like Greg Linden’s Findory used to do a lot more than what Persai even does today and have not done too well at all. In fact, Findory, rather sadly, shut shop recently. The only recommendation system (which works in a stealthy manner) is Google News, which works because they don’t blatantly involve you in the recommendation process.

Once you find content on Persai, there is not much to do with it. Fulfillment is a term that is at best very vague on Persai. You can, as they claim, track the topics, but those links lead out the website anyway. Individual interests have RSS feeds that you can subscribe to, but you can already do that with Google News Alerts and other products. I do doubt if anyone is going to use Persai just to have search term driven RSS feeds.

Accuracy: The approach that Persai has taken to classification involves the usage of training data. This approach works well on similar data sets, but the moment you deviate from the similarity, the entropy will be of a magnitude which will send the classifier on a wild goose chase. And as expected, this has an adverse impact on the accuracy of the results. For instance, one of my interests — “mameo” — throws back results at me which has nothing to do with Mameo in the first five results. I could, of course, reject these sources and help improve Persai, but why would anyone do that when there are other avenues that provide me with much more accurate results?

Speed: To do classification, Persai is already using Hadoop’s MapReduce. Mapreduce does an amazing job of distributively processing huge chunks of data (freshly crawled data to be indexed and classified in this case), but it may only help Persai to a certain extent. The reasons for this are simple: If they process interests as unique to each user, it just won’t scale up. There will be numerous threads doing classification for the same interests since they are unique.

And if the interests are not tracked as a unique item per user, it can play havoc with the results with different users rejecting different sources for different reasons. Of course, there are workarounds for it by using a mix of both approaches (classify as non-unique, filter on display by excluding user-specific rejection criteria), but in the end it ends up being a hack.

In any case, the approach results in tremendously outdated results. Some of the interests have really old articles on top. This could also be due to the fact that the sources are manually added into the system, which means that the quality and spread of the sources will be dependent on the bias of the person who is selecting them. Moreover, it another issue that sites without RSS feeds will not be able get into Persai.

Splogs: Possibly the group that will be over the moon about Persai would be the thugs who run splogs. With Persai it becomes ridiculously easy to set up automated blogs based on topics and, honestly, I see more people using Persai for this than anything else.Considering that Persai is still in beta, I would not give it the “FAIL” rating, but I would certainly give it the “FRAIL” rating. I hope it becomes a much better by the time it comes out of private beta.