Why we chose JXTA

In my last blog posting, I talked about why we decided to build Kerika using Java, and how that experience turned out for us.

The other key decision that we made mid-way through our product development was to switch from a traditional client-server model targeting large enterprises to a more flexible, inexpensive peer-to-peer system that could be made available as a inexpensive subscription service.

We concluded that we had a better market opportunity among small professional services firms than with the Fortune 500, since the latter would have entailed much higher development costs, a much longer sales cycle, a much larger support organization, and much more competition. In other words, much of everything that we wanted to avoid!

For the peer-to-peer networking, we turned to JXTA, another open-source technology that had originally been developed by Sun Microsystems (under the guidance of the great Bill Joy, no less!) There were several reasons for using JXTA:

1. The overall architecture made a lot of sense: in fact, we had independently designed our own peer-to-peer architecture along the same lines before finding out about JXTA, such as using rendezvous and relay serversto implement a hybrid rather than pure peer-to-peer network.
   
   A pure peer-to-peer network is rarely practical in collaboration scenarios, since communication cannot take place unless both parties are online at the same time. We wanted to connect people working in different time zones, or simply according to different work schedules, so a hybrid peer-to-peer architecture was important, with a storage server kicking in automatically when you need to send messages to a team member who isn't online.
   
   The hybrid P2P architecture gives Kerika's users the best of both worlds: P2P's ability to quickly transfer large files directly from one computer to another, when both parties are online, and the store-and-forward reliability of email and other client-server systems.


2. JXTA was written entirely in Java, which meant we would have an easier integration job than if we had chosen a different protocol, such as Jabber, and the Java implementation also meant we could stay true to our cross-platform vision for Kerika.


3. JXTA looked like it could better handle large files and more diverse message types than other protocols, which was important for Kerika. JXTA's weakest point was in handling voice traffic, but this was not a design feature for Kerika so this weakness was not very relevant.
   

After hearing from our users, during our beta trials last year, that privacy was an important consideration, we went one step further and packaged up our storage server for use by our customer – at no additional cost. This enables our users to create private networks, i.e. a ring of trust that secures their data more thoroughly than they can get from any Web 2.0 free service like those from Google.

Why we chose Java

We made a key decision early in Kerika's product development to use Java and develop a true cross-platform collaboration system, although it would undoubtedly have been cheaper and faster to build a plain old .NET application that ran only on Windows. There were two strategic reasons for choosing the harder, but ultimately more rewarding, path:

1. Our vision was always about bringing together distributed teams regardless of where people are located or what sort of computer they are using. This narrowed our choices to Java, Flash, SVG, C++ and C. (Neither Python nor Ruby would have given us the user interface richness we considered essential for Kerika's success – at least back in 2003 when we made our language choices – to have been serious contenders.)
   
   We opted for Java because we believed that it was the richest cross-platform language available, one that would allow us to build an elegant user interface:
   
   
   • Using C or C++ would have resulted a lot of platform-specifc coding for the graphics and file management: in other words, we wouldn't be building a cross-platform application, but rather building the same application repeatedly, and painfully, for each platform.
   
   
   • Using Flash or SVG would likely have given us a great user interface, but there was, and remains, a dearth of programmers who are proficient in these platforms, which would have made recruiting a harder task, and, more importantly, there wasn't a rich ecosystem of third-party and open-source tools for Flash and SVG.
   
   
   • Using the .NET platform would have resulted in Kerika looking like just another WinForm application – in other words, looking like all our clunky competitors!


2. We were also mindful of the need to compete with Groove, and since Groove was very much a Windows-centric application – even before they were acquired by Microsoft – we needed to find terrain that would favor us should we have to go head-to-head with Groove.
   
   Although Ray Ozzie would occasionally make noises about creating a Mac version of Groove, we were fairly certain that this wouldn't happen, because the longer you develop a Windows-only application, the harder it becomes to retroactively make it a cross-platform software.

Our biggest concern as we started down the Java path was simple: was Java really going to be a build once, run everywhere application, or would it turn out to be a build once, debug everywhere application as skeptics warned us?

Our conclusion, after having written over 100,000 lines of Java: Java is really a build once, run everywhere platform!! We did find some minor glitches along the way, but these were almost always the consequence of two factors:

1. Using third-party tools, particularly open-source code, that hadn't been properly tested in the first place.


2. Trying to integrate platform-specific C or C++ code using the Java Native Interface(JNI).

Our own code, as well as code we got from more robust sources likes Piccolo, was worked just fine on multiple platforms. In fact, we were able to create a truly distributed team, with individuals programming on Windows, Mac or Linux according their preference, and then pull everything together as a single set of Jars. We do all our builds on a Mac, create installation packages for Windows and Linux, and then test all three versions simultaneously.

And it all just works...

Microsoft Entourage causing sign-up problems

We have had sporadic reports from users who were unable to sign-up for Kerika, mainly because the confirmation email they were getting contained a bad Web link (URL). We have traced the problem to be coming from Microsoft's Entourage email client on the Mac.

For some reason, Entourage is trying to interpret some part of the Web link as a HTML display tag. In other words, trying to be too clever.

We will have to figure out a way to code around this problem...

Update June 21, 2007: we have fixed this problem!

Help Wanted

Users who like our product, and particularly people who like our customer service, frequently ask us how they could help us in our efforts, and our answer is always the same: help us get the word out about Kerika, the world's greatest graphical Wiki application!

Much of our effort and resources continue to be expended towards product development and support – and, to some extent, legal services, e.g. our patent applications – which means we have little to devote to marketing. Our greatest challenge at the moment, therefore, is to get the word out, and that's how you can help!

The Rip Van Winkle effect in Kerika

A user writes in:

Sometimes when wake my laptop I get a dialogue suggesting that Kerika is trying to connect to a gmail account. Can you say something about what is going on as it looks rather suspicious.

The dialog box looks like this:

Here's whats happening:

• Remember, Kerika is a peer-to-peer application, which means that your machine connects directly to every other Kerika user who happens to be online. (This is similar to how other P2P applications work, like Skype and instant messaging.)
• The way your computer finds every other Kerika user who happens to be online is simple: when you start up Kerika, it automatically checks in with the central rendezvous server that we maintain at our data center.
• The rendezvous server lets your Kerika know who else is online, and how to reach them (i.e. their IP addresses). If you look at the Network Status area, on the lower-right corner of your Kerika application window, you will occasionally see these messages go by as your Kerika connects with other online users. However, the IP addresses of other Kerika users are never shown to you!

So, let's say you have been running Kerika for a while. While you were active, Kerika was keeping track of other users as they came online and left. But when your computer goes to sleep because of inactivity, things can go awry and it could end up waking in a confused state:

• While your Kerika was asleep, some of the other Kerika users may have gone away. But your Kerika missed these goodbye messages while it was sleeping.
• Also, the connection that your computer was maintaining with our rendezvous server was dropped because the rendezvous hadn't hear from your computer in a while and assumed that you had gone away.
• So when your Kerika wakes up, it may find itself in a confused state, just like Rip Van Winkle waking up from his long sleep: things aren't as they were before he took his little nap!
• Some users that he thought were around don't seem to be responding to him, and even the rendezvous server seems to have disappeared. Oh, dear! And like old Rip waking up, Kerika can sometimes be a little incoherent until she gets her bearings.
• The best thing to do in this situation is to simply restart Kerika: that will quickly synch up your machine with the rest of the world.
  

These error messages are quite rare, and obviously not very helpful in this particular context. We can try to eliminate them in a future version of Kerika.

Update: our user wrote back to ask another good question:

Thanks. I'm intrigued how this scales. If you get a million or so customers online does my client end up caching a table with a million or so email address/ip address entries although the community I am sharing with might only be a few tens of people?

The answer is No, because of the way JXTA works: you can set up several rendezvous servers, each of which knows how to connect to the other rendezvous [OK, I admit it: I don't know what the plural of rendezvous is...], either directly or through another rendezvous. Right now we don't have a million or so users, so we have just a single rendezvous ;-) but as we scale up, we can add more rendezvous.

Another feature of JXTA that we haven't fully exploited yet is the ability to create private groups. Right now we have just one big private group for all Kerika users; in the future we could create more fine-grained groups that create mini-communities.

Marketing advice from Tulsa, Oklahoma...

David Walker from the Tulsa Java Developers community very helpfully pointed out that we were doing a rather poor job of telling our Web site visitors that the Kerika software is actually available free for stand-alone use – we charge only for the value-added service that comes with being connected to other Kerika users:

• Sharing your projects with other Kerika users, and all the good stuff that comes with using the world's first (?) graphical Wiki application: sharing content in context, sharing large files, keeping everyone on the same page, etc.
• Sending project updates by email to people who don't use Kerika: which is one of the killer features in our new version.
• Setting up a private network, which is another killer feature in v1.1 (we have people trying this out right now in the US and Europe)

We have updated the sign-up page to clarify this point. Thanks, David!

Shorter Demos = Better Demos (I hope!)

We have a large number of Flash demos available, mostly because we hope these can serve as tutorials for users who want to learn more about a particular feature. These demos were all built using Wink, which is a fairly simple, almost low-tech tool for building Flash demos. (It is available free for Windows.)

Building demos is tremendously time-consuming: a 2-minute demo could take several hours to put together using Wink, if you want to do it well. When you are trying to showcase a collaboration tool like Kerika, part of the challenge is figuring out the exact storyline you want to have in a demo, and then setting up different machines to play their parts in developing this story.

For example, if you are showing how to share a project, you need to set up the project, set up the other machines to simulate other Kerika users, and then carefully walk through the process of capturing the action. You end up with a tremendous amount of screen captures because Wink, by default, captures 4 frames per second. And if you are not careful Wink's memory consumption can kill your PC altogether leaving you empty-handed!

As a result of frequent use I have become fairly adept at using Wink, and we have customized it somewhat to match Kerika's color palette and style. The problem of editing down the hundreds of frames (screen captures) to make a coherent, smooth presentation is still a very time-consuming one. It is possible that some of the more professional tools make some of this easier, but the top-end tools from companies like Macromedia are not only expensive, they can also be very intimidating for newbies.

After spending all these hours and days getting our demos ready, I hadn't really considered that perhaps our demos were too long-winded, in part because I had always considered them to be tutorials as much as marketing collateral. (And this was undoubtedly a mistake in hindsight because tutorials and marketing collateral are intended for very different audiences.)

It was only in the past week that I realized that our most important demo, the Overview, was way too long to serve as marketing collateral. It turned out to be the longest of all our demos since it clocked in at over 8 minutes.

I spent the past couple of days timing each demo's duration (which I had never really done before), and then editing them as ruthlessly as I could to make them shorter. I hope the results are worth the loss of frames: each demo is approximately half as long as it was before!

I have also created a new Intro demo which I hope will work better as marketing collateral, particularly for first-time visitors to our web site. This is a hacked-down version of the Overview demo, and it clocks in at 2 minutes and 45 seconds... (Do you think this is short enough?)

Adding Keynote and Pages files to Idea Pages

We just found out about a new problem: if you add Keynote or Pages documents to Idea Pages, you cannot open them from within Kerika by double-clicking on them, in the way you can open other files (e.g. Excel spreadsheets). In fact, Kerika cannot handle Keynote (.key) or Pages (.pages) files very well, and unfortunately this is not a problem that we can't fix in the short-term. (This problem was reported by one of our users, who therefore qualifies for the bug bounty.)

Here are the gory details for those who might be interested:

Kerika's document management system was designed only to handle files. Some fileson a Mac, e.g. Keynote pages, are actually folders. When you add these to an Idea Page, you cannot open them by double-clicking them – as you could with other files, e.g. Excel spreadsheets.

We don't allow file folders to be added to Idea Pages because we don't have any good way of knowing when a "file" that's actually a folder has been updated. (We need to know that a file has been updated so that we can automatically send the new version to the rest of the team.)

You might think that knowing when a file has been updated is a simple matter: just take a look at the timestamp or the size of the file and see if it has changed. That's true in most cases, but not all, and the problem gets particularly difficult when you are trying to create a true cross-platform application like Kerika that provides the same level of functionality on Windows, Macs and Linux.

The problem is actually with Microsoft Office files on Windows, at least with Office XP (and possibly later editions). Microsoft Word, for example, uses a bit of padding when it saves your document. So if you make just a minor change to a document, e.g. add a word or two, the size of the file – as it shows up on disk – may not change at all because the extra words you added were accommodated within the extra padding that Word used the last time it saved the file.

With Microsoft Excel, on the other hand, there is a different problem: as soon as you open a spreadsheet using Excel, the timestamp on the file immediately changes – even if you haven't made any changes to the file. So if you open a spreadsheet and keep it open for a long time, during this time the file will appear to have been modified – at least from the viewpoint of the timestamp – even though you haven't made any changes yet.

As a result, you cannot rely upon either the timestamp or the size of the file to be sure that it has really changed since the last time you took a look at it – at least for Office XP documents, which are a big chunk of all the documents that our users deal with on a daily basis.

So Kerika does some more complicated stuff with digital signatures to figure out whether a file really has been modified, but this method would not apply very well to folders. Hence, the decision to not allow users to add folders to Idea Pages.

Fixing this problem is possible no doubt, but not something we can do in the short-term, which is why this has been added to the list of Known Problems. (And our intrepid bug finder gets the bug bounty!)

Saying goodbye to the girl...

The picture of the little girl that featured in our "About Us" page is going to be replaced soon with something a lot less fun... We always liked the picture, because it seemed to capture a sense of fun and adventure and optimism, but it was becoming a bit of distraction (too many people asking too many irrelevant questions), so I guess we are going to have to look more "corporate" now :-(

In the Wikipedia

There's an article on Kerika in Wikipedia; check it out.

Free Software, Free Beer and a Free Lunch

The folks at GPL like to describe the open-source movement as being about providing "free software – free as in free speech, not free as in free beer".

And how should we describe the "free software" that we get from Web 2.0 hosted services? Would that be "free as in free lunch"? Only if you believe in free lunches:

Microsoft has begun combining personal data from the 263 million users of its free Hotmail email service – the biggest in the world – with information gained from monitoring their searches.

"We're in the early days of behavioral targeting but it's an idea whose time has come," says Simon Andrews, chief digital strategy officer for WPP Group's MindShare, a large buyer of ad time. Wall Street Journal

And we are not just picking on Microsoft...

Eric Schmidt, Google's chief executive, said last week that being able to acquire more personal data was a key element in the company's expansion plans. Financial Times

Or pretending to be the smartest guys around: Kerika has a very useful feature called the storage server: it automatically holds project updates for your absent team members. In other words, when your buddies are online, they get their updates directly from you, and when they are offline, these get stored on the server. Pretty cool, huh?

Cool maybe, but not smart enough: we found, during our beta trials last year, that a privacy backlash was already underway, and some people hated the idea that their content would sit on our servers, however briefly – even though we didn't serve up any advertisements, use cookies, or do anything else to track usage and behavior!

So we decided to create your own private networks: a "ring of trust" consisting of just your team's computers so that no one else gets hold of your content. Once you set up your private network, all your messages (project updates, files, etc.) go directly from one team member's computer to another: they don't go through any intermediaries, and if any of your buddies are offline, their messages are stored on your private server.

And what's more, we offer this free to all subscribers, even trial users. On the other hand, if you believe in free beer and free lunches, just sit back and relax, it won't hurt a bit...

We recognize that seeing ads based on the content of an email message can be unsettling at first. Our experience has been that this feeling recedes as users become more familiar with Gmail. Gmail's privacy policy.