Errors happen

I have noticed a few DatastoreTimeoutException in logs of my application hosted on Google App Engine. It looks like below:

com.google.appengine.api.datastore.DatastoreTimeoutException: Unknown
com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:42)
com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:60)
com.google.appengine.api.datastore.DatastoreServiceImpl$PreparedQueryImpl.runQuery(DatastoreServiceImpl.java:346)
com.google.appengine.api.datastore.DatastoreServiceImpl$PreparedQueryImpl.access$100(DatastoreServiceImpl.java:272)
com.google.appengine.api.datastore.DatastoreServiceImpl$PreparedQueryImpl$1.iterator(DatastoreServiceImpl.java:306)
org.datanucleus.store.appengine.query.RuntimeExceptionWrappingIterable.iterator(RuntimeExceptionWrappingIterable.java:42)
org.datanucleus.store.appengine.query.StreamingQueryResult.(StreamingQueryResult.java:77)
org.datanucleus.store.appengine.query.DatastoreQuery.newStreamingQueryResultForEntities(DatastoreQuery.java:324)
org.datanucleus.store.appengine.query.DatastoreQuery.fulfillEntityQuery(DatastoreQuery.java:310)
org.datanucleus.store.appengine.query.DatastoreQuery.performExecute(DatastoreQuery.java:242)
org.datanucleus.store.appengine.query.JDOQLQuery.performExecute(JDOQLQuery.java:84)
org.datanucleus.store.query.Query.executeQuery(Query.java:1489)
org.datanucleus.store.query.Query.executeWithArray(Query.java:1371)
org.datanucleus.store.query.Query.execute(Query.java:1344)
org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:221)
com.dendrytsoft.chat.server.MessageDAO.readAfterDate(MessageDAO.java:69)
com.dendrytsoft.chat.server.GreetingServiceImpl.getAllMessages(GreetingServiceImpl.java:38)

I found a hint about this from Jason Cooper(Google):

The majority of exceptions are thrown because two or more requests come in at the same time that attempt to write to a single entity or entity group. This results in write contention, and an exception may be thrown if a given request can't complete its write within the deadline. So, during your design stage, be sure to identify areas where this may be a problem and mitigate as much as possible. You can do this by keeping your entity groups small and

sharding single entities that may updated a lot, such as a global counter: http://code.google.com/intl/pl/appengine/articles/sharding_counters.html

Due to the distributed nature of App Engine's datastore, these exceptions will happen from time to time; in practice, this effects between 0.1 and 0.2 percent of all datastore operations, and we are always working to make this percentage even lower. For critical datastore operations, you should continue to catch the exception so you can provide a custom error if necessary or perform your own retries.

[...]

I would recommending two or three times [execute retry] maximum before showing the user an error message.

I have implemented retry when DatastoreTimeoutException occures and it looks that works great now.

There is an interesting speach about quality of applications hosted on App Engine: Best Practices - Building a Production Quality Application on Google App Engine.

Ken Ashcraft diveded problems that could happen on few clasess:

• Out-of-memory

• DeadlineExceeded

• OverQuotaError

• Server crash

• Datastore crash

• Identical entity already exists

I've recently read Coders At Work (great book), where Ken Thompson wrote that he is guessing that about 50 percent of code in Google infrastructure is connected with handling exceptions/errors/failures that occures during runtime.

I have some experience with real-time distributed telecommunication systems. I haven't thinking before how many percent of this systems code is the special situations handling mechanism. But surprisingly, realy: it could be even an half. It looks like we should be more aware of underlying distributed infrastructure while developing applications on top od GAE.

Building a Google Wave Gadget with GWT (Java + JavaScript)

I am very interested in Google's products for developers. I have been doing a few projects in Google Web Toolkit since version 1.5 when its developement environment and support was rather poor. I must agree that now there are many mature tools, good eclipse plug-in, many JS API wrappers (for example: gwt-google-apis), full Java development stack: Google App Engine(great and free for people starting with GWT) and nice community around this technology. But in this post I will focus on new Google product: Wave and I will shortly describe how to write a Gadget for it.

0.Introduction

I will write a gadget that will count how many pizzas should I order for the evening party. Adding all party's guests to the wave I can enable them to vote, how much they will eat, so I will be able to estimate the total number of pizzas I should order ;) That's tiny example, but it illustrates the way to implement a Wave Gadget.

After adding the gadget to a wave it will look like this:

After voting: 1pizza and adding two more participants to the wave, it will look like this:

After they vote, it will look like this:

So that is gadget that is described below. Lets code it.

1.Configuration

First of all you need to add a line below to your GWT modul descriptor. In my case its DendrytGadget.gwt.xml.

<script src="http://wave-api.appspot.com/public/wave.js"/>

Now you can start creating your gadget's main class. In my case it is DendrytGadget class.

@Gadget.ModulePrefs(title = "DendrytGadget", height=400, scrolling=true)
@Gadget.InjectModulePrefs(files = {"ModulePrefsRpc", "ModulePrefsDynamicHeight"})
public class DendrytGadget extends Gadget<UserPreferences>{

You can set preferences connected with the gadget using Gadget. Annotation Gadget.InjectModulePrefs enables developers to define hand-written content for the ModulePrefs section of the gadget XML manifset file. Each Require XML node should be holded in separate file. In this case we have two files ModulePrefsDynamicHeight:

<Require feature="dynamic-height"/>

and ModulePrefsRpc:

<Require feature="rpc"/>

2.Gadget's initialization

Our gadget class must extends Gadget<UserPreferences> abstract class and implement one abstract method (below). This method will be called after all of the feature initialization hooks have been called.

protected abstract void init(T preferences);

We will implement it in the way below:

@Override
protected void init(UserPreferences userPreferences) {
    instance = this;
    RootPanel.get().add(generateAmountPanel());
    WaveWrapper.registerStateChangeCallback();
}

Be careful at init()-method body: non-catched RuntimeExceptions make your gadget works wrong. Its GUI may be built, but every wave-Object's method will return null. The same problem occurs when you call $wnd.wave.getState().get() from the init()-method. Method getState() will return null and RunTimeException will be thrown:

com.google.gwt.core.client.JavaScriptException: (TypeError): $wnd.wave.getState() is null

That bug could be quite hard to find when you are starting Wave Gadget development. So be aware :) It looks that the good idea here is to surround every native JSNI method call with a try-catch block.

3.Registering state-change callback

Variable instance is DendrytGadget's static field. I use this workaround to set state-change callback via JSNI code (below). This native JS code registers static DendrytGadget.stateUpdated() method as a callback. Our gadget will be interested in the state of amount of pizzas that each user wants to order.

public native static void registerStateChangeCallback() /*-{
    var wave = $wnd.wave;
    if (wave) {
        wave.setStateCallback(@com.dendrytsoft.wave.dendrytgadget.client.DendrytGadget::stateUpdated());
    }
}-*/;

4.Generation GUI

generateAmountPanel()-method generates Gadget's GUI using standard GWT widgets. It creates 4 RadioButton's ("1 pizza, 2 pizzas..."), HTML object(showing choice for each user) and one Label("To sum it up..." - total number of pizzas to make order for). Each RadioButton has implemented ClickListener that calls setChoice()-method (below) that persist the pair (viewerId, choice) in Wave's persistency key-value state map. Only value that we want to persist and share with other Wave's clients is obviously our choice - number of pizzas that we will eat.

public void setChoice(int choice){
    Map<String, String> map = new HashMap<String, String>();
    map.put(WaveWrapper.getViewerId(), String.valueOf(choice));
    WaveWrapper.submitDelta(map);
}

5.Wave's state persistency (WRITE)

Persistency write handling methods should be implemented in the way below:

/**
* Updates the state object with delta, which is a map of key-value pairs representing an update
*/
public static void submitDelta(Map<String, String> delta) {
    JavaScriptObject map = JavaScriptObject.createObject();
    for (Map.Entry<String, String> entry : delta.entrySet()) {
        addEntryToMap(map, entry.getKey(), entry.getValue());
    }
    internalSubmitDelta(map);
}

private static native JavaScriptObject addEntryToMap(JavaScriptObject map, String key, String value) /*-{
    map[key] = value;
    return map;
}-*/;


/**
* Updates the state object with delta, which is a map of key-value pairs representing an update
* (internal call)
*/
private static native void internalSubmitDelta(JavaScriptObject delta) /*-{
    $wnd.wave.getState().submitDelta(delta);
}-*/;

It could be written with less JSNI-fun using com.google.gwt.event.dom.client.PrivateMap class that provides lightweight map implementation with API for both Java and JavaScript, but package is protected due to non-final API. It shows us that clearly that GWT is not yet ready and there are situations where you must use Javascript Native Interfaces.

6.Wave's state persistency(READ)

So, that was about changing wave's state. Now lets talk about reading state. Every time the state changes (one of the participants make a decision about number of pizzas he/she want to order), out callback method is called.

public static void stateUpdated() {
    // ...
    StringBuilder sb = new StringBuilder();
    int counter = 0;
    for(Participant p : WaveWrapper.getParticipants()){
        sb.append(p.getParticipantsInHtml());
        int choice = Integer.parseInt(instance.getUserChoice(p.getId()));
        sb.append(" : " + choice);
        sb.append("<br/>");
        counter += choice;
    }
    String s = counter + ((counter == 1) ? UNIT : UNIT + "s");
    instance.label.setText("To sum it up, we will need: " + s);
    instance.htmlField.setHTML(sb.toString());
}



public String getUserChoice(String userID){
    String s = WaveWrapper.get(userID);
    return (s != null) ? s : "0";
}


/**
* Get a value for the given key from Gadget persistentency key-value map
*/
public static native String get(String key) /*-{
    return $wnd.wave.getState().get(key);
}-*/;

7.Mapping of Javascript object to Java object

So, as you see it is implemented in the same way as the writing changes was. Next interesting thing is mapping of Javascript object to Java object. It is easy. As you see in stateUpdated()-method, there was WaveWrapper.getParticipants() JSNI-method call which returns array of Participant Javascript objects. To use it in Java code we need to implement class that extends JavaScriptObject class. Access to all its fields/methods via JSNI-methods is easy (see below).

public static native Participant[] getParticipants() /*-{
    return $wnd.wave.getParticipants();
}-*/;

public class Participant extends JavaScriptObject {
    protected Participant(){}
    public final native String getId() /*-{ return this.getId(); }-*/;
    public final native String getThumbnailUrl() /*-{ return this.getThumbnailUrl(); }-*/;
    public final native String getDisplayName() /*-{ return this.getDisplayName(); }-*/;

    public final String getParticipantsInHtml(){
        return "<img  src=\"" + getThumbnailUrl() + "\" height=\"30\" width=\"30\" />" + getDisplayName();
    }
}

And I think that's all what you need to know to develop Wave Gadget. So that's all, thanks for attention :) Sourcecode is available here.

BTW I've just found cobogwave project that is quite interesting. I haven't used it yet. It looks it wraps whole Google Wave Gadgets JS API, so using it could help you write less JSNI code.

Michał Radziwon