Codename One iOS Offline Build Project

To build upon my last post, I have encapsulated the steps to build an iOS application offline (i.e. without having to send to the CodenameOne build server) into a single ANT script that can easily be installed on your local system, and imported into the build.xml file of your CodenameOne application.

The CodenameOne folks have provided me with space in the CodnameOne incubator subversion repository to host this project so it will be available to everyone.

Synopsis

The Offline Build Tools project includes some scripts to perform the building of CodenameOne applications on any computer running Mac OS X 10.7 and Xcode 4+. The applications it produces should be equivalent to those apps produced by the build server in every way (including Native Interface support). Please let me know if you run into any cases that the build tools do not support).

Installation

Before you can use the offline build tools in your Netbeans project, you need to install the build tools project. You only need to do this once. Installation involves only two steps:

  1. Check out the offline-build-tools project from the SVN repository:
svn checkout https://codenameone-incubator.googlecode.com/svn/trunk/shannah/offline-build-tools codenameone-build-tools
  1. Run the ant install script:
$ cd codenameone-build-tools
$ ant install

This will check out the entire CodenameOne SVN repository into the “tools” subdirectory so that it can be used for building your projects. It may take some time to complete as the SVN repository is quite large.

When the installation is complete, you should notice that it created a file named “build-ios.xml” inside the codenameone-build-tools/dist directory. This contains a ready-to-use ANT target that can be imported into the build.xml files of your CodenameOne applications.

Adding the “build-for-ios-device-locally” Target to Your Netbeans Project

Now that the Offline Build Tools have been installed, you can freely add the resulting ANT target to your CodenameOne application. If you haven’t already done so, you’ll need to create a CodenameOne project in Netbeans (e.g. File > New Project, and select “Codename One” as the project type).

Under the “Files” tab in Netbeans, you should be able to see and open the “build.xml” file for your project. Just add the following <import> statement anywhere inside the <project> tags of the build.xml file:

<import file="/path/to/codenameone-build-tools/dist/build-ios.xml"/>

Now, if you right click on the build.xml file, and select “Run Target” > “Other Targets”, you will see a target named “build-for-ios-device-locally”. Select this target to build your project.

Alternatively, you can build from the command line using:

ant build-for-ios-locally

Note: If building from the command line, you may need to set your JAVA_HOME environment variable to point to your JDK’s Home directory. e.g.

export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.7.0u10.jdk/Contents/Home

If you’re not using CodenameOne yet, for building mobile apps, you really need to start. It is the only solution that currently allows you to write apps for all major platforms in a performant way.

Disclaimer

This project is meant for development purposes. I created it to allow me to be able to test and develop patches for CodenameOne without having to submit them to the CodenameOne people to be applied to the server. The current settings work directly off of the CodenameOne trunk, which is probably not ideal for production apps … it is bleeding edge.

I recommend using the CodenameOne build server for all production builds as it has been refined with (probably) some optimizations, and it makes it much easier to build on many different platforms with great ease.

Building CodenameOne iOS Apps Locally

Note: This post discusses the mechanics around building CodenameOne iOS projects locally. See my subsequent post for information about a project that automates this entire process.

I’ve been playing around with Codename One quite a bit recently, as it appears to offer enormous potential in the field of mobile development. One initial stumbling block that I had with CodenameOne was that you needed to use their build server to actually build your application. Since the code is open source, it is actually possible to build it yourself, but there were no build scripts available to automate this task, and the only instructions on how to do still required a fair bit of tinkering to get it to work.

Last month, as an exercise, I created an iOS port for CodenameOne that runs on Avian (a light-weight AOT compiler for Java), just to see if it was possible. It was indeed possible, and I was able to successfully get applications built with CodenameOne up and running using Avian on my iPhone. After some benchmarking, I found that there wasn’t a compelling performance difference between my Avian port and the XMLVM port that they are currently using on the build server. Therefore, rather than sink a lot of time into a competing port, it would be better to start working with the “official” XMLVM port for my development.

Today, as another exercise, I built an ANT target that would automate the building of a CodenameOne application for iOS on my local machine. To use it, you simply copy and paste the target into your Netbeans CodenameOne project’s build.xml, and modify some of the properties to point to the proper locations in my environment. Most of the logic is based on the build instructions that had previously posted in the CodenameOne forum.

Installation Prerequisites

You will need the following components to be in place on your system before you can use the ANT target:

After these are installed, the ANT target should work for any application you try to build.

Step 1: Checking out the CodenameOne repository

You simply need to open Terminal and check out the codename one repository. The following example creates a directory called “src” inside your home directory, and checks out codenameone in a directory named codenameone-read-only inside this src directory.

$ cd ~
$ mkdir src
$ cd src
$ svn checkout http://codenameone.googlecode.com/svn/trunk/ codenameone-read-only

At this point, the CodenameOne repository trunk will be located at ~/src/codenameone-read-only.

Step 2: Install XMLVM

The iOS Port project of the CodenameOne repository includes a copy of XMLVM that you can install if you don’t have it installed yet. It is located at:

codenameone-read-only/Ports/iOSPort/xmlvm

You can find installation instructions for XMLVM at http://xmlvm.org/documentation/

Update Dec. 14/2012: Currently you need to use the xmlvm distribution that is included inside the iOSPort directory. Installation instructions are still the same:
$ cd /path/to/codenameone-read-only/Ports/iOSPort/xmlvm
$ sudo ant
$ sudo ant install

The above instructions would install xmlvm at /usr/local/bin/xmlvm

Once you have XMLVM installed, you should be able to use the ANT target in your own projects.

Step 3: Add Build Scripts to iOSPort

The ANT target depends on a Python script to be able to add files to Xcode projects from the command line. You will need to download this buildscripts directory, and copy it into the codenameone-read-only/Ports/iOSPort directory (i.e. the final location would be

/path/to/codenameone-read-only/Ports/iOSPort/buildscripts

You can download these build scripts here

You also need to add a template for the Main application entry point into the iOSPort directory. Create a file at codenameone-read-only/Ports/iOSPort/MainStub.java with the following contents:

Adding the Ant Target to Your Project

  1. Create a CodenameOne application project in NetBeans (if you haven’t already).
  2. Open the build.xml file for your project. (e.g. you can click on the “Files” tab, then expand the tree node for your project, and double click the “build.xml” file.
  3. Paste the following snippet into the build.xml file. It can appear anywhere inside the <project>…</project> tags.
  4. Modify the codename1.repo.path property to point to the codenameone-read-only directory (i.e. where you checked out the CodenameOne repository to).
  5. Modify the xmlvm.path property to point to the XMLVM binary on your system.
  6. Modify the ant.path property to point to the ANT binary on your system.
  7. Modify the python.path property to point to the python binary on your system.

Building your Project in NetBeans

Once you have successfully added the the ANT target to your build.xml file, you should be able to build the project.

  1. Click on the “Files” tab.
  2. Right click on the “build.xml” file in your project, and select “Run Target” > “Other Targets” > “build-for-ios-device-locally”
  3. Wait. It should open Xcode with your project when it is finished building.
  4. You should be able to just run or build your project in Xcode like it is a normal iOS project. It will take some time (usually about 5 minutes per build), because XMLVM produces a couple thousand Objective-C files and includes them in the Xcode project. (Don’t worry, LLVM will strip out all unused code so that the resulting binary isn’t too big).

*Note: It doesn’t seem to like building for the simulator. You can only build for an iOS device currently. I believe this is because it includes a library, zbar, that is only compiled for arm devices (not the i386 emulator).

iOS Signing Identity & Provisioning Profile

It is worth noting that you will need to set up Xcode to use your iPhone Developer identity, or it will stop you in your tracks. You will also have to set up a provisioning profile if you are doing development builds. This was a real pain in the ** to get right (in fact it was probably harder than writing the ANT script), but there is lots of documentation on this on both the CodenameOne site and the Apple site (and all around Google too).

You’ll need to make sure that your App ID (set in Project Properties in NetBeans) will need to begin with your the unique code that your provisioning profile is set up with.

Current Limitations : Native Interfaces

This build script doesn’t currently build native interfaces that you may have developed as part of your application. Native Interfaces need to have some special glue generated at build time for them to work, and the code and specs for this glue have not been released. If you are using native interfaces, you’ll need to use the CodenameOne build server in order to access them.

I am working on replicating that glue so that Native interfaces will work in local builds, and will post it when complete. An inability to develop native interfaces locally is a major impediment to being able to contribute to the CodenameOne project. It is critical that *all* build functions can be performed locally or I fear that it will be difficult for a community to form around this wonderful project.

More Thoughts on Avian vs XMLVM for iOS

I’ve only been using CodenameOne for a couple of weeks now, and I’ve been experimenting with both Avian and XMLVM on iOS. As I mentioned above, the fact that XMLVM actually outperforms Avian on many benchmarks means that I may not pursue the Avian port much further. However there are a few aspects of using Avian that have come to my attention in the short time that I’ve been playing with CodenameOne:

  1. Debugging and Stack Traces are better on Avian. XMLVM currently isn’t a whole lot of help if you run across an exception. It states that stack traces are currently unavailable and simply prints the error message. Avian, on the other hand, will give you a full stack trace, which makes it easy to debug problems.
  2. A full Java runtime environment. In XMLVM there are a few things that don’t work the way you expect if you’re used to the Java runtime environment. E.g. I was trying to print debugging information to the console using the familiar System.out.println() only to find out that this doesn’t seem to output anything in XMLVM. (Not sure if there’s anything special I need to do to get this working). Also, things like loading resources from Jar files using Class.getResource() and Class.getResourceAsStream() don’t work on XMLVM. You need to use equivalents from the CodenameOne API which are thin wrappers over native methods that do the same thing.
  3. Well that’s all I have come up with so far..

Building Locally vs Sending to the CodenameOne build server

It is important for me to be able to build my CodenameOne projects locally because I want to start contributing to the project and I need to be able to test my changes to the CodenameOne core without sending them into Shai to add to the build server. However, for production releases, I still recommend using the build server. They have taken the time to sort of all of the kinks and add many optimizations that I just don’t have time to work out. Their build server is also a bit faster than building locally.

Avian vs XMLVM: AOT Java on iPhone Benchmarks

In my last post, I discussed the results of a benchmark comparison between XVMLVM and Avian both solving the Towers of Hanoi problem on an iPhone 4s. Originally it looked like XMLVM ran slightly faster, solving the problem in 35 seconds (vs 42 seconds for the Avian version).

The code that I used for the benchmark are as follows:

Codename One controller contains the actual benchmark timing code:

However, today I decided to try writing the same code directly in C using CodenameOne’s native interfaces. This way I would have a baseline with which to compare the performance – so that we know just how much performance we are giving up by using Java instead of C to write an iPhone App.

The native code for this benchmark is as follows:

And it was used in a CodenameOne controller to actually perform the benchmark timing as follows:

I was initially frustrated to find that the C version of my benchmark ran in 0 milliseconds (compared to about 35,000 ms for my Java version compiled with XMLVM/GCC and 42,000ms for my Java version compiled with Avian). Something had to be wrong. I checked to make sure that it was actually calling my function. It was indeed running the function. I tried increasing n to ridiculously large values (3000000), but it still ran in 0ms.

As it turns out the GCC/LLVM compiler was outsmarting my benchmark. Because the benchmark function didn’t return any values to to the caller, and didn’t affect any state outside of its local variables, it decided that the entire function could be replaced with No Op. This is sort of like a compiler’s version of “if a tree falls in the woods and nobody sees it…”. As far as GCC is concerned, the tree didn’t fall.

To solve this problem, I added a static variable that is incremented each time the function is called. This way the compiler wouldn’t be able to eliminate the code as dead. The revised C code looks like:

After modifying the benchmark to update a static variable (so that the compiler could no longer cheat), I was able to finally get a benchmark result. However, this discovery cast a shadow over my previous benchmark results, so I decided to modify the Java version to update a static variable (so that no cheating could take place).

The modified Java version looks like:

This time around, I found that Avian was marginally faster than XMLVM (about 15% faster).

The results of the revised benchmark are as follows:

Hand-coded C version (GCC/LLVM) 31.7 seconds
XMLVM version 65.8 seconds
Avian version 57.1 seconds

Changing to Use Object Methods Instead of Static Method Calls

After posting the above results in the XMLVM forum, Arno (the creator of XMLVM) suggested I try changing the TowerOfHanoi.move() method to not be static. I made the change as follows:

And the the actual benchmark timing code changed to:

Clearly, Arno had some insider knowledge about how the method dispatching works because this produced wildly different results for both XMLVM and Avian:

XMLVM version 40.2 seconds
Avian version 77.4 seconds

So XMLVM, in this test, narrows the gap between native C function calls, and its own method dispatching performance. (Less than 33% difference, which is negligible in the scheme of things). Avian clearly doesn’t handle object methods as quickly as it does static methods, and this really brings to a forefront Joel Dice’s comment that little benchmarks like this don’t really mean much. Applications do much more than perform recursive functions with simple integer arithmetic. The true performance of an application comes down to many factors including graphics, networking, memory allocation, floating point calculations, etc…

These experiments, for me, at least reaffirm that Java is a viable platform for developing for mobile. It should *not* suffer from the same performance problems that current HTML5 solutions do.

A Victory Lap for Java

The benchmarks above make it look like Java is at a disadvantage when compared to purely native apps written in Objective-C (even if in some tests XMLVM appears to have almost caught up). But these benchmarks are comparing to C function calls, which are very fast. What if we change the benchmark to use Objective-C messages. This, is not really fair, since it is well-known that Objective-C message passing is much slower than typical function and method dispatching. Nonetheless, let’s make try, if for no other reason than to know just *how* slow it is.

I changed the Native code as follows, so that it is using message calls for all recursive calls.

Not surprisingly, this benchmark ran much more slowly than the previous version:

Objective-C Message Calls 154.5 seconds

Thoughts in Summary

AOT-compiled Java (either by Avian or via XMLVM) produces performance that is comparable to native C code. It may actually perform better, but these benchmarks are very narrow in scope. The relative performance of XMLVM and Avian seem to vary widely depending on which operations are being performed. XMLVM seems to handle object methods very well, whereas Avian seemed to have slightly quicker dispatching of static methods.

In any case, I don’t have any reservations about the performance of CodenameOne in relation to native apps. Whether it uses XMLVM, Avian, or some other solution for deploying Java to iOS, it should be able to provide sufficient performance for any sort of application.

See Also

  1. CodenameOne + Avian = Java on iOS – My blog post discussing my experiment to create an Avian port for CodenameOne to run on iOS
  2. XMLVM is Actually Pretty Fast – My blog post discussing my initial benchmark results comparing Avian to XMLVM
  3. Codename One Avian Project on Github

XMLVM is Actually Pretty Fast

DISCLAIMER: The Benchmark results in this post turned out to be flawed due to GCC/LLVM optimizations unduly favouring XMLVM. Please see the next post where I discuss the corrected benchmark and associated results.

In a previous post, I described some of my experiments in creating an Avian port for CodeNameOne. As mentioned in the post, one of my motivations was to see if I could improve on the performance of the current default iOS port which uses XMLVM to convert the Java code into C code (and ultimately compiled with Xcode). I had a hunch that binaries produced via the XMLVM translation would be slower than an AOT compiled Java implementation because of the way it works (It converts VM byte code operations into C stack operations in the C language – which would presumably produce much less succinct code).

After some tedious conversion of XMLVM runtime library calls to JNI calls, I was able to successfully build my Avian iOS port for CodeNameOne. Next I created a simple application (based on the CodenameOne Tabbed Application template) that runs the Towers of Hanoi problem, and built two versions of the app:

  • One using the CodenameOne build server (which uses XMLVM in its build process)
  • – Another one using my Avian, Proguard, and Xcode.

The Benchmark Results

The results were surprising.

I ran both apps on my iPhone 4s which is on iOS 5. The app is set to solve the tower of Hanoi problem for n=30 moving from pole 1 to pole 3. The time required to complete the problem was pretty consistent. On average, the XMLVM app would complete the problem in 35 seconds, and the Avian app would complete it in 42 seconds.

NOTE: After fixing the benchmarks so that GCC/LLVM *had* to run all of the code, it turned out that Avian was actually a bit faster.

This is quite the *opposite* of what I expected. While I’m a little disappointed in the results, I am also encouraged. This means that the performance of apps developed in CodeNameOne for iOS (using their build server) is actually quite good. I can be confident that I am building on a solid foundation.

I posted these results on both the CodenameOne forum and the Avian forum and a few explanations for the outcome were proposed. One explanation, that makes sense is that the XMLVM build benefits from optimizations of GCC and LLVM such as method inlining, loop unrolling, autovectorization, code motion, and intelligent register allocation. Avian’s AOT compiler, being simpler, and far less mature doesn’t implement any of these optimizations so the resulting binary is actually working at a slight disadvantage.

Joel Dice (creator of Avian) did note that a simple benchmark like Towers of Hanoi is not really helpful for determining real-world performance. A full comparison on real-world tasks would be more informative.

UPDATE: The Benchmark results in this post turned out to be flawed due to GCC/LLVM optimizations unduly favouring XMLVM. Please see the next post where I discuss the corrected benchmark and associated results.

The Executable Size

Another important factor when producing a mobile app, is the actual executable size. Why install a 100MB application when you can get the same app in under a meg. I used Proguard on the Avian build to trim down the code size (so we don’t need to include the entire JRE. XMLVM, also uses a number of optimizations to ensure that dead code isn’t included in the final executable.

My test app came out at 3.5MB for the XMLVM build, and 6.0MB for the Avian build. So XMLVM seems to have won here again.

What Now For the Avian Port

Now that I have established that the current XMLVM implementation of the iOS port is quite fast indeed, there is little need to continue to develop an Avian port. This exercise was academic in nature, and I’m satisfied that I have achieved my goals, which were:

  1. Can it be done? (i.e. Create an Avian port for CodenameOne). The answer was YES!
  2. To learn about the CodenameOne architecture. I learned a lot.
  3. To learn about the Avian architecture. I learned a lot here too.
  4. Find out whether the performance would be dramatically improved by using an AOT java compiler instead of XMLVM -> C -> Xcode. The answer, for now, was NO. XMLVM is pretty fast as it is.

Despite that fact that my experiment didn’t yield any performance improvements, there still might be some advantages to developing apps on the Avian stack. Off the top of my head, these include:

  1. A more familiar Java environment. JNI instead of XMLVM’s runtime when interacting with the native environment.
  2. JDK7 support. I’m not actually sure what version of Java I could use with CodenameOne’s XMLVM implementation. It is at least Java 5, but I don’t think it is at Java 7. I notice that they use Apache Harmony for the iOS port.
  3. Perhaps better debugging and exception handling. Shai Almog noted that XMLVM doesn’t support ClassCastException and doesn’t check cast validity. Its stack traces are also a bit cryptic when you get an exception.

Up Next For Avian/iOS

I am keen to try my next experiment with Avian and iOS: Combining my Java-Objective-C bridge with Avian to produce a Java solution that is similar to MonoTouch. I.e. Where you write the code in Java but still use the Apple development tools for the UI. This wouldn’t be a cross-platform solution like CodenameOne produces, but it would be useful, I think.

Up Next for CodenameOne

Time to develop some real-world apps!

7A942R3ZJ88P

Apple’s Walled Prison

I endure it because they make great software. I endure it because I have become dependent on their ecosystem. I endure it because there’s money in developing software for their platform.

But I am ever so close to just saying “Forget it” and moving on.

Apple’s simplistic DRM solutions for iPhone reduce the utility of the device by about 90%. They don’t even seem to consider what happens when you have 2 or 3 people in a family, all with separate iPhones all synced to the same computer. They don’t seem to care about people who have a desktop and a laptop computer and need to be able to copy files and apps to their phone from either of these devices.

Just a glimpse of my life with Apple’s walled prison last night. I was developing some sample Apps using CodeNameOne, NetBeans and Xcode. I have the full dev environment on my laptop, but my iPhone happens to be paired with my Desktop. When it comes time to “test” out my app on MY phone, I need to first register my phone with Apple as a development device, then set up and install a provisioning profile onto the phone. This part was about an hour of mucking around that I would like to have back.

Then it comes to copying my test app to my phone. Of course, I can’t do that from my laptop. I need to kick my wife off the desktop computer so that I can first copy the app to the desktop computer. From there I should be able to copy the app to my phone with no problems, right? Well not so fast. Apple is a firm believer in Murphy’s law, and a simple “copy” just wouldn’t cut it.

First attempt it tells me that my phone is not authorized for the apps on this computer, so i would need to erase my entire phone to continue. I discovered that iTunes was signed into the iTunes store under my wife’s account – which was the source of the problem.

After logging out of the iTunes store and logging back in with my account, I make another attempt. At this point it stopped pushing me to erase my entire phone, but it seemed to be complaining that several apps were not authorized for this computer. E.g. My Gmail app and my Netflix app. I guess I installed them straight from the App store. It prompted me to authorize them, at which point I entered my iTunes username and password. It took the password and then informed me that the apps still weren’t authorized. 4 or 5 times around this circle, I finally decided to just go with the “Don’t Authorize” button. After all, my goal here is to simply copy ONE test app that I had written onto my iPhone.

After opting to “not authorize” my apps I was informed that I would be required to delete the apps. At this point I just wanted to proceed so I said “fine, delete them!”.

After that I was able to copy my test app onto the phone and it ran quite nicely.

So at the end of the day, my test app written with CodenameOne runs great, and the number of installed apps on my phone has been reduced by 25%. No gmail, No netflix…. I guess I’ll just have to install them again and go through this dance once more.

This is not an isolated incident. Just the most recent in my lifetime of troubled interaction with my iDevices. In the past I have given up on such difficult tasks as
1. Copying a video from my computer to my iPad.
2. Copying a song from my computer to my iPhone
3. Copying a PDF from my computer to my iPad.

Up until now, I had solved the problem by just reducing the number of things that I do with these devices. I was down to just checking email. But since the Apple gods have deemed my unauthorized to use my Gmail app, I guess I won’t be doing that either.