Redis vs RavenDB – Benchmarks for .NET Client NoSQL Solutions

These Redis vs RavenDB benchmarks have been made into a bar chart for a better visualization.

Seeing that Redis v2.0 has been just been released and Oren Eini (aka @ayende) has just checked in performance optimization improvements that show a 2x speed improvement for raw writes in RavenDB, I thought it was a good time to do a benchmark pitting these 2 popular NoSQL data stores against each other.

Benchmarks Take 2 – Measuring write performance

For the best chance of an Apples to Apples comparison I just copied the RavenDB’s benchmarks solution project and modified it slightly only to slot in the equivalent Redis operations. The modified solution is available here. Redis was also configured to run in its ‘most safest mode’ where it keeps an append only transaction log with the fsync option so the operation does not complete until the transaction log entry is written to disk. This is so we can get Redis to closely match RavenDB’s default behaviour. Enabling this behaviour in Redis is simply a matter of uncommenting the lines below in redis.conf:

appendonly yes
appendfsync always

To use this new configuration simply run ‘redis-server.exe /path/to/redis.conf’ on the command line.
Other changes I made for these new set of benchmarks was to remove batching from the Redis benchmark since its an accidental complexity not required or useful for the Redis Client.

Here are the benchmarks with these new changes in place:

Which for this scenario show that:

Redis is 11.75x faster than RavenDB

Note: The benchmarks here are of Redis running on a Windows Server through the Linux API emulation layer – Cygwin. Expect better results when running Redis on Unix servers where it is actively developed and optimized for. It is understood that the Cygwin version of redis-server is 4-10x slower than the native Linux version so expect results to be much better in production.

I attribute the large discrepancy between Redis and RavenDB due to the fact that Redis doesn’t use batches so only pays the ‘fsync penalty’ once instead of once per batch.

The ‘appendfsync always‘ mode is not an optimal configuration for a single process since Redis has to block to wait for the transaction log entry to be written to disk, a more sane configuration would be ‘appendfsync everysec‘ which writes to the transaction log asynchronously. Running the same benchmark using the default configuration yields the following results:

Which is a 39% improvement over the previous benchmarks where now:

Redis is 16.9x faster than RavenDB

Which unless I hear otherwise? should make this the fastest NoSQL solution available for .NET or MONO clients.

Measuring raw write performance using Redis is a little unfair since it has a batchful operation MSET specifically optimized for this task. But that is just good practice, whenever you cross a process boundary you should be batching requests to minimize the number of your calls minimizing latency and maximizing performance.

Even though performance is important, its not the only metric when deciding which NoSQL database to use. If you have a lot of querying and reporting requirements that you don’t know up front then a document database like RavenDB, MongoDB or CouchDB is a better choice. Likewise if you have minimal querying requirements and performance is important than you would be better suited to using Redis – either way having a healthy array of vibrant choices available benefits everybody.

Notes about these benchmarks

Since these benchmarks just writes entities in large batches to a local Redis or RavenDB instance using a single client, I don’t consider this to be indicative of a *real-world* test rather a measure is raw write performance, i.e. How fast each client can persist 5,163 entities in their respective datastore.

A better *real-world* test would be one that accesses the server over the network using multiple concurrent clients that were benchmarking typical usage of a real-world application rather than just raw writes as done here.

So why is Redis so fast?

Based on the comments below there appears to be some confusion as to what Redis is and how it works. Redis is a high-performance data structures server written in C that operates predominantly in-memory and routinely persists to disk and maintains an Append-only transaction log file for data integrity – both of which are configurable.

For redundancy each instance has built-in support for replication so you can turn any redis instance into a slave of another, which can also be trivially configured at runtime. It also features its own Virtual Machine implementation so if your dataset exceeds your available memory, un-frequented values are swapped out to disk whilst the hot values remain in memory.

Like other high-performance network servers e.g. Nginx (the worlds fastest HTTP server), Node.js (a popular, very efficient web framework for JavaScript), Memcached, etc it achieves maximum efficiency by having each Redis instance run in a single process where all IO is asynchronous and no time is wasted context-switching between threads. To learn more about this architecture, check out Douglas Crockford (of JavaScript and JSON fame) imformative speech comparing event-loops vs threading for simulating concurrency.

It achieves concurrency by being really fast and achieves integrity by having all operations atomic. You are not just limited to the available transactions either as you can compose any combination of Redis commands together and process them atomically in a single transaction.

Effectively if you wanted to create the fastest NoSQL data store possible you would design it just like Redis and Memcached. Big kudos to @antirez for his continued relentless pursuit of optimizations resulting in Redis’s stellar performance.

The Redis Client,  JSON and the Redis Admin UI

Behind the scenes the Redis Client automatically stores the entities as JSON string values in Redis. Thanks to the ubiquitous nature of JSON I was easily able to develop a Redis Admin UI which provides a quick way to navigate and introspect your data in Redis. The Redis Admin UI runs on both .NET and Linux using Mono – A live demo is available here.

Download Benchmarks

The benchmarks (minus the dependencies) are available in ServiceStack’s svn repo.

I also have a complete download with including all dependencies available here:
http://servicestack.googlecode.com/files/NoSqlPerformance.zip (18MB)

Gaining in Popularity

Redis is sponsored by VMWare and has a vibrant pro-community behind it and been gaining a lot of popularity lately. Already with a library for every popular language in active use today, it is gaining momentum outside its Linux roots with twitter now starting to make use of it as well as popular .NET shops like the StackOverflow team taking advantage of it.

Unlike RavenDB and MongoDB which are document-orientated data stores, Redis is a ‘data structures’ server which although lacks some of the native querying functionalities found in Document DBs, encourage you to leverage its comp-sci data structures to maintain your own custom indexes to satisfy all your querying needs.

Try Redis in .NET

If these results have piqued your interest in Redis I invite you to try it out. If you don’t have a linux server handy, you can still get started by trying one of the windows server builds.

Included with ServiceStack is a feature-rich C# client which provides a familiar and easy to use C# API which like the rest of Service Stack runs on .NET and Linux with Mono.

Useful resources for using the C# .NET Client

I also have some useful documentation to help you get started:
Designing a NoSQL Database using Redis
+ A refactored example showing how to use Redis behind a repository pattern
Painless data migrations with schema-less NoSQL datastores and Redis
How to create custom atomic operations in Redis
Publish/Subscribe messaging pattern in Redis
Achieving High Performance, Distributed Locking with Redis

Fastest JSON Serializer for .NET released

New! Benchmarks graphs are now available to better visualize the performance of ServiceStack’s JSON and JSV text serializers.

Due to my unhealthy obsession for producing software that runs fast, I’m releasing a shiny new JSON Serializer for .NET into the wild!

ServiceStack JsonSerializer is based upon my previous efforts of inventing a fast, new compact text serializer with `TypeSerializer`
and its JSV Format. Essentially I just refactored the current JSV code-base to support multiple serializers and then simply added an
adapter for the JSON format. Unfortunately in my quest of adding a JSON serializer to the feature list I’ve given up a little perf in the JSV TypeSerializer by
not being able to apply more aggressive static-type optimizations and method in-lining.  However I ended up preferring this option rather than having to branch the existing code-base to support
two fairly large almost identical code-bases doubling my efforts whenever I want to add new features or fix a bug. Although the good news is that the library is still Reflection.Emit code-free
so future optimizations are still possible!

Anyway based on the latest Northwind database benchmarks perf didn’t suffer too much
as JSV is still the fastest text serializer for .NET with the newly released JSON serializer not too far behind ūüôā
The benchmarks is showing the new JSON Serializer now over 3.6x faster than the BCL
JsonDataContractSerializer and is around 3x faster than NewtonSoft JSON.NET (the previous fastest JSON serializer benchmarked).
(Other popular JSON serializers LitJSON and JayRock were also benchmarked although both were found to be slower and more buggier than the previous options).

It also happens to be 2.6x faster and 2.6x more compact than the fastest Type serializer in the BCL – Microsoft’s Xml DataContractSerializer, giving yet another
reason for JSON lovers to prefer it over XML.

Serializer Payload size Larger than best Avg Slower than best
Microsoft DataContractSerializer 4097 4.68x 838.1957 6.93x
Microsoft JsonDataContractSerializer 1958 2.24x 1125.8554 9.31x
Microsoft BinaryFormatter 4927 5.62x 1113.4011 9.21x
NewtonSoft.Json 2014 2.30x 947.2970 7.83x
ProtoBuf.net 876 1x 120.9475 1x
ServiceStack TypeSerializer 1549 1.77x 270.0429 2.23x
ServiceStack JsonSerializer 1831 2.09x 312.6265 2.58x

(Combined results based on the Northwind database benchmarks. Payload size in bytes / Times in milliseconds)

Features and Usages
Effectively the JSON serializer is optimized for one task which is to Serialize/Deserialize types fast. Where possible I try to remain compatible with the BCL’s JsonDataContractSerializer by for example choosing to serialize DateTime using the WCF JSON format (i.e. /Date(1234+0000)/).

Although that being said the serializer tries to serialize as much as possible while at the same time being non-invasive and configuration free:

  • Serializes / De-serializes any .NET data type (by convention)
    • Supports custom, compact serialization of structs by overriding ToString() and static T Parse(string) methods
    • Can serialize inherited, interface, anonymous types or ‘late-bound objects’ data types
    • Respects opt-in DataMember custom serialization for DataContract dto types.

Developers wanting more features like outputting ‘indented JSON’ or building a dynamic JSON structure with LINQ 2 JSON would still be better off with the popular NewtonSoft JSON.NET.

In keeping with tradition I’ve retained a simple API:

[csharp]
string JsonSerializer.SerializeToString<T>(T value);
void JsonSerializer.SerializeToWriter<T>(T value, TextWriter writer);
void JsonSerializer.SerializeToStream<T>(T value, Stream stream);
T JsonSerializer.DeserializeFromString<T>(string value);
T JsonSerializer.DeserializeFromReader<T>(TextReader reader);
T JsonSerializer.DeserializeFromStream<T>(Stream stream);
[/csharp]

Basic Usage Example

[csharp]
var customer = new Customer { Name=”Joe Bloggs”, Age=31 };
var json = JsonSerializer.SerializeToString(customer);
var fromJson = JsonSerializer.DeserializeFromString<Customer>(json);
[/csharp]

Reasons for yet another .NET JSON Serializer

I only recently decided to develop a JSON serializer as I was pretty happy with my JSV format in fulfilling its purpose quite well by providing a fast, human readable, version-able, clean text-format ideal for .NET to .NET web services or for serializing any text blob (e.g. in an RDBMS or Redis, etc).

Unfortunately I recently hit a few issues which called for the use of JSON over JSV:

Ajax Benchmarks

The benchmarks after porting the JSV format over to JavaScript (to be able to use it inside Ajax apps) on the upside showed that it was actually a little quicker to deserialize than ‘safe JavaScript’ in advanced browsers (read: any browser NOT Internet Explorer). On the downside Native evaluation of ‘unsafe JavaScript’ was still quicker in those browsers. Unfortunately the biggest problem was performance in Internet Explorer sucked in comparison as at times was over 20x slower compared to its own eval. Now I’ve long ago become a silent proponent of the ‘Death to IE6’ group by electing not to test/support it, unfortunately given its significant market share I really couldn’t do the same for IE7 and IE8 so for overall performance reasons using the existing BCL JSON serializer was still the way to go.

Embracing the future Web (Ajax / HTML5)

It appears that dynamic web applications using Ajax and HTML5 are quickly becoming the first-choice platform for developing advanced client UI applications. I believe that pure ajax applications (i.e. static html/js/css talking directly to JSON data web services) will quickly supersed rich plugin frameworks like Flash and Silverlight with the help of sophisticated JavaScript frameworks like jQuery, the Google Closure Library and the upcoming browsers broad support for HTML5. I consider performance to be one of the most important features for an application so having a fast JSON Serializer would increase ServiceStack’s appeal as a high-performance Ajax server to power these new breed of apps.

Redis Interoperability

Although Redis potentially supports storing any binary data, it does have first class support for UTF8 strings. There have started to be some talk in the redis mailing groups for client library maintainers to choose to standardize on storing text blobs using JSON for simplicity and interoperability reasons. At the moment ServiceStack’s C# Redis Client is currently using the JSV format to store complex types because of its inherent perf and versionability characteristics, however this is likely to change to use JSON in a future release.

JSON support in Mono

Unfortunately over the years Mono’s implementation of the BCL’s JsonDataContract hasn’t improved much and is still the number 1 reason why some ServiceStack examples don’t work in Mono. This ended up being the motivating factor as I recently added preliminary REST support in ServiceStack (details in a future post) and was not able to run the live examples since I only have access to a Linux web host (thus requiring mono).

In the end, I decided to bite the bullet and jump on the NIH bandwagon again and develop a JSON serializer which would ultimately yield a few benefits mainly by making the ServiceStack web framework a very fast .NET Ajax/JSON server and hopefully positioning it as the preferred platform for developing high-performance cross-platform web services.

Now the default Json Serializer in ServiceStack

I’ve refactored all the ServiceStack.Text.Tests to support and test both the JSON and JSV formats and added a fair few JSON-specific tests as well – so even though its new I consider the new JsonSerializer to fairly stable and bug-free so much so that I’ve made it the default Json serializer in ServiceStack. As a result all my live ServiceStack examples are now working flawlessly on Mono!

If new serializer is causing some problems for existing ServiceStack users please file any issues you have or alternatively you can revert back to using the .NET’s default JsonDataContractSerializer by setting the global config option in your AppHost, i.e.

[csharp]
SetConfig(new EndpointHostConfig { UseBclJsonSerializers = true});
[/csharp]

Download JsonSerializer

JsonSerializer is available in the ServiceStack.Text namespace which like the rest of Service Stack it is released under the liberal Open Source New BSD Licence which you can get:

NewtonSoft

.NET’s new fast, compact Web Service endpoint: The JSV Format

New! Benchmarks graphs are now available to better visualize the performance of ServiceStack’s JSON and JSV text serializers.

Service Stack’s Git repo is still hot from the fresh check-in that has just added TypeSerializer’s text-serialization format as a first class Web Service endpoint.

JSV Format (i.e. JSON-like Separated Values) is a JSON inspired format that uses CSV-style escaping for the least overhead and optimal performance.

Service Stack’s emphasis has always been on creating high-performance, cross-platform web services with the least amount of effort. In order to maximize performance, our web services are effectively raw text/byte streams over light-weight IHttpHandler’s (Note: SOAP endpoints still use WCF bindings). This approach coupled with extensive use of cached-delegates (eliminating any runtime reflection) has proven to provide superior performance in itself.

So why the new format?

Well up until now the de/serialization for all web service endpoints were done using the DataContract serializers in the .NET BCL. The XML DataContractSerializer looks to be a well written library providing good performance for serialization of XML. Unfortunately for reasons articulated in my previous post on the history of web services, XML although great for interoperability, does not make a good ‚Äėprogrammatic fit‚Äô for integrating with many programming language models ‚Äď e.g. for AJAX applications JSON is a much more suitable format. The verbosity and strict-extensibility of XML also does not make it the ideal format in performance-critical, or bandwidth constrained environments.

The problem with JSON in .NET is that although being 2x more compact than XML it is 1.5x slower (based on Northwind model benchmarks). Unfortunately that is the fastest JSON implementation in the BCL there are others like JavaScriptSerializer which are over 40x slower still. The other blocker I encountered was that although the JSON implementation in .NET was slow, the equivalent one in MONO just doesn’t work for anything but the most simple models. Effectively Mono users had no choice but to use the XML endpoints, which is clearly not a good story for bandwidth-constrained environments as found in iPhone/MonoTouch apps.

Quite simply if I want a fast, compact, cross-platform serialization format that’s ideal to use in bandwidth-constrained, performance-critical environments as found in iPhone networked applications I had to code it myself. Drawing on years of experience in handling different serialization formats I had a fair idea on what I thought the ideal text-format should be. Ultimately the core goals of being fast and compact is the major influence in the choice of syntax. It is based on the familiar JSON format but as it is white-space significant, does not require quotes for normal values, which made it the most compact text-format that was still lexically parseable.

Other key goals was that it should be non-invasive to work with any POCO-type. Due to the success of schema-less designs in supporting versioning by being resilient to schema-changes, it is a greedy format that tries to de-serialize as much as possible without error. Other features that sets it apart from existing formats makes it the best choice for serializing any .NET POCO object.

  • Fastest and most compact text-serializer for .NET (5.3x quicker than JSON, 2.6x smaller than XML)
  • Human readable and writeable, self-describing text format
  • Non-invasive and configuration-free
  • Resilient to schema changes (focused on deserializing as much as possible without error)
  • Serializes / De-serializes any .NET data type (by convention)
    • Supports custom, compact serialization of structs by overriding ToString() and static T Parse(string) methods
    • Can serialize inherited, interface or ‚Äėlate-bound objects‚Äô data types
    • Respects opt-in DataMember custom serialization for DataContract DTO types.

For these reasons it is the preferred choice to transparently store complex POCO types for OrmLite (in RDBMS text blobs), POCO objects with ServiceStacks’ C# Redis Client or the optimal serialization format in .NET to .NET web services.

Simple API

Like most of the interfaces in Service Stack, the API is simple and descriptive. In most cases these are the only methods that you would commonly use:

[csharp]

string TypeSerializer.SerializeToString<T>(T value);
void TypeSerializer.SerializeToWriter<T>(T value, TextWriter writer);

T TypeSerializer.DeserializeFromString<T>(string value);
T TypeSerializer.DeserializeFromReader<T>(TextReader reader);

[/csharp]

Where T can be any .NET POCO type. That‚Äôs all there is – the API was intentionally left simple ūüôā

By convention only public properties are serialized, unless the POCO is a DataContract in which case only DataMember properties will be serialized. Structs can provide custom (e.g. more compact) serialization value by overloading the ToString() instance method and providing a static TStruct.Parse(string).

The JSV Web Service Endpoint

The home page for TypeSerializer on code.google.com goes into more detail on the actual text-format. You can get a visual flavour of it with the screen shots below

Note: the results have been ‚Äėpretty-formatted‚Äô for readability, the actual format is white-space significant.

In comparison here is the equivalent data formatted in XML (under a nice syntax highlighter):

View JSV live Web Services

One of the major features of Service Stack is that because JSV is a supported out of the box endpoint, it doesn’t require any code for all your web services to take advantage of it. You can access all your web services with the JSV endpoint by simply changing the base URL. Below are live web service examples from the Service Stack’s Examples project:

GetNorthwindCustomerOrders XML JSON JSV | debug
GetFactorial?ForNumber=3 XML JSON JSV | debug
GetFibonacciNumbers?Skip=5&Take=10 XML JSON JSV | debug

*Live webservices hosted on CentOS / Nginx / Mono FastCGI

You can view all web services available by going to Service Stack’s web service Metadata page:

http://www.servicestack.net/ServiceStack.Examples.Host.Web/ServiceStack/Metadata

Download TypeSerializer for your own projects

The JSV-Format is provided by the TypeSerializer class in the ServiceStack.Text namespace. It is perfect for anywhere you want to serialize a .NET type, ideal for storing complex types as text blobs in a RDBMS. Like the rest of Service Stack it is Open Source, released under the New BSD Licence:

ServiceStack eclipses jquip!

This weekend I’m happy to find we passed an important internal milestone:

Where for the first time since its release, the 4+ year ServiceStack has finally surpassed my 3 week hobby js project jquip in followers!

What’s jquip?

For those that don’t know, jquip is a modularized version of jQuery optimized to have the lightest footprint possible whilst retaining the most useful parts of jQuery. It’s useful for bandwidth constrained environments where you don’t want an external jQuery¬†reference (e.g. widgets, offline stand-alone web app) but still want to code against a pleasant jQuery-like API. We use it in the¬†Careers 2.0 Apply Button widget¬†that can be hosted on external sites and provides faster start times by being able to execute straight after the in-lined jquip. Another area jquip has proven useful is to power¬†ServiceStack’s built-in MiniProfiler¬†Web UI allowing it to work in offline environments without internet access. Embedding the full version of jQuery would’ve significantly increased the size of¬†ServiceStack.dll¬†by more than 100kb. This lets you develop your own lightweight offline single page apps that have access¬†/ssr-jquip.all.js¬†using nothing other than a ServiceStack.dll ūüôā

Why is this important?

This is a major milestone for us as one of the reasons to get into Open Source is for your creations have a positive impact on the world and the number of downloads and watchers your project has are tangible metrics that can be used to measure the reach your software has. These metrics re-affirms that contributions continue to be useful to a wide audience motivating the continued dev efforts.

I’ve always thought going Open Source immortalizes your code and as a developer provides the best chance for your creations to go on living long after you’ve left this planet. It’s also immune from Market threats where not even your commercial competitors can kill it. Although both projects now have an equal number of followers, ServiceStack has received an order of magnitude more effort with more than 2,000+ commits since first moving to GitHub. The unfortunate reality is the OSS culture is relatively weak in .NET¬†whose market share is primarily ruled by commercial entities. I attribute this to the deep roots the .NET culture has in the enterprise and the fear and stigma¬†attached from¬†straying from the prescribed default¬†.NET stack into the many quality OSS alternatives. Unless we see a significant shift in .NET culture I expect the adoption for .NET OSS projects to continue to remain lack-lustre and¬†see the top¬†leading talent of web developers and start-ups continue to erode to the more popular JavaScript, Ruby and Python platforms and their thriving OSS ecosystems. Which is justifiable at this stage, if your motivation for contributing to Open Source is to make an impact and maximize reach and utility of your software, it’s hard to ignore the leading platforms which sees a considerable more OSS activity and code-sharing of OSS libraries and frameworks.

.NET is a quality platform, and C# / F# are semantically world-class designed languages with the fewest warts and a rich balance of intuitive features. They’re not always the most productive option (e.g. slow iteration times of ASP.NET) but they do have the richest IDE and tooling experience. I see the biggest threat to the platform is its waning adoption and network community effects of the strengthening alternative web platforms. The outlook is not all grim, e.g.¬†xamarin.com¬†is bucking the trend and pushing C# in uncharted territories as a growing platform for use in cross-platform mobile development allowing the same shared C# code-base and libraries to run on the leading iOS / Android mobile platforms.

How can you help?

Essentially the goal is to increase .NET OSS activity and anything you can contribute to this effort will have a net-positive effect, some examples of this might include:

  • Create more .NET OSS projects: Consider Open Sourcing your in-house internal libraries and frameworks (which if you think of the benefits of OSS makes more sense to be OSS). One of the few benefits of having a weak ecosystem is the opportunity to fill in the missing features! ūüôā .NET could do with more love around NoSQL solutions, MQ’s, web tooling and libraries for alternative cloud providers, etc. These are great places to start.
  • Contribute to existing .NET OSS projects: Start with the ones you’re already using otherwise here’s the leading list of C# projects on GitHub you can work through.
  • Evaluate if .NET existing projects are a better fit:¬†If you have the opportunity for your next project, consider spending some time to evaluate if more viable OSS alternatives exist that may provide a better fit for your current default closed-source stack.
  • Increase the collective knowledge around existing projects:¬†i.e. blog, submit talks, help out on Stack Overflow, participate in mailing lists or JabbR. If you or your team has spent a lot of time implementing a solution for a popular use-case consider blogging about it, providing insight to others following in a similar path.
Currently C# stands as the 12th and F# the 43rd most popular languages on GitHub (the home of OSS), it would be great to see if we can all beat these numbers next year!

ServiceStack is on InfoQ! TL;DR and Links

Adding to¬†ServiceStack’s great run in 2012¬†I was approached by InfoQ to find some details about ServiceStack vs WCF, why we created it, it’s advantages over WCF and the philosophical differences in development styles and approach between ServiceStack and high-level Microsoft¬†libraries and application frameworks. The resulting interview contains my thoughts on the best development and remote services practices, WCF’s architecture, its use of SOAP/WS-*, why heavy abstractions and¬†artificial server-side programming models are not ideal, how pragmatic and¬†simpler serialization formats are better suited for services, why message-based designs are preferred by most other frameworks, the state of .NET Open Source and much more.

The entire interview is quite long which ran to 8,000 words that as a result was split up and published in two¬†articles. To make it easier to find and¬†navigate the content, I’ve published a TL;DR summary, including related questions, table of contents and all embedded¬†links used¬†below:

Interview With Demis Bellot, Project Lead of ServiceStack – Part 1:

InfoQ: What problems do you see in Microsoft’s approach to Services and how does ServiceStack try to tackle it?

Interview With Demis Bellot, Project Lead of ServiceStack – Part 2

InfoQ: What exactly is a message-based Web service?
InfoQ: You recently introduced a razor engine making ServiceStack a more complete web framework than just a web services framework – what was the motivation behind that?
InfoQ: Are there any scenarios where you think WCF/Web API/MVC might be better suited than ServiceStack?
  • MVC better at¬†large write-heavy server-generated websites
  • ServiceStack is optimizing for SPA’s like Backbone.js, AngularJS, Dart Web Components, etc
  • WebApi better choice for building server-side driven systems with HATEOS Restrictions
  • ServiceStack better choice for maximum re-use and utility of your services
  • MVP and Microsoft Gold Partners will want to continue to choose the prescribed Microsoft stacks
  • ServiceStack will optimize for alternative platforms (e.g. Mono), Clouds (EC2, Google CE), RDBMS (Sqlite, MySql, Pgsql, etc), Redis/NoSQL
InfoQ: Microsoft has collaborated with open source projects in the past (JQuery, NuGet for e.g.) and MS folks like Scott Hanselman seem quite open about adopting good open source solutions into the MS default stack wherever possible – do you foresee any such collaborations that could benefit the .NET community in general?
InfoQ: You made a comment recently on one of the forums – “I’m hoping next year to be even better, as I have a few things planned that should pull us away from the other frameworks” – would you elaborate what features you have in mind for the future?
ServiceStack set to make exciting new product announcements in 2013. Current road map includes:
  • Merging the Async branch and its async pipeline
  • Create new fast Async TCP Endpoints
    • Enable fast, native adapters for node.js and Dart server processes
  • Enable integration with more MQ Endpoints (i.e. RabbitMQ and ZeroMQ)
  • VS.NET Integration and our improved solution for WCF’s ‘Add Service Reference’
  • Integrated Development workflow and story for Mono/Linux
    • Enable automated deployment story to Amazon EC2 and Google Compute Engine clouds
  • Signed and Long-term stable commercial packages
  • Starter templates around popular Single Page App Stacks: Backbone.js, AngularJS, Yeoman and Dart
  • Starter templates for creating CRM and SharePoint-enabled support systems
  • Website re-design and improved documentation

If you have the time I recommend reading the entire interview, otherwise hopefully these bulleted points can help you find the bits you’re interested in sooner.

1,284 Words

After 480 days uptime, lights out for old servicestack.net server

From today, after having served its 484th day of uptime we’re now relenquishing the old servicestack.net server into a well-earned retirement.

Over 480 days uptime and 10M+ commands processed

Since its last restart it has processed more than 10+ Million redis operations resulting from ServiceStack’s Redis-powered live demos and our other websites we had that were using Redis for front-level caching.

It was a CentOS/Mono powered 2GB RAM dedicated server from leaseweb which we put to work to host the servicestack.net domain (inc. all the live demos) as well as the other side projects I had running e.g: ajaxstack.com, freelamusica.com.

Unfortunately time became its ultimate undoing, where newer spec’ed hardware and profileration of in-expensive hosting providers have now diluted its once great value.

The big move

Fortunately moving between Linux servers is easy and we were able to transfer to a different hosting provider in 1 weekend. Configuring Nginx and Apache has gotten a lot better where both now have “no touch” virtual host configuration and simple sym-links to decide which sites should be enabled.

Transferring the GB’s of website content was done with a single rsync command that completed in minutes, thanks to both servers having blindingly fast bandwidth.

WordPress migrations still painful on Linux

Most of the time ended up being spent on migrating this WordPress blog, where I had to manually create the new database, configure host settings, export and import the blog content and as well as hunt down and enable the website template and all the existing plugins. Its funny that despite being developed on the LAMP stack, it’s still easier to install WordPress on Windows than Linux.

Final Memento

I’ll share details of the new server in a future post. But for now I’ll leave this final screenshot of the RedisAdmin UI info page as reminder of the little work horse that was, captured in its final hours – showing its impressive redis-server stats ūüôā

In cloud speak, that’s more than 99.99999% uptime!

330 Words

Async, Cached Twitter API Proxy in F#

Following on from showing how easy it was to create web services in F# with Service Stack, it’s now a good time to step it up a bit and do something useful to showcase some F#’s async goodness!

The final solution is available as a stand-alone HttpListener or an ASP.NET host which I just happen to have deployed earlier on a Linux server at¬†http://www.servicestack.net/ftweetstack/.¬†Here’s a couple example urls to see the stats of all my friends and followers:

http://www.servicestack.net/ftweetstack/followers/demisbellot
http://www.servicestack.net/ftweetstack/friends/demisbellot

Edit Screenshots added:

As I’m now getting rate limited by twitter, the live services have stopped working so I’ve included screenshots of the web service responses, reminiscent of happier times ūüôā

[format=html]

[format=csv]

[format=json]

Continue reading

2,144 Words

F# Web Services on any platform in and out of a web server!

Following on from my previous post where I took a first look at F# through my C# tinted eyes, I looked into some of the advantages of what this versatile language brings to a C# devs table. One of the major benefits I mentioned in passing but failed to explain in detail was it’s exceptionally good async story, which in many respects provides arguably the best async programming development experience to date on any platform.

F# and .NET Web Services

If F# only had to interface with other code as its inputs and outputs, the async world of F# is pristine and beautiful. Such is rarely the case so we’re stuck with having to interface with existing frameworks to provide these additional user facing services. Unfortunately the Web Services story for .NET isn’t pretty, where if for whatever reason you’re forced to use Microsoft-only technologies, you’re left with either having to deal with the WCF problem or extend MVC’s framework and roll your own REST-ful framework-in-a-framework yourself – which although less capable, I’ve witnessed to be a viable alternative providing a better iterative experience than WCF in the wild. Another web framework worth considering along similar lines, but developed in true Open Source spirit is NancyFX.

However if like me you like Web Services and still want to use a framework optimized for the task but consider WCF to be an anti-pattern promoting, over architected abstraction I’m happy to highlight a more elegant option in Service Stack which does the same job as WCF in allowing you to expose XML, JSON and SOAP services (as well as CSV, JSV & HTML formats)¬†but does so cleanly, out-of-the-box, without any configuration required! It’s typed, DTO-first development model gets out of your way providing a friction-free dev experience that encourages the development of clean,¬†best-practice web services – easily consumable from any client.

I should note the latest Web Service offering coming out of Redmond: WCF Web API is not be confused with WCF (even if the naming tries) does a much better job of ‘exposing HTTP and REST’ rather than hiding it like the original WCF framework upon which it is built. This may work better for you,¬†although my preference for Open source solutions favouring light, DRY, typed, performance-focused¬†API’s with the ability to run cross platform keeps me using and continue to recommend and maintain Service Stack for the¬†foreseeable¬†future.

Hello, F# World!

To get started we just need the Service Stack dlls that can be downloaded from GitHub. In the unzipped folder you can spark up your favourite text editor and write your first Hello, World web service which looks like [gist]:

[csharp]
open System
open ServiceStack.ServiceHost
open ServiceStack.WebHost.Endpoints

type Hello = { mutable Name: string; }
type HelloResponse = { mutable Result: string; }
type HelloService() =
interface IService<Hello> with
member this.Execute (req:Hello) = { Result = "Hello, " + req.Name } :> Object

//Define the Web Services AppHost
type AppHost =
inherit AppHostHttpListenerBase
new() = { inherit AppHostHttpListenerBase("Hello F# Services", typeof<HelloService>.Assembly) }
override this.Configure container =
base.Routes
.Add<Hello>("/hello")
.Add<Hello>("/hello/{Name}") |> ignore

//Run it!
[<EntryPoint>]
let main args =
let host = if args.Length = 0 then "http://*:8080/" else args.[0]
printfn "listening on %s …" host
let appHost = new AppHost()
appHost.Init()
appHost.Start host
Console.ReadLine() |> ignore
0
[/csharp]

The rest of the code except for the 5 lines of code needed to create the actual service need only be written once which is just used to setup the AppHost and run it, in this case as a Console Application.

Great, once you copy all that in, and hit save using the filename Hello.fs – you can compile with the 1-liner:

[csharp]fsharpc -r:ServiceStack.Common.dll -r:ServiceStack.Interfaces.dll -r:ServiceStack.Text.dll -r:ServiceStack.dll Hello.fs[/csharp]

Which cuts you a nice Hello.exe that you can double-click to run on Windows, otherwise run on OSX or Linux with:

[csharp]sudo mono Hello.exe[/csharp]

And Voila! your stand alone F# self-hosting Web Service is ready to serve you at: http://localhost:8080/hello/FSharp!

Convert into an ASP.NET Web Service

To convert this into an ASP.NET application we just need to do some minor tweaks add the appropriate files to deal with the nuances of having an ASP.NET application. For this we need to venture back to a time before VS.NET templates existed that did all this for us.
Note: All files including HelloAsp.fs, Global.asax and Web.Config files used in the walkthrough below are available on this [gist].

First lets take a copy of the file and call it HelloAsp.fs

1. As we want to create a .NET assembly we wrap the entire script in a namespace adding the declaration on top and indenting the rest of the file

[csharp]namespace HelloFSharp[/csharp]

2. Change the AppHost from a Http Listener to an ASP.NET AppHost by removing the word HttpListener from AppHostHttpListenerBase e.g:

[csharp]
type AppHost =
inherit AppHostBase
new() = { inherit AppHostBase("Hello F# Services", typeof<HelloService>.Assembly) }
override this.Configure container =
base.Routes
.Add<Hello>("/hello")
.Add<Hello>("/hello/{Name}") |> ignore
[/csharp]

3. Replace starting the AppHost from the command line with initialising it from HttpApplication Application_Start() event, i.e:

[csharp]
type Global =
inherit System.Web.HttpApplication
new() = { }
member x.Application_Start() =
let appHost = new AppHost()
appHost.Init()
[/csharp]

4. Create the Global.asax stub to point it to the Global type, i.e:

[csharp]echo "<%@ Application Inherits="HelloFSharp.Global" %>" >> Global.asax[/csharp]

5. Compile HelloAsp.fs into a library since its now an ASP.NET app instead of a stand-alone executable:

[csharp]fsharpc -r:ServiceStack.Common.dll -r:ServiceStack.Interfaces.dll -r:ServiceStack.Text.dll -r:ServiceStack.dll –target:library HelloAsp.fs[/csharp]

6. make a bin directory and copy all the dlls into it:

[csharp]mkdir bin && cp *.dll bin[/csharp]

7. Copy the desired Web.config mapping from Hello World Tutorial

[csharp]
<?xml version="1.0"?>
<configuration>
<system.web>
<httpHandlers>
<add path="*" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*"/>
</httpHandlers>
</system.web>
<!– Required for IIS 7.0 –>
<system.webServer>
<handlers>
<add path="*" name="ServiceStack.Factory" type="ServiceStack.WebHost.Endpoints.ServiceStackHttpHandlerFactory, ServiceStack" verb="*" preCondition="integratedMode" resourceType="Unspecified" allowPathInfo="true" />
</handlers>
</system.webServer>
</configuration>
[/csharp]

8. Run the .NET 4.0 version of xsp in the current directory

[csharp]xsp4[/csharp]

If all went well you should see the following output:

[csharp]
Listening on address: 0.0.0.0
Root directory: /Users/mythz/src/fsharp/fsharpstack
Listening on port: 8080 (non-secure)
Hit Return to stop the server.
[/csharp]

and you’ll be able to hit the service again using the same url at http://localhost:8080/hello/FSharp!

You now have a fully-fledged ServiceStack ASP.NET application that can be hosted in any ASP.NET compatible web server e.g:

  • IIS
  • Apache + mod_mono
  • Nginx + MonoFastCGI
  • XSP
  • WebDevServer.exe

Now that you know how to create functional F# web services, you can read more about the goodies you get for free when using Service Stack, complete with live links to all those juicy XML, JSON, SOAP, HTML, CSV and JSV Formats check out: http://www.servicestack.net/ServiceStack.Hello/

Next – Async Goodness!

With the basics out of the way tomorrow we’ll tackle a more complex service taking advantage of F#’s async and parallel good ness to provide nice high-level aggregate features off twitters public API.

Stay Tuned!

A Look at F# from C#'s corner

For a while now I’ve been hearing many great things that have been coming out of Microsoft Research’s popular .NET language F#. For the unfamiliar, F# is a strongly-typed, functional-based language for .NET – originally created by¬†Don Syme (better known to most as¬†the father of generics in .NET) it has now become a fully supported language in .NET with soon to be first-class support in VS.NET.

Despite being deeply inspired by its functional roots, it stands out from other languages in that it also supports imperative and OOP paradigms as well. Boasting both interactive and compile modes, it’s been holding over C# lately, sporting a more succinct syntax and already comes complete with features like ¬†async workflows and an interactive mode¬†we wont see in C# until V.Next.

The announcement of F# 3.0 pushes the envelope even further where the new Type Providers promises to be even more productive by allowing you to build against a strong-typed API (with intelli-sense) against a runtime datasource.

So not wanting to fall too far behind a good thing I’ve made F# on this years must-learn-list. Before looking at F# I have to admit I thought C# was close to the perfect server-side language with its biggest weaknesses just being the lack of string interpolation and¬†Go’s-like interfaces. Given that, I was quite surprised to find how much more elegant F# is in a number of ways:

Type Inference to the Max

Where some C# developers are still reluctant to use var in their works, F# takes type inference to the extreme where you can effectively omit type declarations whenever the type is not ambiguous and can be safely inferred.

Less code, the better

I’m a firm believer a smaller code base is a good one and we should be striving for a DRY, tight code-base where the solution permits. Having less code means there is less to maintain and less chance for something to go wrong, where a high signal-to-noise ratio is generally more readable as you’re able to focus more on the solution and less on the needless abstractions that get in the way.

In addition to type inference, F# has many features to tackle code boat including:

Out of all its features I believe what promotes the least code bloat is the quality of F#’s community (see: The Python Paradox) and its preference to simple, elegant composable solutions. ¬†This is contrast to C#’s abstraction fetish it copied from Java and its¬†relentless need to cater for the lowest common Drag n’ Drop developer and impress them with 5 minute demos. Although it’s a subject for another post, this rarely leaves us with quality frameworks or APIs.

Functional vs Imperative

Even though F# allows for programming in procedural, Object Orientated and functional paradigms, its roots and optimized syntax lends itself towards functional-style of programming. In many ways functional programming provides more elegant and robust solutions. Luca Bolognese explains this best in his excellent Intro to F#,¬†where the instincts of a C# programmer attempts to solve a problem imperatively resulting in mutable variables and disjointed logic allowing for moving parts to¬†go wrong. ¬†In his example Luca uses the ‘sum of squares’ as an example where the budding C# developer would approach it into something like this:

[csharp]
public int Square(int num)
{
return num * num;
}

public int SumOfSquares(IEnumerable<int> list)
{
var sum = 0;
foreach (var num in list)
{
sum += Square(num);
}
return sum;
}

var nums = Enumerable.Range(1,100);
Console.WriteLine("Sum of squares of 1-100: {0}", SumOfSquares(nums));
[/csharp]

The F# functional approach would lend it more to the original question, i.e. square the numbers, then sum them:

[csharp]
let squares x = x * x
let sumOfSquares nums =
nums
|> Seq.map squares
|> Seq.sum

printfn "Sum of squares of 1-100: %d" (sumOfSquares [1..100])
[/csharp]

Although oddly enough despite its already succinctness, it can be even further reduced to:

[csharp]
let sumOfSquares nums = nums |> Seq.sumBy squares
[/csharp]

I recommend watching the rest of Luca’s video as he goes on to show how the F# solution lends itself to easy parallelization, without interrupting the existing flow of logic.

In a real world problems closer to home, @ayende recently posted one of his interview questions online asking for example solutions to calculate israels tax. As expected most C# solutions were similarly written the same way complete with mutable variables, tightly coupled solutions Рmany of them having custom types and other artefacts.  Comparatively the few F# solutions posted had significantly less code, yet was easier to read and maintain.

My attempt at solving this problem in a purely functional style resulted in this [gist]:

[csharp]
let taxOf salary taxRates =
((0m,0)::taxRates, taxRates)
||> Seq.zip
|> Seq.map(fun ((_, prevBand),(rate, band)) -> (prevBand, rate, band))
|> Seq.sumBy(fun (prevBand, rate, band) ->
match salary with
| x when x < prevBand -> 0m
| x when x > band -> decimal(band – prevBand) * rate
| x -> decimal(x – prevBand) * rate
)

let israelTaxRates = [
0.10m, 5070;
0.14m, 8660;
0.23m, 14070;
0.30m, 21240;
0.33m, 40230;
0.45m, System.Int32.MaxValue]

let taxOfIsrael salary = israelTaxRates |> taxOf salary

//Usage: taxOfIsrael 5800
[/csharp]

The nice aspects of this solution was having the tax rates and bands in an easily readable and maintainable collection optimized for the reader (like all other F# solutions) separate from its implementation. The internal logic is neatly coupled together into 3 readable scenarios, making it easy to work out how the tax was calculated.

Another nice feature is being able to easily combine the implementation and input tax rates to create a high order taxOfIsrael function that ultimately performs the task. This in-built ability to curry functions makes functional composition a breeze and after using it for a while I can quickly see how it’s more elegant to OOP programming style in a lot of areas.

Interactive Mode

Inside Visual Studio 2010 (or by using¬†fsi on the command line) is F# Interactive mode which works very much like Ruby’s irb or ¬†Pythons interactive mode, it’s just rare to see this from a typed .NET language, although¬†Boo was likely the first to do this.

F# open and F# everywhere!

Despite F# being the latest creation¬†forged in the deep trenches of Microsoft’s R&D department, F# is surprisingly and arguably the most open of all of Microsofts languages with their entire implementation available on GitHub and free to use,¬†released under the liberal and OSS approved Apache 2.0 licence! This wouldn’t mean much if it didn’t immediately work elsewhere however Don Syme and his team have done a good job actively supporting Mono, going as far as reporting blocking Mono bugs, ensuring it continues to work flawlessly. It’s even being currently distributed with the latest release of Mono on OSX.

Being this open is a treat, we can finally build GTK desktop applications with a typed, fast functional language using Open Source components end-to-end!

I’m actually surprised how well it works where I’m doing all current F# development on an OSX Macbook Air, deploying my finished solutions onto¬†http://www.servicestack.net CentOS linux server for hosting.

Getting Started with F#

Although not mentioned here, it should be noted F# is great for concurrent programming tasks as well. It is also particular good at creating composable¬†asynchronous¬†work flows – which happened to be the source of inspiration for C#’s 5.0 async/await feature.

Getting started with F# is easy where if you have Visual Studio 2010 – you’ve already got it! Just create an F# project and your good to go. For all other platforms follow this link.

For the newbie budding developer, I recommend the following learning resources:

Blogs

Next up: Easy Self Hosted, Cross Platform F# Web Services

Although not a surprise for a .NET language, I’m happy to report F# works flawlessly with Service Stack where its thin API doesn’t get in F#’s way – allowing it to easily create elegant¬†self-hosted solutions. In my next¬†instalment¬†I’ll show how you can easily create an async+parallel cached twitter proxy that works cross platform on Windows/OSX and Linux in =~ 100 LOC.

First day on the job, after 1 crazy week!

Wow what a week! it’s been an eventful one to remember: there are Rebellions overthrowing dictatorships in Libya, Earthquakes in Washington with tremors being felt in New York and hurricane Irene swirling around keen¬†on paying us a visit anytime now. All this just setting the stage for the sad news I hoped not to hear for many more years with Mr. Apple (aka Steve Jobs) bowing out on top, relinquishing command as king of the tech world.

Meanwhile all this is happening I’ve undergone some radical lifestyle changes as well with just having just moved to the city of New York after a 2 month hiatus from programming, travelling the wrong way round the world from London. I’m now currently settling into my new apartment in Brooklyn whilst just having finished the first day at my new job after having joined the talented team at: http://stackexchange.com!

My new, New York life is largely thanks to the opportunity I received from the eloquent Joel Spolsky. Which was pretty surreal in itself , given I’ve been following and enjoying his entertaining articles on his world famous blog since the dawn of my computing career >10 years ago!

My first call to action at StackExchange will be helping “Stand Up Matt” @clipperhouse and “Mad Typist”¬†@JasonPunyon (my first impressions – sorry guys! ūüôā ) enhance the new StackOverflow Careers 2.0 site – Something I’ll be looking forward to as It’ll have the potential to help and reach a very large audience.¬†This is in stark contrast to the standard enterprise developer positions I’ve found myself in over the years – developing custom backend solutions that are usually only destined to support a handful of people.

Day 1

With having survived Day 1 (and not having done much), the biggest change I’ve noticed so far¬†is the view. Where back in London I had a quiet desk next to a peaceful, lifeless cemetery where on a winters day I’m greeted with a ghostly calming atmosphere:

In New York that image is a distant memory where sitting next to a window on the 26th floor, neighbouring skyscrapers hit you in the face:

Although this changes very quickly¬†with the bit of rain today, you still get a ghostly picture but instead of snow tipped tombstones – it’s the city of New Jersey!

That’s about it for my first day, I’m told my second will be even better with some in-house Team StackOverflow beer festival on Friday to close off the week!

412 Words

Dark Pastel Visual Studio Theme

Since I’m pretty happy with the Dark Visual Studio Theme I’ve been rolling for a long time with I thought I’d share it with the wider .NET world so others can bask in the glow of an eye friendly dark¬†theme when they’re doing their midnight code runs!

I’ve had it for so long that I’m no longer sure where it was originally based, but looking at Scott Hanslemans VS.NET theme gallery it looks like it’s a mix of Rob Conery’s TextMate and John Lam’s Vibrant Ink.
Over the years where I’ve felt the contrast could be improved, I’ve spiced it up with my own unique blend of¬†colours¬†from the pastel famiglia palette.

And since this theme isn’t complete without a name I’m calling it….

Dark Pastel

Here’s a code screenshot:

C# code screenshot

Some Web.config love:

Web.config screenshot

A splash of HTML:

HTML screenshot

And the all important download.

.

Enjoy!

147 Words

REST Media Types, SOAP, Heavy frameworks and Literate Programming

Obviously the title of this post doesn’t make sense to be all¬†in same article, yet I’ve still managed to carry an underlying theme interlinking the above topics : -). Effectively I’m hoping to encourage the pursuit of end user value over and above all else.

Disadvantages of Custom Mediatypes

Although I’m going to focus on Custom Media Types in this instance (since it reflects my most recent public discussion),¬†the essence of my argument can be applied to many other areas, protocols, libraries, patterns, etc being advocated¬†in software development today.

The main problem I have with Custom Media Types, is the same I have with any other technologies asking me to devote more effort to produce to what I ultimately conceive as a less valueable output. Where the only time when it proves to be beneficial over raw canonical data forms is when the consuming client has built-in support for the custom media format saving the integration point since the client already understands how to process the response.

Although this post may contain a critique on certain¬†facets of REST, I’m by no means advocating against it as overall it’s a far superior more elegant approach to designing services than the SOAP and WS-* stack (which I’m happy to see now starting to slowly¬†fade out of existence). In this light I instead consider myself more a devils advocate providing critique on parts which provide minimal value yet require significant more effort to implement. So consider this a public service announcement as due to the taboo nature of REST you’re unlikely to find many posts like this critiquing it which is partly my motivation for writing it.

There are times when custom mediatypes are useful however what’s not being discussed in these cases (and what I continue¬†to have issue with) is the amount of airtime given to what I consider a small % of applicability and real-world use.¬†This has a tendency to result in a significant body of knowledge flowing around new developers information streams¬†and if they didn’t know any better would think that this is the right way to go about developing web services.¬†What we’re going to end up with are solutions looking for problems and a swath of invented custom formats – more then there deserves to be. Given the unhealthy amount of coverage given to REST frameworks used to flex different parts of REST theory, I fully expect solutions developed just because developers want to obtain full-compliance REST¬†reputation badges implementing non-trivial CONNEG + Custom Media Types multi-hop solutions when far simpler, more direct solutions can elegantly solve the same problem space. Of course I fully expect people to argue that a superior solution can be achieved even with full-compliance of¬†REST HATEOS restrictions, and I simply want to re-iterate for developers to consider, to understand and weigh all options, and that Getting Things Done in the minimum LOC and effort possible is also a worthy goal to strive for, as when we get older, time becomes more¬†precious¬†and being able to hone this skill will ultimately pay dividends later : -)

Web Services should be optimized for Data Formats

In my 10+ years as a backend systems developer, I can count on one hand the number of times when¬†creating custom media types has proven beneficial or was a customer requirement.¬†The majority of my time is¬†spent developing systems with a primary purpose of servicing clients, thick, thin or otherwise.¬†In my web services world view, not all media types are created equal and its the popular data formats XML, JSON, SOAP, CSV, etc ¬†that are ultimately exposed by web services and consumed by its clients.¬†It’s these formats I believe Web Service Frameworks should be optimized for as they provide the most utility, re-usability and value.

As expected my real-world experience is also the primary influence on¬†http://www.servicestack.net where the above data formats are built-in and¬†are configured to work with all your services automatically.¬†Your C#¬†logic remains DRY and is fused with the expected HTTP result providing a free logic block to be able to return any C# POCO or throw any C# Exception. At the same time making all services available on all above data formats. These are the traits I’ve found to be the most productive and ultimately provide a friction-free¬†development¬†experience.

The other notable mediatypes which are actually more prominent than web services data formats are the media types understood by web browsers. By this I mean HTML/CSS/JavaScript /img/vids/etc. The development of web sites have long been the focus for most development platforms where websites have now become the most important deployment target for most new applications. These web formats are first class formats where entire web frameworks are built around and optimized for, which you should be using instead of Web Service Frameworks.

Custom formats are still important, just not of primary importance

Despite my criticisms, I still think the production of custom formats is important when it’s required, and ServiceStack actually¬†has very good support for it with one of the DRY-est API’s you’re likely to see in accomplishing this task.¬†I just don’t personally give prominence to this functionality since it’s not where I believe web service developers should¬†be speding their time in implementing solutions with. Although for the interested this is how ServiceStack implements the¬†VCard Format. Where its non-invasive layered API¬†makes it trivial to support any custom media format that is well supported since its the same integrated pipeline the newer built-in¬†HTML and¬†CSV formats are using.

Note: at the lowest HTTP API level in every web framework (e.g. IHttpHandler, HttpServletRequest, etc) this has always been possible, so all web service frameworks are showcasing its high-level API to achieve this end. Some frameworks make this functionality available using APIs mapping directly to REST concepts, ServiceStack however prefers to make it as DRY, typed and expressive as possible.

Data formats the preferred choice in SOA systems

A key objective in¬†SOA systems and web services today is to servicify and¬†free your data making it as¬†universally available and re-usable as possible. Given these objectives it is common to see SOA systems today¬†make use of HTTP for their transport protocol and utilizing either XML or JSON for their data formats.¬†This is largely because the data formats contain the essence of your data where should they need to at a later date,¬†your clients can use this canonical form of your data and create the custom media types themselves.¬†The same is not true the other way around where if the only format your service provided were custom media formats (e.g.¬†the VCard format) it not only takes much more effort to extract the raw data, but the specific media type¬†will only contain the subset of your contact data as supported by the format. So if your developers only have an option to deliver 1 format for your resource, you’re best to make it a raw data format since the development of other formats can still be achieved by your clients long after the developers have moved on.

In summary you should only be accessing the specific VCard format (or other specific mediatype) only if your client already has built-in support for it. Since it requires more effort on the client and server to implement and access otherwise.

It is therefore important to remain vigilant and keep the¬†80/20 Rule in mind (or in the case of custom media formats the 95/5 rule : -) and treat custom media types as complimentary formats that you should only add¬†in addition to your data formats. When development of your SOA services have finished¬†you want to ensure it’s data is as accessible and re-usable as possible in the formats that provide the most¬†utility to your current and future consumers of your services, i.e. the data formats.

The guiding value light

With the current demise of SOAP,¬†REST is making it’s way back into the forefront of IT minds.
The value of REST has always been available in HTTP/1.1 since its inception but between the marketing and committees¬†pushing their own agendas, the Enterprise development world was largely oblivious to its possibilities as they were taught¬†if your data didn’t come in a XML decorated envelope it’s not a real web service.

Its come-back can largely be attributed to developers and companies ignoring the blind advice of big iron Company’s¬†and instead discovered that they are better able to provide value by embracing the inherent properties in HTTP¬†rather than the friction-encumbered dev experience of building SOAP web services.

Displacing big iron-backed monolithic standards is hard, it takes courageous companies willing to discard current enterprise teachings who are able to re-think the essence of what¬†they’re trying to achieve.

Focusing on max value and min effort

It’s therefore important we don’t repeat the same mistakes and don’t lose sight of what the value proposition on¬†the technologies we ultimately choose to adopt, can provide.¬†I’m constantly seeing blind embracing of software process, approach or technology where at the most extreme levels its¬†treated akin to a religion, instead of what it should be: productive tools serving a means to an end.

HATEOS

REST at its extreme conformance imposes some hefty HATEOS restrictions which coupled with most Hyper media driven applications fixation on XML, does not leave you with a very productive window¬†to develop in. If you see real-world value in conforming to these restrictions then by all means adopt them, I personally¬†don’t, and since collecting stamps of approvals from the REST community is not my #1 priority, I routinely discard they exist.

JSON vs XML

XML is another one of those funny technologies handed down from standard committees as the holy grail data format. ¬†Unfortunately this only holds true when you don’t have to develop with it, as it’s a poor programattic fit, making it harder to work with¬†than JSON, that’s both more verbose and slower as well.¬†A good reference that explores these differences in more detail is Douglas Crockford’s entertaining talk on¬†Heresy & Heretical Open Source.

XML as a interoperable structured data format has its place, at the same time I’m finding cases where it would be the ideal data format¬†harder to find. I no longer store it in databases, make use of it for efficient messaging or let it go near my mobile or web apps.

Choosing the right technology

The way I defend myself from adopting the wrong technology is to take a cynical view against all promises, hype and marketing materials.¬†I usually form a picture of the easiest way to develop something and use that as a benchmark to compare other solutions against.¬†The onus then becomes on other frameworks to prove their value by either reducing the amount of effort or code required or¬†in someway enhancing re-usability, performance or promote better maintainability by reducing coupling. If it does not provide any realizable real-world value I don’t spend too much time researching it and continue what I was doing before this Hot Tech Tip landed on my lap.

Basically don’t take any advice at face value, understand the benefits and amount of effort to achieve the desired outcome¬†of each approach and choose the one that offers the maximum end user utility given the least effort to implement.¬†If you don’t understand the full value that your architecture/approach/framework is providing, you won’t be able to¬†maximize its benefits and will soon find you’ve inadvertently subscribed to a solution looking for a problem.

So if you’re spending a lot of time fighting with your framework just to try to get it to prescribe to some artificial standard or religion¬†it might be time to explore other approaches/technologies. When evaluating technologies think about the concept of¬†Last principles where you first look at the ultimate end-user result you want to deliver, then look backwards at the tools,¬†technoligies and frameworks that will best meet this end with the least effort and maximum efficiency.

Heavy weight libraries and frameworks

I personally believe that the Java/.NET worlds are plagued with leaky, heavy abstractions that introduce more¬†friction an inertia that they’re trying to solve. The pain in configuring these monstrosities can be attributed to:

“So much complexity in software comes from trying to make one thing do two things.”

— Ryan Singer

Whilst the un-intended, unknown or magic behaviour I frequently see at runtime whilst developing in said frameworks I’m attributing to its poor, leaky abstractions.

Prefer thin abstractions promoting DRY code

In recent years I’ve been more active in proactively shunning heavy frameworks and look to libraries allowing¬†me to write less code and configuration. Configuration is actually a form of inferior code which happens to be more awkward¬†to define object graphs with than your general purpose programming language. What’s left is a fragile, un-typed mess that you¬†can usually only determine correctness at runtime.

I have many reasons to prefer shorter code-bases which follow my general mantra of¬†“You never have to maintain¬†any line of code you didn’t write”.¬†However here a couple of my favourite quotes I’ve collected on the subject that put this more eloquently than I:

“I happen to hold a hard-won minority opinion about code bases. In particular I believe, quite staunchly I might add,¬†that the worst thing that can happen to a code base is size.”

Steve Yegge

“A programmer knows he has achieved perfection not when there is nothing left to add,¬†but when there is nothing left to take away.”

— Antoine St Exupery

With that said, my general preference is for light-weight development libraries and frameworks with thin abstractions allowing¬†me to capture my logic’s intent as DRY as possible. Usually this means these libraries have a low artificial construct / concept count¬†whilst remaining expressive at its core.

The rise of Ruby and Literate Programming

Despite its poor runtime performance and no sight of any corporate backing, Ruby has been making deep inroads into developers mindshare¬†simply because it allows developers to be more productive. Although the language itself is quite expressive, I’m attributing¬†it’s large productivity gains on the communities focus on Simplicity, Convention, DRY-ness and it’s ultimate goal of its¬†ability to compress logic into pseudo code that runs.
In this goal, it seems either by accident or in a way that was cleverly conceived by the languages author it appears that¬†much of Rubys libraries follow a deep symmetry to Donald Knuth’s¬†Literate programming which enables programmers to develop programs in the order demanded by the logic and flow of their thoughts.

Code as literature

“Meaning belongs to a text or to a program to the extent that it acts upon intelligence in a particular way.”

We say that a textual work is “literary” if it contains a high density of meaning.

The above quotes are from @jashkenas enlightening¬†talk at Ruby’s SunnyConf 2010 exploring the subject.

Literate programming is an intriguing topic that not only looks to enhance developer productivity but at the same time¬†increases developer enjoyment and satisfaction. I can definitely see its appeal as it seems like a breadth of fresh air¬†especially if you spend your time fighting heavy frameworks in your day job.¬†Having said that Ruby’s lack of good performance characteristics¬†which I consider important keeps me from using the language in any capacity.

JavaScript’s star new language – CoffeeScript

Luckily CoffeeScript is a hot new language that looks to embrace this vision and since its built on JavaScript, can take advantage of the arms race going on amongst browser vendors in recent years trying to make JavaScript as fast as possible.

Unlike other JavaScript frameworks tackling this domain, it’s not interpreted at runtime and simply compiles 1:1 with¬†efficient JavaScript (in some cases more efficient than a developer would write).

This gives it some unique qualities where it is able to run in a browser without any plugins and thanks to node.js the same CoffeeScript code can run very efficiently on servers as well.

We’re now starting to see this powerfull and expressive combination powering new servers like Ruby’s new¬†Pow Rack server for OSX.
Its recent inclusion in the default Rails install is causing the biggest rift I recall in recent times for a single line of config change.

So while everyone may not be in love with it yet, they at least have an opinion on it ūüôā

What’s yours?

2,764 Words

The App Store bubble: How much does an App earn today?

The rise and fall of Internet riches

Within the last 15 years on the rise of the Internet we have seen many a phenomena gracing technology circles promising great riches to early adopters and investors. It first started out with a gold rush land grab for domain names where in this era a 3-letter domain name like sex.com was able to earn millions a year just by providing a holding page hosting adult-related advertising banners and was eventually sold for 14M.  Then came the dotcom boom where anything Internet related were the most sought after prize on the stock market where traffic and the size of your user base was the hard currency of the day and was rewarded with over-inflated valuations never mind that many had no visible path to profitability. The preferred sport of many startups at the time was who had the highest burn-rate which were fueled by expensive Aeron chairs and Sparc servers.

The Bigger They Come, The Harder They Fall

But just as quick as the riches came to dotcom companies, they vanished again almost entirely within one weekend (which this author remembers quite well as an eager investor on the friday before black monday). Eventually the market corrected itself and only businesses with viable business models were the focus of cautious investors whose fingers were still smarting from the bursting of their dotcom investments.

In time faith was restored in the online advertising market spurred largely by Google’s foray into the area of targeted advertising. Confidence was restored in Internet companies and a smaller bubble is starting to emerge that many casual observers are labelling Web 2.0.

Cometh the App Store, Cometh the riches again

Out of the all the events to show up on the technology radar recently, none has been as exciting and show as much promise as the introduction of the App Store by Apple. With the introduction of their revolutionary smart phone, Apple for the first time put the power of a desktop operating system in the palms of technophiles (with above-average discretionary incomes). Although its potential wasn’t realized until Apple put the first version of the iPhone SDK in developers hands and opened the much publicized App Store on July 10, 2008. Within months of its introduction stories or riches started to be told of hobbyist programmers earning $600,000 in just 1-month enough to quit their full-time job to pursue a lucrative career in developing mobile apps for the App store. Soon enough a new economy was created and in pretty short order a growing number of developers made it to the App store millionaire club. This brings us to where we are today where as of September 1, 2010 there were 250,000 third-party applications with over 6.5 billion total downloads generating its developers over USD $1 billion in revenue.

The reports of App Stores death are greatly exaggerated

With now over 250,000 applications available for sale and many of them free many people have chimed in with their opinions some proclaiming the app bubble is bursting and the path to easy riches is over while others maintain the App store maintains a fruitful future. Unfortunately the ubiquity of the Internet has made opinions cheap and plentiful where it looks like anyone with a keyboard can pluck 2 numbers out of the air to show a downward trend.

An App Store Developers story…

As a developer on the App Store I wanted to weigh in on the state of the App store with some empirical evidence of my own as it evolves. My story dates back 18 months ago when it took me 3 weeks to learn enough Objective-C to submit my first app. Within a couple of weeks, inspired by the success of iFart apps I put aside my morals and pursuit of software excellence and ended up submitting my own cookie cutter apps. Although my own fart apps didn’t do nearly as well as notable others my translator app earned a noticeable side income where it managed to pay off its development effort many times over. Unfortunately around this time I joined my first startup and between the all nighters and 12 hour work days my free time was sapped and my once App making machine days were over. That is until my recent resignation which saw the return of my free time and I was finally able to open xcode again after an 18 month hiatus. After tying up some loose ends on my open source project I got to work on a new App.

How much does an App built today earn?

So that Non-App developers can get an idea of what they can expect when developing an App today, I’m going to be completely open and transparent about the development and the financial results of my latest App that has just been approved today (14th Sept UK). I will keep this blog updated with live financial results as I get them.

Coming up with an idea for an App

Developing a successful app in this age can be a tricky balance to get right where the subject and appeal of your app is every much as important as its features and quality. Because of this, I’m not a true believer of investing a lot of time in developing an App before I see some ROI. So I’m always on the lookout for small, quick wins with short development cycles. It just so happen that at the time I was seeking inspiration for my new app, I also spent a large portion of my free time playing Words with friends – an enjoyable, network-enabled SCRABBLE¬ģ clone. Although many would see my poor scrabbling skills the result of a long absence from participating in any academic curriculum, I saw it as an opportunity to build an App to improve my scrabble game! (read: cheat) After a quick sanity check to make sure there weren’t a hundred free apps already available on the App store that did the same thing: Word Mate was born – a collection of word finding utilities that help you play SCRABBLE¬ģ like a computer ūüôā

Development efforts and total cost

Word Mate took me roughly a week full-time to develop which included downloading all the latest SDK’s and tools, a bit of reading designed to re-ignite my love with the Objective-C language and its Cocoa inspired frameworks as well as creating all the App and website graphics. As the website is freely hosted on Google App Engine and the only 3rd party icons used were free, no costs were incurred into making this app other than the $99/year iOS Developer Program fee and my own time.

App Store submission process

Although the language and tools haven’t improved much, the app submission and code-signing process has improved significantly. When it was first released, code-signing your app so it can be submitted to the App store was considered a black-art which thanks to my careful literal following of every word in an online tutorial caused me to be stuck with the embarrassing title of ‘iPhone Developer: Demis Bellot’ for the name of my developer profile. Anyway I submitted my app on the 1st September and it was only approved at midnight on the 13th September. Where the QA staff that Mr Jobs says approves most apps within 7 days are? I don’t know – I’ve never had an any of my apps approved within 2 weeks.

Regardless today will be my first full day on the App Store market. Like all my apps, Word Mate will be free for the first week. I like to do this as it gives a chance for my friends and anyone else who wants one a chance to download my app for free. Making an App for free and then charging for it will also give you an idea of the Free vs Premium download ratio. From the experience of my first app the difference was about 6-9% where for every 100 free apps that were downloaded (when it was free) only 6-9 were paid for (when I started charging USD $0.99 for it). It will be interesting to see if that trends continues with Word Mate.

Live Financial Results

I’m going to maintain a live list of results on the table below, so you can check back on this post to see how it develops. iTunes Connect (where we login to get our sales reports) doesn’t publish daily results until 12pm GMT the next day. You can follow @demisbellot on twitter to find out as soon as the latest results have been added.

Period Ending Cost Qty Sold Earnings
14th Sept $0.00 118 $0.00
15th Sept $0.00 118 $0.00
16th Sept $0.00 41 $0.00
17th Sept $0.00 28 $0.00
18th Sept $0.00 24 $0.00
19th Sept $0.00 21 $0.00
20th Sept $0.99 15 free/1 $0.99
21st Sept $0.99 1 $0.99
22nd Sept $0.99 0 $0.00
23rd Sept $0.99 2 $1.98
20-26th Sept $0.99 4 $3.96
27 Sept-3rd Oct $0.99 5 $4.95
4th-10th Oct $0.99 2 $1.98
11th-17th Oct $0.99 3 $2.97
18th-24th Oct $0.99 2 $1.98
25th-31st Oct $0.99 5 $4.95
1st-28th Nov $0.99 5 $4.95

*NOTE: Word Mate will be free until 20 Sept 2010, so if you think you might find this app useful now would be a good time to get it for free ūüôā

Follow the conversation

A discussion thread has started in hacker news which you can get involved in at:

SCRABBLE¬ģ is a registered trademark of Hasbro, Inc. in the United States and Canada, and Mattel, inc. elsewhere. I’m not affiliated with either company.

Update 23/09/210

It’s just come to my attention that I may have stuffed up the SEO of my app in the App Store as Word Mate can’t be found with the words ‘Word Finder’ or ‘Word Cheat’  2 of the most popular searches for finding like apps. Unfortunately Apple wont let you change it after you’ve added it so I’m going to have to submit a new app. Unfortunately SEO is just as important as having a quality app as if users can’t find it, they can’t buy it.